July 2008 July 2008
Data Services: Develop Robust and Scalable Apps with SQL Server Data Services
Here the author introduces SQL Server Data Services, which exposes its functionality over standard Web service interfaces. David Robinson
ADO.NET: Achieve Flexible Data Modeling with the Entity Framework
Here the author answers questions regarding the Entity Framework and provides an understanding of how and why it was developed. Elisa Flasko
Data and WPF: Customize Data Display with Data Binding and WPF
Here we present techniques for programmatic and declarative data binding and display with Windows Presentation Foundation. Josh Smith
Transactions: Build Scalable Systems That Handle Failure Without Losing Data
Systems that handle failure without losing data are elusive. Learn how to achieve systems that are both scalable and robust. Udi Dahan
WCF P2P: How To Design State Sharing in a Peer Network
Learn how you can peer-enable business applications by allowing them to share state in a serverless peer network. Kevin Hoffman
Columns
Editor's Note: Flex Your Data: Filling in for Editor-in-Chief Howard Dierking, Ted Neward lends some insight into the state of data collection and manipulation.Ted Neward | Toolbox: Easy Wiki Hosting, Scott Hanselman’s blog, and Snagging Screens: Host a wiki the easy way, get the screen shots you need and mark them up, and read about LINQ this month in Toolbox.Scott Mitchell | CLR Inside Out: Extend Windows Forms Apps Using System.AddIn: See how Windows Forms applications can be adapted to use the new .NET Add-in framework (System.AddIn) this month.Mueez Siddiqui
Data Points: The Entity Framework in Layered Architectures: Here we build a solution that fits the Entity Framework into an n-tier architecture that uses WCF and WPF and the MVP pattern.John Papa | Advanced Basics: The LINQ Enumerable Class, Part 1.: Presented here is the LINQ Enumerable class, which allows you to manipulate data in any class that implements IEnumerable(Of T).Ken Getz | Office Space: Automated SharePoint Site Branding: Here's a custom branding solution for SharePoint sites that integrates Master Pages and CSS files at the level of the site collection.Ted Pattison
Cutting Edge: The HTML Message Pattern: In this installment, the author provides an enhanced implementation of the BST pattern and compares it to HTM solutions.Dino Esposito | Test Run: Introduction to WCF Testing: This month James McCaffrey builds a test harness for WCF applications that really puts them through the paces.Dr. James McCaffrey | Security Briefs: Reinvigorate your Threat Modeling Process: In this column the author outlines some approaches to threat modeling that can be employed by development teams of any size.Adam Shostack
Foundations: Adding Code Access Security to WCF, Part 2: This month's column continues the discussion around code access security in WCF and partially trusted services.Juval Lowy | .NET Matters: Asynchronous I/O with WebClient: In this month’s installment of .NET Matters, columnist Stephen Toub answers reader questions concerning asynchronous I/O .Stephen Toub | { End Bracket }: Customize Windows Home Server: Chris Gray discusses custom applications he’s developed for Windows Home Server and explores applications for energy savings.Chris Gray


Data Services
Develop Robust and Scalable Apps with SQL Server Data Services
David Robinson

This article discusses:
  • The SSDS data model
  • Managing entities, containers, and authorities
  • Creating a sample Web application
  • Class serialization and deserialization
This article uses the following technologies:
SQL Server
This column is based on a prerelease version of SQL Server Data Services. All information herein is subject to change.
Contents
SSDS Data Model
Building a Classified Ads System
Adding a City
Adding Categories
Updating and Deleting Entities
Adding and Deleting Listing Schemas
The Classifieds Web App
Class Deserialization
Using the Custom Listing Schema.
Looking Forward
The Microsoft data platform is made up of a rich set of products and technologies for all types of data, from unstructured binary data on one end of the spectrum to highly structured online analytical processing (OLAP) cubes on the other. While these technologies address any application scenarios you may have, typically it involves an up-front investment of time and hardware resources before you are able to start coding your solution.
It is often difficult for application developers to estimate the storage and processing requirements that an application will require over its lifecycle. This can be particularly acute when you consider a startup company where funds are tight and the data used to make hardware investment decisions is scarce. To address concerns about these up-front investments, Microsoft has recently added a new technology to its already robust data platform: SQL Server® Data Services (SSDS).
Not a product in the traditional boxed-software terms, SSDS is a robust, scale-free data service that internally uses proven SQL Server technology and exposes its functionality over industry standard Web service interfaces. SSDS provides an easy-to-use, flexible data model that you access via open, industry standard protocols.
Regardless of whether you are developing with the Microsoft® .NET Framework, Java, or other technologies, you can employ SSDS as your data store. In addition, SSDS is designed to allow developers to provision an account quickly and start developing against it immediately.
SSDS provides developers with a data back end that is not constrained by available drive bays or rack space. By using SSDS as your data platform, your application is free to consume as much data as you need. Regardless of whether it's a gigabyte or a petabyte of storage, you will only pay for the resources you consume.
Many Web sites and apps have cyclical access patterns where you need enough capacity for peak loads, even if those loads aren't sustained. By using SSDS as your data back end, you are free to concentrate on your app, not on data platform capacity planning.
In this article I will introduce you to the basics of developing a data solution around SSDS. I will begin by mapping out the data model that is employed by SSDS. Then I will dive into the development details by showing you how I built the simple online classified ads system, shown in Figure 1, using SSDS.
Figure 1 Sample Classifieds App Hosted by SSDS

SSDS Data Model
SSDS provides a flexible entity-based data model. This model has three main elements—authority, container, and entity—referred to as the ACE concepts (see Figure 2 ). An SSDS authority can be related to a database in the relational world. When you provision an authority, SSDS will create a DNS name for you to access the authority. For instance, if you provision an authority called ssdsdemo, you will access it at the following URI:
ssdsdemo.data.beta.mssds.com
Figure 2Basic SSDS Components
An authority is also the unit of geo-location, meaning that the created DNS name will map to a specific Microsoft datacenter where your data is hosted. If you create two authorities—americasdemo.data.beta.mssds.com and europedemo.data.beta.mssds.com—each one will be associated with a different datacenter that is located closest to your users.
An authority contains a collection of containers—a container is similar to a table in a relational database. The key difference is that you attach a schema to a database table to make all rows in the table homogeneous. A container in SSDS does not require a schema, which allows you to store heterogeneous entities in one convenient location. It is simply a collection of entities. In the current release of SSDS, all queries are scoped to a single container.
One other important point to make here is that each container is placed on a different node in the SSDS cluster. In SQL Server, to gain additional performance, you would place different tables on separate spindles to maximize your read/write capabilities. This is similar in SSDS except that instead each container will be on a separate machine. Additional performance gains can be realized by partitioning your data across multiple containers and multithreading your requests since your reads and writes won't be constrained by a single machine.
A final note about containers is that while each container is placed on a separate node in the SSDS cluster, the data is also replicated on several other nodes for disaster recovery purposes. If the machine your container is on fails, one of the backup replicas are promoted automatically to ensure your applications doesn't observe any data or performance loss.
An entity can be compared to a row in a table in a relational database. An entity is simply a property bag of name/value pairs. These name/value pairs are grouped into two categories: distinguished system properties and flexible properties.
Distinguished system properties are common to all entities and include ID, Kind, and Version. ID uniquely identifies the entity. IDs must be unique within the container it exists in, but different containers can have entities with the same ID. Kind is used to categorize similar entities together. There is no schema attached to the entity, so having entities of the same Kind does not guarantee the same structure, and Version is used to identify the current version of the entity. This value gets updated on each operation.
Flexible properties are where the developer stores the application data. Flexible properties support simple types: string, decimal, bool, datetime, and binary. Each flexible property is indexed up to the first 256 bytes.

Building a Classified Ads System
To demonstrate the features of SSDS and to give an overview of the development experience, I will walk you through the implementation of a sample online classifieds system, Contoso Classifieds. The sample comprises two separate applications: a Windows® Forms application to administer the system and an ASP.NET application—the main Contoso Classifieds site—that users would visit. The application accesses SSDS via a SOAP interface in the Windows Forms application and through the Representational State Transfer (REST) interface within the ASP.NET application.
When you sign up for a SSDS account, you will receive a user name and password. From there it is up to you to begin building your authorities, containers, and entities. For Contoso Classifieds, I set up the authority as contosoclassifieds and the service URI endpoint in the beta release of SSDS is data.data.beta.mssds.com.
The contosoclassifieds authority includes two containers: Categories and Cities. Categories will contain the entities related to the Listing categories. Cities will contain an entity for each city defined in the system. Those entities will simply be pointers to city-specific authorities. Each city-specific authority will have a container per listing category (see Figure 3).
Figure 3 Contoso Classifieds Elements
There are a few reasons I went with this design. I chose to implement on authority per city so that I can ensure the authority is provisioned in a datacenter that is geographically close to the population it will serve. I chose one container per listing category so that I have a simple query pattern and I can spread the load across multiple machines in the cluster (remember a container is scoped to a specific node in the back-end cluster). Figure 4 shows the user interface for the Contoso Classifieds Administration client.
Figure 4 Classifieds App Administrative Client
As previously mentioned, it is up to the developer to create any authorities, containers, and entities the application will use. In the sample app, all this work is contained within the click event for the Perform Initial Setup button. (The full code, including error handling, is included in the code download for this article.)
Provisioning an SSDS authority is extremely easy. In this example code I use the SitkaSoapServiceClient, for which I had previously added a service reference to the SSDS SOAP interface:
using (SSDSClient.SitkaSoapServiceClient ssdsProxy =
  new SSDSClient.SitkaSoapServiceClient())
You have already seen an Authority, but what is an SSDS scope? The scope object is used within SSDS to provide a way of addressing objects in the SOAP service similar to the way URIs are used in the REST service.
The first thing you should do is set the user name and password for the service:
ssdsProxy.ClientCredentials.UserName.UserName =
  txtUserName.Text;
ssdsProxy.ClientCredentials.UserName.Password =
  txtPassword.Text;
You need to create an authority, which is at the highest level of the ACE model, so you first create an empty scope:
SSDSClient.Scope serviceScope = new SSDSClient.Scope();
Now create the Contoso Classifieds Authority. This will hold general configuration information in the system
SSDSClient.Authority contosoAuth =
  new SSDSClient.Authority();
contosoAuth.Id = "contosoclassifieds";
and submit the creation to SSDS
ssdsProxy.Create(serviceScope, contosoAuth);
Now that the main Contoso Classifieds Authority has been created, point your scope to it, then create the container to hold the cities you will be serving classified ads to, and submit the container creation to SSDS:
serviceScope.AuthorityId = contosoAuth.Id;

SSDSClient.Container citiesContainer = new SSDSClient.Container();
citiesContainer.Id = "Cities";

ssdsProxy.Create(serviceScope, citiesContainer);
The process to create a container is similar to the process of creating an authority. The only difference is that you create a Container object instead of an Authority object and, instead of an empty scope, you update the scope to point to the authority in which the container is to be created.
Create the container that will hold all the Header and Listing categories you will support within the application:
SSDSClient.Container categoriesContainer =
  new SSDSClient.Container();
categoriesContainer.Id = "Categories";

ssdsProxy.Create(serviceScope, categoriesContainer);
Once the authority is created you can point a Web browser to it, using either HTTP or HTTPS, and you can view the container's contents using the REST interface. Simply use the new DNS name that SSDS created when setting up the authority. In this case, the URL would be:
http://contosoclassifieds.data.data.beta.mssds.com/v1
After this URL is entered into the browser, you will be prompted to authenticate. Once you successfully authenticate, you will see something like this in your browser:
<s:Authority xmlns:s="http://schemas.microsoft.com/sitka/2008/03/"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:x="http://www.w3.org/2001/XMLSchema">
  <s:Id>contosoclassifieds</s:Id>
  <s:Version>11</s:Id>
</s:Authority>
The authority name is lower-cased because since SSDS creates a DNS entry for the authority, you are required to follow DNS naming conventions.
After creating the container, if you refresh the browser and add an empty query to your URL in the form of ?q="", you will see a single Container object being returned in the EntitySet. An EntitySet is simply a collection of entities that are returned as the response to a query. Upon completion of initial setup the sample application will have created one authority (contosoclassifieds) and two containers (Categories and Cities):
<s:EntitySet xmlns:s="http://schemas.microsoft.com/sitka/2008/03/"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:x="http://www.w3.org/2001/XMLSchema">
  <s:Container>
    <s:Id>Categories</s:Id>
    <s:Version>1</s:Id>
  </s:Container>
  <s:Container>
    <s:Id>Cities</s:Id>
    <s:Version>1</s:Id>
  </s:Container>
</s:EntitySet>
Now that the initial setup is complete, let's move ahead and add the cities-served functionality.

Adding a City
As mentioned earlier, Contoso Classifieds places each city's listings in its own authority, which enables you to choose the datacenter it is hosted in. (While that feature is not currently supported in the beta of SSDS, it will be supported by the time the product is released.)
The code for adding the city is similar to the code used for the initial setup. Using the SOAP proxy, set the credentials, create an empty scope, set the authority ID, and issue the create:
SSDSClient.Authority cityAuth =
  new SSDSClient.Authority();
cityAuth.Id = cityAuthorityName;

ssdsProxy.Create(serviceScope, cityAuth);
Now that the new city authority is created, add a pointer entity to the Cities container in the main contosoclassifieds authority:
serviceScope.AuthorityId = "contosoclassifieds";
serviceScope.ContainerId = "Cities";

//Create the City Entity, and set its properties appropriately
SSDSClient.Entity cityEntity = new SSDSClient.Entity();
cityEntity.Id = cityAuthorityName;
cityEntity.Kind = "CityServed";
cityEntity.Properties = new Dictionary<string, object>();
cityEntity.Properties["AuthorityUri"] =
  string.Format("{0}.data.data.beta.mssds.com/v1/", cityAuthorityName);
cityEntity.Properties["Name"] = txtCity.Text;

//Issue the create to SSDS
ssdsProxy.Create(serviceScope, cityEntity);
Let's recap for a moment. I've created a main contosoclassifieds authority that holds all of the systemwide data. In that authority, there is a Cities container that has an entity for each city the application will serve. That entity contains the display name of the city, along with a pointer to the authority that contains all the listings.

Adding Categories
Up to this point, you've just added authorities, containers, and entities. The next step is to implement the listing categories feature. To do so involves using the query, update, and delete functionality of SSDS.
Contoso Classifieds will support a category header, with sub categories below it to hold the actual postings from users. If you were to use a relational model for this data, you would typically use a header/line pattern for the data. The flexible entity data model used by SSDS, however, allows your data to take whatever shape you like. In the case of the Contoso Classifieds application, I chose to use a single entity to represent each category header and all the subcategories that belong to it (see Figure 5).
Figure 5 Entity Data Model Used by the App
Please note that I am using this pattern to simply show the flexible nature of an entity and to illustrate that, although multiple entities can have the same Kind, that does not force the entities to be homogeneous. The entity can have whatever shape you would like.
A better approach to this scenario might be to have two different entity Kinds, Category and Listing Category, and to have a flex property of CategoryID in all entities. That would allow you to issue a query such as:
from e in entities where e["CategoryID"] ==    "For Sale" select e
Notice I say flex property. As previously mentioned, entities in a container must have unique IDs that will limit you from using the distinguished system property ID. Also notice the query syntax I have issued. SSDS uses a LINQ-like query syntax, which should look familiar to most developers who use the .NET Framework.
If you look back at the administrative application shown in Figure 4, you will notice a tree view that represents the listing categories and two textboxes for adding category headers and listing categories. Let's dive into the implementation behind adding a category header.
As with the previous code, you use the SitkaSOAPServiceClient, set the credentials, and set the scope to the contosoclassifieds authority and the Categories container. Now all you need to do is create the entity:
SSDSClient.Scope serviceScope = new SSDSClient.Scope();

serviceScope.AuthorityId = "contosoclassifieds";
serviceScope.ContainerId = "Categories";

SSDSClient.Entity categoryHeaderEntity = new SSDSClient.Entity();
categoryHeaderEntity.Id = CategoryID;
categoryHeaderEntity.Kind = "Category";
You can also add a single flex property that will contain the category name. Notice that this only adds a single property to the flex properties collection:
categoryHeaderEntity.Properties =
  new Dictionary<string, object>();
categoryHeaderEntity.Properties["CategoryName"] = CategoryName;
The next step is to add the listing categories. Since Contoso Classifieds uses a single entity to represent the entire collection of listing categories within a category header, you need to retrieve the category entity, add the additional listing categories (which will be added as additional flex properties), and submit the update to SSDS.
To retrieve the category entity, set the scope to point directly to it and call the Get method, which will retrieve the entity directly without having to issue a query:
SSDSClient.Scope serviceScope = new SSDSClient.Scope();
serviceScope.AuthorityId = "contosoclassifieds";
serviceScope.ContainerId = "Categories";
serviceScope.EntityId = CategoryID;
//Retrieve the Category Entity
SSDSClient.Entity categoryEntity = ssdsProxy.Get(serviceScope);
Now just determine the index of the new property, add it as an additional flex property, and issue the update (see Figure 6).
Figure 6 Adding Flex Properties
//Determine whether this is the first listing
//category being added to this header
if (categoryEntity.Properties.Count > 2) {
  propCount = ((categoryEntity.Properties.Count - 1) / 2);
  propCount++;
}
else {
  propCount = 1;
}

//Create the FlexProperties for the new listing category
string listingIdPropName = string.Format  ("ListingCategoryID{0}", propCount);
string listingNamePropName = string.Format  ("ListingCategoryName{0}", propCount);

categoryEntity.Properties[listingIdPropName] = ListingCategoryID;
categoryEntity.Properties[listingNamePropName] = ListingCategoryName;

//Issue the update to SSDS
ssdsProxy.Update(serviceScope, categoryEntity);

Updating and Deleting Entities
Updating listing categories is as simple as adding an additional flex property. For each update, you are required to retrieve the entity, make any updates locally and submit the update, passing the updated entity along with the scope to the Update method.
Begin by setting the service scope to the main contosoclassifieds authority. Also point it to the Categories container and the entity for the category you are updating:
SSDSClient.Scope serviceScope = new SSDSClient.Scope();
serviceScope.AuthorityId = "contosoclassifieds";
serviceScope.ContainerId = "Categories";
serviceScope.EntityId = currentHdrNode.Name;
Get the entity you are going to be updating:
SSDSClient.Entity categoryEntity = ssdsProxy.Get(serviceScope);
Now you can reinitialize the flex properties collection, re-adding all the flex properties for the listing categories. Alternatively, you could have looped through each flex property and made the updates, but due to the simple structure of this entity, it's easier to just recreate the entity as shown in Figure 7.
Figure 7 Updating Properties
categoryEntity.Properties = new Dictionary<string, object>();
categoryEntity.Properties["CategoryName"] = currentHdrNode.Text;

int propCount = 1;

if (e.Node.Parent != null) {
  //Loop through each node and add its category ID and
  //category name flex property.
  foreach (TreeNode node in e.Node.Parent.Nodes) {
    string listingIdPropName = string.Format      ("ListingCategoryID{0}", propCount);
    string listingNamePropName = string.Format      ("ListingCategoryName{0}", propCount);

    categoryEntity.Properties[listingIdPropName] = node.Name;
    categoryEntity.Properties[listingNamePropName] = node.Text;

    //if we are re-adding the updated category, the tree view
    //will still have its old value, so set the flex property
    //to the updated value
    if (node.Name == e.Node.Name) {
      categoryEntity.Properties[listingIdPropName] =         m_OldSelectNode.Name;
      categoryEntity.Properties[listingNamePropName] = e.Label;
    }

    propCount++;
  }
}

//Submit the update to SSDS
ssdsProxy.Update(serviceScope, categoryEntity);
To delete an entity or a container, point ServiceScope to the item and call the Delete method of SitkaSoapServiceClient. If it is a Header, delete the entire entity:
if (m_OldSelectNode.Parent == null) {
  ssdsProxy.Delete(serviceScope);
}
For other entities, just as you did with the update, recreate the flex property collection and rebuild, performing the add for all nodes except the deleted node:
SSDSClient.Entity categoryEntity = ssdsProxy.Get(serviceScope);
categoryEntity.Properties = new Dictionary<string, object>();
categoryEntity.Properties["CategoryName"] = CategoryName;

int propCount = 1;

foreach (TreeNode node in entityNode.Nodes) {
  string listingIdPropName = string.Format    ("ListingCategoryID{0}", propCount);
  string listingNamePropName = string.Format    ("ListingCategoryName{0}", propCount);

  if (node.Text != m_OldSelectNode.Text) {
    categoryEntity.Properties[listingIdPropName] = node.Name;
    categoryEntity.Properties[listingNamePropName] = node.Text;

    propCount++;
  }
}
Finally, you will issue the update to SSDS and remove the node from the tree view:
ssdsProxy.Update(serviceScope, categoryEntity);

tvCategories.Nodes.Remove(m_OldSelectNode);

Adding and Deleting Listing Schemas
The current beta of SSDS does not support schemas. While there are plans to add schema support in the future, adding schemas yourself within SSDS is an easy task. The only caveat is that you need to manage the schema from within your application because, at this time, SSDS will not enforce it.
For Contoso Classifieds, it seemed like a good idea for the administrator to define a custom schema for certain listing categories. You can define a schema inside the administrative client and use it later on when implementing the Web site.
Up to this point there are no heterogeneous entities inside the same container. Since each entity has a Kind property, you can use this to store different entity types within the same container and also to query for entities of a specific type or, in this case, Kind. While there is no schema contract associated with Kind, it is easy to use this mechanism to group similar data. Now I will add a new entity of Kind ListingSchema to the Categories container.
The dialog for adding a listing schema is quite simple, as shown in Figure 8. It allows the administrator to define additional fields that are associated with a listing and to identify certain fields as being required. I will go into more detail about this later when I show you how to implement the actual Web site.
Figure 8 Schema Customization for a Listing
The process used to add the ListingSchema entity is the same one you've already seen, but this time using a different entity (Kind). The code for the adding the ListingSchema is shown in Figure 9. After adding a listing schema for a For Sale Cars category, notice that the contents of the Categories container include two different entity types, with completely different data in the same container.
Figure 9 Adding a Schema
listingSchemaEntity.Kind = "ListingSchema";
listingSchemaEntity.Properties = new Dictionary<string, object>();
listingSchemaEntity.Properties["ListingID"] = m_ListingID;
listingSchemaEntity.Properties["ListingName"] = m_ListingName;

int propCount = 0;
bool required = false;

foreach (DataGridViewRow row in dataGridView1.Rows) {
  if (row.IsNewRow) continue;
  propCount++;

  listingSchemaEntity.Properties[string.Format    ("Property{0}Name",propCount)] =
    row.Cells["colFieldName"].Value.ToString();
  listingSchemaEntity.Properties[string.Format    ("Property{0}DataType", propCount)] =
    row.Cells["colDataType"].Value.ToString();

  if (row.Cells["colRequired"].Value == null) {
    required = false;
  }
  else {
    required = Convert.ToBoolean(row.Cells      ["colRequired"].Value.ToString());
  }

  listingSchemaEntity.Properties[string.Format    ("Property{0}Required", propCount)] = required;
}

if (propCount > 0) {
  using (SSDSClient.SitkaSoapServiceClient ssdsProxy =     new SSDSClient.SitkaSoapServiceClient()) {
    //Set username and password for the service
    ...

    //Set service scope to the main contoso classifieds
    //authority. Point it to the categories container
    ...

    //If this entity doesn't exist, create it
    if (listingSchemaID == "" || listingSchemaID == null) {
      ssdsProxy.Create(serviceScope, listingSchemaEntity);
    }
    //Otherwise update it
    else {
      serviceScope.EntityId = listingSchemaID;
      ssdsProxy.Update(serviceScope, listingSchemaEntity);
    }
  }
  MessageBox.Show("Custom Schema Saved", "Custom Schema Saved",
    MessageBoxButtons.OK, MessageBoxIcon.Information);
  this.Hide();
}
The last piece of the admin client is to delete a custom listing schema when a listing category is deleted. I previously discussed deleting entities and containers, but then I knew the ID of the entity or container I wanted to delete. Deleting a schema means first issuing a LINQ query to retrieve all listing schemas that are attached to the listing categories and deleting them. The query looks like this:
string.Format(@"from e in entities
  where e.Id == ""{0}"" select e", CategoryID);
That query will return a list of ListingSchema entities, and all you need to do at this point is delete them. The entire DeleteListingSchema method is shown in Figure 10.
Figure 10 DeleteListingSchema
//Retrieve the custom schema entity for this listing
string linqQuery =   string.Format(
  @"from e in entities where e.Id == ""{0}"" select e", CategoryID);

List<SSDSClient.Entity> entities =
  ssdsProxy.Query(serviceScope, linqQuery);

foreach (SSDSClient.Entity entity in entities) {

  //If more than 1 flex property on the entity, the
  //header has listing categories attached to it.
  if (entity.Properties.Count > 1) {
    //Calculate the number of Listing Categories
    int propCount = ((entity.Properties.Count - 1) / 2);
    for (int x = 1; x < propCount + 1; x++) {
      //Delete the listing schema for each listing
      //in this category
      DeleteListingSchema(entity.Properties["ListingCategoryID" +
        x.ToString()].ToString());
    }
  }
}
At this point the Contoso Classifieds Administration client is complete and I can move on the implementing the Web site using the REST interface of SSDS.

The Classifieds Web App
As mentioned earlier, you can access SSDS via its REST interface as well as its SOAP interface (as I did for the Contoso Classifieds Administration client). The concepts and features are the same regardless of the interface you choose, but with REST you will make direct HTTP (or HTTPS) calls.
To start, look back at the main interface of the Web app shown in Figure 1. There is a tree view that represents the listing categories, a main content area, and a Datalist of all the cities that the application is currently configured to support. Looking into each one, I'll show you how easy it is to develop an ASP.NET app against SSDS.
If you look at the ASPX code for the tree view, you'll see that it is quite simple:
<h3>Groups</h3>
<asp:TreeView ID="TreeView1" runat="server"
  onselectednodechanged="TreeView1_SelectedNodeChanged">
</asp:TreeView>
The magic happens in the code behind, as you can see in Figure 11. The first thing for you to be concerned with is the building of the URI, which is the pointer to the Categories container. Since this data is not city-specific, I have chosen to store it in the main contosoclassifieds authority, within the Categories container.
Figure 11 Building the Tree View
TreeView1.Nodes.Clear();

appDataUri = string.Format(@"http://{0}.{1}{2}",
  conSSDSAuthName, conSSDSUri, "Categories");
query = @"from e in entities where e.Kind == ""Category"" select e";

UriBuilder newUri = new UriBuilder(appDataUri);
newUri.Query = String.Format("q='{0}'", Uri.EscapeDataString(query));

string xmlResults =
  HTTPHelper.GetHTTPWebRequest(newUri.Uri.ToString(),
  new System.Net.NetworkCredential(conSSDSUsername, conSSDSPassword));

XmlDocument categoriesDoc = new XmlDocument();
categoriesDoc.LoadXml(xmlResults);
XmlNodeList nodeList = categoriesDoc.SelectNodes("//Category");

int nodeIndex = 0;

foreach (XmlNode node in nodeList) {
  if (node.ChildNodes.Count > 3) {
    int propCount = ((node.ChildNodes.Count - 1) / 2);

    TreeNode tn = new TreeNode(node.ChildNodes[2].InnerText,
      node.ChildNodes[0].InnerText);
    TreeView1.Nodes.Add(tn);

    for(int x=1;x<propCount;x++) {
      tn = new TreeNode(node.ChildNodes[(x*2)+2].InnerText,
        node.ChildNodes[(x*2)+1].InnerText);
      TreeView1.Nodes[nodeIndex].ChildNodes.Add(tn);
    }
  }
  else {
    TreeNode tn = new TreeNode(node.ChildNodes[2].InnerText,       node.ChildNodes[0].InnerText);
    TreeView1.Nodes.Add(tn);
  }

  nodeIndex++;
}
Following that, I proceed to use a LINQ query to retrieve all entities with a Kind category, and then I use a UriBuilder object to put it all together and add the necessary HTTP escaping. Next, I go ahead and call the GetHTTPWebRequest method, passing in the escaped URI and a NetworkCredential object with the SSDS user name and password. GetHTTPWebRequest returns a string representation of an EntitySet with all Entities that satisfy the query predicates.
GetHTTPWebRequest is a static helper method to hide some of the HTTP details:
WebRequest request =
  HttpWebRequest.Create(Uri.EscapeUriString(serviceUri));
request.Credentials = requestCredential;
request.Method = "GET";
request.ContentType = XmlContentType;

// Get the response and read it in to a string.
using (HttpWebResponse response =
  (HttpWebResponse)request.GetResponse()) {

  return ReadResponse(response);
}
Now all that needs to be done is to create a new WebRequest object, set the Credentials that were passed to it, set the appropriate HTTP verb, set the ContentType to XML, and call the GetResponse method. I then call ReadResponse, which is another static helper method which uses a stream reader to read the HTTPResponse and return it to the caller. The code behind then takes the returned XML, loads it up into a XmlDocument, and loads the tree view with it.

Class Deserialization
Up to this point, the code has simply been manually manipulating XML, but it's easy to take an entity and deserialize it into a class. In this case, the application will be using a CityServed class.
The base Entity class is pretty basic. The class contains properties for both the ID and Version, which are common to all entities. It also contains the necessary attributes so that the class is serializable.
CityServed, shown in Figure 12, inherits from the base Entity class. A generic Query method returns a list of entities of type CityServed:
foreach (CityServed i in HTTPHelper.Query<CityServed>(
  appDataUri, query, new System.Net.NetworkCredential(
  conSSDSUsername, conSSDSPassword)))
Figure 12 CityServed
[XmlRoot(ElementName = "CityServed", Namespace = "")]

public class CityServed : Entity {
  [XmlElement(ElementName = "AuthorityUri")]
  public object AuthorityUriField;

  [XmlElement(ElementName = "Name")]
  public object NameField;

  public override string ToString() {
    return Name;
  }

  [XmlIgnore]
  public string AuthorityUri {
    get { return (string)AuthorityUriField; }
    set { AuthorityUriField = value; }
  }

  [XmlIgnore]
  public string Name {
    get { return (string)NameField; }
    set { NameField = value; }
  }
}
The Query method, included in the code download for this article, is simply a generic static helper method that calls the same GetHTTPWebRequest method as we had used earlier. The Serialize and Deserialize methods are shown in Figure 13. The Query method, combined with the Serialize and Deserialize methods, gives you the ability to use strongly typed .NET objects and save them in SSDS.
Figure 13 Serializing and Deserializing Entities
private static T Deserialize<T>(Stream stm, string xmlPayload) {
  XmlSerializer ser = new XmlSerializer(typeof(T));

  T flex = (T)ser.Deserialize(stm);

  XmlDocument xDom = new XmlDocument();
  xDom.LoadXml(xmlPayload);

  return flex;
}

public static T Deserialize<T>(String xmlPayload) {
  using (MemoryStream stm = new MemoryStream()) {
    Encoding encoding = new UTF8Encoding(false);
    stm.Write(encoding.GetBytes(xmlPayload), 0, encoding.GetByteCount(xmlPayload));
    stm.Position = 0;
    return Deserialize<T>(stm, xmlPayload);
  }
}

public static string Serialize<T>(T flex) {
  using (MemoryStream stm = new MemoryStream()) {
    Serialize(stm, flex);

    stm.Position = 0;

    using (StreamReader reader = new StreamReader(stm)) {
      return reader.ReadToEnd();
    }
  }
}
private static void Serialize<T>(Stream stm, T flex) {
  XmlSerializer ser = new XmlSerializer(typeof(T));
  Encoding encoding = new UTF8Encoding(false);
  XmlWriterSettings settings = new XmlWriterSettings();
  settings.CloseOutput = false;
  settings.ConformanceLevel = ConformanceLevel.Document;
  settings.Encoding = encoding;
  settings.Indent = true;
  settings.OmitXmlDeclaration = true;

  using (XmlWriter writer = XmlWriter.Create(stm, settings)) {
    ser.Serialize(writer, flex);
  }
}

Using the Custom Listing Schema.
As previously mentioned, SSDS entities are flexible, meaning that each entity can have whatever shape you require. The Contoso Classifieds Administration client, for example, gave you the ability to add a schema to a listing category. You can also pull the custom schema out of SSDS and dynamically create an entry form that can be used to add listings to the system.
The first step is to create a DataTable with four columns: Field, Value, DataType and Required. This table will hold the fields the administrator has defined for the listing. Next I will query to see whether a customs schema has been defined for the listing:
string appDataUri = string.Format(@"http://{0}.{1}{2}",
  conSSDSAuthName, conSSDSUri, "Categories");
string query = string.Format(
  @"from e in entities where e[""ListingID""] == ""{0}"" select e",
  listingCategoryID);

UriBuilder newUri = new UriBuilder(appDataUri);
newUri.Query = String.Format("q='{0}'", Uri.EscapeDataString(query));

string xmlResults = HTTPHelper.GetHTTPWebRequest(newUri.Uri.ToString(),
  new System.Net.NetworkCredential(conSSDSUsername, conSSDSPassword));

XmlDocument categoriesDoc = new XmlDocument();
categoriesDoc.LoadXml(xmlResults);
XmlNodeList nodeList = categoriesDoc.SelectNodes("//ListingSchema");
I can then load the returned XML into the DataTable and bind it to the DataList.
In the REST interface, you need to build the XML representation of the entity and issue a HTTP POST to the container into which you want insert the entity. Building the entity is easy—you can simply use a boilerplate entity and loop through each field, adding it as a node in the XML document:
foreach (DataListItem fieldItem in dlAddFields.Items) {
  Label lblField = (Label)fieldItem.FindControl("lblAddField");
  TextBox tbItem = (TextBox)fieldItem.FindControl("txtAddValue");
  Label lblFieldType = (Label)fieldItem.FindControl("lblAddFieldType");
  entity = entity + String.Format(
    @" <{0} xsi:type='x:{1}'>{2}</{0}>", lblField.Text.Replace(" ", ""),
    lblFieldType.Text, tbItem.Text);
}
Finally, build the URI that points to the container to which you want to POST and issue the POST:
string serviceUri = string.Format(@"http://{0}.{1}{2}",
  postingAuthority, conSSDSUri, postingContainer);

HTTPHelper.PostHTTPWebRequest(serviceUri, entity,
  new System.Net.NetworkCredential(
  conSSDSUsername, conSSDSPassword));

Looking Forward
SSDS is easy to develop against. The one point I would like to reiterate is that SSDS is built upon proven SQL Server and Windows Server® technology. While the team has chosen to only reveal a limited set of features in the initial beta release of SSDS, there is a lot more to come. The subset of functionality available now will assist many of the scenarios that customers need to address.
Additionally, you can expect to see support for SSDS being added to the other products in the SQL Server suite of products and tools. So, whether you're a developer using C#, Visual Basic®, Java, Ruby, or even Microsoft Office Access®, SSDS with its support for open protocols is an ideal place for you to store your application's data. To learn more about it, visit the SSDS site at microsoft.com/sql/dataservices and sign up for the beta. If there are any additional features you would like to see in the product, please contact me at david.robinson@microsoft.com.

David Robinson is a Senior Program Manager on the SQL Server Data Services team at Microsoft. He spends his time driving new and compelling features into the product. He also enjoys doing presentations at community events and getting feedback from customers on SSDS.


ADO.NET
Achieve Flexible Data Modeling With The Entity Framework
Elisa Flasko

This article discusses:
  • The philosophy behind the Entity Framework
  • The Entity Data Model
  • Querying, mapping, and n-tier development
This article uses the following technologies:
ADO.NET, LINQ, Entity Framework

Code download available at: EntityFramework2008_07.exe (6,602 KB)
Browse the Code Online

Contents
Why Another Data Model?
Why Describe the EDM with XML?
Who Needs Another New Query Language?
Implementing the EDM
EntityClient
Mapping
Object Services
Just Another ORM?
The ADO.NET Entity Framework is almost here! First introduced as ADO.NET vNext in 2006, the framework is now ready for prime time with the upcoming release of Visual Studio® 2008 SP1. After a couple of unsuccessful attempts at similar products over the years, Microsoft released two technologies with Visual Studio 2008 that fit, in part, into the Object Relational Mapping (ORM) space: LINQ to SQL and the ADO.NET Entity Framework. With the adoption of these technologies in the marketplace, developers now want to know how we got here and where Microsoft is going next. They also want to know what was behind the development of these technologies, what makes the Entity Framework different from other ORM technologies in the market, and how is Microsoft investing in these technologies going forward?
There were numerous articles following the initial release of Visual Studio 2008 focusing on LINQ to SQL, as well as articles addressing which technology to use (see msdn.microsoft.com/data). Here I'll focus on the Entity Framework and provide a deeper understanding of how and why choices were made during development.
The Microsoft® Entity Data Model (EDM), based on Dr. Peter Chen's Entity Relationship (ER) model, is really the driving force behind the ADO.NET Entity Framework. The EDM is also the feature that most significantly differentiates the Entity Framework from other ORM-style technologies in the marketplace. The EDM builds on the ER model to raise the abstraction level for models an order higher than logical models while still preserving the concepts of entities and relationships.

Why Another Data Model?
So why was another model needed? Well, as the amount of data companies were working with increased, it became very difficult to understand and develop applications on top of that data. Database schemas are designed with storage concerns, such as data integrity, performance, and management, in mind and are not always easy to understand. Often these schemas also do not align with the structure of your application, making development and maintenance even more complex.
Custom solutions that separated the structure of data from the application being built were common. Unfortunately, with the number of custom solutions, various approaches, and steps required to model data all being different for each application, the problem continued to grow. There was a consistent need across the industry for a way to define and develop against an application-level domain model that can be clearly separated from the logical model of the store. Enter the Entity Framework.
The EDM (see the sample illustrated in Figure 1) allows the definition of a domain model that is consistent with the way an organization thinks about and uses its data, rather than the way that data is stored. The EDM was also developed with the primary goal of becoming the core data model across a suite of developer and server technologies within Microsoft.
Figure 1 Sample Entity Data Model for a Blogging Database
With one core data model, application maintenance is simplified. When this goal is realized, the EDM can be used to define a model not only for custom applications built on the ADO.NET Entity Framework but also as the input to reporting and visualization applications, intranet portal applications, or workflow applications.
Similar to the ER model, the EDM uses two primary concepts: entities (things) and relationships (or associations) among those entities. When reasoning about the storage of entity and association instances, or the closure of set operations over those instances, the concept of sets is also required. Therefore, to complete the picture, entities live in EntitySets and associations live in AssociationSets.
The final structural concept defined in the EDM is that of an EntityContainer, which defines a closure around the sets of instances and relationships previously described. These simple concepts, used together, allow developers to define a domain model that can be mapped back to the persistence layer and to classes used in the application itself. (Just note that the persistence layer of the EDM doesn't need to be relational even though today it is.)
Each Entity Type defined in an EDM can contain two different types of members—properties defining the entity (analogous to columns in the database) and navigation properties, which enable navigation of relationships in which the Entity Type participates (often represented as foreign keys in the database). In addition to these, each Entity Type must have a distinct identity or key. The XML fragment in Figure 2 describes a "Blog Post" Entity Type.
Figure 2 BlogPost Entity Type Definition
<EntityType Name="BlogPost">
   <Key>
      <PropertyRef Name="BlogPostID" />
   </Key>
   <Property Name="BlogPostID" Type="Int32" Nullable="false" />
   <Property Name="BlogEntry" Type="String" Nullable="false" MaxLength="Max"
             Unicode="true" FixedLength="false" />
   <Property Name="BlogDate" Type="DateTime" Nullable="false" />
   <Property Name="BlogTitle" Type="String" Nullable="false" MaxLength="500"
             Unicode="true" FixedLength="false" />
   <Property Name="BlogType" Type="Int32" Nullable="false" />
   <Property Name="CityVisited" Type="String" MaxLength="200"
             Unicode="true" FixedLength="false" />
   <Property Name="CountryVisited" Type="String" MaxLength="200" Unicode="true"
             FixedLength="false" />
   <Property Name="DateVisited" Type="DateTime" />

   <NavigationProperty Name="Blog" Relationship=
                       "MyTravelPostModel.FK_BlogPosts_Blogs"
                       FromRole="BlogPosts" ToRole="Blogs" />
   <NavigationProperty Name="Pictures" Relationship=
                       "MyTravelPostModel.FK_Pictures_BlogPosts"
                       FromRole="BlogPosts" ToRole="Pictures" />
   <NavigationProperty Name="Comments" Relationship=
                       "MyTravelPostModel.BlogComments"
                       FromRole="BlogPosts" ToRole="Comments" />
</EntityType>
Properties on an Entity Type can be primitive types or complex types (see Figure 3), but in the Entity Framework 1.0 they cannot be other Entity Types or Collections of primitive or complex types. The Entity Type's key is then composed of a subset of these properties. Navigation properties enable navigating a relationship from one entity to another.
Figure 3 Address Complex Type Definition
<ComplexType Name="Address">
    <Property Name="StreetAddress"
              Type="String" MaxLength="50" />
    <Property Name="Address2"
              Type="String" MaxLength="50" />
    <Property Name="City"
              Type="String" MaxLength="50" />
    <Property Name="Region"
              Type="String" MaxLength="50" />
    <Property Name="PostalCode"
              Type="String" MaxLength="50" />
    <Property Name="Country"
              Type="String" MaxLength="50" />
</ComplexType>
As discussed previously, relationships may be surfaced as navigation properties on each Entity Type that is a party to the relationship. Relationships themselves are first-class citizens within the EDM and are explicitly defined as Associations within the EDM:
<Association Name="FK_Friends_People">
   <End Role="People" Type="MyTravelPostModel.Person" Multiplicity="1" />
   <End Role="Friends" Type="MyTravelPostModel.Friend" Multiplicity="*" />
   <ReferentialConstraint>
      <Principal Role="People">
         <PropertyRef Name="PersonID" />
      </Principal>
      <Dependent Role="Friends">
         <PropertyRef Name="PrimaryPersonID" />
      </Dependent>
   </ReferentialConstraint>
</Association>
So in short, why did we create a new data modeling technology in the first place? Although a number of data modeling technologies or languages existed prior to the EDM, none were able to satisfy the primary goals Microsoft was trying to accomplish, and none served to make the much-used Entity Relationship model executable. The team investigated numerous existing data modeling technologies but, realizing that all were fairly specific or focused to certain problem areas, began developing the EDM to create a model that met these goals and could truly be used as a core data model across a suite of developer and server technologies.

Why Describe the EDM with XML?
After much consideration, XML was chosen as the first serial representation for the EDM. Having a well-defined XML format enables developers and third parties to do transformations into this format and to load into the Entity Framework's metadata runtime, either through generating XML files (or resources) or by loading from dynamically generated XML representations. It is conceivable, however, that other representations of the EDM could be created, and it is likely that alternate representations will be seen as the product moves forward with future releases.
The current EDM grammar is defined in an XML Schema Definition Language (XSD) that ships with the product. It is not expected, however, that most people will develop the XML by hand, but rather will use the tools provided in Visual Studio. That said, the team has heard of interest in Domain-Specific Languages (DSLs) and in alternative persistence mechanisms (databases being the common one) for EDM models, and it is evaluating the options for expansion in upcoming releases.

Who Needs Another New Query Language?
The last question concerning the development of the EDM is why create a new query language? Why not use an existing one? The answer will become clearer as I dig slightly deeper into the EDM.
So far I've talked about why the EDM was created and the various constructs used in the EDM, as well as the fact that the model is descendant from the Entity Relationship model. In creating a model that was not only able to map cleanly to the underlying data store but also represent the application-level domain model developers would like to program against, the EDM needed to be capable of modeling concepts such as inheritance and polymorphism. Since current relational query languages do not support querying based on inheritance, relationship navigation, or the return of polymorphic results, a new query language was needed to satisfy this requirement.
Thus was born Entity SQL (ESQL), a new SQL dialect that adds the ability to query based on the concepts that are not supported in previous SQL dialects. ESQL extends the existing SQL language in much the same way as the EDM extends the Relational model used in databases. Additionally, ESQL is also not tied to the syntax of any specific back-end database, allowing the queries (and/or the application) to be written once, regardless of the back-end database being targeted. In the following example, I look at a simple ESQL query that will retrieve all blogs with at least one blog post and the associated Person (or in the case of my model, the blog owner):
select c, c.Person
  from travelEntitiesGeneral.Blogs as c
  where c.BlogPosts.Count > 0

Implementing the EDM
The ADO.NET Entity Framework is an evolution of ADO.NET and the first concrete implementation of the EDM, providing a higher level of abstraction when developing against a relational database. In version 1.0, the team has been focused on building up the foundation of a platform, more than just a simple ORM, which will allow developers to work against a conceptual or object model with a very flexible mapping and the ability to accommodate a high degree of divergence from the underlying store.
This high degree of flexibility and divergence from the underlying store is the key to allowing the database and applications to evolve separately. When a change is made in the database schema, the application is insulated from the change by the Entity Framework, and you are often not required to rewrite portions of the application, but rather to simply update the mapping files if necessary to accommodate the change.
To begin evolving the ADO.NET platform, the Entity Framework is built on top of the existing ADO.NET 2.0 provider model, with existing providers being updated slightly to support the new Entity Framework and ADO.NET 3.5 functionality. We chose to implement on top of the existing ADO.NET provider model to ensure a provider model that is familiar to the development community.
The architecture is illustrated in Figure 4. You'll note that acceptable schemas include Conceptual Schema Definition Language (CSDL), Mapping Schema Language, and Storage Schema Definition Language (SSDL). Also note that the Entity Framework includes an updated SqlClient Data Provider that supports canonical command trees (CCT).
Figure 4 ADO.NET Entity Framework Architecture

EntityClient
The Entity Framework then introduces a new ADO.NET provider, EntityClient, on top of these ADO.NET 3.5 providers. EntityClient looks very much like the ADO.NET providers you are used to, and it provides the first abstraction allowing developers to execute queries in terms of the EDM using the standard Connection, Command, and DataReader objects. It also adds the additional Client View Engine required to map the domain model, defined in terms of the EDM, to the underlying relational database schema. EntityClient provides the ability for developers, when necessary, to work against entities in the form of rows and columns using ESQL query strings without the need to generate classes to represent the conceptual schema.
If you look at the use of EntityClient in Figure 5, you can see that I have created an EntityCommand taking in an ESQL query string, and that command is then executed against my EDM. The query text supplied as part of the EntityCommand is parsed, and a CCT is created.
Figure 5 Use of ESQL to Query against EntityClient
using (EntityConnection conn = new
         EntityConnection("name=travelEntitiesGeneral"))
{
      conn.Open();
      EntityCommand cmd = conn.CreateCommand();
      cmd.CommandText = @"select c.BlogID
         from travelEntitiesGeneral.Blogs as c
         where c.BlogPosts.Count > 0";
      EntityDataReader reader =
         cmd.ExecuteReader(CommandBehavior.SequentialAccess);
      while (reader.Read())
      {
         Console.WriteLine("BlogID = {0}", reader["BlogID"]);
      }
     conn.Close();
}
At this first stage, the command tree is still represented in terms of the EDM. The Client View Engine, borrowing from the theories for materialized views in database systems but applying these theories to the data access layer, applies a mapping transformation to the tree, producing a tree that represents the same operation in terms of the underlying logical storage model and removing any non-relational concepts such as relationships, inheritance, and polymorphism. This newly transformed tree is handed to the ADO.NET 3.5 provider services, which return a DbCommand that encapsulates native SQL for the underlying store, which is then executed and the results propagated back up through the stack.

When defining the mapping that is used in the Client View Engine to transform between the EDM and the logical database schema, there are a couple of different options. The mapping can be specified using a declarative XML grammar, the Mapping Specification Language (MSL), which can be created and edited either by hand coding the XML or using the Entity Mapping Tools included in Visual Studio (see Figure 6).
Figure 6 MSL—EntitySetMapping Example
<EntitySetMapping Name="BlogPosts">
   <EntityTypeMapping TypeName="IsTypeOf(MyTravelPostModel.BlogPost)">
      <MappingFragment StoreEntitySet="BlogPosts">
         <ScalarProperty Name="BlogPostID" ColumnName="BlogPostID" />
         <ScalarProperty Name="BlogEntry" ColumnName="BlogEntry" />
         <ScalarProperty Name="BlogDate" ColumnName="BlogDate" />
         <ScalarProperty Name="BlogTitle" ColumnName="BlogTitle" />
         <ScalarProperty Name="BlogType" ColumnName="BlogType" />
         <ScalarProperty Name="CityVisited" ColumnName="CityVisited" />
         <ScalarProperty Name="CountryVisited"
                         ColumnName="CountryVisited" />
         <ScalarProperty Name="DateVisited" ColumnName="DateVisited" />
      </MappingFragment>
   </EntityTypeMapping>
</EntitySetMapping>
When compiled, the MSL allows the Entity Framework to generate the necessary query and update views that are then used within the Client View Engine to accomplish the transformation of the query defined in terms of the EDM to the logical storage schema.
An alternative option for expressing the mapping or a portion of the mapping is through the use of an ESQL query. In this case, when the developer expresses the Query View using ESQL, the infrastructure requires that they also define the accompanying Create, Update, and Delete mappings in the mapping specification. This is required, as the mapping infrastructure is not able to generate a corresponding update view for the query view given that one can leverage the power of ESQL in the Query View, making it possible to have views defined for which there is not a single valid update view.

Object Services
On top of the EntityClient provider, the Entity Framework adds another set of abstractions in order to allow development against objects, rather than untyped Data Records returned by EntityClient. This is the layer that is most often thought of as an ORM, producing CLR instances of the types defined in one's data model and allowing developers to query against those objects using either LINQ or ESQL. This also happens to be the layer of the Entity Framework that is responsible for initially attracting many developers to the technology when looking at the available ORM technologies in the marketplace.
As you saw in Figure 1, the high-level function of the Object Services layer is to take in either ESQL or LINQ queries from the application, pass a query expression to the EntityClient below, and return an IEnumerable<T>. To look a bit deeper, however, you see that at the heart of the Object Services layer is the ObjectContext, representing the session of interaction between the application and the underlying data store.
The ObjectContext is the primary construct that the developer will work with to query, add, and delete instances of his entities and to save new state back to the database. Creation and use of ObjectContext to query, manipulate, and SaveChanges to an Entity is shown in Figure 7. This example uses ESQL as the query language.
Figure 7 Using ObjectContext
using (ObjectContext context = new ObjectContext("name=travelEntities"))
    {
        //--- create a query for customers
        ObjectQuery<Person> personQuery = context.CreateQuery<Person>(
                     @"select value c from travelEntitiesGeneral.People
                     as c where c.PersonID == 1");
        //--- by enumerating the query will be implicitly executed
        //--- against the store and you can now work with an
        //--- IEnumerable<Customer>
        foreach (Person c in personQuery)
        {
            //--- dereference anything you like from Customer
            Console.WriteLine(c.PersonID + ": " + c.Name);
            c.Name = "New Name";
        }
        try
        {
            context.SaveChanges();
        }
        catch (OptimisticConcurrencyException opt)
        {
            // catching this exception allows you to
            // refresh travelEntities with either store/client wins
            // project the travelEntities into this failed travelEntities.
            var failedEntities = from e3 in opt.StateEntries
                                 select new { e3.Entity };

            // Note: in future you should be able to just pass
            // the opt.StateEntities
            // in to refresh.
            context.Refresh(RefreshMode.ClientWins, failedEntities.ToList());
            context.SaveChanges();
        }
    }
The process of tracking changes as they are made to the objects in memory and the process of saving those changes back to the database is simplified for the developer through the use of Object Services. Object Services makes use of the ObjectStateManager to track not only the current state of the instances in memory but also the initial state of each instance as it was retrieved from the store, allowing the Entity Framework to apply optimistic concurrency as data is pushed back to the database. Tracked changes are easily saved and pushed back to the data store with invocation of the SaveChanges method on the ObjectContext.
Up to this point, I have been speaking about the ObjectContext in general, and my examples have shown the use of a base ObjectContext, which is often used in the scenario where you have a dynamic tool or application that consumes EDM models. When using Visual Studio as the development environment, however, developers see the added benefit of a strongly typed ObjectContext, adding properties and methods to surface capabilities that may be specific to the EDM being targeted.
Figure 8 shows a query built using a strongly typed ObjectContext. This example demonstrates the use of LINQ as the query language. Using a strongly typed ObjectContext exposes properties for each EntitySet, making them more discoverable; for example, travelEntities.BlogPosts instead of travelEntities.CreateQuery<BlogPost>("travelEntitiesGeneral.BlogPosts").
Figure 8 Query Built Using a Strongly Typed ObjectContext
using (MyTravelPostEntities travelEntities = new MyTravelPostEntities())
{
    // get the latest blog post, with the comments and the people
    // I'm querying for all the blog posts that are related to this blog.
    // I want to include the comments and the people who wrote the
    // comments.
    // I also want only the most recent posting.
    // Note: Since we use the EntityKey that is put on the EntityReference
    // we can either do a tracking query or use span.
    BlogPost post = (from bp in
        travelEntities.BlogPosts
                              .Include("Comments.Person")
                              .Include("Blog")
                     where bp.Blog.BlogID == requestedBlog.BlogID
                     orderby bp.BlogDate descending
                     select bp).First();
    return post;
}
LINQ to Entities can be seen as a fairly thin layer over Object Services, providing query facilities directly within the programming language (see Figure 8) as opposed to a string-based query. In this case, the ObjectQuery class implements IQueryable, allowing it to take a LINQ expression tree, pushing the query through the Entity Framework as a CCT query expression in the same manner as Object Services would pass an ESQL query to the EntityClient provider.
N-Tier Development with the Entity Framework
While it is not my primary goal to fully address n-tier development here, as one of the more interesting scenarios for development with the Entity Framework it should be touched on. In version 1.0, the Entity Framework supports n-tier development in a couple of primary scenarios. These include the use of ADO.NET Data Services or the use of Windows® Communication Foundation (WCF) with the ability to serialize entities and attach and detach entities from an ObjectContext. These are obviously not the only approaches to n-tier development; however, these are the solutions the team chose to concentrate on in V1, with additional scenarios added in V2 and beyond, such as a more dataset-like experience. Figure 9 illustrates what I'm talking about.
Figure 9 ADO.NET Data Services in N-Tier Apps
static Uri baseService = new
   Uri("http://localhost:17338/MyTravelPostService.svc");
MyPeople2Entities context = new MyPeople2Entities(baseService);
    // get the comment that is being marked for deletion
    // and get the view state blog post.
BlogPost post = (BlogPost)ViewState["BlogPost"];

    // move the comment to the deleted comment selection.
Comment deletedComment = post.Comments[e.RowIndex];

    // call the DeleteComment service
context.AttachTo("Comments", deletedComment);
context.DeleteObject(deletedComment);
DataServiceResponse r = context.SaveChanges();

    // reload page so that F5, refresh doesn't update all this data.
ReloadPage();
ADO.NET Data Services is a concrete realization of a Representational State Transfer (REST) architecture style (each resource represents a "noun" in the system—a thing that can be uniquely addressed via a Uniform Resource Identifier, or URI), which enables n-tier application development over any arbitrary IQueryable implementation. With ADO.NET Data Services, you can do more than just query for instances across the wire. ADO.NET Data Services support the various HTTP verbs for Create, Read, Update, and Delete and provide the client-side abstractions to help developers implement their solutions.
The second option for n-tier scenarios is the use of WCF with the Entity Framework, taking advantage of the ability to serialize entities and attach and detach entities from an ObjectContext. Figure 10 shows how to attach to an ObjectContext in this scenario.
Figure 10 Attaching to ObjectContext
// the creation of the travel MyTravelPostEntities opens the connection
// and sets up all the metadata information automatically for you.
using (MyTravelPostEntities travelEntities = new MyTravelPostEntities())
{
    // attach the comment and delete.
    travelEntities.Attach(deleteComment);

    // call delete on the object
    travelEntities.DeleteObject(deleteComment);

    try
    {
       travelEntities.SaveChanges();
    }
    catch (OptimisticConcurrencyException opt)
    {
      // catching this exception allows you to
      // refresh travelEntities with either store/client wins
      // project the travelEntities into this failed travelEntities.
      var failedEntities = from e3 in opt.StateEntries
                           select new { e3.Entity };

     travelEntities.Refresh(RefreshMode.ClientWins, failedEntities.ToList());
     travelEntities.SaveChanges();
    }
}
By default, any CLR classes that are generated from an EDM in Visual Studio or using edmgen.exe (the command-line tool that ships with the Entity Framework) are XML serializable, binary serializable, and are Data Contracts with the Navigation Properties attributed as DataMembers by default, making it possible to create ASMX Web services and use Entity instances in view state or WCF services.
Like most ORMs, the Entity Framework currently does not support Data Manipulation Language (DML) operations for create, update, or delete. Changes must be applied to objects in memory, and building the entire graph to be persisted could require a number of round-trips to the database.
One way that this can be avoided is through the use of the attach functionality offered by the ObjectContext. Using Attach allows you to instruct the infrastructure that the Entity already exists and that a set of operations should be performed in memory and then the changes pushed down. For additional information on n-tier development with the Entity Framework, search the MSDN® Library, as more content will be added soon.

Just Another ORM?
So far, the Entity Framework has been considered by many to be just another ORM in the marketplace, which is understandable when looking simply at the first version of the product. In that direction, much of what has been included in the product to this point enables a core set of scenarios that ORMs begin to tackle. Much of the analysis to this point, however, has pointed out that the Entity Framework does not in all cases cover what you might expect from some of the other ORMs in the marketplace, and this is a valid observation.
The investment that Microsoft is making in this space is meant to extend well beyond that of a traditional ORM product, and the Entity Framework, as I will discuss shortly is the first step in a much broader strategy around the EDM. The EDM, as I discussed at the beginning of this article, creates a higher level domain model that will be applicable beyond just Entity Framework and the world of traditional ORMs. The expectation is that over the next few releases of the Microsoft .NET Framework, Visual Studio, SQL Server®, and other Microsoft technologies, you will begin to see increased adoption of the EDM.
This expectation and the overall vision of where the EDM is headed has been the primary influence for this, as seen in many of the product decisions discussed throughout this article. Many decisions have been made with the explicit intent of enabling adoption by technologies such as Reporting Services and Analysis Services. This will bring a strong benefit to customers as services are able to be offered across a common and consistent domain model.
The first realization of this vision will ship alongside the Entity Framework in Visual Studio 2008 SP1 as ADO.NET Data Services. ADO.NET Data Services, which delivers a compelling developer experience for REST-based applications, will be the first released product (outside of the Entity Framework) to be built with the EDM as its metadata exchange format.
In coordination with this release, Microsoft demonstrated a number of different Windows Live™ properties at MIX 2008, which expose their data using the ADO.NET Data Services protocol and the EDM. Similarly, as we begin now to plan for the next release of SQL Server and Visual Studio, the team is working hard on better end-to-end development experiences with EDM and Entity Framework at the core.

Elisa Flasko is a Program Manager in the Data Programmability team at Microsoft, focusing on ADO.NET, XML, and SQL Server Connectivity technologies. She can be reached via her blog at blogs.msdn.com/elisaj.


Data and WPF
Customize Data Display with Data Binding and WPF
Josh Smith
This article discusses:
  • WPF data binding
  • Data display and hierarchical data
  • Using templates
  • Input validation
This article uses the following technologies:
WPF, XAML, C#

Code download available at: AdvancedWPFDatabinding2008_07.exe (171 KB)
Browse the Code Online
Contents
Binding in Code
Working with Templates
Working with an Inherited DataContext
Working with Collection Views
Working with Hierarchical Data
Using Many Controls to Display XML Data
Using Many Controls to Display Business Objects
One Control to Display an Entire Hierarchy
Using Hierarchical Data Templates
Working with User Input
Input Validation via ValidationRules
Displaying Validation Errors
Input Validation via IDataErrorInfo
Wrapping Up
Experiment in the virtual lab:
When the Windows® Presentation Foundation (WPF) first appeared on the .NET radar, most of the articles and demo applications touted its superb rendering engine and 3D capabilities. While fun to read and play with, such examples do not demonstrate the powerful real-world functionality of WPF. Most of us do not need to create applications with rotating video cubes that burst into fireworks when you click on them. Most of us earn a living by creating software to display and edit large amounts of complex business or scientific data.
The good news is that WPF offers excellent support for managing the display and editing of complex data. In the December 2007 edition of MSDN® Magazine, John Papa wrote "Data Binding in WPF" (msdn.microsoft.com/magazine/cc163299), which did a great job of explaining essential WPF data binding concepts. Here, I'll explore more advanced data binding scenarios, building on what John presented in the aforementioned Data Points column. At the end of the day, you will be aware of various ways to implement common data binding requirements seen in most line-of-business applications.

Binding in Code
One of the biggest changes that WPF introduces for desktop application developers is the extensive use of and support for declarative programming. WPF user interfaces and resources can be declared using Extensible Application Markup Language (XAML), an XML-based markup language. Most explanations of WPF data binding only show how to work with bindings in XAML. Since anything that is possible to do in XAML is also possible to achieve in code, it is important that professional WPF developers learn how to work with data binding programmatically as well as declaratively.
In many situations, it is more convenient and expedient to declare bindings in XAML. As systems become more complex and dynamic, it sometimes makes more sense to work with bindings in code. Before going further, let's first review some common classes and methods involved with programmatic data binding.
WPF elements inherit both the SetBinding and GetBindingExpression methods, from either FrameworkElement or FrameworkContentElement. These are just convenience methods that call into methods with the same names in the BindingOperations utility class. The following code illustrates how to use the BindingOperations class to bind the Text property of a textbox to a property on another object:
static void BindText(TextBox textBox, string property)
{
      DependencyProperty textProp = TextBox.TextProperty;
      if (!BindingOperations.IsDataBound(textBox, textProp))
      {
          Binding b = new Binding(property);
          BindingOperations.SetBinding(textBox, textProp, b);
      }
}
You can easily unbind a property by using the code seen here:
static void UnbindText(TextBox textBox)
{
    DependencyProperty textProp = TextBox.TextProperty;
    if (BindingOperations.IsDataBound(textBox, textProp))
    {
        BindingOperations.ClearBinding(textBox, textProp);
    }
}
By clearing out the binding, you will also remove the bound value from the target property.
Declaring a data binding in XAML conceals some of the underlying details. When you start working with bindings in code, those details begin to come out of the woodwork. One of those details is the fact that the relationship between a binding source and target is actually maintained by an instance of the BindingExpression class, not Binding itself. The Binding class contains high-level information that multiple BindingExpressions can share, but the enforcement of a link between two bound properties comes from an underlying expression. The following code shows how you can use BindingExpression to programmatically check if the Text property of a textbox is being validated:
static bool IsTextValidated(TextBox textBox)
{
    DependencyProperty textProp = TextBox.TextProperty;

    var expr = textBox.GetBindingExpression(textProp);
    if (expr == null)
        return false;

    Binding b = expr.ParentBinding;
    return b.ValidationRules.Any();
}
Since a BindingExpression does not know if it is being validated, you'll need to ask its parent binding. I'll examine input validation techniques later on.
Virtual Lab: Advanced WPF Data Binding
WPF offers excellent support for managing the display and editing of complex data. With relatively few lines of XAML, you can display a hierarchical data structure or validate user input. You can experiment with WPF data binding in our preconfigured Virtual Lab. Everything is installed and ready to go, including the projects discussed in the article. Just start the lab and code along as you read the article.

Experiment in the Virtual Lab

Working with Templates
An effective user interface presents raw data in such a way that the user can intuitively discover meaningful information in it. This is the essence of data visualization. Data binding is just one piece of the data visualization puzzle. All but the most trivial WPF programs require a means to present data in a more powerful way than just binding one property on a control to one property on a data object. Real data objects have many related values, and those various values should aggregate into one cohesive visual representation. This is why WPF has data templates.
The System.Windows.DataTemplate class is just one form of template in WPF. In general, a template is like a cookie cutter that the WPF framework uses in order to create visual elements that aid in rendering objects that have no intrinsic visual representation. When an element attempts to display an object that does not have an intrinsic visual representation, such as a custom business object, you can tell the element how to render the object by giving it a DataTemplate.
The DataTemplate can generate as many visual elements as necessary to display the data object. Those elements use data bindings to display the data object's property values. If an element does not know how to display the object it is told to render, it simply calls the ToString method on it and displays the result in a TextBlock.
Suppose you have a simple class called FullName, which stores a person's name. You want to display a list of names and have each person's last name appear more prominently than the other parts of the name. To do this you could create a DataTemplate that describes how to render a FullName object. The code listed in Figure 1 shows the FullName class and the codebehind for a window that will display a list of names.
Figure 1 Display FullNames with a DataTemplate
public class FullName
{
    public string FirstName { get; set; }
    public char MiddleInitial { get; set; }
    public string LastName { get; set; }
}

public partial class WorkingWithTemplates : Window
{
    // This is the Window's constructor.
    public WorkingWithTemplates()
    {
        InitializeComponent();

        base.DataContext = new FullName[]
        {
            new FullName
            {
                FirstName = "Johann",
                MiddleInitial = 'S',
                LastName = "Bach"
            },
            new FullName
            {
                FirstName = "Gustav",
                MiddleInitial = ' ',
                LastName = "Mahler"
            },
            new FullName
            {
                FirstName = "Alfred",
                MiddleInitial = 'G',
                LastName = "Schnittke"
            }
        };
    }
}
As seen in Figure 2, there is an ItemsControl in the window's XAML file. It creates a simple list of items that the user cannot select or remove. The ItemsControl has a DataTemplate assigned to its ItemTemplate property, with which it renders each FullName instance created in the window's constructor. You should notice how most of the TextBlock elements in the DataTemplate have their Text property bound to properties on the FullName object that they represent.
Figure 2 Display FullName Objects Using a DataTemplate
<!-- This displays the FullName objects. -->
<ItemsControl ItemsSource="{Binding Path=.}">
  <ItemsControl.ItemTemplate>
    <DataTemplate>
      <StackPanel Orientation="Horizontal">
        <TextBlock FontWeight="Bold" Text="{Binding LastName}" />
        <TextBlock Text=", " />
        <TextBlock Text="{Binding FirstName}" />
        <TextBlock Text=" " />
        <TextBlock Text="{Binding MiddleInitial}" />
      </StackPanel>
    </DataTemplate>
  </ItemsControl.ItemTemplate>
</ItemsControl>
When this demo app runs, it looks like Figure 3. By using a DataTemplate to render the name, it is easy to accentuate each person's last name because the corresponding TextBlock's FontWeight is bold. This simple example demonstrates the fundamental relationship between WPF data binding and templates. As I dig deeper into this topic, I will combine these features into more powerful ways of visualizing complex objects.
Figure 3 FullNames Rendered by a DataTemplate

Working with an Inherited DataContext
Unless told otherwise, all bindings implicitly bind against an element's DataContext property. An element's DataContext references its data source, so to speak. There is something special to be aware of regarding how DataContext works. Once you understand this subtle aspect of DataContext, it vastly simplifies the design of complex data-bound user interfaces.
An element's DataContext property does not have to be set in order to reference a data source object. If the DataContext of an ancestor element in the element tree (technically, the logical tree) is given a value for its DataContext, the value will automatically be inherited by every descendant element in the user interface. In other words, if a window's DataContext is set to reference a Foo object, by default the DataContext of every element in the window will reference that same Foo object. You can easily give any element in the window a different DataContext value, which causes all of that element's descendant elements to inherit that new DataContext value. This is similar to an ambient property in Windows Forms.
In the previous section, I examined how to use DataTemplates to create visualizations of data objects. Elements created by the template in Figure 2 have their properties bound to properties of a FullName object. Those elements are implicitly binding to their DataContext. The DataContext of elements created by a DataTemplate reference the data object for which the template is used, such as a FullName object.
There is no magic involved in the value inheritance of the DataContext property. It is leveraging the support for inherited dependency properties that is built-in to WPF. Any dependency property can be an inherited property, simply by specifying a flag in the metadata provided when registering that property with the dependency property system of WPF.
Another example of an inherited dependency property is FontSize, which all elements have. If you set the FontSize dependency property on a window, by default all elements in that window will display their text at that size. The same infrastructure used to propagate the FontSize value down the element tree is what propagates the DataContext.
This usage of the term "inheritance" is different from its meaning in the object-oriented sense, where a subclass inherits the members of its parent class. Property value inheritance refers only to the propagation of values down the element tree at run time. Naturally, a class can inherit a dependency property that happens to support value inheritance, in the object-oriented sense.

Working with Collection Views
When WPF controls bind to a collection of data, they do not bind directly to the collection itself. Instead, they implicitly bind to a view automatically wrapped around that collection. The view implements the ICollectionView interface and can be one of several concrete implementations, such as ListCollectionView.
A collection view has several responsibilities. It keeps track of the current item in the collection, which usually translates to the active/selected item in a list control. Collection views also supply a generic means of sorting, filtering, and grouping items in a list. Multiple controls can bind to the same view around a collection so they are all coordinated with each other. The following code demonstrates some features of ICollectionView:
// Get the default view wrapped around the list of Customers.
ICollectionView view = CollectionViewSource.GetDefaultView(allCustomers);

// Get the Customer selected in the UI.
Customer selectedCustomer = view.CurrentItem as Customer;

// Set the selected Customer in the UI.
view.MoveCurrentTo(someOtherCustomer);
All list controls, such as listbox, combobox, and list view, must have their IsSynchronizedWithCurrentItem property set to true in order to remain synchronized with the collection view's CurrentItem property. The abstract Selector class defines that property. If it is not set to true, selecting an item in the list control will not update the CurrentItem of the collection view, and assigning a new value to CurrentItem will not be reflected in that list control.

Working with Hierarchical Data
The real world is full of hierarchical data. A customer places multiple orders, a molecule is made of many atoms, a department consists of many employees, and a solar system contains a set of celestial bodies. You're undoubtedly familiar with this common master-detail arrangement.
WPF provides various ways to work with hierarchical data structures, each of which is appropriate for different situations. The choices essentially boil down to either using many controls to display data or displaying multiple levels of the data hierarchy in one control. Next, I am going to examine both of those approaches.

Using Many Controls to Display XML Data
A very common way to work with hierarchical data is to have a separate control display each level of the hierarchy. For example, suppose we have a system that represents customers, orders, and order details. In that situation, we might have a combobox to display customers, a listbox to display all of the selected customers' orders, and then an ItemsControl to display details about the selected order. This is a great way to display hierarchical data, and it's quite easy to implement in WPF.
Based on the scenario I described previously, Figure 4 shows a simplified sample of the data an application might deal with, wrapped up in the WPF XmlDataProvider component. A user interface similar to Figure 5 can display that data. Notice how the customers and orders are selectable, but the details about an order exist in a read-only list. This makes sense because a visual object should only be selectable if it affects the state of the application or if it is editable.
Figure 4 Customers, Orders, and OrderDetails XML Hierarchy
<XmlDataProvider x:Key="xmlData">
  <x:XData>
    <customers >
      <customer name="Customer 1">
        <order desc="Big Order">
          <orderDetail product="Glue" quantity="21" />
          <orderDetail product="Fudge" quantity="32" />
          </order>
          <order desc="Little Order">
            <orderDetail product="Ham" quantity="1" />
            <orderDetail product="Yarn" quantity="2" />
          </order>
        </customer>
        <customer name="Customer 2">
          <order desc="First Order">
            <orderDetail product="Mousetrap" quantity="4" />
          </order>
        </customer>
      </customers>
    </x:XData>
</XmlDataProvider>
Figure 5 One Way to Display the XML Data
The XAML in Figure 6 describes how to use those various controls to display the hierarchical data just shown. This window requires no code; it exists entirely in XAML.
Figure 6 XAML to Bind Hierarchical XML Data to a UI
<Grid DataContext=
    "{Binding Source={StaticResource xmlData},
    XPath=customers/customer}"
    Margin="4"
  >
  <Grid.RowDefinitions>
    <RowDefinition Height="Auto" />
    <RowDefinition />
    <RowDefinition />
  </Grid.RowDefinitions>

  <!-- CUSTOMERS -->
  <DockPanel Grid.Row="0">
    <TextBlock DockPanel.Dock="Top" FontWeight="Bold" Text="Customers" />
    <ComboBox
      IsSynchronizedWithCurrentItem="True"
      ItemsSource="{Binding}"
      >
      <ComboBox.ItemTemplate>
        <DataTemplate>
          <TextBlock Text="{Binding XPath=@name}" />
        </DataTemplate>
      </ComboBox.ItemTemplate>
    </ComboBox>
  </DockPanel>

  <!-- ORDERS -->
  <DockPanel Grid.Row="1">
    <TextBlock DockPanel.Dock="Top" FontWeight="Bold" Text="Orders" />
    <ListBox
      x:Name="orderSelector"
      DataContext="{Binding Path=CurrentItem}"
      IsSynchronizedWithCurrentItem="True"
      ItemsSource="{Binding XPath=order}"
      >
      <ListBox.ItemTemplate>
        <DataTemplate>
          <TextBlock Text="{Binding XPath=@desc}" />
        </DataTemplate>
      </ListBox.ItemTemplate>
    </ListBox>
  </DockPanel>

  <!-- ORDER DETAILS -->
  <DockPanel Grid.Row="2">
    <TextBlock DockPanel.Dock="Top" FontWeight="Bold"
      Text="Order Details" />
    <ItemsControl
      DataContext=
         "{Binding ElementName=orderSelector, Path=SelectedItem}"
      ItemsSource="{Binding XPath=orderDetail}">
      <ItemsControl.ItemTemplate>
        <DataTemplate>
          <TextBlock>
            <Run>Product:</Run>
            <TextBlock Text="{Binding XPath=@product}" />
            <Run>(</Run>
            <TextBlock Text="{Binding XPath=@quantity}" />
            <Run>)</Run>
          </TextBlock>
        </DataTemplate>
      </ItemsControl.ItemTemplate>
    </ItemsControl>
  </DockPanel>
</Grid>
Notice the extensive use of short XPath queries to tell WPF where to get the bound values. The Binding class exposes an XPath property to which you can assign any XPath query supported by the XmlNode.SelectNodes method. Under the hood, WPF uses that method in order to execute XPath queries. Unfortunately, this means that since XmlNode.SelectNodes does not currently support the use of XPath functions, WPF data binding does not support them either.
The combobox of customers and the listbox of orders both bind to the resultant node set of the XPath query performed by the root Grid's DataContext binding. The listbox's DataContext will automatically return the CurrentItem of the collection view wrapped around the collection of XmlNodes generated for the Grid's DataContext. In other words, the DataContext of the listbox is the currently selected Customer. Since that listbox's ItemsSource is implicitly bound to its own DataContext (because no other source was specified), and its ItemsSource binding executes an XPath query to get the <order> elements from the DataContext, the ItemsSource is effectively bound to the selected customer's list of orders.
Bear in mind that when binding to XML data you are actually binding to the objects created by a call to XmlNode.SelectNodes. If you are not careful, you can end up with multiple controls binding to logically equivalent, but physically distinct, sets of XmlNodes. This is due to the fact that each call to XmlNode.SelectNodes generates a new set of XmlNodes, even if you pass the same XPath query to the same XmlNode each time. That is a concern specific to XML data binding, so when binding to business objects you can blissfully ignore it.

Using Many Controls to Display Business Objects
Now suppose you wanted to bind to the same data as in the previous example, but the data exists as business objects instead of XML. How would that change the way you bind to the various levels of the data hierarchy? How similar or different would the technique be?
The code in Figure 7 shows the simple classes used to create business objects that store the data to which we will bind. These classes form the same logical schema as the XML data used in the previous section.
Figure 7 Classes to Create Hierarchy of Business Objects
public class Customer
{
    public string Name { get; set; }
    public List<Order> Orders { get; set; }

    public override string ToString()
    {
        return this.Name;
    }
}

public class Order
{
    public string Desc { get; set; }
    public List<OrderDetail> OrderDetails { get; set; }

    public override string ToString()
    {
        return this.Desc;
    }
}

public class OrderDetail
{
    public string Product { get; set; }
    public int Quantity { get; set; }
}
The XAML of the window that displays these objects is in Figure 8. It is very similar to the XAML seen in Figure 6, but there are some important differences worth noting. Something not seen in the XAML is that the window's constructor creates the data objects and sets the DataContext, instead of the XAML referencing it as a resource. Notice that none of the controls have their DataContext explicitly set. They all inherit the same DataContext, which is a List<Customer> instance.
Figure 8 XAML to Bind Hierarchical Business Objects into a UI
<Grid Margin="4">
  <Grid.RowDefinitions>
    <RowDefinition Height="Auto" />
    <RowDefinition />
    <RowDefinition />
  </Grid.RowDefinitions>

   <!-- CUSTOMERS -->
   <DockPanel Grid.Row="0">
     <TextBlock DockPanel.Dock="Top" FontWeight="Bold" Text="Customers"
     />
     <ComboBox
       IsSynchronizedWithCurrentItem="True"
       ItemsSource="{Binding Path=.}"
       />
   </DockPanel>

   <!-- ORDERS -->
   <DockPanel Grid.Row="1">
     <TextBlock DockPanel.Dock="Top" FontWeight="Bold" Text="Orders" />
     <ListBox
       IsSynchronizedWithCurrentItem="True"
       ItemsSource="{Binding Path=CurrentItem.Orders}"
       />
   </DockPanel>

   <!-- ORDER DETAILS -->
   <DockPanel Grid.Row="2">
     <TextBlock DockPanel.Dock="Top" FontWeight="Bold"
        Text="Order Details" />
     <ItemsControl
       ItemsSource="{Binding Path=CurrentItem.Orders.CurrentItem.
       OrderDetails}"
       >
       <ItemsControl.ItemTemplate>
         <DataTemplate>
           <TextBlock>
             <Run>Product:</Run>
             <TextBlock Text="{Binding Path=Product}" />
             <Run>(</Run>
             <TextBlock Text="{Binding Path=Quantity}" />
             <Run>)</Run>
           </TextBlock>
         </DataTemplate>
       </ItemsControl.ItemTemplate>
     </ItemsControl>
   </DockPanel>
</Grid>
Another significant difference when binding to business objects instead of XML is that the ItemsControl that hosts the order details does not need to bind to the SelectedItem of the order's listbox. That approach was necessary in the XML binding scenario because there is no generic way to reference the current item of a list whose items come from a local XPath query.
When binding to business objects instead of XML, it is trivial to bind against nested levels of selected items. The ItemsControl's ItemsSource binding utilizes this convenient feature by specifying CurrentItem in the binding path twice: once for the selected customer, again for the selected order. The CurrentItem property is a member of the underlying ICollectionView wrapped around the data source, as discussed previously in the article.
There is one more point of interest regarding a difference between how the XML and business object approaches work. Since the XML example binds to XmlElements, you must provide DataTemplates to explain how to render the customers and orders. When binding to custom business objects, you can avoid this overhead by simply overriding the ToString method of the Customer and Order classes and allowing WPF to display the output of that method for those objects. This trick only suffices for objects that can have simple textual representations. It might not make sense to use this convenient technique when dealing with complex data objects.

One Control to Display an Entire Hierarchy
Until now, you have only seen ways to display hierarchical data by showing each level of the hierarchy in separate controls. It is often useful and necessary to show all levels of a hierarchical data structure within the same control. The canonical example of this approach is the TreeView control, which supports the display and traversal of arbitrary levels of nested data.
You can populate the WPF TreeView with items in one of two ways. One option is manually adding the items, in either code or XAML, and the other option is to create them via data binding.
The XAML that follows shows how one can manually add some TreeViewItems to a TreeView in XAML:
<TreeView>
  <TreeViewItem Header="Item 1">
    <TreeViewItem Header="Sub-Item 1" />
    <TreeViewItem Header="Sub-Item 2" />
  </TreeViewItem>
  <TreeViewItem Header="Item 2" />
</TreeView>
The manual technique of creating items in a TreeView makes sense for situations where the control will always display a small, static set of items. When you need to display large amounts of data that can vary over time, it becomes necessary to use a more dynamic approach. At that point, you have two options. You can write code that walks over a data structure, creates TreeViewItems based on the data objects it finds, and then adds those items to the TreeView. Alternatively, you can take advantage of hierarchical data templates and let WPF do all of the work for you.

Using Hierarchical Data Templates
You can declaratively express how WPF should render hierarchical data via hierarchical data templates. The HierarchicalDataTemplate class is a tool that bridges the gap between a complex data structure and a visual representation of that data. It is very similar to a normal DataTemplate but also allows you to specify where a data object's child items come from. You can also provide the HierarchicalDataTemplate a template with which to render those child items.
Suppose you now want to display the data presented in Figure 7 within one TreeView control. The TreeView might look something like Figure 9. Implementing this involves the use of two HierarchicalDataTemplates and one DataTemplate.
Figure 9 Display Entire Data Hierarchy in a TreeView
The two hierarchical templates display Customer and Order objects. Since OrderDetail objects have no child items, you can render them with a non-hierarchical DataTemplate. The TreeView's ItemTemplate property uses the template for objects of type Customer, since Customers are the data objects contained at the root level in the TreeView. The XAML listed in Figure 10 shows how all the pieces of this puzzle fit together.
Figure 10 XAML Behind the TreeView Display
<Grid>
  <Grid.DataContext>
    <!--
    This sets the DataContext of the UI
    to a Customers returned by calling
    the static CreateCustomers method.
    -->
    <ObjectDataProvider
      xmlns:local="clr-namespace:VariousBindingExamples"
      ObjectType="{x:Type local:Customer}"
      MethodName="CreateCustomers"
      />
  </Grid.DataContext>

  <Grid.Resources>
    <!-- ORDER DETAIL TEMPLATE -->
    <DataTemplate x:Key="OrderDetailTemplate">
      <TextBlock>
        <Run>Product:</Run>
        <TextBlock Text="{Binding Path=Product}" />
        <Run>(</Run>
        <TextBlock Text="{Binding Path=Quantity}" />
        <Run>)</Run>
      </TextBlock>
    </DataTemplate>

    <!-- ORDER TEMPLATE -->
    <HierarchicalDataTemplate
      x:Key="OrderTemplate"
      ItemsSource="{Binding Path=OrderDetails}"
      ItemTemplate="{StaticResource OrderDetailTemplate}"
      >
      <TextBlock Text="{Binding Path=Desc}" />
    </HierarchicalDataTemplate>

    <!-- CUSTOMER TEMPLATE -->
    <HierarchicalDataTemplate
      x:Key="CustomerTemplate"
      ItemsSource="{Binding Path=Orders}"
      ItemTemplate="{StaticResource OrderTemplate}"
      >
      <TextBlock Text="{Binding Path=Name}" />
    </HierarchicalDataTemplate>
  </Grid.Resources>

  <TreeView
    ItemsSource="{Binding Path=.}"
    ItemTemplate="{StaticResource CustomerTemplate}"
    />

</Grid>
I assign a collection of Customer objects to the DataContext of a Grid that contains the TreeView. This is possible to do in XAML by using the ObjectDataProvider, which is a convenient way of calling a method from XAML. Since the DataContext is inherited down the element tree, the TreeView's DataContext references that set of Customer objects. That is why we can give its ItemsSource property a binding of "{Binding Path=.}", which is a way of indicating that the ItemsSource property is bound to the TreeView's DataContext.
If you did not assign the TreeView's ItemTemplate property, the TreeView would only display the top-level Customer objects. Since WPF has no idea how to render a Customer, it would call ToString on each Customer and display that text for each item. It would have no way to figure out that each Customer has a list of Order objects associated with it, and that each Order has a list of OrderDetail objects. Since there is no way that WPF can magically understand your data schema, you must explain the schema to WPF so that it can render the data structure properly for you.
Explaining the structure and appearance of your data to WPF is where HierarchicalDataTemplates fit into the picture. The templates used in this demonstration contain very simple visual element trees, mostly just TextBlocks with a small amount of text in them. In a more complicated application, the templates could have interactive rotating 3D models, images, vector graphic drawings, complex UserControls, or any other WPF content to visualize the underlying data object.
It is important to note the order in which the templates are declared. You must declare a template before referencing it via the StaticResource extension. This is a requirement imposed by the XAML reader, and it applies to all resources, not just templates.
It is possible to reference the templates by using the DynamicResource extension instead, in which case the lexical order of template declarations is irrelevant. However, using a DynamicResource reference, as opposed to a StaticResource reference, carries along with it some runtime overhead because they monitor the resource system for changes. Since we do not replace the templates at run time, that overhead is unnecessary, so it is best to use StaticResource references and properly arrange the order of the template declarations.

Working with User Input
For most programs, displaying data is only half of the battle. The other big challenge is analyzing, accepting, and rejecting data entered by the user. In an ideal world, where all users always enter logical and accurate data, this would be a simple task. In the real world, however, this is not at all the case. Real users create typos, forget to enter required values, enter values in the wrong place, delete records that should not be deleted, add records that should not be added, and generally follow Murphy's Law whenever humanly possible.
It is our job, as developers and architects, to combat the inevitable erroneous and malicious input entered by our users. The WPF binding infrastructure has support for input validation. In the next few sections of this article, I'll examine how to make use of the WPF support for validation, as well as how to display validation error messages to the user.

Input Validation via ValidationRules
The first version of WPF, which was part of the Microsoft® .NET Framework 3.0, had only limited support for input validation. The Binding class has a ValidationRules property, which can store any number of ValidationRule-derived classes. Each of those rules can contain some logic that tests to see if the bound value is valid.
Back then, WPF only came with one ValidationRule subclass, called ExceptionValidationRule. Developers could add that rule to a binding's ValidationRules and it would catch exceptions thrown during updates made to the data source, allowing the UI to display the exception's error message. The usefulness of this approach to input validation is debatable, considering that the bedrock of good user experience is to avoid unnecessarily revealing technical details to the user. The error messages in data parsing exceptions are generally too technical for most users, but I digress.
Suppose that you have a class that represents an era of time, such as the simple Era class seen here:
public class Era
{
    public DateTime StartDate { get; set; }
    public TimeSpan Duration { get; set; }
}
If you want to allow the user to edit the start date and duration of an era, you could use two textbox controls and bind their Text properties to the properties of an Era instance. Since the user could enter any text he wants into a textbox, you cannot be sure that the input text will be convertible to an instance of DateTime or TimeSpan. In this scenario, you can use the ExceptionValidationRule to report data conversion errors, and then display those conversion errors in the user interface. The XAML listed in Figure 11 demonstrates how to achieve this task.
Figure 11 A Simple Class that Represents an Era of Time
<!-- START DATE -->
<TextBlock Grid.Row="0">Start Date:</TextBlock>
<TextBox Grid.Row="1">
  <TextBox.Text>
    <Binding Path="StartDate" UpdateSourceTrigger="PropertyChanged">
      <Binding.ValidationRules>
        <ExceptionValidationRule />
      </Binding.ValidationRules>
    </Binding>
  </TextBox.Text>
</TextBox>

<!-- DURATION -->
<TextBlock Grid.Row="2">Duration:</TextBlock>
<TextBox
  Grid.Row="3"
  Text="{Binding
         Path=Duration,
         UpdateSourceTrigger=PropertyChanged,
         ValidatesOnExceptions=True}"
  />
Those two textboxes demonstrate the two ways that an ExceptionValidationRule can be added to the ValidationRules of a binding in XAML. The Start Date textbox uses the verbose property-element syntax to explicitly add the rule. The Duration textbox uses the shorthand syntax by simply setting the ValidatesOnExceptions property of the binding to true. Both bindings have their UpdateSourceTrigger property set to PropertyChanged so that the input is validated every time the textbox's Text property is given a new value, instead of waiting until the control loses focus. A screenshot of the program is seen in Figure 12.
Figure 12 ExceptionValidationRule Displays Validation Errors

Displaying Validation Errors
As seen in Figure 13, the Duration textbox contains an invalid value. The string it contains is not convertible to a TimeSpan instance. The textbox's tooltip displays an error message, and a small red error icon appears on the right side of the control. This behavior does not happen automatically, but it is easy to implement and customize.
Figure 13 Render Input Validation Errors to the User
<!--
The template which renders a TextBox
when it contains invalid data.
-->
<ControlTemplate x:Key="TextBoxErrorTemplate">
  <DockPanel>
    <Ellipse
      DockPanel.Dock="Right"
      Margin="2,0"
      ToolTip="Contains invalid data"
      Width="10" Height="10"
      >
      <Ellipse.Fill>
        <LinearGradientBrush>
          <GradientStop Color="#11FF1111" Offset="0" />
          <GradientStop Color="#FFFF0000" Offset="1" />
        </LinearGradientBrush>
      </Ellipse.Fill>
    </Ellipse>
    <!--
    This placeholder occupies where the TextBox will appear.
    -->
    <AdornedElementPlaceholder />
  </DockPanel>
</ControlTemplate>

<!--
The Style applied to both TextBox controls in the UI.
-->
<Style TargetType="TextBox">
  <Setter Property="Margin" Value="4,4,10,4" />
  <Setter
    Property="Validation.ErrorTemplate"
    Value="{StaticResource TextBoxErrorTemplate}"
    />
  <Style.Triggers>
    <Trigger Property="Validation.HasError" Value="True">
      <Setter Property="ToolTip">
        <Setter.Value>
          <Binding
            Path="(Validation.Errors)[0].ErrorContent"
            RelativeSource="{x:Static RelativeSource.Self}"
            />
        </Setter.Value>
      </Setter>
    </Trigger>
  </Style.Triggers>
</Style>
The static Validation class forms a relationship between a control and any validation errors it contains by the use of some attached properties and static methods. You can reference those attached properties in XAML to create markup-only descriptions of how the user interface should present input validation errors to the user. The XAML in Figure 13 is responsible for explaining how to render input errors messages for the two textbox controls in the previous example.
The Style in Figure 13 targets all instances of a textbox in the UI. It applies three settings to a textbox. The first Setter affects the textbox's Margin property. The Margin property is set to a value that provides enough space to display the error icon on the right side.
The next Setter in the Style assigns the ControlTemplate used to render the textbox when it contains invalid data. It sets the attached Validation.ErrorTemplate property to the ControlTemplate declared above the Style. When the Validation class reports that the textbox has one or more validation errors, the textbox renders with that template. This is where the red error icon comes from, as seen in Figure 12.
The Style also contains a Trigger that monitors the attached Validation.HasError property on the textbox. When the Validation class sets the attached HasError property to true for the textbox, the Style's Trigger activates and assigns a tooltip to the textbox. The content of the tooltip is bound to the error message of the exception thrown when attempting to parse the textbox's text into an instance of the source property's data type.

Input Validation via IDataErrorInfo
With the introduction of the Microsoft .NET Framework 3.5, WPF support for input validation vastly improved. The ValidationRule approach is useful for simple applications, but real-world applications deal with the complexity of real-world data and business rules. Encoding business rules into ValidationRule objects not only ties that code to the WPF platform, but it also does not allow for business logic to exist where it belongs: in business objects!
Many applications have a business layer, where the complexity of processing business rules is contained in a set of business objects. When compiling against the Microsoft .NET Framework 3.5, you can make use of the IDataErrorInfo interface to have WPF ask business objects if they are in a valid state or not. This removes the need to place business logic in objects separate from the business layer, and it allows you to create UI platform-independent business objects. Since the IDataErrorInfo interface has been around for years, this also makes it much easier to reuse business objects from a legacy Windows Forms or ASP.NET application.
Suppose that you need to provide validation for an era beyond just ensuring that the user's text input is convertible to the source property's data type. It might make sense that an era's start date cannot be in the future, since we do not know about eras that have yet to exist. It might also make sense to require that an era last for at least one millisecond.
These types of rules are similar to the generic idea of business logic in that they are both examples of domain rules. It is best to implement domain rules in the objects that store their state: domain objects. The code listed in Figure 14 shows the SmartEra class, which exposes validation error messages via the IDataErrorInfo interface.
Figure 14 IDataErrorInfo Exposing Validation Error Messages
public class SmartEra
    : System.ComponentModel.IDataErrorInfo
{
    public DateTime StartDate { get; set; }
    public TimeSpan Duration { get; set; }

    #region IDataErrorInfo Members

    public string Error
    {
        get { return null; }
    }

    public string this[string property]
    {
        get
        {
            string msg = null;
            switch (property)
            {
                case "StartDate":
                    if (DateTime.Now < this.StartDate)
                        msg = "Start date must be in the past.";
                    break;

                case "Duration":
                    if (this.Duration.Ticks == 0)
                        msg = "An era must have a duration.";
                    break;

                default:
                    throw new ArgumentException(
                        "Unrecognized property: " + property);
            }
            return msg;
        }
    }

    #endregion // IDataErrorInfo Members
}
Consuming the validation support of the SmartEra class from a WPF user interface is very simple. The only thing you have to do is tell the bindings that they should honor the IDataErrorInfo interface on the object to which they are bound. You can do this in one of two ways, as seen in Figure 15.
Figure 15 Consuming the Validation Logic
<!-- START DATE -->
<TextBlock Grid.Row="0">Start Date:</TextBlock>
<TextBox Grid.Row="1">
  <TextBox.Text>
    <Binding Path="StartDate" UpdateSourceTrigger="PropertyChanged">
      <Binding.ValidationRules>
        <ExceptionValidationRule />
        <DataErrorValidationRule />
      </Binding.ValidationRules>
    </Binding>
  </TextBox.Text>
</TextBox>

<!-- DURATION -->
<TextBlock Grid.Row="2">Duration:</TextBlock>
<TextBox
  Grid.Row="3"
  Text="{Binding
         Path=Duration,
         UpdateSourceTrigger=PropertyChanged,
         ValidatesOnDataErrors=True,
         ValidatesOnExceptions=True}"
  />
Similar to how you can add the ExceptionValidationRule explicitly or implicitly to a binding's ValidationRules collection, you can add the DataErrorValidationRule directly to the ValidationRules of a binding, or you can just set the ValidatesOnDataErrors property to true. Both approaches result in the same net effect; the binding system queries the data source's IDataErrorInfo interface for validation errors.

Wrapping Up
There is a reason why many developers say that their favorite feature of WPF is its rich support for data binding. The use of binding in WPF is so powerful and pervasive that it requires many software developers to recalibrate their thinking around the relationship between data and user interfaces. Many core features of WPF work together to support complex data-bound scenarios, such as templates, styles, and attached properties.
With relatively few lines of XAML, you can express your intentions for how to display a hierarchical data structure and how to validate user input. In advanced situations, you can tap into the full power of the binding system by accessing it programmatically. With such a powerful infrastructure at our disposal, the perennial goal of creating great user experiences and compelling data visualizations is finally within reach for developers creating modern business applications.

Josh Smith is passionate about using WPF to create great user experiences. He was awarded the Microsoft MVP title for his work in the WPF community. Josh works for Infragistics in the Experience Design Group. When he is not at a computer, he enjoys playing the piano, reading about history, and exploring New York City with his girlfriend. You can visit Josh's blog at joshsmithonwpf.wordpress.com.


Transactions
Build Scalable Systems That Handle Failure Without Losing Data
Udi Dahan

This article discusses:
  • Durable messaging
  • Transactions and system consistency
  • Handling problems with error queues
  • Message size and timing
 This article uses the following technologies:
WCF, MSMQ
Contents
HTTP and Message Loss
Durable Messaging
Systems Consistency
Transactional Messaging
Transient Conditions
Deserialization Errors
Messages in the Error Queue
Time and Message Loss
TimetoBeReceived
Call Stack Problems
Large Messages
Small Messages from Large
Idempotent Messaging
Long-Running Processes
Learning from Mistakes
Designing distributed systems has never been easier. With the increasing power coming with every version of the CLR, the productivity of Visual Studio®, and the fine-grained control found in frameworks like Windows® Communication Foundation (WCF), developers have all the tools they need to build scalable systems. Unfortunately, it is not enough.
When working on a large-scale distributed system, my team found out that developing robust systems that handle failure scenarios without losing data is far from trivial. It's not a failing in the toolset, though; these tools need to be employed in very specific patterns to achieve systems that are both scalable and robust.

HTTP and Message Loss
The first challenge the team faced was message loss. When first designing our system, we had decided on HTTP as the transport for our WCF-based services. As a part of the message processing, our services would often write data to the database. Not to belabor the point, but many systems are designed this way.
Before taking our system to the staging environment, rigorous stress testing was done—the system was run under heavy load for a period of one week. The result of the stress testing showed that our system lost messages.
During the analysis of what had occurred, we identified that the messages lost were primarily those containing order information—not surprisingly since most of the load on the system had to do with order information. However, we were surprised that all of this message loss occurred during one 10-minute interval. After that, the system operated normally.
After going through various log files, we discovered that, during that week, a critical Windows patch was issued and the servers in the staging lab restarted themselves after automatically installing the patch. When considering the implications of this revelation—servers do restart every once in a while—we realized that we couldn't ignore this. The system would need to be operational for several years and needed to weather many server upgrades and restarts.
When processing a message over HTTP, a service would open a transaction against the database, attempt to write the data, and commit the transaction. Under regular circumstances this would succeed. If the message-processing server restarted mid-transaction, the database would detect the timeout of the transaction and roll back its changes, maintaining its consistent state. However, when the server would start up again, the data of the original message would be available neither in memory nor in the network processing stack of the server—that message would be lost.
For order processing systems, this would mean losing money. One can barely entertain the nightmarish thought of this scenario occurring in air-traffic control systems should the event "aircraft A and B are on a collision course" be lost.

Durable Messaging
After deciding to move from HTTP to a durable transport, we had to choose between Microsoft® Message Queue (MSMQ) and SQL Server® Service Broker. The advantage of Service Broker over MSMQ was that the messaging could be kept in the database, avoiding distributed transactions between the messaging layer and the database, thus resulting in potentially higher performance. The advantage of MSMQ over Service Broker was its availability on Windows directly (at no cost), the tight integration with WCF, and the ability to do peer-to-peer communication between servers without going through the already overloaded database. In the end, we decided on using private MSMQ queues on each of our servers.
Note that using MSMQ does not automatically cause every message to be written to disk—which is a good thing as it comes with a performance overhead. In order to define that a message be durable, when using MSMQ directly set the Recoverable property of the Message class to true or use NetMsmqBinding since by default it has the Durable property set to true.
When using NetMsmqBinding across an entire solution, make sure that messages not requiring durable messaging are transferred on separate endpoints. Configure those endpoints as non-transactional queues and set the Durable property to false in the <netMsmqBinding> section.
In order for the queue to roll back the message in case of failure, the Receive operation must be performed within a transaction. This is done quite simply by using the TransactionScopeRequired property of the OperationBehaviorAttribute—with a queued transport, the transaction used to dequeue the message is used to process the message.

Systems Consistency
As a result of processing a message, a service can reply with one or more messages as well as send any number of other messages to other systems. The flexibility gained by being able to have any code communicate via messaging at any point in the service is extremely valuable. However, doing this with HTTP, TCP, or any other non-transaction-aware technology is dangerous.
Since the transaction around processing a message may be rolled back as a result of a database deadlock, we may be sending other systems information that will be untrue once the transaction rolls back. Consider the scenario illustrated in Figure 1. Deadlocks can occur in databases where multiple threads work with data from the same tables. One thread successfully locks table 1 and attempts to lock table 2. At the same time, a different thread successfully locks table 2 and attempts to lock table 1. The database detects this deadlock, selects one thread as a victim, and aborts its transaction.
Figure 1 Transaction Encounters a Database Deadlock
The problem here is that we have notified system A about a change to data in table 1 that has been rolled back. This may, in turn, lead to system A changing its internal data and further propagating misinformation, ultimately leading to global inconsistency.
In the order-processing scenario, the behavior we witnessed was the following:
  • Purchase Order arrives.
  • Purchase Order table updated (status received).
  • Warehouse System notified about pending order.
  • Customer Relationship Management (CRM) database accessed to update value of customer.
  • CRM database deadlock detected.
  • Transaction rolled back.
When the transaction was retried, the customer data indicated that no business was to be done with that customer, as it had defaulted on its payments. Consequently, the ordering system was left in a consistent state, as was the CRM database.
Unfortunately, the warehouse system was not notified about the other databases rolling back—how could it have been? This resulted in produce being taken out of refrigeration and sitting on the loading docks for more than an hour until someone called and check what was going on with the order. The produce could not be sold at that point, and the time it took to replenish the inventory led to several other orders being delivered hours late.
The cost of that single incident brought down that week's profit by 30 percent and brought these down-in-the-weeds technical issues into the core business spotlight. When we transitioned to MSMQ, these problems were solved without any development effort on our part, as I'll explain next.

Transactional Messaging
When sending a message to a queue using MSMQ, the message is not sent when the method call returns. This is very different from the behavior HTTP and other connected technologies exhibit. When using MSMQ, before a message is sent it is stored locally on the same machine in an outgoing queue. When the message is sent within the context of a transaction, only after the transaction commits is the message released so that MSMQ can actually send it.
In the previous example, when the ordering system sent a message to the warehouse system notifying it about the new order, that message was not sent right then. Only when the entire message processing transaction committed could MSMQ send the message to the warehouse system. Should the transaction abort as the result of a database deadlock or for any other reason, the message to the warehouse system would have been deleted from the MSMQ outgoing queue. In other words, in order to prevent a total breakdown of system consistency, transactional messaging is needed.
Note that transactional messaging does not imply that both the sender of a message and the service processing that message share a transaction. The fact that a message arrives at a service for processing implies that a separate transaction at the sender committed successfully. However, the fact that transaction was committed says nothing about what may transpire when the receiving service handles that message.
At times, a message may arrive at a service where the processing of that message results in an exception. There are many possible root causes for this exception, and it's important to understand the differences between them and how to handle them. Sometimes an exception may occur only once or twice in the processing of a message; other times it appears that no matter how many times the service will try to process that message, it will always fail.
A message that is identified as belonging to the second group is called a poison message and requires special treatment. As I go through the various root causes, you'll see that a single simple solution is enough to handle them all effectively.

Transient Conditions
You've already seen that a well-formed message whose processing would otherwise succeed may result in an exception due to database deadlocks, or perhaps the database connection pool is maxed out handling other transactions. What is common to all these scenarios is that the exception results from transient environmental conditions.
Since the default behavior in transactional messaging when an exception bubbles up through the transaction scope is to roll everything back, the original message returns to the queue. At that point, a thread—possibly the same thread that had just been hit by the exception—can process it again. Since rolling back a transaction takes some time, there is a high likelihood that the previous environmental conditions have changed now allowing the message processing to transactionally complete successfully.
One often-overlooked case where message processing fails is quite simply that the database used by the service is unavailable—not all databases have five nines of availability. This problem is non-transient from the perspective of the code using the database—it is not known for how long the database will be unavailable. Simply rolling back the message and trying again will most likely result in the same exception.
The message itself may be perfectly valid and may contain very valuable data like, say, a million dollar purchase order that you wouldn't want to lose. Letting the message stay stuck in the processing-rollback loop is one way of keeping the message around, but it causes other problems.
Consider the case where the service logic makes use of more than one database or even makes use of more than one table. One database may be available while the other isn't. The data file backing one table may be available while the file backing the other table may be on a remote server that is down. Now consider that processing messages of type A only needs table A, and messages of type B only need table B. What will occur when table B (or database B) becomes unavailable?
Messages of type A will be processed successfully, but messages of type B will keep rolling back. This will continue until there are more messages of type B in the service's input queue than there are threads processing messages. At that point, the service effectively becomes unavailable even for messages of type A until all resources come back online. In other words, the service is only as available as its least available resource—a worrisome (and costly) proposition from a systems availability perspective.
The most intelligent thing that a service can do after a message has consistently failed to be processed multiple times is to move the message out of the input queue into some other queue—let's call that an error queue for now, yet there's nothing inherently different about that queue than any other queue. In this scenario, when one resource goes offline, messages of type B will be moved to the error queue after rolling back n times, but messages of type A will continue to be processed successfully. While the service's latency and throughput may be hurt by that resource going down, the service will continue to be available.

Deserialization Errors
When a bunch of bytes arrives at an endpoint, the service's underlying technology tries to turn those bytes into a regular object—a message. This deserialization process can fail for a number of reasons. It could be that the data type that the XML instructs the technology to create hasn't been deployed at that endpoint. It could be that the wrong version of that data type was deployed. Another more common situation is that an existing client is sending messages to an endpoint of a service that has been recently upgraded and that the new version of the data type isn't compatible with the previous version.
The problem is that although the service may have not been able to understand the message sent to it, there may be valuable data in that message and, as such, you don't want to throw the message away. You also wouldn't want it to clog up the service's input queue.
The solution in this case is quite similar to the one that is used for database unavailability: simply move the message to a different queue, possibly even the exact same error queue. The slight difference in this case is that there is no uncertainty as to the future result of processing this message—it will always result in an exception. Being such, it is unnecessary for you to wait until the message has rolled back n times before moving it to the error queue. A message that fails deserialization should just be moved directly to the error queue.

Messages in the Error Queue
From the scenarios described earlier you can see that various messages can be identified as poison messages in many circumstances. Some of these messages may indeed be garbage that is best thrown away, yet the only definitive way to make that decision is by having a human operator examine these messages.
In the case of a deserialization exception, the operator may route the message to an endpoint running a previous version of the software or possibly reformat the message to the new version and return it to the input queue.
Hopefully, operators would know about databases going offline, but if a single data file goes bad and only a few tables are affected, messages in the error queue, along with the corresponding log files, can raise a flag about these hidden issues so that they can be identified and resolved quickly.
What can be learned from the topic of poison messages is that the time it takes a service to respond to a message sent to it can be quite long. In fact, the issue of time becomes a primary concern in designing message-based systems—sometimes a late response might as well never have come at all.

Time and Message Loss
While you might think that by using durable and transactional messaging, message loss can be mitigated entirely, the reality, as always, is more complicated than that.
Let's consider a business-to-business scenario where our ordering system is communicating with an external shipping company to take care of the shipment of orders to customers. For every order we accept, a message needs to be sent to the shipping system. In order to prevent messages from getting lost if either system goes down, we use durable messaging along with transactional handling of messages to transfer the order information.
As we analyze the specifics of this scenario, we'll see that message loss is not only inevitable but cost effective. If we take an average message size of 1MB for each order sent, and our order system is processing 10 orders per second (nothing too heavy), and the shipping servers become unavailable, these messages will be durably stored in an outgoing queue. In terms of I/O usage, that's 10MB per second, or 600MB per minute.
This scenario was only noticed after a communication outage of three hours where our systems tried to write over 100GB of data to a partly full three-disk RAID 5 array of 36GB SCSI disk. Servers tend to lose stability when they aren't able to write to their hard drives anymore—not only that, it's almost impossible to bring them back up.
The three hours of backlogged orders, despite being the source of the problem, were not our main concern at that point. The fact that for the two hours that it took us to get the system back up we were losing orders to the tune of several hundred thousand dollars—well, that was enough to get serious management pressure to both solve the immediate problem and prevent it from occurring again. The lesson we learned was that durable messaging is no silver bullet and has an associated cost.
To maintain stability, our servers would need to throw away messages that had not yet been sent to their destination successfully. The difficulty was in deciding which messages should be kept and for how long.

TimetoBeReceived
The order system received requests from many other systems, published events, received events from other systems—each of these resulting in messages either in incoming queues or in outgoing queues. Some services made use of data coming in on a feed where the validity of the data was only a few minutes. It didn't make sense to keep those messages around for any longer than that—even if it took longer than that for all server processes (IIS, SQL Server, other Windows services) to come up. The same thing was true for data feeds that our systems were putting out—if a subscriber was offline for longer than the time the data in a message was valid, it didn't make sense for that message to take up space in our outgoing queues.
As we went through the various messages that made up each service's contract, we noticed that for those messages whose data didn't necessarily have a period of validity, the service had response time requirements as a part of its service-level agreement. In other words, if the service wasn't able to finish processing a message in a given period of time (including the time the message waited in a queue), that service was already malfunctioning from a business perspective.
This meant that a period of validity could be defined for each message type. This time needs to take into account time spent in the sender's outgoing queue, time in transit, and time spent in the receiver's queue. MSMQ allows us to define this value using the TimeToBeReceived property of the Message class (see the "Inside TimeToBeReceived" sidebar for some implementation details).
At this point it was clear the message loss was an unavoidable reality. From a client-side perspective, a message that is lost, a lost response, or just a slow server are indistinguishable from each other. If our service employs other services as clients as a part of our message processing, then to ensure appropriate response times as indicated by our service-level agreement, our service needs to protect itself. Our decision was to employ timeouts so that if responses don't arrive within a specified time, our service can at the very least respond with a message indicating that the request was received and that a response will be sent later, possibly via e-mail if we were interacting with a user.
Inside TimeToBeReceived
As we began looking at how programmers would be working with these APIs, we realized that the value of TimeToBeReceived would be defined as a part of the service's contract and that a framework could access this metadata when sending amessage, saving the programmer from specifying this every time he sent a message.
We defined a TimeToBeReceivedAttribute that could be placed on classes as a part of the message contract (though WCF data contracts are just as suitable). As a part of the implementation of the SendMessage method, the framework would access the type of the message object passed and loop over the custom attributes of that type using reflection. After getting the data from the attribute, the framework passed that data to the TimeToBeReceived property of the MSMQ message that was sent.

Call Stack Problems
Another technical issue that came up was the fact that we had the state of the interaction with the external system represented simply as the call stack of the thread managing the process, as shown in Figure 2.
Figure 2 Interaction with External Systems
The technical problem had to do, once again, with servers restarting. If a thread was waiting for a response from either our internal CRM system or the external shipping partner system when the server restarted, that state would be lost. In other words, if the system was supposed to wait a maximum of five minutes for a response, and the server with the knowledge of how much time had elapsed restarted, and even got back to handling requests within that time window, the system would not know that it needed to notify the business activity monitoring (BAM) system.
Given that the reason a response may not have arrived may have been that the original request was lost or thrown away, this scenario would cause an occasional order to get stuck. The system could not proceed without a response from some other system, but forgot that it was waiting for a response because of a restart.
In retrospect, this didn't occur very often—at the time, we didn't know that it was happening at all. The BAM system wasn't showing anything problematic. However, when the COO shows up in the IT department, furious that an urgent order from a strategic partner wasn't delivered on time and that nobody even knew about it, well, we were in for another round of refactoring to solve that problem.
It turns out that making these kinds of inter-system integration processes robust requires joining state management, messaging, and time into one unified model.

Large Messages
Before discussing how long-running processes connect to the solution, it's important to understand the effect of large messages on system performance and availability.
One of the most difficult performance problems that we ran into was caused by messages received from strategic partners—our most profitable market segment. While the purchase orders received from most of our customers were fairly small, our strategic partners would send orders for many items and many kinds of each item. The difficulty was compounded by the requirement for item customization—something that our strategic partners did often. Together, this resulted in strategic partners sending us messages that could easily be between 20MB and 50MB of XML.
Just the deserialization of those quantities of XML was enough to tie up a Xeon core for up to a minute. We tried offloading the schema validation to hardware XML appliances, but that didn't solve the overall problem—these huge messages were processed orders of magnitude slower than the linear multiplication of corresponding small messages:
TimeToProcess(BigMessage) >> N x TimeToProcess(SmallMessage) where
Size(BigMessage) == N x Size(SmallMessage)
We had always thought about scalability in terms of the ability to scale to larger numbers of messages per second, and that's what we had tested for. It turned out that scaling to larger message sizes is a much more difficult problem.
Network congestion issues barely affect small messages, but large messages are much more susceptible. The same goes for writing this data to disk when using durable messaging. While its fairly easy to find a contiguous strip of free disk to write a small message, after the system has been running for a while without defragmenting a disk, writing a large message can take much longer. As users, we're familiar with the 99 percent done phenomenon when copying large files between machines. Servers suffer the same problems.
Obviously while all this would be going on, smaller messages would be affected as well. Although less of an issue from a percent-profitability perspective, handling 10 percent fewer orders due to throughput problems is an unacceptable hit to the bottom line. We needed some way to turn one large message into many smaller messages.

Small Messages from Large
The problem was that the data flowing in those messages had uniqueness requirements from a business perspective. We can't allow duplicate purchase order numbers. When designing banking systems, transaction numbers need to be sequential. Parallelism can't be handled on a purely technological level—business rules need to be designed around this as well.
After analyzing the data in these large purchase orders and speaking with the development teams at our strategic partners, we discovered that these large purchase orders were built up gradually over time and were only sent when completed. The most interesting thing we heard was that these large messages were problematic on the sender side as well and that there was universal interest in finding ways to break them up into multiple small messages.
The problem was that the current interface and processing logic tied a single purchase order number to a single message. If a solution could be found where that connection could be broken, it was clear that the other problems could then also be solved.
A simple solution was devised, though it turned our previously stateless architecture into a statefull one—a decision not to be taken lightly. Stateless architectures keep all state in the database. Although simple to build and having good tool support, these architectures exacerbate the load on the database.
After building a prototype and putting it through a bout of stress testing we found that latency, throughput, and overall resource usage was so much improved that we decided to move forward with the architecture.
In the new architecture, our partners could send us multiple messages with the same purchase order number. Those messages would only include changes to data already sent or additional data. When all the data from the purchase order had been sent, the partner system would set a Boolean field indicating that the current message was the last one in the purchase order. In terms of the message contract, the change was really quite small—just an additional field. However, the order processing logic needed to be fundamentally changed.
Instead of regular request/response semantics, our service was now supporting multiple-request/single-response semantics. The logic didn't know how many requests it was going to receive for a given purchase order. Further, those requests were not to be processed until a request arrived with the purchase order complete field set to true. However, the data from those requests needed to be stored until that last request arrived.
We separated out the storage of the data at this stage so as to not increase the load on the database any further. After analyzing the data access patterns of this state, we discovered there was no need for rolling up reports on all purchase orders at that point. In other words, the choice of a relational database for storing the data was overkill in terms of transactional consistency guarantees and too slow to handle the high write-to-read ratio of this kind of logic.
After moving to a distributed caching solution, performance greatly improved. (Distributed caching products keep data in memory on multiple servers for fault tolerance and high availability and offer a high-performance alternative to conventional database technologies.)
Although the static structure of the order processing contract did not change very much, on the other hand the dynamic behavior of its message exchange patterns was very different. Since our service could now accept the same purchase order in multiple messages, we could now leverage that enhanced protocol in infrastructure decision.

Idempotent Messaging
The original decision to use durable messaging for purchase orders from strategic partners hinged on preventing message loss. Unfortunately, we saw that the fact that a message was written to disk did not necessarily provide a 100 percent guarantee. However, with the new message protocol, if a partner sent us a purchase order message and did not get a message back from us in a reasonable period of time, they could just send the same message again.
Although we hadn't explicitly set out to create idempotent message contracts, their benefits were felt very quickly. (An idempotent message is one that can be sent multiple times to a service and the resulting service state is the same as if the message was processed just once.) Using durable messaging involved writing each message to disk. Even expensive high-performance disks are much slower than storing messages in memory. The performance gain was two-fold: not only were we sending and processing smaller messages, but the messages didn't need to be written to disk at all. The cost savings in terms of message storage can be significant.
There is an additional element of complexity that comes with idempotent messaging. Disregarding the difficulty in designing idempotent contracts, clients can no longer just fire-and-forget a message to a service. Clients need to manage timers on their end in order to send the same message again if a response does not come back from the server in time. In other words, not only do servers need to statefully process messages, but it is necessary for clients to statefully send messages.

Long-Running Processes
Another architectural shift that occurs when splitting up large messages into multiple smaller messages is that previously simple request/response semantics become long-running processes as well. A long-running process is one that handles multiple messages. Often, state from one message needs to be available to logic processing a subsequent message.
As previously discussed, the processing of a single message is a single transactional unit of work and includes changes to local state as well as sending out other messages. A long-running process handles multiple messages, each of which represents a transactional unit of work. When a response arrives from a previously sent request, it is handled in a unit of work all its own.
Between handling messages, though, there is no reason for the state of the long-running process to take up memory, so it is saved to some highly available store (typically a database or possibly a distributed cache). When a message containing the process ID arrives at an endpoint, the data from the highly available store is retrieved, and the long-running process is dispatched the message.
One other common requirement that leads to designing long-running processes is complying with response times while needing a response from an external system. As you've already seen, when communicating with an external system or even just processes on a different machine, there are many things that can go wrong—the machine may be down, the process may be down, the message may have been lost or thrown away.
Consider the previously described process of shipping an order to a customer. Our company may work with multiple shipping partners, yet one is usually a preferred partner. There are cases where our preferred partner's systems may be down, slow, or just not responding. In those cases, logic dictates that we should wait for up to five minutes for a response, and if no response arrives, to choose a different shipping partner. At times, we would ship only part of an order if certain products were unavailable, leaving our shipping process to continue processing the order when inventory would become available.
We previously introduced the problem with implementing these timers using regular in-memory constructs such as System.Threading.Timer. In the case of a server restart, the data about time elapsed would be lost. The challenge that we faced was to store the information about time in such a way that it would be highly available and fault tolerant. A related issue was how our process would be notified of a timeout event.
After realizing that a timeout event is just like any other business event in the system, and that business events were represented as messages arriving at our process, we decided to use messaging to handle the issue of time as well. Our long-running process would send a timeout message to a Timeout Manager Service. The message would contain the ID of the long running process and the date and time at which to be notified.
The Timeout Manager Service would maintain this data in some way, either in a database that it would poll at specific intervals or possibly just by keeping the message in a durable queue. When we reached the time specified in the message, the Timeout Manager Service would send that very same message back to the endpoint from which it came—the endpoint of the long-running process. The long-running process would handle that message just like any other—in its own unit of work. While handling the timeout message, the process would have all its state from all the previous interactions available.
When looking at this shipping process, the logic handling the timeout message would check whether a response had arrived from our preferred shipping partner. The reason for this check is that we may have received a response from the partner system just before the timeout had occurred, and now our process is waiting for the rest of the inventory to become available. In this manner, a timeout message is not necessarily an indication of something going wrong, but rather a wake-up call for our process. In a single process, we may make use of multiple timeouts each corresponding to a business requirement around response times and service-level agreements.
It is important to understand that these timeouts are not a technological phenomenon like those that arise from trying to contact a database or an HTTP Web service call timing out. These timeouts are business events that may well be days and weeks in the future.
After we began addressing time as a first-level concern in our interactions, we began to see that previously trivial synchronous, blocking request/response semantics became processes in their own right. Each interaction eventually had some restriction as to the maximum time it should take. Every time we thought that we had a simple request/response interaction on our hands, we asked ourselves, "What if it takes two days to get a response back? What if it takes two weeks? Would our system still comply with its service-level agreement?" The result was the always the same: time cannot be ignored.

Learning from Mistakes
Developing reliable services is not easy. The problems our team faced may have appeared to be a failing of the technology at first, but the solutions often came from re-examining the problem domain and taking into consideration the harsh realities systems face in a production environment. We iterated through the project, trying durable messaging technology, adding full unit-of-work transactional message handling, handling versioning issues and poison messages, splitting up large messages, and finally addressing time robustly using long-running process. One thing became clear; the core patterns were the same:
  • Never assume that just because you sent a request you'll get a response.
  • Time is a business event and like other events can be handled like any other messages.
  • The handling of any message—a request from a client, a response back from another system, the arrival of a published message to which we had previously subscribed—is a single unit-of-work encompassing all local activities and the sending of messages as well.
The mindset that served us well at every decision point was "how could this fail?" This lead to good judgment around the choice of technologies and how to design service contracts for statefull interactions. Good judgment comes from experience, and experience comes from bad judgment. Only after we had been burned by our previous design decisions did our vision become clearer as to how our design-time decisions impacted runtime behavior.

Udi Dahan is The Software Simplist, an MVP and Connected Technologies Advisor working on WCF, WindowsWF, and "Oslo." He provides training, mentoring, and high-end architecture consulting services, specializing in service-oriented, scalable, and secure .NET architecture design. Udi can be contacted at www.udidahan.com.


WCF P2P
How To Design State Sharing In A Peer Network
Kevin Hoffman

This article discusses:
  • Fixed state servers
  • Elected state servers
  • Serverless data blast
  • Nearest peer synchronization
This article uses the following technologies:
WCF

Code download available at: P2P2008_07.exe (171 KB)
Browse the Code Online

Contents
Fixed State Server
Transient Elected State Server
Pure Peer Serverless Data Blast
Nearest Peer Synchronization
The WeSpend Sample Application
Next Steps
In recent years, people have started to discover the amazing power and functionality that can be provided by an application that communicates over a peer network. So-called peer applications cover the gamut from simple file sharing to instant messaging (IM) to full-on collaborative applications such as shared white boarding, Voice over IP (VoIP) calling and conferencing, social networking, and much more.
Think about how your Xbox 360® automatically detects the presence of another computer running Windows Media® Center and can immediately begin sharing audio and video with that computer, allowing you to play music from your computer on your Xbox®. Wouldn't it be great if business applications sported the same kind of peer awareness and capability? Think about how pleased your customers would be if someone created a new record from their desktop and everybody else within the organization using that application immediately saw that new record and you didn't have to install and configure a central server to make that happen.
This article is all about peer-enabling business applications by allowing them to share state in a serverless peer network. Before getting into the details of a sample app, I want to talk about some of the various ways that people typically try to share state in a peer network, as well as the pros and cons of those approaches. Finally, I'll show you a sample application called WeSpend, a tool for tracking your bank book ledger. This application automatically synchronizes data with all other instances of itself on your home network.
There are as many ways to implement a peer network as there are potential application ideas. Each implementation carries with it certain benefits and drawbacks. Some of those aspects are related to infrastructure, some are related to the ease with which the solution can be maintained over time, and yet others are related to how easily the solution can be developed initially. In this section I take a look at some of the most common ways people develop peer networks.

Fixed State Server
When many development teams get past the initial concept phase of their peer-enabled application and start looking at implementation, one of the first questions that arises is usually: how do we share state?
One of the simplest solutions is to create a hybrid peer network. This is where all broadcast and peer traffic takes place on the peer mesh, but there is a well-known central server that is maintaining state, often through a Web service or another Windows® Communication Foundation (WCF) service. This allows the peers on the network to be aware of the singleton state for the entire application, as well as being able to take advantage of optimized peer mesh communication.
The most common examples of this type of network are social networking and IM applications. IM applications often require a central server that is maintaining the list of who is logged in, their presence and availability, and the buddy lists for all of the registered users (see Figure 1). Communication between users on the network takes place in peer-to-peer fashion. There is an extremely useful and powerful split of functionality between classic client-server and peer-to-peer features.
Figure 1 Central State Server in a Peer Network
The benefit of this pattern is that it is easy to implement. The peer networking component has low complexity, and communication with a central server has been made extremely easy in recent years with the Microsoft® .NET Framework and WCF.
The drawback is that the infrastructure requires a central server. The peer applications need to know the address of that server, which could cause a configuration and maintenance problem. Also, many application developers don't have the resources to deploy central state servers globally, and the requirement of a central state server is often something that cannot be accommodated by an application's specifications.

Transient Elected State Server
The next step in the evolution of shared state patterns is to alleviate the infrastructure concern from the previous pattern by electing one of the nodes in the peer mesh to be the state server. You often see this pattern in networked real-time strategy games where the person who initiated the game has the "one true copy" of game state, and everyone else has synchronized duplicates.
This pattern works by using an algorithm that elects a node to be the state server for the mesh. Once this node is the state server, all the other nodes in the mesh follow the same pattern as in the fixed state server scenario for updating and querying singleton state.
While this removes the problem of requiring a fixed central server, it trades that problem for the complexity of an elections system. To implement this peer network, you need to determine how you elect the state server. One really easy solution is to pick either the oldest node (the one in the mesh longest) or the youngest node. The problem arises when nodes leave the mesh. What happens if your elected state server goes offline? If the mesh is aware of it being offline, the peers can elect a new state server. But what happens if the state server is just really slow, hasn't gone offline, and the mesh hasn't decided to re-elect?
As you can see, the drawback of this solution is that you are trading the infrastructure requirement of a central server for the potentially project-halting complexity of building an elections system that not only picks the best candidate to be state server but is resilient enough to deal with the state server becoming a black hole (messages come in but don't come out, which is slightly different from simply being offline) or the state server going offline.
If the complexity of the election system is actually handled well, this can be an powerful and versatile system. If the election system is done poorly, this kind of network will fail miserably and cause untold problems with state sharing.

Pure Peer Serverless Data Blast
The first two scenarios attempt to provide a solution in which there is one and only one authoritative copy of state. Whether the location of the state server is fixed or elected, the means by which peers query and manipulate state remains the same: contact with a single remote server.
A popular alternative to the central state server is what I call the "data blast" network. In this scenario, peers broadcast copies of their data to all other peers in the mesh each time a new peer joins the mesh or when a peer specifically asks for a data update. Each time a peer receives a broadcast of state data, they compare the contents of the broadcast with local state and update accordingly.
There are two rather significant downsides to this approach. The first is that the amount of redundant data being sent across the network can become extremely large. The more computers on the peer network, the more bandwidth will be wasted. In fact, the amount of redundant data transmission in this scenario actually grows exponentially as the number of nodes in the mesh increases.
The other downside to this approach is that, since each application is broadcasting a copy of its own data, the reception of a broadcast piece of data must now perform a synchronization operation to compare the received data against local data and reconcile the differences. As the size of the peer network increases, it could reach a point where it takes longer to process a state broadcast than it does for the next message to arrive, creating a bottleneck and potentially degrading the performance of the application itself. Data sent in broadcasts must have some way of being identified as unique, because each node in the peer mesh will need to check to see if the data is duplicated.
The trade-off for the somewhat wasteful use of network resources is that there is no central state server and you don't incur the complexity penalty that an election pattern carries with it. In some situations, especially when the number of nodes is known to never grow above a certain size, the duplicate data transmissions are worth the reduced implementation effort.

Nearest Peer Synchronization
When researching the various ways in which an election scenario might be implemented, I discovered that there is an attribute in WCF that allows you to indicate the maximum number of hops that a particular message will travel. After seeing this, it became obvious that there was a means for sharing state in a peer network that not only required no central server, but was resilient to node drop-off and did not require election. I call it Nearest Peer Synchronization.
The route any message takes between any two nodes in a peer mesh is entirely dictated by WCF. This allows WCF to optimize the communication paths in the network. Knowing this, and knowing that you can limit the distance a message travels to only one hop, it's possible to implement a neighbor synchronization pattern for state sharing.
First, after joining a mesh, the newly joined node will send out a one-hop message requesting a list of unique record identifiers for shared state records. All nodes in the mesh that are one hop away will receive this message. These nodes will then use the callback contract to reply directly to the mesh. The reply contains a list of the unique IDs for all records of data being maintained by that node. The first one to reply wins and all of the other replies are ignored. Keep in mind that you have to write the code that ignores duplicate messages. WCF won't do that for you.
The newly joined node then compares that list of unique IDs with its own list of unique IDs (because it has probably maintained offline state on disk and has a partial view of shared state). The unique IDs received in the reply that are not contained in local state are then sent out to the mesh in a one-hop message. The first node to reply to that message (again, using the callback contract) replies with the details for each of those unique IDs. The details are usually a fully constituted, serializable object.
The beauty of this pattern is that it is automatically able to correct for the disappearance of faulty nodes. If a node in the mesh crashes, it will not respond to requests for data. All other one-hop nodes then have a chance to reply to that data. If the mesh is organized such that the newly joining node only communicated with the one node that failed, WCF now considers nodes that used to be two hops away to be one hop away. There is no election logic required, and automatic failover is taken care of directly in the WCF peer channel layer without you having to write any additional code. Because each neighbor is guaranteed to be synchronized with each neighbor's state, state propagates automatically among all peers in the mesh. Figure 2 is an illustration of a pure, serverless peer mesh in which nodes auto-sync with neighbors.
Figure 2 A Peer Mesh with No State Server

The WeSpend Sample Application
To put these ideas into action, let's look at a simulation of a personal finance manager. It allows the user to keep track of financial transactions such as checks written or debit transactions made against a checking account.
WeSpend is a collaborative peer application because any other copy of the application running on the network automatically shares its state with other copies. The user-perceived effect of this is that when a family member downstairs enters a transaction in WeSpend, another family member upstairs immediately sees the new transaction. When a third person in the same home starts a fresh copy of WeSpend, he automatically receives transactions created when his application was not active. Adding this kind of peer awareness and collaborative capability could make the difference between a fantastic, popular application and a simply mediocre application.
WeSpend uses the nearest-neighbor synchronization shared state pattern. There are several key technological components of WCF that make this application possible. The first, which has already been mentioned, is an attribute called PeerHopCount that allows the contents of a message to only be transmitted a fixed number of times before being dropped. This attribute can turn a property of a data message into something very similar to the IP concept of time to live (TTL).
Here is a sample of a WCF message contract in the WeSpend application that includes a field controlling the peer hop count. It is important to note here that the peer hop count attribute is only used by the NetPeerTcpBinding protocol binding. The attribute is ignored by all other bindings:
[MessageContract]
public class TransactionIdRequest {
    [MessageBodyMember]
    public string Requester;
    [PeerHopCount]
    public int Hops;

    public TransactionIdRequest() {
        Hops = 1;
    }
}
When you put this message out onto the mesh, you don't get the normal flood behavior that sends the message to every node available. Instead, the message is sent out and delivered once and then the message ceases to exist. This allows you to request a list of transaction IDs only from the closest neighbor in the mesh.
Another key component that makes this scenario possible is the use of callback contracts. When a method is invoked on a WCF peer by another peer, the target code can call back to the peer mesh that initiated the method call. This allows a single node to reply to a neighbor request and obviates the need to host separate WCF services on every peer node. Here is a sample of the code that replies to a request for transaction IDs using a callback contract:
public void RequestTransactionIds(TransactionIdRequest request) {
    List<Guid> outboundIds = new List<Guid>();

    foreach (Transaction tx in ModelRoot.Current.Transactions) {
        outboundIds.Add(tx.TransactionID);
    }

    CallbackChannel.TransactionIdsReply(new TransactionIdReply() {
        ReplyFrom = AppController.Current.Username,
        TransactionIDs = outboundIds
    });
}
Here the callback channel is being used to reply to the peer that initiated the method call RequestTransactionIds. Remember that the TransactionIdRequest message is limited to one hop only, so the initiating peer will not be inundated with replies even in the largest of peer networks.
When a peer node receives a list of transaction IDs (by having its TransactionIdsReply method invoked), it uses the callback contract again to request details for those IDs that it does not already have in local storage, as shown in Figure 3.
Figure 3 Requesting Transaction Details
public void TransactionIdsReply(TransactionIdReply reply)
{
    // Compare the list of TX IDs with the list of IDs in local storage
    // for each item that is "missing" from local storage, put that item
    // in the request for details.

    List<Guid> output = new List<Guid>();
    foreach (Guid id in reply.TransactionIDs)
    {
        if (!ModelRoot.Current.Transactions.Any(
                tx => tx.TransactionID.Equals(id)))
        {
            output.Add(id);
            Console.WriteLine("[LTS] Need to request detail for TX " +
                id.ToString());
        }
    }
    CallbackChannel.RequestTransactionDetails(output);
}
When the other half of the synchronizing pair has its RequestTransactionDetails method invoked, it too uses the callback contract to reply to the mesh, as shown in Figure 4.
Figure 4 Replying with the Details
public void RequestTransactionDetails(List<Guid> transactionIds)
{
    List<LedgerTransaction> output = new List<LedgerTransaction>();
    foreach (Guid id in transactionIds)
    {
        Transaction tx = (from Transaction lt in
        ModelRoot.Current.Transactions
                         where lt.TransactionID.Equals(id)
                         select lt).FirstOrDefault();
        if (tx != null)
            output.Add(tx.ToLedgerTransaction());
    }
    CallbackChannel.TransactionDetailsAck(new TransactionDetailReply()
    {
        ReplyFrom = AppController.Current.Username,
        Transactions = output
    });
}
The property that has been used in the previous code listings, CallbackChannel, is defined as follows on the implementation of the WCF peer service contract:
protected ILedgerTransactionServiceChannel CallbackChannel {
    get {
        return OperationContext.Current.GetCallbackChannel<
            ILedgerTransactionServiceChannel>();
    }
}
Figure 5 illustrates the flow of messages between the peer that just joined the mesh and the closest neighbor to answer the call to initiate a synchronization operation.
Figure 5 Nearest-Neighbor Synchronization Order
One thing that is missing from the preceding method implementations is a barrier against re-entrant execution. In other words, if multiple peers want to start a synchronization operation at the same time, production code would need to guard against that. Also note that the code makes heavy use of the callback contract for the synchronization operations. The peer node will need to guarantee that no other peer nodes can synchronize at the same time, but this can be accomplished with a simple semaphore.

Adding peer networking capabilities to an application can dramatically improve the experience your customers have with your software. All too often, however, the perceived complexity of doing so is a barrier that can result in having peer networking features taken out of the product's design. Some implementations quickly become such maintenance nightmares that people remove peer networking from their specifications—the complexity simply spirals out of control.
The goal of this article is to show you that not only is peer networking possible and easy to implement, but you can even implement a serverless state sharing system that is efficient, fault tolerant, and easy to code. Also keep in mind that there is a nearly infinite supply of variations on the patterns discussed here.
I suggest you download the code sample and start experimenting with it. It works best when you have two computers and you modify the app.config to simulate two different user names. After launching one instance with the txhost application setting set to true (this means it creates a few sample transactions), launch the second instance with txhost set to false (doesn't automatically create transactions). You will see the transactions from the first instance automatically propagated to the second instance. When you click the button to create a new transaction, you will see that transaction appear on both running instances of the sample application. To see the synchronization effect even more clearly, quit the second instance and then restart it. When you restart the second instance, all of the transactions will show up. Depending on how Peer Name Resolution Protocol (PNRP) is working on your computer, it could take up to 30 seconds for the sample applications to start communicating for the first time.

Kevin Hoffman is a Research Developer at Liquidnet Holdings, Inc., who also blogs frequently at dotnetaddict.dotnetdevelopersjournal.com. He has always been passionate about networking, communications, and distributed application architecture. He has been working with WCF since the early days when it was still known by the codename "Indigo."


Editor's Note
Flex Your Data
Ted Neward

Do not attempt to adjust your magazine—everything is under control. Howard Dierking is away this month and we asked our guest editor, Ted Neward, to write the Editor's Note for this issue. Howard will return in time for Tech•Ed and the August issue. —Ed.
We've devoted this month's issue to the topic of data storage, manipulation, and access. Few topics in programming elicit debate the way questions of data access do. Whether the question surrounds the efficacies of the object/relational-mapper, the need for an ad-hoc query syntax, or the performance drawbacks of relational normal form, the data debate never disappoints.
The continuing advances in cheap storage and the increases in access speed mean that even individuals have access to more data than industry visionaries could have imagined just a few years ago. I recall wondering how I would ever fill up the first 1GB drive I owned. Now we're swiftly encroaching on the 1TB barrier.
The nature of that data has changed, too. Simple business information such as purchase orders, customer names, and the like once represented the majority of data storage. Now it includes personal information along with larger and more complex file formats such as photos, songs, and voicemails. Such data is not easily captured in the relational format. One of the challenges we face is the need to integrate this new kind of data into the traditional business space—how to index it, search it, and relate it to other critical data.
To meet the challenges, either relational databases will have to evolve to store (and allow queries on) these new data formats, or we will have to discover new non-relational ways to store data. David Robinson presents one such approach this month in his article on SQL Server Data Services.
From now on, data storage will not be confined to centrally located, managed, optimized relational databases. In fact, it's likely that more and more data will live on the edge of the network, either on the user's machine or in some other local store. Thus we'll need to devise new ways to query across devices and tiers, a subject John Papa addresses this month in DataPoints.
As users grow more accustomed to owning their own data, they will get better at managing it. If my mother's portable music player dies, it takes the music stored there with it. And my mother, a non-technical user if ever there was one, understands that. She knows she, not the company that sold her the music player, is responsible for backing up her data. The burden of managing data is shifting somewhat, from the developer to the user herself. This shift will require other changes as well.
We'll need to create automated scenarios in which data on a device is automatically synchronized with, say, a desktop machine or external drive and then backed up. These kinds of automated scenarios will appeal to users because they make data safety so easy.
As you flip through the pages of this issue, take note of all the changes in data programming and management on the horizon. We hope they'll light a few sparks and get you thinking differently about your future data projects.
In other news, we are hosting two new Virtual Labs in this issue. The first includes the projects discussed in Joshua Smith's article this month, "Data and WPF: Customize Data Display with Data Binding and WPF," at go.microsoft.com/fwlink/?LinkId=120301. Just start the lab and code along as you read.
We're also featuring a Virtual Lab for Dr. James McCaffrey's Test Run column on testing SQL stored procedures from the April 2008 issue of MSDN Magazine. You can get started by pointing your browser to go.microsoft.com/fwlink/?LinkId=120461.
Access, Active Directory, Excel, IntelliSense, Internet Explorer, Microsoft, MSDN, PowerPoint, SharePoint, Silverlight, SQL Server, Visio, Visual Basic, Visual Studio, Windows, Windows Media, Windows Server, Windows Vista, Xbox, and Xbox 360 are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

Visit us at msdn.microsoft.com/msdnmag. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Melitta Andersen, Spotty Bowles, Alex DeJarnatt, T.J. Goltermann, Kapil Gupta, Michael Howard, Shalini Joshi, Steve Maine, Tim Mallalieu, Kevin Ransom, Daniel Simmons, Beatriz Stollnitz, Jon Wheeler, and Jesse Yurkovich.

Ted Neward is an independent consultant specializing in high-scale enterprise systems. He is the author and co-author of several books, a Microsoft MVP Architect, a BEA Technical Director, an INETA speaker, and a PluralSight instructor. Reach Ted at ted@tedneward.com or visit his blog at blogs.tedneward.com.


Toolbox
Easy Wiki Hosting, Scott Hanselman's blog, and Snagging Screens
Scott Mitchell

Contents
Hosting a Wiki
Blogs of Note
Capture, Edit, and Share Screenshots
The Bookshelf

Hosting a Wiki
A wiki is a Web application whose content is collaboratively added, updated, and organized by its users. As is the case with blogs and other content management systems, a wiki's content is editable through a Web page interface. What separates wikis from blogs and traditional content management systems is that wikis are inherently amorphous. The site's users create the content, define the relationships, and establish the links between the site's Web pages. Perhaps the most well-known wiki is Wikipedia.org.
Wikis are especially useful for creating knowledge management portals. Within a company, a team of developers may maintain a wiki that includes information about ongoing development projects and their status, coding conventions, and tutorials for new hires on how to set up a baseline development environment.
A customer-facing support site could include a wiki allowing trusted support staff to proactively create and manage answers to commonly asked questions. And many community-oriented sites include wikis to allow members to update the site or contribute to the knowledge base. For instance, the open-source code projects hosted at CodePlex.com are managed as a wiki.
There are a variety of commercial and open-source wiki platforms available. One open-source wiki created with ASP.NET and C# is ScrewTurn Wiki version 2.0. By default, ScrewTurn Wiki's pages, modification history, and user accounts are saved on the Web server's file system, meaning the wiki does not require a database. Consequently, deploying ScrewTurn Wiki to a Web server is as simple as copy and paste.
But the wiki's storage system is built atop the provider model, so you can plug in an alternative provider to persist system information elsewhere. You'll find SQL Server® and MySQL storage providers at the ScrewTurn Wiki Web site.
When it is first installed, ScrewTurn Wiki allows anyone to create an account and any authenticated user to create, edit, or delete content. These access levels can be restricted through the wiki's administration pages.
Visitors may be barred from registering, or may require the approval of an administrator before being able to log in. And administrators can limit what user actions are allowable; for instance, the site can be configured to allow users to create and edit pages but not delete them. Of course, like other wikis, ScrewTurn Wiki maintains a complete history of every page modification with tools to compare different versions. And administrators can roll back a page to a previous version.
ScrewTurn Wiki enables content to be created and displayed in multiple languages, with support for English, French, Spanish, and German, among others.
Price: Free, open source.
screwturn.eu
Open-Source ScrewTurn Wiki

Blogs of Note
Reading other developers' source code is a great way to hone your own programming skills, as it introduces you to different techniques for accomplishing common tasks. This sentiment is shared by Microsoft Senior Program Manager and prolific blogger Scott Hanselman, who reads source code like ordinary people read the newspaper. He shares some of the most interesting code he's read—along with commentary—in a series of blog entries titled "The Weekly Source Code."
In addition to "The Weekly Source Code," every developer should take the time to peruse Scott's Ultimate Developer and Power Users Tool List (hanselman.com/tools)—an expansive list of tools and utilities Scott uses on a day-to-day basis. There are hundreds of recommendations categorized by topic—from working with XML to Visual Studio® productivity enhancers to managing your e-mail inbox and getting the most out of your Web browser. The Ultimate Developer and Power Users Tool List was inaugurated in 2003 and has been updated annually since.
In addition to his frequent blog posts, Scott hosts "Hanselminutes," a weekly audio podcast. There Scott reviews developer tools, dispenses programming advice, and shares war stories. Sometimes he interviews prominent Microsoft staff and members of the Microsoft® .NET Framework dev community. The podcasts are great for your commute to and from work, as each episode is approximately 30 minutes long.
Blog: hanselman.com/blog
Podcast: hanselminutes.com

Capture, Edit, and Share Screenshots
During a typical work day, I take upwards of a dozen screenshots. While most of them are for a blog entry, article, or book I'm working on, I also send screenshots to coworkers or customers.
Often it's easier to send a screenshot of a flawed UI than try to explain the problem in an e-mail. With Windows®, the Print Screen key copies the current screen, while Alt+Print Screen copies the current window. You can then paste the screenshot into an e-mail message, an image editing program, or a Microsoft Office Word document.
I recently decided I needed to upgrade from the Print Screen key to a more robust screen capture technique and decided to check out TechSmith's SnagIt version 8.2, which was first reviewed here in July 2003. A lot has changed since 2003. SnagIt has an updated user interface, many more image editing tools, and the ability to save to a number of new file formats.
SnagIt can take screenshots of a user-defined region, of a specified window, or of the entire screen. And when capturing a scrolling window, SnagIt can be configured to automatically scroll through the document, thereby capturing its entire contents in one image. This is especially helpful when capturing lengthy Web pages.
After capturing the window or selected region, SnagIt loads the screenshot into a powerful editing and sharing tool. With a few clicks of the mouse you can add text, draw shapes, cut and move regions, highlight areas, define border colors and widths, and add callouts. The dimensions of the screen capture can be further tweaked by trimming, resizing, or rotating the captured image.
More advanced tasks, such as adding watermarks, spotlighting or magnifying important regions, and adding hotlinks, are possible as well. Best of all, SnagIt is straightforward and easy to use—you don't need to be a digital imaging expert.
Once the screenshot has been edited, sharing it is as easy as clicking a button. There are one-click options for saving the capture as an image, copying it to the clipboard, printing it, sending it via e-mail, sending it via FTP, sending it via Instant Messenger, and loading it into a Microsoft Office Excel®, Word, or PowerPoint® file.
In addition, they can all be saved to a variety of file formats. Along with standard image formats such as BMP, JPG, and PNG, SnagIt can alternatively save your screen captures as PDF documents, Web archive files (.MHT), or Adobe Flash files, which are also new capabilities in this version.
Price: $39.95 per single-user license.
techsmith.com/screen-capture.asp
SnagIt Has an Updated User Interface

The Bookshelf
Querying data is one of the most common tasks a programmer faces. Yet interfacing with different data types requires learning unique syntax and semantics; and querying external data sources, such as XML files and databases, typically involves serializing query constructs to a string.
Because the compiler cannot parse a string literal like it can parse code, features like IntelliSense® and compile-time type checking are unavailable. To remedy this, Microsoft created LINQ, which offers developers a unified data querying syntax. LINQ extends the C# and Visual Basic® programming languages to include a rich, SQL-like syntax that can be used to query collections of objects, XML documents, databases, and more.
LINQ is made possible thanks to a number of new language enhancements and several new namespaces and classes in the .NET Framework. Because of the breadth and depth of the topic, to really learn LINQ you'll need more than a few online articles. One LINQ book, which explains the various facets through rich examples that build on previous chapters, is LINQ in Action by Fabrice Marguerie, Steve Eichert, and Jim Wooley.
LINQ in Action presents one of the best introductions to LINQ I've read, walking readers through its purpose and design goals and the language enhancements added to C# and Visual Basic that make LINQ possible. The book then looks at LINQ to Objects, LINQ to SQL, and LINQ to XML, which are libraries that enable LINQ queries against objects, relational data, and XML data. The book's last two chapters examine how to extend LINQ to query alternate data sources (such as Web services) and how to best utilize LINQ at different layers in your application's architecture.
What I liked best about LINQ in Action is how the authors start each new section with straightforward, easy-to-follow examples that highlight the most important features. Next, they pull back the curtains and show how the LINQ queries and features are actually implemented.
With the groundwork in place, the authors move on to more advanced features and scenarios. For example, the chapters covering LINQ to SQL start with setting up the object mapping and using the DataContext class to read and update data. Then they look at how LINQ's query syntax is translated into T-SQL. It concludes with advanced LINQ to SQL features, such as concurrency control, using stored procedures and user-defined functions, and so on.
Price: $44.99.
manning.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Extend Windows Forms Apps Using System.AddIn
Mueez Siddiqui

Code download available at: CLRInsideOut2008_07.exe (472 KB)
Browse the Code Online
Contents
Adapting the Host
Exposing the Object Model
Adapters, Contracts, and Add-In Views
Writing the Add-In
The Collaboration Add-In
Wrap-Up
Did you know that you can now make Windows® Forms applications extensible by using the new Microsoft® .NET add-in framework (System.AddIn)? In this column I will modify the ShapeApp drawing application to expose its object model through the add-in framework. This will allow me to create add-ins that automatically perform tasks on the host application (automation add-ins). Also, I'll discuss the kinds of problems commonly encountered in this scenario and present solutions.
If you need background information on the add-in framework before you get started, a good place to begin is the resources page on the CLR Add-In Team Blog at go.microsoft.com/fwlink/?LinkId=117519. You might also consider reading the previous CLR Inside Out columns on the add-in framework.
ShapeApp is a drawing application that lets you make drawings using basic shapes. You can insert new shapes, move them around, change their color and size, and so on. It allows you to save drawings to files and to open multiple drawings in tabs. A simple drawing created in ShapeApp can be seen in Figure 1.
Figure 1 Screenshot of a ShapeApp
ShapeApp was developed by the Visual Studio® Tools for Applications (VSTA) team as a sample, which lets you embed the Visual Studio IDE into your application. The IDE can then be used to write extensions for the application. They have provided versions of the application with and without their extensibility code. In this sample, I start from the base version and adapt it to work with the add-in framework.

Adapting the Host
In this section I'll adapt the application to expose its object model, similar to the Document Object Model (DOM) in HTML. This will allow me to create add-ins that automate tasks in the application.
The first step in adapting the host is to define an object model that the host will expose (note that I don't need to write any code yet). The isolation and versioning capabilities of the add-in framework impose constraints on the object model. In general, the object model should define all the types that the add-in will use that aren't built-in to the .NET Framework; in other words, the view assembly should not reference any assemblies other than the .NET Framework assemblies. MarshalByRefObject types are explicitly disallowed. For a more detailed breakdown of the allowable types, see the blog post on contracts at go.microsoft.com/fwlink/?LinkId=117520.
Figure 2 lists the types in the ShapeApp's object model. All of the classes you see there exist internally in the host application. I will be exposing them through the pipeline later.
Figure 2 ShapeApp Types
Type | Description
ShapeApplication | The main application object.
Drawing | An object that represents a drawing in the application.
Shape | An object that represents a shape in a drawing.
DrawingCollection | A collection of drawing objects.
ShapeCollection | A collection of shape objects.
EventArgs-related types | Several types derived from System.EventArgs, used for custom event arguments.
The next step is to add a way for the host to discover and activate add-ins. For this I use a simple add-in manager on the host. This includes both a static class, AddInManager, that loads and keeps track of add-ins and a form that lets the user manage add-ins. The form is activated through a menu item on the host, requiring minimal changes to the host. The form is shown in Figure 3.
Figure 3 The Add-In Manager Form
The add-in manager code is very simple. Here are the three lines required to find available add-ins:
// update the add-in store to include add-ins located in the
// current program folder
string[] warnings = AddInStore.Update(PipelineStoreLocation.ApplicationBase);
// get the view type defined in the host views assembly
System.Type hostViewOfAddIn = typeof(ShapeAppHostViews.IShapeAddIn);
// find all add-ins that implement the view
ICollection<AddInToken> addIns = AddInStore.FindAddIns(hostViewOfAddIn,
    PipelineStoreLocation.ApplicationBase);
I use the first line, AddInStore.Update, to search for available add-ins and pipeline components. PipelineStoreLocation.ApplicationBase tells it to look in the application's folder. For a detailed description of the folder structure, see go.microsoft.com/fwlink/?LinkId=117521. Next, I get the type of the add-ins I want to activate. In this case it's the IShapeAddIn type defined in the host views assembly. Finally, I call AddInStore.FindAddIns, which returns a collection of available add-in tokens that I can inspect and activate. Once I have selected a token from the collection, I can use one line of code to activate it:
// activate the add-in in full trust mode
ShapeAppHostViews.IShapeAddIn addIn =
    token.Activate<ShapeAppHostViews.IShapeAddIn>(
    AddInSecurityLevel.FullTrust);
I can also choose to activate add-ins out of process or under a different trust level. For a list of isolation levels, you can go to go.microsoft.com/fwlink/?LinkId=117522. For a performance comparison of isolation levels, see the blog post on performance, available at go.microsoft.com/fwlink/?LinkId=117523.

Exposing the Object Model
Once the empty pipeline and add-in manager are in place, I can start exposing functionality from the host to the add-in. To expose the types included in the object model, I need to first create a host view in the HostViews assembly for each type that I want to expose. I can expose properties, methods, and events in the view. The host view can be either an abstract base class or an interface.
In the ShapeApp, I have used interfaces for three reasons. First, they are needed for events to work correctly when used from Visual Basic®. Second, C# does not support multiple inheritance, so a class cannot inherit from two base classes. Therefore, the view should be an interface to allow all host classes to implement it (even if they already have a base). Third, interfaces allow EIMI (explicit interface method implementation). I will explain the significance of this later.
Figure 4 shows what the host view looks like for the ShapeApplication object. The types used in members can either be built-in framework types, such as EventHandler<T>, or other views, such as IDrawing. You can download FxCop rules from our CodePlex site at go.microsoft.com/fwlink/?LinkId=117524 to verify that the views are easily adaptable into a complete pipeline.
Figure 4 ShapeApplication Object Host View
namespace ShapeAppHostViews
{
    public interface IShapeApplication
    {
        // the drawing present in the selected tab
        IDrawing ActiveDrawing { get; }
        // a collection of available shapes (square, circle, etc)
        IShapeCollection AvailableShapes { get; }
        // a collection of drawings currently open in the application
        IDrawingCollection Drawings { get; }
        // main window visibility
        bool Visible { get; set; }

        // create a new drawing in the application
        IDrawing NewDrawing();
        // completely exit the application
        void Quit();

        // event fired when a drawing is created
        event EventHandler<CreatedDrawingEventArgs> CreatedDrawing;
    }
}
Once I have created the views assembly, I modify the host's classes to inherit from corresponding views. The changes to the ShapeApplication class are shown bolded below:
public class ShapeApplication : System.Windows.Forms.IWin32Window
{
...
}

public class ShapeApplication : ShapeAppHostViews.IShapeApplication,
    System.Windows.Forms.IWin32Window
{
...
}
Next, I need to implement each member from the host view. For members that use built-in types as arguments and return values, there is no special implementation required (other than ensuring the member is public in the host's class). For example, the Visible property in the ShapeApplication class has the same signature as the Visible interface member:
public bool Visible
{
    get {...}
    set {...}
}
This means that the existing implementation of Visible in the host's ShapeApplication class will be used as the view implementation.
However, members that use other types defined in the host views need to be implemented explicitly. For example, the ActiveDrawing property has different signatures in the ShapeApplication class and in the IShapeApplication host view. The original version in the host's class returns a Drawing object:
public Drawing ActiveDrawing
{
    get {...}
}
The version in the host view, IShapeApplication, can only return the corresponding view, IDrawing:
IDrawing ActiveDrawing
{
    get;
}
To implement the second version in the host's class, I need to use EIMI. I do this by adding a new property in the host's class with the same name, ActiveDrawing. Now the host's ShapeApplication class has two properties called ActiveDrawing (see Figure 5).
Figure 5 ActiveDrawing Property
// original property
public Drawing ActiveDrawing
{
    get {...}
}

// implementation of the host view's version of ActiveDrawing
ShapeAppHostViews.IDrawing
ShapeAppHostViews.IShapeApplication.ActiveDrawing
{
    get
    {
       // a call to the original property
       return this.ActiveDrawing;
    }
}
The new implementation implicitly casts the Drawing object to the IDrawing type. If you are exposing an object model for a Windows Forms application, it is likely that some of the exposed members will access and modify Windows Forms controls. If such members are called from a thread other than the one that created the control, it causes unpredictable behavior in the GUI.
Since members exposed to the add-in can be called from any thread, I must ensure that any exposed members that access or modify GUI controls do so safely by using Control.InvokeRequired and Control.Invoke. For example, the ShapeApplication.NewDrawing function accesses the ApplicationForm object and therefore requires the use of Invoke to be safely implemented.
Here's the old implementation:
public Drawing NewDrawing()
{
    Drawing newDrawing = new Drawing(this);
    ...
    this.ApplicationForm.drawingsTabControl.TabPages.Add(
        newDrawing.DrawingSurface);
    ...
    return newDrawing;
}
The new implementation is shown in Figure 6; the changes are bolded.
Figure 6 NewDrawing Method with Invoke
                                                            // delegate added to allow invoke
private delegate Drawing NewDrawingDelegate();

public Drawing NewDrawing()
{
    // check if we need an invoke
    if (this.ApplicationForm.InvokeRequired)
    {
        // invoke this method using the ApplicationForm object
        NewDrawingDelegate del = new NewDrawingDelegate(NewDrawing);
        return (Drawing) ApplicationForm.Invoke(del);
    }
    else
    {
        Drawing newDrawing = new Drawing(this);
        ...
        this.ApplicationForm.drawingsTabControl.TabPages.Add(
            newDrawing.DrawingSurface);
        ...
        return newDrawing;
    }
}

Adapters, Contracts, and Add-In Views
To allow the add-in to access functionality exposed by the host, I need to create adapters, contracts, and add-in views. The pipeline components can be automatically generated from the host view. The Pipeline Builder does just that, and it can be found at go.microsoft.com/fwlink/?LinkId=117525. This works great for the first version of the application.
Alternately, the pipeline can be coded by hand. This is required when writing cross-version adapters. In Visual Studio, one can set up a pipeline by adding projects for each required assembly (views, adapters, contracts, and so forth) into the ShapeApp solution. A step-by-step guide to creating a pipeline in Visual Studio can be found at go.microsoft.com/fwlink/?LinkId=117526.
In this sample, I coded the pipeline manually. Since this is the first version of the ShapeApp object model, the host and add-in views are identical. Therefore, I can use one assembly for both views. When coding by hand, it is helpful to set up an empty pipeline initially. The view for such a pipeline looks like this:
public interface IShapeAddIn
{
    void Initialize(IShapeApplication application);
}

public interface IShapeApplication
{
    // TODO: Implement this.
}
The Initialize method is used to pass the main application object to the add-in when it is loaded. This allows the add-in to control the host at any time, without the host explicitly asking for any services (that's why it's called an automation add-in).
After I have created an empty pipeline, I can expose the rest of the functionality from the host one bit at a time, compile it, and test it. This iterative approach helps avoid a seemingly endless string of compile errors. Of course, once the interfaces have been finalized and the application is released, I can't add any more methods to the existing interfaces. What adapters I'll need depends on where an object lives and where it is accessed. Generally, objects can fall into one of the following three categories:
Host-Side Objects These objects live on the host side and are accessed from the add-in (for example, the ShapeApplication class). To expose them to the add-in, I need the pipeline components shown in Figure 7. Since the object exists on the host side, I start from the host view on the top. I use a view-to-contract host adapter to convert it to a contract, and contract-to-view add-in adapter to convert it to the add-in view. These two adapters let the add-in access the host-side object.
Figure 7 Pipeline Components Required
Host-Side Objects | Add-In-Side Objects
Host (object lives here) | Add-in (object lives here)
Host view | Add-in view
View-to-contract host adapter | View-to-contract add-in adapter
Contract | Contract
Contract-to-view add-in adapter | Contract-to-view host adapter
Add-in view | Host view
Add-in (object is accessed here) | Host (object is accessed here)
Add-In-Side Objects These objects live on the add-in side and are accessed from the host (for example, the ShapeAddIn class). The pipeline components are similar here, but the direction of the adapters is reversed to allow the host access to the add-in-side object. The view-to-contract adapter is now on the add-in side instead of the host side, and the contract-to-view adapter is now on the host side. Note that the rest of the pipeline (the views and the contract) is identical.
Either-Side Objects These are objects that can live on either side and can be accessed from either side. These require four adapters, two for each side, since I need to adapt them either way. Note that if you're using the same assembly for the host and add-in view, you can re-use adapters as well. In that case you would need only two.
Contracts do not support events natively. However, events can be simulated in contracts by using the following pattern (example shown from the IShapeApplication.CreatedDrawing event):
Host views:
public interface IShapeApplication
{
       ...
       // event fired when a drawing is created
       event EventHandler<CreatedDrawingEventArgs> CreatedDrawing;
}
Corresponding contracts:
public interface IShapeApplicationContract : IContract
{
    ...
    void CreatedDrawingAdd(ICreatedDrawingEventHandlerContract handler);
    void CreatedDrawingRemove(ICreatedDrawingEventHandlerContract
        handler);
}

public interface ICreatedDrawingEventHandlerContract : IContract
{
    void Handler(ICreatedDrawingEventArgsContract args);
}
You can see that I have created a new contract for the event handler, ICreatedDrawingEventHandlerContract, because delegates like System.EventHandler<CreatedDrawingEventArgs> cannot be used in a contract.
The add-in-side adapter registers a handler on the host side through the Add and Remove methods, and it maintains a local event that the add-in can subscribe to. The host-side adapter calls the Handler function when the event is fired.
This added complexity is mostly transparent to the add-in developer, except for one caveat: the add-in must register and unregister itself on the same adapter instance of an object. Otherwise, unregistering will not work.
Next, I will describe how a host object is able to have two (or more) sets of adapter objects that reference it. When an object on the host side is returned to the add-in (through a property access or function call), two adapter objects (host-side adapter and add-in-side adapter) are created that allow access to the host object. When the same object is returned again to the add-in, two new adapters are created.
For example, accessing ShapeApplication.ActiveDrawing twice will return two different object references to the add-in, and ShapeApplication.ActiveDrawing.ReferenceEquals(ShapeApplication.ActiveDrawing) will return false. The presence of two (or more) adapters for the same host object can be a problem when registering/unregistering events and storing host objects in collections, among other things.
To help solve these problems, the .Equals and .GetHashCode functions can be overridden on the adapters to call the corresponding functions on the real host object. This allows us to put host objects into collections in the add-in, and methods like .Contains work fine. Of course, it's also true that the add-in developer still has to ensure that events are registered and unregistered on the same object. And you should know that the .Equals function can help with this.
As you may have guessed, another option is to cache adapters. However, this is difficult in practice, because there is no easy way to store a weak mapping of objects to adapters (here, weak refers to weak object references, which the garbage collector ignores). Using a regular dictionary will prevent adapters and objects from being garbage collected.

Writing the Add-In
Once the pipeline is in place, I can write add-ins against the add-in view. Writing the add-in is as simple as possible. All an add-in developer needs to do is to create a class that inherits from the add-in view, and mark the add-in implementation with the AddIn attribute. The rest of the code can be written almost as if there were no pipeline between the host and the add-in. There are a couple of caveats, however. One is the object identity issue I discussed earlier, and the second is performance, depending on what isolation boundary is used. For performance benchmarks, see go.microsoft.com/fwlink/?LinkId=117527.
Currently, add-ins cannot directly display Windows Forms controls on the host application's forms. However, they can use any of the following three methods: they can display their own forms (note that if the add-in is running in some partial trust scenarios—like the Internet trust level—the user will see a security warning on the add-in form when it is loaded); they can display Windows Presentation Foundation (WPF) controls directly on the host application's forms (please see go.microsoft.com/fwlink/?LinkId=117528); and they can use Windows Forms controls that are wrapped in WPF containers (see go.microsoft.com/fwlink/?LinkId=117529).
Windows Forms have some threading requirements that must be met. For example, an add-in extending a command-line application must create a new thread to construct a form and process its events. This is because command-line applications use the multithreaded apartment (MTA) model by default, and Windows Forms require the single-threaded apartment (STA) model for UI threads. The solution is simple: add-ins extending a Windows Forms application can just include two lines to display forms (given the add-in is activated and used from the host's UI thread), like so:
AddInForm form = new AddInForm();
form.Show();

The Collaboration Add-In
Using the exposed interface, one can create really powerful add-ins. One example is the collaboration add-in included with this sample. It allows two users of the ShapeApp using two different machines to edit a drawing together in real time. The collaboration add-ins connect to each other through Windows Communication Foundation (WCF), allowing this interaction to work globally over the Internet. A screenshot of the connect screen can be seen in Figure 8.
Figure 8 Collaboration Add-In UI
Once two add-ins are connected to each other, any new documents created or opened on either end are shared; that is, changes on one machine are sent in real time to the other. This is accomplished using events. When the collaboration add-in is loaded, it subscribes to all the application's events. When a new drawing is created, the CreatedDrawing event is fired. The add-in receives this event and subscribes to all events for the new drawing. Similarly, it subscribes to all shape-related events when shapes are created. This allows the add-in to track all user actions and propagate them over to its peer.
Figure 9 shows the path taken by an event through the collaboration add-in. On Machine 1, the user performs an action (such as changing the position of a shape). This makes the host fire an event, and the collaboration add-in receives it. The collaboration add-in creates a message and sends it through WCF to the add-in on Machine 2. This add-in then performs the same action on the host. Note that it temporarily unhooks event handlers when performing actions on the host. This prevents the event from coming back to the add-in, which would result in an infinite cycle.
Figure 9 Path Taken by an Event through the Collaboration Add-In
More interesting scenarios are made possible by using WCF. Since the collaboration add-in hosts a service, any number of clients can connect to it. This allows more than two people to work on a drawing together, seamlessly. Figure 10 shows the connections among three machines. The collaboration add-in on each machine is connected to all the others.
Figure 10 Three Instances of the ShapeApp Connected Together

Now you've seen how the ShapeApp is adapted to host add-ins using the .NET add-in framework. You should have a good idea of the capabilities of the add-in framework and how it can be used to create an add-in that seamlessly turns the ShapeApp into a real-time collaborative editor. Please feel free to leave feedback or ask questions on the Add-In Team Blog at go.microsoft.com/fwlink/?LinkId=117530.

Mueez Siddiqui is a Software Development Engineer in the CLR security and extensibility team at Microsoft.


Data Points
The Entity Framework In Layered Architectures
John Papa

This column is based on a prerelease version of the ADO.NET EntityFramework. All information herein is subject to change.

Code download available at: DataPoints2008_07.exe (3,549 KB)
Browse the Code Online
Contents
Defining Layers
Building the Model
How It Works
Persisting Changes
Deleting and Adding
Wrapping It Up

When architects of an n-tiered architecture evaluate any new technology, pattern, or strategy, they have to consider how that new piece of the puzzle is going to mesh with the architecture. With the Entity Framework, integration is not a problem. It can be integrated into an n-tier architecture as well as a single-tier architecture.
In this month's column I will demonstrate how the Entity Framework can fit into an n-tier architecture that uses Windows® Communication Foundation (WCF) and Windows Presentation Foundation (WPF) technologies and the Model View Presenter (MVP) pattern. I will present a sample architecture that contains layers for a logical store database, data access, a domain model, a business manager, a service layer, a presentation layer, and a passive UI layer, and I will demonstrate how these layers integrate using the Entity Framework. All of the code examples I use are available for download from the MSDN® Magazine Web site.

Defining Layers
The application I'll present allows a user to search for customers in the NorthwindEF sample database and view, add, edit, or delete them. Before diving into the code and the examples, let's discuss the overall architecture of the sample. Because my focus is not the architecture itself, but how to integrate the Entity Framework with an architectural design, I chose a relatively common architecture that can be modified and integrated fairly easily with other strategies.
Figure 1 shows a high-level view of a typical layered architecture. The top two layers handle the user interface presentation and navigation using a UI layer and a presentation layer. The UI layer can be implemented in any one of a variety of technologies; however, for this column and its examples I will use WPF. The UI layer follows the MVP pattern with a passive view, which means that the views (the top UI layer) are managed and directed by the presentation layer. The presenters are responsible for giving the views their data, pulling data from the views to be saved in the lower layers, and in general for responding to events raised by the views.
Figure 1 Architectural Overview
The presenters communicate with the lower tiers through WCF in my example here. A presenter will invoke a service through WCF using the service's contract as a guide. The service layer exposes its services through service contract interfaces. These contracts allow the presenters to be certain of how to call the services.
The service layer is responsible for receiving the communications from the presenters and calling the appropriate business layer methods that will execute appropriate business logic and data gathering or modifications. The business layer is where the business logic and the LINQ to Entities code for this project would reside. The LINQ to Entities code will reference the entity model that is generated from the Entity Framework. When the LINQ queries are executed, the Entity Framework will translate the LINQ query to the conceptual entity model (the Entity Data Model, or EDM), map the entity aspects to the storage layer, and generate a SQL query to execute against the database.

Building the Model
Now that I've provided a high-level overview of how the layers work in the architecture, let's examine the key aspects of each layer as they pertain to the Entity Framework. Since the database already exists for the application, I generated the entity model from the NorthwindEF database as a starting point.
I could have built the entity model first and then mapped the entities to the database as well. The EDM wizard helps generate the base entity model, which can then be modified as needed to incorporate inheritance, entity splitting, and other domain modeling concepts. Figure 2 shows the EDM wizard with all tables and stored procedures selected to be imported into the EDM.
Figure 2 Generating the Model from the Database
One topic that often causes people confusion in regard to EDMs is the default naming convention for the EntitySets and the EntityTypes. I like to use singular names for all of my entities in my domain models. I create an instance of a Customer or return a list of Order instances using List<Order>. Each entity is a singular instance of a blueprint that has properties defining the entity.
On the other hand, I like my EntitySets to use a plural naming convention. The EntitySets are often used in LINQ queries when asking the ObjectContext to refer to its set of Customers or Orders.
As an example of this, take a look at the following LINQ to Entities query:
var q = from c in context.Customers
        select c;
List<Customer> customerList = q.ToList();
This query tells LINQ to Entities to access the Customers EntitySet and return all Customer entity instances when executed. The second line executes the query and returns the List<Customer> to the local variable named customerList. The EntitySet is plural in this example, making it clear that it is querying EntitySets and returning instances of the Customer (notice this is singular) entity.
Is it necessary to have this naming convention? Of course not. However, I find it helps make my code more readable. Otherwise, if you take the default of what is returned from the EDM wizard, you get an EntitySets named Customers and an EntityType named Customers, which makes your LINQ to Entities query look like this:
var q = from c in context.Customers
          select c;
  List<Customers> customerList = q.ToList();
When the EDM wizard generates the model, the EntitySet and EntityType names can easily be modified. This can be done by selecting the entity in the diagram, viewing its properties in the Properties window, and modifying the desired setting (see Figure 3). For this application, I modified all of the EntityTypes to be singular by setting the Name property. I did not change the EntitySet Name property, since it was already plural.
Figure 3 Changing the EntityType Names

How It Works
Now I will demonstrate the application and discuss how it operates from the top layer down, starting with the views (located in the NWUI project) and the presenters (located in the NWPresentation project). Both projects are available in the code download that accompanies this column. The application loads a customer search view, which allows the user to search for customers by matching the company name criteria (see Figure 4). The view is implemented using WPF, and when a user interacts with the view it raises events that are listened to by its presenter, who can then take appropriate action.
Figure 4 Searching for Customers
When the user searches for all customers starting with the letter D, as shown in Figure 4, the view raises an event when the user clicks the Search button. The presenter listens for this event and responds by calling the service layer through WCF to get the list of customer entities that will be displayed on the CustomerSearchView. Here is the code in the view when the user clicks the Search button:
private void btnSearch_Click(object sender,     RoutedEventArgs e)  {
      if (FindCustomerSearchResults != null)          FindCustomerSearchResults();
  }
This code does not interact with the list of entities that are returned, but, rather, it leaves that up to the presenter. The view uses WPF data binding to reference the entity's properties so it knows how to bind the list of entities to the list view control's elements. The only interaction the views have with the entities is through the data binding.
The CustomerSearchView raises the event FindCustomerSearchResults, and the CustomerSearchPresenter listens for the event and responds by taking the baton and performing the search. The following code shows how the CustomerSearchPresenter class creates an instance of the NWServiceClient class, which is a proxy to the WCF service that is exposed on the lower tier:
public void view_FindCustomerSearchResults()
{
    if (this.view.CompanyNameCriteria.Length > 0)
        using (var svc = new NWServiceClient())
        {
            IList<Customer> customerList = svc.FindCustomerList(                view.CompanyNameCriteria);
            view.CustomerSearchResultsList = customerList;
        }
}
The NWServiceClient is referenced using a ServiceReference from the NWPresentation project so the presenters are aware of how to call the services and what types of data will be returned. The presentations layer does not and should not reference the EDM directly. Instead, it is told what types of entities to expect by the DataContracts exposed via WCF. This allows the Entity Framework's entities to be passed across a physical network boundary via WCF to the presenters.
Notice that once this list of Customer entities is returned, it is set to a public property on the view. This property of the view then accepts the List<Customer> and binds it to the DataContext of the view. The presenter serves up the data, passes it along, and the view handles any view-specific binding (since that code is very specific to the technology, whether it be WPF, Silverlight®, Windows Forms, or ASP.NET).
This technique allows the same presenter to be used to interact with any view that implements the ICustomerSearchView interface. In this application, the binding is handled by the WPF binding techniques using the DataContext.
The contracts expose the method that can be called in the service layer as well as the entities that will be returned. In this application, I only have a method to return the Customer and Order entity types. This means that only these entity types will be included in the contract.
WCF handles the serialization of the entities by applying the WCF DataContract attributes to them where needed. By exposing the entities through DataContracts the entities can be used in the UI layers without referencing the EDM directly.
Note that as of the .NET Framework 3.5 SP1 Beta 1, the Entity Framework supports automatic graph serialization. For example, if a parent entity has associated child entities, the parent and its child entities will be serialized. In the sample app, because the OrderManager's FindOrderList method uses a LINQ to Entities query that eager loads the Order Details for each Order, each Order entity returned from the middle tier will contain a List<OrderDetail> accessible through its navigation property.
While the serialized entities can be passed between the presenters and the service layer via WCF, the ObjectContext is neither serialized nor passed to the presenter. This means the entities can be used in the UI layers but the ObjectContext is left in the lower tiers where it will be able to access the EDM and the full resources of the Entity Framework.
Leaving the ObjectContext behind means that it can't be used to retrieve or modify entities in the UI layer directly, and it can't be used to manage change tracking in the UI layer. These roles are ideally left to the lower layers anyway. But when the entities are passed back down to the lower layers, the application has to synchronize with the ObjectContext so it can persist any changes in the entities.
When a user clicks the Search button shown in Figure 4, the presenter calls the service layer, which then calls the business layer (found in the NWBusinessManagers project) to retrieve the List<Customer>. This layer has two major roles. The first role is to get or put any data to or from the EDM. The second role is to handle any business logic that may exist.
The CustomerManager uses the ObjectContext to handle the interaction with the EDM, so it defines a local field called context and creates an instance of it in its constructor. The ObjectContext could be created and destroyed in each method. However, it is optimized to open and close database connection resources as needed. Also, by making the ObjectContext accessible throughout the class, it can maintain change tracking without having to be passed around a series of private methods within the class:
public CustomerManager()
{
    context = new NWEntities();
}
Note though that for this type of app, the ObjectContext should not be held but created/destroyed as required. Due to identity resolution, holding on to the same object context will eventually lead to inconsistent, stale data and (as more and more data gets tracked) degraded performance when doing identity resolution, and it can lead to problems with updates in a multithreaded environment.
The following code shows the CustomerManager class's FindCustomerList method in the business layer. This method declares a LINQ to Entities query that accesses the context to request a list of Customer entities that start with the criteria. When this query is executed, it evaluates the mappings from the conceptual layer to the storage layer and generates an appropriate SELECT command:
public List<Customer> FindCustomerList(string companyName)
  {
      var q = from c in context.Customers
              where c.CompanyName.StartsWith(companyName)
              select c;
      return q.ToList();
  }
If you would like, you can view the queries as they execute using the SQL Server® Profiler.

Persisting Changes
Now that I have walked through the application using a simple retrieval, it is time to examine how modifications to data can be persisted. When a user edits a customer, the CustomerView view appears bound to the appropriate Customer entity instance (see Figure 5). The CustomerView raises an event to the presenter, who in turn requests the Customer entity instance from the lower layers.
Figure 5 Editing Customers
When a user makes a modification to the customer and saves it, the entity is passed to the lower layers from the presenter using the code shown in Figure 6. This code evaluates whether the user was adding or modifying a customer and then calls the appropriate service layer method, passing the entity along for the ride.
Figure 6 SaveCustomer in the Presenter
public virtual void view_SaveCustomer()
{
    Customer customer = view.CurrentCustomer;
    var svc = new NWServiceClient();
    switch (view.Mode)
    {
        case ViewMode.EditMode:
            svc.UpdateCustomer(customer);
            break;
        case ViewMode.AddMode:
            svc.AddCustomer(customer);
            break;
        default:
            break;
    }
    view.CurrentCustomer = FindCustomer();

}
The service layer will then pass control on to the business layer, which will then save the customer entity to the database. Since the customer entity is no longer part of an ObjectContext, it must first be reunited with one by using the ObjectContext's Attach method, as shown in the code shown below. Once the entity has been attached to the context, the entity's properties must be marked as modified. This can be done by using the context's ObjectStateManager and invoking the SetModified method for each property. Now that the context knows the entity is modified, the SaveChanges method can be issued, which will then generate a SQL UPDATE command and execute it against the database:
public void UpdateCustomer(Customer customer)
{
    context.Attach(customer);
    customer.SetAllModified(context);     // custom extension method
    context.SaveChanges();
}
Notice that the code in the UpdateCustomer method uses an extension method I named SetAllModified<T>, which makes it easier to set the state of all of the properties for an entity to be modified. SetAllModified<T> gets an instance of the ObjectStateEntry for the given entity T. It then retrieves a list of all of the property names for the entity and iteratively calls SetModifiedProperty for each property:
public static void SetAllModified<T>(this T entity, ObjectContext  context)
where T : IEntityWithKey
{
    var stateEntry = context.ObjectStateManager.      GetObjectStateEntry(entity.EntityKey);
    var propertyNameList = stateEntry.CurrentValues.DataRecordInfo.      FieldMetadata.Select
      (pn => pn.FieldType.Name);
    foreach (var propName in propertyNameList)
        stateEntry.SetModifiedProperty(propName);
}
Another way to ultimately save the entity is to call the context's Refresh method. This tells the context to go get the data for the entity instance and refresh its property values from the database's values. The RefreshMode enumerator of ClientWins will replace the original values with the latest values from the database, thus allowing a last-in-wins strategy.
The RefreshMode of StoreWins will overwrite both original and current values in the entity cache with the values from the database. ClientWins is the appropriate strategy for last-in-wins, while StoreWins is an appropriate strategy when you want to cancel the changes and refresh the UI's view with the latest database values:
context.Refresh(RefreshMode.ClientWins, customer);  // Last in wins
The Entity Framework enforces optimistic concurrency when generating update and delete commands. It does this by including original values in the WHERE clause for any properties with a ConcurrencyMode attribute value set to Fixed.
By default, models are generated with no fields specified as concurrency fields. This means that when a user saves his changes, he possibly could be overwriting another user's changes inadvertently. If another user changes a value while the CustomerView is open and you want to use optimistic concurrency, you can do so by setting the ConcurrencyMode attribute of the EntityType in the conceptual model.
Editing the EDM file and setting the ConcurrencyMode to Fixed tells the Entity Framework to add this column to the WHERE clause of any Update or Delete commands. Thus, an OptimisticConcurrencyException is raised if a matching row is not found. Figure 7 shows this exception being raised when I modified the region of a customer in the database, just before the user tried to modify the same region.
Figure 7 OptimisticConcurrencyException
You can trap this exception and take whatever action is appropriate. For example, you could trap the exception, log it, and then overwrite the user's changes anyway, as shown here:
catch (OptimisticConcurrencyException e){
    context.Refresh(RefreshMode.ClientWins, customer); // Last in wins
    logger.Write(e);
    context.SaveChanges();
}

Deleting and Adding
When a user deletes a customer, the CustomerManager's DeleteCustomer method gets the customer entity and applies the delete:
context.Attach(customer);
context.DeleteObject(customer);
context.SaveChanges();
First, the Customer entity instance must be reunited with an ObjectContext using the Attach method. Then the customer must be deleted from the ObjectContext. Done this way, the change tracking mechanism within the ObjectContext is aware the Customer entity instance has been deleted. Finally, when the SaveChanges method is called, the ObjectContext is now aware the entity was deleted and it should generate a DELETE SQL command and execute it.
When adding a customer, the CustomerManager's AddCustomer method gets the customer entity and applies the insert, like so:
context.AddToCustomers(customer);
context.SaveChanges();
This entity instance is new, so it must be added to the context and flagged as a new instance of a Customer entity by associating the Customer entity instance with an ObjectContext using the AddToCustomer method. Finally, when the SaveChanges method is called, the ObjectContext is aware that the entity was added and that it should generate an INSERT SQL command and execute it.

Wrapping It Up
Here I've demonstrated how the Entity Framework can integrate into an architecture, use modern patterns such as the MVP pattern, and handle the common architectural concerns. Key components of the Entity Framework in layered architectures are its change tracking mechanisms, integration with LINQ to Entities, ability to disconnect and reconnect with its ObjectContext, and how it provides a means for a developer to handle concurrency issues.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa (johnpapa.net) is a Senior Consultant with ASPSOFT (aspsoft.com) and a baseball fan who spends summer nights rooting for the Yankees with his family. John, a C# MVP and INETA speaker, has authored several books and is now working on his latest, titled Data Access with Silverlight 2. He often speaks at conferences such as DevConnections and VSLive.


Advanced Basics
The LINQ Enumerable Class, Part 1.
Ken Getz

Code download available at: AdvancedBasics2008_07.exe (184 KB)
Browse the Code Online
Contents
Inside Enumerable
Creating Sequences
Selecting Sequences
Retrieving a Single Element
Filtering Sequences
Ordering Sequences
Verifying Sequences
As part of a recent programming project, I needed to create a randomly ordered list of integers between 1 and 100. Yes, this is obviously a basic Computer Science 101-type problem. And I originally used the same solution I had written back in the classroom, in 1985, using lots of looping and comparisons and generating random numbers and several arrays. But in these days of declarative programming, the solution really bothered me.
After some research, I managed to whittle the problem down to just two lines of code:
Dim rnd As New System.Random()
Dim numbers = Enumerable.Range(1, 100). _
  OrderBy(Function() rnd.Next)
After running these two lines, the numbers variable contains the integers between 1 and 100, in random order.
The behavior of this simple yet elegant solution counts on the System.Linq.Enumerable class, which provides a few dozen shared methods called extension methods that allow you to manipulate data in any class that implements the IEnumerable(Of T) interface. In addition, as you see in this example, these methods also can generate new collections of data.
In this and my next column, I will provide examples of the Enumerable class's extension methods, showing off multiple overloaded versions of the members and using various data structures to demonstrate the capabilities. Because you can apply these methods in so many places, and because they're so incredibly useful, it makes sense to take some time and investigate each—that way, when you need to operate on a collection, you'll know what tools you have available to you.
I've created a simple Windows®-based application as a sample. In the project, I've added simple classes representing the Customers, Categories, and Products data, and I've added a class named SimpleDataContext that fills instances of the these classes with data from the SQL Server® Northwind sample database. In addition, this class exposes public properties named Customers, Categories, and Products; each containing a generic List of Customer, Category, and Product instances, respectively. (Although I could have used the LINQ to SQL designer to create a "real" DataContext class, doing so wouldn't allow me to demonstrate the Enumerable class's extension methods: when you call the described methods in context of LINQ to SQL, you're actually calling extension methods on the LINQ to SQL provider's Queryable class that have the same signatures as the methods shown here. Although such examples would illustrate the same behavior, they wouldn't be using the Enumerable class.) I strongly suggest that you download the sample application and follow along using this application—the code will make much more sense if you can experiment with it directly, rather than just reading it here.
To make the long list of methods more tractable, I've broken them down into logical groupings. You'll investigate methods that deal with creating, selecting, retrieving a single element, filtering, converting, positioning within sequences, ordering, verifying, calculating, and performing set operations of sequences of data. In each of the following sections, you'll try out various examples that demonstrate one or more methods of the Enumerable class. By the time you're done, you will have worked through all the methods of the Enumerable class, and you will have investigated many of the overloads for the more complicated methods.
Note that the examples generally allow Visual Basic® to infer data types, rather than having the code specify the type of the return value for calls to the extension methods on the Enumerable class. Also, the sample relies on the Northwind sample database, installed on SQL Express. If you want to run the sample, you'll either need to replicate this environment or modify the sample application.

Inside Enumerable
The Enumerable class plays an important role in every LINQ query you create. For example, imagine that your project includes a Product class and a generic List of Product instances (as in the sample project). You've created an instance of the SimpleDataContext class from the sample that fills data in the Products list, and you write a query like this:
Dim db As New SimpleDataContext()
Dim query = _
  From prod In db.Products _
  Where prod.CategoryID = 1 _
  Order By prod.ProductName _
  Select prod.ProductName
If you've thought much about this, you've probably realized that, behind the scenes, the Visual Basic compiler must convert the SQL-like syntax into some set of method calls on some instance of a type. Indeed, in this example, the Products property of the SimpleDataContext class is of the List(Of Product) type and, as you might have guessed, the List(Of T) type implements the IEnumerable(Of T) interface. The compiler converts the LINQ keywords into corresponding calls to extension methods on the Enumerable type. Specifically, you could create the same exact set of instructions by rewriting the previous LINQ query like this:
Dim query = _
  db.Products _
  .Where(Function(prod) CBool(prod.CategoryID = 1)) _
  .OrderBy(Function(prod) prod.ProductName) _
  .Select(Function(prod) prod.ProductName)
If you examine the intermediate language (IL) created by the compiler, using the LINQ query that employs Visual Basic keywords, you will see that the compiler converts the Visual Basic keywords into calls to the corresponding methods of the Enumerable class. Doing this results in the extension of the System.Collections.Generic.List(Of T) class behavior. (As a matter of fact, it's easy to determine this—look in the documentation for this particular class's members, and you'll see a list of extension methods, most of which are defined by the Enumerable class.)
What does this all mean to you? Because the Enumerable class's extension methods can process many other classes—including Array and List—you can use methods of the Enumerable class not only to create LINQ queries but also to manipulate the behavior of arrays and other data structures.
If you examine the documentation for the extension methods on the Enumerable class, you'll find that each member is Shared. As a result, you have two options for calling a method. First, you can call it as a method of the declaring type, like this, passing in the instance of the extended class (assuming that the results variable is of a type can be processed by the Count method):
Dim itemCount = _
  Enumerable.Count(results, _
  Function(prod) prod.CategoryID = 3)
You can also call Count as if it were an instance method on the collection itself. When you call the extension method this way, you don't need to supply the first parameter:
Dim itemCount = _
  results.Count(Function(prod) _
  prod.CategoryID = 3)
Visual Basic and C# handle the conversion from keywords into extension method calls differently. Visual Basic provides more keywords that you can use within your LINQ queries than does C#. For example, you can create a LINQ query in Visual Basic using the Take keyword, like this:
Dim db As New SimpleDataContext()
Dim query = _
  From prod In db.Products _
  Where prod.CategoryID = 1 _
  Order By prod.ProductName _
  Take 10 _
  Select prod.ProductName
You could also use the Take extension method, provided by the Enumerable class, like this:
Dim query = _
  (From prod In db.Products _
  Where prod.CategoryID = 1 _
  Order By prod.ProductName _
  Select prod.ProductName).Take(10)
Because C# doesn't supply the Take keyword, you would have no other choice than to call the Take method explicitly. Figure 1 lists the mappings between Visual Basic LINQ query keywords and the corresponding methods of the Enumerable class.
Figure 1 LINQ Keyword Links to Enumerable Methods
Visual Basic LINQ Query Keyword | Enumerable Method
Aggregate … Into … All | All
Aggregate … Into … Any | Any
Aggregate … Into … Average | Average
Aggregate … Into … Count | Count
Aggregate … Into … LongCount | LongCount
Aggregate … Into … Max | Max
Aggregate … Into … Min | Min
Aggregate … Into … Sum | Sum
Distinct | Distinct
Group By | GroupBy
Group Join | GroupJoin
Order By | OrderBy
Order By … Descending | OrderByDescending
Order By (with multiple fields) | ThenBy
Order By … Descending (with multiple fields) | ThenByDescending
Select | Select
Skip | Skip
Take | Take
Take While | TakeWhile
Where | Where

Creating Sequences
The Enumerable class provides several Shared methods that are not extension methods but exist for you to create new sequences. I will start by examining these simple methods.
The Enumerable.Range method creates a new sequential list of integers. You specify the starting value and the number of items in the list. The method returns an IEnumerable(Of Integer) sequence. You can use this method to solve the original problem in this article (that is, retrieving a list of numbers between two endpoints, in random order).
The following code fills the list using the Range method, then calls the Enumerable.OrderBy method with a simple lambda expression to arrange the sequence in random order:
' From RangeDemo in the sample:
Dim rnd As New System.Random
Dim items = Enumerable.Range(1, 10)
Dim randomList = _
  items.OrderBy(Function() rnd.Next())
Running the sample procedure multiple times returns different results each time, but you should see a list like the following each time you execute the code:
6, 8, 7, 1, 2, 9, 10, 4, 3, 5
The Enumerable.Reverse method returns its input sequence in reverse order. The following code calls the Reverse method both as a Shared method of the Enumerable class and as an instance method on the list resulting from calling the Enumerable.Range method:
'From ReverseDemo in the sample:
Dim items = Enumerable.Range(1, 10)
Dim reversed = items.Reverse()
reversed = Enumerable.Reverse(items)
However you call the Reverse method, you'll end up with the following list:
10, 9, 8, 7, 6, 5, 4, 3, 2, 1
The Enumerable.Repeat method creates a list containing the value you supply, repeated as many times as you specify:
'From RepeatDemo in the sample:
Dim repeated = Enumerable.Repeat("Hello", 5)
After calling the previous code, the repeated variable contains the following items:
Hello, Hello, Hello, Hello, Hello
Use the Enumerable.Empty method to create an empty IEnumerable(Of T) that's set up to contain the specified data type. Using this method, you could create the empty instance of the collection and add items to it as necessary. By initializing the collection to an empty collection, you never have to worry about whether the reference is Nothing:
'From EmptyDemo in the sample:
Dim emptyList = _
 Enumerable.Empty(Of Customer)()

Selecting Sequences
The Enumerable class provides two different extension methods that allow you to select a sequence of elements, projecting the elements from the input sequence into an output sequence. The Enumerable.Select method simply projects each element into a new form. The following example takes a sequence containing customers in the U.S. from the Northwind Customers table, then projects the results into a sequence that contains a numbered list of contact names:
' From SelectDemo in the sample:
Dim db As New SimpleDataContext
Dim results = _
  From cust In db.Customers _
  Where cust.Country = "USA"
Dim selectResults = results.Select( _
  Function(cust, index) _
  String.Format("{0}. {1}", _
  index + 1, cust.ContactName))
Note that the Select method has multiple overloaded versions. After running this code, selectResults contains a list of values resembling the following:
1. Howard Snyder
2. Yoshi Latimer
3. John Steel
4. Jaime Yorres
5. Fran Wilson
6. Rene Phillips
7. Paula Wilson
8. Jose Pavarotti
9. Art Braunschweiger
10. Liz Nixon
11. Liu Wong
12. Helvetius Nagy
13. Karl Jablonski
The Enumerable.SelectMany method takes a collection of items, each of which can be a collection of items, and collapses the two-dimensional input into a single-dimensional output sequence. This method can be useful if you need to take, for example, a list of categories, each of which contains products, and end up with a list of products with corresponding category information. The following example retrieves a list of categories for which the category contains less than seven products, and projects the result into a single list containing information about each product, along with category information:
' From SelectManyDemo in the sample:
Dim db As New SimpleDataContext
Dim categories = _
  From cat In db.Categories Where cat.Products.Count < 7
Dim manyResults As IEnumerable(Of String) = _
  categories.SelectMany(Function(cat, index) _
  cat.Products.Select( _
  Function(prod As Product) String.Format("{0}. Category {1}: {2}", _
    index, prod.CategoryID, prod.ProductName)))
In this case, the lambda expression passed to the SelectMany method accepts a reference to a specific category, working through each category in turn. This particular overload of the SelectMany method passes the index of the category along with the category to the lambda expression. Given the category and its index, the lambda expression calls the Select method on the Products property of the category, allowing it to work on each of the products within the category. The sample returns a sequence containing the following list of strings. The call to SelectMany flattened the original category/product hierarchy into a single list of values:
0. Category 6: Mishi Kobe Niku
0. Category 6: Alice Mutton
0. Category 6: Thüringer Rostbratwurst
0. Category 6: Perth Pasties
0. Category 6: Tourtière
0. Category 6: Pâté chinois
1. Category 7: Uncle Bob's Organic Dried Pears
1. Category 7: Tofu
1. Category 7: Rössle Sauerkraut
1. Category 7: Manjimup Dried Apples
1. Category 7: Longlife Tofu

Retrieving a Single Element
Often, you will need to retrieve and work with a single element from a sequence. The Enumerable class provides several methods that allow you to filter the contents of a sequence to a single element. The Enumerable.First and Enumerable.Last methods return the first or last elements in the sequence, respectively. The Enumerable.Single method returns an element given a function or lambda expression that specifies a single element. The Enumerable.ElementAt method returns a single element at a particular index within the sequence.
Each of these methods throws an exception if the requested element doesn't exist in the sequence—or, for the Single method, if the restriction results in either zero elements or multiple elements. Each of these methods also allows you to specify a lambda expression or function to restrict the input data. In any case, the resulting sequence must contain a single element.
The Enumerable class also provides similar methods that return a single element if possible and a default member of the sequence's type if not. These methods are Enumerable.FirstOrDefault, Enumerable.LastOrDefault, Enumerable.SingleOrDefault, and Enumerable.ElementAtOrDefault. Figure 2 demonstrates the behavior of these methods. Running the sample places the following data into the output stream:
ALFKI: Maria Anders
ALFKI: Maria Anders
Last in USA: Karl Jablonski
Sequence contains no elements
Specified argument was out of the range of valid values.
Parameter name: index
CustomerID = XXXXX is Nothing: True
Customer 2 in USA is Nothing: False
Customer 200 in USA is Nothing: True
Figure 2 Retrieving Elements with Enumerable Methods
' From SingleElementDemo in the sample:

Dim oneCustomer As Customer
Dim sw As New StringWriter

Dim db As New SimpleDataContext

Dim query0 = From cust In db.Customers

Dim query1 = _
  From cust In db.Customers _
  Where cust.CustomerID = "ALFKI"

Dim query2 = _
  From cust In db.Customers _
  Where cust.Country = "USA"

Dim query3 = _
  From cust In db.Customers _
  Where cust.CustomerID = "XXXXX"

' This works fine, finds the customer whose CustomerID is ALFKI:
oneCustomer = query1.Single
sw.WriteLine("ALFKI: {0}", oneCustomer.ContactName)

' You can supply an expression to filter, when calling
' the First, Last, Single, and so on, methods:
oneCustomer = query0.Single( _
  Function(cust) cust.CustomerID = "ALFKI")
sw.WriteLine("ALFKI: {0}", oneCustomer.ContactName)

' This works fine, finds the last customer in the USA:
oneCustomer = query2.Last
sw.WriteLine("Last in USA: {0}", oneCustomer.ContactName)

' These two raise exceptions:
Try
  ' There's no customer use customer ID is "XXXXX":
  oneCustomer = query3.Single
Catch ex As Exception
  sw.WriteLine(ex.Message)
End Try

Try
  ' There aren't 200 customers in the USA:
  oneCustomer = query2.ElementAt(200)
Catch ex As Exception
  sw.WriteLine(ex.Message)
End Try

' These don't raise exceptions, but return default values
' if they fail:
oneCustomer = query3.SingleOrDefault
sw.WriteLine("CustomerID = XXXXX is Nothing: {0}", _
             oneCustomer Is Nothing)

' Because there are at least two customers in the USA,
' method returns a valid, single customer:
oneCustomer = query2.ElementAtOrDefault(2)
sw.WriteLine("Customer 2 in USA is Nothing: {0}", _
             oneCustomer Is Nothing)

oneCustomer = query2.ElementAtOrDefault(200)
sw.WriteLine("Customer 200 in USA is Nothing: {0}", _
             oneCustomer Is Nothing)

Filtering Sequences
Use the Enumerable.Where, Enumerable.Distinct, or Enumerable.OfType method to filter the contents of an existing sequence, returning a subset of the original data in an output sequence. The OfType method filters the input sequence based on a specified type. Imagine that you want to perform an operation on only a particular type of control on a form. Using the OfType method, you can restrict the collection of controls exposed by the form and iterate through only the relevant subset of controls.
The following example fills a generic List with data and retrieves an output list of integers, and then retrieves a list of strings:
' From OfTypeDemo in the sample:
Dim items As New List(Of Object)
items.Add("January")
items.Add(0)
items.Add("Monday")
items.Add(3)
items.Add(5)
items.Add("September")

Dim numbers = items.OfType(Of Integer)()
Dim strings = items.OfType(Of String)()
After running the code, the two output lists contain the following data:
0, 3, 5
January, Monday, September
The Enumerable.Where method allows you to specify a condition that filters the input sequence. A second overloaded version provides access to the index of each item in the collection so you can filter based on the index as well. The following example uses both the Where and Select methods, first filtering and then projecting the results, retrieving a sequence of files in the C:\Windows folder smaller than 100 bytes:
' From WhereDemo in the sample:
Dim files As IEnumerable(Of FileInfo) = _
  New DirectoryInfo("C:\Windows").GetFiles()
Dim fileResults = _
  files _
  .Where(Function(file) file.Length < 100) _
  .Select(Function(file) _
    String.Format("{0} ({1})", file.Name, file.Length))
Because arrays implement the IEnumerable interface, you can use the Where method to filter array contents just as with any other collection. In this case, the code filters the array of FileInfo objects returned by the GetFiles method. Also note that the example cascades calls to the Where method and then the Select method. On my computer, the sample returned the following results:
Addrfixr.ini (62)
bthservsdp.dat (12)
iltwain.ini (36)
S86D5A060.tmp (48)
setuperr.log (0)
You can use the second overload of the Where method to access the index of each input item that is processed. For example, the following Where method call returns files less than 100 bytes in size that also appear within the first 20 input files:
' From WhereDemo in the sample:
fileResults = _
  files _
  .Where(Function(file, index) _
    (file.Length < 100) And (index < 20)) _
  .Select(Function(file) _
    String.Format("{0} ({1})", file.Name, file.Length))
In this case, on my computer, the results looked like this (the missing files fell outside the first 20 files processed):
Addrfixr.ini (62)
bthservsdp.dat (12)
iltwain.ini (36)
The Enumerable.Distinct method allows you to filter a list so that the output sequence contains only the unique items from the input list. For simple values (strings, numbers, and so on), this process is easy.
For lists of complex objects, how can the runtime engine perform the comparisons to determine whether or not two objects are duplicates? It can't simply compare the instance variables, as this would, in practice, simply compare the memory addresses of the objects. Instead, for complex objects, you must supply a comparer—an instance of a class that implements IEqualityComparer(Of T) to perform the comparison. (This same issue applies to several methods of the Enumerable class, and it will show up again later in this column.)
Suppose you had a sequence containing customer information and you wanted a unique list of countries from where those customers came. If you had a simple list of countries, you could use the default comparer to compare the strings. In this case, you have a list of customers instead. (Yes, you could use the Select method to project the list so that you only had a list of countries, but then you'd lose all the remaining data for other calculations using the list.)
The sample project includes the CustomerCountryComparer class, which implements the IEqualityComparer(Of Customer) interface, as shown in Figure 3. The sample code demonstrates two different ways to use the Distinct method.
Figure 3 IEqualityComparer
Public Class CustomerCountryComparer
  Implements IEqualityComparer(Of Customer)

  Public Function Equals1( _
    ByVal x As Customer, ByVal y As Customer) As Boolean _
    Implements IEqualityComparer(Of Customer).Equals

    Return x.Country.Equals(y.Country)

  End Function

  Public Function GetHashCode1( _
    ByVal obj As Customer) As Integer _
    Implements IEqualityComparer(Of Customer).GetHashCode

    Return obj.Country.GetHashCode

  End Function
End Class
The first projects a list of customers to a list of country names first, and then uses the default string comparer to reduce the list so that it's unique. The second uses the specific CustomerCountryComparer class to perform the comparisons and create the unique list, and then projects the resulting distinct list of customers into a list of strings:
' From DistinctDemo in the sample:
Dim db As New SimpleDataContext

Dim countries As IEnumerable(Of String) = _
  db.Customers _
  .Select(Function(cust) cust.Country) _
  .Distinct()

countries = customers _
  .Distinct(New CustomerCountryComparer) _
  .Select(Function(cust) cust.Country)
You may need to handle empty sequences gracefully, and the Enumerable.DefaultIfEmpty method returns either the entire input sequence or a default-valued instance if the sequence is empty. Using this method, you're guaranteed that you always have at least a single item of the sequence's type.
The procedure in Figure 4 demonstrates one use for the DefaultIfEmpty method. This code fills the noCustomers variable with an empty list (there are no customers whose country is "XXX"), then creates the noCustomers1 list calling the DefaultIfEmpty method. Running the sample code produces the following output:
noCustomers contains 0 element(s). noCustomers1 contains 1 element(s).
XXXXX
Figure 4 Using DefaultIfEmpty
' From DefaultIfEmptyDemo in the sample:
Dim db As New SimpleDataContext
Dim sw As New StringWriter

Dim noCustomers = _
  From cust In db.Customers Where cust.Country = "XXX"
Dim noCustomers1 = _
  noCustomers.DefaultIfEmpty()

Dim results = _
  String.Format( _
  "noCustomers contains {0} element(s). " & _
  "noCustomers1 contains {1} element(s).", _
  noCustomers.Count, noCustomers1.Count)
sw.WriteLine(results)

' You can specify the exact value to use if the
' collection is empty:
noCustomers1 = noCustomers.DefaultIfEmpty( _
  New Customer With {.CustomerID = "XXXXX"})
For Each cust In noCustomers1
  sw.WriteLine(cust.CustomerID)
Next
noCustomers1 contains one element (a Customer with no information filled in). You can call the DefaultIfEmpty method supplying a default value for the singleton item, as in the second method call.

Ordering Sequences
You can use Enumerable.OrderBy, Enumerable.ThenBy, Enumerable.OrderByDescending, and Enumerable.ThenByDescending to supply an initial ordering and then one or more secondary orderings (in ascending or descending order). This sample starts with a sequence of customers and then applies ordering using several of these methods:
' From OrderByDemo in the sample:
Dim db As New SimpleDataContext
Dim customers = _
  From cust In db.Customers _
  Where cust.ContactTitle = "Owner"

' Sort first by country, then by city descending, then by name:
Dim results = _
  customers _
  .OrderBy(Function(cust) cust.Country) _
  .ThenByDescending(Function(cust As Customer) cust.City) _
  .ThenBy(Function(cust As Customer) cust.ContactName) _
  .Select(Function(cust) String.Format("({0}, {1}) {2}", _
  cust.Country, cust.City, cust.ContactName))
Note that Visual Basic won't complain if you don't specify the data types for the parameter to the lambda expression in the ThenBy and ThenByDescending method calls, but if you don't, you won't get any IntelliSense® to help you. Only the first ordering method can infer the data type of the lambda expression parameter. Oddly enough, the Select method correctly infers the type, even at the end of the long chained list of methods. The sample sorts the customers whose title is "Owner" first by country, then descending by city, and, finally, by contact name within each city:
(Denmark, Kobenhavn) Jytte Petersen
(France, Paris) Marie Bertrand
(France, Nantes) Janine Labrune
(France, Marseille) Laurence Lebihan
(Germany, Köln) Henriette Pfalzheim
(Mexico, México D.F.) Ana Trujillo
(Mexico, México D.F.) Antonio Moreno
(Mexico, México D.F.) Miguel Angel Paolino
(Norway, Stavern) Jonas Bergulfsen
(Poland, Warszawa) Zbyszek Piestrzeniewicz
...

Verifying Sequences
Some applications require you to verify that specific conditions are met within a sequence. Do any of the elements meet a criterion? Do all the elements meet a criterion? Does a particular item appear within the sequence? Are two sequences equal? The Enumerable class provides methods that provide all this information.
To determine whether a sequence contains any elements at all, call the Enumerable.Any method without any parameters. To supply a condition and to determine whether any elements in the sequence match the condition, pass a function to the method. The method returns True if any elements of the input sequence exist or match the supplied condition (see Figure 5).
Figure 5 Finding a Sequence
' From AnyDemo in sample:
Dim db As New SimpleDataContext
Dim results = _
  From product In db.Products _
  Where product.CategoryID = 1

' Determine if a list has any elements:
Dim anyElements = results.Any()

' Call as extension method or shared method
' of IEnumerable:
Dim matchingElements = _
  results.Any(Function(prod) prod.ProductName.StartsWith("M"))
matchingElements = _
  Enumerable.Any(results, _
  Function(prod As Product) prod.ProductName.StartsWith("M"))
To determine whether all members of a sequence meet a criterion, call the Enumerable.All method, supplying a function that specifies the condition. This example determines if all the elements in the results collection have a UnitsInStock value greater than 5:
' From AllDemo in sample:
' Is it true that all products have more than 5 units in stock?
Dim db As New SimpleDataContext
Dim results = _
  From product In db.Products _
  Where product.CategoryID = 1
Dim allElements = _
  results.All(Function(prod) CBool(prod.UnitsInStock > 5))
To determine whether a sequence contains a particular element, call Enumerable.Contains. If you're searching for a simple value, like a string or an integer, you can use the default comparer, and you only need to supply the value you're seeking. If you're attempting to determine whether a more complex object exists within a sequence, you must supply an instance of a class that implements IEqualityComparer(Of T).
The sample code in Figure 6 demonstrates both calls to Enumerable.Contains. The first example simply determines whether the number 5 appears within a list; the second determines whether an element corresponding to Chai appears within the list of products (this example uses the same ProductComparer class you saw earlier in the column).
Figure 6 Finding Items in a Sequence
' From ContainsDemo in the sample:
' Simple check to see if list contains a value:
Dim numbers = Enumerable.Range(1, 10)
Dim contains5 = numbers.Contains(5)

' More complex objects require more complex comparison:
Dim db As New SimpleDataContext
Dim results = _
  From product In db.Products _
  Where product.CategoryID = 1

Dim item As New Product _
  With {.ProductID = 1, .ProductName = "Chai"}
Dim containsChai = results.Contains(item, New ProductComparer())
To determine whether two sequences are equal, call the Enumerable.SequenceEqual method. Just like the Enumerable.Contains method, you can either compare two sequences of simple values using the default comparer or two sequences of more complex objects using a custom comparer.
If the two sequences don't contain the same types or are of different lengths, the comparison fails immediately. If they contain the same type and are of the same length, the Enumerable.SequenceEqual method compares each item using the specified comparer. Figure 7 demonstrates both ways of calling the Enumerable.SequenceEqual method. In both cases, the return value is False.
Figure 7 Comparing Sequences
' From SequenceEqualDemo in sample:
Dim rnd As New Random
Dim start = rnd.Next

Dim count = 10
Dim s1 = Enumerable.Range(start, count)

' Choose a different random starting point:
start = rnd.Next
Dim s2 = Enumerable.Range(start, count)
Dim sequencesEqual = s1.SequenceEqual(s2)
sw.WriteLine("sequencesEqual = {0}", sequencesEqual)

' What if there's no default comparer? Must use your own:
Dim products1 = _
  From prod In db.Products _
  Where prod.CategoryID = 1
Dim products2 = _
  From prod In products1 _
  Where prod.UnitPrice > 5

sequencesEqual = _
  products1.SequenceEqual(products2, New ProductComparer())
sw.WriteLine("sequencesEqual = {0}", sequencesEqual)
In my next column, I'm going to focus on the remaining methods of the Enumerable class, including some fun methods that perform useful set-based operations. Until then, you can download the sample application and give the Enumerable class's methods a try. They'll certainly enrich your work with data structures in Visual Studio® 2008.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a Senior Consultant with MCW Technologies and a courseware author for AppDev (www.appdev.com). He is coauthor of ASP .NET Developers Jumpstart, Access Developer's Handbook, and VBA Developer's Handbook, 2nd Edition. Reach him at keng@mcwtech.com.


Office Space
Automated SharePoint Site Branding
Ted Pattison

Code download available at: OfficeSpace2008_07.exe (1,315 KB)
Browse the Code Online
Contents
Introducing the LitwareBranding Solution
Creating a Utility Class for Brand Management
Swapping Out the Master Page
Registering an HttpModule in the Web.config File
Initializing Branding during Feature Activation
There is one request I have heard often since I began working with SharePoint® technologies. Rather than using standard SharePoint sites, companies often want to create unique sites. Whether you are using SharePoint technologies to create your company's Internet-facing Web sites or an intranet environment with hundreds of team collaboration sites, you might be asked to manage the visual aspects of your SharePoint sites such as page layout and the use of fonts and colors.
Windows® SharePoint Services (WSS) 3.0 provides branding through themes. A user can apply a WSS theme to a site to change its appearance. Note that themes in WSS are entirely different from themes in ASP.NET. WSS implements its infrastructure for themes by copying a cascading style sheet (CSS) file from the file system of the Web server into the context of the current site and dynamically linking to this CSS file from each page.
WSS themes have limitations that make them less interesting to developers. The first limitation is that WSS themes cannot integrate a custom Master Page to customize the page layouts used within a site. Furthermore, a WSS theme can only be applied on a per-site basis as opposed to across a site collection. There is no support to perform a single action that applies a theme to all the sites within a site collection. Instead, you would have to go to each site separately and apply the same theme to maintain a consistent appearance and functionality across a site collection.
The SharePoint Designer provides another means for branding sites in WSS or Microsoft® Office SharePoint Server (MOSS) 2007. You can use the SharePoint Designer to customize a site's Master Page and to edit and integrate custom CSS files. However, when working with a site collection based on standard WSS sites, the SharePoint Designer doesn't provide a way to synchronize all the sites within a site collection to use the same version of a customized Master Page. This can possibly lead to scenarios where you are forced to copy and paste your Master Page edits across multiple Master Page files.
Things get better when you use the SharePoint Designer together with MOSS because MOSS provides a Web Content Management (WCM) infrastructure through the use of publishing portals. A MOSS publishing portal is a site collection containing publishing sites that can automatically synchronize pages across all the sites to use the same Master Page and the same CSS file.
However, it is important to keep in mind that using the SharePoint Designer has some noteworthy limitations when it comes to creating a branding solution intended for use across multiple site collections or in enterprise scenarios. Remember that changes made with the SharePoint Designer are always made as one-off customizations within the context of a particular site. This means a customization made with the SharePoint Designer can be tricky or impossible to migrate between site collections or between farms. Also, it's impractical to integrate site customizations made with SharePoint Designer into a source code management system such as Visual Studio® Team System.
This month I am going to walk through developing a custom branding solution for SharePoint sites that leverages the capabilities of WSS to integrate custom Master Pages and custom CSS files at the level of the site collection. I am going to walk through the code within a sample branding solution named LitwareBranding that has been developed using a Visual Studio project and that is deployed using a solution package. You can find all the code in the code download for this issue.
Employing Visual Studio and generating a solution package makes it possible to deploy your branding solution to any farm running WSS 3.0 or MOSS 2007. It also makes it easy to create a single source file for each Master Page and CSS file that can be reused across multiple site collections or across farms. Furthermore, the Visual Studio approach can be far more appealing than using the SharePoint Designer because it allows you to check all your source files into a source code management system. You can also more easily move your work from a development farm to a staging farm for testing and then into production.

Introducing the LitwareBranding Solution
This month's column is accompanied by a sample Visual Studio project named LitwareBranding. Figure 1 shows the structure of the LitwareBranding project within Visual Studio. As you can see, the solution has been designed using three different features whose purpose and implementation will be explained throughout this month's column. The central feature within this solution is named LitwareBranding, which has been designed to be activated by a user within the scope of a WSS or MOSS site collection. Activation of the LitwareBranding feature forces the LitwareBranding solution to apply all the branding techniques that will be covered in this month's column.
Figure 1 LitwareBranding Solution in Visual Studio
Let's begin our code walkthrough by looking at the feature.xml file for the central feature named LitwareBranding, shown in Figure 2. The LitwareBranding feature has been defined with a Scope attribute value of Site, which means it is scoped to the site collection level. The idea is that a site collection owner should be able to activate a single feature to apply the Litware corporate branding to all the sites within the current site collection.
Figure 2 Feature.xml
<Feature
  Id="065E2243-B968-4F14-BAAE-610BB975EFB7"
  Title="A sample feature: LitwareBranding"
  Description="Demoware created for Ted Pattison's OfficeSpace column"
  Hidden="FALSE"
  Scope="Site"
  ImageUrl="LitwareBranding\AfricanPith32.gif"
  ReceiverAssembly="LitwareBranding, [4-part assembly name]"
  ReceiverClass="LitwareBranding.FeatureReceiver"
  xmlns="http://schemas.microsoft.com/sharepoint/">

  <ElementManifests>
    <ElementManifest Location="elements.xml"/>
    <ElementManifest Location="stapling.xml" />
    <ElementFile Location="LitwareBranding.master"/>
  </ElementManifests>

</Feature>
The LitwareBranding feature contains a Master Page template named LitwareBranding.master and an element manifest named elements.xml. Within the elements.xml manifest there is declarative XML provisioning logic to create an instance of this Master Page template. This logic is defined by a Module element with an inner File element:
<Module Name="MasterPages"
        Path="" List="116"
        Url="_catalogs/masterpage" >

  <File Url="LitwareBranding.master"
        Type="GhostableInLibrary" />

</Module>
During feature activation, this Module element forces WSS to provision an instance of the Master Page template named LitwareBranding.master in the Master Page gallery of the top-level site. After this Master Page instance has been provisioned, it's then possible to supply code that programs against the WSS object model to synchronize pages through the site collection to link to this Master Page, as I will show in the next section.

Creating a Utility Class for Brand Management
The LitwareBranding project contains a class named BrandManager. This utility class encapsulates all the code that programs against the WSS object model to apply and remove various branding elements. There are static methods exposed by the BrandManager class that accomplish the following tasks:
  • Synchronizing site pages to link to LitwareBranding.master
  • Synchronizing pages to use an alternate CSS file
  • Synchronizing pages to use a custom graphic for the site logo
  • Adding support to swap out the Master Page for application pages instead
Before drilling down into specific members, take a look at Figure 3, which shows all the members of the BrandManager class. This class contains several static properties that parse together URLs pointing to various resources such as Master Pages and a CSS file. For example, the SiteCollectionUrl property returns the Web application-relative path to the current site collection. The DefaultMasterPageUrl returns a Web application-relative path to the default master page named default.master in the Master Page gallery of the current site collection's top-level site. This path is parsed together using the SiteCollectionUrl property together with the following site-relative path:
_catalogs/masterpage/default.master
Figure 3 BrandManager Class
public class BrandManager {
  // read-only properties
  public static string SiteCollectionUrl
  public static string DefaultMasterPageUrl
  public static string CustomMasterPageUrl
  public static string CustomCssUrl
  public static string CustomSiteLogoUrl
  // utility branding methods
  public static void ConfigureMasterUrl(bool ApplyMasterUrl) {}
  public static void ConfigureCustomMasterUrl(bool ApplyCustomMasterUrl) {}
  public static void ConfigureAlternateCss(bool ApplyCustomCss) {}
  public static void ConfigureSiteLogo(bool ApplySiteLogo) {}
  public static void ConfigureApplicationPageMaster(bool ApplyApplicationPageMaster) {}
}
The CustomMasterPageUrl property returns a Web app relative path to the Master Page instance named LitwareBranding.master in the Master Page gallery of the current site collection's top-level site. This path is parsed together using the SiteCollectionUrl property together with the following site-relative path:
_catalogs/masterpage/litwarebranding.master
Now, let's examine the ConfigureMasterUrl method, which enumerates through every site in the current site collection and updates each site's MasterUrl property to point to either the custom Master Page or the default Master Page:
public static void ConfigureMasterUrl(bool ApplyMasterUrl) {
  // determine MasterUrl property setting
  string MasterUrlPath =
    (ApplyMasterUrl ?
    CustomMasterPageUrl :
    DefaultMasterPageUrl);

  // update MasterUrl property for all sites
  foreach (SPWeb site in SPContext.Current.Site.AllWebs) {
    site.MasterUrl = MasterUrlPath;
    site.Update();
  }
}
Note that the MasterUrl property affects all site pages that have been created with a MasterPageFile property setting of ~masterurl\default.master. This includes the standard default.aspx page template that provides the home page for many SharePoint site templates including Team Site and Blank site. It also includes the form pages such as AllItems.aspx and NewItem.aspx that are used by standard WSS lists types.
Page layouts in a MOSS publishing site will not be affected when you update the MasterUrl property. This includes all the content pages within the Pages library of a MOSS publishing site.
Instead, these content pages are designed to use a dynamic token for the MastePageFile attribute, which follows the form of ~masterurl\custom.master. This token is different from the ~masterurl\custom.master token because it is switched out using the CustomMasterUrl property instead of the MasterUrl property. Therefore, the BrandManager class provides a second method named ConfigureCustomMasterUrl that can be used to switch out the Master Page for content pages created in the Pages library of MOSS publishing sites:
public static void ConfigureCustomMasterUrl(
  bool ApplyCustomMasterUrl) {
  // determine MasterUrl property setting
  string CustomMasterUrlPath =
    (ApplyCustomMasterUrl ?
    CustomMasterPageUrl :
    DefaultMasterPageUrl);

  // update MasterUrl property for all sites
  foreach (SPWeb site in SPContext.Current.Site.AllWebs) {
    site.CustomMasterUrl = CustomMasterUrlPath;
    site.Update();
  }
}
In addition to swapping out Master Pages, the LitwareBranding solution also integrates a custom CSS file named styles.css. The LitwareBranding solution uses a strategy of deploying the CSS file named styles.css in a directory nested inside the LAYOUTS directory so that it can be deployed a single time on the file system of each front-end Web server. The value of deployment within the LAYOUTS directory is that a file such as style.css can be deployed once on the Web server's file system and yet still be accessible from any site within the current farm using a standard URL that is returned by the CustomCssUrl property:
public static string CustomCssUrl {
  get {
    return "/_layouts/1033/STYLES/LitwareBranding/styles.css";
  }
}
The BrandManager class provides a method named ConfigureAlternateCss that uses the CustomCssUrl property to assign the URL to the AlternateCSS property of each site within the current site collection:
public static void ConfigureAlternateCss(bool ApplyCustomCss) {
// determine MasterUrl property setting
  string AlternateCssUrl =
    (ApplyCustomCss ?
    CustomCssUrl :
    string.Empty);

  // update AlternateCssUrl for all sites
  foreach (SPWeb site in SPContext.Current.Site.AllWebs) {
    // make sure no theme is enabled
    site.ApplyTheme(string.Empty);
    // apply custom CSS file
    site.AlternateCssUrl = AlternateCssUrl;
    site.Update();
  }
}
The branding capabilities built into WSS also provide an easy way for you to replace the standard WSS site logo that displays on the top-left portion of its pages. The LitwareBranding solution takes advantage of this capability by including a custom graphics file named SiteLogo.gif, which is deployed in a directory nested inside the IMAGES directory. Like the CSS file named styles.css, the graphics file named SiteLogo.gif is also deployed in such a way that it is accessible from any site in the current farm using the URL returned by the CustomSiteLogoUrl property:
public static string CustomSiteLogoUrl {
  get {
    return "/_layouts/images/LitwareBranding/SiteLogo.gif";
  }
}
The ConfigureSiteLogo method of the BrandManager class has been written to enumerate through each site of the current site collection and to update the SiteLogoUrl property:
public static void ConfigureSiteLogo(bool ApplySiteLogo) {
  // determine SiteLogoUrl property setting
  string SiteLogoUrl =
    (ApplySiteLogo ?
    CustomSiteLogoUrl :
    string.Empty);

  // update AlternateCssUrl for all sites
  foreach (SPWeb site in SPContext.Current.Site.AllWebs) {
    site.SiteLogoUrl = SiteLogoUrl;
    site.Update();
  }
}

Swapping Out the Master Page
While SharePoint makes it relatively easy to swap out the Master Page for site pages, it does not provide an obvious way to swap out the Master Page for built-in application pages running out of the LAYOUTS directory such as the standard WSS Site Settings page (settings.aspx). These application pages are linked to a standard WSS Master Page named application.master that WSS deploys within the LAYOUTS directory. However, if you create a branding solution that swaps out the Master Page for all the site pages within a site collection, you might want to create a consistent appearance by swapping out the Master Page for application pages such as settings.aspx so that they also use a customized Master Page.
The LitwareBranding solution demonstrates a technique for swapping out the Master Page for all application pages that link to the standard Master Page file named application.master. The technique used for swapping out the Master Page involves implementing a custom HttpModule that registers an event handler for one of the events in the ASP.NET page lifecycle named PreInit. This is the event that the ASP.NET programming model requires you to implement whenever you want to swap out the Master Page during the initial processing of a page request.
The class LitwareBrandingHttpModule provides the HttpModule used in the LitwareBranding solution to swap out the Master Page (see Figure 4). This HttpModule class registers a handler for the PreRequestHandlerExecute event inside the Init method. Within the method implementation of the PreRequestHandlerExecute event handler, the HttpModule class determines whether the request is based on an HttpHandler object that derives from the ASP.NET Page class. Only in cases where the request is based on a Page-derived object will the HttpModule class register an event handler from the PreInit event. The code in the PreRequestHandlerExecute event handler also checks to make sure the incoming request targets a page inside the _layouts directory, which will always be the case when processing an application page in WSS.
Figure 4 Swap the Master Page with an App Page
using System;
using System.Web;
using System.Web.UI;
using Microsoft.SharePoint;

namespace LitwareBranding {
  public class LitwareBrandingHttpModule : IHttpModule {

    public void Init(HttpApplication context) {
      context.PreRequestHandlerExecute
        += new EventHandler(context_PreRequestHandlerExecute);
    }

    void context_PreRequestHandlerExecute(object sender, EventArgs e) {
      Page page = HttpContext.Current.CurrentHandler as Page;
      if (page != null) {
        // register handler for PreInit event
        page.PreInit += new EventHandler(page_PreInit);
      }
    }

    void page_PreInit(object sender, EventArgs e) {
      // IF – requested page links to application.master
      // THEN – dynamically modify page to link to custom .master file
    }

    public void Dispose() { /* empty implementation */ }

  }
}
It's important to remember that an HttpModule cannot be deployed in a WSS farm for an individual site collection. Instead, an HttpModule must be configured as an all-or-nothing proposition at the Web application level. However, a Web application may contain hundreds of site collections and only certain site collections might have enabled the LitwareBranding feature. Therefore, the PreInit event handler for the HttpModule class must be able to determine whether the current site collection has been configured with the behavior to swap out the Master Page for its application pages.
The LitwareBranding solution solves this problem by creating a custom property in the site collection's top-level site to indicate that swapping out the Master Page for application pages should be enabled. Examine the implementation of the ConfigureApplicationPageMaster method defined within the BrandManager class:
public static void ConfigureApplicationPageMaster(
  bool ApplyApplicationPageMaster) {
  SPWeb TopLevelSite = SPContext.Current.Site.RootWeb;
  if (ApplyApplicationPageMaster) {
    TopLevelSite.Properties["UseCustomApplicationPageMaster"] = "True";
  }
  else {
    TopLevelSite.Properties["UseCustomApplicationPageMaster"] = "False";
  }
  TopLevelSite.Properties.Update();
}
This method creates a custom property named UseCustomApplicationPageMaster on the top-level site and assigns the property a value of True. Now, the method implementation for the PreInit event handler in the HttpModule can look for this property within the current site collection to see whether it has been configured to enable swapping out the application page master. Note that the PreInit event handler also performs several other checks to determine whether it is appropriate to swap out the Master Page within the context of the current request (see Figure 5).
Figure 5 PreInit Event Handler
void page_PreInit(object sender, EventArgs e) {
  Page page = sender as Page;
  if ((page != null) &&
      (page.MasterPageFile != null) &&
      (page.Request.Url.AbsolutePath.Contains("_layouts")) &&
      (SPContext.Current != null)  ) {
    // inspect UseCustomApplicationPageMaster property
    SPWeb site = SPContext.Current.Site.RootWeb;
    string UseCustomApplicationPageMaster =
      site.Properties["UseCustomApplicationPageMaster"];
    if ((!string.IsNullOrEmpty(UseCustomApplicationPageMaster)) &&
       (UseCustomApplicationPageMaster.Equals("True"))) {
      // now replace application.master with customized version
      if (page.MasterPageFile.Contains("application.master")) {
        page.MasterPageFile = "/_layouts/LitwareBranding/application.master";
      }
    }
  }
}
Once the PreInit event handler determines the current site collection has been configured to enable swapping out the Master Page for application pages and also that the current page links to application.master, it modifies the MasterPageFile property of the current ASP.NET Page object to use a custom Master Page located within a solution-specific directory within the LAYOUTS directory at the following path:
/_layouts/LitwareBranding/application.master

Registering an HttpModule in the Web.config File
When deploying a business solution with WSS and MOSS, it is good to distribute your development efforts within a solution package. For example, the Visual Studio project for LitwareBranding builds a single-solution package named LitwareBranding.wsp, which makes it easy and reliable to deploy this solution out to any farm running WSS 3.0 or MOSS 2007. However, the LitwareBranding solution requires adding an HttpModule entry to the web.config file for each Web application that will be running site collections that activate the LitwareBranding feature.
To update the web.config file within specific Web applications, the LitwareBranding solution uses a second feature named LitwareBrandingWebApplication. This feature is scoped to the Web application level and is configured with a feature receiver class to fire event handlers as it is activated and deactivated within the scope of a particular Web application:
<Feature
  Id="FF739C76-0B08-4bc2-A3A2-F61524B492D8"
  Title="Litware Branding Support Feature (WebApplication)"
  Scope="WebApplication"
  Hidden="False"
  ReceiverClass="LitwareBranding. FeatureReceiverWebApplication"
  ReceiverAssembly="LitwareBranding, [4-part assembly name]"
  xmlns="http://schemas.microsoft.com/sharepoint/">

  <!-- no declarative elements -->
  <ElementManifests />

</Feature>
Like any other feature receiver class, the FeatureReceiverWebApplication class inherits from the SPFeatureReceiver class and overrides the four handler methods named FeatureInstalled, FeatureActivated, FeatureDecactivating and FeatureUninstalling. The implementation for FeatureActivated adds the HttpModule entry to the web.config file for the current Web application. The implementation for FeatureDeactivating reverses that operation by removing the HttpModule entry.
When you need to add an entry to the web.config file, it is best to use the SPWebConfigModification class from the WSS object model. The FeatureReceiverWebApplication class contains a utility method named CreateHttpModuleModification that creates and initializes an instance of SPWebConfigModification and passes it back as its return value (see Figure 6).
Figure 6 CreateHttpModuleModification
public SPWebConfigModification CreateHttpModuleModification() {
  SPWebConfigModification modification;
  string ModName = "add[@name='LitwareBrandingModule']";
  string ModXPath = "configuration/system.web/httpModules";
  modification = new SPWebConfigModification(ModName, ModXPath);
  modification.Owner = "LitwareBranding";
  modification.Sequence = 0;
  modification.Type =
    SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
  modification.Value =
    @"<add name=""LitwareBrandingModule"" " + @ "type="
    "LitwareBranding.LitwareBrandingHttpModule, [4-part assembly name]"
    " />";
  return modification;
}
Once you have a utility method such as CreateHttpModuleModification that returns an initialized SPWebConfigModification object, you can simply call this method from event handlers such as FeatureActivated and FeatureDeactivating to add the required HttpModule entry to or remove it from the web.config file for the current Web application (see Figure 7).
Figure 7 Enabling and Disabling the HttpModule
public override void FeatureActivated(
  SPFeatureReceiverProperties properties) {
  SPWebApplication WebApp =
    (SPWebApplication)properties.Feature.Parent;
  WebApp.WebConfigModifications.Add(
    CreateHttpModuleModification());
  WebApp.WebService.ApplyWebConfigModifications();
  WebApp.WebService.Update();
}

public override void FeatureDeactivating(
  SPFeatureReceiverProperties properties) {
  SPWebApplication WebApp =
    (SPWebApplication)properties.Feature.Parent;
  WebApp.WebConfigModifications.Remove(
    CreateHttpModuleModification());
  WebApp.WebService.ApplyWebConfigModifications();
  WebApp.WebService.Update();
}

Initializing Branding during Feature Activation
It's time to put all these pieces together. When a user activates the LitwareBranding feature within a specific site collection, there is a feature receiver with a FeatureActivated event handler that uses the BrandManager class to apply all the various branding elements. There is also a FeatureDeactivating event handler that removes all the branding elements during feature deactivation (see Figure 8).
Figure 8 Applying and Removing Branding Elements
public override void FeatureActivated(SPFeatureReceiverProperties properties) {
  EnsureWebApplicationFeatureEnabled();
  BrandManager.ConfigureMasterUrl(true);
  BrandManager.ConfigureCustomMasterUrl(true);
  BrandManager.ConfigureAlternateCss(true);
  BrandManager.ConfigureSiteLogo(true);
  BrandManager.ConfigureApplicationPageMaster(true);
}

public override void FeatureDeactivating(SPFeatureReceiverProperties properties) {
  BrandManager.ConfigureMasterUrl(false);
  BrandManager.ConfigureCustomMasterUrl(false);
  BrandManager.ConfigureAlternateCss(false);
  BrandManager.ConfigureSiteLogo(false);
  BrandManager.ConfigureApplicationPageMaster(false);
}
Note the call to the EnsureWebApplicationFeatureEnabled method at the beginning of the FeatureActivated event handler. This method has been written to ensure that the Web application-level feature named LitwareBrandingWebApplication has been activated so that the HttpModule, which swaps out the Master Pages for application pages, is properly registered with ASP.NET:
public void EnsureWebApplicationFeatureEnabled() {
  // make sure feature which adds HttpModule to web.config is active
  SPSecurity.RunWithElevatedPrivileges(delegate() {
    using (SPSite siteCollection = new SPSite(SPContext.Current.Site.ID)) {
      try {
        Guid FeatureId = new Guid("FF739C76-0B08-4bc2-A3A2-F61524B492D8");
        siteCollection.WebApplication.Features.Add(FeatureId);
      }
      catch { }
    }
  });
}
WSS and MOSS pose another design problem when creating a site collection-scoped branding solution: you must decide how to deal with properly initializing the branding for child sites as they are created. The code inside the BrandManager class only affects the branding properties of existing sites. Therefore, the LitwareBranding solution includes a third feature named LitwareBrandingChildSiteInitializer that contains the following event handler for the FeatureActivated event:
// fired whenever a new site is created
public override void FeatureActivated(SPFeatureReceiverProperties properties) {
  SPWeb ChildSite = (SPWeb)properties.Feature.Parent;
  SPWeb TopLevelSite = ChildSite.Site.RootWeb;
  ChildSite.MasterUrl = TopLevelSite.MasterUrl;
  ChildSite.CustomMasterUrl = TopLevelSite.CustomMasterUrl;
  ChildSite.AlternateCssUrl = TopLevelSite.AlternateCssUrl;
  ChildSite.SiteLogoUrl = TopLevelSite.SiteLogoUrl;
  ChildSite.Update();
}
This event handler fires during feature activation and copies the top level site's branding properties into the current child site. Now you must figure out how to get this feature to activate automatically whenever a new child site is created inside a site collection in which the LitwareBranding feature has been activated.
The answer is feature stapling. The LitwareBranding feature provides a FeatureSiteTemplateAssociation element that staples the LitwareBrandingChildSiteInitializer feature to the GLOBAL site definition:
<!-- staple GLOBAL site definition to   itwareBrandingChildSiteInitializer -->
<FeatureSiteTemplateAssociation
  Id="1204A425-D105-46c5-BB2C-473A2F27B563"
  TemplateName="GLOBAL" />
This stapling technique forces automatic feature activation. By stapling the LitwareBrandingChildSiteInitializer feature to the GLOBAL site definition, you are in effect configuring the feature to activate automatically whenever a new site is created no matter what site template has been used.
Note that the standard WSS site template named Blank Site does not participate in features stapling to the GLOBAL site definition. This behavior was added to SharePoint so that sites created from the Blank Site template will work with the MOSS content deployment strategy. If you want to automate the activation of the LitwareBrandingChildSiteInitializer feature in sites created from the Blank Site template, you must add explicit stapling instructions for that site definition and configuration as well:
<!-- staple blank site template to LitwareBrandingChildSiteInitializer -->
<FeatureSiteTemplateAssociation
  Id="1204A425-D105-46c5-BB2C-473A2F27B563"
  TemplateName="STS#1" />

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. Ted has just completed his book Inside Windows SharePoint Services 3.0 for Microsoft Press. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group (www.TedPattison.net).


Cutting Edge
The HTML Message Pattern
Dino Esposito

Code download available at: CuttingEdge2008_07.exe (514 KB)
Browse the Code Online

Contents
AJAX Service Layer
General Pattern for an HTML UI
Enhanced BST Implementation
InnerHTML and DOM
HTML Message Pattern
Performance and Design Considerations
DynamicPopulate Extender
Solutions for Now
A true AJAX architecture is characterized by a neat separation between the presentation and service layers. As I discussed last month, the existence of an explicit contract between the AJAX front end and back end opens up a whole new world of programming possibilities but raises a number of architectural questions (see msdn.microsoft.com/magazine/cc546561). I covered client-side data binding and templates and discussed an implementation of the Browser-Side Templating (BST) pattern. I also briefly introduced the HTML Message (HTM) pattern as an alternative model for rendering the user interface of a Web client. This month I'll provide an enhanced implementation of BST and compare it to HTM solutions.

AJAX Service Layer
I'll refer to the server-side part of a typical AJAX presentation layer as the AJAX service layer to distinguish it from the service layer that generally represents the point of contact between the presentation and middle tier in a standard multitier architecture. Figure 1 illustrates the model.
Figure 1 Typical Multitier System with an AJAX Front End
The communication between the AJAX service layer and the client front end occurs through HTTP endpoints that are exposed by Windows® Communication Foundation (WCF) services and are invoked by JavaScript proxy classes embedded in client pages. Especially when ASP.NET AJAX is the reference platform, designing and consuming services in the AJAX service layer is not a big deal. The trouble begins when you create or update the user interface.
In particular, you need powerful tools, such as JavaScript-based data binding and templates, to manipulate data on the client effectively. In this context, ASP.NET partial rendering is only a short-term solution that doesn't imply a real architectural shift. However, ASP.NET partial rendering saves you from generating the user interface programmatically. Partial rendering preserves the view state and server page lifecycle and allows you to declaratively design the user interface using controls and properties.
Such a model is still probably the best option for relatively simple Web sites, but I doubt it can be effective in an enterprise scenario where the AJAX front end is just the top tier of a deeper service-oriented system. As shown in Figure 1, the AJAX service layer represents the intermediate layer between a JavaScript front end and a set of business services. It smooths the impedance mismatch between the two tiers by adding a security barrier to protect core services in a customer-to-business (C2B) scenario and massaging data to and from JavaScript objects.
The key role played by the AJAX service layer in an AJAX architecture makes the issue of accommodating the HTML user interface based on raw data (the data returned by core services and massaged by the AJAX service layer) virtually unavoidable for architects and developers. How would you dynamically create and update a browser user interface based on raw JavaScript data?

General Pattern for an HTML UI
Years of work with ASP.NET server controls may have blurred the notion of what's really needed to build an HTML user interface. If you've had exposure to custom control development, you probably remember that it is all about accumulating HTML markup in some buffer and then outputting it to the response stream. There's no other way around this general pattern. Enhancements exist only to make it less error prone and more manageable. So the buffer where the HTML is accumulated may be a plain memory stream where you write HTML literals or a more sophisticated hierarchy of components each of which abstracts out a chunk of HTML.
In classic ASP.NET, the response of a Web page is obtained by composing a control tree that is then recursively visited. Each member of the tree receives a stream where it writes down its own HTML markup. In an AJAX model, the response of a request may be raw data serialized as JavaScript Object Notation (JSON), XML, syndication, or whatever else you like, as well as HTML generated on the server.
The BST pattern refers to a situation when a request brings raw data back to the client. The HTM pattern refers to a situation in which the request carries back ready-to-display markup.

Enhanced BST Implementation
In last month's source code, I created a JavaScript class that takes three HTML templates as input, iterates them over a collection of data, and returns the resulting HTML text. The JavaScript code in Figure 2 shows the essence of this code.
Figure 2 Get Templates
function pageLoad()
{
    if (builder === null)
    {
        builder = new Samples.MarkupBuilder();
        builder.loadHeader($get("header"));
        builder.loadFooter($get("footer"));
        builder.loadItemTemplate($get("item"));
    }
}
function getLiveQuotes()
{
    Samples.WebServices.LiveQuoteService.Update(onDataAvailable);
}
function onDataAvailable(results)
{
    var temp = builder.bind(results);
    $get("grid").innerHTML = temp;
}
HTML templates can either be defined as an HTML string or loaded from XML data islands interspersed in the page:
<xml id="item">
    <tr>
        <td align="left">#Symbol</td>
        <td align="right">#Quote</td>
        <td align="right">#Change</td>
     </tr>
</xml>
The arbitrary syntax in the HTML template allows you to find placeholders for the data-bound items. In the sample code included here, #Quote represents the value of the property Quote in the bound data item object.
There are a few ways to improve this code and make it more useful. An essential improvement would be the ability to style items individually. So suppose that the data downloaded to the client represents the current quote and price change of a number of stocks. In this case, you might want to render in green any stocks on the rise and in red those stocks that are on a downswing. For this to happen, though, you need to inject some logic into the rendering process. Figure 3 shows an excerpt from the Samples.MarkupBuilder class used to bind a collection of stock data to a few HTML existing templates (the full code is available in this month's download).
The heart of the MarkupBuilder class is the _generate method, which is in charge of merging templates with data. In Figure 3, the _generate method takes two arguments: data and callback. In the code I presented last month, the same method takes only the data argument.
Figure 3 MarkupBuilder Class
function Samples$MarkupBuilder$_generate(data, itemCallback)
{
    var pattern = /#\w+/g; // Finds all #word occurrences

    var _builder = new Sys.StringBuilder(this._header);

    for(i=0; i<data.length; i++)
    {
        var dataItem = data[i];
        var template = this._itemTemplate;

        var matches = template.match(pattern);
        for (j=0; j<matches.length; j++)
        {
            var text = matches[j];
            var memberName = text.slice(1);

            //Invoke a callback to further modify data to be bound
            var memberData = dataItem[memberName];
            var temp = memberData;
            if (itemCallback !== undefined)
            {
                temp = itemCallback(memberName, dataItem);
            }
            template = template.replace(matches[j], temp);
          }

          _builder.append(template);
    }

    _builder.append(this._footer);

    // Return the markup
    var markup = _builder.toString();
    return markup;
}
As a client developer, you also specify a JavaScript function that the builder class will call back while processing the template for a given item. The callback function is expected to have the following prototype:
function applyFormatting(memberName, dataItem)
The first argument is the name of the placeholder without the initial # symbol. In most cases, this first argument matches the name of a public property on the bound object. The second argument is the entire data item object being bound to the current instance of the template.
The availability of the entire data object makes it possible for you to fully check runtime conditions to determine variations on the markup. Designed in this way, the callback function is logically equivalent to the server-side DataBound event of ASP.NET server controls. The following code snippet shows the JavaScript code required to change the color of the quote based on the rise or fall of the stock price:
function applyFormatting(memberName, dataItem)
{
    var temp = dataItem[memberName];
    if (memberName == "Change" && x.charAt(0) == "+")
    {
        return "<span style='color:green;'>" + temp + "</span>";
     }
     if (memberName == "Change" && x.charAt(0) == "-")
     {
         return "<span style='color:red;'>" + temp + "</span>";
     }
     return temp;
}
The logic in the code is applied only when the callback is invoked for the #Change placeholder. The callback just returns the raw member value if invoked for a different member. Figure 4 shows the final effect in the page.
Figure 4 Customized Data Binding in the Browser
You're probably wondering how some cells in Figure 4 have a different background color. Well, I presented the aforementioned callback as a data-binding function, but in reality, the callback is invoked for each match found in the item template with the #xxx expression. This means that by inserting a #xxx placeholder in a template tag you can have the callback invoked and inject HTML code virtually anywhere. Here's an example:
<xml id="item">
   <tr>
      <td align="left">#Symbol</td>
      <td #Style1 align="right">#Quote</td>
     <td align="right">#Change</td>
   </tr>
</xml>
The #Style1 expression is interpreted as a placeholder to process through the callback. The callback invocation is accompanied by the name of the pseudo-member (in this case, Style1) and the current data item. Based on the provided information, the callback can, say, change the background of the host tag to light yellow if the cell is used to render a stock whose price is on the rise:
function applyFormatting(memberName, dataItem)
{
    if (memberName == "Style1")
    {
        if (dataItem["Change"].charAt(0) == "+")
            return "style='background-color:lightyellow;'";
        else
            return "";
    }
 ...
}
In ASP.NET, most template-based controls don't apply data binding rules to header and footer regions. The Samples.MarkupBuilder component is no exception. However, there might be situations in which you want to display in the footer information that is derived from displayed data. A quick trick to do so consists of defining a scriptable element in the footer (or header) template:
<xml id="footer">
      <tr>
         <td colspan="3" align="right"
            style="background-   color:#eeeeee;">
            <small><i>provided by <b id="lblProvider"></b></i></small>
         </td>
      </tr>
   </table>
</xml>
The content of the TD tag includes a <b> tag with a unique ID. This is sufficient to make that element further scriptable. It should be noted that the content of the XML data island is merged with raw data collected from a remote WCF service and then, when ready, inserted into the page object model via the innerHTML property of HTML browser elements.
When this happens, the browser automatically parses the contents of the HTML and updates the Document Object Model (DOM). In this way, any literal element that contains a unique ID string becomes scriptable. The following sample function is the callback associated with the call to the remote WCF service that provides stock quotes:
function onDataAvailable(results)
{
    // Bind data and update the UI
    var temp = builder.bind(results, applyFormatting);
    $get("grid").innerHTML = temp;

    $get("lblProvider").innerHTML = results[0].ProviderName;
}
Once the innerHTML property of a page element has been updated to include an element with a given ID (say, lblProvider), you can start scripting that element.
The BST pattern forces you to generate any HTML you need in the browser using JavaScript. In general, this is a good thing as it allows you to isolate all presentation logic in the one tier, as you can see in Figure 1.
In addition, by using templates and JavaScript callbacks you can keep up with the inherently dynamic nature of HTML and manage to accommodate characteristics of the data and user's expectations. Templates help you a lot in wedding code flexibility and ease of maintenance. A general-purpose class like the MarkupBuilder presented here completes the offering and closes the circle. You can't reasonably rely on JavaScript alone to generate HTML and mix generation with sprinkles of presentation logic. Code will soon get too complex, hard to read, and inevitably full of errors. Helper classes, developed with the help of the Microsoft® Client AJAX library, come to the rescue.

InnerHTML and DOM
Before I invoke the HTML Message pattern—the opposite way to generate HTML markup in an AJAX application—I want to discuss innerHTML. The DOM is a standard API through which the browser programmatically exposes the contents of the currently displayed page. The DOM represents the page as a tree of elements. Each node in the logical tree corresponds to a live object with a known behavior and its own identity.
There are three fundamental operations you can accomplish on DOM nodes: find a node, create a node, and manipulate a node. Identifying a particular node is easy as long as you know the ID of the corresponding element. The API offers a handy function:
var node = document.getElementById(id);
In ASP.NET AJAX, the getElementById function is wrapped by the $get function. If there are multiple elements with the same ID, the function returns the first that appears in the collection. To update a DOM subtree, you should remove unwanted elements and add new ones. While this approach is neat and clean from a design perspective, it can have performance problems.
Nearly all browsers also support the innerHTML property on their DOM elements. It sets or retrieves the HTML between the start and end tags of the given element. The property was introduced with the DHTML object model of Internet Explorer® 4.0 but never made its way to the official DOM API. Compared to the DOM API, though, innerHTML is much faster, especially when it comes to creating complex structures of elements. A good place to read about performance of innerHTML versus DOM is at go.microsoft.com/fwlink/?LinkId=116828. The innerHTML is not free of issues. Read about possible problems to be aware of at go.microsoft.com/fwlink/?LinkId=116827.

HTML Message Pattern
The goal of HTM pattern is having the server generate blocks of HTML markup to be displayed in the browser. A possible implementation consists of making a call to a remote URL (either a service or an HTTP handler) and receiving an HTML snippet ready for display.
The implementation of HTM relies entirely on the code you have on the server—specifically, the AJAX service layer. This is yet another good reason to favor the creation of an AJAX-specific middle layer that isolates your core services from AJAX and presentation needs and concerns (see Figure 1).
To support the HTML Message pattern, an AJAX application needs services that accomplish the task for which they have been created and can translate their calculated results to HTML snippets. Figure 5 illustrates this.
Figure 5 Service that Supports HTML Message
The service composability principle is evident here. It's just a variation of the principle of reusability. In general, it applies to service-oriented architectures (SOAs) where you orchestrate business processes through composition languages like Web Services Business Process Execution Language (WS-BPEL) and get a parent service process resulting from the concatenation of a few others.
An AJAX service that outputs HTML may be seen as the composition of a core service that gets its data and a renderer service that massages it into HTML. Figure 6 shows the composed architecture of a service that returns stock quotes.
Figure 6 Stock Quote Service that Supports HTML Message
In this implementation, I'm just composing classes to obtain a parent and wrapper service. From a design perspective, several classes in Figure 6 are treated as services—stock quote provider, data finder, output renderer and, as you'll see later in this column, also an input adapter.
Implementing the HTM Pattern
Figure 7 shows the contract of the stock quote service I'm using in the sample implementation of the HTM pattern. The service is built around offline and online data providers. The offline data provider returns stale values for quotes and changes, whereas the online provider connects to a real finance service and returns live data.
Figure 7 Contract of Sample StockQuote Service
namespace Samples.Services.FinanceInfo
{
    [ServiceContract(Namespace="Samples.Services",
         Name="FinanceInfoService")]
    public interface IFinanceInfoService
    {
        [OperationContract]
        StockInfo[] GetQuotes(string symbols, bool isOffline);

        [OperationContract(Name="GetQuotesOffline")]
        StockInfo[] GetQuotes(string symbols);

        [OperationContract(Name="GetQuotesFromConfig")]
        StockInfo[] GetQuotes(bool isOffline);

        [OperationContract(Name = "GetQuotesFromConfigOffline")]
        StockInfo[] GetQuotes();

        [OperationContract(Name = "GetQuotesOfflineAsHtml")]
        string GetQuotesAsHtml(string symbols, bool isOffline);

        [OperationContract(Name = "GetQuotesFromConfigAsHtml")]
        string GetQuotesAsHtml(bool isOffline);

        [OperationContract(Name = "GetQuotesFromConfigAsHtmlEx")]
        string GetQuotesAsHtml(string contextKey);
    }
}
Both providers rely on an internal finder component to actually get data. The finder component is characterized by an interface and the actual finder class is read from the configuration file. The default finder component for the offline provider uses the Microsoft .NET Framework Random class to generate just random numbers. A finder component for the online provider may use any public Web services that return financial information.
Any data that is obtained through a finder class is then composed into an HTML snippet using a renderer class. The renderer component exposes an interface and can be replaced by simply changing a setting in the configuration file. The default HTML renderer builds a table with some hardcoded styles. In this month's source code, you'll find out that the actual HTML renderer is a class derived from the default renderer that also adds a last update label. The code snippet below shows the interfaces of the finder and rendered classes:
namespace Samples.Services.FinanceInfo
  {
      public interface IFinanceInfoFinder
      {
          string ProviderName { get; }
          StockInfo[] FindQuoteInfo            (string symbols);
      }
      public interface IFinanceInfoRenderer
      {
          string GenerateHtml            (StockInfo[] stocks);
      }
  }
The interfaces guarantee smooth interoperability as the data obtained by the finder flows directly into the methods of the renderer.
In this implementation, the GenerateHtml method of the renderer builds a table based on some predefined settings. In general, it can use any other style information that may be passed around from the client. However, the stock quote service is designed to pick up any renderer "service" that is configured on the server as the official HTML generator. All that this "service" has to do is implement the previous IFinanceInfoRenderer interface.
A sample using the markup generated via the stock service will look like this:
function getLiveQuotes()
{
    var isOffline = $get("chkOffline").checked;
    Samples.Services.FinanceInfoService.GetQuotesFromConfigAsHtml
      (isOffline,
      onDataAvailable);
}
function onDataAvailable(results)
{
    // Update the UI
    $get("grid").innerHTML = results;
}
The getLiveQuotes function is attached to a client-side event such as a button click or perhaps to a timer callback. Figure 8 shows a sample page in action. The HTML markup travels back to the client using a JSON package.
Figure 8 Executing an HTML Message Pattern

Performance and Design Considerations
The HTML Message pattern moves UI generation to the server and, in particular, to the service you call from the client. This model has pros and cons. On one hand, it allows you to use managed code to implement any complex logic required to generate the markup. On the server, you can read configuration files, connect to remote services, and access databases of HTML templates with programming power that is just impossible to obtain in the browser.
At the same time, you write the code that generates the markup without much help from visual tools such as designers. Any required changes to the markup have to be addressed with C# code and there's no clear separation between layout, data, and codebehind.
A possible workaround can be found in the definition of some internal ASP.NET pages that the service queries for markup in a server-to-server scenario. These pages would work as templates; they can be created with Visual Studio® 2008 and deployed in the same IIS application that hosts the AJAX service layer. These pages will be invoked programmatically and the markup they return will be forwarded to the client.
The HTML Message pattern tends to generate more traffic than plain calls going to a service that returns raw data. It should be noted, though, that the HTML Message pattern results in less traffic than partial rendering. The more you add styles and HTML enhancements, the larger the size of the packet you return grows.
In light of this, you can decouple HTML styling from HTML layout and embed in the markup only references to client-side CSS classes for styling. If you reduce the HTML markup to just layout and data, the percentage of the extra stuff that is transferred beyond raw data may be significantly lower. During my experiments, I noted that a page built with HTML Message may sometimes result in even lower traffic than a page built with BST if you only need to display a few fields and you limit yourself to referencing CSS client classes for styling.

DynamicPopulate Extender
To wrap up, I'd like to spend some time on one of the extenders available in the AJAX Control Toolkit—the DynamicPopulate extender that goes well with an HTML Message service.
When bound to a client trigger control (say, a button), the extender invokes a service method and attaches the results to the innerHTML property of a DOM element. Needless to say, the DynamicPopulate extender requires an HTML Message service in the AJAX service layer:
<act:DynamicPopulateExtender runat="server"
    ID="DynamicPopulateExtender1"
    BehaviorID="DynamicPopulateExtender1"
    ClearContentsDuringUpdate="false"
    TargetControlID="grid"
    UpdatingCssClass="updating"
    ServicePath="LiveQuotes.svc"
    ServiceMethod="GetQuotesFromConfigAsHtmlEx"
/>
The DynamicPopulate extender also imposes another requirement on the service that is highlighted by the input adapter service in Figure 6. The method referenced through the ServiceMethod attribute is required to have the following prototype:
string MethodName(string contextKey);
The contextKey parameter can contain any data serialized in any format that the service method knows how to process. In an input adapter service class, you take the input string and transform it into more specific parameters that other classes in the service know how to process.
One of the issues you may run into when using the extender is that it doesn't prevent the default event when the user clicks a button. So if the button is an ASP.NET button, the postback still occurs, which invalidates the service call. Here's a more common way of using the DynamicPopulate extender:
<asp:Button runat="server" id="btnRefresh" text="Live Quotes"
    onclientclick="invoke();return false;" />
The attached JavaScript simple functions does the following:
function invoke()
{
    var extender = $find("DynamicPopulateExtender1");
    var isOffline = $get("chkOffline").checked;
    extender.populate(isOffline.toString());
}
Based on this code, the UI is updated by merging the HTML response with the element specified through the TargetControlID property of the extender.

Solutions for Now
HTML is used for presentation whereas XML, JSON, and RSS are formats that move server-generated data to the client where it can be manipulated for presentation. In the context of AJAX, this neat model is hindered by the use of the JavaScript language. Once you have downloaded data to the client, you only have JavaScript to build the UI. The BST pattern in the form of custom data binding and template techniques helps you create the UI you need.
What if you have a server-centric mindset and just hate JavaScript? What if the UI is particularly complex and you prefer to use more reliable and powerful development and debugging tools? What if both on the server and the client you end up duplicating the same heavyweight algorithm on some large data structure? Wouldn't an HTML response eliminate some concerns on the client?
In general, I believe that until we get a powerful set of controls with a rich client-side object model, we may not always employ the otherwise desirable model in which HTML is used for presentation and JSON is used for data. Today, this model may not be applicable in all situations as far as an AJAX presentation is concerned. That's why the HTML Message pattern is worth a look. It may not serve all purposes, but it has its place.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is the author of Programming ASP.NET 3.5 Core References. Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Test Run
Introduction to WCF Testing
Dr. James McCaffrey

Code download available at: TestRun2008_07.exe (173 KB)
Browse the Code Online

Contents
The System under Test
The Test Harness
Additional Considerations
Whether you are new to Windows® Communication Foundation (WCF) or have worked with it a bit, there are some testing techniques and principles that will make your WCF work easier. There are several ways to think about what WCF is—I tend to think of WCF services as a major extension of Web services. Like Web services, WCF services allow you to create distributed systems using a service-oriented architecture. However, WCF services provide much greater flexibility (such as choice of transport protocol) and additional features (such as transactions and security). WCF is much more than merely an extension of Web services, but if you are new to WCF, initially thinking about WCF services in this way is a reasonable approach.
Figure 1 shows a simple but representative WCF scenario. Here Internet Explorer® is acting as a client program and is accessing an ASP.NET Web application that accepts some text from the user and computes a crypto-hash of it. Behind the scenes, the ASP.NET Web app is calling a WCF service to actually perform the hashing computation. In this particular scenario, the WCF service is being hosted by IIS and being consumed by an ASP.NET Web app, but as I will explain shortly, WCF services can be hosted in several ways in addition to IIS and can be consumed by virtually any type of application or other service.
Figure 1 Typical WCF Application Scenario
The most basic type of WCF service testing involves verifying the functional correctness of the service operations. One approach would be to manually test a WCF service through an application UI. But, although manual testing is necessary, using this approach to test the basic functionality of a WCF service would be time consuming, error prone, inefficient, and just plain boring.
A better approach would be to write test automation similar to the program shown running in Figure 2. The screenshot shows a console application test harness I wrote that feeds input text directly to the back-end WCF service, fetches the response message from the service, and determines a pass/fail test case result. The diagram in Figure 3 is a simplified view and summary of the relationships between the programs shown inFigures 1 and 2. In many cases, WCF services retrieve information from a back-end database, or retrieve information from Web services or WCF services, but I have not included those scenarios in Figure 3.
Figure 2 Testing a WCF Service
Figure 3 Simplified Relationships
Next, I'll explain the back-end WCF service so you'll understand exactly what is being tested, briefly discuss the ASP.NET Web application shown in Figure 1 which consumes the WCF service, and describe the test harness in detail. I'll conclude by touching upon some other WCF testing scenarios.

The System under Test
The system under test consists of a back-end WCF service and an ASP.NET Web application that uses the WCF service. WCF services are remarkably flexible. One of your key design decisions when creating a WCF service involves choosing a hosting mechanism for the service. There are four main options: using IIS, using a Windows® Service, self hosting, and using Windows Activation Service (WAS). You're probably familiar with the use of IIS and Windows. Self-hosting involves hosting WCF within a Microsoft® .NET Framework managed program such as a console application. WAS is a new process activation mechanism available in Windows Server® 2008 and in Windows Vista®. Each WCF hosting option has advantages and disadvantages depending upon your particular development scenario. For the sample WCF service in this column I decided to use IIS. This choice leverages IIS advantages such as built-in integrated management and monitoring, process recycling, idle shutdown, and message-based activation.
Creating a WCF service hosted by IIS is stunningly easy. I begin by firing up Visual Studio® 2008 on Windows Server 2003. Note that if you decide to develop a WCF service on a machine running Windows Server 2008 or Windows Vista, during development you will have to deal with issues related to their enhanced security features. However, I do not have space to describe them here.
Virtual Lab: Testing with LINQ
LINQ can dramatically simplify test automation when testing SQL-stored procedures. You can experiment with these LINQ testing techniques in our Virtual Lab. Everything is installed and ready to go, including the projects discussed in the column and a preconfigured SQL Server® database. Just start the lab and code along as you read.

Experiment in the virtual lab:
go.microsoft.com/fwlink/?LinkId=120461
Next, I select the File | New | Web Site option from the Visual Studio menu. Then I choose the WCF Service template (installed by default in Visual Studio 2008) from the New Web Site dialog and target the .NET Framework 3.5. For the Location field, I select HTTP and specify localhost/WCF/CryptoHashService. This approach creates a full Web Application and IIS virtual directory at C:\Inetpub\wwwroot\WCF\CryptoHashService on my development machine; instead I could have selected a location on my machine's file system and used the built-in Visual Studio Web Development Server.
I decided to use C# as my implementation language; however, WCF services can be implemented using Visual Basic® .NET, too. After clicking the OK button on the dialog box, Visual Studio creates a fully functional WCF service with two example operations named GetData and GetDataUsingDataContract. If you look at the Solution Explorer window, you'll see that Visual Studio generates four key files: IService.cs, Service.cs, Service.svc, and web.config. File IService.cs holds the interface definitions for WCF operations and File Service.cs holds the actual implementations of your operations. In this case, I decide to rename these two files as ICryptoHashService.cs and CryptoHashService.cs, respectively. Next, I load ICryptoHashService.cs into the Visual Studio code editor, delete the sample interface code, and replace it with this code:
[ServiceContract]
public interface ICryptoHashService
{
    [OperationContract]
    string GetCryptoHash(string s);
}
Here I have only a single operation, GetCryptoHash, but I could have added other operations. Notice that the [SeviceContract] and [OperationContract] attributes will be doing most of the actual code generation for me behind the scenes. Next, I edit the implementation file CryptoHashService.cs by adding a using statement reference to the System.Security.Cryptography namespace, and I write this code:
public class CryptoHashService : ICryptoHashService
{
    public string GetCryptoHash(string s)
    {
        byte[] ba = Encoding.Unicode.GetBytes(s);
        MD5CryptoServiceProvider sp = new MD5CryptoServiceProvider();
        byte[] hash = sp.ComputeHash(ba);
        string result = BitConverter.ToString(hash);
        return result;
    }
}
First, I change the name of my CryptoHashService class and the class from which it derives to match the name I use in my interface definition. In my GetCryptoHash method, I simply use the GetBytes method to convert the input argument to a byte array, instantiate an instance of the MD5CryptoServiceProvider class, and use the ComputeHash method to convert my byte array to a 16-byte MD5 crypto-hash. I convert the resulting hash from a byte array to a human-friendly string using the static BitConverter.ToString method, and then return that string. To keep my example short I have omitted the normal error checking you'd use in a production environment, such as checking my input argument, catching exceptions, and so on. Next, I edit the Service.svc file to reflect my naming changes from "Service" to "CryptoHashService":
<%@ ServiceHost Language="C#" Debug="true"   Service="CryptoHashService"
           CodeBehind=           "~/App_Code/CryptoHashService.cs" %>
I finished my name editing by updating two name references in web.config:
<system.serviceModel>
         <services>
         <service name="CryptoHashService"     behaviorConfiguration="Servic Behavior">
    <!-- Service Endpoints -->
    <endpoint address=""binding="wsHttpBinding"         contract="ICryptoHashService">
Notice the binding="wsHttpBinding" entry. A WCF binding is a collection of information that specifies how a WCF service communicates with clients including the transport protocol used by the service, the text encoding scheme used, and so on. You can use built-in bindings or create custom bindings. The wsHttpBinding is a pre-configured binding used by default when you create a WCF service hosted by IIS. Another part of the web.config file generated by Visual Studio is:
<endpoint address="mex" binding="mexHttpBinding"
    contract="IMetadataExchange"/>
This entry instructs my WCF service to expose metadata about itself so that client programs can probe the service to determine how to interact with it. At this point I can successfully build my WCF service by selecting Build | Build Solution from the main menu. I could have also hit the F5 shortcut key to both build my WCF service and get instructions for creating a client program. Because my service is being hosted by IIS, I don't need to start the service explicitly; the WCF service will be ready to accept incoming WCF messages as long as IIS is running.
Now let's walk through the creation of the ASP.NET Web application shown in Figure 1. I launch a new instance of Visual Studio and issue a File | New | Web Site command. On the New Web Site dialog, I select the ASP.NET Web Site template and select the .NET Framework 3.5 in the framework target dropdown control. I use Location HTTP and select C# as my language. I type localhost/WCF/UtilitiesAndTools in the location field to implicitly name my Web app. I could have specified a file system location on my development machine and used the development Web server instead of IIS. Next, I add some minimal UI code to my Web app.
The code in Figure 4 is the basic UI, without formatting details such font style and <hr/> tags. You can get the exact UI code, along with all the code presented in this article, from the code download that accompanies this column.
Figure 4 ASP.NET Web Application UI Code
<body>
   <form id="form1" runat="server">
      <div>
      <asp:Label runat="server" ID="Label"
         Text="Demo Utilities Featuring WCF Services" />
      <asp:Label runat="server" ID="Label2"
         Text="Enter text here:" />
      <asp:TextBox runat="server" ID="TextBox1"
         Height="100px" Width="320px" />
      <asp:Button runat="server" ID="Button1"
         Text="Get MD5 Crypto-Hash" onclick="Button1_Click"
         Width="150px" />
      <asp:Button runat="server" ID="Button2"
         Text="Get SHA1 Crypto-Hash"
         Width="150px" />
      <asp:Label runat="server" ID="Label3"
      Text="Crypto-Hash of your text (computed by WCF Service) is:" />
      <asp:TextBox runat="server" ID="TextBox2"
         Width="320px" />
      </div>
   </form>
</body>
At this point, my ASP.NET Web application has no knowledge of my WCF service; however, WCF and ASP.NET technologies were designed to work together seamlessly. In the Solution Explorer window of my Web application project, I right-click on the project name and select Add Service Reference from the context menu. Notice this is a new option that complements the older Add Reference (typically used for DLL libraries and .NET namespaces) and Add Web Reference (typically used for ASP.NET Web services) options.
In the resulting Add Service Reference dialog, I type localhost/WCF/CryptoHashService/Service.svc and click on the Go button. The Add Service Reference tool will then scan the specified location for available services and display the services found; in this case the CryptoHashService WCF service. In the Namespace field on the dialog, I accept the simple, but not very descriptive, default name, ServiceReference1, then click the OK button. Visual Studio generates all the proxy code my application needs to connect to my WCF service. In particular, I get a class named CryptoHashServiceClient (that is, "Client" appended to the name of the WCF service), which allows me to communicate with the CryptoHashService service.
In design view, I double-click on the Button1 control to instruct Visual Studio to register the control's event handler. Then I add this code to the Button1_Click method:
try
{
    string s = TextBox1.Text;
    ServiceReference1.CryptoHashServiceClient c =
       new ServiceReference1.CryptoHasahServiceClient();
    string h = c.GetCryptoHash(s);
    TextBox2.Text = h;
}
catch(Exception ex)
{
    TextBox2.Text = ex.Message;
}
It's almost too easy. I grab the text in TextBox1, instantiate an instance of the autogenerated CryptoHashServiceClient class, call the object's GetCryptoHash method, and display the resulting MD5 (Message Digest version 5) crypto-hash in TextBox2. The user interface of the Web application in Figure 1 shows a button control that computes a SHA-1 (Secure Hash Algorithm 1) crypto-hash, but I did not implement that functionality to simulate a system under development. After building the Web application, a user can launch Internet Explorer and navigate to the app, enter some text, and get the MD5 crypto-hash of the text, computed by the back-end WCF service, as shown in Figure 1.

The Test Harness
Now let's take a look at the code for the simple test harness shown in Figure 2. In essence, the test harness program is just a WCF client that sends an input message to the WCF service under test and verifies a correct return message. I begin by launching a new instance of Visual Studio and creating a C# console application program named TestHarness. In the example shown in Figure 2, notice I set the location of my harness as a subdirectory of C:\Inetpub\wwwroot just to keep my harness code located near my system under test. However, I could have placed my harness anywhere, including on a separate test host machine. Figure 5 illustrates the overall structure of the test harness.
Figure 5 Test Harness Structure
using System;
using System.Collections.Generic;
using System.Text;
using System.ServiceModel;

namespace TestHarness
{
    class Program
    {
        static void Main(string[] args)
        {
        try
            {
                 // display startup messages
                 // set up test case data collection, testCases
                 // set up WCF binding object, wsb
                 // set up WCF address object, epa

                 CryptoHashServiceClient c =
                     new CryptoHashServiceClient(wsb, epa);

                foreach (TestCaseData tcd in testCases)
                {
                     // echo case ID, input, expected values
                     // call GetCryptoHash method, fetch return
                     // compare actual return with expected
                     // print pass/fail result
                }
                Console.WriteLine("\nDone");
            }
            catch (Exception ex)
            {
                Console.WriteLine("Fatal error: " + ex.Message);
            }
        }
    }

    class TestCaseData
    {
        public readonly string caseID;
        public readonly string input;
        public readonly string expected;
        public TestCaseData(string caseID, string input, string expected)
        {
            this.caseID = caseID; this.input = input; this.expected =
              expected;
        }
    }
}
I begin coding my test harness by adding a using statement that points to the System.ServiceModel namespace. This namespace holds the core WCF service functionality. Next, I set up my test case data. In Figure 5 you can see that I create a simple TestCaseData class to hold test case ID, input, and expected value. In a production scenario you could add other fields too, including WCF service binding information, as I'll explain shortly. Now I embed my test case data directly into my harness by creating a generic List object and populating it with TestCaseData objects:
List<TestCaseData> testCases = new List<TestCaseData>();
testCases.Add(new TestCaseData("001", "Hello world",
    "E6-76-0D-55-5C-32-F6-6F-5E-15-93-31-DB-20-FD-8E"));
testCases.Add(new TestCaseData("002", "Goodbye world",
    "1A-05-3C-C0-4A-18-13-06-0E-AC-EA-BA-46-EC-CF-B1"));
testCases.Add(new TestCaseData("003", "",
    "D4-1D-8C-D9-8F-00-B2-04-E9-80-09-98-EC-F8-42-7E"));
I could have set up my test case data in an external store such as an XML file or SQL table. Here I am using three test cases, but in the real world I could have thousands. Identifying good test case inputs that thoroughly exercise the system under test, and determining expected results is the most difficult part of software testing.
Now, although you can write WCF client code from scratch, it's much easier to use the svcutil.exe command-line tool that ships with Visual Studio 2008 to generate proxy code and a WCF configuration file. In this case, I launch a Visual Studio command shell (which knows the location of svcutil.exe) and enter the command:
> svcutil.exe http://localhost/WCF/CryptoHashService/Service.svc
Svcutil.exe running with the location of a WCF service, and without any additional arguments, reads WCF metadata from the service and generates two files for me. The first file is CryptoHashService.cs (the name of the targeted WCF service followed by a .cs extension), which holds C# proxy code that will allow me to send messages to and receive messages from my WCF service. The second file is output.config, which holds the WCF binding information (transport protocol, time-out setting, text encoding, and so on) used by the targeted WCF service. After generating these two files, I right-click on my test harness project and add the CryptoHashService.cs file to my project. Alternatively, I could have copied and pasted the code directly into my test harness.
Now there are two ways to use the binding information in the output.config file. One way is to rename the file to app.config and then add the file to your client project. The second approach is to examine the data in the output.config file and then write code that programmatically assigns the values shown in output.config to a binding object. When writing a normal WCF client program, the app.config approach is usually a better choice than the programmatic approach. Because an app.config file is ordinary text and is read by the client program at run time, if the binding information of the associated WCF service changes, you can change the corresponding binding information of your client simply by changing your app.config file, without recompiling your client. However, in the special case of a WCF test harness client, it's sometimes better to specify binding information programmatically. The programmatic approach allows you to easily pass in binding information as part of your test case input. Here I use the programmatic approach. I instantiate a WSHttpBinding object:
WSHttpBinding wsb = new WSHttpBinding();
Now I can assign values to the Name property and various time properties of the binding object:
wsb.Name = "WSHttpBinding_ICryptoHashService";
wsb.CloseTimeout = TimeSpan.Parse("00:01:00");
wsb.OpenTimeout = TimeSpan.Parse("00:01:00");
wsb.ReceiveTimeout = TimeSpan.Parse("00:10:00");
wsb.SendTimeout = TimeSpan.Parse("00:01:00");
I determine these values by visually examining the attributes of the binding entry in the output.config file, which was generated by the svcutil.exe tool. In this case, I am using all the default values that were generated by Visual Studio when I created my WCF service. The remaining binding properties are given values similarly:
wsb.BypassProxyOnLocal = false;
wsb.TransactionFlow = false;
wsb.HostNameComparisonMode =
    System.ServiceModel.HostNameComparisonMode.StrongWildcard;
wsb.MaxBufferPoolSize = 524288;
wsb.MaxReceivedMessageSize = 65536;
wsb.MessageEncoding =
    System.ServiceModel.WSMessageEncoding.Text;
wsb.TextEncoding = System.Text.Encoding.UTF8;
wsb.UseDefaultWebProxy = true;
wsb.AllowCookies = false;
Now I can set up my proxy object:
string uri =
    "http://vte014.vte.local/WCF/CryptoHashService/Service.svc";
EndpointAddress epa = new EndpointAddress(uri);
CryptoHashServiceClient c =
    new CryptoHashServiceClient(wsb, epa);
The CryptoHashServiceClient class is defined inside the auto-generated CryptoHashService.cs file I created using svcutil.exe. That class constructor accepts a binding object (which I just set up programmatically) and an EndPointAddress object that points to the WCF service. Now that my client object is instantiated, I can iterate through each test case and exercise the WCF service under test.
In Figure 6 I simply send a pass/fail message to the console for each test case. In a production environment you would likely do more—track the total number of cases that pass and the number that fail, perhaps send an e-mail message programmatically if one or more test cases fail, save results to an external data store, and so on. If you are using Team Foundation Server, you can manage this test harness using the techniques in my Test Run column "Custom Test Automation with Team System" from the 2008 Launch issue (see msdn.microsoft.com/magazine/cc164248).
Figure 6 Display Status for Each Test Case
foreach (TestCaseData tcd in testCases)
{
    Console.WriteLine("Case ID  = " + tcd.caseID);
    Console.WriteLine("Input    = " + tcd.input);
    Console.WriteLine("Expected = " + tcd.expected);

    string actual = c.GetCryptoHash(tcd.input);
    Console.WriteLine("Actual   = " + actual);
    if (actual == tcd.expected)
        Console.WriteLine("* Pass *");
    else
        Console.WriteLine("** FAIL **");
}

Additional Considerations
The techniques I've presented this month provide a solid foundation for getting started with basic WCF service testing. However, there are many other aspects of WCF testing that I will cover in future columns. Most of these additional testing topics are made possible by the great flexibility of WCF. For example, WCF allows systems to use security at the transport level (using HTTPS, for instance) as well as at lower levels. Although WCF services can use HTTP, WCF allows systems to communicate using several other mechanisms including TCP and named pipes. WCF services can be hosted in IIS, as I've shown, but WCF services can also be hosted in other ways including via Windows services and self-hosted managed applications. WCF services can support multiple endpoints, each of which can have a different address, binding, and contract. WCF supports request-reply-style messaging and duplex-style messaging. All these WCF scenarios, and many others, have interesting implications for thorough testing.
The basic WCF functional testing scenario presented in this column represents just one part of thorough WCF testing. Because my dummy WCF crypto-hash service is so simple, the entire logic is contained in a single GetCryptoHash method. In more realistic scenarios you may have code that encapsulates business logic and separate code that encapsulates the service functionality. This approach allows you to test business logic and services separately, which can simplify your test effort.
When creating WCF services using Visual Studio Team System, you can take advantage of the built-in unit testing support if you use a test driven development philosophy. You can also use the WcfTestClient.exe test client utility which ships with Visual Studio 2008 to perform manual testing of your WCF services to complement the type of automated testing I've presented in this column (see the column written by my MSDN® Magazine colleague Juval Lowy at msdn.microsoft.com/magazine/cc163289). In addition to purely functional testing, you may also want to perform load testing using the integrated load-testing tools in Visual Studio.
Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes. He can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Security Briefs
Reinvigorate your Threat Modeling Process
Adam Shostack

Contents
Approaches to Threat Modeling
A Quick and Dirty Threat Model
Knowing When You're Finished
Driving the Threat Modeling Process
Keeping It in Sync: Dealing with Change
Help, I'm Lost!
My colleague Ellen likes to say that everyone threat models all the time. We all threat model airport security. We all threat model our homes. We think about threats against our assets: our families, our jewelry, and our sentimental and irreplaceable photographs (well, those of us old enough to have photos that never existed in digital form do). We model threats based on architecture: there's a wall here, a picture window there, and an easily climbed tree that we can use when we forget our keys. And we model threats based on attackers. We worry about burglars and kids falling into pools. We also worry about the weather, be it earthquakes, snow, or tornadoes.
If I wanted to sound like a management consultant, I'd say you employ a mature, multi-dimensional assessment process, with a heavy reliance on heuristics and low reproducibility across instances. At the same time, it's likely you won't have thought of everything or implemented defenses against every possible attack. It's very unlikely you have a home defense management plan or have ever run a penetration test against your home.
As we build software, regardless of whether we're in an agile or a waterfall world, we need agreement on what we're building, what we're not building, and what we're doing to ensure we're building the right thing. In the past few years, a perception that threat modeling is a heavy, bureaucratic process has been generated. There are some good reasons to move toward adding processes; I'd like to talk about them, some lessons learned from these processes, and how to put the fun back in threat modeling while making it an efficient, agile-friendly activity that anyone can do.

Approaches to Threat Modeling
There are many things called threat modeling. Rather than argue about which is "the one true way," consider your needs and what your skills, abilities, and schedules are, and then work with a method that's best for you. As part of that approach, some people ask, "What's your threat model?" and "Have you threat modeled that component?" One is requirements elicitation, the other design analysis. At Microsoft, we almost always mean the latter technique.
There are more threat modeling methods out there than I can dream of covering in one column. There's also a tremendous diversity of goals. Should your threat modeling process be fast or deep? Should it focus on assurance and completeness, or ease of use? Should you involve experts or developers in every meeting? Do you have organizational or industry rules you need to follow, such as the Microsoft® Security Development Lifecycle (SDL) or the rules for medical device manufacturers? The high level objective should be to understand security issues early so you can address them in the design rather than try to overcome design flaws later.
Some of the major ways to approach threat modeling activity include the following:
Assets Asset-driven threat modeling is much like thinking about what you want to protect in your house. You start by listing what assets your software has associated with it, and then you think about how an attacker might compromise those assets. Examples include a database that stores customer credit cards or a file that contains encrypted passwords.
Some people may interpret an asset as an element of the threat modeling diagram, thinking that a Web server itself is an asset. That way lies madness. Digital assets are things an attacker wants to read, tamper with, or deny you the use of.
Attackers Attacker-driven threat modeling involves thinking about who might want your assets, and it works from an understanding of their capabilities to an understanding of how they might attack you. This works great when your adversary is a foreign army with a known strategic doctrine, physical world limits, and long-lead-time weapons systems development. This works less well when your adversary is a loosely organized group of anonymous hackers.
More generally, it's not clear this is useful in software threat modeling. There are certainly people for whom "think like an attacker" is an effective part of design analysis. It's less clear that this is a reproducible process in which people can get training. If you're going to start from attackers, it's probably worth using a standard set. It will be helpful to have a small set of these anti-personas written out.
Software Design Design-driven threat modeling is threat modeling based on where your fences and windows are. You draw diagrams and worry about what can go wrong with each thing in your diagram. (This is the essence of the SDL threat modeling process today because everyone in software knows how to draw diagrams on a whiteboard.)
The software equivalents of fences and windows are the various forms of attack surface, such as file parsers or network listening services—sockets, remote procedure call (RPC) services, Web services description language (WSDL) interfaces, or AJAX APIs. They're the trust boundaries where you should expect an attacker to first get a foothold.

A Quick and Dirty Threat Model
Threat modeling doesn't have to be a chore. Following the process illustrated in Figure 1, here is the outline of a basic threat modeling process that will get you going quickly and painlessly:
  1. Diagram your application, and use this to tell your app's story in front of the whiteboard (see Figure 2). Use circles for code, boxes for things that exist outside of it (people, servers), and drums for storage. Our team uses funny looking parallel lines for data stores. Draw some trust boundaries using dotted lines to distinguish domains.
  2. Identify threats by brainstorming. When you get stuck, apply the STRIDE threat model, described in Figure 3, on each element of your app. Don't worry about the fixes, just get a brainstorming flow going.
  3. Consider redesigns by considering whether the threats are congregating or are totally scattered. Either way, you might want to design a component out or add one to address them centrally. All the threats in one place may mean you're worried about the front door and not worrying about anything else.
  4. Mitigate the first order threats. Go broad, not deep. It can be helpful to think of threats and mitigations as ordered. A first order threat is opening the door. The first order mitigation is a lock. The second order threat is that someone picks the lock or breaks it down. Second order mitigations include good locks and solid doors, properly installed. A third order defense might be an alarm system on the door, and to mitigate the threat of someone cutting the wire, you send a regular message down the wire. If you find yourself worrying about the software equivalent of what happens when someone cuts the phone wire to the alarm system before you worry about locks on the doors, you're worrying about the wrong things.
  5. File bugs so you can fix what you found threat modeling.
Figure 1 The Threat Modeling Process
Figure 2 Diagramming Your App Helps Threat Modeling
Figure 3 STRIDE Threat Model
Threat | Property | Definition | Example
Spoofing | Authentication | Impersonating something or someone else. | Pretending to be any of these: billg, microsoft.com, or ntdll.dll.
Tampering | Integrity | Modifying data or code. | Modifying a DLL on disk or DVD, or a packet as it traverses the LAN.
Repudiation | Non-repudiation | Claiming to have not performed an action. | "I didn't send that e-mail," "I didn't modify that file," "I certainly didn't visit that Web site, dear!"
Information Disclosure | Confidentiality | Exposing information to someone not authorized to see it. | Allowing someone to read the Windows source code; publishing a list of customers to a Web site.
Denial of Service | Availability | Deny or degrade service to users. | Crashing Windows or a Web site, sending a packet and absorbing seconds of CPU time, or routing packets into a black hole.
Elevation of Privilege | Authorization | Gain capabilities without proper authorization. | Allowing a remote Internet user to run commands is the classic example, but going from a limited user to admin is also EoP.

Knowing When You're Finished
When your diagrams show all of the relevant parts of the system, and you've walked through the STRIDE model for every element in the diagrams, you're doing pretty well. (You're doing even better if you've filed bugs for each vulnerability.)
However, that's somewhat like saying you're finished when you've checked that all the doors and all the windows have locks. That may be enough. It may be what you have time to do. Or maybe you're planning to go deeper, to think about bypasses for the defenses you've put in place.
How deep you go depends a lot on your organization's risk tolerance, or that of your customers. It also depends on your budget and experience. It may be that you want to enter into a threat modeling process, prove success, and exit reasonably quickly. It may be that you're designing the control system for something safety critical and need to dive deep and spend a good deal of time on it.
So knowing when you're done involves a set of trade-offs. First you need to consider the policies, laws, or regulations mandating your level of effort. Then you need to assess the risk for the system being developed. Finally, you need to account for the availability of time and resources both for your threat modeling process and any resulting mitigation and testing.

Driving the Threat Modeling Process
Threat modeling doesn't just happen by itself. Someone needs to drive it. Who's driving it will have a major impact on how it works. If you have one or even a few people who love security, they can drive threat modeling across an organization.
There's a question of how or whether to involve other people. This is both a cultural and a resources question. If the enthusiasts can threat model everything, share the results, and get the issues detected fixed, then great! Why involve others?
Unfortunately, sometimes there are more projects than security enthusiasts to take them on, and you need to bring others in. Microsoft has found that threat modeling works better with a security expert in the room, but there isn't always one available.
Your expert is someone who thinks about security all the time and can do so in a structured way. It's someone who can talk about what a repudiation threat might mean in the context of your design. Your expert might have a certification of some sort. He might be that someone who points out security flaws in your code and operations. She might attend (or want to attend) conferences such as SANS or BlackHat, or she may have been thanked in a Microsoft Security Bulletin.﻿
You can get decent results by giving people structure and feedback on their work, and by breaking it down into small, easy pieces with rules and self-checks in each one. However, breaking threat modeling down into too many little pieces and having too many rules (or the wrong rules) can take all the fun out of the process, and that turns threat modeling into a chore. So how you break the pieces out depends on how many experts you have and how much time they can dedicate to the process.

Keeping It in Sync: Dealing with Change
Plans change. It's unavoidable. When they do, there's a cost to keeping the documentation synchronized with the many real plans that are in everyone's heads. No matter how agile you are, a project of any substantial size is definitely going to have some architecture documentation somewhere. Maybe it's a set of Visio® diagrams the size of a large wall, or maybe it turns out to be a whiteboard that can't be erased.
For threat modeling purposes, you want to know when things you do add trust boundaries to that architecture. When you do, you have to consider the implications of those new threats.
Sometimes, the threats are direct. Say you change a component to listen on port 80. When threats are direct to the component, you can easily discover them, address them, and move on.
Other times, the threats are indirect. Say you decide to get configuration data from Active Directory® and Lightweight Directory Access Protocol (LDAP). This probably entails a few new components (the directory interface and an abstraction layer so you can switch), but it also entails a change to the configuration data. Before, the data might have been entered by an administrative user into your GUI. Now it's coming in over the network, using several different protocols. It's probable that you'll want to validate the data very carefully in the abstraction layer, and perhaps also become more cautious about how you read and act on configuration data.

Help, I'm Lost!
You may find yourself sitting around a table saying, "What do we do now?" Maybe you're having trouble understanding the process. Maybe one step has you completely stumped. Here are some problems we've seen on the SDL team and how to get unstuck:
Some teams have problems with diagramming. If you don't know what the system does, find someone who does, or break the threat modeling process and spend time understanding this thing you've inherited.
If you don't know what the system should do—perhaps you're working on a new system, and you have two competing designs—threat modeling can help unstick your decision-making progress. Draw both designs and ask, "Which has the worst vulnerabilities?"
If you don't know how to represent an element, don't sweat it too much. You may use the wrong element type, and you'll notice. But in essence, all the code that you are responsible for, no matter who wrote it, is a process. All data that you read and write to is stored in data stores.
You may also come across problems enumerating threats. First, it's important you don't let yourself get stuck on a single threat. Flag it and come back to it later. If that doesn't work, check out Larry Osterman's great series on threat modeling (go.microsoft.com/fwlink/?LinkId=118281). There's also good stuff in Michael Howard's books Writing Secure Code and The Security Development Lifecycle.
For problems validating the threat model and your mitigation plan, look to see whether the diagrams represent the code and whether you have agreement between developers and testers on that. Look to see if you've enumerated a threat for each threat type against each element (see the November 2006 MSDN® Magazine article "Uncover Security Design Flaws Using the STRIDE Approach," available at msdn.microsoft.com/magazine/cc163519). And finally, check to see if you've filed and fixed a bug per threat.
If you happen to have an expert handy, ask him for help. Even our experts ask each other, "How would you approach this issue?" If you don't have an expert handy, try getting one, either by hiring him or blogging about your problem.
Don't let yourself get stymied by a single issue. Threat modeling is too important for you to skip. And remember to have fun. Breaking stuff before you ship is not only fun, it's a heck of a lot more fun than fixing it afterward.

Send your questions and comments to briefs@microsoft.com.

Adam Shostack is a Program Manager on the Security Development Lifecycle (SDL) team at Microsoft. He is responsible for the threat modeling component of the SDL.


Foundations
Adding Code Access Security to WCF, Part 2
Juval Lowy

Code download available at: Foundations2008_07.exe (497 KB)
Browse the Code Online

Contents
Host-Side CAS in the .NET Framework 3.5
Partially Trusted Services
App Domain Host
Implementing AppDomainHost
Partially Trusted Hosts
Structured Host-Side Security Demands
Implementing Structured Host Demands
Additional Aspects of AppDomainHost
In the April 2008 installment of this column, I discussed the lack of adequate support for code access security (CAS) and the motivation for adding proper CAS support to the Windows® Communication Foundation (WCF). I then proceeded to show how to add that support on the client side, enabling partially trusted clients to call WCF services, all without compromising on security or on the WCF programming model (for the April column, see msdn.microsoft.com/magazine/cc500644).
In this issue, I will show you what it takes to complete the picture by allowing partially trusted services and partially turreted hosts. As with the client side, I will share with you not just the mechanics of the solution but also the approach and the thought process leading to it. You will also get to see some advanced programming techniques in the WCF and Microsoft® .NET Framework.

Host-Side CAS in the .NET Framework 3.5
In the .NET Framework 3.5, WCF only allows partially trusted code to host a service over the BasicHttpBinding, WSHttpBinding, and WebHttpBinding bindings, and only with no security at all or with transport security. Furthermore, in the case of WSHttpBinding, aspects such as message security, reliable messaging, and transactions are disallowed. All partial-trust enabled bindings must use text encoding. A service running under partial trust cannot use additional facilities such as diagnostics and performance counters.
Enforcing the limited set of supported features is the responsibility of the bindings. Each non-HTTP binding actively demands full trust of its service host. The allowed HTTP bindings themselves do not demand full trust; they instead demand permissions according to the context of use, as listed in Figure 1.
Figure 1 Allowed HTTP Bindings Demands on the Host
Scenario | Permissions
Basic, Web, and WS- with no security or transport security | Security permission with execution and Infra-structure; Web permission to accept calls to URI.
Above bindings with internal service type in separate assembly | Unrestricted reflection.
Above bindings with authenticated calls | Security permission to control principal.
The allowed HTTP bindings all demand permission to execute and permission to modify the infrastructure (security permission with infrastructure flag), as well as permission to accept calls on the URI configured for their endpoint (Web permission with accept flag to the URI). When the calls are authenticated, the allowed HTTP bindings also demand permission to control the thread principal (security permission with principal control flag).
This is done because, after authentication, WCF will install a new principal with identity matching the provided client's credentials for the duration of the call and regardless of the binding. If the service type provided to the host constructor is defined as an internal type of another assembly, the host also demands unrestricted reflection permission to be able to load that type using reflection. (The decision to demand infrastructure permission is puzzling since there is no obvious need for it.)
There are additional limitations on configuration that you need to consider. For example, the .config file cannot contain any reference to any certificate store (for service-side certificates), since touching the certificate store will cause WCF to demand full trust. Administrators have to configure these certificates separately using tools such as HttpConfig.exe.
A deficiency of basing the host-side permissions demand solely on the binding is that the hosted service instance is implicitly given these permissions, even if it does not require them to function. For example, the service instance could control the principal or reflect internal types of other assemblies on the host machine. It would have been better to design the bindings to demand their permissions and yet somehow provide the host with a way of granting different permissions to the service, while ensuring that the service permissions are a subset of the hosting code permissions.
Ideally, you would like to tap into the full power of WCF, from distributed transactions to reliable calls, to various security credentials types, to intranet (or even same-machine) applications with TCP and the interprocess communication—IPC (named pipes)—binding, not to mention duplex callbacks, asynchronous calls, diagnostics and tracing, instrumentation, and, of course, queued calls over Microsoft Message Queue (MSMQ). And you would like do all that without compromising on CAS (that is, resorting to full trust).

Partially Trusted Services
In the .NET Framework 3.0, the only way for a service to execute in partial trust was to explicitly permit only the permissions required for it to operate and therefore to implicitly deny all other permissions. One way of achieving that was to apply the matching permission attributes with the SecurityAction.PermitOnly flag. Consider the service in the following:
[SecurityPermission(
  SecurityAction.PermitOnly,
  Execution = true)]
[UIPermission(SecurityAction.PermitOnly,
  Window =
    UIPermissionWindow.SafeTopLevelWindows)]
class MyService : IMyContract {
  public void MyMethod() {
    Form form = new TestForm();
    form.ShowDialog();
  }
}
The service requires permission in order for it to execute (as does all managed code), and it also must have permission in order for it to display safe windows. If you were to stack multiple permit-only permission attributes on a class, it would yield a single permission set at run time.
The .NET Framework uses it in order to allow only those permissions and to actively deny all other permissions by installing a dedicated stack walk modifier. As a result of this, even if the assembly the service resides in (as well as the app domain) is granting the service full trust, all other permission will be denied.
When the service tries to perform an operation such as opening a file, that will trigger a security exception, since the demand for file I/O will encounter the stack walk modifier that will actively deny having that permission. All the service can do is execute in its virtual sandbox and display safe windows. The .NET Framework will change the window caption and display a warning tag on the displayed form to inform the user that the application is partially trusted, as seen in Figure 2.
Figure 2 Window Displayed from Partially Trusted Code
Instead of specifying the permissions as attributes, you can list them in a permission set XML file and provide that file name to the PermissionSetAttribute, as shown in Figure 3.
Figure 3 Using Permission Set File for Partially Trusted Service
<!-- MyServicePermissions.xml -->
<PermissionSet class = "System.Security.PermissionSet">
  <IPermission
    class = "System.Security.Permissions.SecurityPermission"
    Flags = "Execution"
  />
  <IPermission
    class = "System.Security.Permissions.UIPermission"
    Window = "SafeTopLevelWindows"
  />
</PermissionSet>

[PermissionSet(SecurityAction.PermitOnly,File =
  "MyServicePermissions.xml")]
class MyService : IMyContract {
  public void MyMethod() {
    Form form = new TestForm();
    form.ShowDialog();
  }
}
Note that the permission set file is only used at compile time, and it has no use for deployment. The compiler will fuse the permitted permissions into the class metadata as individual attributes. If the file is not present, the build will fail.

App Domain Host
There are several problems with the attribute-based approach to partially trusted services. First, every time there is a change in what the service does—or for that matter any of the downstream classes it uses—you will need to modify the stack of attributes or change the file. This will couple the service to those classes and increase the maintenance cost.
Second, because the permissions are part of the service definition, there is no way to use the service with different permissions as a product of the hosting environment. You cannot give it more or fewer permissions.
Third, and more important, it is unlikely most services will actually go through the trouble of analyzing the security permissions required for them to operate and meticulously reduce their granted permissions in order to run with the least-possible privileges and permissions. If anything, it is the host that should be concerned about what exactly the service it loads is up to, and yet the host by default has no ability to affect that. The ServiceHost of WCF will host all services in full trust.
It would be better for the host, as well as for the service's own protection, to allow the host to grant whichever permissions it deems as appropriate for the service. To that end, I wrote the class AppDomainHost defined in Figure 4.
Figure 4 AppDomainHost
public class AppDomainHost : IDisposable {
  //Create new app domain in full trust
  public AppDomainHost(Type serviceType,params Uri[] baseAddresses);
  public AppDomainHost(Type serviceType,string appDomainName,
    params Uri[] baseAddresses);

  //Create new app domain with specified permission set
  public AppDomainHost(Type serviceType,PermissionSet permissions,
    params Uri[] baseAddresses);
  public AppDomainHost(Type serviceType,PermissionSet permissions,
    string appDomainName,params Uri[] baseAddresses);

  //Additional constructors that take standard permission set,
  //permission set filename, and an existing app domain

  public void Open();
  public void Close();
  public void Abort();

  //More members
}
The class AppDomainHost can be used just like the WCF-provided ServiceHost:
AppDomainHost host =
  new AppDomainHost(typeof(MyService));
host.Open();
The difference is that AppDomainHost will host the provided service type in a new app domain, not in the app domain of its caller. The new app domain name will default to "AppDomain Host for" suffixed with the service type and a new GUID. You can also specify a name for the new app domain:
AppDomainHost host =
  new AppDomainHost(typeof(MyService),"My App Domain");
host.Open();
The new app domain is created with full trust by default. However, placing the service in a separate app domain is the key for managing partially trusted services.
AppDomainHost allows you to provide the permissions for the new app domain. For example, Figure 5 shows a service and a host granting the service just enough permissions required in order for it to operate.
Figure 5 Partially Trusted Service
class MyService : IMyContract {
  public void MyMethod() {
    Form form = new TestForm();
    form.ShowDialog();
  }
}

//Hosting code:
PermissionSet permissions =
  new PermissionSet(PermissionState.None);
permissions.AddPermission(new SecurityPermission(
  SecurityPermissionFlag.Execution));
permissions.AddPermission(new UIPermission(
  UIPermissionWindow.SafeTopLevelWindows));

AppDomainHost host =
  new AppDomainHost(typeof(MyService),permissions);

host.Open();
You can also use a permission set file, and, unlike that which was shown in Figure 3, the file is only needed at run time and can be modified post deployment. It is important to note that you also have the ability to specify one of the standard named permission sets. The virtual sandbox imposed on partially trusted services affects any downstream class it may use along with its ability to call outside the sandbox over any technology, from the .NET Framework 1.0 to WCF.
A single host process can have many instances of AppDomainHost, each hosting the same or a different service type, each with any arbitrary set of permissions, as shown in Figure 6 (using the same service definition as in Figure 5).
Figure 6 Same Service Type with Different Permissions
//Default is service with full trust
AppDomainHost host0 =
  new AppDomainHost(typeof(MyService),
  "Full Trust App Domain",
  new Uri("net.tcp://localhost:6000"));

host0.Open();

//With just enough permissions to do work
PermissionSet permissions1 =
  new PermissionSet(PermissionState.None);
permissions1.AddPermission(new SecurityPermission(
  SecurityPermissionFlag.Execution));
permissions1.AddPermission(new UIPermission(
  UIPermissionWindow.SafeTopLevelWindows));

AppDomainHost host1 =
  new AppDomainHost(typeof(MyService),permissions1,
  "Partial Trust App Domain",
  new Uri("net.tcp://localhost:6001"));

host1.Open();

//With not enough permissions to do work
PermissionSet permissions2 =
  new PermissionSet(PermissionState.None);
permissions2.AddPermission(new SecurityPermission(
  SecurityPermissionFlag.Execution));

AppDomainHost host2 =
  new AppDomainHost(typeof(MyService),permissions2,
  "Not enough permissions",new Uri("net.tcp://localhost:6002"));

host2.Open();

//Using one of the named permission sets
AppDomainHost host3 =
  new AppDomainHost(typeof(MyService),
  StandardPermissionSet.Internet,
  "Named permission set",
  new Uri("net.tcp://localhost:6003"));

host3.Open();

Implementing AppDomainHost
Implementing AppDomainHost is done in two parts: creating a new app domain and injecting a service host instance, and then having the service host (and the service instance) execute in partial trust. To create a service host instance and to activate it in a separate app domain, I wrote the class ServiceHostActivator shown in Figure 7.
Figure 7 ServiceHostActivator
class ServiceHostActivator : MarshalByRefObject {
  ServiceHost m_Host;

  public void CreateHost(Type serviceType,Uri[] baseAddresses) {
    m_Host = new ServiceHost(serviceType,baseAddresses);
  }
  public void Open() {
    m_Host.Open();
  }
  public void Close() {
    m_Host.Close();
  }
  public void Abort() {
    m_Host.Abort();
  }

  //Rest of the implementation
}
ServiceHostActivator is a simple wrapper around a standard WCF-provided ServiceHost instance. ServiceHostActivator derives from MarshalByRefObject so that AppDomainHost could call it across the app domain boundary. The CreateHost method encapsulates constructing a new ServiceHost instance. The rest of the methods of ServiceHostActivator simply forward the remote calls to the underlying host instance.
AppDomainHost offers several overloaded constructors. These constructors all call each other (see Figure 8), even creating a new app domain along the way and eventually the construction ends up using a protected constructor that takes the service type, the new app domain instance, the permission set for the new domain, and the base addresses.
Figure 8 Implementing AppDomainHost
public class AppDomainHost : IDisposable {
  ServiceHostActivator m_ServiceHostActivator;

  public AppDomainHost(Type serviceType,
    params Uri[] baseAddresses) :
    this(serviceType,"AppDomain Host for "+
    serviceType+" "+Guid.NewGuid(),
    baseAddresses) {
  }

  public AppDomainHost(Type serviceType,
    string appDomainName,
    params Uri[] baseAddresses) : this(serviceType,
    new PermissionSet(PermissionState.Unrestricted),
    appDomainName,baseAddresses) {
  }

  public AppDomainHost(Type serviceType,
  PermissionSet permissions,
  string appDomainName,
  params Uri[] baseAddresses) :
  this(serviceType,AppDomain.CreateDomain(appDomainName),
  permissions,baseAddresses) {
  }

  //More constructors

  protected AppDomainHost(Type serviceType,
    AppDomain appDomain,
    PermissionSet permissions,Uri[] baseAddresses) {

    string assemblyName = Assembly.GetAssembly(
      typeof(ServiceHostActivator)).FullName;
    m_ServiceHostActivator = appDomain.CreateInstanceAndUnwrap(
      assemblyName,typeof(ServiceHostActivator).ToString()) as
      ServiceHostActivator;

    CodeAccessSecurityHelper.SetPermissionsSet(appDomain,permissions);

    m_ServiceHostActivator.CreateHost(serviceType,baseAddresses);
  }

  public void Open() {
    m_ServiceHostActivator.Open();
  }
  public void Close() {
    m_ServiceHostActivator.Close();
  }
  public void Abort() {
    m_ServiceHostActivator.Abort();
  }
  void IDisposable.Dispose() {
    Close();
  }
}
The protected constructor of AppDomainHost uses .NET remoting to inject into the new app domain an instance of ServiceHostActivator, ending with a remoting proxy to it, stored in the m_ServiceHostActivator member.
The new app domain is created with full trust by default. AppDomainHost uses the SetPermissionsSet method of my CodeAccessSecurityHelper class in order to install a new CAS policy in the new app domain—permitting only the supplied permissions and denying the rest:
public static class CodeAccessSecurityHelper {
  public static void SetPermissionsSet(
    AppDomain appDomain,
    PermissionSet permissions) {

    PolicyLevel policy = PolicyLevel.CreateAppDomainLevel();
    policy.RootCodeGroup.PolicyStatement =
      new PolicyStatement(permissions);
    appDomain.SetAppDomainPolicy(policy);
  }
  //More members
}
It is as simple as creating a new security policy at the app-domain level and calling the SetAppDomainPolicy method of the AppDomain class:
public sealed class AppDomain :
  MarshalByRefObject,... {

  public void SetAppDomainPolicy(
    PolicyLevel domainPolicy);
  //More members
}
Incidentally, a similar technique is used by ClickOnce to enforce the partial-trust execution of the ClickOnce-deployed applications.
Calling the other methods of AppDomainHost such as Open or Close uses the proxy to ServiceHostActivator to call to the other app domain and have it open or close its host instance. Since the service instance will execute in the app domain that happened to open its host, the service will also execute under that app domain's security policy.

Partially Trusted Hosts
The approach presented thus far for partially trusted services was predicated on having the code that uses AppDomainHost running under full trust. This is because ServiceHost and the bindings will demand full trust in any hosting scenario that is not listed in Figure 1.
But what if the code launching the host is only partially trusted? You could place AppDomainHost and ServiceHostActivator in a fully trusted assembly, allow partially trusted callers and have them both assert full trust:
[PermissionSet(SecurityAction.Assert,Unrestricted = true)]
public class AppDomainHost : IDisposable
{...}

[PermissionSet(SecurityAction.Assert,Unrestricted = true)]
class ServiceHostActivator : MarshalByRefObject
{...}
While this approach works, it circumvents CAS and disables a crucial security mechanism, and by doing so you introduce two security problems. First, it is incorrect to assume that the code opening the host has permissions to accept calls on the transport channels or participate in any of the WCF activities such as distributed transactions. Second, by asserting full trust and suppressing the stack walk, the partially trusted code that launched the host can actually be used to create a service with higher privileges to do its dirty deeds for it. For example, the hosting code may not have file I/O permissions, but it will use AppDomainHost to accept calls to a service with file I/O permissions.
The solution is simple: AppDomainHost and ServiceHostActivator should not employ a blank assertion of full trust. Instead, they should only assert locally, when needed, and then revert to the existing permissions. In addition, AppDomainHost should challenge the code using it and demand appropriate hosting permissions (such as accepting calls). It should also verify that the hosting code is granted at least the permissions it wishes to run the service under. This will prevent a restricted and partially trusted code from using a service hosted in a separate app domain with more permission. The net result is structured host-side security demands.

Structured Host-Side Security Demands
The reworked ServiceHostActivator, suitable for partial trust callers, is shown in Figure 9. The CreateHost method of ServiceHostActivator cannot declaratively assert full trust, since this will prevent it from demanding the appropriate host permission of the calling code. Instead, it programmatically uses CodeAccessSecurityHelper to create a full-trust permission set and assert it, then proceeds to create the host. When WCF demands full trust, the assertion prevents that demand from going up the call stack.
Figure 9 Reworked ServiceHostActivator
class ServiceHostActivator : MarshalByRefObject {
  public void CreateHost(Type serviceType,Uri[] baseAddresses) {
    CodeAccessSecurityHelper.PermissionSetFromStandardSet(
    StandardPermissionSet.FullTrust).Assert();

    m_Host = new ServiceHost(serviceType,baseAddresses);

    PermissionSet.RevertAssert();

    m_Host.DemandHostPermissions();
  }

  //Behavior demands happen here, must assert
  [PermissionSet(SecurityAction.Assert,Unrestricted = true)]
  public void Open() {
    m_Host.Open();
  }
  ...
}
After creating the host, CreateHost explicitly reverts the assertion. CreateHost then calls the extension helper method DemandHostPermissions of CodeAccessSecurityHelper on the just-created host instance in order to demand the hosting permissions. DemandHostPermissions examines the host object and demands the appropriate permissions.
For example, if the host supports a TCP endpoint, then DemandHostPermissions will demand permission to execute and permission to accept TCP calls on the endpoint URI. If reliable messaging is used over TCP, DemandHostPermissions will demand permission to control the policy. There are quite a few additional possible demands, as shown in Figure 10.
Figure 10 Structured Host-Side Demands with AppDomainHost
Scenario | Permissions
TCP | Security permission with execution, Socket permission to accept TCP call on URI.
IPC | Security permission with execution, control policy, and control evidence.
MSMQ | Security permission with execution, MSMQ permission to read from queue.
WS, WS-Dual, Basic, Web | Security permission with execution and Web permission to accept calls on URI.
RM over TCP | Security permission to control policy.
Authenticated calls | Security permission to control principal.
Transaction propagation | Unrestricted distributed transaction permission.
User name credentials with message security, certificate credentials with validation, and service certificate | Store permission to enumerate stores, open store, and enumerate certificates.
Diagnostic tracing | Environment permission to read COMPUTERNAME, File I/O permission to path discovery, append, and write to log files.
WCF service performance counters | Performance counter permission to write to service counter, write to endpoint counter, and write to operation counter.
WCF all performance counters | Performance counter permission to write to service counter, write to endpoint counter, write to operation counter, and write to host counter.
ASP.NET providers | Minimal ASP.NET hosting permission.
Use of the IPC binding demands permission to control policy and evidence. Use of the MSMQ binding triggers a demand for permission to read from the endpoint queue. Use of any of the HTTP bindings triggers demand to accept calls on the endpoint address. If the binding uses authenticated calls, DemandHostPermissions will demand permissions to control the principal.
Use of a transaction-aware binding, with transaction flow enabled and having at least one operation on the endpoint's contract with a transaction flow allowed (or mandatory), will cause a demand for unrestricted distributed transaction permission.
Any access to the certificate store, such as with message security, client certificate validation, or use of a service certificate to protect the message, will cause demand for store enumeration to open a store and enumerate the certificates.
If the serviced performs diagnostics, DemandHostPermissions will demand access to the computer name environment variable and file I/O permission to that file. If the host uses WCF performance counters, DemandHostPermissions will demand permissions to write to the counters at the reported level.
If the host relies on the ASP.NET providers for caller authentication or role membership, DemandHostPermissions will demand minimal ASP.NET hosting permission (all providers require that permission). When calling back over the WS-Dual binding, the binding itself will demand Web permission to connect to that callback endpoint, so there is no need for AppDomainHost to explicitly demand it when launching the host.

Implementing Structured Host Demands
Figure 11 shows the partial implementation of DemandHostPermissions, which first performs endpoint-specific demands. It iterates over the collection of the service's endpoints, and for each endpoint it demands its respective connectivity, security, and transactions permissions.
Figure 11 Implementing DemandHostPermissions
public static class CodeAccessSecurityHelper {
  internal static void DemandHostPermissions(this ServiceHost host) {
    foreach(ServiceEndpoint endpoint in host.Description.Endpoints) {
      DemandHostConnectionPermissions(endpoint);
      DemandHostSecurityPermissions(endpoint);
      using(TransactionScope scope = new TransactionScope()) {
        DemandTransactionPermissions(endpoint);
      }
    }
    DemandHostStorePermissions(host);
    DemandPerformanceCounterPermissions();
    DemandTracingPermissions();
    DemanAspNetProvidersPermissions(host);
  }
  internal static void DemandHostConnectionPermissions(
    ServiceEndpoint endpoint) {

    PermissionSet connectionSet =
      new PermissionSet(PermissionState.None);
    if(endpoint.Binding is NetTcpBinding) {
      connectionSet.AddPermission(new SocketPermission(
        NetworkAccess.Accept, TransportType.Tcp,
        endpoint.Address.Uri.Host,endpoint.Address.Uri.Port));
    }

    /* Checking the other bindings */

    connectionSet.Demand();
  }

  static void DemanAspNetProvidersPermissions(ServiceHost host) {
    bool demand = false;

    foreach(IServiceBehavior behavior in host.Description.Behaviors) {
      if(behavior is ServiceCredentials) {
        ServiceCredentials credentialsBehavior =
          behavior as ServiceCredentials;

        if(credentialsBehavior.UserNameAuthentication.
          UserNamePasswordValidationMode ==
          UserNamePasswordValidationMode.MembershipProvider)  {

          demand = true;
          break;
        }
      }
      if(behavior is ServiceAuthorizationBehavior) {
        ServiceAuthorizationBehavior serviceAuthorization =
          behavior as ServiceAuthorizationBehavior;
        if(serviceAuthorization.PrincipalPermissionMode ==
          PrincipalPermissionMode.UseAspNetRoles &&
          Roles.Enabled) {

          demand = true;
          break;
        }
      }
    }
    if(demand) {
      IPermission permission =
        new AspNetHostingPermission(
        AspNetHostingPermissionLevel.Minimal);
      permission.Demand();
    }
  }
  //Rest of the implementation
}
For the transactions demand, it creates a temporary ambient transaction using a transaction scope. This enables it to use the same method as on the client side, which demands permissions at call time based on the presence of an ambient transaction.
After the endpoints demands, DemandHostPermissions demands permissions that are a product of the service host configuration, specifically certificates store access permission, performance counters permissions, tracing permissions, and ASP.NET providers permissions. For example, to demand the ASP.NET providers permissions, it obtains the collection of service behaviors and sees whether the host's service credentials behavior uses the membership provider or the host's service authorization behavior uses the roles providers, and if so it demands the ASP.NET hosting permission.
AppDomainHost too needs some rework to support structured demands in a partial-trust environment, as shown here:
[SecurityPermission(
  SecurityAction.Assert,ControlAppDomain = true)]
[ReflectionPermission(
  SecurityAction.Assert,Unrestricted = true)]
public class AppDomainHost : IDisposable {
  protected AppDomainHost(Type serviceType,
    AppDomain appDomain,
    PermissionSet permissions,
    params Uri[] baseAddresses) {
    //Cannot grant service permissions
    // the host does not have
    permissions.Demand();

    //Rest of constructor
    }
}
Since AppDomainHost creates a new app domain, it asserts the permission to control the app domain. Since it uses reflection to inject the host in the other app domain, it asserts unrestricted reflection permission. Most importantly, before creating the service host in the new app domain, AppDomainHost demands the requested permissions for the service of the code that calls it. This ensures that partially trusted code can only launch and host a service that is permitted to do as much as, and no more than, the calling code.

Additional Aspects of AppDomainHost
While AppDomainHost does not support ICommunicationObject, it does offer its events model and state machine management. AppDomainHost also copies a few static variables from the calling app domain to the new app domain it created. Specifically, it uses ServiceHostActivator to copy to the other app domain the ASP.NET providers application name. AppDomainHost temporarily asserts the ASP.NET hosting permission, accesses the providers, copies the values of the membership and roles application names, and reverts the assertion.

Juval Lowy is a software architect with IDesign providing WCF training and architecture consulting. His recent book is Programming WCF Services. He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at www.idesign.net.


.NET Matters
Asynchronous I/O with WebClient
Stephen Toub

Q I really like the static helper methods on the System.IO.File class for reading and writing data: ReadAllText, ReadAllBytes, WriteAllText, WriteAllBytes, and so on. However, these methods are synchronous, and I'd like to be able to use them asynchronously such that under the covers they're also using asynchronous I/O. I know the System.IO.FileStream class supports asynchronous I/O; does File?

A File's methods only support synchronous operation. But the functionality to implement asynchronous methods like the ones you describe certainly exists. In this column, I'll walk through two ways to do it, starting with the more complicated, but more efficient, of the two.
First, I'll begin by defining the APIs I actually want to implement. The original signatures you cite look like this:
public static byte [] ReadAllBytes(string path);
public static string ReadAllText(string path);

public static void WriteAllBytes(string path, byte [] bytes);
public static void WriteAllText(string path, string contents);
I want asynchronous versions of these, and I'll define them as:
public static void ReadAllBytesAsync(
    string path, Action<byte[]> success, Action<Exception> failure);
public static void ReadAllTextAsync(
    string path, Action<string> success, Action<Exception> failure);

public static void WriteAllBytesAsync(
    string path, byte[] bytes,
    Action success, Action<Exception> failure);
public static void WriteAllTextAsync(
    string path, string contents,
    Action success, Action<Exception> failure);
These signatures are very similar to the originals. Instead of returning data synchronously, the Read* methods accept two delegates, one for successful execution and one for exceptional execution, where the delegate for the former is passed the read data and where the delegate for the latter is passed any resulting exception. The Write* methods also accept two delegates, but the success delegate is parameterless, since there's no expected output (the original Write* methods return void).
The more complicated approach involves implementing these directly with the Asynchronous Programming Model (APM) pattern exposed by the System.IO.Stream class, from which FileStream derives. In my column in the March 2008 issue of MSDN® Magazine (msdn.microsoft.com/magazine/cc337900), I demonstrated implementing a method CopyStreamToStream, which asynchronously copies from one Stream to another Stream using each Stream's APM implementation; for reference, that implementation is shown in Figure 1, and I'm going to reuse it here. (The implementation is slightly simplified and changed from the last time you saw it. This included removing the usage of System.ComponentModel.AsyncOperationManager and AsyncOperation, preferring to move their usage into the higher level APIs being implemented in this column. I'll discuss why in a moment.)
Figure 1 Asynchronous Stream Copying
public static void CopyStreamToStream(
    Stream source, Stream destination, Action<Exception> completed)
{
    byte[] buffer = new byte[0x1000];
    if (completed == null) completed = delegate {};

    AsyncCallback rc = null;
    rc = readResult =>
    {
        try
        {
            int read = source.EndRead(readResult);
            if (read > 0)
            {
                destination.BeginWrite(buffer, 0, read, writeResult =>
                {
                    try
                    {
                        destination.EndWrite(writeResult);
                        source.BeginRead(
                            buffer, 0, buffer.Length, rc, null);
                    }
                    catch (Exception exc) { completed(exc); }
                }, null);
            }
            else completed(null);
        }
        catch (Exception exc) { completed(exc); }
    };

    source.BeginRead(buffer, 0, buffer.Length, rc, null);
}
That implementation represents the most difficult aspect of implementing these asynchronous methods. On top of CopyStreamToStream, I'll implement two helper methods, one for reading bytes from a file asynchronously and one for writing bytes to a file asynchronously. They do the bulk of the work (see Figure 2).
Figure 2 Asynchronous Helper Methods
private static void ReadAllBytesAsyncInternal(string path,
    Action<byte[]> success, Action<Exception> failure)
{
    var input = new FileStream(path, FileMode.Open,
        FileAccess.Read, FileShare.Read, 0x1000, true);
    var output = new MemoryStream((int)input.Length);
    CopyStreamToStream(input, output, e =>
    {
        byte [] bytes = e == null ? output.GetBuffer() : null;
        output.Close();
        input.Close();

        if (e != null) failure(e);
        else success(bytes);
    });
}

private static void WriteAllBytesAsyncInternal(
    string path, byte[] bytes,
    Action success, Action<Exception> failure)
{
    var input = new MemoryStream(bytes);
    var output = new FileStream(path, FileMode.Create,
        FileAccess.Write, FileShare.None, 0x1000, true);
    CopyStreamToStream(input, output, e =>
    {
        input.Close();
        output.Close();

        if (e != null) failure(e);
        else success();
    });
}
ReadAllBytesAsyncInternal creates an input FileStream to work with the underlying stream asynchronously and a MemoryStream to store the bytes read in from the file. Then the CopyStreamToStream method is used to copy all of the data asynchronously from the FileStream to the MemoryStream. When the operation is complete, the streams are closed. The failure delegate is called if an exception was thrown; if not, the success delegate is called, provided with the data read into the MemoryStream from the file.
WriteAllBytesAsyncInternal is very similar. Here, an input MemoryStream is created that wraps the provided byte array, and an output FileStream is created, again one that supports asynchronous I/O. As with ReadAllBytesAsyncInternal, upon completion the streams are closed, and the failure delegate is called if an exception occurred.
Implementing each of the public signatures shown earlier now requires just a few additional lines on top of the methods I created inFigures 1 and 2. These are shown in Figure 3.
Figure 3 Implementing the Public Methods
public static void ReadAllBytesAsync(
    string path, Action<byte[]> success, Action<Exception>
        failure)
{
    AsyncOperation asyncOp =
        AsyncOperationManager.CreateOperation(null);
    ReadAllBytesAsyncInternal(path,
        bytes => asyncOp.Post(delegate { success(bytes); }, null),
        exception => asyncOp.Post(
            delegate { failure(exception); }, null));
}

public static void ReadAllTextAsync(
    string path, Action<string> success, Action<Exception>
        failure)
{
    AsyncOperation asyncOp =
        AsyncOperationManager.CreateOperation(null);
    ReadAllBytesAsyncInternal(path,
        bytes => {
            string text;
            using (var ms = new MemoryStream(bytes))
                text = new StreamReader(ms).ReadToEnd();
            asyncOp.Post(delegate { success(text); }, null);
        },
        exception => asyncOp.Post(
            delegate { failure(exception); }, null));
}

public static void WriteAllBytesAsync(
    string path, byte[] bytes, Action success, Action<Exception>
       failure)
{
    AsyncOperation asyncOp =
        AsyncOperationManager.CreateOperation(null);
    WriteAllBytesAsyncInternal(path, bytes,
        () => asyncOp.Post(delegate {success(); }, null),
        exception => asyncOp.Post(
            delegate { failure(exception); }, null));
}

public static void WriteAllTextAsync(
    string path, string contents, Action success, Action<Exception>
       failure)
{
    AsyncOperation asyncOp =
        AsyncOperationManager.CreateOperation(null);
    ThreadPool.QueueUserWorkItem(delegate
    {
        var bytes = Encoding.UTF8.GetBytes(contents);
        WriteAllBytesAsyncInternal(path, bytes,
            () => asyncOp.Post(delegate {success(); }, null),
            exception => asyncOp.Post(
                delegate { failure(exception); }, null));
    });
}
For the most part, each of these methods is a simple wrapper around the internal implementations shown in Figure 2. However, there are some interesting subtleties as well. First, I mentioned earlier that I was stripping the AsyncOperationManager support out of CopyStreamToStream. AsyncOperation is itself a wrapper around System.Threading.SynchronizationContext, and it uses the underlying SynchronizationContext captured in the call to AsyncOperationManager.CreateOperation to Post delegates back in a manner appropriate to the synchronization context current at the time of creation. For example, on the UI thread in a Windows® Forms application, SynchronizationContext.Current will likely return a WindowsFormsSynchronizationContext that supports marshaling delegate calls back to the UI thread. Thus, if AsyncOperationManager.CreateOperation is called on the UI thread, using the resulting AsyncOperation's Post method will result in the provided delegate being marshaled to and executed in the UI thread.
However, I want to minimize the amount of work that's done on the UI thread. Consider now the ReadAllTextAsync method. When I'm finished loading the data from the file, I want to convert the read bytes into a string (this would be more efficient if I had a writeable StringStream class and could pass an instance of that to CopyStreamToStream rather than passing a MemoryStream). The conversion could incur more of a cost than I'd like on the UI thread. Thus, I want to do the conversion before posting back to the UI. But in the original implementation of CopyStreamToStream, the completion delegate was run under the captured SynchronizationContext, and thus I'd already be on the UI thread by the time I wanted to do the conversion. Instead, I've pulled the AsyncOperation work into each of these outer methods; thus I can delay calling Post until I'm done with the real computational work.
Another interesting implementation detail exists in WriteAllTextAsync. WriteAllTextAsync doesn't return until the asynchronous operation has been kicked off. But before I can call WriteAllBytesAsyncInternal to do so, I need to convert the provided text string into a byte array. So, rather than block the calling thread, I queue a work item to the ThreadPool. That work item does the conversion from string to bytes, and then starts the internal copy.
In the end, this isn't a horrible amount of code, but it's also not simple. One would hope that these common asynchronous patterns for reading and writing file data would already exist somewhere in the Microsoft® .NET Framework, especially when you consider that there are other things you'd likely want to add on top of this, such as progress notifications as the data is being read or written. In fact, such functionality does already exist, but in an unlikely place: System.Net.
The System.Net.WebClient class is incredibly handy for a variety of purposes. Given its name and placement in the System.Net namespace, it's not surprising that most folks think of it as being only for Web-related activities: downloading files from an HTTP server, uploading files to an FTP site, and the like. But WebClient is nicely abstracted on top of WebRequest and WebResponse, which support a pluggable factory model for creating concrete implementations of each of these types. Calling WebRequest.Create with an HTTP URL will return an instance of an HttpWebRequest, just as calling it with an FTP URL will return an instance of an FtpWebRequest. WebClient uses WebRequest and WebResponse internally to implement a whole slew of useful functionality. Here's a sampling of some of the more relevant methods:
public void DownloadDataAsync(Uri address);
public event DownloadDataCompletedEventHandler DownloadDataCompleted;

public void DownloadStringAsync(Uri address);
public event
    DownloadStringCompletedEventHandler DownloadStringCompleted;

public void UploadDataAsync(Uri address, byte[] data);
public event UploadDataCompletedEventHandler UploadDataCompleted;

public void UploadStringAsync(Uri address, string data);
public event UploadStringCompletedEventHandler UploadStringCompleted;
So now if you call DownloadDataAsync providing it with the URL of the data to be downloaded, it will do so asynchronously; when it completes it'll raise the DownloadDataCompleted event. Similarly, call UploadStringAsync providing it with a text string and the location to which the string should be uploaded, and it will do so asynchronously; when it completes it will raise the UploadStringCompleted event. Nice and easy.
Now, the cool part is that one of the WebRequest providers built into the .NET Framework is FileWebRequest (and a corresponding FileWebResponse). With this, I can write code like:
WebRequest wr = Webrequest.Create(@"file://C:\test.txt");
This code gives me a WebRequest I can work with like any other, except this WebRequest is targeting a file on disk rather than some file on a Web site somewhere. Combine that with WebClient's usage of WebRequest, and I'm sure you see where I'm headed: I can use WebClient to read and write files asynchronously!
To implement the read and write methods I did earlier requires only a few lines of code per method, as shown in Figure 4 (in fact, at this point, I question whether it's even worth having these wrappers, since most of the code is just for adapting the made-up API signatures to those provided by WebClient for the same tasks). WebClient internally uses AsyncOperationManager to ensure the callbacks happen in the right context, and FileWebRequest/FileWebResponse will use asynchronous I/O when the asynchronous methods on WebClient are being used (WebClient also provides synchronous versions of these methods).
Figure 4 Using WebClient to Read and Write Files
public static void ReadAllBytesAsync(
    string path, Action<byte[]> success, Action<Exception> failure)
{
    var wc = new WebClient();
    wc.DownloadDataCompleted += (sender, e) =>
    {
        if (e.Error != null) failure(e.Error);
        else success(e.Result);
    };
    wc.DownloadDataAsync(new Uri("file://" + path));
}

public static void ReadAllTextAsync(
    string path, Action<string> success, Action<Exception> failure)
{
    var wc = new WebClient();
    wc.DownloadStringCompleted += (sender, e) =>
    {
        if (e.Error != null) failure(e.Error);
        else success(e.Result);
    };
    wc.Download﻿StringAsync(new Uri("file://" + path));
}

public static void WriteAllBytesAsync(
    string path, byte [] bytes,
    Action success, Action<Exception> failure)
{
    var wc = new WebClient();
    wc.UploadDataCompleted += (sender, e) =>
    {
        if (e.Error != null) failure(e.Error);
        else success();
    };
    wc.UploadDataAsync(new Uri("file://" + path), bytes);
}

public static void WriteAllTextAsync(
    string path, string contents,
    Action success, Action<Exception> failure)
{
    var wc = new WebClient();
    wc.UploadStringCompleted += (sender, e) =>
    {
        if (e.Error != null) failure(e.Error);
        else success();
    };
    wc.UploadStringAsync(new Uri("file://" + path), contents);
}
What's even cooler is that WebClient provides useful additional functionality on top of this; specifically, it already has built-in support for progress tracking. It provides two relevant events:
public event
    DownloadProgressChangedEventHandler DownloadProgressChanged;
public event
    UploadProgressChangedEventHandler UploadProgressChanged;
These are raised when there are progress updates to provide with regard to an in-process download or upload, respectively. And the event arguments provided to the handlers for these events provide useful information for progress reporting. For example, here's the DownloadProgressChangedEventArgs type:
public class DownloadProgressChangedEventArgs :
    ProgressChangedEventArgs
{
    public long BytesReceived { get; }
    public long TotalBytesToReceive { get; }

    /* from the base ProgressChangedEventArgs type
    public int ProgressPercentage { get; }
    public object UserState { get; }
    */
}
All of this lets me integrate asynchronous I/O very nicely into my GUI applications. Consider a Windows Forms application that needs to load a large file from disk into memory before working with it. I can load it asynchronously, with progress notifications that update a progress bar, and with completion notification:
private void button1_Click(object sender, EventArgs e) {
    WebClient wc = new WebClient();
    wc.DownloadDataCompleted += (s, de) => {
        _fileData = de.Result;
        MessageBox.Show("File loaded");
    };
    wc.DownloadProgressChanged +=
        (s, de) => progressBar1.Value = de.ProgressPercentage;
    wc.DownloadDataAsync(new Uri(@"file://c:\largeFile.dat"));
}
private byte [] _fileData;
Unfortunately, while the APIs are quite slick, there's a fairly significant problem here. While FileWebRequest/FileWebResponse do use asynchronous I/O under the covers, for whatever reason the implementation also relies on the worker ThreadPool for callback notifications; sometimes it blocks those threads from the pool waiting for the asynchronous I/O to complete. (Whether this is a code defect or just a bad performance problem is a matter of debate.) In the end, however, it means that if you're trying to use this WebClient technique to read or write a large number of files asynchronously and concurrently, you'll likely find your performance severely hampered by the number of threads in the ThreadPool.
Blocking threads in the ThreadPool is typically a bad idea, as it will force the ThreadPool to inject new threads, which not only consumes additional resources but also slows down the application as the ThreadPool throttles injection with time delays. All in all, while the WebClient approach for reading and writing files asynchronously may be good for a few files at a time, until FileWebRequest/FileWebResponse are modified to fix this issue, you're probably better off with the hand-coded versions shown earlier in this column. Another solution is to provide your own implementations of WebRequest/WebResponse that do the right thing, and then register those with WebClient.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager Lead on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.


{ End Bracket }
Customize Windows Home Server
Chris Gray

For the past few years, I've been working on Windows® Home Server, an application that backs up all of your home computers via a simple user interface. The idea behind Windows Home Server is that every member of the household stores their pictures, videos, and music in one safe location, making it easy to find the files. The server is really useful straight out of the box, of course, but developing custom applications that will make my home more fun and friendly is what has me most excited.
I am convinced that one day most of us will have a computer stashed somewhere out of the way that makes our home a little more useful and intuitive. Inexpensive devices and appliances will take advantage of the networking, processing, and storage capacities of this server to communicate with other appliances to make our homes a little more comfortable.
Over the past few months some friends of mine and I have been experimenting with these possibilities by taking advantage of Windows Home Server and our programming skills. We've had the most fun with three areas: media, Internet connectivity, and power consumption.
We've been centralizing our television and music onto Windows Home Server and viewing the content with an assortment of inexpensive network-capable media devices. Happily, we've found all sorts of cool devices that support UPNP, SMB, or HTTP, or can be used on the server just by installing some software, and we've written some custom software for our media centers that automatically copies recorded television to our servers at night.
Once on the server, we've written code that uses the server's spare CPU cycles to transcode the content into formats suitable for streaming to video and audio devices that look like stereo components. By centralizing in this way, we can choose network-capable devices that fit our needs without having to rewire our houses for stereo or televisions—we just use a normal wireless router that easily goes through walls.
Having all types of media on one computer and accessing it through the Internet has been a source of endless creativity. Building this solution wasn't too difficult—we just used the domain name that comes with the server to make finding the server easier and we built a Web site that has the ability to search for a particular piece of content.
This project got me thinking about using Web services to talk to an Internet-capable cell phone. I haven't created anything with it yet, but I bet I can have some fun linking the servers storage and processing power to a phone.
Another idea I've recently been dabbling with is reducing the total power consumption of my home by integrating lights and power outlets with the Windows Home Server. This idea isn't new (home automation has been around for years), but this is the first that I have grappled with it. The general idea is that your computer can control the power switch and shut off your TV, lights, thermostat, or other equipment when you're not home.
Several companies make really great light switches, power outlets, and dimmers that are computer-controlled. However, one of the major blocks to home automation has been the total price—these computer-controlled switches cost more than standard switches and usually require a special-purpose computer. If you've already got a Windows Home Server, though, all you need to do is install the included Windows-based application.
Looking ahead, I can imagine hooking up the Windows Home Server, adding a few light switches, and controlling everything from the server—and even controlling it remotely. I could then chart the total power usage of the entire house over a set period of time in order to evaluate my family's power usage.
Harnessing the power of the Web, I would also like to link to a real-time weather application so that if it was going to rain, for example, then the lawn sprinklers wouldn't turn on. Likewise, the system could download daily sunset and sunrise times so that the outside lights would go on at sunset and shut off at daybreak. Furthermore, if your Exchange calendar showed that you were on vacation, the furnace would stay at a low maintenance temperature.
From a developer perspective, experimenting with Windows Home Server has offered some exciting challenges and it's great to see the immediate results. I've enjoyed creating new applications that link my appliances in ways that make them more useful and accessible to me even when I'm away from home. For more possibilities, see msdn.microsoft.com/library/bb425866.

Chis Gray is a Development Lead for Windows Home Server at Microsoft. In his spare time he enjoys hiking and making furniture with his wife, Melinda.