December 2008
Team System: Streamline Team Projects With Process Templates
See how you can streamline your team projects using Process Templates in Visual Studio Team System (VSTS) Brian A. Randell
Real-World WF: Best Practices For Windows Workflow Foundation Apps
The heart of Windows Workflow Foundation is its declarative programming model. Here are some best practices to consider when using WF to realize software solutions in the real world. Josh Lane
Visual Studio OBA Tools: Simplify OBA Development With Interop API Extensions
To introduce you to VSTO Power Tools Office interop API extensions, we’ll walk through the development of an application that automates Outlook, Excel, and Word. Andrew Whitechapel, Phillip Hoff, and Vladimir Morozov
SOA Data Access: Flexible Data Access With LINQ To SQL And The Entity Framework
Building a data access layer using LINQ to SQL and the ADO.NET Entity Framework allows you to decouple your application from the persistence technology you're using. Anthony Sneed
Geneva Framework: A Better Approach For Building Claims-Based WCF Services
Here we introduce Microsoft Code Name “Geneva,” the new framework for building claims-based applications and services, and federated security scenarios. Michele Leroux Bustamante
Columns
Editor's Note: I Am The Business: At an open spaces conference in Austin, Texas, Editor-in-Chief Howard Dierking realizes that software development and business goals can both coexist and benefit each other.Howard Dierking | Toolbox: Static Analysis Tools For .NET, Matt Berseth’s Blog: This month we take a look at FxCop and other tools that enforce your design rules, along with jQuery.Scott Mitchell | CLR Inside Out: Thread Management In The CLR: Getting the performance you want in concurrent applications is not as straightforward as you might think. See how common threading issues can affect your application.Erika Fuentes and Eric Eilebrecht
Advanced Basics: The ObservableCollection Class: Ken Getz shows how the CollectionChanged event lets you reflect changes to your underlying data source in your bound data controls.Ken Getz | Cutting Edge: ASP.NET Presentation Patterns: Choosing the right design pattern for your ASP.NET Web application can help you achieve the separation of concerns between your presentation layer and the layers beneath it.Dino Esposito | Patterns in Practice: Design For Testability: Designing testability into your app means smaller tests that are cheaper to create, easier to understand, faster to run, and much simpler to debug.Jeremy Miller
Test Run: Configuration Testing With Virtual Server, Part 2: Because Virtual Server is built upon a set of COM modules, you can automate the creation and testing of virtual machines. Here we use Windows PowerShell to run the tests.Dr. James McCaffrey and Paul Despe | Foundations: Render Text On A Path With WPF: With Windows Presentation Foundation (WPF) you can lay out text on a path, then animate the individual points defining the path and watch the characters bounce around in response.Charles Petzold | Windows With C++: X64 Debugging With Pseudo Variables And Format Specifiers: This month we explain how pseudo variables and format specifiers provide a wealth of information for use in debugging.Kenny Kerr
Going Places: How Connection Manager Connects: Marcus Perryman explains the correct use of Connection Manager when a Windows Mobile application requires network data.Marcus Perryman | Usability in Practice: The Human Face Of Software: A great user experience is more than just a pretty face. In this new column we’ll look at some of the subtleties of building great user experiences.Dr. Charles B. Kreitzberg and Ambrose Little | { End Bracket }: What Makes A Good Software Tester?: In this month's installment, James McCaffrey talks about the qualities and skills he looks for when searching for great software testers.Dr. James McCaffrey


Team System
Streamline Team Projects With Process Templates
Brian A. Randell

This article discusses:
  • Process template basics
  • The MSF Agile and MSF CMMI templates
  • Customizing process templates
  • Creating your own templates
This article uses the following technologies:
Visual Studio Team Foundation Server 2008
Contents
Process Templates
Inside the MSF Templates
Process Template Customization
Creating Your Own Templates
Editing Customized Templates
Today you can collect and track all of your work and artifacts in Team Foundation Server (TFS) within a team project. A team project is simply a bucket that stores and partitions all of the artifacts you track and use during a development project. And using the New Team Project wizard from the Team Foundation Client (TFC) can help you on your way.
The wizard is deceptively simple. Clicking File | New team project starts the wizard. Once it has opened, you must provide a name for your team project. Click Next and pick a process methodology template using the combobox. Click Next yet again and add a description for your team project that SharePoint displays on your portal's home page. Clicking Next once more gives you the option to decide how you want version control defined. You can choose to create a new trunk, create a new trunk by branching from an existing trunk, or not create a trunk. After you have made your choices, click Finish and, after a few moments, you'll have a new team project ready for use.
If that were all there was to it, then this would be a very short article. My editor would most likely withhold payment and even reconsider my column's status. However, it's not that simple. In fact, the page of the wizard where you choose the process templates is critical to getting your development project off to a good start when using Visual Studio Team System (VSTS). This is because a process template defines the initial structure and contents of your team project.

Process Templates
As mentioned, a team project provides a collection mechanism for all of your team's artifacts. Naturally, you store version-controlled items, such as source files, in a team project. You also manage and store all of your work items and work item queries. You store documents in SharePoint document libraries that are a part of your team project's portal. And finally, you run reports against the data warehouse to track how well your team is working.
A process template defines the default artifact templates included in your team project. Specifically, it defines the work item types you'll have available to you. It also defines the user interface and structure of your SharePoint site as well as the default documents loaded into the document libraries—both files and templates. It also provides the default reports for your reporting services site and defines your default security groups, areas, iterations, and version control settings. And finally, a process template provides prescriptive guidance as part of your SharePoint site.
When you first install TFS, you will find that it provides you with two process templates. Their official names in the 2008 release of TFS are Microsoft Solutions Framework (MSF) for Agile Software Development (MSF Agile) and MSF for CMMI Process Improvement (MSF CMMI).
MSF promotes an iterative development process with heavy customer interaction. The current release of MSF is version 4.0 and influences the TFS process guidance, though it has its own rich set of MSF guidance resources. You can find further useful information in the book Microsoft Solutions Framework Essentials (Microsoft Press, 2006) by Michael Turner.
The prescriptive guidance included with the Microsoft process templates dives into how to run a development project and how to structure your work streams. This guidance is focused on the process of development, not the mechanics of Visual Studio or TFS. You access the guidance through the Process Guidance link off the home page of your team project's SharePoint portal home page. The guidance is also available for download on its own from Microsoft for both MSF Agile and MSF CMMI if you want to peruse it without needing live access to TFS.

Inside the MSF Templates
An easy way to contrast the MSF Agile and MSF CMMI process templates is agile versus formal. Both templates define different roles based on a team of peers broken down into seven logical areas. The MSF Agile template defines 8 roles as part of its team model. In contrast, MSF CMMI defines a whopping 20 roles. As shown in Figure 1, the CMMI template expands each logical area with more specific roles. These include managers and additional stakeholders that are not part of the core development team.
Figure 1 MSF Agile and MSF CMMI Team Member Roles
Area | Role | CMMI | Agile
Architecture | Infrastructure Architect | Y | N
Architecture | Solution Architect | Y | N
Development | Build Engineer | Y | N
Development | Development Manager | Y | N
Development | Lead Developer | Y | N
Product Management | Auditor | Y | N
Product Management | Product Manager | Y | N
Product Management | Sponsor | Y | N
Product Management | Subject Matter Expert | Y | N
Program Management | Integrated Project Management (IPM) Officer | Y | N
Test | Test Manager | Y | N
User Experience | User Education Specialist | Y | N
User Experience | User Experience Architect | Y | N
Development | Database Developer | Y | Y
Development | Developer | Y | Y
Product Management | Business Analyst | Y | Y
Program Management | Project Manager | Y | Y
Release Operations | Database Administrator | Y | Y
Release Operations | Release Manager | Y | Y
Test | Tester | Y | Y
Architecture | Architect | N | Y
User Experience | Business Analyst | N | Y
Beyond the roles, you find work items have a number of differences (see Figure 2), including differences in the actual work item types provided for each template. Microsoft composes each work item type from four facets: purpose, fields, workflow, and layout. Thus, while both templates have a bug work item, fundamentally they are different types. In fact, a key difference between the MSF Agile and the MSF CMMI work items is their workflow expressed as states and transitions. If you drill into the process guidance for an MSF Agile bug and examine the States and Transitions section, you'll note that an MSF Agile bug has three states and 23 transitions (see Figure 3). In contrast, the MSF CMMI bug has four states and 17 transitions (see Figure 4).
Figure 2 MSF Agile and MSF CMMI Work Item Types
Work Item Type | Description | CMMI | Agile
Change Request | A change request identifies a proposed change to some part of the product or baseline. | Y | N
Issue | The issue work item documents an event or situation that may block work or is currently blocking work on the product. | Y | N
Requirement | Requirements capture and track what the product needs to do to solve the customer problem. | Y | N
Review | The review work item documents the results of a design or code review. | Y | N
Bug | A bug is a work item that communicates that a potential problem exists or has existed in the system. | Y | Y
Risk | A risk is any probable event or condition that can have a potentially negative outcome on the project in the future. | Y | Y
Task | A task work item communicates the need to do some work. | Y | Y
Quality of Service Requirement | Quality of service requirements document characteristics of the system such as performance, load, availability, stress, accessibility, serviceability, and maintainability. | N | Y
Scenario | A scenario is a type of work item that records a single path of user interaction through the system. | N | Y
Figure 3 MSF Agile Bug States and Transitions
Figure 4 MSF CMMI Bug States and Transitions
You'll notice, for example, that the CMMI bug starts in a proposed state while an MSF Agile bug is active once created. Another difference you'll find between the two types of bugs is purpose. Microsoft made the MSF Agile bug a catch-all for all defects. If you examine a non-customized MSF Agile bug, you'll see that it contains a Boolean field named Issue. For the CMMI process template, Microsoft created a separate Issue work item type. This is just one example of the differences. However, differences such as these will affect your team's workflow when using a team project.
Each of the MSF work item types has a section in the documentation labeled Process Guidance. The process guidance is broken down into two sections: work streams and activities. Work streams are a logical flow of activities typically related to a given role. An activity is a group of steps focused on a particular task.
Each activity has both entry and exit criteria and is associated with one or more roles and a work stream. For example, if you look at the Reproduce the Bug activity in MSF Agile, you'll see it's a part of the Fix a Bug work stream and developer is the only participating role. Other activities such as Determine Iteration Length, which is also from the MSF Agile template's Plan an Iteration work stream, have both a responsible role (the project manger) and participating roles (developers and database developers).
Depending on how mature your team and process already is, you might adopt all, some, or none of the advice provided in the guidance. However, it's important to realize that Microsoft has designed the work items, reports, and documents included with each template to work within the design of each template's guidance. Thus, you should review the documentation and figure out what works and what doesn't for your team. You can then decide if you want to live within the framework presented or adapt it for your team.

Process Template Customization
While it may be possible for your team to work with one of the Microsoft templates out of the box, more than likely you will want to change something. Number one on most teams' lists is customization of work items. Beyond that there are check-in policies, the SharePoint portal site, and reports to create and customize.
Since my very first MSDN Magazine column, I have extolled the benefits of Team System's rich customization feature set, and process templates are no exception. Just about every single piece of a process template is changeable and customizable. In fact, a good way to find out just what's possible is for you to take the time to explore two of the more popular third-party templates that extol the virtues of the Scrum development lifestyle.
Conchango published the first Scrum public template that I'm aware of, and there are template versions for TFS 2005 and TFS 2008. As we go to press, Conchango provides both templates free to the community but requires that you register to get the download link.
Once you've downloaded the template, you extract the contents and run the setup package. Once configured, you'll find the template as a choice in the New Team Project wizard. If you create a team project, you'll note that the TFS 2008 version of the template defines six custom work item types, 16 team work item queries, 15 reports, a SharePoint template, and its own process guidance.
A second popular template is the VSTS Scrum Process Template (Lightweight Scrum). VSTS MVPs Mike Azocar, Steven Borg, and Willy-Peter Schaub started this as a community-driven project. Like the Conchango template, the project owners have made the template available for both TFS 2005 and TFS 2008. In addition, they've provided a version that supports working with the Microsoft Project Server 2007 VSTS Connector.
The TFS 2008 version of the Lightweight Scrum template comes with six custom work item types, 18 team work item queries, 5 reports, a SharePoint template, and its own process guidance. To install the template, you must first install the SharePoint template. Once you've done this, install the rest of the Lightweight Scrum template using the Process Template Manager, accessible via the Team | Team Foundation Server Settings menu inside Visual Studio. Unzip the Lightweight Scrum template into a folder, then open the Process Template Manager and click the Upload button. Navigate to the folder containing the ProcessTemplate.xml file and click the Upload button (see Figure 5).
Figure 5 Importing the Lightweight Scrum Template
Both Scrum templates give you a good idea as to the level of customization that is possible. If you search the Internet, you'll find additional templates out there. A few VSTS MVPs have set up a CodePlex project called Templex to create a community-driven Process Template Library.

Creating Your Own Templates
While it's possible for you to grab the documentation and build your own process template by hand, most customers I've worked with find it easier to take an existing Microsoft template and start from there. I'll be describing this approach to get you started.
Begin by creating a directory in order to download one of the Microsoft process templates. If you're serious about doing this, you should configure a location in version control to check in the template after you download it. Using the TFS Settings menu item off the Team menu, download the MSF Agile template using the Process Template Manager to C:\PMT. The Process Template Manager creates a folder called MSF for Agile Software Development–v4.2. Within this folder, you will find six folders and an XML file—ProcessTemplate.xml (see Figure 6).
Figure 6 The Results of Downloading the MSF Agile Process Template
Most of the folders and their contents are self-explanatory. You describe a process template and its contents using a number of XML files. These XML files, in conjunction with the actual artifacts (report definition files, HTML documentation, and so on), come together to define the process template. Figure 7 lists the configuration files, their locations, and their purpose.
Figure 7 Process Template Configuration Files
Folder\File | Purpose
%root%\ProcessTemplate.xml | ProcessTemplate.xml defines a template's core structure. It references all the other files except FileMapping.xml.
Classification\Classification.xml | Classification.xml defines a template's Areas and Iterations. In addition, it references FileMapping.xml.
Classification\FileMapping.xml | FileMapping.xml maps work item fields to fields in Microsoft Project.
Groups and Permissions\GroupsandPermissions.xml | GroupsandPermissions.xml defines the default groups and their permissions.
Reports\ReportsTasks.xml | ReportsTasks.xml defines the list of Reporting Services report definitions.
Version Control\VersionControl.xml | VersionControl.xml defines the permissions in the version control repository, default check-in notes, as well as default checkout behavior.
Windows SharePoint Services\WssTasks.XML | WssTasks.xml defines what SharePoint site template to use (something you must already have installed). In addition, it defines document libraries, folders, and any files including the template's process guidance.
WorkItem Tracking\workitems.xml | Workitems.xml defines the list of work item types and your process template's team work item queries. In addition, it contains the definitions for any default work items that the new team project should create during team project creation.
You can modify the XML files a number of ways. Naturally, any text editor, such as Notepad, will work. If you're comfortable editing XML by hand and like the Visual Studio XML editor, you can hook up the XSD files by downloading the Visual Studio 2008 SDK. You'll find the XSD files at %SDK Install Folder%\VisualStudioTeamSystemIntegration\Process Templates\ProcessTemplateSchemas. In addition, the XSD files for work item definitions and global lists are found at %SDK Install Folder%\VisualStudioTeamSystemIntegration\Work Item Tracking\WorkItemTrackingSchemas. You can find the process template schemas documentation on MSDN.
You may find it more convenient (especially when you're new to the whole process) to use the visual Process Template Editor that comes as part of the VSTS 2008 TFS Power Tools (discussed in more detail in "Essential Power Tools"). You access the Process Template Editor from the Tools menu within Visual Studio.
Before you open any of the files from the download template, you should rename the folder to the name you want to use for the template—for example, "MSDN Magazine Agile 1.0." Once you've done this, with the latest version of the Power Tools installed, open the ProcessTemplate.xml file by clicking Tools | Process Templates | Open Process Template in Visual Studio. The template designer will open. Change the name and description to something reasonable (see Figure 8) and then save your changes. If you close the designer and reopen the file, the editor will show your new name. The tree view on the left side of the editor provides access to the core files described in Figure 7 as well as all of the work item type definitions and work item queries.
Figure 8 Edit the Name and Description of Your New Template
Most of the items exposed with the main Process Template Editor are self-explanatory when it comes to editing. As mentioned earlier, work items tend to be the first big item changed. In the tree view, select the Bug work item under the Work Item Types Definitions node. Visual Studio opens the work item type in the Work Item Type definition editor. The editor lets you change the name and description using normal textbox controls.
The work item editor has three tabs (see Figure 9) that provide access to the work item type's definition: Fields, Layout, and Workflow. In addition, the View XML button provides you with an easy way to see the XML representation for the type. You'll see this button throughout the editors, and it is a good way for you to learn the effects of your changes to the raw XML files. As you get more comfortable, you might find editing the XML files is quicker and easier than using the Process Template Editor.
Figure 9 Editing the Bug Work Item Type
The Fields tab lists all of the fields used by the work item type. Microsoft defines a field as having a name, a type, and a ref name. You can think of a ref name as a field's fully qualified name. It must have at least a two-part name separated by a period. Ref names serve as column display names in work item queries.
Click the New button to add a new field. In the Field Definition dialog, fill in the fields, as in Figure 10, and click OK. The name and type fields are obvious. Other valid data types you can choose include DateTime, Double, History, HTML, Integer, PlainText, String, and TreePath. History and TreePath are specialized controls for history fields and the areas and iterations fields, respectively. The Help Text shows up when you hover over the field when working with a work item instance.
Figure 10 Values for a New Field Definition
The Reportable field controls whether or not TFS exports the field to the TFS data warehouse so that you are then able to include it in your reports. If you decide to make it a Reportable field, it is important that you choose whether it's a dimension, detail, or measure. You use a dimension for DateTime, Double, Integer, and String fields. You choose a dimension field if you want to filter using the field. This choice is best when the field uses a standard set of values.
Use the detail option when you have large text fields on which you want to report. TFS does not put detail fields into the data warehouse cube, only the relational warehouse database. You can use DateTime, Double, Integer, and String fields for detail fields.
Finally, you use a measure field, which only supports Double and Integer data, when you want to use a formula (such as sum or count) or do a similar operation on a field. If you choose measure, you'll want to pick an option from the formula field. It's important to note that the current documentation incorrectly states that only sum is valid.
It's important to remember that once you apply the Reportable attribute to a field, you cannot change it. However, you can apply the attribute to a field after you've already used it. Any revisions to a work item after you've made its field Reportable will be in the warehouse (if applicable). However, TFS will not backfill data into the warehouse for changes made prior to setting the field as Reportable.
Once you've created the field, you'll want to specify where it should appear on the form. Click the Layout tab in the Work Item Type editor. You can click the Preview Form button to examine the current work item. In the tree view, move down until you see the Tab Page—Details node. Under it you'll find a Group node and, below that, two Column nodes.
Under the first column node, beneath the Group—General node, you'll find a single Column node. Right-click on it and select New Control. Select the newly added control node and change the FieldName to MSDN.Mag.Beverage using the Property Grid to the right. Change the label to Beverage and then preview the results. If you're happy, save your work and close any open editors.
Using the Process Template Editor, click the Upload button and navigate to the folder that has your modified ProcessTemplate.xml file. Once you've uploaded your template, run the New Team Project wizard and create a team project called MSDNMagDemo. (I recommend you test this using a Virtual PC image, as discussed in the "Essential Power Tools" installment of Team System mentioned earlier.) Once you've created your team project, add a new Bug work item and switch to the Details tab. You should see your new field. Close the new Bug without saving it.

Editing Customized Templates
Imagine you'd like to change this newly created field to have a specific list of allowed beverages. In addition, you want to be able to run reports on the field. Because you'll want this change to take effect for both the team project you just created as well as any new team projects, you'll need to do a little extra work.
First, reopen the Bug work item template by clicking Process Editor | Work Item Types | Open WIT from File. Navigate to C:\PMT\MSDN Magazine Agile 1.0\WorkItem Tracking\TypeDefinitions and select the Bug.xml file. Locate the Beverage field, select it, and then select the Open command on the toolbar at the top of the fields list. Change the Reportable field value to dimension.
Next, click the Rules tab and then click New on the toolbar. As you can see from the Select a rule type dialog, each field can have a number of rules applied to it. Select ALLOWEDVALUES and click OK. In the ALLOWEDVALUES dialog, click New. In List Item Edit, enter Coffee and click OK. Repeat two more times adding Soda and Juice to the list (feel free to be as creative as you like, or as your project allows). Once you've completed the list, click OK to close the ALLOWEDVALUES dialog. Click OK to close the Field Definition dialog and save your changes using the Save command on the Visual Studio File menu.
At this point, you've changed the file on disk. You need to re-upload the template as before. When you attempt to do so without any additional changes to the ProcessTemplate.xml file, you'll receive a warning that the template already exists. Click Yes to overwrite the previous copy of the template.
This solves the problem for new team projects. However, you need to fix the existing Bug work item template in the MSDNMagDemo team project. Click Process Editor | Work Item Types | Import WIT from File Server. In the Import Work Item Type Definition dialog, click the Browse button and select the Bug.xml file at C:\PMT\MSDN Magazine Agile 1.0\WorkItem Tracking\TypeDefinitions. Then, in the list of team projects in the Project to import to tree, select the MSDNMagDemo team project and click OK. If the MSDNMagDemo team project doesn't appear, you'll need to close and restart Visual Studio.
Before you try to add a new Bug, you must refresh the Work Item definition cache by right-clicking on the Work Items folder under the MSDNMagDemo team project node in the Team Explorer window and select Refresh. Figure 11 shows what you should see.
Figure 11 The Completed Bug Work Item with the New Beverage Field
I hope you've seen that Microsoft put a ton of effort into providing not only a tool but a tool with prescriptive guidance. I recommend you spend time getting to know both of the built-in templates (or even those produced by the community). Just give yourself time to understand the implications that picking a template will have on your team and your overall development progress.

Brian A. Randell is a Senior Consultant with MCW Technologies LLC. He spends his time speaking, teaching, and writing about Microsoft technologies. Brian is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


Real-World WF
Best Practices For Windows Workflow Foundation Apps
Josh Lane
Code download available from the MSDN Code Gallery
Browse the Code Online

This article discusses:
  • The workflow programming model
  • Workflows and services
  • Unit testing workflows
  • Plugging WF into your solution
This article uses the following technologies:
Microsoft .NET Framework 3.5, Windows Workflow Foundation, Visual Studio 2008
Contents
The Workflow Programming Model
Side Effect-Driven Programming
Services, Services, Services
Episodic Execution
Unit Testing Is Still Your Friend
A Runtime within a Runtime
It's Not All or Nothing
Plug and Play
Domain Modeling and Program Design
Additional Advice
Since its release as part of the Microsoft .NET Framework 3.0, I've spent a lot of time with Windows Workflow Foundation (WF), implementing systems with the technology and teaching others to do the same. These experiences have given me some insights into some best (and less-than-best) practices to consider when using WF to realize software solutions in the real world.
My experience has been that WF suffers from something of an identity crisis. Many developers I talk to understand, at some level, what's going on in WF: conditional logic, flow control, atomic actions, and so on.
"Yes, I think I understand. You drag shapes onto the design surface. These represent actions that execute in some sort of 'flowchart' sequence. Yes, that certainly is clever!" This can be followed by mild protests along the lines of "But what would I use this for?" or "But I can do this already in C# or Visual Basic!"
In the abstract, these are not unreasonable objections. But while it's true that the public face of WF is the drag-and-drop designer experience in Visual Studio, and that a superficial look at the features of WF will uncover capabilities seemingly found elsewhere, the real benefits of WF are found underneath the surface.
To appreciate the value of WF, it's useful to consider some historical context. Since its inception, a primary goal of the .NET Framework has been to raise the level of abstraction at which Windows software programs are constructed. COM, HRESULTs, smart pointers, MTAs, message pumps, thunking layers … the sheer number of non-domain-specific details a Windows programmer needed to grasp could be overwhelming.
Thankfully, the .NET Framework has succeeded in turning most of these details into historical trivia. Indeed, most of us developers take for granted that we can open up Visual Studio and immediately start writing code that maps directly to the problem at hand. We're happy to let the .NET runtime worry about issues such as memory management, enforcement of security policy, runtime resolution of code dependencies, and countless other details. As a result, many developers are far more productive today than they've ever been before.
WF represents a logical next step in this march toward higher levels of programming abstraction. WF programs execute within a special runtime environment that itself runs on the .NET CLR. The WF runtime environment imposes some restrictions on the developer authoring programs that target it. But in return WF offers a powerful, flexible, and extensible set of runtime services such as support for long-running code, where execution might span days or even months; support for pausing/resuming/canceling running programs; support for auditing and tracing program execution; and even modification of running program logic!
The heart of WF is its declarative programming model. That is, in WF you describe what you want your program to do, typically via the drag-and-drop WF designer in Visual Studio. However, you don't specify exactly how this work should be accomplished. Instead, the WF runtime uses your program description (ultimately, a hierarchical graph of control flow and activities) as a blueprint for execution of your desired logic.

The Workflow Programming Model
Spend five minutes with the workflow designer in Visual Studio and it becomes obvious that WF is a component technology. A component in WF is an activity. More concretely, it is any subclass of the Activity base class.
Activities are the unit of composition and execution in WF. Some activities—those that subclass CompositeActivity, which itself subclasses Activity—have child activities that execute as part of the parent activity's execution sequence. Indeed, a workflow in WF is nothing more than a graph of such activities, starting with a single root activity and expanding to arbitrary depth and complexity. The innate logic of each activity in the graph, combined with data manipulated by this logic, ultimately determines execution order and overall workflow behavior.
I like to think of WF activities as Lego blocks. With Legos, you combine individual blocks of varying color, size, and function (minimally useful on their own) to build a larger whole, greater than the sum of its parts. Similarly, in WF, you combine atomic activities that perform discrete functions to create higher-order sets of logic, which are then combined to realize still larger business processes, and so on.
One pattern common to component programming in general (and WF in particular) is to prefer composition over inheritance to achieve reuse. As you work with WF over time and build increasingly complex workflows, you'll more often find yourself composing programs from smaller, autonomous activities rather than creating deep class hierarchies along the lines of complex systems written in C# or Visual Basic.
You're still able to employ class inheritance as a reuse technique in WF. There's nothing wrong with deriving from SendEmailActivity to create, say, SendEncryptedEmailActivity. In this case, you're altering the fundamental nature of the action being performed, and inheritance is a natural technique for this. But if you need to conditionally send the e-mail only under certain circumstances or do other work before and after sending the e-mail, a better choice is to make SendEmailActivity a child of another CompositeActivity subclass (like SequenceActivity or IfElseActivity) and define the additional logic outside SendEmailActivity.
Figure 1 shows a composite activity that encapsulates a conditional e-mail action. This allows SendEmailActivity to maintain a single responsibility (sending e-mails), thus increasing its chance for reuse in other workflows. It also allows you to reuse existing constructs in WF (support for conditional branching and sequential activity execution) instead of writing new code from scratch.
Figure 1 A Custom Composite Activity with Conditional Logic

Side Effect-Driven Programming
In programming, a side effect is defined as an observable change to shared state that results from execution of some part of the program. Examples of this include a change in value to an object property, or a change to one or more values in a database table. Side effects are used all the time to accomplish real work in software.
In contrast to pure functional programming where shared state is avoided and method return values are the primary means of data flow and manipulation, everything interesting in a workflow involves a side effect of some kind. Within a workflow program there is no such thing as a return value; instead, you compare pre- and post-execution state for changes. You dictate control flow not by calling methods, but rather by composing activities alongside and within one another. Activity inputs and outputs are implicit in the design of the activity class (as properties), rather than explicitly defined via method parameters or return values. Data flows from one activity to another using built-in support for activity data binding. You also have the option of manipulating shared state by interacting with configured services in the logic of your custom WF activities.
The basic execution sequence for WF activities is to initialize activity state, execute activity, examine resultant activity state for changes, then repeat for all other activities. So it is precisely the side effects of activity execution ("What is the new value of the PurchaseOrderTotal property?" or "What database value just changed?") that tell you what just happened and what will happen next. You do have the potential for activities to manipulate state in ways undesirable for other downstream activities (however, note that the risk here is no worse than that inherent in any imperative language like C#), but you don't pay the performance costs associated with data copying.

Services, Services, Services
As a general purpose programming environment, WF is designed to integrate with external subsystems and code. The most direct way to accomplish this is to author custom workflow activities that make database or Web service calls or call into third-party libraries. In WF terminology, such external systems are known as services.
The idea behind WF services is simple: any library or functional element that is required by an activity in your workflow, but is not contained within the activity definition, is registered as a service with the WF runtime:
// create an instance of your service type...
  LoggingService logSvc = new LoggingService();

  // register it with the WorkflowRuntime instance...
  workflowRuntime.AddService( logSvc );

  WorkflowInstance instance =
    workflowRuntime.CreateWorkflow( typeof( Workflow1 ) );

  instance.Start();
During execution, an activity can request access to the services it needs through the ActivityExecutionContext (activities never interact directly with their host runtime) and use the configured services to perform work, while letting the WF runtime and host manage service lifetimes:
protected override ActivityExecutionStatus Execute(
    ActivityExecutionContext aec ) {
  // ActivityExecutionContext provides access to services...
  LoggingService logSvc = aec.GetService<LoggingService>();

  Debug.Assert( logSvc != null );
  logSvc.LogEvent( EvtType, Message );
  return ActivityExecutionStatus.Closed;
}
Typical service examples include logging components, data access layers, and proxies to external subsystems, though the possibilities are limitless.
The power of the WF service model stems from its simplicity. Literally any .NET object can be configured as a service. Note that some extra behaviors, such as service lifetime management, are available to services explicitly designed to subclass WorkflowRuntimeService. Also, I usually find it useful to expose services via an interface instead of through a concrete type; this makes it easier to vary the service implementation independent of the activities that depend on it.
So the key best practice here is the decoupling of a custom WF activity's core logic from the constituent pieces needed to accomplish its work. Additionally, you should refactor into a service logic that is stateful and is shared among multiple activities in a workflow (logging, for example); it would be cumbersome to pass a reference to the logging component down the activity graph during execution. This is further complicated for workflows serialized at some point during their execution cycle.
I'll mention one other bit of advice regarding services. Windows WF ships with an infrastructure known as local services for calling out to .NET class instances from activities, as well as handling .NET events inside an activity. You interact with these local services via the CallExternalMethod and HandleExternalEvent activities. They are essentially an extra layer on top of the base WF service infrastructure and provide the same benefits as raw services in a declarative style friendlier to WF host environments such as SharePoint. However, for more general-purpose hosts such as Windows Presentation Foundation (WPF) or ASP.NET, this extra layer can be more cumbersome to use than the equivalent raw service configuration. Keep in mind your anticipated hosting requirements before building solutions with local services.
A key principle to employ here is to leverage the power and flexibility of the WF service model to design well-factored, reusable workflow activities.

Episodic Execution
One feature that makes WF particularly well suited for general-purpose programming tasks is its support for asynchronous, non-blocking work. The very nature of modern Web service- and database-centric applications lends itself to asynchronous invocation, where a request is made to the service and then local resources associated with the requesting call are freed to process other work. Later, when the dispatched work is completed, execution can pick back up where it left off. ASP.NET asynchronous pages are one example of this technique.
When first executed by the WF runtime, the activity's Execute method is invoked. Execute returns ActivityExecutionStatus.Completed if the activity has performed all of its work within Execute. It returns ActivityExecutionStatus.Executing if it reaches a point where it's done as much work as possible, and is now waiting on some external stimulus, such as a response from a Web service or database call or an e-mail sent to a specific address, to continue.
Figure 2 shows the Execute method for an activity that queries an external service for a credit score, given a customer ID. Since credit scoring might take minutes or even hours to complete, the workflow invokes the service asynchronously and then goes idle waiting on a result. Before going idle, the activity registers a delegate to be invoked by the runtime when the stimulus arrives. This delegate is known as a continuation (literally, a "pointer to the rest of the program") and allows the activity author to define any logic needed to continue activity processing. The lifetime of a single workflow may consist of several such idle-continue state transitions.
Figure 2 The Execute Method
protected override ActivityExecutionStatus Execute(
  ActivityExecutionContext aec ) {

  if ( string.IsNullOrEmpty( SSN ) ) {
    throw new InvalidOperationException( "SSN property value is invalid." );
  }

  ICreditScoring creditSvc = aec.GetService<ICreditScoring>();
  Debug.Assert( creditSvc != null );

  WorkflowQueuingService queueSvc = aec.GetService<WorkflowQueuingService>();
  Debug.Assert( queueSvc != null );

  Guid queueName = Guid.NewGuid();
  WorkflowQueue queue = queueSvc.CreateWorkflowQueue( queueName, false );
  queue.QueueItemAvailable += CreditScoreComputed;
  creditSvc.ComputeScoreAsync( SSN, queueName );

  return ActivityExecutionStatus.Executing;
}
The runtime receives notice of the external stimulus needed to wake an idle workflow through arrival of an item (any .NET object) on a workflow queue associated with the workflow instance. In the case of an asynchronous Web service call, the code waiting on the response obtains a reference to the queue and pushes any relevant data (presumably whatever was finally received from the Web service) onto the queue. The runtime monitors the queue and invokes the registered delegate; the delegate logic can obtain the queued item and perform any necessary processing.
The power of this approach stems from what happens to the workflow during idle time. Activities that return ActivityExecutionStatus.Executing from their Execute method (and are thus considered idle) are obviously doing no work for some period of time. This might be seconds, hours, days or even months of waiting on some external stimulus. During this time, there's no need for the workflow to remain in memory consuming machine resources that could otherwise be put to better use, so the WF runtime supports the ability to automatically unload such workflows from memory and serialize them to persistent storage using a persistence service, such as SqlWorkflowPersistenceService. When the external stimulus arrives, the WF runtime reloads the workflow and continues execution as normal.

Unit Testing Is Still Your Friend
Unit testing remains relevant to your life as a programmer targeting WF. In fact, the explicit compositional nature of WF is a natural fit for authoring a battery of comprehensive tests that exercise each piece of your solution in isolation of other pieces. I'll give you some higher-level information here, but be sure to check out Matt Milner's Foundations column in the November 2008 issue of MSDN Magazine ("Unit Testing Workflows And Activities") for a deeper look at unit testing workflows.
A common misconception when working with WF is that you can only execute entire workflows (meaning, subclassed instances of SequentialWorkflowActivity). In fact, you can execute any activity instance as a standalone workflow, which makes it easy to use a generic activity test harness to execute and validate activities in isolation or in small subsets of a larger whole. Figure 3 shows such a test harness.
Figure 3 A Generic Workflow Test Harness
protected Dictionary<string, object> ExecuteActivity(
  Type activityType,
  IEnumerable<object>
  services,
  params object[] inputs ) {

  Dictionary<string, object> outputs = null;
  Exception ex = null;

  using ( WorkflowRuntime workflowRuntime = new WorkflowRuntime() ) {
    AutoResetEvent waitHandle = new AutoResetEvent( false );

    workflowRuntime.WorkflowCompleted +=
      delegate( object sender, WorkflowCompletedEventArgs e ) {

      outputs = e.OutputParameters;
      waitHandle.Set();
    };

    workflowRuntime.WorkflowTerminated +=
      delegate( object sender, WorkflowTerminatedEventArgs e ) {

      ex = e.Exception;
      waitHandle.Set();
    };

    foreach ( object svc in services ) {
      workflowRuntime.AddService( svc );
    }

    Dictionary<string, object> parms = new Dictionary<string, object>();

    for ( int i = 0; i < inputs.Length; i += 2 ) {
      Debug.Assert( inputs[ i ] is string );

      parms.Add( (string) inputs[ i ], inputs[ i + 1 ] );
    }

    WorkflowInstance instance =
      workflowRuntime.CreateWorkflow( activityType, parms );

    instance.Start();

    waitHandle.WaitOne();

    if ( ex != null ) {
      Assert.True( false, ex.Message );
      return null;
    }
    Else {
      return outputs;
    }
  }
}
The basic state-oriented test pattern of "data in/execute code/make assertions against data" works just fine for WF with a generic test harness. The WF runtime supports auto-population of root activity property values using a generic Dictionary keyed by property name. It also allows you to examine root activity property values after activity execution has completed. Even if your activity will ultimately be nested several levels inside a workflow hierarchy and relies on data binding to initialize its property values, you can test the activity in isolation of others and simulate the full range of expected and error conditions for that activity.
WF also plays nicely with more advanced unit testing techniques such as object mocking, where the emphasis is on isolating tested code from its dependencies, as well as validation of expectations on those dependencies (for example, "When executing the code for this test, I expect the Foo() method for this dependency to be called exactly 3 times."). Here's an example:
// mock the IBanking interface...
var bankingMock = new Mock<IBanking>();

// specify that you expect AddCustomer to
// be called with the customer instance...
bankingMock.Expect( bank => bank.AddCustomer( customer ) );

// execute WF harness, configuring the mock as a service...
ExecuteActivity( typeof( AddCustomerActivity ),
  new List<object>() { bankingMock.Object }, "Customer", customer );
If you're familiar with unit testing and object mocking, you might have already guessed that WF services are perfect candidates for mocking in tests that target WF activities. WF services are dependencies by their very nature and should be factored out of such unit tests.

A Runtime within a Runtime
WF is more than a component-based extensible programming environment. It also incorporates an execution engine built on top of the CLR. This engine provides many services useful to workflow programs, such as work scheduling, automatic serialization/deserialization during idle states, and propagation of data from parent to child activities via data binding, among other things. But you must take care to play by the WF rules; naïve workflow implementations can perform poorly in this environment, or not at all.
Perhaps the most important thing to understand about the WF runtime is that it imposes no specific threading requirements on its host code. Rather, it's designed to integrate with any existing threading model, be it single-threaded or multithreaded.
Scheduling workflow execution occurs via a service configured against the WF runtime. The DefaultWorkflowSchedulerService executes workflows on the .NET ThreadPool, while the ManualWorkflowSchedulerService allows you to specify exactly which thread you want to use (useful with WF hosts that themselves dispatch work to the ThreadPool, like ASP.NET). Though rarely needed in practice, you also have the option of providing your own custom scheduling implementation.
Note that, regardless of threading model, individual workflows never execute on more than one thread at any given moment. This is a conscious design choice in the WF runtime execution model. If you need to invoke multithreaded code within a workflow, your best bet is to do it in a custom WF service.
Unlike traditional stack-based programming environments such as C# or Visual Basic, workflows execute asynchronously with respect to their calling code. That is, when a workflow completes execution, control is not automatically returned to the point of invocation of the workflow. Rather, each workflow instance becomes a separate logical fork in your program.
Managing state for workflows running in WF introduces the potential for problems without an understanding of the WF runtime. Traditional POCO (Plain Old CLR Objects) programming uses class member fields and properties for data storage. You build WF activities in the same manner, but in the case of episodic workflows configured to persist during idle states, a single logical workflow lifetime might be realized via multiple physical activity graph instantiations (as depicted in Figure 4) and across multiple applications or even multiple machines.
Figure 4 Logical vs. Physical Workflow Lifetime, for Episodic Workflows
While WF supports automatic serialization of activity graphs and state during persistence, it's important to note that it only works for a subset of data types (no arrays, and limited support for generics). Hand-authored custom serialization is possible via serialization surrogates. You can also override Activity.OnActivityExecutionContextLoad and Activity.OnActivityExecutionContextUnload to do custom processing per activity for each instantiation and teardown of a workflow object graph.
Remember that WF is a runtime unto itself. Avoid common pitfalls by developing a basic understanding of the WF threading, scheduling, and serialization concepts.

It's Not All or Nothing
It's a common anti-pattern when first adopting a new technology to apply it everywhere, sometimes to the exclusion of other options, the proverbial "when wielding a hammer, everything looks like a nail" syndrome. We've all been guilty of this at some point. And while this is certainly possible with WF, its nature as a low-level plumbing technology with many integration points and extensibility hooks allows you to apply it in many distinct problem domains.
WF works equally well in highly multithreaded environments, such as ASP.NET, or in desktop client applications built with Windows Forms or WPF. Common uses for WF in these scenarios include implementation of UI-driven logic, where you implement user interface event-handling logic in a workflow instead of imperative code; task-oriented asynchronous background work, such as fetching and processing Web search results or manipulating large files on disk, typically invoked on a separate thread; and page flow, in which workflow logic dictates UI navigation for ASP.NET or smart clients.
Figure 5 shows the design surface for a Microsoft sample PageFlow application. Each top-level activity in the workflow defines the view (the Web page or application form) to display, as well as any custom processing for each view. The allowed transitions between views are also depicted here. Contrast a clean graphical representation like this with dozens of lines of C# or Visual Basic to codify the same information and you can see where WF has obvious benefits.
Figure 5 A Workflow for Governing User Interface Navigation Steps
WF and Windows Communication Foundation (WCF) are natural technology mates in service-oriented applications. A workflow is, in essence, a service that you invoke to perform a specific function, and WCF is designed for exposing services for consumption. To make it easy on you, the .NET Framework 3.5 ships with the SendActivity and ReceiveActivity activities that make WF-to-WCF integration possible right out of the box. You use ReceiveActivity to declaratively expose WF workflows behind a WCF address/binding/contract facade. The workflow is the service implementation, and it is invoked by the WCF integration plumbing as requests are received.
You can also expose long-running workflows this way, where a single workflow instance might expose multiple service operations and go through several execute/idle sequences. WCF binding types (BasicHttpContextBinding, WsHttpContextBinding, or NetTcpContextBinding) are used to correlate messages sent from clients to specific in-flight workflow instances.
Finally, as you might guess, SendActivity allows you to configure outbound WCF client connections from your workflow to external services with relative ease.
As a .NET integration technology, WF programs can interact with any library or technology consumable by .NET code. Your existing technology investments needn't be discarded when implementing application logic in WF; to the contrary, the real value of the technology lies in enabling greater and more productive use of existing technology assets.
WF is as much an integration technology as a means unto itself. WF is being used as a foundation technology within the Microsoft products that employ a workflow product. These include SharePoint, the Dynamics products, and the upcoming release of Office Communications Server and Office Project. This common usage allows for much easier integration between the technologies and your WF application. Expect to use it as part of a greater whole.

Plug and Play
In order to maximize integration opportunities, extensibility is a fundamental tenet of WF. Key pieces of the core infrastructure are configurable—or even replaceable with custom implementations. And while WF ships with numerous built-in activities for control flow, transactional semantics, and services communication (among others), none of these activities are required for use in your WF programs, and none of them uses special features unavailable to the WF programming community at large. Your own custom activities (or those of third-party vendors) are inherently no more or less capable than anything that ships out of the box.
Action-oriented activities that map to domain-specific behaviors are obvious choices for custom implementation. A workflow used by a mortgage company to originate loans might contain custom activities for requesting flood insurance quotes, computing property taxes, and obtaining applicant credit scores, among others. These are useful and necessary, but the more interesting cases are custom composite activities where the emphasis is on customized control flow instead of realizing a specific domain function. WF ships with several standard control flow composites such as IfElseActivity and WhileActivity. But with WF, you can realize more exotic cases, such as execution of composite child activities based on some configurable priority level, or perhaps use of a dependency network to determine execution order (see Figure 6).
Figure 6 Using a Dependency Network
[Designer( typeof( DependencyNetworkActivityDesigner ) )]
[ActivityValidator( typeof( DependencyNetworkActivityValidator ) )]
public partial class DependencyNetworkActivity : SequenceActivity {

  DependencyProperty.RegisterAttached( "RunAfter",
    typeof( string ), typeof( DependencyNetworkActivity ),
    new PropertyMetadata( DependencyPropertyOptions.Metadata ) );

  public static object GetRunAfter( object depObj ) {
    return ( (DependencyObject) depObj ).GetValue( RunAfterProperty );
  }

  public static void SetRunAfter( object depObj, object value ) {
    ( (DependencyObject) depObj ).SetValue( RunAfterProperty, value );
  }

  public DependencyNetworkActivity() {
    InitializeComponent();
  }

  private IList<Activity> _orderedList = null;
  private int _currentIndex = 0;

  protected override ActivityExecutionStatus Execute( ActivityExecutionContext aec ) {
    if ( this.EnabledActivities.Count == 0 ) {
      return ActivityExecutionStatus.Closed;
    }
    else {
      _orderedList = DependencyNetworkActivityValidator.ComputeOrderedList( this );

      Debug.Assert( _orderedList != null );
      Debug.Assert( _orderedList.Count == this.EnabledActivities.Count );

      ScheduleNextChild( aec );

      return ActivityExecutionStatus.Executing;
    }
  }

  private void ScheduleNextChild( ActivityExecutionContext aec ) {
    Debug.Assert( aec != null );

    Debug.Assert( _orderedList != null );
    Debug.Assert( _currentIndex < _orderedList.Count );

    Activity child = _orderedList[ _currentIndex++ ];

    Debug.Assert( child != null );

    child.Closed += ChildDone;

    aec.ExecuteActivity( child );
  }

  private void ChildDone( object sender, ActivityExecutionStatusChangedEventArgs e ) {
    ActivityExecutionContext aec = ( sender as ActivityExecutionContext );
    e.Activity.Closed -= ChildDone;

    if ( _currentIndex < _orderedList.Count ) {
      ScheduleNextChild( aec );
    }
    else {
      aec.CloseActivity();
    }
  }
}
In addition, almost all of the various runtime services that ship with WF are extendible or replaceable. You're free to (for example) implement your own workflow persistence layer on top of another storage medium, or perhaps implement a workflow scheduler that executes against a custom thread pool implementation instead of the built-in .NET ThreadPool. Any such custom service is configured in exactly the same way as all other WF services, so the model is consistent and straightforward.
Ultimately, the goal for WF is to remain flexible in the face of evolving technologies and application architectures, instead of mandating a rigid set of requirements that might otherwise render the technology less broadly applicable.
Additional WF Resources

Workflow Tips and Tricks

Workflow Services (Windows Workflow Foundation and Windows Communication Foundation)

Build Workflows To Capture Data And Create Documents

Loading Workflow Models in WF

Domain Modeling and Program Design
One of the more difficult aspects of building software with WF is mapping elements of your problem domain to elements of a workflow. The primary issue is the control flow-centric programming model of WF in which—unlike traditional imperative, or object-oriented programming—the atomic elements of your workflow (activities) don't represent entities in your problem domain, but rather actions or processing steps. It's a subtle but important distinction, and it makes what are already non-trivial tasks (domain modeling and program design) even more challenging, especially for WF newcomers.
It's one thing to understand the mechanics of the WF programming model—activities, services, episodic execution, and the rest—but it's another thing entirely to take a functional specification or a bunch of use cases and turn that into one or more efficient, well-factored, robust workflows. What domain concepts map to an activity versus service versus an entire workflow? What data should flow through the workflow, and how? Where can work be done in parallel, and where must the workflow be designed to go idle, waiting on user input?
Truth be told, it's impossible to sufficiently answer such questions in the space of a few paragraphs. But I offer you here a few general guidelines that have served me well. As with any discussion of modeling and design, take the advice that follows as a starting point and not a complete treatment of the topic, and don't assume this is the only way to achieve WF design nirvana. It isn't.
First, start your modeling phase as you normally would with use cases, actors, and domain elements given up-front design consideration suitable for your project. Use any approach that works for you; there's no need for WF-specific accommodations here.
Next, model your actors (Customer, BankManager, and so on) and your domain elements (PurchaseOrders, InsurancePolicies, AcmeWidgets) as objects. As always, give careful consideration to the data encapsulated by each item. Expose data with properties, as necessary. Stub out actor methods, but don't implement them just yet.
At this point, you have a decision to make. Is your intended design WF-centric, such that all significant behavior will be represented as distinct activities in one or more workflows? Or do you want to use WF in a more targeted fashion, implementing key aspects in WF while keeping the rest as more traditional .NET code? Don't consider it a binary yes-or-no question; instead, give deliberate consideration to where on the spectrum between all WF and no WF you want to be. The strengths of WF—modularity, visibility, and feature set—will be more beneficial to you in some areas (complex and continuously changing business logic) than in others (user interface layout or data access layers).
Now consider the operations defined by each domain element. In traditional object-oriented programming, these will map to public methods of concrete types. In WF, such actions (say, "withdraw money from bank account" or "add item to shopping cart") usually map to individual custom activities that operate on one or more defined objects. The more you move toward WF on the continuum, the more actions you'll implement as WF activities, and the fewer you'll implement as traditional class methods.
Use cases will define an ordered set of actions performed by an actor to achieve a specific goal, as well as contingencies for error conditions. In C# or Visual Basic, this might manifest itself through use of the Strategy or Template Method patterns, resulting in methods hanging off of actor type definitions. With WF, I like to map use cases to custom composite activities consisting of domain element action activities and other activities that define conditional logic in the use case. Figure 7 shows a custom composite activity (AddCustomerWithAuditActivity) that contains no code-based logic; all steps are implemented as a series of custom activity invocations.
Figure 7 A Custom Composite Activity with No Code-Based Logic
Since your use cases are now defined as custom composite activities, you can then define larger business processes ("do month-end reconciliation" or "project next fiscal-year budget") as workflows that aggregate these custom composites. Actor behaviors can be defined as class methods that accept all necessary input and launch the appropriate workflow, passing data into the workflow as needed and harvesting results. Data can be passed via data binding or service invocation, as appropriate.
Domain modeling and code design in WF have some similarities, but also several key differences to the equivalent tasks in imperative .NET programming. Anticipate a learning curve and some refactoring opportunities as inevitable by-products of your first WF projects.

Additional Advice
I'll round out my advice with some important points that didn't otherwise merit a full section of their own:
First, minimize use of CodeActivity. It can be useful for one-off "glue" code to ensure proper data flow between activities, but otherwise should be replaced by a custom activity specific to the task at hand.
When integrating workflows with other subsystems, it is better to expose WF functionality indirectly through an interface. This promotes loose coupling and makes your workflow code easier to consume and easier to test. It also then becomes easier to vary the implementation of WF logic over time.
Don't neglect to implement validators for your custom activities. The WF activity composition model supports optional validation logic for allowed versus disallowed configuration of activities. You implement this by subclassing ActivityValidator and attributing the target activity type. See the accompanying code download for an example validator implementation.
Know that the boilerplate Visual Studio wizard-generated workflow hosting code is useful for ad hoc invocation and testing of workflows, but little else. The most interesting and useful workflows might never execute from a command line!
For maximum runtime agility, you should consider use of a dependency injection container. Check out StructureMap or Spring.NET for WF custom service configuration.
If your workflow executes business rules using PolicyActivity, consider extending the available grammar with domain-specific keywords by subclassing RuleAction. This increases expressiveness and comprehension of authored rules.
And last, but not least, remember to design your workflow hosting environment so that it is continuously available to service responses from external systems, bound for idle workflows. Your workflow may be idle, but if your workflow host is not running, the external system response may never be handled, and your idle workflow never awoken!
For a broad technology like WF, any list of best practices or key principles is liable to be incomplete, or applicable to varying degrees depending on circumstance. The guidance found here comes from my own experiences, as well as those of my colleagues and students; it is broadly, but not universally, applicable. Consider your specific problem domain and constraints when deciding how my advice might best fit (or not fit) your needs.

Josh Lane is Principal Architect for business rule engine vendor InRule Technology Inc. and a .NET instructor for DevelopMentor. He lives in Atlanta, Ga.


Visual Studio OBA Tools
Simplify OBA Development With Interop API Extensions
Andrew Whitechapel, Phillip Hoff, and Vladimir Morozov
Code download available from the MSDN Code Gallery
Browse the Code Online

This article discusses:
  • VSTO Power Tools and Office interop APIs
  • Using the Outlook, Word, and Excel extensions
  • Extending the extensions
  • Using the extensions in Visual Basic
This article uses the following technologies:
Office, VSTO Power Tools, LINQ
Contents
Installing the Extensions
Using the Extensions
Word Extensions
Excel Extensions
Outlook Extensions
Extension Internals
Extending the Extensions
Using the Extensions in Visual Basic
Wrapping Up
Developing applications in C# against the Microsoft Office client application object models can be awkward because these object models use features—such as optional parameters and parameterized properties—that are handled internally in Visual Basic for Applications (VBA) and Visual Basic, but not in C#. Furthermore, Office makes extensive use of loosely typed parameters and return values, which are anathema to a strongly typed system like the Microsoft .NET Framework.
The Microsoft Visual Studio Tools for the Office System (VSTO) Power Tools include a set of Office interop API extension libraries that provide C# developers with support for these features. Developers using Visual Basic will also find the libraries useful, specifically for querying Microsoft Office Outlook items in a type-safe manner. Using these extensions can help to make your code more robust, as well as increase developer productivity.
To introduce you to these Visual Studio Power Tools API extensions, we'll walk through the development of an application that uses the Power Tools to automate Office applications. We'll also look at the .NET Framework 3.0 features being employed by the extensions, as well as extending the extensions for more customized use. We'll even take a look at using the extensions with your Visual Basic code.

Installing the Extensions
The VSTO Power Tools are documented on MSDN and can be obtained from the VSTO System Power Tools v1.0.0.0 download page. The Office Interop API extensions are all in the download file named VSTO_PTExtLibs.exe. When you execute this self-extracting file, it will install the documentation for the extensions under %PROGRAMFILES%\Microsoft VSTO Power Tools 1.0\Office Interop Extensions. The extension assemblies themselves are installed in the Global Assembly Cache (GAC).
The first release of these extensions includes support for Microsoft Office Excel, InfoPath, Project, Outlook, PowerPoint, Visio, and Word in both Office 2003 and the 2007 Office system. The extensions all have a consistent naming convention based on the following pattern: Microsoft.Office.Interop.<application>.Extensions.dll. For example, the extensions for Excel are defined within the namespace Microsoft.Office.Interop.Excel.Extensions and built into an assembly named Microsoft.Office.Interop.Excel.Extensions.dll.
Note that the extensions are essentially very thin wrappers to the Office primary interop assemblies (PIAs), and you therefore need to have the appropriate Office PIAs installed in order for the extensions to work. The PIAs are installed by default with the 2007 Office system, and optionally with Office 2003. Alternatively, you can use the PIA redistributables. Although the extensions are produced by the same team that produces VSTO, you can use the extensions in any solution that uses Office regardless of whether you are building a VSTO solution.

Using the Extensions
To explore the extensions, we'll build a solution that invokes functionality in three Office applications: Outlook, Excel, and Word. Our application is not a VSTO solution; rather it is a simple Windows Presentation Foundation (WPF) application that automates the Office applications in turn. First, it fetches e-mail data from Outlook and performs some simple calculations on that data; then it feeds the results into Excel to produce a chart; and finally it pastes the chart into a Word document and saves it.
These three tasks have been separated out so that they can be controlled individually by the user via buttons in the main window (see Figure 1). This is obviously an artificial mechanism, but it serves an important teaching purpose: to separate each operation so that it can be examined in isolation. Also, any of the tasks can be performed using either the raw PIA objects or the extensions library, in any permutation.
Figure 1 Sample Application Main Window
The final output from the application is a Word document in HTML format that contains an Excel chart generated from Outlook data. The Outlook dataset of interest is the size of a selected set of e-mail items, as well as the date and time they were sent and received. The chart plots the size of each item against the elapsed time between sending and receiving it. This document is shown in Figure 2.
Figure 2 HTML Document Produced by the Application
As it happens, the sequence of operations in this solution starts with the most complex use of the extensions and ends with the simplest use of the extensions. For this reason, we'll examine the major steps in reverse order, before putting everything together in the correct order. Figure 3 summarizes the solution requirements and the features of the extensions used to meet each requirement.
Figure 3 Solution Features Mapped to Extension Features
Solution Feature | Office Application | Problem Addressed | Extension Feature
Fetch selected data from e-mail Inbox. | Outlook | Build a filtered query without using an error-prone DASL query string. | LINQ-to-DASL.
Use e-mail data to create a chart. | Excel | Specify Excel cell ranges without using the parameterized get_Range property. Find all negative cell values and set them to zero, in a performant manner. | Parameterized properties.Collections exposed via parameterized properties. LINQ-to-Objects.
Paste the chart into a document and save it. | Word | Invoke Office functionality using strongly typed parameters. Invoke Office functionality without having to pass optional parameters. Use Word features without having to pass parameters explicitly by reference. | Strongly typed method overloading. Strongly typed optional parameters. Nullable types. Object initializers.

Word Extensions
In our example solution, the operations we perform using Word are simple: insert some text and set its style, then move to the end of the document and paste in an Excel chart from the Clipboard. One of the awkward features of the Office object models, which is particularly prevalent in Word, is the use of reference parameters. For example, the Document.Range method in Word takes two parameters: the start position and the end position of the range you want to fetch. In C#, both these parameters must be passed as explicit variables by reference.
The following code shows how to get the range that corresponds to the start of the document, using the raw PIA approach:
object startPosition = 0;
object endPosition = 0;
Word.Range r = (Word.Range)doc.Range(ref startPosition,
  ref endPosition);
One of the simplest features provided by the extensions is an overload of the Range method that takes two simple parameters which do not need to be passed by reference. Using the extensions, the call to get the start of the document is far simpler. In this example, the extensions avoid the need to pass the parameters by reference—and they also enforce the requirement to pass strongly typed parameters. The start and end position will always be integers, but the Word object model only specifies that they are loosely typed objects. The extensions impose strong typing on the method signature, which therefore brings the benefits of design-time IntelliSense and compile-time type-checking. It also reduces the code from three lines to one:
Word.Range r = doc.Range(0, 0);
Another minor inconvenience in the Office object models is the use of parameterized properties. It is common in Office to expose properties that take parameters. Bear in mind that all properties exposed from COM servers are really methods. Methods, of course, can take multiple parameters.
In C#, although methods can take parameters, properties cannot. For this reason, the Office PIAs expose parameterized properties in C# as a pair of get_ and set_ methods. For example, the PIA code to set the style property requires you to pass an object parameter (and by reference again). The following raw PIA code continues on from the previous snippet: here we insert some text after the current Range and set its style:
r.InsertAfter("Email Send Time vs Size");
r.InsertParagraphAfter();
r = (Word.Range)doc.Paragraphs[1].Range;
object style = Word.WdBuiltinStyle.wdStyleTitle;
r.set_Style(ref style);
To do the same thing using the extensions, the code is far more intuitive. As before, it uses strongly typed parameters and a simple method call, without the need for an interim variable and without the set_ or get_ prefix:
r.InsertAfter("Email Send Time vs Size");
r.InsertParagraphAfter();
r = (Word.Range)doc.Paragraphs[1].Range;
r.Style(Word.WdBuiltinStyle.wdStyleTitle);
Many methods in the Office object models take optional parameters. In Visual Basic, you do not need to supply any parameters that are optional, because the Visual Basic runtime provides a generic parameter on your behalf with a value that tells the Office object to use the default value for the missing parameter. C# does not perform this operation for you. This is another area where the extensions take care of the underlying operation and provide C# developers with an experience that is very similar to the experience of using Visual Basic.
For example, to move to the end of the document and paste something in from the Clipboard, code using the raw PIAs must pass any missing parameters explicitly as System.Type.Missing:
object gotoItem = Word.WdGoToItem.wdGoToLine;
object gotoDirection = Word.WdGoToDirection.wdGoToLast;
object missing = Type.Missing;
r = r.GoTo(ref gotoItem, ref gotoDirection,
  ref missing, ref missing);
r.Paste();
Using the extensions, on the other hand, we can use the object-initializers feature introduced in C# 3.0 to provide strongly typed named parameters for the values we want to specify and to completely omit any parameters for which we want to use default values. The use of named parameters is another feature that brings a Visual Basic experience to C# developers:
r = r.GoTo(
  new DocumentGoToArgs {
    What = Word.WdGoToItem.wdGoToLine,
    Which = Word.WdGoToDirection.wdGoToLast
  });
r.Paste();
The usefulness of the optional parameter feature in the extensions becomes more obvious with a method like Document.SaveAs in Word. Using PIA code, you need to supply all 16 parameters. In our example, we want to save the document as an HTML file, which requires us to specify the file name and the enum value for HTML file format in Word, both as loosely typed objects. We don't care about the remaining 14 parameters, but we have to pass them anyway, and we have to pass all parameters by reference:
object saveFormat = Word.WdSaveFormat.wdFormatHTML;
doc.SaveAs(ref fileName, ref saveFormat,
  ref missing, ref missing, ref missing, ref missing,
  ref missing, ref missing, ref missing, ref missing,
  ref missing, ref missing, ref missing, ref missing,
  ref missing, ref missing);
The extension method again uses strong typing and allows us to completely omit optional parameters:
doc.SaveAs(fileName, Word.WdSaveFormat.wdFormatHTML);

Excel Extensions
In the example solution, we use Excel to put a chart into the Clipboard so that we can subsequently paste it into the Word document. First, we get hold of the first worksheet in the workbook. It is common in the Office object models to return specific objects typed as generic objects. For example, the Sheets collection may contain both Worksheet objects and Chart objects, so when you specify a particular item in the collection, the Excel PIA returns it to you as a generic object, which you then need to cast to the specific Sheet type you are expecting:
Excel.Worksheet sheet = (Excel.Worksheet)book.Sheets[1];
A general theme in the extensions is the use of strong typing throughout, and this extends to the type of object returned from heterogeneous collections, without forcing the developer to make a speculative cast on the returned object. Apart from simplifying the developer's code, this also moves the type-checking from runtime to compile time, which improves the robustness of the application and reduces the cost of the test. With the extensions, you can index into the collection using either an integer or the name of the sheet:
//Excel.Worksheet sheet =
//  book.Sheets.Item<Excel.Worksheet>("Sheet1");
Excel.Worksheet sheet =
  book.Sheets.Item<Excel.Worksheet>(1);
In the example, we want to get specific cell ranges and insert some data. We will have two columns of data: one for the elapsed time in milliseconds between sending and receiving an e-mail message and one for the size of the e-mail in bytes. After inserting the column headings, we then insert the e-mail data into a grid of cells.
Using the PIA approach, we again have to use parameterized properties for Range and Offset. Note that the emailData variable is an array of strings, which we will populate in the complete solution from e-mail data in Outlook:
sheet.get_Range("A1", missing).Value2 = "mSec";
sheet.get_Range("B1", missing).Value2 = "Bytes";
Excel.Range firstCell = sheet.get_Range("A2", missing);
Excel.Range lastCell;

for (int r = 0; r < this.emailData.Length; r++) {
  String[] cellValues = this.emailData[r].Split(',');
  for (int c = 0; c < cellValues.Length; c++) {
    lastCell = (Excel.Range)firstCell.get_Offset(r, c);
    lastCell.Value2 = cellValues[c];
  }
}
Using the extensions, we can use the strongly typed Range method. Note that the extensions library covers most of the Office object model, but not all of it. So, in this example, we can use the extension for the Range property, but we have to use the PIA Offset property—unless we extend the extensions (more on that later):
sheet.Range("A1").Value2 = "mSec";
sheet.Range("B1").Value2 = "Bytes";
firstCell = sheet.Range("A2");
Next, we want to find any negative values in the sheet and set them to zero. We can do this in a simple way, using the PIA:
Excel.Range chartData = sheet.get_Range("A2", lastCell);
foreach (Excel.Range cell in chartData) {
  if (((double)cell.Value2) < 0) {
    cell.Value2 = 0;
  }
}
Alternatively, we can do this using a SQL-like query, using the extensions. This uses the LINQ-to-Objects functionality in the extensions library. Note that LINQ is optimized for performance and uses a delayed execution model where the query is not actually executed until you begin to enumerate the resultset. In some scenarios, this can dramatically improve the performance of your application. That said, it is sometimes more efficient to force execution of the query by calling ToList or ToArray and then operate on the result—for instance if you intend to iterate over the resultset several times. In this example, the search condition is very simple (cell value less than zero), but you can see how a SQL-like query would become more useful as the complexity of the search conditions increases:
Excel.Range chartData = sheet.Range("A2", lastCell);
var cells = (from c in chartData.Items()
             where ((double)c.Value2) < 0
             select c).ToArray();
foreach (Excel.Range cell in cells) {
  cell.Value2 = 0;
}
Next, we create a simple line chart from the mSec and Bytes columns and copy it to the Clipboard. With the PIA approach, we have to use loose typing, explicit optional parameters, and parameterized properties:
Excel.Shape chart = sheet.Shapes.AddChart(
  Excel.XlChartType.xlLine,
  missing, missing, missing, missing);
chart.Select(missing);
excel.ActiveChart.SetSourceData(
  sheet.get_Range("A1", lastCell), missing);
excel.ActiveChart.ChartArea.Copy();
Out of the box, the extensions only help us here with the parameterized Range property—they don't provide overloads for Shapes.AddChart or Chart.SetSourceData. This is another case where we might want to extend the extensions.

Outlook Extensions
You can search, filter, and query Outlook items using the DAV Searching and Locating (DASL) query language. The language is similar to SQL and is quite powerful. However, it has one serious drawback: it requires you to compose a string and pass this as a filter to the Outlook Restrict or Find methods, and because the query is a string, there can be no design-time or compile-time support. This means that you cannot be sure that the string really represents the query you intended until you execute it at run time.
You cannot even be sure that the string is a correctly formed query. DASL queries use property URNs, which are non-intuitive, typically include obscure hex values, are not well-documented, and are easily mistyped. If your query uses string patterns, date, or time conditions, you must observe wildcard and formatting rules, which are even more easily mistyped. Clearly, these query strings can rapidly become complex and difficult both to construct and to parse, slowing down productivity, increasing test cost, and making code maintenance difficult.
In our example, we set up a DASL filter, shown in Figure 4, to find all Inbox items that are e-mail messages (that is, their MessageClass is based on IPM.Note), where the subject starts with "RE:" and were received in the last 30 days. We use Windows Desktop Search (WDS) if it is available, because this provides faster searching. WDS is installed by default on Windows Vista, but is optional on Windows XP and Windows Server 2003. Note that DASL queries always perform date-time comparisons in Coordinated Universal Time (officially abbreviated as UTC), so if you use a date literal in a comparison string, you must use its UTC value for the comparison.
Figure 4 DASL Filter to Find E-Mail Messages
string filter;
if (outlook.Session.DefaultStore.IsInstantSearchEnabled) {
  filter =
    @"@SQL=(((""http://schemas.microsoft.com/mapi/proptag/0x001a001e"""
    + "CI_STARTSWITH 'IPM.Note')"
    + @" AND (""urn:schemas:httpmail:subject"" CI_STARTSWITH 'RE:'))"
    + @" AND (""urn:schemas:httpmail:datereceived"" >= '"
    + (DateTime.Now.ToUniversalTime()
    - new TimeSpan(30, 0, 0, 0)).ToString("g") + @"'))";
}
else {
  filter =
    @"@SQL=(((""http://schemas.microsoft.com/mapi/proptag/0x001a001e"""
    + "LIKE 'IPM.Note%')"
    + @" AND (""urn:schemas:httpmail:subject"" LIKE 'RE:%'))"
    + @" AND (""urn:schemas:httpmail:datereceived"" >= '"
    + (DateTime.Now.ToUniversalTime()
    - new TimeSpan(30, 0, 0, 0)).ToString("g") + @"'))";
}

Outlook.Items folderItems = folder.Items;
Outlook.Items filteredItems = folderItems.Restrict(filter);
StringBuilder builder = new StringBuilder();

foreach (Outlook.MailItem item in filteredItems) {
  builder.AppendLine(String.Format("{0},{1},{2}",
    item.SentOn, item.ReceivedTime, item.Size));
}

CalculateElapsedTime(builder.ToString());
Once we have retrieved all the items of interest, we will compose a string for each item that consists of the time taken to send the e-mail and the size of the e-mail (using a custom method, CalculateElapsedTime, which simply performs string and DateTime operations, and does not use the Office object models—this method is not listed here, but you can find it in the accompanying source-code download).
Using the extensions, we can dispense with the string query and instead perform the same search using a LINQ query, which will bring all the benefits of design-time IntelliSense and compile-time type and syntax-checking—very different from the DASL string query experience. Note one limitation: the extensions do not expose the Outlook item's Size property, so we use the Length of the Body property instead (this is another case where extending the extensions would make sense):
var filteredItems = (
  from item in folder.Items.AsQueryable<Mail>()
  where item.MessageClass.StartsWith("IPM.Note")
  && item.Subject.StartsWith("RE:")
  && item.DateReceived >=
  DateTime.Now.ToUniversalTime() - new TimeSpan(30, 0, 0, 0)
  select item).ToList();
StringBuilder builder = new StringBuilder();

foreach (Mail item in filteredItems) {
  builder.AppendLine(String.Format("{0},{1},{2}",
    item.Date, item.DateReceived, item.Body.Length));
}

CalculateElapsedTime(builder.ToString());

Extension Internals
Now that we've seen how much simpler and more error-free the developer's coding experience is with the extensions, let's look at how this simplicity and robustness are achieved under the covers. Consider first the Document.SaveAs method in Word. This takes 16 optional parameters, and in most cases you only need to specify a very small number of these parameters (or, in some cases, none).
To write an extension method, you simply write a static method that takes the object you want to extend as its first (or only) parameter. So, to provide a strongly typed SaveAs method for the Document object, you can write a method called SaveAs that takes a Document object as its first parameter. You can also provide overloads for your extension methods so that each overload is used to simplify the developer's coding experience, but the real functionality is ultimately invoked on the underlying PIA Document object itself.
This is how the extensions are implemented, as shown in the code listings for the SaveAs method in Figure 5. At design-time in Visual Studio, the extension methods appear to the developer to be methods of the object that they extend, and are available through IntelliSense and autocomplete. IntelliSense will flag extension methods with "(extension)," as shown in Figure 6.
Figure 5 Overloaded SaveAs Methods
public static void SaveAs(this Document doc) {
  SaveAs(doc, (string) null);
}

public static void SaveAs(this Document doc, string fileName) {
  SaveAs(doc, fileName, null);
}

public static void SaveAs(this Document doc, string fileName,
   WdSaveFormat? fileFormat) {
  SaveAs(doc, new DocumentSaveAsArgs {
    FileName = fileName, FileFormat = fileFormat });
}

public static void SaveAs(this Document doc, DocumentSaveAsArgs args) {
  doc.SaveAs(
    ref args.FileNameInternal,
    ref args.FileFormatInternal,
    ref args.LockCommentsInternal,
    ref args.PasswordInternal,
    ref args.AddToRecentFilesInternal,
    ref args.WritePasswordInternal,
    ref args.ReadOnlyRecommendedInternal,
    ref args.EmbedTrueTypeFontsInternal,
    ref args.SaveNativePictureFormatInternal,
    ref args.SaveFormsDataInternal,
    ref args.SaveAsAOCELetterInternal,
    ref args.EncodingInternal,
    ref args.InsertLineBreaksInternal,
    ref args.AllowSubstitutionsInternal,
    ref args.LineEndingInternal,
    ref args.AddBiDiMarksInternal);
}
Figure 6 IntelliSense for an Extension Method in Visual Studio
You can also take advantage of the nullable types feature of C#, where a strongly typed parameter can be passed as null (note that this is not the same as passing a loosely typed object with the System.Type.Missing value). In the third overload SaveAs method shown in Figure 5, the WdSaveFormat parameter is specified as nullable (declared using the ? syntax), which allows us to invoke this method and pass null for this parameter. The real work of saving the document is ultimately done in the overload that takes a DocumentSaveAsArgs object. As we have seen, this uses object initializers, to allow the developer to provide values for the interesting parameters by name and to omit any that can be defaulted.
The DocumentSaveAsArgs class is derived from the SaveAsArgs extensions class. The argument values themselves are stored within internal fields of the class, such as FileNameInternal, and these are used directly by the extension methods. The developer uses the public properties, such as FileName, rather than the internal fields. This design arises because of the requirement in most methods in the Word object model to pass parameters by reference. The argument classes in the extensions library for the other Office applications do not use this extra layer:
public abstract class SaveAsArgs : ExtensionArgs {
  internal object FileNameInternal = Type.Missing;

  public string FileName {
    get {
      return ToReference<string>(FileNameInternal);
    }
    set {
      FileNameInternal = ToObject(value);
    }
  }
  // code omitted for brevity.
}
SaveAsArgs is, in turn, derived from ExtensionArgs, which includes code to convert between real types and their boxed equivalents and includes logic for handling System.Type.Missing:
protected T ToReference<T>(object obj) where T : class {
  return (obj != null && obj != Type.Missing) ? (T) obj : null;
}
Parameterized properties are also handled by overloaded extension methods in the extensions library—as before, these use the underlying PIA objects for the ultimate functionality. For example, the Excel.Range extensions provide three overloads that use the get_Range property method under the covers:
public static Range Range(this Worksheet worksheet, string name) {
  return worksheet.get_Range(name, Type.Missing);
}

public static Range Range(this Worksheet worksheet,
  string name1, string name2) {
  return worksheet.get_Range(name1, name2);
}

public static Range Range(this Worksheet worksheet, string range1, Range range2) {
  return worksheet.get_Range(range1, range2);
}
To support LINQ-to-DASL queries in Outlook, the extensions library provides an abstract base class to represent the data source for the query: OutlookItemSource. This class implements the IQueryable<T> and IQueryProvider interfaces used by LINQ. The extensions also provide derived classes ItemsSource and ApplicationSource for performing queries on the Items collection of a single folder and queries across multiple folders, respectively:
public abstract class OutlookItemSource<T> : IQueryProvider,    IQueryable<T> {
  public IQueryable<TElement> CreateQuery<TElement>    (Expression expression) {
    MethodCallExpression call = expression as MethodCallExpression;
    switch (call.Method.Name) {
      case "Where": return new
        OutlookItemSourceWhereHandler<TElement>(
          this, expression, _builder, _search);
      case "Select": return new
        OutlookItemSourceSelectHandler<TElement>(
          this, expression);
    }
  }
  // code omitted for brevity.
}
The OutlookItemSource class uses other internal extensions classes to build a DASL query string from the LINQ query. For example, the ParseMethodCallExpression method converts LINQ expressions such as Contains to DASL query sub-strings such as (LIKE '%'):
private static void ParseMethodCallExpression(MethodCallExpression call) {
  if (call.Method.DeclaringType == typeof(string)) {
    string format = null;
    switch (call.Method.Name) {
      case "Contains": format = @"({0} LIKE '%{1}%')"; break;
      case "StartsWith": format = @"({0} LIKE '{1}%')"; break;
      case "EndsWith": format = @"({0} LIKE '%{1}')"; break;
    }
    // code omitted for brevity.
  }
}
The OutlookItemProperty attribute provides a mapping between DASL properties and properties on the Outlook item interfaces. This attribute is used on the public properties exposed from the OutlookItem class, and this is how the extensions allow the developer to avoid the use of the obscure URN/hex tag strings to specify target properties:
public class OutlookItem : IOutlookItemWrapper {
  [OutlookItemProperty(
    "http://schemas.microsoft.com/mapi/proptag/0x001a001e")]
  public virtual string MessageClass
    { get { return GetProperty("MessageClass") as string; } }
  // code omitted for brevity.
}

Extending the Extensions
Knowing how the extensions are implemented internally, you can easily extend them. Some objects and methods in the Office object model are not covered by the extensions, and in some cases you might even want to provide user-defined extensions.
From our example application, the most obvious cases are the Excel methods Range.get_Offset, Shapes.AddChart, Shape.Select and Chart.SetSourceData. For all of these, it would be trivial to provide extension methods. Recall that these simply need to be static methods that take as their first parameter an object of the type which you want to extend as shown in Figure 7.
Figure 7 Extending the Excel Methods
public static class RangeExtensions {
  public static Excel.Range Offset(
    this Excel.Range cells, int rowOffset, int columnOffset) {
    return cells.get_Offset(rowOffset, columnOffset) as Excel.Range;
  }
}

public static class ShapesExtensions {
  public static Excel.Shape AddChart(
    this Excel.Shapes shapes, Excel.XlChartType chartType) {
    return shapes.AddChart(chartType,
      Type.Missing, Type.Missing, Type.Missing, Type.Missing);
  }
}

public static class ShapeExtensions {
  public static void Select(this Excel.Shape shape) {
    shape.Select(Type.Missing);
  }
}

public static class ChartExtensions {
  public static void SetSourceData(this Excel.Chart chart,
    Excel.Range range) {
    chart.SetSourceData(range, Type.Missing);
  }
}
Note that, taking the Shapes.AddChart method as an example, this extension method works by virtue of having a different number of arguments from the PIA AddChart method (or any of the extensions overloads). If you tried to provide an extension method that had the same arguments, but was simply strongly typed, the compiler would never call it (because it favors native methods, regardless of the strength of the argument types). We can do this with Word because of its use of by-reference arguments; by providing by-value extensions of Word methods, the compiler is always able to differentiate between the two.
The extensions library for Excel actually includes a number of methods that are unreachable by the compiler in this way. It might seem strange to provide methods that the compiler will not use, but they will still show up in IntelliSense and will therefore help the user identify the real types of the arguments. Also, the developer could use these methods with normal static method syntax if desired. With these custom extensions, we can simplify the application code:
//lastCell = (Excel.Range)firstCell.get_Offset(r, c);
lastCell = (Excel.Range)firstCell.Offset(r, c);

//chart = sheet.Shapes.AddChart(
//    Excel.XlChartType.xlLine, missing, missing, missing, missing);
chart = sheet.Shapes.AddChart(Excel.XlChartType.xlLine);
//chart.Select(missing);
chart.Select();
//excel.ActiveChart.SetSourceData(sheet.Range("A1", lastCell), missing);
excel.ActiveChart.SetSourceData(sheet.Range("A1", lastCell));
Apart from simply allowing the developer to provide additional extension methods, the extensions library itself supports a degree of extensibility. For example, the LINQ-to-DASL features support pluggable logging (see Figure 8). The ItemsSource class provides a Log property that you can set to any object derived from System.IO.TextWriter. In the example, we can write a simple class that outputs to the debug window. This will then be used internally when the extensions code has completed building the DASL query string from the LINQ query—and this string will be output to the debug window.
Figure 8 Logging with LINQ-to-DASL
// Provide a logging class for our LINQ-to-DASL queries.
internal class DebuggerWriter : TextWriter {
  public override Encoding Encoding {
    get { throw new NotImplementedException(); }
  }

  public override void WriteLine(string value) {
    Debug.WriteLine(value);
  }
}

// Subclass the extensions Mail class so that we can expose a Size
// property that maps to the real Outlook size property.
internal class MailEx : Mail {
  [OutlookItemProperty("http://schemas.microsoft.com/mapi/proptag/0x0E080003")]
  public int Size { get { return Item.Size; } }
}

// Create the ItemsSource manually (this is
// what Items.AsQueryable() does implicitly).
var source = new ItemsSource<MailEx>(folder.Items);

// Set the Log property of the ItemsSource to a TextWriter.
// It will be given the DASL query string immediately before
// the query is executed in Outlook.
source.Log = new DebuggerWriter();

//var filteredItems = (from item in folder.Items.AsQueryable<Mail>()
var filteredItems = (from item in source
                     where item.MessageClass.StartsWith("IPM.Note")
                     && item.Subject.StartsWith("RE:")
                     && item.DateReceived >=
                     DateTime.Now.ToUniversalTime() -                        new TimeSpan(30, 0, 0, 0)
                     select item).ToList();

//foreach (Mail item in filteredItems)
foreach (MailEx item in filteredItems) {
  builder.AppendLine(String.Format("{0},{1},{2}",
    //item.Date, item.DateReceived, item.Body.Length));
    item.Date, item.DateReceived, item.Size));
}
At the same time, we can subclass the extensions Mail class to expose a public property for the underlying Outlook Size property, using the OutlookItemProperty attribute to establish the mapping. Recall that we previously had to use Body.Length as a workaround because Size is not exposed by the extensions by default.
With these additional extension enhancements in place, our example application is complete. We use LINQ-to-DASL to fetch filtered Outlook mail item data, feed that data into Excel to produce a chart, and copy that chart into a Word document, which we finally save as an HTML file.
Note that Outlook 2007 introduced the Table object, which is a performance-optimized mechanism for enumerating folder items. It allows you to specify properties as table columns, and to filter and sort items as rows in the table. From Outlook 2007 onwards, you should use the Table mechanism in preference to the traditional method of working with collections of items.
To get a table, you use the same filter string that you would use with the Restrict or Find methods, but you use this with the GetTable method instead. From the code shown in Figure 9, it should be clear that there would be little advantage in using the existing version of the LINQ-to-DASL extensions with an Outlook Table, because you still need to use a query string to get a Table, and Table filtering is done by specifying Table Columns. The LINQ-to-DASL approach in the extensions library can be used for either Outlook 2003 or Outlook 2007, but if you are targeting Outlook 2007 only, the Table approach is likely to offer better performance, especially for large data sets. Example code for this approach is in Figure 9.
Figure 9 Using Table in Outlook 2007
//Outlook.Items folderItems = folder.Items;
//Outlook.Items filteredItems = folderItems.Restrict(filter);
//foreach (Outlook.MailItem item in filteredItems) {
//  builder.AppendLine(String.Format("{0},{1},{2}",
//    item.SentOn, item.ReceivedTime, item.Size));
//}

// Get a table of mail items, remove the default column set,
// and add the specific columns we're interested in instead.
table = folder.GetTable(filter, Outlook.OlTableContents.olUserItems);
table.Columns.RemoveAll();
table.Columns.Add("SentOn");
table.Columns.Add("ReceivedTime");
table.Columns.Add("Size");

// Iterate the table rows.
while (!table.EndOfTable) {
  Outlook.Row nextRow = table.GetNextRow();
  builder.AppendLine(String.Format("{0},{1},{2}",
    nextRow["SentOn"], nextRow["ReceivedTime"], nextRow["Size"]));
}
VSTO Resources
To learn even more about using VSTO to build your own Office-based applications, see these articles from the archives of MSDN Magazine:

VSTO 3.0: Developing Office Business Apps with Visual Studio 2008 by Steve Fox

VSTO: Build Office-Based Solutions Using WPF, WCF, and LINQ by Andrew Whitechapel

Office Apps: Extend Your VBA Code with VSTO by Paul Stubbs and Kathleen McGrath

Using the Extensions in Visual Basic
The main benefit of the extensions is to provide C# developers with a coding experience that is similar to Visual Basic. That said, some of the extensions—in particular, the LINQ-to-DASL extensions—also bring benefits to developers working in Visual Basic (or indeed any other managed language). For example, it is beneficial in any language to use a strongly typed LINQ query instead of a string-based DASL query. Figure 10 shows a Visual Basic version of the Outlook query operation.
Figure 10 Outlook Query in Visual Basic
Dim source As New ItemsSource(Of MailEx)(folder.Items)
Dim builder As New StringBuilder()

Dim filteredItems = ( _
  From item In source _
  Where item.MessageClass.StartsWith("IPM.Note") _
  AndAlso item.Subject.StartsWith("RE:") _
  AndAlso item.DateReceived >= _
  (DateTime.Now.ToUniversalTime() - New TimeSpan(30, 0, 0, 0)) _
  Select item).ToList()

For Each filteredItem In filteredItems
  builder.AppendLine(String.Format("{0},{1},{2}", _
  filteredItem.Date, filteredItem.DateReceived, filteredItem.Size))
Next
MessageBox.Show(builder.ToString())
Note one of the differences in the way C# and Visual Basic generate expression trees. C# uses the base type—OutlookItem, in our example—which defines a mapped property for MessageClass. On the other hand, Visual Basic uses the most derived type—Mail—which defines an override for MessageClass that is not attributed with a property mapping. For this reason, we have to define another override of MessageClass in our MailEx class (see Figure 11) that is attributed with a property mapping (this is another possible place for improvement in future releases).
Figure 11 MailEx for Visual Basic
Friend Class MailEx
  Inherits Mail

  <OutlookItemProperty(     "http://schemas.microsoft.com/mapi/proptag/0x0E080003")> _
  Public ReadOnly Property Size() As Integer
    Get
      Return MyBase.Item.Size
    End Get
  End Property

  <OutlookItemProperty(     "http://schemas.microsoft.com/mapi/proptag/0x001a001e")> _
  Public Overrides ReadOnly Property MessageClass() As String
    Get
      Return MyBase.Item.MessageClass
    End Get
  End Property
End Class

Wrapping Up
In this article, we showed how C# developers can use the Office interop API extensions to streamline Office development. The extensions provide a thin, strongly typed layer over the loosely typed Office object models, which makes your code less error-prone and more robust, and can significantly reduce the cost of testing and maintenance. We also showed how to extend the extensions for custom scenarios and how to use them in Visual Basic.

Andrew Whitechapel is a Program Manager in the Visual Studio Business Applications team, responsible for architecting new Office development features in Visual Studio.

Phillip Hoff is a Software Development Engineer in the Visual Studio Business Applications team and is one of the original authors of the Office Interop API Extensions library. He is currently working on SharePoint tool integration with Visual Studio.

Vladimir Morozov is a developer in the Visual Studio Business Applications team. He worked together with Phillip Hoff on envisioning and creation of the Office Interop API Extensions library. He is currently working on SharePoint tooling for Visual Studio.


SOA Data Access
Flexible Data Access With LINQ To SQL And The Entity Framework
Anthony Sneed
Code download available from the MSDN Code Gallery
Browse the Code Online

This article discusses:
  • LINQ to SQL
  • ADO.NET Entity Framework, service-oriented architecture
  • Data transfer objects
  • Persistence, updates, and change tracking
This article uses the following technologies:
LINQ to SQL, ADO.NET Entity Framework, and SOA
Contents
Creating the Data Access Layer
Persisting Individual Objects
Tracking Changes across Service Boundaries
Configuring the Client
Moving Forward
With the releases of LINQ to SQL and the ADO.NET Entity Framework, developers now have two products from Microsoft designed to reduce the impedance mismatch that exists between the worlds of relational data and object-oriented programming. Each of these eliminates much of the plumbing code you would otherwise have to write to perform object persistence. However, incorporating these object relational mapping (ORM) technologies into a service-oriented application architecture presents a whole new set of challenges for the application developer.
For instance, how do you create a data access layer (DAL) that abstracts object persistence from other parts of the application, so that you can swap out one ORM provider for another should the need arise? How do you track changes to entities without requiring LINQ to SQL or the Entity Framework on the client? How do you perform inserts, updates, and deletes for multiple entities in a single transaction with just one trip to the service?
In this article, I will provide you with a roadmap and make some suggestions for how you can address these issues. I'll start by creating a DAL for processing orders from the Northwind sample database. The DAL will rely on an interface with two implementations, one using LINQ to SQL, and the other using the Entity Framework (see Figure 1). Both LINQ to SQL and the Entity Framework have tools that generate entities based on a database schema, but rather than using these entities, the DAL will only expose data transfer objects (DTOs) that are basically ignorant when it comes to their own persistence.
Figure 1 Service-Oriented Architecture for an Order Processing App
LINQ to SQL and the Entity Framework each have different methods for persisting disconnected entities, depending on whether you are creating a new entity or updating an existing one. Therefore, in order to invoke the appropriate API, you need to know the state of the entity beforehand. If, for example, you have an UpdateOrder method that accepts an order with order details, you could add three parameters to the UpdateOrder method accepting created, updated, and deleted order details. However, this tends to pollute the method signature with plumbing details.
Alternatively, you could pass change state information out-of-band, for instance, in a SOAP or HTTP header, but this could couple change tracking to the communications protocol. Another approach, and the one I am advocating here, is to make change state part of the data contract for each entity:
[DataContract]
public enum TrackingInfo
{
    [EnumMember]
    Unchanged,
    [EnumMember]
    Created,
    [EnumMember]
    Updated,
    [EnumMember]
    Deleted
}
In addition to other properties, each DTO would have a TrackingState property using the TrackingInfo enum. All that's required is for both sides to agree on this data contract.
Even though I've added a TrackingState property to each DTO, I must still rely on the client, which has no knowledge of LINQ to SQL or the Entity Framework, to track which objects have been created, updated or deleted and to set the TrackingState property to the corresponding value (see Figure 1). I've crafted a generic change-tracking collection to perform this task. There are two types of the collection, one that extends ObservableCollection<T> for Windows Presentation Foundation (WPF) applications, and another that extends BindingList<T> for use with Windows Forms applications. Each caches deleted items before removing them from the collection and has a GetChanges method which returns only items that have been inserted, updated, or deleted.

Creating the Data Access Layer
The DAL helps insulate other parts of the application from the details of object persistence. For this reason, it is important that objects exposed through the DAL exclude remnants of any specific data access technology. Both LINQ to SQL and the Entity Framework allow you to manufacture entities using both command-line tools and Visual Studio designers; however, these code-generated entities possess some artifacts that betray their origins.
LINQ to SQL, for example, represents data relations using the EntityRef<T> and EntitySet<T> collection types in order to support deferred loading of associated objects, while the Entity Framework represents navigation properties using EntityReference<T> and EntityCollection<T>. Likewise, these entities incorporate code elements intended to support client-server scenarios, such as validation (using partial methods) and data binding (with INotifyPropertyChanged), which are unnecessary when you only use them for communication with a Windows Communication Foundation (WCF) service.
Because both LINQ to SQL and the Entity Framework support serialization of entities using the DataContractSerializer (by marking them with DataContract and DataMember attributes), you may be tempted to use them in your service operations anyway, despite the extra baggage they carry. After all, collection types such as those used by LINQ to SQL and the Entity Framework show up as simple arrays on the client when you set a WCF service reference.
Consider, however, that when you set the SerializationMode of the LINQ to SQL DataContext from None to Unidirectional, only the "many" side of one-to-many relations will be marked with the DataMember attribute. This means that if you have an Order entity with Customer and Order_Details properties, Order_Details will be marked with DataMember and included in the data contract, but Customer will not. The Entity Framework, on the other hand, will include more fields than you may want in the data contract, generating classes on the client for the Entity Framework-specific features, such as EntityKey, EntityKeyMember, EntityObject, and EntityReference.
For these reasons, you should avoid exposing either LINQ to SQL or Entity Framework tool-generated entities through the DAL and instead return simple DTOs whose sole purpose in life is to carry data across service boundaries. For example, the Order DTO (shown in Figure 2) contains only properties and no methods. All the DTOs are defined in a DataTransferObjects namespace, to distinguish them from tool-generated classes with the same name.
Figure 2 DTO Representing an Order from Northwind
namespace DataTransferObjects
{
    [DataContract]
    public class Order
    {
        [DataMember]
        public int OrderID { get; set; }

        [DataMember]
        public string CustomerID { get; set; }

        [DataMember]
        public string CustomerName { get; set; }

        [DataMember]
        public DateTime? OrderDate { get; set; }

        [DataMember]
        public List<OrderDetail> OrderDetails { get; set; }

        [DataMember]
        public byte[] Updated { get; set; }

        [DataMember]
        public TrackingInfo TrackingState { get; set; }
    }
}
A flexible DAL should buffer the Order Service from changes to the underlying persistence technology. Suppose, for example, you decide to use LINQ to SQL to persist objects because you've selected SQL Server for your data store and your object model bears a close resemblance to the database schema. But later on, after you've written the LINQ to SQL persistence logic, you decide to use another database system, such as Oracle, or you want to take advantage of the Entity Framework's mapping capabilities to decouple your conceptual and logical schemas. Perhaps a new data access technology comes along that you'd like to use. If you've designed a flexible DAL that's based on a plug-in architecture, you should be able to switch providers on the fly, based on an entry in app.config.
To achieve this flexibility, I've created an interface called IDataProvider, which has methods for retrieving and updating customer orders, as well as retrieving supporting customer and product information (see Figure 3). The sample application includes two classes that both implement IDataProvider: SqlDataProvider, which uses LINQ to SQL, and EntityDataProvider, which uses the Entity Framework and LINQ to Entities (see Figure 1). You can simply enter the fully qualified class name for the "DataProvider" setting in the DAL's app.config file, and use the CreateInstance method of the Assembly class to create an instance of the class, casting it to IDataProvider. The Order Service doesn't care at all about which class implements IDataProvider, giving you the option to use whatever data provider you choose:
IDataProvider provider =
    Assembly.GetExecutingAssembly()
    .CreateInstance(Settings.Default.DataProvider, true)
    as IDataProvider;
Figure 3 Decoupling the DAL with an Interface
namespace DataAccessLayer
{
    // Alias the namespace containing Data Transfer Objects
    using DTO = DataTransferObjects;

    public interface IDataProvider
    {
        // Retrieve Customers, Products, Orders
        DTO.Customer[] GetCustomers();
        DTO.Product[] GetProducts();
        DTO.Order[] GetCustomerOrders(string customerID);

        // CRUD operations for Order
        DTO.Order GetOrder(int orderID);
        DTO.Order CreateOrder(DTO.Order order);
        DTO.Order UpdateOrder(DTO.Order order);
        void DeleteOrder(DTO.Order order);
    }
}
By default, both LINQ to SQL and the Entity Framework will generate entities within the same namespace as the project. But by placing a new "LINQ to SQL Classes" item in a project folder called L2S, the tool will use the folder name as a nested namespace, which helps you differentiate LINQ to SQL entities from your DTO entities. Similarly, placing a new "ADO.NET Entity Data Model" in an L2E project folder causes the Entity Framework entities to be placed in a nested L2E namespace. Alternatively, you could specify the namespace in the property window of the LINQ to SQL and the Entity Framework visual component designers.
Here I am making a choice to generate entities both for LINQ to SQL and the Entity Framework, converting the query results to DTOs. There is also the possibility of dispensing with tool-generated entities altogether and mapping the DTOs using XML files. The Entity Framework already uses XML mapping files, but in the first version of the Entity Framework it is not yet possible to achieve pure persistence ignorance (PI) using plain old C# objects (POCOs). You can get close to this goal in version 1, but entities are required to implement two interfaces, IEntityWithRelationships and IEntityWithChangeTracking, so you're left with IPOCO (POCO + Interfaces). Expect to see improved support for POCOs and persistence ignorance in subsequent versions of the Entity Framework.
LINQ to SQL does have good support for mapping POCOs to a database schema using an XML mapping file. However, projecting from an L2S.Order or L2E.Order to a DTO.Order gives you an added degree of flexibility. For example, Figure 4 shows the GetOrder method of SqlDataProvider, which populates the CustomerName property of DTO.Order from the ContactName property of the L2S.Order's Customer property, in effect flattening the relationship between Customer and Order. I use the same approach to set the DTO.OrderDetail's ProductName property from the L2S.Order_Detail's Product property. In each case, I need to configure the LoadOptions property of the DataContext to eager-load LINQ to SQL entities.
Figure 4 GetOrder Method Returns a DTO.Order Using LINQ to SQL Framework
namespace DataAccessLayer
{
    using DataAccessLayer.L2S;
    using DTO = DataTransferObjects;

    public class SqlDataProvider : IDataProvider
    {
        public DTO.Order GetOrder(int orderID)
        {
            using (NorthwindDataContext db = new NorthwindDataContext
                (Settings.Default.NorthwindSqlConnection))
            {
                // Set load options
                DataLoadOptions opts = new DataLoadOptions();
                opts.LoadWith<Order>(o => o.Customer);
                opts.LoadWith<Order>(o => o.Order_Details);
                opts.LoadWith<Order_Detail>(od => od.Product);
                db.LoadOptions = opts;

                // Create a DTO.Order from the L2S.Order
                DTO.Order order =
                    (from o in db.Orders
                     where o.OrderID == orderID
                     select new DTO.Order
                     {
                         OrderID = o.OrderID,
                         CustomerID = o.CustomerID,
                         CustomerName = o.Customer.ContactName,
                         OrderDate = o.OrderDate,
                         Updated = o.Updated.ToArray(),
                         OrderDetails =
                            (from od in o.Order_Details
                             select new DTO.OrderDetail
                             {
                                 OrderID = od.OrderID,
                                 ProductID = od.ProductID,
                                 ProductName = od.Product.ProductName,
                                 Quantity = od.Quantity,
                                 UnitPrice = od.UnitPrice,
                                 Updated = od.Updated.ToArray()
                             }).ToList()
                     }).Single();
                return order;
            }
        }
    }
}
Now let's turn to the GetOrder method of EntityDataProvider, which uses LINQ to Entities to return a DTO.Order based on an L2E.Order (see Figure 5). Notice the Include operator used to eager-load Order_Details and Order_Details.Product properties. (For more on LINQ operators, see the Data Points column "Standard Query Operators with LINQ.") The Entity Framework does not provide lazy-loading capability out of the box, as does LINQ to SQL, so you are required to indicate explicitly which related entities you wish to include in your query results. After retrieving the specified order, you can simply use it to create a new DTO.Order.
Figure 5 GetOrder Method Returns a DTO.Order Using LINQ to Entities
namespace DataAccessLayer
{
    using DataAccessLayer.L2E;
    using DTO = DataTransferObjects;

    public class EntityDataProvider : IDataProvider
    {
        public DTO.Order GetOrder(int orderID)
        {
            using (NorthwindEntities db = new NorthwindEntities
                (Settings.Default.NorthwindEntityConnection))
            {
                // Get the specified L2E.Order
                Order nwOrder =
                    (from o in db.OrderSet
                        .Include("Order_Details")
                        .Include("Order_Details.Product")
                     where o.OrderID == orderID
                     select o).First();

                // Create a DTO.Order from the L2E.Order
                DTO.Order order = new DTO.Order
                {
                    CustomerID = nwOrder.CustomerID,
                    CustomerName = (from c in db.CustomerSet
                                 where c.CustomerID == nwOrder.CustomerID
                                    select c).First().ContactName,
                    OrderID = nwOrder.OrderID,
                    OrderDate = nwOrder.OrderDate,
                    Updated = nwOrder.Updated,
                    OrderDetails =
                        (from od in nwOrder.Order_Details
                         select new DTO.OrderDetail
                         {
                             OrderID = od.OrderID,
                             ProductID = od.ProductID,
                             ProductName = od.Product.ProductName,
                             Quantity = od.Quantity,
                             UnitPrice = od.UnitPrice,
                             Updated = od.Updated
                         }).ToList()
                };
                return order;
            }
        }
    }
}

Persisting Individual Objects
LINQ to SQL and the Entity Framework take different approaches to persisting disconnected entities. Let's say, for example, you would like to create a new order that has one or more order details. As shown in Figure 6, LINQ to SQL requires at least two lines of code: one to insert the order by calling InsertOnSubmit and another to insert the order's detail items using InsertAllOnSubmit. The Entity Framework, on the other hand, only requires that you add the order to the OrderSet of the ObjectContext, which will automatically add the entire object graph (see Figure 7). Both LINQ to SQL and the Entity Framework do the right thing by inserting the parent order first, obtaining the value for OrderID, which is an identity column in the database, then inserting child order details.
Figure 6 Creating an Order Using LINQ to SQL
public DTO.Order CreateOrder(DTO.Order order)
{
    // Insert new order
    using (NorthwindDataContext db = new NorthwindDataContext
        (Settings.Default.NorthwindSqlConnection))
    {
        // Create an L2S.Order
        Order nwOrder = new Order
        {
            OrderID = order.OrderID,
            CustomerID = order.CustomerID,
            OrderDate = order.OrderDate,
            Updated = order.Updated
        };

        // Add order details
        nwOrder.Order_Details.AddRange(
            from od in order.OrderDetails
            select new Order_Detail
            {
                OrderID = od.OrderID,
                ProductID = od.ProductID,
                Quantity = od.Quantity,
                UnitPrice = od.UnitPrice,
                Updated = od.Updated
            });

        // Insert order and order details separately
        db.Orders.InsertOnSubmit(nwOrder);
        db.Order_Details.InsertAllOnSubmit(nwOrder.Order_Details);
        db.SubmitChanges();

        // Return the updated order
        return GetOrder(nwOrder.OrderID);
    }
}
Figure 7 Creating an Order Using LINQ to Entities
public DTO.Order CreateOrder(DTO.Order order)
{
    using (NorthwindEntities db = new NorthwindEntities
        (Settings.Default.NorthwindEntityConnection))
    {
        // Create an L2S.Order
        Order nwOrder = new Order
        {
            OrderID = order.OrderID,
            CustomerID = order.CustomerID,
            OrderDate = order.OrderDate,
            Updated = order.Updated
        };

        // Add order details
        foreach (DTO.OrderDetail od in order.OrderDetails)
        {
            Order_Detail detail = new Order_Detail
            {
                Order = nwOrder,
                Product = (from p in db.ProductSet
                            where p.ProductID == od.ProductID
                            select p).First(),
                Quantity = od.Quantity,
                UnitPrice = od.UnitPrice,
                Updated = od.Updated
            };
            nwOrder.Order_Details.Add(detail);
        }

        // Add order and order details
        db.AddToOrderSet(nwOrder);
        db.SaveChanges();

        // Return the updated order
        return GetOrder(nwOrder.OrderID);
    }
}
Before looking at deleting or updating entities, I need to discuss how to deal with concurrency issues. LINQ to SQL requires you to attach a disconnected entity prior to calling DeleteOnSubmit. Doing so prompts LINQ to SQL to perform optimistic concurrency checks, causing deletes to fail with a ChangeConflictException if a user tries to delete the item after another user has changed it. There really isn't any easy way to turn off this behavior, because LINQ to SQL will throw an InvalidOperationException if you attempt to invoke DeleteOnSubmit without first having called Attach. The Entity Framework, on the other hand, doesn't force you to handle change conflicts when deleting entities (see the code download accompanying this article).
Figure 8 shows the code for updating an order using LINQ to SQL. Notice that the Attach method accepts an asModified parameter of type bool. Setting this parameter to true indicates to LINQ to SQL that you wish to use a timestamp column for concurrency management. Doing so will cause LINQ to SQL to include the current timestamp value in the WHERE clause for the SQL Update statement. If the record was updated by another user, the update will not affect any rows and LINQ to SQL will throw a ChangeConflictException.
Figure 8 Updating an Order Using LINQ to SQL
public DTO.Order UpdateOrder(DTO.Order order)
{
    using (NorthwindDataContext db = new NorthwindDataContext
        (Settings.Default.NorthwindSqlConnection))
    {
        // Create an L2S.Order
        Order nwOrder = new Order
        {
            OrderID = order.OrderID,
            CustomerID = order.CustomerID,
            OrderDate = order.OrderDate,
            Updated = order.Updated
        };

        // Attach and save order
        db.Orders.Attach(nwOrder, true);
        db.SubmitChanges();

        // Return the updated order
        return GetOrder(order.OrderID);
    }
}
This strategy relieves the client from needing to preserve original values and submit them along with the update, which would need to be passed to one of the other overloads of the Attach method. It doesn't matter what you name the column, so long as it has a timestamp data type. When you add a table with a timestamp column to the LINQ to SQL designer in Visual Studio, the UpdateCheck attribute will be set to Never for all entity fields and LINQ to SQL will use the timestamp to check for change conflicts.
Updating entities and managing concurrency with the Entity Framework is presently a bit more involved than with LINQ to SQL, because it requires the entity's original values, even if you are using a timestamp column for concurrency (see Figure 9). (This is likely to improve in the next version of the Entity Framework.) Because the client has not provided the original order, you need to retrieve the order from the database and set its Updated property to the value provided by the client, which is the only "original" value you really care about. Be sure to commit the change to Updated, either by detaching and re-attaching the order to the object context or by calling AcceptAllChanges. Then call ApplyPropertyChanges to update the original order with values from the updated order. There is one more step required in order for the Entity Framework to use the timestamp column for concurrency checking: you will need to set the ConcurrencyMode for the Updated property on the Order entity from None to Fixed (see Figure 10).
Figure 9 Updating an Order Using LINQ to Entities
public DTO.Order UpdateOrder(DTO.Order order)
{
    using (NorthwindEntities db = new NorthwindEntities
        (Settings.Default.NorthwindEntityConnection))
    {
        // Get original order using key from updated order
        EntityKey key = db.CreateEntityKey("OrderSet",
            new Order { OrderID = order.OrderID });
        Order origOrder = db.GetObjectByKey(key) as Order;

        // Set Updated to match DTO.Order - (for concurrency)
        origOrder.Updated = order.Updated;

        // Commit change to the Updated property
        db.Detach(origOrder);
        db.Attach(origOrder);

        // Create a NW order for updating the original order
        Order updatedOrder = new Order
        {
            OrderID = order.OrderID,
            OrderDate = order.OrderDate,
            CustomerID = order.CustomerID,
            Updated = order.Updated
        };

        // Apply changes to the original order
        db.ApplyPropertyChanges("OrderSet", updatedOrder);

        // Persist changes
        db.SaveChanges();

        // Return the updated order
        return GetOrder(order.OrderID);
    }
}
Figure 10 Concurrency Mode of Updated Set to Fixed

Tracking Changes across Service Boundaries
If your goal is to persist changes for multiple entities in the same transaction with one call to the service, you need a way to determine the change state of each entity. That is why each DTO in the sample application contains a TrackingState property. Assuming this property has been set by the client, the DAL can then apply the appropriate LINQ to SQL or the Entity Framework API for persisting inserts, updates, and deletes for disconnected entities. This is what allows you to pass an array of DTOs to a service operation for batch updating, teasing out inserted, updated, or deleted order details with a simple LINQ to Objects query. For example, to obtain only order details that have been added to an order, you can query the collection for items with a TrackingState of Created:
List<Order_Detail> insertedDetails =
    (from od in order.OrderDetails
     where od.TrackingState == TrackingInfo.Created
     select new Order_Detail
     {
         OrderID = od.OrderID,
         ProductID = od.ProductID,
         Quantity = od.Quantity,
         UnitPrice = od.UnitPrice,
         Updated = od.Updated == null ? new byte[0] : od.Updated
     }).ToList();
You need to convert a null value for Updated to an empty byte array because timestamp columns in the database table are ordinarily not nullable. Once you have the list of inserted order details, you can use the convenient LINQ to SQL API for adding new entities by passing a collection of order details:
db.Order_Details.InsertAllOnSubmit(insertedDetails);
As you might guess, there are matching AttachAllOnSubmit and DeleteAllOnSubmit methods. When you call SubmitChanges on the DataContext, LINQ to SQL will make sure to commit these changes in the correct order (parent-children for inserts, children-parent for deletes) and wrap everything in a transaction. You can take the same approach to persisting order details using LINQ to Entities. See the download for the full code listing of UpdateOrder for both LINQ to SQL and the Entity Framework.
It should be obvious by now that the DAL is relying on the client to track change state for each DTO. The most convenient mechanism for tracking changes to objects on the client is a generic collection. In order for the collection to update the TrackingState property, it should constrain the type argument to implement an interface. (You could also use Reflection to get and set the TrackingState property, but a generic type constraint will provide both type safety and better performance.) Enter the ITrackable interface, which has just one member, the TrackingState property:
public interface ITrackable
   {
       TrackingInfo TrackingState { get; set; }
   }
As I mentioned at the beginning, there are two types of the change tracking collection, one for use with Windows Forms that extends BindingList<T> and another for WPF applications that extends ObservableCollection<T>. But rather than duplicate change-tracking logic in both these collections, I have centralized it in a class called ChangeTrackingHelper<T>, which derives from Collection<T> and constrains T to implement ITrackable (see Figure 11). It also constrains T to implement INotifyPropertyChanged so that it can handle each item's PropertyChanged event and set the TrackingState property to Updated.
Figure 11 Change Tracking Helper Class
internal class ChangeTrackingHelper<T> : Collection<T>
    where T : ITrackable, INotifyPropertyChanged
{
    Collection<T> deletedItems = new Collection<T>();

    // Listen for changes to each item
    private bool tracking;
    public bool Tracking
    {
        get {  return tracking; }
        set
        {
            foreach (T item in this)
            {
                if (value) item.PropertyChanged += OnItemChanged;
                else item.PropertyChanged -= OnItemChanged;
            }
            tracking = value;
        }
    }

    void OnItemChanged(object sender, PropertyChangedEventArgs e)
    {
        if (!Tracking) return;
        ITrackable item = (ITrackable)sender;
        if (e.PropertyName != "TrackingState")
        {
            // Mark item as updated
            if (item.TrackingState == TrackingInfo.Unchanged)
                item.TrackingState = TrackingInfo.Updated;
        }
    }

    // Mark item as created and listen for property changes
    protected override void InsertItem(int index, T item)
    {
        if (!Tracking) return;
        item.TrackingState = TrackingInfo.Created;
        item.PropertyChanged += OnItemChanged;
        base.InsertItem(index, item);
    }

    // Mark item as deleted and cache it
    protected override void RemoveItem(int index)
    {
        if (!Tracking) return;
        T item = this.Items[index];
        if (item.TrackingState != TrackingInfo.Created)
        {
            item.TrackingState = TrackingInfo.Deleted;
            item.PropertyChanged -= OnItemChanged;
            deletedItems.Add(item);
        }
        base.RemoveItem(index);
    }

    public ChangeTrackingHelper<T> GetChanges()
    {
        // Create a collection with changed items
        ChangeTrackingHelper<T> changes = new ChangeTrackingHelper<T>();
        foreach (T existing in this)
        {
            if (existing.TrackingState != TrackingInfo.Unchanged)
                changes.Add(existing);
        }

        // Append deleted items
        foreach (T deleted in deletedItems)
            changes.Add(deleted);
        return changes;
    }
}
When an item is added to the collection, its TrackingState is set to Created. When an item is removed, its TrackingState is set to Deleted and it is cached in a collection of deleted items. The user indicates that tracking should begin on a collection by setting the Tracking property to true, at which point the collection will subscribe to each item's PropertyChanged event. The GetChanges method creates a new ChangeTrackingHelper collection, comprised only of items marked as Created, Updated, or Deleted.

Configuring the Client
Now that we have a set of generic change tracking collections, it's time to use them! I've placed these in a separate class library, so all you have to do is reference it from the client application. Nevertheless, you need to take some additional steps so that the collection can use classes generated by svcutil.exe when you add a service reference to the client project. Classes such as Customer, Product, Order, and OrderDetails will already possess the TrackingState property, but they will not implement the ITrackable interface required by ChangeTrackingHelper<T>.
Furthermore, the TrackingInfo enum needs to be placed in a namespace where the change tracking collection can find it, instead of within the nested namespace where svcutil.exe will place it when you add a service reference. Lastly, it would be nice if arrays of objects returned by service operations would appear as change tracking collections in your client application. This would set the data type for Order.OrderDetails to ChangeTrackingCollection<T> or ChangeTrackingList<T> instead of an array of OrderDetail, which would save you some extra work when you're sending changed order details to the service.
Here are the steps you should follow to set up a Windows Forms or WPF client application to use a change tracking collection to track entity changes and send them to the service. Be sure to follow these steps in the order given:
  1. From a Windows Forms or WPF client application, set a reference to the ClientChangeTracker class library project or assembly, where the ITrackable interface and change tracking collections are located.
  2. With the WCF service application running, add a service reference pointing to the metadata endpoint or Web Services Description Language (WSDL) URL. It is important that you reference the ClientChangeTracker assembly prior to adding the service reference. By default, a service reference will reuse types in all referenced assemblies, and because ClientChangeTracker contains the TrackingInfo enum with DataContract and EnumMember serialization attributes, svcutil.exe will reuse it instead of copying it into the Reference.cs code file created by the service reference.
  3. After setting the service reference, add partial classes for each DTO class created by the service reference. For example, if your service operations expose Customer, Product, Order, and OrderDetail objects, these will be included in the service reference. If you show all files for the project and expand the service reference and Reference.svcmap node, you'll see these classes located in Reference.cs. Make note of the namespace in which they are located (which should match what you specified when adding the service reference). Now add a class file to the project and change the namespace to reflect the namespace in Reference.cs. Within this namespace, add public partial classes for each DTO class and implement the ITrackable interface:
    namespace WpfClient.OrderService
{
    public partial class Customer : ITrackable { }

    public partial class Product : ITrackable { }

    public partial class Order : ITrackable { }

    public partial class OrderDetail : ITrackable { }
}
    Note the absence of the TrackingState property. You won't have to insert it here, because the code-generated classes already contain this property—it is part of the data contract for each DTO. You may also insert additional code in these classes to make it easier to use them in your application, for example, constructors to initialize the class (which are called when you create the class using new, but not when WCF creates them). See the code download for an example.
  4. Lastly, you will probably want to configure the service reference to use ChangeTrackingCollection<T> or ChangeTrackingList<T> as the collection type instead of System.Array. This means that when a service operation returns an Order array, it will materialize on the client as ChangeTrackingCollection<Order> or ChangeTrackingList<Order>. The OrderDetails property of each Order will likewise show up as a change tracking collection type. How cool is that? To make this magic happen, you'll need to open the Reference.svcmap file (show all project files and open the service reference to see it) and dive into the XML. For a WPF client, configure the CollectionMappings element as follows:
<CollectionMappings>
    <CollectionMapping TypeName="ChangeTracker.ChangeTrackingCollection'1" Category="List" />
</CollectionMappings>
For a Windows Forms client, you should replace ChangeTracker.ChangeTrackingCollection`1 with ChangeTracker.ChangeTrackingList`1.
  1. Now, with the service application still running, right-click the service reference and select "Update Service Reference" and recompile the client application. If all goes well, the project will build successfully and you'll be on your way.
After performing these steps, all that's left is to write code for the client application to retrieve some DTOs, bind them to the UI where the user can update them, and send them back to the service, which will persist them to the database. Retrieving objects from the service is simply a matter of creating an instance of the service proxy and invoking operations to get Customers, Products, and Orders. For example, if the service has a GetOrder method that accepts an int for orderId, then your client code might look like this:
using (OrderServiceClient proxy = new OrderServiceClient())
{
   CurrentOrder = proxy.GetOrder(orderId);
}
If you've set up data binding properly, the order should appear in the user interface, together with the order details. Figure 12 shows what the WPF client looks like for the sample application. The "Get Order" button allows the user to retrieve an existing order, choosing from a list of customer orders. The "New Order" button simply creates a new order, and the "Delete Order" button calls DeleteOrder on the service proxy, passing the current order.
Figure 12 Sample WPF Client App
The "Save Order" button will call the service CreateOrder operation to persist a newly created order, or it will call the UpdateOrder operation to save changes to an existing order. Both these operations return an order object, which will contain the current Updated property (for concurrency management), and a new OrderId property for inserted orders (the Order table in Northwind uses an identity column). After obtaining the updated order, you will start change tracking and set data binding to use the order.
However, before saving an existing order, you have a little more work to do. We only want to pass order details that have been created, updated, or deleted and not waste network bandwidth by sending details that are unchanged. ChangeTrackingCollection<T> and ChangeTrackingList<T> make this relatively easy by exposing a GetChanges method, which will return only order details that have been inserted, modified or removed. The code to invoke UpdateOrder should look like Figure 13.
Figure 13 Invoking UpdateOrder
Order changedOrder = new Order
{
    OrderID = CurrentOrder.OrderID,
    CustomerID = CurrentOrder.CustomerID,
    OrderDate = CurrentOrder.OrderDate,
    Updated = CurrentOrder.Updated,
    TrackingState = CurrentOrder.TrackingState,
    OrderDetails = CurrentOrder.OrderDetails.GetChanges()
};

using (OrderServiceClient proxy = new OrderServiceClient())
{
    CurrentOrder = proxy.UpdateOrder(changedOrder);
}
Because I've configured the service reference to use ChangeTrackingCollection<T> as the collection type, the OrderDetails property of the Order class will be of type ChangeTrackingCollection<OrderDetail>, and I can call GetChanges on it directly to get just the order details that were added, modified or removed, passing those on to the UpdateOrder operation. And because ChangeTrackingCollection<T> extends ObservableCollection<T>, it supports data binding features conducive to WPF applications (for example, implementing INotifyCollectionChanged so that the UI can update an items control when items are added or removed from the collection). Similarly, because ChangeTrackingList<T> extends BindingList<T>, it serves as a good data source for a Windows forms DataGridView, which leverages interfaces implemented by BindingList<T> to enhance the user experience.

Moving Forward
You now have a flexible service-oriented application architecture where the client is completely ignorant of how objects will be persisted. As tempting as it might be to pass tool-generated entities across service boundaries, we have seen that DTOs allow more control over the structure and shape of your object model. In addition, building a data access layer will decouple the application from whatever persistence technology you may be using, allowing you to replace a data access implementation without affecting other parts of the application.
LINQ to SQL and the ADO.NET Entity Framework represent a quantum leap forward in the ORM space and go a long way toward reducing the impedance mismatch between objects and relational data. They support various scenarios, including traditional client-server and service-oriented architectures. However, updating the database using DTOs requires more work on the part of the developer, especially if you use a timestamp value to detect conflicts among concurrent users. Nevertheless, incorporating change state into the data contract between client and service allows you to attach entities that have been disconnected from their original context. Lastly, we have seen how a generic change tracking collection can do the heavy lifting of managing change state on the client, allowing you to process batch updates with a single round trip to the service.
For more on the Entity Framework, see "ADO.NET: Achieve Flexible Data Modeling with the Entity Framework" by Elisa Flasko.

Anthony Sneed is an instructor for the developer training company DevelopMentor, where he authors and teaches courses on the .NET Framework 3.5, LINQ, and the Entity Framework. In his spare time he enjoys cooking his world-famous chili and producing family videos. You can reach him at tony@tonysneed.com or visit his blog at blog.tonysneed.com.


Geneva Framework
A Better Approach For Building Claims-Based WCF Services
Michele Leroux Bustamante
Code download available from the MSDN Code Gallery

Browse the Code Online

This article is based on a prerelease version of the "Geneva" Framework. All information is subject to change.

This article discusses:
  • Implementing security with the Geneva Framework
  • WCF security
  • Claims-based security
  • Security Token Services
 This article uses the following technologies:
Windows Communication Foundation
Contents
Why Claims-Based Security?
Security Token Services
Support for Claims-Based Security
WCF Services, before the Geneva Framework
Issued Tokens and WSFederationHttpBinding
Enable the Geneva Framework for WCF Services
Demanding Claims
Configuring <microsoft.identityModel>
Hooking the Geneva Framework with IIS Hosting
From WCF to the Geneva Framework
Migrate Applications to the Geneva Framework
The "Geneva" Framework, formerly called "Zermatt," is the code name for the new framework for building claims-based applications and services, and for implementing federated security scenarios. Features include the plumbing to build a custom Security Token Service (STS); a mechanism to require federated authentication from ASP.NET applications; and an object model that facilitates claims-based authorization for ASP.NET applications and Windows Communication Foundation (WCF) services.
The Geneva Framework also includes features that support Windows CardSpace such as managed information card issuance and ASP.NET controls that simplify the creation of a Windows CardSpace login experience. (Read more about Windows CardSpace in "Identity: Secure Your ASP.NET Apps and WCF Services with Windows CardSpace".) Clearly, the Geneva Framework spans the breadth of security features, but at its core is claims-based security.
While WCF has always had native support for a claims-based security model, the Geneva Framework enhances this experience by simplifying access to claims at run time and providing a mechanism to support claims-based authorization in a manner that is consistent with the role-based authorization principals already available in the Microsoft .NET Framework. ASP.NET applications utilize the Geneva Framework for claims-based authorization features compatible with existing ASP.NET login controls that enforce role-based security. In this article I will focus on the value of implementing a claims-based security model, describe how to approach claims-based WCF services using the Geneva Framework, and discuss how this compares to the way in which WCF handles claims-based security without the Geneva Framework.
Before you continue reading, I recommend that you take a look at the Geneva Framework white paper for developers, written by Keith Brown and Sesha Mani. This will provide you with an overview of the features in the Geneva Framework, as well as some background on claims-based security concepts, and explain how to enable these features in ASP.NET applications and WCF services, with an emphasis on the former. In addition, you can learn more about WCF and claims-based security in Keith Brown’s Security Briefs column published in September 2007.

Why Claims-Based Security?
Why do you want to move to a claims-based security model? Before you consider implementing a solution with the Geneva Framework you must know the answer to this question. It is true that role-based security is often sufficient if you assume that the definition of application roles never changes and that only a single authentication mechanism will map a security principal to those roles. But it helps to design applications and services so that they aren't tied to a particular credential type or to a particular set of roles. This is one of the value propositions of a claims-based security model.
Decoupling applications and services from roles allows the name and meaning of roles to change without affecting the system. Authenticated users can be assigned a more granular artifact for authorization—a claim. Claims can be assigned based on the authenticated user, as shown in Figure 1, or assigned based on the authenticated user's roles, as shown in Figure 2.
Figure 1 Assigning Claims Based on the Authenticated User
Figure 2 Assigning Claims Based on the Authenticated User’s Roles
The latter still supports a model where authorization is performed based on claims, while using roles as a way to group claims more easily behind the scenes. From this perspective, claims-based security has properties similar to permission-based security, although claims are guaranteed by a trusted issuer, which has added security benefits. (Note that Figures 1 and 2 illustrate the use of the assignment of a name claim for the authenticated user and the assignment of several custom permission claims—Create, Read, Update, and Delete.)
Authorizing access based on claims is most useful when the association between a claim and the functionality or resources to which it grants rights does not change (at least not frequently). Meanwhile, the rules for how users and roles are assigned claims can change freely without impacting authorization logic. For example, the Delete claim is likely to be required by all delete operations in the system, while the Administrators role may not always grant that claim if a new SuperUsers role is introduced.
A claims-based security model can also help applications and services support multiple credential types. A simple (and common) example is an application that relies on Windows credentials for internal users on the same domain, and custom username and password accounts for external users outside of the domain. Supporting multiple credential types can complicate the authentication process and related authorization code. ASP.NET applications that support multiple credential types add complexity in terms of login support, configuration, and initialization of roles for each credential type.
WCF services also require additional work to support different credential types in terms of behavior configuration for the different credentials, and proper initialization of the AuthorizationContext. If ASP.NET applications and WCF services can receive a credential that carries required claims (be they roles or otherwise), authorization efforts can be normalized to a common set of rules regardless of how the user was authenticated.
Ideally, an STS would handle authentication of the different credential types and handle the generation of a security token that carries those claims. This token, likely a SAML (Security Assertion Markup Language) token, can then be used to authenticate to the service based on its digital signature—and the claims it yields can be used to authorize access.
Federated security scenarios are also a natural fit for claims-based security. In a federated security scenario users are authenticated by their own security domain and a token is issued by that domain to be used to authenticate to another domain.

Security Token Services
An STS plays an important role in a claims-based security scenario—whether or not it's a federated scenario. To authorize access based on claims, a trusted set of claims must be available to the application. An STS can issue a security token with the requested claims after authenticating the user. The application need only trust the security token—which usually means trusting its digital signature. Figure 3 illustrates two applications (relying parties)—a WCF service and an ASP.NET application—that trust SAML tokens issued by an STS. The high-level flow of communication is as follows:
Figure 3 Relying Party Trusting Tokens Issued by an STS
  1. Users authenticate to the STS.
  2. The STS assigns claims for the authenticated user and builds a SAML token to carry those claims (as SAML attributes). The STS signs the token with its private key (IPKey) and encrypts it for the application using the public key (RPKey) provided with the request.
  3. The client app or browser presents the token to the app (relying party). The token is passed with the message in the WCF scenario and passed as a cookie in the ASP.NET scenario. If the token signature is trusted (IPKey), the claims within are trusted for authorization.
If multiple credential types are to be supported by an application or service, the STS can handle the authentication step for each credential type and issue a security token carrying claims appropriate for each authenticated user. Once again, the application need only trust the security token—and is blissfully unaware of the credential types that the STS supported to authenticate each user. Figure 4 illustrates an STS that accepts Windows, username and password, certificate, or SAML tokens for authentication—and generates appropriate claims for the authenticated user.
Figure 4 An STS Authenticating Multiple Credential Types
Figures 3 and 4 illustrate scenarios where an STS is employed within the same security domain. In a federated scenario, a trust relationship is established between two or more security domains so that users can authenticate to the domain that manages their credentials while still being granted access to resources in another domain.
Federation reduces many risks related to identity management. It removes the need to maintain user credentials across multiple applications or domains—which helps to reduce risks associated with provisioning and deprovisioning accounts across domains such as forgetting to delete an account in multiple places. Password synchronization when multiple copies of an account aren't managed will also cease to be a problem. In addition to these benefits, federation also facilitates Single Sign-On (SSO) scenarios since users can log in to one application and be granted access to another (possibly across security domains) without having to authenticate again. Figure 5 illustrates a federated scenario for a Web application. The flow is as follows:
  1. User browses to the Web application in Domain B and is authenticated by the STS in Domain B.
  2. The STS issues a SAML token for the user, which is returned as a cookie to the browser.
  3. User browses to the Web application in Domain A, passing the SAML token as a cookie.
  4. The token signature is trusted by the STS in Domain A because of the trust relationship with the STS in Domain B.
  5. The STS in Domain A issues a new SAML token for the user, with claims relevant to Domain A. This token is returned as a cookie to the browser.
Figure 5 A Federated Scenario Illustrating Trust between Domains
The trust relationship between the STS in Domain A and the STS in Domain B means that the former will accept tokens issued by the latter as proof of authentication. Claims issued by Domain B may include user information (username, e-mail address, and so forth) and rights that are understood by Domain A (that is DomainAReadOnly). Domain A may convert these partner claims to claims meaningful to the application hosted by Domain A—for example, granting the Read claim.
Claims issued by an STS are open-ended, which means there are many potential configurations to these scenarios involving delegation of authentication and federation—but this should provide you with a fundamental understanding of why claims-based security is useful and how those claims might be issued. Now I will move on to some implementation details for WCF services and ASP.NET applications.

Support for Claims-Based Security
The Geneva Framework has many features, as discussed in the white paper I mentioned earlier. You can use the Geneva Framework to build a custom STS without writing all of the plumbing to expose WS-Trust endpoints or build SAML tokens that carry claims. With this framework you can also issue managed Information Cards to support identity selectors such as Windows CardSpace on the Windows platform. You can more easily support Windows CardSpace login in ASP.NET applications, and you can leverage integrated claims-based support for WCF services and ASP.NET applications.
The primary goal of the Geneva Framework is to make it easier to build claims-based applications. This is done with claims-based extensions to the existing role-based security model common to all .NET-based apps, with hooks that process security tokens to extract claims for each respective runtime to interact with, and in the case of ASP.NET, with a mechanism for triggering calls to an STS that can issue tokens carrying the required claims of the application.

WCF Services, before the Geneva Framework
WCF services were designed to support claims-based security from the beginning. When a message is sent to a service operation, each security token supplied with the message is converted to a set of claims accessible through the ServiceSecurityContext for the executing operation. If the security token is a Windows or UserName token, developers would typically rely on the security principal attached to the request thread to authorize calls using classic role-based security. If the security principal is a certificate or SAML token, the claims become more meaningful and developers are likely to access the AuthorizationContext to evaluate claims.
Consider an example in which a WCF service exposes Create, Read, Update, and Delete (CRUD) operations and expects to authorize calls based on the user's CRUD rights to the application. The DeleteSomething operation requires the authenticated caller to have delete rights, indicated by a custom permission claim like those shown in Figures 1 and 2. The code to search the AuthorizationContext for the Delete claim is shown in Figure 6. The code traverses the claim sets in the AuthorizationContext looking for this claim. If it isn't found in any of the trusted claim sets, a SecurityException is thrown. The code to search the AuthorizationContext this way is cumbersome, to say the least. The Geneva Framework provides an alternate approach to evaluating claims that greatly simplifies claims-based authorization in WCF.
Figure 6 WCF AuthorizationContext for a Permission Claim
public string DeleteSomething() {
    AuthorizationContext authContext =
      ServiceSecurityContext.Current.AuthorizationContext;

    bool foundClaim = false;
    foreach (ClaimSet cs in authContext.ClaimSets) {
        Claim claim = new Claim("http://schemas.contoso.com/samples/
          2008/09/claims/permission","Delete", Rights.PossessProperty);

        if (cs.ContainsClaim(claim)) {
            foundClaim = true;
            break;
        }

    }

    if (!foundClaim)
        throw new SecurityException(
          "Access is denied. Required claims not satisfied.");

    return String.Format("DeleteSomething() called by user {0}",
      System.Threading.Thread.CurrentPrincipal.Identity.Name);
}

Issued Tokens and WSFederationHttpBinding
WCF supplies federation bindings to support scenarios in which the WCF service expects a token issued by an STS. Typically this implies a SAML token, although this is not a strict requirement. WSFederationHttpBinding is the original standard binding that supports issued tokens, and WS2007FederationHttpBinding is an update to this binding supporting the final release of the WS-* protocols underlying the binding. For the purpose of this discussion, I will use WSFederationHttpBinding to configure WCF service endpoints that require a SAML 1.1 token issued by a custom STS built with the Geneva Framework.
For this example the service contract, ICrudService, has four service operations, as shown in Figure 7. These operations are indicative of CRUD operations, and the service implementation will rely on CRUD permission claims like those shown inFigures 1 and 2 to authorize access to each operation.
Figure 7 WCF Service Contract Exposing CRUD Operations
[ServiceContract(Namespace="http://www.contoso.com/samples/2008/09")]
public interface ICrudService
{
  [OperationContract]
  string CreateSomething();

  [OperationContract]
  string ReadSomething();

  [OperationContract]
  string UpdateSomething();

  [OperationContract]
  string DeleteSomething();
}
Figure 8 illustrates a sample WCF configuration with a single service exposing a WSFederationHttpBinding endpoint and its associated behavior configuration. The binding configuration includes settings indicating the required token format (SAML 1.1), required Claim types (name and permission claim), and the STS WS-Trust endpoint address and metadata exchange address. This configuration provides enough information for clients to generate a proxy that is capable of authenticating to the STS to retrieve the SAML token, and using that token to authenticate to the WCF service.
Figure 8 WCF Configuration with a WSFederationHttpBinding Endpoint
<system.serviceModel>
  <services>
   <service name="ClaimsBasedServices.CrudService"
    behaviorConfiguration ="serviceBehavior">
    <endpoint contract="ClaimsBasedServices.ICrudService" binding=
     "wsFederationHttpBinding" bindingConfiguration="wsFed"/>
      <endpoint contract="IMetadataExchange" binding="mexHttpBinding"
          address="mex"/>
   </service>
  </services>
  <bindings>
   <wsFederationHttpBinding>
    <binding name="wsFed" >
      <security mode="Message">
        <message issuedTokenType="http://docs.oasis-open.org/wss/
          oasis-wss-saml-token-profile-1.1#SAMLV1.1" >
          <claimTypeRequirements>
            <add claimType="http://schemas.xmlsoap.org/ws/2005/05/
              identity/claims/name" isOptional="false"/>
            <add claimType="http://schemas.contoso.com/samples/2008/09/
              claims/permission" isOptional="false"/>
          </claimTypeRequirements>
           <issuer address="http://localhost:51213/TokenIssuer/Service.svc" />
           <issuerMetadata address="http://localhost:51213/TokenIssuer/
              Service.svc/mex" />
        </message>
      </security>
    </binding>
   </wsFederationHttpBinding>
  </bindings>
  <behaviors>
   <serviceBehaviors>
    <behavior name="serviceBehavior">
      <serviceMetadata httpGetEnabled="true"/>
      <serviceCredentials>
        <issuedTokenAuthentication allowUntrustedRsaIssuers="false">
          <knownCertificates>
            <add findValue="IPKey" storeLocation ="LocalMachine"
             storeName="TrustedPeople" x509FindType "FindBySubjectName"/>
          </knownCertificates>
        </issuedTokenAuthentication>
        <serviceCertificate findValue="RPKey" storeLocation=
         "LocalMachine" storeName="My" x509FindType="FindBySubjectName"/>
      </serviceCredentials>
      <serviceAuthorization principalPermissionMode="None" />
    </behavior>
   </serviceBehaviors>
  </behaviors>
</system.serviceModel>
The service behavior configuration has some settings worth pointing out. Inside <serviceCredentials>, the <issuedTokenAuthentication> section indicates how the SAML token should be authenticated. This configuration indicates that only trusted issuers are allowed, which means that the SAML token must be signed by a known certificate. The known certificate in this scenario has the subject name IPKey. The STS will use IPKey to sign the SAML token; thus, any claims contained within the SAML token will be trusted. This section is still relevant when the Geneva Framework is enabled, even though the Geneva Framework also has a configuration for specifying trusted issuers.
The <serviceAuthorization> section influences the type of security principal attached to the request thread after authentication takes place. Normally, for an issued token scenario, you would configure this to "Custom," which requires you to specify a custom authorization policy (a type that implements IAuthorizationPolicy). The authorization policy is responsible for creating the security principal (the IPrincipal type) for the request thread and for attaching any relevant claims to the AuthorizationContext. This is set to None in this case because it will no longer be useful once the WCF service is Geneva Framework-enabled.

Enable the Geneva Framework for WCF Services
A claims-based security model based on Geneva Framework features can be achieved with just a few simple steps. The first step is to add a reference to the Microsoft.Identity assembly—the core assembly belonging to the Geneva Framework. Next, the ServiceHost for each WCF service must be initialized for the Geneva Framework runtime features, which includes registering a list of trusted token issuers to the ServiceHost. The end result of configuring this framework is that a security principal is attached to the request thread for each call—carrying a collection of claims from trusted issuers for authorization.
FederatedServiceCredentials is a derivative of the ServiceCredentials type that exposes a static method to initialize the ServiceHost instance for the Geneva Framework. Figure 9 illustrates where the code to initialize the ServiceHost fits in the context of initialization.
Figure 9 Initialize the ServiceHost for Geneva Framework
ServiceHost host = new ServiceHost(typeof(Services.RelyingParty));
try
{
  FederatedServiceCredentials.ConfigureServiceHost(host, new
                                      TrustedIssuerNameRegistry());
  host.Open();
  Console.ReadLine();
}
finally
{
  if (host.State != CommunicationState.Faulted)
    host.Close();
  else
    host.Abort();
}

public class TrustedIssuerNameRegistry : IssuerNameRegistry
{
    public override string GetIssuerName(SecurityToken securityToken)
    {
        X509SecurityToken x509Token = securityToken as X509SecurityToken;
        if (x509Token != null)
        {
            if (x509Token.Certificate.SubjectName.Name == "CN=IPKey")
            {
                return x509Token.Certificate.SubjectName.Name;
            }
        }
        throw new SecurityTokenException("Token signature is not trusted
                                         by the issuer name registry. ");
    }
}
The ConfigureServiceHost method is called before the ServiceHost instance is opened. This method performs the following key actions. It reads the <microsoft.identityModel> configuration section to initialize aspects of the runtime such as token handlers, trusted issuers, and maximum clock skew for token validity. Then it sets the principalPermissionMode (see Figure 8) to Custom and adds a custom authorization policy (IAuthorizationPolicy type) named ServiceAuthorizationPolicy. ServiceAuthorizationPolicy is responsible for creating the ClaimsPrincipal type and initializing Thread.CurrentPrincipal. It also removes other authorization policies and clears the claim sets added to the AuthorizationContext for the request. That's because the Geneva Framework expects you to rely on the ClaimsPrincipal to evaluate claims going forward, rather than using the AuthorizationContext.
The code in Figure 9 passes an implementation of the IssuerNameRegistry type to ConfigureServiceHost, which eliminates the need to specify this type in the <microsoft.identityModel> configuration section. The override for GetIssuerName is responsible for checking the signing token against a list of trusted issuers. This example uses a hardcoded trusted issuer name, IPKey. Later I will discuss elements of the <microsoft.identityModel> section as it relates to configuring the Geneva Framework for WCF services.

Demanding Claims
Once the service is Geneva Framework-enabled, a ClaimsPrincipal instance will be attached to each request thread, carrying claims for the authenticated user. To authorize access to application code and resources, you can traverse the collection of claims for a particular Claim type and value. The following code accesses the Claims collection exposed by the ClaimsIdentity type, searching for the role claim with a Uri value for the Administrators role:
ClaimsIdentity identity =     Thread.CurrentPrincipal.Identity as ClaimsIdentity;
if (!identity.Claims.Exists(c=>
    c.ClaimType=="http://schemas.microsoft.com/ws/2006/04/identity/
    claims/role" && c.Value=="http://schemas.contoso.com/
    samples/2008/09/roles/administrators"))
throw new SecurityException("Access is denied.");
The Claims collection may carry claims from different issuers, so it is possible to check for a claim from a particular issuer. This is usually an unnecessary step, however, since once it is established that the issuer is trusted, it shouldn't matter who issued the claim when authorizing access.
Although direct access to the Claims collection through the ClaimsIdentity instance is convenient, the ClaimsPrincipal provides an IsInRole implementation that can be used to demand role claims as follows:
if !Thread.CurrentPrincipal.IsInRole
    ("http://schemas.contoso.com/samples/2008/09/roles/administrators"))
  throw new SecurityException("Access is denied.");
This approach assumes that the STS issues role claims with the appropriate values relevant to the application. The role claim in the Geneva Framework is indicated by the following Uri by default: "http://schemas.microsoft.com/ws/2006/04/identity/claims/role." The actual role can be any string, but in this case it's a custom Uri indicating an application role: "http://schemas.contoso.com/samples/2008/09/roles/administrators."
While IsInRole is a convenient way to perform claims-based security checks, it is also possible to perform classic permission demands using PrincipalPermission and PrincipalPermissionAttribute. These both rely on an IPrincipal type attached to the request thread—in this case a ClaimsPrincipal will do the trick.
Programmatic demands can be performed dynamically:
PrincipalPermission p = new PrincipalPermission(null,
    "http://schemas.contoso.com/samples/2008/09/roles/administrators",
    true);
    p.Demand();
Demand executes an IsInRole check against the thread's ClaimsPrincipal and throws an exception if the demand can't be satisfied—or, put another way, if the specified role claim is not found. A collection of PrincipalPermission instances can be grouped into a PermissionSet to check for one of several roles being satisfied.
An even more elegant solution, of course, is to apply the PrincipalPermissionAttribute to service operations as follows:
[PrincipalPermission(SecurityAction.Demand, Role =
    "http://schemas.contoso.com/samples/2008/09/roles/administrators")]
    public string CreateSomething()
This attribute constructs the PrincipalPermission type and performs the same demand, but it does so before executing the operation to which it is applied. The beauty of this model is its declarative nature—developers can see the roles required by their service operations. This attribute can also be stacked so that more than one role will satisfy the permission demand.

Configuring <microsoft.identityModel>
The Geneva Framework relies on a new configuration section to initialize its environment: <microsoft.identityModel>. As discussed in the white paper mentioned earlier, this configuration section is used by Geneva Framework features for ASP.NET and WCF applications—but in this section I'll focus on core elements relevant to Geneva Framework-enabled WCF services.
To enable support for this new section, the following must appear in your app.config or web.config (depending on the hosting model for your WCF services):
<configSections>
    <section name="microsoft.identityModel"
      type="Microsoft.IdentityModel.Configuration.
      MicrosoftIdentityModelSection,
      Microsoft.IdentityModel,
      Version=0.4.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
</configSections>
The <microsoft.identityModel> configuration section does not have any required elements, but you can supply any of the core settings to provide declarative initialization of the elements listed in Figure 10.
Figure 10 Initialize the ServiceHost for Geneva Framework
Name | Description
audienceUris | Indicate the allowed audience Uris for SAML token processing. Refers to a feature of the SAML token that can restrict where the token can be used by providing a specific Uri. If this is present in the SAML token, the service can check it against its list of allowed Uris to verify that the token was intended for one of those trusted Uris. Prevents malicious use of tokens.
issuerNameRegistry | Used to declaratively indicate the issuerNameRegistry type that will be used to validate token signatures.
maxClockSkew | Provides control over the valid time window for a token to prevent replay attacks.
securityTokenHandlers | Used to customize token handler type for issued tokens. For example, there's a SAML 1.1 and SAML 2.0 handler. In this section you can customize the settings for the token handler or replace the token handler. A customization might be to provide alternate role claim types.
An example of the configuration section to initialize these values is shown in Figure 11. When the issuerNameRegistry type is specified in configuration, initialization of the service host shown in Figure 9 would change to the following:
FederatedServiceCredentials.ConfigureServiceHost(host);
Figure 11 Core Geneva Framework Features for WCF
<microsoft.identityModel>
  <issuerNameRegistry type="Services.TrustedIssuerNameRegistry,
    Services"/>
  <SecurityTokenHandlers>
    <remove type="Microsoft.IdentityModel.Tokens.Saml11.
      Saml11SecurityTokenHandler, Microsoft.IdentityModel,Version=0.4.1.0,
      Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    <add type="Microsoft.IdentityModel.Tokens.Saml11.
      Saml11SecurityTokenHandler, Microsoft.IdentityModel,
      Version=0.4.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
      <samlSecurityTokenRequirement audienceUriMode="Never">
        <nameClaimType value="http://schemas.contoso.com/ws/2005/05/
          claims/name"/>
        <roleClaimTypes>
          <add value="http://schemas.contoso.com/samples/2008/09/claims/
          permission"/>
        </roleClaimTypes>
      </samlSecurityTokenRequirement>
    </add>
  </SecurityTokenHandlers>
</microsoft.identityModel>
As for the token handlers, you can either create a custom securityTokenHandler or use this section to initialize features for an existing token handler—the latter of which is shown in Figure 11. In this example, the SAML 1.1 token handler is initialized with custom name and role claim types (recall from the earlier white paper that the role claim type collection is likely to become a single value rather than a collection in a future release of the Geneva Framework). The role claim type specified here is "schemas.contoso.com/samples/2008/09/claims/permission", which means that the STS can issue permission claims like those indicated in Figure 1 or 2—and the WCF service can execute permission demands for the specified permission using the classic role-based security methods discussed earlier.

Hooking the Geneva Framework with IIS Hosting
Depending on your experience level with WCF, you may or may not be familiar with the mechanism for custom ServiceHost initialization when services are hosted in IIS, which you'll need to know so that you can enable the Geneva Framework for the ServiceHost. To hook ServiceHost initialization, you provide a custom ServiceHostFactory type in the @ServiceHost directive as follows:
<%@ ServiceHost Factory="ClaimsBasedServices.
    ClaimsBasedServiceHostFactory"
    Service="ClaimsBasedServices.CrudService" %>
In this example, the ClaimsBasedServiceHostFactory type is responsible for constructing the ServiceHost instance for the specified service type. You can either construct the ServiceHost type directly in the factory and initialize it for Geneva Framework services prior to passing the instance to the runtime, or you can create a custom ServiceHost type and override the InitializeRuntime method to supply custom code prior to opening the service channel. The latter is shown in Figure 12.
Figure 12 Enable Geneva on IIS-Hosted WCF Services
public class ClaimsBasedServiceHostFactory: ServiceHostFactory
{
        public ClaimsBasedServiceHostFactory()
        {
        }

        public override System.ServiceModel.ServiceHostBase CreateServiceHost(
                                   string constructorString, Uri[] baseAddresses)
        {
            Type t = Type.GetType(string.Format("{0}, {1}", constructorString,
                constructorString.Substring(0, constructorString.IndexOf("."))));
            return new ClaimsBasedServiceHost(t, baseAddresses);
        }

        protected override System.ServiceModel.ServiceHost CreateServiceHost(
                                           Type serviceType, Uri[] baseAddresses)
        {
            return new ClaimsBasedServiceHost(serviceType, baseAddresses);
        }
}

public class ClaimsBasedServiceHost: ServiceHost
{
        public ClaimsBasedServiceHost(object singletonInstance, params Uri[]
                           baseAddresses): base(singletonInstance, baseAddresses)
        {
        }

        public ClaimsBasedServiceHost(Type serviceType, params
                          Uri[] baseAddresses) : base(serviceType, baseAddresses)
        {
        }

        protected override void InitializeRuntime()
        {
            FederatedServiceCredentials.ConfigureServiceHost(this, new
                                                    TrustedIssuerNameRegistry());
            base.InitializeRuntime();
        }
}

From WCF to the Geneva Framework
There are some common questions that are likely to surface when you enable the Geneva Framework for your WCF services—particularly if you are a seasoned WCF developer who has already implemented claims-based solutions with pre-Geneva Framework techniques. One question you'll likely have relates to why the Geneva Framework doesn't use the ClaimSet and Claim type from the System.IdentityModel.Claims namespace. The answer is simplicity. In classic WCF, to authorize callers based on their claims you must traverse the collection of ClaimSet instances in the AuthorizationContext (shown earlier), which is a cumbersome programming model. The Geneva Framework improves upon this model by removing some of the complexity related to the ClaimSet and adding necessary features to the Claim type.
The ClaimSet type groups claims by issuer. This model has some limitations such as the inability to merge two ClaimSet instances if the issuers are different. It isn't possible to inspect a single collection of all claims. The Geneva Framework, on the other hand, exposes a single collection of claims through ClaimsIdentity that can be inspected during authorization. Once the runtime has established that claims from a particular issuer can be trusted (because the issuer is trusted), it is the Claim type and value that matters to authorization code—not the issuer.
The collection of claims produced by the Geneva Framework use a new Claim type from the Microsoft.IdentityModel.Claims namespace. This Claim type has an Issuer property, so you can still perform a LINQ query over the claims collection to gather all claims from a particular issuer, if you really want to.
You'll most likely also want to know why the Issuer property for a claim is a simple String type instead of a ClaimSet. The Issuer property in WCF is a recursive property—each issuer is defined by a ClaimSet, which may have an issuer defined by another ClaimSet, and so on up to the root (self-signed) issuer. Not only is this a difficult programming model to work with, but this chain is not truly important at run time once you have established that you trust the immediate issuer during token authentication. For this reason, the Geneva Framework changed the Issuer property to be a string representation of the issuer. It is still the token signature that is used to establish trust—the string is merely a way to describe the issuer in a friendly manner.
You may also be surprised to see that the AuthorizationContext is cleared when the Geneva Framework is enabled. Claims-based security features of WCF rely specifically on the AuthorizationContext, which provides access to available claim sets for the request, extracted from trusted security tokens. Unfortunately, the AuthorizationContext has no meaning outside of WCF services, which bifurcates the claims programming model.
The goal of the Geneva Framework is to make it easier to support claims-based security in all .NET-based applications, while leveraging existing facets of .NET security. By assigning a ClaimsPrincipal to the request thread, existing role-based security implementations can be easily migrated to claims-based security. In addition, this provides a consistent experience for WCF services and ASP.NET applications. Another benefit of working with security principals is that when new threads are created, they are initialized with the same security principal as the originating thread. AuthorizationContext, on the other hand, is not automatically passed to new threads.
As a result, the Geneva Framework clears the AuthorizationContext under the assumption that if you enable the Geneva Framework for your WCF services, you are no longer relying on claim sets in the AuthorizationContext to perform claims-based security checks. And, therefore, WCF services that currently leverage custom authorization policies to populate the AuthorizationContext and to create the security principal for the request thread must be migrated when the Geneva Framework is enabled.

Migrate Applications to the Geneva Framework
After reading the aforementioned white paper for developers and this article, you should have a good understanding of the requirements for building claims-based WCF services and ASP.NET applications using the Geneva Framework. Let me summarize the impact of this on both WCF and ASP.NET.
To enable the Geneva Framework for your WCF services requires, at a minimum, the following steps: exposing an endpoint for the service using one of the federated security bindings; initializing the ServiceHost for the Geneva Framework; providing a list of trusted issuers; configuring the role claim type according to your claims-based security model; and applying permission demands to operations to authorize calls against role claims. WCF services that are already claims-based will require changes, since claims will be accessible directly through the security principal attached to the request thread, no longer through the AuthorizationContext.
ASP.NET applications can leverage the Geneva Framework for implementing federated security scenarios that delegate authentication to an STS, and to initialize the request thread with claims from trusted issuers—accessible through ClaimsPrincipal and ClaimsIdentity instances. Applications that are already role-based will expect claims that match those roles during the move to a claims-based scheme, but new applications have the option of working with more granular claims such as permissions discussed in this article. In either case, the ASP.NET login controls and other role-based security permission demands will look to the ClaimsPrincipal attached to the request thread to authorize calls. In a future article, I will dive deeper into the Geneva Framework to explore building a custom STS for your claims-based and federated security scenarios.

Michele Leroux Bustamante is Chief Architect of IDesign Inc., Microsoft Regional Director for San Diego, and a Microsoft MVP for Connected Systems. Her latest book is Learning WCF. Reach her at mlb@idesign.net or visit idesign.net. Michele blogs at dasblonde.net.


Editor's Note
I Am The Business
Howard Dierking

I'm writing this month's editor's note while at an open spaces conference in Austin, Texas. You may remember the same conference last year that kicked off a firestorm in the blogosphere. Many of the participants from that conference, which was called ALT .NET, have returned to Austin this year to talk about continuous improvement in software development.
Unlike last year, where the focus was on sharing techniques related to design and code, this year's meeting is focused on the methods and philosophies behind measuring and managing development projects. There is quite a bit of interest in incorporating ideas and methods from other disciplines, such as manufacturing and engineering, and there is a growing interest in trying to understand development efforts from a business perspective. I've heard terms like ROI used here so often, I had to walk outside and double-check the sign to make sure I was at the right conference.
While all of this discussion is beneficial for us as a community, what has been most meaningful for me is the realization of just how dramatically my own perspectives have changed. A year ago, I remember lamenting the fact that my day-to-day job seemed to be moving farther away from the technology—away from everything with which I had experience and confidence, and away from the people with whom I shared so many common stories and frustrations.
However, as I sat in the back of the room last night listening to the discussion about techniques, successes, and frustrations related to the friction between developers and the business, I realized something for the first time: I am the business. And I'm OK with that.
You may be surprised that it took me a year to come to such a simple conclusion, but it's hard to let go of such a large component of one's identity. In my job with MSDN Magazine, I have two main priorities: strategy and management (both budget and people). Part of our strategy includes software development projects, to be used internally and externally. And those projects are usually funded out of my budget—so from my perspective, characteristics like maintainability are not end goals worthy of funding. This is not to say that maintainability is never worth funding—just that it has to be driven by larger, more measurable business goals.
Speaking of maintainability, when thinking of such good practices, my sense is that we tend to elevate them to the level of end goals because we understand the problems they solve, and we are not nearly as comfortable with the overall business context. Therefore, when we don't understand the nature of the business that our software will support, our natural tendency is to generalize the business and focus on the non-functional "goodness" of the system. This creates an immediate break between the development group and the business that the software is supposed to be helping, and it ultimately leads to the failure of so many projects.
The simple fact that developers are meeting to talk about things such as ROI is a great step to start addressing this pattern. I also think there are some creative methods that can be adapted from other industries and applied to software projects. For example, I'm considering having my developers actually do the job of my users in lieu of a formal "requirements interview." This may help to avoid some assumption-making in the requirements-gathering process.
In the end, I think it takes both successful projects and failures to mature as a profession. That said, I'm grateful that there are developers thinking proactively about moving us along with more wins than losses.

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Mukundan Agaram, Spotty Bowles, Paul Despe, Elisa Flasko, Buck Hodges, Bryce Jonasson, Irena Kennedy, Ravi Krishnamurthy, Sesha Mani, Beth Massi, Dwayne Need, Michael Puleio, Karl Reinsch, Gerhard Schneider, Cliff Simpkins, and Don Smith.


Toolbox
Static Analysis Tools For .NET, Matt Berseth’s Blog
Scott Mitchell

All prices were confirmed October 30, 2008, and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

Contents
Improve Software Quality with Static Code Analysis Tools
Blogs of Note

Improve Software Quality with Static Code Analysis Tools
Many software teams use code reviews to ensure that developers are writing correct, secure code that adheres to the company's design guidelines. These guidelines might outline naming conventions, patterns to use for accessing data or other external resources, and so on. Many aspects of the code review process are rather mechanical and can be automated. Static code analysis tools scan through source code or intermediate code and search for violations of defined design guideline rules.
One such static analysis tool for applications within the Microsoft .NET Framework is FxCop (version 1.36), a free tool created by Microsoft. FxCop analyzes the intermediate code of a compiled .NET assembly and provides suggestions for design, security, and performance improvements. By default, FxCop analyzes an assembly based on the rules set forth by Design Guidelines for Developing Class Libraries. The design guideline rules are divided into nine categories, including design, globalization, performance, and security, among others. An example of a naming rule is, "Events should not have 'before' or 'after' prefix." If FxCop identifies an event named BeforeUpdate, it will recommend replacing BeforeUpdate with a present-tense version of the event name (namely, Update). You can also plug in a custom rules class that reflects your company's internal design guidelines.
To analyze an assembly, launch FxCop, create a new project, and add the assembly to the project. FxCop shows the 200+ rules that are used when analyzing the assembly; you may turn off existing rules or add your own. Click the Analyze button to begin the analysis. After enumerating the types, classes, methods, and members of your assembly, FxCop displays the analysis results, which list the offending code and the rule that was violated. Select a result for a more detailed description and solution.
FxCop is available as a standalone app; it also includes a command-line implementation that makes it easy to plug into an automated build process. (Code Analysis, a tool much like FxCop, ships with Visual Studio Team System and is integrated into the Visual Studio shell.) For more on how to use FxCop, see the Bugslayer columns by John Robbins: "Bad Code? FxCop to the Rescue" and "Three Vital FXCop Rules."
FxCop Analyzes an Assembly Based on Rules Set by .NET Design Guidelines
Another static code analysis tool from Microsoft is StyleCop (version 4.3). Whereas FxCop evaluates design guidelines against intermediate code, StyleCop evaluates the style of C# source code. Style guidelines are rules that specify how source code should be formatted. They dictate whether spaces or tabs should be used for indentation and the format of for loops, if statements, and other constructs. Example StyleCop rules include: the body of for statements should be wrapped in opening and closing curly brackets; there should be white space on both sides of the = and != operators; and calls to member variables within a class must begin with "this.".
StyleCop is not integrated into Visual Studio Team System—you must install it yourself. Executing StyleCop from within Visual Studio analyzes the source code in the currently opened solution, displaying the results as warnings in the error list window. StyleCop can also be integrated with MSBuild.
While FxCop and StyleCop pinpoint rule violations, the developer is still responsible for implementing these tools' suggestions. CodeIt.Right (version 1.1) from SubMain takes static code analysis to the next level by enabling rule violations to be automatically refactored into conforming code. Like FxCop, CodeIt.Right ships with an extensive set of predefined rules, based on the design guidelines document mentioned earlier, with the ability to add custom rules. But CodeIt.Right makes it much easier to create and use custom rules.
Using custom rules in FxCop requires building and compiling a rules class and plugging it into FxCop. With CodeIt.Right, custom rules are generated from a graphical user interface; defining a new rule entails picking the base behavior pattern and then customizing a couple of properties. CodeIt.Right integrates within the shells of Visual Studio .NET 2003, Visual Studio 2005, and Visual Studio 2008, and offers a command-line implementation as well.
CodeIt.Right Updates Selected Violations to Conform to Rules
To use CodeIt.Right from within Visual Studio, choose the Start Analysis option from the CodeIt.Right menu. After the solution is analyzed, CodeIt.Right displays the results in a window within the Visual Studio IDE, listing each rule violation. This report can be exported to an XML or Microsoft Office Excel file.
CodeIt.Right's biggest benefit is the automatic code refactoring. From the results screen you can check which violations to fix and then click the Correct Checked button. All changes made by CodeIt.Right are highlighted within Visual Studio, and the automated changes can be rolled back with the click of a button.
Static code analysis tools provide a fast, automated way to ensure that your source code adheres to predefined design and style guidelines. Following such guidelines helps produce more uniform code and also can point out potential security, performance, interoperability, and globalization shortcomings. Static code analysis tools are not a replacement for human-led code reviews. Rather, they can generate a first pass of the code base and highlight areas that require more attention from a senior developer.
Price: FxCop (free); StyleCop (free); CodeIt.Right ($250 per user license).
FxCop
StyleCop
CodeIt.Right

Blogs of Note
Many Web sites have embraced AJAX, a set of technologies for building responsive user interfaces. To help with building AJAX-enabled Web applications, ASP.NET ships with the ASP.NET AJAX Framework; this framework provides the core functionality for implementing AJAX scenarios. Additionally, Microsoft has partnered with the AJAX community and released the AJAX Control Toolkit, a set of free, open-source AJAX-enabled Web controls.
Every Web developer building AJAX-enabled sites needs to check out Matt Berseth's blog. Most of the blog entries show how to create a particular Web-based UI and do so in a very detailed fashion with downloadable code, live demos, and a plethora of code snippets and screenshots. For example, in one blog entry Matt shows how to use a ListView control and the Accordion control from the AJAX Control Toolkit to create a UI that mirrors the Microsoft Office Outlook navigation bar. Another entry shows how to use the ModalPopup control with the GridView and DetailsView controls to create an editable grid whose editing interface appears in a modal window. Matt's blog also includes a demo gallery, where all of the live demos scattered throughout his blog entries are consolidated on one page.
You'll also find posts covering technologies such as Silverlight, JavaScript, LINQ, and the ASP.NET data Web controls. With new posts every couple of days, Matt's blog is a wealth of information for cutting-edge Web developers using ASP.NET.
www.mattberseth.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Thread Management In The CLR
Erika Fuentes and Eric Eilebrecht

This column is based on prerelease versions of the CLR Threading System and the Task Parallel Libraries. All information is subject to change.

Contents
Common Issues in Concurrency
Lock Contention
Synchronization
Scheduling
Types and Sizes of Workloads
Memory Hierarchy Awareness
Dangerous Tuning
Noise
Concurrency Control
A Final Note on Extracting Parallelism
The ongoing technological evolution from single-core to manycore architectures is bringing great benefits. Consider, for instance, a threaded environment where the efficient use of multiple threads can promote better performance through the use of multiple cores and parallelism, for example, an ASP.NET application using multiple threads to make many independent queries to a database.
Using multiple cores does bring about new problems, however. You'll likely see an increase in the complexity of programming and synchronization models, you'll need to control concurrency, and performance tuning and optimization will be more difficult. Additionally, many of the new factors that affect the performance and behavior of concurrent applications are not well understood. Thus, tuning for performance has become challenging, especially when the targeted optimizations are not application-specific but rather general purpose.
Threading in the CLR is an example of a concurrent environment where many factors (such as those introduced by multiple-core architectures) can influence the behavior and performance of concurrency. Lock and cache contention and excessive context switching are just a few. Ongoing work on the CLR ThreadPool aims to make it easier for developers to exploit concurrency and parallelism for higher concurrency levels and better performance by integrating the Parallel Frameworks into the runtime and internally dealing with some of the most common problems.
In this column, we will describe some of the common issues developers encounter and the important factors to be considered when tuning multithreaded managed code, particularly in multiprocessor hardware. We also describe how future changes in the ThreadPool should deal with some of these issues and help alleviate the programmers' work. Furthermore, we will discuss some important aspects of concurrency control, which can help you to understand the behavior of apps in highly multithreaded environments. We will assume that you are familiar with the basic concepts of concurrency, synchronization, and threading in the CLR.

Common Issues in Concurrency
Most software was designed for single-threaded execution. In part, this is because the programming model of single-threaded execution reduces complexity and is easier to code. Existing single-threaded applications scale poorly for use with multiple cores. Even worse, it is not straightforward to adapt the single-threaded model to multiple cores. Many of the causes are predictable—synchronization issues such as lock and resource contention, race conditions, and starvation. Other causes are not yet well understood. For example, how do you determine the best concurrency level (number of threads) at any given time for a particular type (and size) of workload?
Consider the threading system in the CLR, which was designed to operate on a single-core machine: the tasks (WorkItems) are queued in a single list protected by one lock. Each thread then sequentially consumes the items from the queue. When this is migrated to a multicore architecture, its performance degrades when there's a lot of contention on the lock for higher concurrency levels. Interestingly, in his October 2008 Windows With C++ column, "Exploring High-Performance Algorithms", Kenny Kerr said that "a well-designed algorithm on a single processor can often outperform an inefficient implementation on multiple processors," implying that simply adding more processors does not always improve performance.
From here on, we'll loosely refer to the existing threading system as the ThreadPool and to the concept of WorkItem as an item in the tasks queue (a QueueUserWorkItem, for example). In single-core machines, managing concurrency could be handled with the careful scheduling of tasks by dividing the processor's time among the threads. Now that there's more than one core, you also must consider how to fairly distribute work among the cores, consider the underlying memory hierarchy for correctness and performance, and decide how to control and exploit higher concurrency levels.
Fairness vs. Performance
The discrepancy between task fairness and performance is critical in a concurrent environment: it's nearly impossible to get fair task scheduling and overall high performance at the same time. Consider, for example, a bunch of WorkItems in a queue waiting to be run. To be fair, each WorkItem can be run (consumed) in the order in which it arrived. But this doesn't deliver optimal performance.
From an operating system's point of view, a good way to achieve great performance gains is by keeping "hot" items (recently used) in the cache to avoid long trips to main memory; however, this approach conflicts with the previous idea of consuming items in order. The best way to maintain recent information in the cache would be to dequeue the last items in the queue first.
The ThreadPool environment has focused mainly on fairness, and the target of upcoming versions is extracting better performance. Strategies such as work-stealing algorithms are being incorporated with this purpose. In a work-stealing approach, there are several queues (such as one per processor), and "cold" items in a queue can be "stolen" and run, for instance, in a different processor. This can improve the performance while preserving some fairness, which can be advantageous for many concurrent applications.

Lock Contention
Concurrent environments can experience frequent lock contention for various reasons. For example, in the ThreadPool, locks are acquired every time there is a WorkItem added to the task queue and on every attempt to dequeue. Contention happens when there are many small WorkItems and the lock has to be taken frequently: when work arrives and is queued in large bursts (because the lock is taken often during these bursts) and when there is faulty concurrency control. The ThreadPool is designed to wake up all threads anytime work becomes available, so if there are many threads but no work in the queue, and then a single WorkItem arrives, all the threads might wake up simultaneously and try to acquire the lock.
This problem can become even more complicated since the threads spin while waiting for a lock, which leads to the belief that these are busy threads; during this time, if more WorkItems are enqueued, more new threads will be added, and they will also try to get the lock—so the cycle continues. Some ASP.NET applications exhibit this problem frequently.
A workaround for this last condition was to further throttle the number of threads we released at a time. Unfortunately, this sometimes led to too few threads being released, stalling the whole ThreadPool. Ongoing work on the ThreadPool to alleviate contention consisted of removing certain locks.

Synchronization
You must optimize your design for synchronization, and the use of synchronization primitives is not trivial. You'll have to choose your synchronization primitives and decide whether to use read or write locks (depending on the frequency of each of these operations). Also, sometimes the use of a primitive is more expensive on a multicore system than it is on a single-processor machine.
An example of this is a mutex on a single processor that is performing reasonably well and can facilitate workload distribution fairness but degrades when the application is moved to a manycore environment. In that case, the fairness enforced by the mutex begins to hurt performance because work has to wait where otherwise it wouldn't—in some cases, as in the use of monitors in the Microsoft .NET Framework, you may be better off spinning rather than waiting for the lock. (For more on performance and thread synchronization, see the Concurrent Affairs column "Performance-Conscious Thread Synchronization.")

Scheduling tasks in a multicore environment is a very large topic, and its full discussion is out of the scope of this column. However, there are some questions common to all implementations. For example, how will you choose your work consumption strategy? The answer will depend on the application. You may decide that those WorkItems queued last are those that embody most recent activity and, thus, consuming those before the ones in the head of the queue may result in better performance.

Types and Sizes of Workloads
Performance can vary depending on the type (or size) of an application's workload. You'll likely notice more scheduling and synchronization issues with CPU-bound workloads, while non CPU-bound ones are more likely to affect concurrency-level control. In addition, as mentioned before, it may be difficult to scale the app for use with multiple cores.
The following example illustrates this issue. Using the ThreadPool, a simple experimental application is run on a different number of cores (on the same machine but using different process affinity masks). Figure 1 shows some representative results. The workload is changed to explore whether the problem scales well using more cores.
Figure 1 Throughput Results for Different Workloads on Increasing Number of Cores
An "experimental" workload of 0 performs better than bigger (and more realistic) workloads; in this sense, the app is not scaling properly to work better with bigger workloads on more cores. However, the lines at the bottom (light blue and dark blue) corresponding to the bigger workloads indicate that the throughput is actually increasing when more cores are added, while the throughput for the experimental empty workload shows a slight decrease. Additionally, larger workloads can experience more lock contention. As you can imagine, addressing all these pitfalls in a single, general-purpose optimization technique is difficult.

Memory Hierarchy Awareness
Taking into account that memory hierarchy is one of the most important issues for performance tuning—exploiting memory locality, especially for higher numbers of cores, is indispensable. In non-uniform memory access (NUMA) architectures, accessing memory locations in "distant" nodes is expensive and can hurt performance significantly. Understanding the underlying memory hierarchy is important for scheduling purposes. For example, a way of keeping memory accesses more localized in a multiprocessor environment would be to maintain threads working on a particular task in the same processor. The ThreadPool is not NUMA-aware; however, the addition of strategies such as work stealing inherently takes advantage of memory locality. Similar tasks tend to remain in the same node due to local queuing and dequeuing.
In general, remember that the efficient utilization of a cache can result in significant performance improvements. (For more information on memory hierarchy, see "Paradigm Shift: Design Considerations for Parallel Programming" at and "CLR Inside Out: Using Concurrency for Scalability."

Dangerous Tuning
To make things more challenging, there are cases where the introduction of some logic designed to provide some performance improvement can have the opposite effect. A typical example is when such an improvement is designed to deliver general-purpose optimizations; for some applications it just adds a lot of overhead, which in turn results in performance degradation.
Picture a case where various synchronization mechanisms are added to an app to function in multiple cores (to distribute work fairly, for instance), doing the synchronization adds complexity, which may result in unnecessary overhead for simple apps where an easier solution may have worked better. In highly concurrent applications, it is hard to identify significant tuning parameters. For example, maximizing the CPU usage does not necessarily result in throughput improvement; while in single-processor machines it does, in manycore machines it is sometimes better to not maximize the CPU usage and, instead, make threads "waste" some time.

Noise
Noise is yet another factor that makes performance tuning difficult. All the experiments presented in this discussion were run on dedicated machines (no other significantly expensive applications were running at the same time). But although there was not much noise, there were unexpected variations that were quite difficult to understand.
In real-life systems, it will often be the case that more than one application (and hence more threads) is competing for the resources. In the case of our ThreadPool, noise may even be introduced by different application domains executing at the same time.

Concurrency Control
The number of threads in an environment (concurrency level) is very important since it affects the behavior and performance of the environment and its applications. For instance, a low concurrency level with one thread per core is desirable for apps that are CPU-intensive (furthermore, the minimum number of threads in a system is usually set to be the same as the number of cores). On the other hand, high concurrency levels can result in excessive context switching or resource contention.
Figure 2 shows a typical example of how the concurrency level affects the performance of an app. (Note that the blue dots are observations captured at each concurrency level. The black line shows the fitted curve for these experiments to help us visualize the type of function that describes the behavior.) The way we measure performance in this context is with throughput, which is measured as number of completed WorkItems per second.
Figure 2 Throughput as a Function of Concurrency Level
This plot was generated using an experimental app that runs WorkItems for a certain time (each WorkItem has 100ms of execution time with 10ms of CPU usage and 90ms of waiting). The experiment was run several times on a dual-core 2GHz machine to collect the data and evaluate possible noise and variations. The theoretical maximum for the throughput on this machine is 200 completions per second at a concurrency level of 20. However, we can see that, experimentally, the throughput peaks at 150 a little bit before the concurrency level of 20 is reached, and then it degrades. The degradation is mainly due to context switching.
During this experiment, the CPU was maximized (however, after a certain point this does yield optimal throughput). A 100% CPU usage can mean that data is being pulled from memory for threads that aren't really being used; this gives the appearance that the app is being productive when in reality it's wasting CPU time. This also happens in single-processor environments, but it becomes more obvious in manycore.
As we've seen, defective concurrency control can increase the occurrence of certain problems, even more if parallelism is used. Additionally, concurrency management for multiple tasks has become even more challenging when multiple cores are present.
One example is related to the memory hierarchy issue. Trying to follow a simple rationale based on the example in Figure 2, increasing concurrency levels up to some point can be beneficial; however, in a multicore environment it is not trivial to determine where the threads are needed. For instance, if a processor is being overloaded for some reason (due to bad scheduling, for example) while the other processors are being under utilized, the overall throughput can be very low, but when new threads are added they could be assigned erroneously to the wrong processing unit.
Conversely, if the concurrency level is very high but the overall throughput is starting to degrade, we could decide to start removing threads, but the threads removed could be taken away from those processors that actually need them and whose throughput in particular could be improved by adding more threads.
Figure 3 shows how naively trying to change the concurrency levels can result in unexpected behavior. The experiment consists of a series of runs of a test application using different amounts of work to execute (WorkItems) and the execution time of each WorkItem (workload).
Figure 3 Changing Concurrency Levels Can Result in Unexpected Behavior
In the top section of Figure 3 are the results for this test running on the ThreadPool. The throughput is poor but shows some consistency along all of the trials. The bottom half of Figure 3 shows the results obtained by making some naive adjustments to the concurrency control logic to try to improve the way threads are added and removed. We can see that there are many variations in the throughput, and regardless of the amount of work, the adjustments result in what seems more like noise than better control.
Some points may appear to be improving for specific combinations of values, which suggests that different types of concurrency control adjustments are necessary depending on the type of app.
One possible approach to address this problem is to incorporate some sort of dynamic-tuning logic to adjust concurrency levels based on the application (for example, using statistical methods). Further inspection of concurrency control and tuning approaches is also out of the scope of our present discussion.
Appropriate tuning may lead to improvements in performance. The following is an encouraging example showing a similar experiment, where many factors and parameters in the ThreadPool were manually tuned specifically for this test application.
The performance results are in Figure 4. The improvement (represented by the red line) is mainly a result of adding more threads, which quickly increases the concurrency levels. In the ThreadPool (represented by the blue line), only one thread is added when increasing concurrency levels is deemed necessary—in this experiment the check happens every interval of 250ms. Moreover, after a certain point it concludes adding more threads because there won't be any further improvement.
Figure 4 Improvement Due to Manually Tuning the ThreadPool
In this experiment, WorkItems are queued, each of which does CPU work then blocks for some amount of time. Every 30 seconds the blocking time is changed: in the first 30 seconds it lasts 1ms, then 2ms, then it switches back to 1ms. At the 90 second mark, the work is stopped for 10 seconds. After this, the previous blocking cycle continues. The blue line shows performance results using the existing ThreadPool. The red line shows the improvements obtained with manual tuning of this ThreadPool.

A Final Note on Extracting Parallelism
The introduction of parallelism can improve performance but is not trivial to incorporate parallel concepts and algorithms into existing applications. (For example, there are many pitfalls, as outlined in "Concurrency Hazards: Solving 11 Likely Problems in Your Multithreaded Code.")
The Task Parallel Libraries (TPL) are extensions to the .NET Framework, developed by the Parallel Computing Platform team, that provide a way to expose concurrency and provide support to take advantage of parallelism. The TPL are currently available, together with more information about this development, on MSDN.
Of course, taking advantage of exposed parallelism is still not seamless for the average programmer since there are still many factors that need to be taken into consideration. And as we saw in Figure 1, applications don't always scale properly. The same is true when parallelizing a problem using divide-and-conquer techniques (recursive by nature). This can be tricky because these algorithms are a common source of deadlocks in the ThreadPool (which does not enable reentrant code). Finally, consider the usual case of a serial app that computes results in a certain order. When this application is parallelized, the results may be computed in a different order by the concurrent tasks. Then it is up to the programmer to sort out the results where necessary to ensure correctness and functionality.
Adjusting programs to perform in concurrent and parallel environments is not trivial. General-purpose tuning is even harder, and it cannot guarantee optimal behavior for all apps. Understanding the underlying structure of the concurrent environment can help you determine which parameters to tune for and the best programming strategies to employ. Ongoing work for future ThreadPool versions includes trying to reduce the need to do manual tuning.

Send your questions and comments to clrinout@microsoft.com.

Erika Fuentes, PhD, is a Program Manager on the CLR team where she works on the Core Operating System in concurrency. She has several academic publications in the area of statistics, adaptive systems, and scientific computing.

Eric Eilebrecht is a Senior Software Development Engineer in the CLR team, where he works on the Core Operating System. He has been at Microsoft for 11 years and had worked on back-end storage systems for Hotmail prior to joining the CLR team.


Advanced Basics
The ObservableCollection Class
Ken Getz
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Introducing the ObservableCollection Class
Examining the Sample
Examining the Code
Imagine that you're creating a Windows Forms application, and you have bound a DataGridView control to a standard List(Of Customer) data structure. You'd like to be able to keep the items in the grid synchronized with the values in the underlying data source. That is, if some other code or some other form changes a customer's data in the List, you'd like the grid to update and display the modified data.
Using Windows Forms, this feat is possible, in a very general sense. You can make the update work, but it will have severe limitations. For example, you can, under the right circumstances, see the updates immediately in the grid, but there's no easy way to add a new row to the grid when someone adds a new row to the data source. Windows Presentation Foundation (WPF) adds functionality to the Microsoft .NET Framework so that you actually can reliably keep bound controls synchronized with their data sources. Here I'll demonstrate how to use the ObservableCollection class provided by WPF.
Although the ObservableCollection class makes it possible for WPF applications to keep bound controls in sync with underlying data sources, it provides even more useful information—specifically, the ObservableCollection class also raises the CollectionChanged event when you add, remove, move, refresh, or replace an item in the collection. This feature makes it possible to react when code outside your window modifies the underlying data. You'll see how you might use this information in this month's sample application, which I'll describe next.

Introducing the ObservableCollection Class
The System.Collections.ObjectModel.ObservableCollection(Of T) class inherits from Collection(Of T), which is the base class for generic collections, and implements both the INotifyCollectionChanged and INotifyPropertyChanged interfaces. It's the INotifyCollectionChanged interface that makes the collection interesting—this is the interface that allows bound objects (and your code) to determine if the collection has changed.
It's important to note that although the ObservableCollection class broadcasts information about changes to its elements, it doesn't know or care about changes to the properties of its elements. In other words, it doesn't watch for property change notification on the items within its collection.
If you need to know if someone has changed a property of one of the items within the collection, you'll need to ensure that the items in the collection implement the INotifyPropertyChanged interface, and you'll need to manually attach property changed event handlers for those objects. No matter how you change properties of objects within the collection, the collection's PropertyChanged event will not fire. As a matter of fact, the ObservableCollection's PropertyChanged event handler is protected—you can't even react to it unless you inherit from the class and expose it yourself. You could, of course, handle the PropertyChanged event for each item within the collection from your inherited collection, although in the sample application I take a simpler approach, allowing the client application to handle individual items' change events.
If you disregard inherited and protected members (assuming that you are already familiar with the base Collection class, from which all of the members of the ObservableCollection class derive), the only interesting members left are the Move method (which allows you to move a member to a new location within the collection) and the CollectionChanged event (which broadcasts information about changes to the contents of the collection). Before reading further, you might want to download and install the sample WPF application that demonstrates these features.
Examining the Sample
The sample solution, ObservableCollectionTest, contains a class named CustomerList, which inherits from ObservableCollection (see Figure 1). As you might imagine, the CustomerList class exposes an ObservableCollection instance containing Customer objects. If you examine the code, however, you'll see that the class exposes only one list, so that multiple consumers of the class each retrieve a reference to the same collection. (That's the point of this particular demonstration, and isn't necessary for other applications.) The class provides a private constructor, so the only way to retrieve an instance of the class is to call the shared GetList method, which hands out the existing collection instance:
Private Shared list As New CustomerList

Public Shared Function GetList() As CustomerList
  Return list
End Function
Figure 1 CustomerList
System.Collections.ObjectModel
Imports System.ComponentModel

Public Class CustomerList
  Inherits ObservableCollection(Of Customer)

  Private Shared list As New CustomerList

  Public Shared Function GetList() As CustomerList
    Return list
  End Function

  Private Sub New()
    ' Make the constructor private, enforcing the "factory" concept
    ' the only way to create an instance of this class is by calling
    ' the GetList method.
    AddItems()
  End Sub

  Public Shared Sub Reset()
    list.ClearItems()
    list.AddItems()
  End Sub

  Private Sub AddItems()
    Add(New Customer("Maria Anders"))
    Add(New Customer("Ana Trujillo"))
    Add(New Customer("Antonio Moreno"))
  End Sub
End Class
The private constructor calls the AddItems method; the public shared Reset method clears the list and then calls the AddItems method. Either way, the result is three customers in the collection:
Private Sub AddItems()
  Add(New Customer("Maria Anders"))
  Add(New Customer("Ana Trujillo"))
  Add(New Customer("Antonio Moreno"))
End Sub
The Customer class, for this example, is extremely simple (that is, just complex enough to demonstrate the necessary features). The class, shown in Figure 2, contains only a Name property and wouldn't even be worth discussing if it weren't for the fact that the class implements the INotifyPropertyChanged interface, which enables consumers of instances of the class (including data-bound controls) to be notified when property values change.
Figure 2 The Customer Class with the PropertyChanged Event
Imports System.ComponentModel

Public Class Customer
  Implements INotifyPropertyChanged

  Public Event PropertyChanged( _
    ByVal sender As Object, _
    ByVal e As PropertyChangedEventArgs) _
    Implements INotifyPropertyChanged.PropertyChanged

  Protected Overridable Sub OnPropertyChanged( _
    ByVal PropertyName As String)

    ' Raise the event, and make this procedure
    ' overridable, should someone want to inherit from
    ' this class and override this behavior:
    RaiseEvent PropertyChanged( _
      Me, New PropertyChangedEventArgs(PropertyName))
  End Sub

  Public Sub New(ByVal Name As String)
    ' Set the backing field so that you don't raise the
    ' PropertyChanged event when you first create the Customer.
    _name = Name
  End Sub

  Private _name As String
  Public Property Name() As String
    Get
      Return _name
    End Get
    Set(ByVal value As String)
      If _name <> value Then
        _name = value
        OnPropertyChanged("Name")
      End If
    End Set
  End Property
End Class
When a class implements the INotifyPropertyChanged interface, it must supply the PropertyChanged event:
Public Event PropertyChanged( _
  ByVal sender As Object, _
  ByVal e As PropertyChangedEventArgs) _
  Implements INotifyPropertyChanged.PropertyChanged
In order to raise the event using the standard .NET design pattern, the Customer class includes a protected, overridable OnPropertyChanged procedure, which raises the event:
Protected Overridable Sub OnPropertyChanged( _
  ByVal PropertyName As String)

  ' Raise the event, and make this procedure
  ' overridable, should someone want to inherit from
  ' this class and override this behavior:
  RaiseEvent PropertyChanged( _
    Me, New PropertyChangedEventArgs(PropertyName))
End Sub
Then, within the definition for the Name property, the property setter calls the OnPropertyChanged method if the new value is different from the current value of the property:
Private _name As String
   Public Property Name() As String
     Get
       Return _name
     End Get
     Set(ByVal value As String)
       If _name <> value Then
         _name = value
         OnPropertyChanged("Name")
       End If
     End Set
   End Property
Given the PropertyChanged event that the class raises, it's possible for code that uses the class, or a collection of instances of the class, to react to the PropertyChanged event and take action based on the fact that the property changed. (Please note that the PropertyChangedEventArgs class adds only a PropertyName property to the standard event arguments and doesn't provide any information on either the old or the new value of the property. The sample application works around this limitation, as you'll see, to at least determine the new value of the changed property.)
The sample also contains a single WPF window named MainWindow, as shown in Figure 3. The only important detail in the markup for the window lies in the definition for the ListBox control, which includes declarative data binding for the ItemsSource property of the control. The binding indicates that the control should get its data from a property of the MainWindow class named Data and that it should display the Name property of each item within the Data property:
<ListBox
  DisplayMemberPath="Name"
  ItemsSource=
    "{Binding ElementName=MainWindow, Path=Data}"
  Grid.Column="3" Grid.RowSpan="3" Name="ItemListBox"
  Margin="5" />
Figure 3 The Sample WPF Window
The codebehind class for MainWindow includes the following declaration:
Public WithEvents Data As CustomerList = CustomerList.GetList()
This code exposes the contents of a CustomerList instance to the window, as you see in Figure 3.
Before examining the remainder of the code within the codebehind class, you should stop at this point and experiment with the application. Because the ListBox on the window has been bound to a class that inherits from ObservableCollection, you would expect that the listbox should always stay current with the contents of the collection, and the demonstration window proves this to be true.
In addition, the sample displays two separate instances of the main window, and because the ListBox controls on both windows are bound to the same ObservableCollection instance, changes you make in one window appear in both windows. To open two instances of the window, the Application.xaml file contains the following markup, indicating that the application should start by running the code in the Application_Startup procedure:
<Application x:Class="Application"
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  Startup="Application_Startup">

  <Application.Resources>

  </Application.Resources>
</Application>
The Application.xaml.vb codebehind file contains the following startup code, which creates two instances of MainWindow.xaml, each with its own title:
Private Sub Application_Startup( _
  ByVal sender As System.Object, _
  ByVal e As System.Windows.StartupEventArgs)

  Dim window As New MainWindow
  window.Title = "Observable Collection 1"
  window.Show()

  window = New MainWindow
  window.Title = "Observable Collection 2"
  window.Show()
End Sub
Follow these steps to give the sample application a workout.
  1. Within Visual Studio 2008, load and run the sample application. You'll see two instances of the same window.
  2. Click to open the combobox on the left side of the window. Note that the control contains the numbers 0 through 2, corresponding to the three current customers. Choose 1, which selects Ana Trujillo and copies her name into the textbox.
  3. In one window, click Delete. Ana Trujillo disappears from both windows, because the two ListBox controls are bound to the same ObservableCollection instance, and the binding causes updates to appear immediately. Open the combobox again, and note that it now indicates that there are only two customers. Try it in both windows to verify that both instances are up-to-date.
  4. Click Delete two more times, removing all the customers. Click Reset Data to refill the lists in both windows.
  5. In the textbox next to the Add New Item button, enter your own name and click Add New Item. The new name appears in both ListBox controls. Click to open the combobox and verify that it now contains numbers from 0 to 3 (one for each of the four customers). Verify that the combobox has changed on both windows—obviously, both windows' classes received the event that indicated that the collection had changed.
  6. In the ListBox in one window, select a name. In the lower textbox on the left, modify the name and click Change. First, you'll see an alert indicating that you have changed the property, and once you dismiss it, you'll see that the name has changed in both windows (see Figure 4).
Figure 4 Capturing Change Events for Data in the Collection
Except for the alert that appeared when you changed the customer name and the ComboBox control whose items change to match the number of customers in the collection, all the code in the sample window deals with the window's user interface. That is, all the work done to keep the ListBox control in sync with the ObservableCollection instance happens for "free" and is managed by WPF.
When you add a new item, the ListBox automatically displays the full list. When you change an item, the ListBox automatically displays the modified list. When you delete items, the ListBox maintains full fidelity with the underlying collection. In other words, in terms of binding the ObservableCollection class to controls in WPF, you can easily say "It just works."
Actually, there is some "magic" going on under the covers. When you bind the ListBox to the ObservableCollection, WPF actually creates a CollectionView instance, which presents a view of the data that handles grouping, sorting, filtering, and so on. You're seeing that default view of the collection within the ListBox controls. You could create multiple CollectionView instances based on the same collection and display the data in different ways (sorted, for example) within one of the ListBox controls. Although it's outside the scope of this discussion, it's worth investigating the CollectionView class if you need to display multiple views of the same collection. For more information, please look at the CollectionView Class information on MSDN.

Examining the Code
The MainWindow class defines the CustomerList instance using the WithEvents keyword so that the code can handle events of the ObservableCollection list without requiring the code to add handlers manually:
Public WithEvents Data As CustomerList = CustomerList.GetList()
In the Change Event Handlers region of the code, you'll find the CollectionChanged event handler, which verifies that you have either added or removed an item from the collection. If so, the code sets the combobox's data source and enables buttons on the window accordingly, as you see in Figure 5.
Figure 5 Check for Collection Changed
Private Sub Data_CollectionChanged( _
  ByVal sender As Object, _
  ByVal e As NotifyCollectionChangedEventArgs) _
  Handles Data.CollectionChanged

  ' Because the collection raises this event, you can modify your user
  ' interface on any window that displays controls bound to the data. On
  ' both windows, if you add or remove an item, all the controls update
  ' to indicate the new collection!

  ' Did you add or remove an item in the collection?
  If e.Action = NotifyCollectionChangedAction.Add Or _
    e.Action = NotifyCollectionChangedAction.Remove Then

    ' Set the list of integers in the combo box:
    SetComboDataSource()

    ' Enable buttons as necessary:
    EnableButtons()
  End If
End Sub
The important part of this simple code lies in the NotifyCollectionChangedEventArgs parameter. This object provides information about what changed within the collection and also provides five interesting properties, which are listed in Figure 6.
Figure 6 NotifyCollectionChangedEventArgs Parameters
Parameter | Description
Action | Retrieves information about the action that caused the event. This property contains a NotifyCollectionChangedAction value, which can be either Add, Remove, Replace, Move, or Reset.
NewItems | Retrieves a list of the new items that were involved in the change to the collection.
NewStartingIndex | Retrieves the index of the collection at which the change occurred.
OldItems | Retrieves a list of the old items that were affected by the Replace, Remove, or Move action.
OldStartingIndex | Retrieves the index of the collection at which the Replace, Remove, or Move action occurred.
Given all this information, your event handler can determine exactly what occurred within the collection that triggered the event. The sample code merely uses the Action property, updating the list of integers in the ComboBox control if the collection changed size:
If e.Action = NotifyCollectionChangedAction.Add Or _
  e.Action = NotifyCollectionChangedAction.Remove Then
Although it's not pertinent to the discussion of the ObservableCollection class, it is interesting to see how the code fills in the list of indexes in the ComboBox control:
Private Sub SetComboDataSource()
  ' Set the list of integers shown in the
  ' combo box:
  ItemComboBox.ItemsSource = _
    Enumerable.Range(0, Data.Count)
End Sub
Rather than performing some loop to generate a list containing integers between 0 and the highest-numbered index in the collection, this code simply calls the Enumerable.Range method to retrieve a collection of integers starting at 0 and containing Data.Count values. The code sets the ItemsSource property of the ComboBox control to the returned collection—and that's all there is to it! (If you are looking for more information on the Enumerable class, you should read my previous two Advanced Basics columns: The LINQ Enumerable Class, Part 1 and The LINQ Enumerable Class, Part 2.)
In order for the sample application to be notified if you change properties of an item within the collection, you have to write a little more code. Each of the properties within the class raises the PropertyChanged event each time some code changes the value of the property. (Of course, it's up to the author of the specific class to make sure that property changes raise the PropertyChanged event, because it doesn't happen automatically. The Customer class's Name property, as you've seen, does this.)
Within the MainWindow class, you'll find the HookupChangeEventHandler procedure, which hooks up the PropertyChanged event for an individual Customer object:
Private Sub HookupChangeEventHandler(ByVal cust As Customer)
  ' Add a PropertyChanged event handler for
  ' the specified Customer instance:
  AddHandler cust.PropertyChanged, _
    AddressOf HandlePropertyChanged
End Sub
The HookupChangeEventHandlers procedure hooks up event handlers for each Customer within the ObservableCollection of customers, like so:
Private Sub HookupChangeEventHandlers()
  For Each cust As Customer In Data
    HookupChangeEventHandler(cust)
  Next
End Sub
When the window loads or you click the Reset button, the code calls the HookupChangeEventHandlers procedure. When you click Delete, the window both removes the item from the collection and removes the event handler:
' From DeleteItemButton_Click

Dim index As Integer = ItemComboBox.SelectedIndex
If index >= 0 Then
  RemoveHandler Data.Item(index).PropertyChanged, _
    AddressOf HandlePropertyChanged

Data.RemoveAt(index)
When you click Add New Item, the code creates the new customer, and it also hooks up its PropertyChanged event:
' From NewItemButton_Click

cust = New Customer(NewItemTextBox.Text)
HookupChangeEventHandler(cust)
Data.Add(cust)
Of course, because the window binds the ListBox control to the ObservableCollection instance, all these changes appear automatically, without any code support, in both instances of the window. In fact, you only need the code support to programmatically add and remove customers from the list, and to hook up and react to changes within an individual customer.
When you do change the value of a property within the Customer class, the client application receives the notification via the PropertyChanged event handler. Note that the HandlePropertyChanged procedure (shown in Figure 7) contains the application's most complex code. Because the change notification supplies only the name of the changed property, you must remember that it's up to your code to retrieve the current value of the property, should you have some need for it.
Figure 7 HandlePropertyChanged Using Reflection
Private Sub HandlePropertyChanged( _
ByVal sender As Object, _
ByVal e As PropertyChangedEventArgs)

  ' In this particular application, you only want to bother with this
  ' code for the first window, although both will run the code. In this
  ' case, if the event was raised by the window whose title is
  ' "Observable Collection 1" then process the event:
  If Me.Title.EndsWith("1") Then
    Dim propName As String = e.PropertyName
    Dim myCustomer As Customer = CType(sender, Customer)

    ' Unfortunately, no one hands you the old property value, or the new
    ' property value. You can use Reflection to retrieve the new property
    ' value, given the object that raised the event and the name of the
    ' property:
    Dim propInfo As System.Reflection.PropertyInfo = _
      GetType(Customer).GetProperty(propName)
    Dim value As Object = _
      propInfo.GetValue(myCustomer, Nothing)

    MessageBox.Show(String.Format( _
      "You changed the property '{0}' to '{1}'", _
      propName, value))
  End If
End Sub
The procedure starts by ensuring that the code only runs once—because you have two instances of the window open, the code would otherwise run for each instance, and there's no point in seeing the alert twice. The code simply checks the final character of the title (assuming that you haven't changed the window's Title property) and restricts the behavior to a single window:
If Me.Title.EndsWith("1") Then
  'Code removed here…
End If
The code retrieves and stores the name of the property that was changed and a reference to the object that raised the event—that is, the current customer:
Dim propName As String = e.PropertyName
Dim myCustomer As Customer = CType(sender, Customer)
The code then uses Reflection to retrieve a System.Reflection.PropertyInfo instance, given the name of the property and the type:
Dim propInfo As System.Reflection.PropertyInfo = _
  GetType(Customer).GetProperty(propName)
Given the PropertyInfo object and a particular Customer instance, the code can then retrieve the current value of the property:
Dim value As Object = _
  propInfo.GetValue(myCustomer, Nothing)
The remainder of the code in the application maintains the user interface, including enabling/disabling buttons, keeping the combobox and listbox in sync, and so on.
Although this application made use of the binding support provided by the ObservableCollection class and also reacted to its CollectionChanged event in order to update the user interface, you needn't use the class this way. Because it notifies listeners that its contents have changed, you can replace any List or Collection instance that you use with an ObservableCollection instance (even if you're not creating a WPF application) and then hook up event handlers to notify clients that the collection's contents have changed.
Just as the sample window updated the list of integers corresponding to collection indexes when the collection changed size, you could react in any necessary way to changes in the collection from a client class. Remember, however, that the collection won't tell you, on its own, whether the properties of its child elements have changed. You'll need to hook up event handlers within the client, so the client gets notified when the properties of a child element within the collection change.
Also remember that the rich data binding support that you saw in the sample application only works within WPF applications—if you're creating a Windows Forms application, you'll still need to manually refresh the binding for any control that's bound to an ObservableCollection instance when the collection changes. On the other hand, because the collection notifies you when it changes, at least it's now possible to make this happen.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a Senior Consultant with MCW Technologies and a courseware author for AppDev (appdev.com). He is coauthor of ASP .NET Developers Jumpstart, Access Developer's Handbook, and VBA Developer's Handbook, 2nd Edition. Reach him at keng@mcwtech.com.


Cutting Edge
ASP.NET Presentation Patterns
Dino Esposito

This column is based on a prerelease version of the ASP.NET MVC Framework. All information is subject to change.
Contents
Inside an ASP.NET Event Handler
The Original MVC Pattern
Model2: A Web Variation of MVC
ASP.NET MVC Framework vs. Manual MVC
The MVP Pattern
The Page Controller Pattern
In a layered Web app, the presentation layer is just as important as navigation logic, business logic, and data access. While you should try to keep the presentation layer (mostly the presentation logic) independent from the UI technology and platform, it's not always easy to do. Design patterns can help.
In this month's column, I'll examine some design patterns for building an ASP.NET presentation layer. I'll start with the Model-View-Controller (MVC) pattern—the root of all patterns specifically targeted to the UI—and discuss its applicability to ASP.NET beyond the ASP.NET MVC Framework.

Inside an ASP.NET Event Handler
An ASP.NET presentation layer is composed mostly of .aspx pages that are orchestrated by the HTTP runtime environment. A typical .aspx page, at some point, may place an HTTP request following a certain user's action (such as a button click or a list selection). In ASP.NET Web Forms programming, these events are usually handled by methods—event handlers—written in the codebehind class of the page. At first glance, it would appear that there's a direct connection between the user action and the system's reaction. While this may be true in desktop applications, it is not the case in ASP.NET.
In ASP.NET, a lot of things happen in the time between a user click and the display of an updated page—regardless of whether or not you're using AJAX. The flow of information is shorter, however, in ASP.NET applications that you build with the ASP.NET MVC Framework.
Let's consider a button click event. As a developer, you handle it by writing some code in the Button control's Click event handler. The code goes in the page's codebehind, like so:
void Button1_Click(object sender, EventArgs e)
{
    // Perform any required action
}
You could place all the code for the required action here. Or, you could group all of this code in a static or instance method exposed by an object:
void Button1_Click(object sender, EventArgs e)
{
    // Static method bound to the user action
    ActionManager.Button1Clicked();
}
You could also organize the code that responds to the user's activity in controllers and actions:
void Button1_Click(object sender, EventArgs e)
   {
       // Static method bound to the user action
       ThisPageController controller = new
       ThisPageController();
       controller.Button1Clicked();
   }
I'm assuming that each page has its own controller component and that each controller has a list of public methods logically bound to a possible user action.
How would you let the controller know the state of the page and how the new view is generated? Button1_Click is invoked on the server after the user has clicked the button and the ASP.NET runtime has received and processed the corresponding HTTP request. The Button1_Click method is exposed by a class derived from System.Web.UI.Page. This class has access to UI controls.
Here's a simple change to the controller's signature that would enable the controller to access any public element in the page:
void Button1_Click(object sender, EventArgs e)
{
    // Static method bound to the user action
    ThisPageController controller = new ThisPageController(this);
    controller.Button1Clicked();
}
The controller's signature now receives a reference to the current ASP.NET page—the view. The controller method performs its work and then simply updates server controls. The standard page lifecycle of ASP.NET will do the rest and produce the HTML for the browser.
The Page Lifecycle
Figure 1 shows the main steps in the page lifecycle. Whenever an ASP.NET request hits the Web server and an HTTP handler is found to service it, the steps outlined in the figure are performed. As you can see, the code for a simple click of a button is only a small fraction of the whole ASP.NET infrastructure. You'll see in a moment that the page lifecycle represents a fraction of the work that the ASP.NET runtime executes to service each request.
Figure 1 The ASP.NET Page Lifecycle
Before the code in Button1_Click can execute, the ASP.NET runtime creates an instance of the Page class that holds the expected behavior for the requested URL. The Page class processes the request according to the steps in Figure 1 and uses your event handlers where appropriate.
In the Button1Clicked method in a controller class, you should be able to access any of the controls in the page with the same syntax you would use in a plain codebehind event handler. But members that reference UI controls (say, TextBox1) are not public members on the Page class. How can you make them accessible to a controller? You can implement a custom interface on the codebehind class that includes public properties for the UI controls you intend to manipulate from within the controller. Here's an example:
public interface IThisPageView
{
    TextBox TextBox1Property;
    ...
}

// In the codebehind class
public TextBox TextBox1Property
{
    get {return TextBox1;}
}
Figure 2 provides an example of the sample controller. In Button1Clicked, you do your regular ASP.NET Web Forms programming and determine the next view by simply updating the state of server controls. The logic of the page is now encapsulated in a separate class that can be developed separately from the page.
Figure 2 A Sample Class Acting as the Controller for a Page
public class ThisPageController
{
   protected IThisPageView thePage;

   public ThisPageController(IThisPageView page)
   {
      thePage = page;
   }

   public void Button1Clicked()
   {
       // Get the text currently stored in control TextBox1
       string text = thePage.TextBox1Property.Text;

       // Do something: i.e., turn to upper case
       string newText = text.ToUpper();

       // Update the text in control TextBox1
       _thePage.TextBox1Property.Text = newText;
   }
}
This solution, though, is still tightly coupled to the controls in the page. Let's consider a refinement that refers to the Model-View-Presenter (MVP) pattern. I'll briefly recall the basics of the MVC and MVP patterns.

The Original MVC Pattern
The MVC pattern was devised back in 1979 as an approach to move away from catch-all, autonomous front ends. An autonomous view is a class that displays content, maintains state information for the view, and incorporates the full logic to handle any user actions from start to finish.
This code snippet lays the groundwork for an autonomous view:
void Button1_Click(object sender, EventArgs e)
   {
       // Perform any required action. All the code you
       // need to handle the event goes here. A page
       // built in this way is an autonomous view.
       // MVC fights autonomous views.
   }
With such a monolithic view, you can hardly build a testable presentation layer that's separate from the underlying layers. Applying the concept of Separation of Concerns (SoC) helps you to get the right combination of low coupling and high cohesion for any component, because it causes you to keep the various functionalities in their own layers rather than intermingled. Second, SoC makes it easier to implement a navigational workflow to decide which page comes next.
The original paper describing the MVC approach, "How to Use Model-View-Controller," is available online. If you've never read it, I suggest you have a look at it regardless of your familiarity with the MVC philosophy. Today, MVC is considered a pattern for building a presentation layer, but it was originally meant for building complete applications. It's helpful to look at it in this light.
MVC is a loosely defined pattern that gives the architect much discretion over implementation details. This is probably why so many variations of MVC exist.
MVC splits the app into three parts—the model, the view, and the controller. The model refers to the application data, manages the application's functionalities, and notifies the view of state changes. The view is concerned with the content to display to users. Finally, the controller maps user gestures to actions on the model and updates the current view or selects the next view. These three actors are often referred to as the MVC triad. Figure 3 provides a graphical comparison between autonomous views and MVC applications.
Figure 3 From Autonomous View to MVC
In MVC, the user interacts with the view, and actions (such as a button click) are captured by the view and forwarded to the controller. The controller decides what to do and does it through an interaction with the model. The model is essentially the business layer plus some extra capabilities. In particular, the model notifies the view about changes that may require an update of the UI.
The model doesn't know any details of the view, but between model and view there's an "observer" relationship. In other words, the view holds a reference to the model and registers itself with the model to receive a notification of changes. When the notification is received, the view gets fresh data from the model and updates the UI. Figure 4 illustrates a sequence diagram of an MVC interaction. Note that in some MVC implementations, it's the controller that notifies the view of the changes.
Figure 4 A Standard MVC Interaction

Model2: A Web Variation of MVC
MVC was devised to target desktop applications, but because of its relatively loose formulation, it was easily adapted for the Web. A popular Web variation of the MVC pattern is Model2. Model2 is the historical name of the pattern that fuels the ASP.NET MVC Framework today.
The Model2 pattern exposes the view to users through the browser. User actions are captured by the browser and transformed into a new HTTP request. In this way, requests flow from the browser to a special component (referred to as the front controller) sitting within the Web server. The front controller coordinates all of the requests that are made to the Web application.
In ASP.NET, the front controller takes the form of the URL-routing HTTP module. The HTTP module captures the request and routes it to the appropriate controller for the requested action to take place. As the action method returns, the controller orders the view to refresh and passes fresh data for the new UI. The output of the view is captured by the front controller HTTP module and sent back to the browser.
Figure 5 shows the sequence diagram for a typical Model2 interaction. You should note that the diagram includes only the general steps. For example, in the ASP.NET MVC implementation of the pattern, the URL routing component does some work in addition to using the controller.
Figure 5 The Model2 Interaction as in the ASP.NET MVC Framework
A few differences exist between the original MVC and Model2. As you can see, there's no contact between the view and the model as in the MVC original formulation based on the aforementioned "observer" relationship. The user action is not captured and handled by the view. The controller renders the view and explicitly passes display data to it.
In ASP.NET, when you say MVC you should specify whether you mean Model2 or a manual implementation of MVC, as briefly explained in the beginning of this column. (If you'd like to read more on the MSDN Magazine article ASP.NET MVC Framework.

ASP.NET MVC Framework vs. Manual MVC
Is there any difference at all between using the ASP.NET MVC Framework and rolling your own implementation of the MVC pattern in order to have a controller class for each page (or for a group of pages)? In terms of SoC, I don't see any significant differences. In both cases, you have a controller class that is neatly separated from the view and the model that represents the gateway to the business or service layer.
In terms of testability, the Model2 pattern is preferable over the original formulation of the MVC pattern because of the neat separation it forces between view and model and because of an extremely thin view. This aspect alone makes code written for the ASP.NET MVC Framework extremely effective to test. Furthermore, in Model2 there's a sort of loose contract that can be established between the view and the controller. In the ASP.NET MVC Framework, this contract is represented by the ViewData container object or, better yet, by the ViewPage<T> base class for view classes.
Compared with ASP.NET Web Forms, a Model2 implementation is also faster to execute the requested action. In a Model2 implementation, all you need is a component that processes the HTTP request and invokes a controller. There's no need to create page classes dynamically; the lookup for the controller is based on a much simpler algorithm than the lookup of an HTTP handler in a Web Forms scenario. Likewise, you don't need the page lifecycle, the viewstate, or server controls. In a Model2 implementation, such as the ASP.NET MVC Framework, you have a much more agile runtime environment. Let's explore it further.
In an MVC scenario implemented over the Web Forms programming model, any user action originates an HTTP post. The Web server captures these requests and maps them to a page class. The Page class goes through its own lifecycle, as shown in Figure 1. Next, the Page class figures out the right controller and invokes a method. The execution of the method modifies the model. The remainder of the page lifecycle involves refreshing the view.
In a Model2 scenario implemented with the ASP.NET MVC Framework, the generation of the next view is a much simpler process. The URL routing module parses the URL, determines from it the controller to use, and invokes an action on it. Then the controller collects new data for the new view and passes this information on. The view assembles some HTML that is then returned to the browser as the response to the captured request. The view is a passive subject that needs virtually no testing. Instead, you focus your testing effort on the controller.
Model2 is an excellent alternative for improving the testability of an application. However, what if you still need the Web Forms runtime environment with its support for Session, Cache, server controls, and even viewstate that you get with the MVC pattern? Does this mean that you have to abandon SoC and testability? Certainly not. You can opt for the other variation of MVC that I mentioned at the outset—the MVP pattern. It applies to Web applications without requiring an ad hoc runtime.

The MVP Pattern
The original formulation of MVC has one key weakness: the mechanism that updates the view. The flow of information you observe in Figure 4 shows that the view communicates the user gesture to the controller but receives a notification of changes from the model. Next, the view needs to leverage its intimate knowledge of the model to grab updated data and refresh itself. MVP is a variation of MVC that separates the elements of the triad more cleanly. Figure 6 displays the typical interaction between the actors in an MVP model.
Figure 6 The MVP Pattern in Action
In MVP, the view forwards user input to the presenter, and from the presenter it receives fresh data for updates. In turn, the presenter services the request by interacting with the model.
In the original MVC, there's no explicit contract that states which data the view needs. In MVC, the view holds a reference to the model and runs its own logic to select the data it needs and massage it into UI elements. Equipped with this logic, the view is not as passive as it should be for testing purposes. In addition, the view depends, to some extent, on the underlying UI platform.
In MVP, the presenter is essentially the mediator between end users and applications. The presenter has the power to render the view and interact with the model. As a result, most of the presentation logic lives within the presenter. Because the presenter is a plain class with no UI, it is an inherently more testable class. You can see a practical implementation of the MVP pattern in ASP.NET in the August 2006 MSDN Magazine Design Patterns column. MVP is natively supported by the Microsoft Web Client Software Factory, which is a collection of ASP.NET-related application blocks. An MVP QuickStart is available on MSDN.
MVP is a general UI pattern that, unlike Model2, is not specifically designed for the Web. Model2 (the ASP.NET MVC Framework) and MVP are nearly the same thing. Interestingly, neither was originally designed to be a general-purpose pattern.
Technically speaking, there's just one difference between Model2 and MVP: the contract between the controller (called the presenter in MVP) and the view. The contract is loosely defined in Model2 and formally defined in MVP.
In MVP, the view implements an interface, and the presenter talks to the view using only the members on the interface. So, for example, the presenter reads input data in the view using methods and getters in the interface and sets new values for the view using methods and/or setters in the interface. In Model2, you use a strongly typed container.
With MVP, the presentation logic gains independence from the UI platform, so it's easier to reuse the same presenter across different platforms. In addition, the same presenter can work with different views of the same application thus enabling Software as a Service (SaaS) scenarios. Finally, with MVP you have a component—the presenter—that may store some logic for navigation between pages. Whether you implement the logic internally or use a Windows Workflow Foundation (WF) component is up to you.

The Page Controller Pattern
MVC, Model2, and MVP are patterns external to the design of the ASP.NET runtime. The Web Forms programming model of ASP.NET is based on yet another design pattern—the Page Controller pattern. The Page Controller pattern entails the creation of a central object (the dynamically created class that inherits from your codebehind class) that handles all HTTP requests directed at a specific Web page (see Figure 7).
Figure 7 The Page Controller Structure
If you customize this pattern, you may get some benefits in terms of logic reuse across pages, including the navigation logic. You can customize it by simply creating a hierarchy of Page classes and incorporating navigation and presentation logic in the classes higher up in the hierarchy to make that logic available to derived classes. A customized page controller may be a good way to improve the reuse of presentation and navigation logic without switching to a full new pattern.
Applying MVP means essentially designing the codebehind class of pages in a different way by implementing an interface on the Page class and using a separate component—the presenter—to handle user gestures. No changes occur to the runtime environment, as the request is routed to the codebehind class in the usual way.
Creating an ASP.NET MVC Framework project means entering changes to the runtime environment to enable a front controller—the URL routing module—to resolve the request in terms of an action on a controller. Both options have an impact on the development of the application.
The first issue to assess is whether you need viewstate and server controls. There might be situations where you prefer not to deal with them. As an example, if you have no server controls, styling the UI using CSS is much easier. Using AJAX with server controls is currently easier than with an ASP.NET MVC Framework solution. Applications with a lot of data entry or complex table-based views are not so easy to build without rich server controls. So it is ultimately a trade-off. The key question is whether you need server controls and viewstate. There's no obvious answer.
If you decide you'd better stick to Web Forms and still want some SoC, then MVP is for you. MVP is an important UI pattern that may be a bit expensive to implement in relatively simple applications. On the other hand, MVP shines in enterprise-class applications where you really need to reuse as much presentation logic as possible, across multiple platforms and in SaaS scenarios.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an architect at IDesign and the coauthor of Microsoft .NET: Architecting Applications for the Enterprise (Microsoft Press, 2008). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


.SidebarContainerA { width: 40%; height:225px; float: right; border: 2px solid #008080; padding: 10px 20px 0px 20px; background-color: #e1e8f5; margin: 20px 0px 20px 10px; } .SidebarContainerB { width: 40%; height:650px; float: right; border: 2px solid #008080; padding: 10px 20px 0px 20px; background-color: #e1e8f5; margin: 20px 0px 20px 10px; } .SidebarContainerC { width: 40%; height: 500px; float: right; border: 2px solid #008080; padding: 10px 20px 0px 20px; background-color: #e1e8f5; margin: 20px 0px 20px 10px; } .SidebarContainerD { width: 40%; height: 225px; float: right; border: 2px solid #008080; padding: 10px 20px 0px 20px; background-color: #e1e8f5; margin: 20px 0px 20px 10px; }
Patterns in Practice
Design For Testability
Jeremy Miller
Contents
The Value Proposition
What Is Testability?
Isolate the Ugly Stuff
Using Fakes to Establish Boundary Conditions
The Gateway Pattern
Separate Deciding from Doing
Small Tests before Big Tests
The Big Picture
Don't adjust your dial; this is still a column about software design fundamentals. This time around I'd like to talk about testability, both as an important quality of your designs and as another tool you can use to create and think through designs. There is undeniable value in doing automated testing, but the question is whether the benefits of automated testing are worth the time and manpower investment. The benefits and costs of automated testing are greatly affected by design and architecture choices. In this column I will examine the design issues, patterns, and principles that can enhance testability.
This column does not assume that you will be using, or want to use, the design practice of Test Driven Development (TDD). I would hope that this column would be useful to anyone who is interested in adding more automated testing to their process or is just interested in software design in general.
The Excessive Test Setup Smell

In the ProcessWorkflowEvent sample, I'm trying to make the point that coupling to infrastructure concerns will increase the unit testing effort. There's a more general concern evident in this sample. Excessive work to set up a unit test is a "code smell." Excessive unit test setup is often caused by harmful coupling or a lack of cohesion. Many times you will be better served to refactor the code in question before writing a unit test with a great deal of setup.

The Value Proposition
Developers write code, but the real mission is to ship software. The effort to ship software is the sum of the time spent designing, coding, debugging, and testing the software. Lean programming has taught that you shouldn't be doing local optimization. Instead of only worrying about the coding effort, keep your eye on the total effort. The value proposition of automated testing is to reduce the total time of coding + debugging + testing. Your team will come out ahead as long as automated testing is saving more time in debugging and testing than it's costing in extra coding time.
In the United States, we say that the only certainties in life are death and taxes. Well, add another one. There will certainly be bugs in the code you and your team just wrote (and mine, too). Maybe it's because of a misunderstood requirement, a genuine programming mistake, an unforeseen edge case, or a third-party API that doesn't quite do what you thought it would do. Regardless of how those bugs got into your code, you've got to find those problems and get rid of them.
In the end, testability is all about creating rapid and effective feedback cycles in your development process to find problems in your code. It's an axiom in software development that problems are cheaper to fix the earlier they are detected. Since detecting and fixing errors is such a major part of the development effort, it's worth our time as developers to invest in feedback mechanisms to find errors more quickly. When comparing the relative ease of developing in one codebase compared to another, I would go so far as to say that the single most important factor is how short the feedback cycle is between writing a little code and getting some sort of feedback about whether the code you just wrote is working correctly.
Mocking Best Practices

Fake objects come in many different flavors. Most of the examples in this column use stubs—simple objects that provide pre-canned answers and return values. Another type of fake is a "mock" object that is used to record or validate the interaction between classes. Mocks are one of the most valuable—but confusing and misused—concepts in automated testing. Here are some best practices for getting the most out of mocks in your tests:
  • Do not mock or fake out calls to fine-grained or chatty interfaces like ADO.NET that require a lot of calls. The effort involved far outstrips the reward, and those tests are generally unreadable. Only mock coarse-grained interfaces like my IEmailGateway example.
  • Do not mock any interface or class that you do not completely understand. Be especially cautious about mocking interfaces from outside your codebase. It's often better to write your own adapter class or gateway around external APIs that expresses the functionality of that external API in terms of your own architecture.
  • If you find yourself repeating the same mocking setup across multiple tests, you may want to change your design to separate the code that forces the repetitive mocking setup.
  • Do use mock objects as placeholders for classes that don't yet exist, especially as a way to help determine what the public interface should be. Mock objects (or stubs) allow you to investigate the shape and signature of an API without having to build the real implementation first.
For more about mocks and test doubles, see Mark Seemann's MSDN Magazine article "Unit Testing: Exploring the Continuum of Test Doubles".

What Is Testability?
I'm going to define testability as the quality of a software design that allows for automated testing in a cost-effective manner. The end goal of testability is to create rapid feedback cycles in your development process in order to find and eliminate flaws in your code. Here are a few aspects of testability:
Repeatable Automated tests must be repeatable. You must be able to measure an expected outcome for known inputs. You cannot do automated testing on systems that cannot be set up into a known state.
Easy to Write The tests should be mechanically easy to write. The equation is very simple: if it takes a lot of work to set up the inputs to an automated test for a unit of code, you aren't going to recoup your investment in writing that test.
Easy to Understand The tests should be readable and intention-revealing. Ideally, the automated tests that you write should also serve as a useful form of design and requirements documentation. In the real world, changing an existing codebase will at some point break the existing automated tests. When that time happens, it's advantageous for the automated tests to be intention-revealing to help developers understand why the test is broken in order to fix the broken tests. Automated testing coverage should make teams more confident in their ability to change code, not afraid of breaking inscrutable tests.
Fast The automated tests should run quickly. Again, the goal is to make the overall development process faster by establishing rapid feedback cycles to find problems. Slow-running tests will retard productivity.
These four points are directly related to design decisions, specifically to being mindful in regard to the classic design qualities of separation of concerns, cohesion, and coupling—things that you generally want anyway. That said, there are certain design patterns, such as inversion of control and the various forms of separated presentation, that can greatly enhance testability.
As in all things, there are some trade-offs. Many of these patterns are controversial because they don't quite fit the current direction of Microsoft tooling, are relatively new, or seem to negate the usefulness of some traditional .NET approaches. I'd like to use the rest of this column to present some sample scenarios and design patterns, and try to explain why these patterns are considered so important by teams that value TDD and automated testing.

Isolate the Ugly Stuff
Five years ago I considered myself to be strong in object-oriented design. Then I worked on a couple of projects that used TDD and learned firsthand just how little I really knew. Before going into these projects, I had read quite a bit about writing unit tests with NUnit that showed examples like this:
[TestFixture]
public class HolyHandGrenadeTester {
  [Test]
  public void users_should_count_to_three_after_pulling_the_pin() {
    new HolyHandGrenade().TheNumberToCountTo().ShouldEqual(3);
  }
}
Armed with these examples, I went into my first TDD project and immediately ran into trouble. My C# code was interacting with databases, Web services, the ASP.NET runtime, and external systems. Those things were a lot harder to test than code that runs completely inside a single CLR AppDomain.
I specifically remember having a lot of trouble writing unit tests for a little custom workflow subsystem. The workflow data was fairly complex and hierarchical, so I decided at the time to store the state of the workflow as XML data in an Oracle CLOB field. The workflow code itself knew to pull out the XML data from the database, work on it, update the XML, then invoke more ADO.NET code to save the changes back to the database. My code looked like the contrived example shown in Figure 1.
Figure 1 Workflow Code
public void ProcessWorkflowEvent(
  string newStatus, long workflowId) {

  // First, go get the XML out of the database
  OracleConnection connection =
    new OracleConnection(ProjectConfiguration.ConnectionString);
  OracleCommand command = connection.CreateCommand();
  command.CommandType = CommandType.StoredProcedure;
  command.CommandText = "sp_that_returns_the_workflow_information";
  // and set the necessary parameters

  command.ExecuteNonQuery();

  string xml = (string) command.Parameters["@xml"].Value;
  XmlDocument document = new XmlDocument();
  document.LoadXml(xml);

  // depending upon what the new status is, make changes
  // to the Xml document, decide whether or not to send an email,
  // then save the changed Xml document with another set
  // of ADO.NET calls
}
SpecUnit

In many of the unit test samples, I'm using the extension methods from the SpecUnit library. Among other things, SpecUnit provides a series of extension methods that help make unit tests much more readable than the classic xUnit Assert.AreEqual(expected, actual) syntax.
Here's a sample from SpecUnit:
public static object ShouldEqual(
    this object actual, object expected) {
  Assert.AreEqual(expected, actual);
  return expected;
}
And this method would be used like this within unit tests:
item.Name.ShouldEqual("Smith");
item.Age.ShouldEqual(9);
SpecUnit is usable from any of the popular xUnit tools for .NET. My team is using SpecUnit plus many extension methods that are specific to our application to make unit tests a little bit easier to write and much easier to read.
Let's think about what you would have to do to write automated unit tests for this code. The only way to feed the test input into this data was to create a properly formed XML document and then put that XML into a database row. There was a lot of business logic that would change the state of the XML file, and therefore a lot of permutations of previous state and user input that I had to test.
As I said before, I struggled mightily when I tried to write unit tests for this code. My real issue was getting the business rules about the workflow state changes correct. The need to set up the XML data in the database was slowing me down. This is when I learned my first important lesson about testability: some things are just plain ugly to deal with in testing.
What I needed to do was isolate the ugly stuff. All I really mean by "ugly stuff" is any kind of code or infrastructure that is complicated or laborious or just plain inconvenient to get into a test harness, or that makes tests run very slowly. My partial list of ugly stuff would be:
  • Database access. As most TDD practitioners will tell you, tests that involve a database will run an order of magnitude slower than tests that can run completely within a single CLR AppDomain. Setting up data in a database is much more time intensive than simply building objects in memory because of referential integrity and data constraints. When you test against the database, you will often find yourself adding setup data that has nothing to do with the actual test to the database just to satisfy referential integrity and value constraints.
  • GUI technologies, such as Windows Presentation Foundation (WPF) or testing Web apps directly in the browser. It is possible to test the user interface itself, but it's a significant investment in time and the tests run even slower than database tests.
  • Active Directory access.
  • Web services. It's hard enough to set up a known state for testing your own code. Coordinating with a completely separate team to set up tests across two or more systems is more difficult. My advice is to make sure that the functionality of your Web services can be tested independent of the actual Web service protocols. For Web services external to your code, I strongly advise having your code depend on an abstracted interface for external Web services rather than use a Web service proxy class directly. This will enable you to replace the external Web service with your own stubbed implementation during internal testing.
  • Configuration files.
Now, all of those things are important, and all of them should be tested at some point in time. The key, though, is to decouple as much of your core application as possible away from this infrastructure to at least make your core application code easy to test. I should be able to test my data access in isolation once, then write simple tests for the business logic without any concern for the database.
Back to my sample workflow subsystem. At the time I was struggling to get the business logic correct, but the XML data setup and data access was slowing me down. What I really needed to do was walk right up to the business logic code and work with it entirely in memory with quick-running tests. The first thing I should have done was to isolate the business logic away from the infrastructure and XML:
public class WorkflowState {
  public DateTime LastChanged { get; set; }
  public string CurrentStatus { get; set; }
  public string Priority { get; set; }
  // Other properties

  public void ChangeStatus(string newStatus) {
    // WorkflowState will update itself based on
    // the current state of the WorkflowState object
    // and the newStatus
  }
}
Let's say that you have a business rule that says you should escalate the priority of the WorkflowState to Urgent if the customer is upset. Now the workflow business logic can be tested in isolation from the database infrastructure and the XML like this:
[TestFixture]
public class WorkflowStateTester {
  [Test]
  public void
    the_priority_should_be_escalated_to_urgent_when_the_customer_is_angry() {
    WorkflowState state = new WorkflowState(){Priority =
      "Low", CurrentStatus = "NotStarted"};
    state.ChangeStatus("CustomerIsVeryAngry");

    state.Priority.ShouldEqual("Urgent!");
  }
}
The workflow business logic is easier to test, but what else did you get out of this change? You've made the business logic easier to write and understand because it's now independent from the data access infrastructure and the XML format. You now have a much greater ability to change the data access and the business logic independently.
Testability is extremely important in areas of the code that have a lot of variability, require many permutations of input to adequately test, or are volatile in regard to changing requirements.
In the original ProcessWorkflowEvent method shown in Figure 1, the act of loading and persisting the XML document to and from the database is basically the same no matter what is happening with the workflow business logic. I can write a small handful of tests against that persistence code and be confident that it works. It took many more test scenarios to adequately test all the code paths through the business logic. I could have saved myself quite a bit of effort on the original project if I'd been able to unit test the business logic in complete isolation.

Using Fakes to Establish Boundary Conditions
You might have noticed that the second version of the workflow logic isn't complete. It still needs to interact with the database and possibly an e-mail server as well. I would rewrite the original ProcessWorkflowEvent method to something like the method on a new WorkflowService class shown in Figure 2. Instead of doing the data access and sending e-mail directly in WorkflowService, I'll have it delegate to some new services named IRepository and IEmailGateway that I'll explain very soon.
Figure 2 Processing Workflow Events
public class ChangeWorkflowStatusMessage {
  public string NewStatus { get; set; }
  public long WorkflowId { get; set; }
}

public class WorkflowService {
  private readonly IEmailGateway _emailGateway;
  private readonly IRepository _repository;

  public WorkflowService(IEmailGateway emailGateway, IRepository repository) {
    _emailGateway = emailGateway;
    _repository = repository;
  }

  public void ChangeStatus(ChangeWorkflowStatusMessage message) {
    // Fetch the correct WorkflowState object from the database
    WorkflowState workflow = _repository.Find<WorkflowState>(message.WorkflowId);
    workflow.ChangeStatus(message.NewStatus);

    // email the owner of this workflow and tell them that something happened
    // if this is an Urgent item
    if (workflow.Priority == "Urgent") {
      // workflow.Owner is the user who is responsible for
      // this workflow
      _emailGateway.SendChangeStatusEmailTo(workflow, workflow.Owner);
    }

    // Save any changes to the WorkflowState object
    _repository.Save(workflow);
  }
}
One of the prerequisites for automated testing is establishing known inputs to the test. To fully test the WorkflowService.ChangeStatus functionality, I need to set up test inputs that would work on WorkflowState objects that are and are not Urgent. I could set up the database for those test inputs on each test run, but there's an easier way. Let's just use a fake database that can be more easily controlled in testing scenarios by using the Repository pattern to mediate between the application's domain model and the data access services.
My team has consistently chosen to use a Repository pattern with an IRepository interface:
public interface Irepository {
  T Find<T>(long id) where T : Entity;
  void Delete<T>(T target);
  T[] Query<T>(Expression<System.Func<T, bool>> where);
  void Save<T>(T target);
}
The real implementation of Repository uses NHibernate and LINQ for NHibernate to transfer information between the Entity objects and the database tables. In unit testing scenarios, all you really care about is that an Entity was saved, deleted, or available from this IRepository interface because that's all that the rest of the application knows about. For testing, you can create a known database state by using an implementation of IRepository (see Figure 3) that simply stores objects in memory and uses LINQ-to-Objects instead of LINQ for NHibernate for querying.
Figure 3 Fake Database from IRepository
public class InMemoryRepository : IRepository {
  private readonly Cache<Type, object> _types;
  private MockUnitOfWork _lastUnitOfWork;

  public InMemoryRepository() {
    _types = new Cache<Type, object>(type => {
      Type listType = typeof(List<>).MakeGenericType(type);
      return Activator.CreateInstance(listType);
    });
  }

  private IList<T> listFor<T>() {
    return (IList<T>)_types.Get(typeof(T));
  }

  public T Find<T>(long id) where T : Entity {
    return listFor<T>().FirstOrDefault(t => t.Id == id);
  }

  public void Delete<T>(T target) {
    listFor<T>().Remove(target);
  }

  public T[] Query<T>(Expression<Func<T, bool>> where) {
    var query = from item in listFor<T>() select item;
    return query.Where(where.Compile()).ToArray();
  }

  public void Save<T>(T target) {
    listFor<T>().Add(target);
  }
}
In unit tests, you construct objects that depend on IRepository with an instance of the InMemoryRepository that was set up with the desired data. Figure 4 shows a sample unit test that uses InMemoryRepository to simulate a database state.
Figure 4 Test that Simulates a Database State
[Test]
public void find_the_user_by_user_name_and_password() {
  // Set up an InMemoryRepository with a User
  InMemoryRepository repository = new InMemoryRepository();
  User user = new User(){UserId = "jeremy"};
  user.ResetPassword("thePassword"); // the password is hashed
  repository.Save(user);

  // Construct a new instance of SecurityDataService using the
  // InMemoryRepository from above
  SecurityDataService service = new SecurityDataService(repository);
  service.Authenticate("jeremy", "wrong").ShouldBeFalse();
  service.Authenticate("jeremy", "thePassword").ShouldBeTrue();
  service.Authenticate("wrong", "thePassword").ShouldBeFalse();
  service.Authenticate("wrong", "and wrong").ShouldBeFalse();
}
The sample tests the functionality of the SecurityDataService class (see Figure 5). Internally, it needs to query an IRepository for the existence of a user with a username and password.
Figure 5 Class to Be Tested
public class SecurityDataService : ISecurityDataService {
  private readonly IRepository _repository;

  // The IRepository dependency is set up via Constructor Injection
  // In our real application, Both the real NHibernateRepository
  // and the SecurityDataService would be constructed by
  // an Inversion of Control container
  public SecurityDataService(IRepository repository) {
    _repository = repository;
  }

  public bool Authenticate(string username, string password) {
    return _repository.FindBy<User>(u => u.UserId ==
      username && u.Password == User.HashPassword(password)) != null;
  }
}
In the real application, the SecurityDataService uses the NHibernate Repository object that accesses the database. But in testing you can happily substitute a fake Repository that can easily be controlled to more quickly simulate multiple test cases. You can only do this type of substitution with fake objects if the classes that use IRepository only depend on the declared interface of IRepository. If the classes that use IRepository depend even indirectly on the fact that the normal IRepository does NHibernate-specific things, you won't be able to cleanly substitute the fake object.
Fake objects, such as stubs or mocks, are a huge advantage for doing emergent or continuous design by allowing you to build, and even design, a system incrementally by standing in for services that you haven't yet created.

The Gateway Pattern
Almost every project I work on contains at least one feature that requires the application to send automated e-mail notifications to the user depending on some sort of action and state in the system. But I most definitely do not want my application code getting hung up with e-mail functionality.
I like to wrap up e-mail functionality by using the Gateway pattern. In the Gateway pattern, an object encapsulates access to an external system or resource. I want to pull every single piece of code it takes to send e-mails, including SMTP-type configuration, and hide it away from the rest of my code inside an IEmailGateway object, like this:
public class EmailMessage {
  public string Body { get; set; }
  public string[] To { get; set; }
}

public interface IEmailGateway {
  void SendChangeStatusEmailTo(WorkflowState workflow, User owner);
}
There are a lot of existing solutions for sending e-mails from .NET code, so I'm not really that concerned about the actual act of sending the e-mail, but I'm very concerned about whether my application code decided to send an e-mail, to whom, and what the message body was. What I need to test in the WorkflowService is the simple fact that it did or did not choose to send an e-mail based on the final status of the WorkflowState object.

Separate Deciding from Doing
One of the best things you can do in your code is to treat performing an action and deciding to take an action as two separate responsibilities. In the case of WorkflowService, it is only responsible for deciding to send the e-mail:
// email the owner of this workflow and tell them
// that something happened if this is an Urgent item
if (workflow.Priority == "Urgent") {
  // workflow.Owner is the user who is responsible for
  // this workflow
  _emailGateway.SendChangeStatusEmailTo(workflow, workflow.Owner);
}
You could just let the code send the e-mail to the right people, but it's generally pretty annoying to set up fake e-mail accounts or run around and ask people if they just got an e-mail from your system. Instead, use a mock object in place of the IEmailGateway just to check whether WorkflowService decided to send the e-mail notification (see Figure 6).
Figure 6 Testing whether E-Mail Was Sent
[Test]
public void send_an_email_if_the_WorkflowState_is_urgent() {
  // setting up the WorkflowState

  var theOwner = new User();
  var theWorkflowState = new WorkflowState{Priority =
    "Urgent", Id = 5, Owner = theOwner};
  repository.Save(theWorkflowState);

  // Exercise the ChangeStatus() method
  service.ChangeStatus(new ChangeWorkflowStatusMessage(){
    NewStatus = "Urgent", WorkflowId = 5});

  // Verify that the change status email was sent
  emailGateway.AssertWasCalled(x =>
    x.SendChangeStatusEmailTo(theWorkflowState, theOwner));
}
Separating the e-mail functionality into IEmailGateway made it easy to unit test the business logic, but did it give you anything else? Pulling all access to the actual e-mail sending code into a single class helps in the following ways:
  • Prevent duplication in the code. Repetitive coding is required to find configuration data to set up the System.Mail.SmtpClient object to send the e-mail in .NET code.
  • It makes the code that needs to send the e-mails somewhat simpler. WorkflowService just needs to say: send this e-mail to this user.
  • You can change the e-mail infrastructure. What if you want to use a third-party tool instead of System.Mail.SmtpClient for e-mailing? What if you want extra logging for each e-mail, or want to implement a retry capability for failures or want to queue up e-mails to send in a batch? If you consistently use the same Gateway pattern interface for access to sending e-mails, you can add more robust infrastructure later without changing the classes that depend on IEmailGateway.
Resources for Increasing App Testability

Unit Testing with Mocks
Better Testing with Mocks
Tame Your Software Dependencies for More Flexible Apps
A Unit Testing Walkthrough with Visual Studio Team Test

Small Tests before Big Tests
You want fast and accurate feedback loops on your code to flush out problems quickly. Any automated test should tell you that there is a problem when something doesn't work as expected, but a small, focused test can tell you exactly where something is wrong. When a big end-to-end test fails, you have a lot of different factors to consider, which often makes debugging harder. You still need the big end-to-end test, but my strong advice is to structure your application in such a way that you can write and test granular units of code first to flush out problems quickly before you attempt to execute end-to-end tests.
In the WorkflowService example, before attempting to run the actual WorkflowService from end to end, I would do the following tests:
  1. Write the WorkflowState object and test all of its business logic with various permutations of state.
  2. Write a couple of tests to prove that I can correctly save and load a WorkflowState object from the database.
  3. Write some interaction tests against the WorkflowService to make sure that it is correctly coordinating the IEmailGateway, IRepository, and WorkflowState objects.
  4. Inside the concrete EmailGateway class, I would write unit tests to verify that it is forming the correct e-mail contents for a given WorkflowState and User object independent of sending the e-mail.
Only once I am reasonably confident that every one of the pieces of WorkflowService function independently will I try to execute a full test of WorkflowService that uses the real database and sends e-mails.
Smaller tests are cheaper to create, easier to understand, faster to run, and much simpler to debug. It sounds like more work to write a bunch of unit tests than a single integrated test, but I've frequently found it more efficient to test small units of code before attempting to integrate those units in a bigger test.

The Big Picture
I would say that my focus on testability over the past five years has had more impact on how I approach software design than anything else. Granted, there are times when I have to go out of my way in a design for no other purpose than to make the code testable. In the end, though, I think that testability goes hand in hand with the classical definition of a good design.
Carefully considering the question, how can I test this in isolation? is yet another tool that will help you arrive at the classic design qualities of cohesion, coupling, and separation of concerns in your codebase.

Send your questions and comments to mmpatt@microsoft.com.

Jeremy Miller, a Microsoft MVP for C#, is also the author of the open-source StructureMap (structuremap.sourceforge.net) tool for Dependency Injection with .NET and the forthcoming StoryTeller (storyteller.tigris.org) tool for supercharged FIT testing in .NET. Visit his blog, "The Shade Tree Developer," part of the CodeBetter site.


Test Run
Configuration Testing With Virtual Server, Part 2
Dr. James McCaffrey and Paul Despe
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Virtual Server Automation with Windows PowerShell
Creating a Virtual Machine
Configuration Tests on a Virtual Machine
Last Word
There are several ways you can perform software configuration testing. One effective approach for certain scenarios is to use Microsoft Virtual Server to create a library of virtual machines. Because Virtual Server is built on a set of COM modules, you can completely automate the process of creating and exercising virtual machines.
Although individual Virtual Server automation tasks are quite well documented, in discussions with our testing colleagues we learned that there is a need for a complete end-to-end example that puts together all the parts of automating Virtual Server for software configuration testing. Additionally, almost all current Virtual Server automation references use the old VBScript language rather than the much more powerful Windows PowerShell.
In this month's column, we're going to walk you through the process of automating software configuration testing using Virtual Server and Windows PowerShell.We will assume that you have a basic familiarity with them, but you should be able to follow our examples even if these technologies are new to you.
(If you do want more information, you can read about Windows PowerShell in the May 2007 installment of Test Run—"Lightweight Testing with Windows PowerShell"—and in "CMDLETS: Extend Windows PowerShell with Custom Commands" by Jim Truher. I also covered configuration testing with Virtual Server in my September 2008 Test run column "Configuration Testing With Virtual Server, Part 1.")
The screenshot in Figure 1 gives you a good idea of where we're headed in this column. Our physical host machine is running Windows Server 2008; however, all the techniques described here also work with Windows Server 2003. The host machine has Virtual Server 2005 R2 SP1 installed. The screenshot in Figure 1 shows a virtual guest machine named VitualMachine-Test that is running Windows XP SP2. What is not apparent from the screenshot is that the virtual machine was created using a Windows PowerShell 1.0 script. In the background of the screenshot, you can see that a second Windows PowerShell script has set up a Scheduled Task on the virtual machine. This task is the test automation, which is running in the cmd.exe shell in the foreground of the virtual machine.
Figure 1 Example of Configuration Testing
In the sections of this column that follow, we'll explain in detail the script that created the virtual machine shown in Figure 1 and the script that set up and executed the test automation. We think you'll find the techniques we present here a useful addition to your software testing, development, and management skill sets.

Virtual Server Automation with Windows PowerShell
Virtual Server is built on a set of objects that use classic DCOM technology. You can automate Virtual Server using any COM-aware language, including VBScript, JavaScript, C#, Visual Basic .NET, and Windows PowerShell. Our preferred approach is to use Windows PowerShell. Its advantages include its ability to directly call into the Microsoft .NET Framework and its command-line capabilities, which allow you to develop scripts interactively. Because VBScript and JavaScript are native Win32 scripting languages, they can directly create and use Virtual Server automation objects. But because Windows PowerShell is a .NET-compliant language, in order to use Virtual Server automation remote objects with Windows PowerShell you must enable impersonation on each object.
There are several approaches you can use. The most straightforward approach is to use Visual Studio and C# to create a custom DLL that contains a method that allows you to set an impersonation level on a native Win32 object, and then call that custom method from Windows PowerShell. We use a somewhat surprising alternative approach, which is to write and compile a custom DLL on the fly using Windows PowerShell.
The script in Figure 2 creates a Windows PowerShell function named SetImpersonation, which we can use to set the impersonation level on Virtual Server automation objects. We decided to place our code inside the special Windows PowerShell $profile startup script. We begin our script with two write-host messages to alert users that a custom startup script is adding additional functionality to Windows PowerShell. Next, we set up references to the C# compiler and a temporary file:
$csc = (join-path ($env:windir) Microsoft.NET\Framework\v2.0.50727\csc.exe)
$tempFile = [IO.Path]::GetTempFileName() + ".cs"
Figure 2 Custom SetImpersonation Function
write-host "Executing custom startup script"
write-host "Creating custom SetImpersonation function'n"
set-location D:\

$csc = (join-path ($env:windir) Microsoft.NET\Framework\v2.0.50727\csc.exe)

$tempFile = [IO.Path]::GetTempFileName() + ".cs"

$source = @"
  using System;
  using System.Runtime.InteropServices;

  public class MySecurity
  {
    [DllImport("Ole32.dll", CharSet=CharSet.Auto)]
    public static extern int CoSetProxyBlanket(IntPtr pProxy,
      uint dwAuthnSvc,
      uint dwAuthzSvcp2,
      IntPtr pServerPrincName,
      uint dwAuthnLevel,
      uint dwImpLevel,
      IntPtr pAuthInfo,
      uint dwCapabilities);

    public static int EnableImpersonation(object objDCOM) {
      return CoSetProxyBlanket(Marshal.GetIDispatchForObject(objDCOM),
      10, 0, IntPtr.Zero, 0, 3, IntPtr.Zero, 0);
    }
  }
"@

set-content $tempFile $source
$targetDLL = [IO.Path]::GetTempFileName() + ".dll"

invoke-expression "$csc /nologo /target:library /out:$targetDLL $tempFile"
[System.Reflection.Assembly]::LoadFrom($targetDLL) | out-null

function SetImpersonation($obj)
{
  [MySecurity]::EnableImpersonation($obj) | out-null
}
We use the join-path cmdlet to create a full path to the .NET Framework 2.0 C# compiler, which we assume is on our host machine. Then we call directly into the System.IO namespace and use the static GetTempFileName method of the Path class to generate a unique file name with a .cs extension, located in the system Temp folder of our host machine.
Next we set up C# source code in order to create our custom impersonation function:
$source = @"
  using System;
  using System.Runtime.InteropServices;

  public class MySecurity
  {
    . . .
  }
"@
We use the here-string feature of Windows PowerShell to create, in essence, a complete C# program as a single string. The idea is to use the C# P/Invoke mechanism to create a managed wrapper method named EnablePersonation around the Win32 CoSetProxyBlanket function. The details of the CoSetProxyBlanket function are outside the scope of this column, but the bottom line is that we create a method that allows objects to impersonate the client's security context on remote systems.
With our C# source code created, next we actually build a managed DLL on the fly:
set-content $tempFile $source
$targetDLL = [IO.Path]::GetTempFileName() + ".dll"
invoke-expression "$csc /nologo /target:library /out:$targetDLL $tempFile"
[System.Reflection.Assembly]::LoadFrom($targetDLL) | out-null
We use the set-content cmdlet to place the C# source code into our temp file. Next we use the GetTempFileName to create a unique name for our DLL file. We could have left this step out, because the C# compiler by default will create a DLL name based on the name of the source code file. Then we use the invoke-expression cmdlet to run the C# compiler.
Because our argument to invoke-expression is a double-quoted string, the objects inside the string will be evaluated. After the C# compiler builds our custom DLL with our EnableImpersonation method, we use the LoadFrom method to load the DLL into our current Windows PowerShell execution environment.
Our script then finishes by creating a Windows PowerShell wrapper function named SetImpersonation around the C# EnableImpersonation method:
function SetImpersonation($obj)
{
  [MySecurity]::EnableImpersonation($obj) | out-null
}
We pipe our call to EnableImpersonation to the out-null cmdlet simply to suppress printing diagnostic information when the function is called. By placing this script in your Windows PowerShell startup program, every time you launch a new instance of Windows PowerShell, the script will execute and you will create a new custom DLL with a SetImpersonation function.
The disadvantage of this approach is that you are recreating the same DLL every time you launch an instance of Windows PowerShell. Alternatively, you can place this code directly into the beginning of any Windows PowerShell script that needs to access Virtual Server COM objects. But we do not recommend that you simply build the custom DLL once (with C# or on the fly with Windows PowerShell) and then place that DLL somewhere on your host machine.
Automation scripts often have a way of persisting much longer than you initially expect. You don't want to be in a position months or even years later where you've lost your DLL and have no idea how to recreate it.

Creating a Virtual Machine
The screenshot in Figure 3 summarizes the steps you perform when using automation to create a Virtual Server virtual machine. We begin by verifying that we have our custom SetImpersonation function available. Next we check to see if our virtual machine configuration files already exist from a previous automation run. If the configuration files exist, we stop the background Virtual Server service and delete the files, then restart the service.
Figure 3 Creating a Virtual Machine
Next we instantiate the primary Virtual Server automation object, which contains all automation functionality. Then we create a virtual machine and provision its RAM, hard disk drive (with operating system), CD/DVD drive, and network adapter. We finish by starting the virtual machine, determining when the virtual machine's operating system is up and running.
Let's walk through the createVM.ps1 Windows PowerShell script shown running in Figure 3. Our script begins by looking for the SetImpersonation function described in the previous section:
write-host "'nChecking to see if SetImpersonation function exists"
$funcs = get-childitem function:\*
$found = $false
foreach ($f in $funcs) {
  if ($f.name -eq "SetImpersonation") {
    $found = $true
  }
}
We use the get-childitem cmdlet to retrieve a collection of all functions known to our current Windows PowerShell environment. We could also have used the get-command cmdlet with a "-commandtype function" argument. We iterate through every item in the collection looking for a function with a SetImpersonation name property. Alternatively, we could have used the Windows PowerShell break statement to exit our iteration loop early if SetImpersonation is found.
After examining the function:\ collection, we can use our Boolean $found variable to determine if the SetImpersonation function is in our current environment:
if ($found) {
 write-host "Found custom SetImpersonation function"
}
else {
 throw "No SetImpersonation function found"
}
Here we decide to throw an exception if the SetImpersonation function is not found, because the rest of our automation will certainly fail. The next phase of our automation prepares to delete a virtual machine configuration file if it already exists. To delete a virtual machine configuration file using Windows PowerShell, you must first stop the background Virtual Server service, so we create a StopService utility function to do so.
The helper function begins by retrieving information about the Virtual Server service:
function StopService
{
  write-host "'nStopping Virtual Server service"
  $s = get-service "Virtual Server"

  if ($s.status -ne "Stopped") {
    $s.stop()
  }
  . . .
}
We use the handy get-service cmdlet to fetch an object that contains a Status property and then call the object's Stop method. At this point we could pause our automation to give the Virtual Server service time to stop, but the problem is that we have no way of knowing how long to pause.
A much better approach is to go into a delay loop:
$ct = 0
while ( ((get-service "Virtual Server").status -ne "Stopped")
          -and ($ct -lt 100) ) {
  $ct++
  start-sleep -m 200
  write-host "Waiting for Virtual Server service to stop . . ."
}
We initialize a counter to keep track of how many times we pause our automation. Then we go into a delay loop that will exit as soon as the service Status is equal to Stopped or if we exceed 100 delays. Inside our delay loop, we use the start-sleep cmdlet to pause for 200 milliseconds, so we will delay at most 20,000 milliseconds (20 seconds). After our delay loop terminates, we check to see how it exited:
if ( (get-service "Virtual Server").status -ne "Stopped" ) {
  throw "Failed to stop Virtual Server service in allowed time"
}
If the Virtual Server status is not Stopped, that means the loop exited because it exceeded 100 iterations, so we throw an exception. We define a StartService helper function, which uses the same logic as the StopService function:
. . .
if ($s.status -ne "Running") {
  $s.start()
}
. . .
while ( ((get-service "Virtual Server").status -ne "Running")
        -and ($ct -lt 100) ) {
. . .
}
if ( (get-service "Virtual Server").status -ne "Running" ) {
  throw "Failed to start Virtual Server service in allowed time"
An alternative to placing the StopService and StartService helper functions directly inside your virtual machine creation script is to place these helper functions inside the Windows PowerShell $profile startup script. With our two helper functions defined, we can now determine if we need to delete any virtual machine configuration files left from previous automation:
$vmcFile = test-path "D:\VMs\VirtualMachine-Test.vmc"
$lnkFile = test-path "D:\Users\All Users\Microsoft\Virtual Server\Virtual Machines\VirtualMachine-Test.lnk"
We use the test-path cmdlet to return a true/false value depending upon whether the cmdlet's file argument exists or not.
When you create a virtual machine, you can place the .vmc configuration file anywhere on your host machine. Virtual Server also creates a shortcut .lnk file to the .vmc file at a standard location, which depends upon your host operating system—for example, %SystemRoot%\Users\All Users\Microsoft\Virtual Server\Virtual Machines\ in the location in Windows Server 2008.
To create a virtual machine with a particular name, you must delete both the .vmc and corresponding .lnk files or Virtual Server will complain that your virtual machine already exists. Notice that our virtual machine name is hardcoded into our script for clarity; in a production environment, you will likely want to parameterize this value and pass it to your script. Once we've determined if any old virtual machines exist we can delete them, as we do in Figure 4.
Figure 4 Delete Old Virtual Machines
if ($vmcFile -or $lnkFile) {
  write-host "'nFound existing .vmc and/or .lnk file"
  StopService
  if ($vmcFile) {
    write-host "Deleting .vmc file"
    remove-item "D:\VMs\VirtualMachine-Test.vmc"
  }
  if ($lnkFile) {
    write-host "Deleting .lnk file"
    remove-item "D:\Users\All Users\Microsoft\Virtual Server\Virtual
      Machines\VirtualMachine-Test.lnk"
  }
  StartService
}
else {
 write-host "'nDid not find existing .vmc or .lnk files"
}
If either our target .vmc or .lnk files exist, we call our StopService helper function to halt the background Virtual Server service. Then we use the remove-item cmdlet to delete the appropriate files and then restart the service.
At this point in our script, we are ready to create the primary Virtual Server COM automation object:
if ((get-service "Virtual Server").status -ne "Running") {
  StartService
}
$vs = new-object -com "VirtualServer.Application"
if ($vs -eq $null) {
  throw "Failed to create main COM automation object"
}
SetImpersonation $vs
First we make sure that the Virtual Server service is running. Next we use the new-object cmdlet to instantiate an instance of the Virtual Server automation library, which has a ProgID of VirtualServer.Application. And if that creation fails, we throw an exception because our automation is doomed. Otherwise, we invoke our custom SetImpersonation function on the automation object so that the object can impersonate the client's security context through Virtual Server's underlying DCOM transport mechanisms.
Creating a virtual machine is almost too easy:
write-host "'nCreating virtual machine"
$vm = $vs.CreateVirtualMachine("VirtualMachine-Test", "D:\VMs")
if ($vs -eq $null) {
  throw "Failed to create virtual machine object"
}
SetImpersonation $vm
We simply call the CreateVirtualMachine method and pass the name of the virtual machine as a string, and the location of the host where we want the resulting .vmc file to reside—in this case a hardcoded D:\VMs\ directory. Because we need to manipulate the virtual machine object, we set its impersonation level.
At this point in the process, we can begin provisioning our newly created virtual machine:
write-host "Assigning 256 MB memory"
$vm.Memory = 256

write-host "'nAdding existing virtual hard disk drive"
write-host "Copying .vhd file from library to working directory"
copy-item "D:\VirtualMachinesLibrary\VirtualMachine-WinXP-SP2.vhd"
  "D:\VirtualMachines\"
First we assign RAM for the guest machine, keeping in mind the fact that the total amount of RAM available for all running virtual machines plus the host machine is limited by the amount of RAM on the host machine. Next we prepare to attach a .vhd virtual hard disk drive file to our virtual machine. We copy a .vhd file that contains our OS and other key software from a backup location to a working directory. The idea is that we want to make sure that if our .vhd file becomes corrupted during configuration testing, we still have an original version of our .vhd file to use later.
Because .vhd files are typically about 2.0GB in size, this file copy operation usually takes a couple of minutes. When you create a .vhd file with Virtual Server, the .vhd file is placed in the same directory as its associated .vmc file. However, when attaching an existing .vhd file to a new virtual machine, the .vhd and .vmc files can be in different directories, as we're doing here. And once we have our working .vhd file, we can attach it to our virtual machine:
$loc = "D:\VirtualMachines\VirtualMachine-WinXP-SP2.vhd"
if (test-path $loc) {
  write-host "Found the .vhd file"
  $hd = $vm.AddHardDiskConnection("D:\VirtualMachines\VirtualMachine-WinXP-SP2.vhd", 0, 0, 0)
}
else {
  throw "Failed to find .vhd file"
}
First we use the test-path cmdlet to make sure our working copy of the target .vhd file exists. Next we use the AddHardDiskConnection method to attach the virtual hard disk. The first parameter to the AddHardDiskConnection method is hard drive bus type, which is an enumeration where vmDriveBusType_IDE = 0 and vmDriveBusType_SCSI = 1.
Interestingly, even if your host machine has only IDE drives, you can specify a SCSI drive for your virtual machine for improved performance. The second parameter is the bus number where the value can be 0 or 1 for IDE drives, and 0 through the number of SCSCI controllers for SCSI drives. The third parameter is the device number where the value can be 0 or 1 for IDE drives and 0 through 6 for SCI drives.
After attaching a .vhd virtual hard drive, we are able to make that drive Undoable:
write-host "Making .vhd disks undoable"
$vm.undoable = $true
By default, Virtual Server automatically saves all changes made to a virtual machine while the machine is running. But if you configure Undo disks, then all changes made to the virtual machine while the machine is running are saved to a .vud virtual undo disk file. And when you shut down the virtual machine, you get the option either to discard the Undo disk (and revert back to the original .vhd disk) or commit the Undo disk (and save changes to the .vhd file.)
Now, with the scheme we're using here—copying a base configuration .vhd file to a working directory—it doesn't matter if our hard disk is undoable or not. But we show you this technique in case you need it for your particular scenario.
Now we provision the CD/DVD drive:
write-host "'nAdding CD/DVD drive"
$dvd = $vm.AddDVDROMDrive(0,1,0)
if ($dvd -eq $null) {
  write-host "Failed to add CD/DVD drive"
}
SetImpersonation $dvd

write-host "Attaching CD/DVD drive to F:"
$dvd.AttachHostDrive("F")
When you create a virtual machine using the Virtual Server GUI interface, Virtual Server automatically adds and attaches a CD/DVD drive to your guest machine if the host machine already has a CD/DVD drive. But when you create a virtual machine with automation, you must explicitly configure a CD/DVD drive.
The first parameter to the AddDVDROMDrive method is the bus type, which must be 0 for a CD/DVD drive. The second parameter is the bus number, which must be 0 or 1. The third parameter is the device number, which also must be 0 or 1. Next we use the AttachHostDrive method to associate the virtual CD/DVD drive on the guest machine with the physical drive on the host machine. Notice we've hardcoded "F:" in our script for clarity, but you may want to parameterize this value and pass it into your script. Now we're ready to configure our network adapters:
$vn = $vs.FindVirtualNetwork('External Network (Broadcom 802.11g Network
   Adapter)')
if ($vn -eq $null) {
  write-host "Failed to find virtual network adapter"
}
SetImpersonation $vn
We use the FindVirtualNetwork method to locate an existing network adapter. Notice that Virtual Server often uses the terms network and network adapter more or less interchangeably. When you install Virtual Server, the setup program scans the host machine for all active adapters and then names the adapters by placing the adapter name in parentheses, preceded by the string "External Network". However, if you add a network adapter after Virtual Server installation, you can name the adapter anything you wish.
After setting the impersonation level on the adapter object, we attach our virtual adapter to our virtual network:
$adapters = $vm.NetworkAdapters
SetImpersonation $adapters
SetImpersonation $adapters.Item(1)
$adapters.Item(1).AttachToVirtualNetwork($vn)
We use the NetworkAdapters property to fetch a collection of all adapters. We must set impersonation on the collection object and each adapter in the collection in order to access an adapter with Windows PowerShell. The first adapter, Item(0), is the built-in "Internal Network" adapter, so Item(1) is the first external adapter. We use the AttachToVirtualNetwork method to connect the adapter to the network. Our configuration is now complete, and we can start our virtual machine:
write-host "'nStarting Virtual Machine"
$startTask = $vm.StartUp()
SetImpersonation $startTask
$startTask.WaitForCompletion(60000) # wait up to 1 minute
The Startup method returns VMTask object that can be used to monitor the method's progress. The task object has a convenient WaitForCompletion method that accepts a maximum value for time, in milliseconds, to wait for the task to complete.
Next we must wait until the operating system on the guest virtual machine boots up:
write-host "'nWaiting for guest virtu al machine to boot                up'n"
   $ct = 0
   while ( ((test-path "\\VM-017\Public") -eq $false) -and $ct
          -lt 100 )
   {
     $ct++
     write-host "Waiting . . . $ct"
     start-sleep -s 10
   }
The approach we take is to use a delay loop. In each iteration through the delay, we pause for 10 seconds. Our while loop condition probes into the virtual machine using test-path and exits if we discover a directory that we know exists on the guest, or if we exceed a maximum number of probing attempts. Our script concludes by determining how our delay loop terminated:
if ((test-path "\\VM-017\Public") -eq $false) {
  throw "'nFailed to discover guest machine"
}
else {
  write-host "'nVirtual machine is up and running"
}
If we cannot discover the virtual guest machine, we throw an exception. Otherwise we display a success message.

Configuration Tests on a Virtual Machine
The previous section describes how to create and provision a virtual machine using a script. At this point you can perform manual testing, but let's look at the key techniques for automating the process of configuration testing. If you look at the screenshot in Figure 5, you can see we have a Windows PowerShell script named exerciseVM.ps1. Our script first verifies that the target virtual machine is running and then copies a test-related file from the host machine to the guest machine. Next our script creates a Windows task on the guest machine that will fire off an automated test at a specified time. The script concludes by turning the virtual machine off and discarding the machine's Undo disk.
Figure 5 Placing Test Automation on a Virtual Machine
The exerciseVM.ps1 script begins by using the test-path cmdlet to look for the target virtual machine:
if ((test-path "\\VM-017\C$") -eq $false) {
  throw "'nFailed to discover guest machine"
}
else {
  write-host "Found virtual guest machine"
}
If we can't find the system root of the virtual machine, we throw an exception to terminate the script. Next we verify that test automation file exists on the host:
if (test-path "D:\Public\testHarness.js") {
  write-host "Found testHarness.js file to copy"
}
else {
  throw "Did NOT find testHarness.js file to copy"
}
Then we verify that the guest has a public share to receive our test automation. This approach assumes that the virtual machine's .vhd file has been configured with a share that allows users to copy files to that share. There is no easy way for a script running on a host machine to perform such a security configuration on a guest.
if (test-path "\\VM-017\Public") {
  write-host "Found \\VM-017\Public directory to copy to"
}
else {
  throw "Did NOT find \\VM-017\Public directory to copy to"
}
copy-item "D:\Public\testHarness.js" "\\VM-017\Public"
We simply use the copy-item cmdlet to copy our single automation file from host to guest. Now we set up a Windows task to execute the test automation on the virtual guest machine:
$dt = new-object -com WbemScripting.SWbemDateTime
$now = [DateTime]::now
$later = $now.addSeconds(90)
write-host "'nTest automation testHarness.js schedeuled to run at " $later
$later.ToUniversalTime() | out-null
$dt.SetVarDate($later)
We are using Windows Management Instrumentation (WMI)/Common Information Model (CIM) technology, so first we create an SWbemDateTime object (as opposed to a .NET DateTime object). We use the .NET static Now property to fetch the current date and time on the host. Then we use the AddSeconds method to create a .NET Date time object that is 90 seconds later. Next we convert the .NET execution time to a Coordinated Universal Time (UTC) object. Then we use the SetVarDate to configure the SWbemDateTime date-time object.
Now we schedule our test automation task:
[System.Management.ManagementClass] $mc =
  new-object System.Management.ManagementClass("\\VM-017\root\cimv2",
    "Win32_ScheduledJob", $null)
$mc.Create("cscript.exe //nologo C:\Public\testHarness.js",
           $dt.value, $false, 0, 0, $true) | out-null
Instead of wrestling directly with WMI technology using the Windows PowerShell get-wmiobject cmdlet, we use the fact that Windows PowerShell can directly call into the .NET Framework, and we use the ManagementClass of the System.Management namespace. You can think of this class as a friendly .NET wrapper around WMI functions.
The first argument to the object's constructor is the scope of the object and essentially identifies the target machine. The second argument is the WMI class name, and the third argument is an object that contains any optional information to use when retrieving the WMI class.
We use the Create method to set up the scheduled test automation. The first parameter is a string that is the command to execute. The second parameter is an object that specifies when to run the task. The third parameter is a Boolean value that specifies if the task is to run more than once or not. The next two parameters specify when multiple jobs should run. The sixth parameter is a Boolean that specifies if the job should run in interactive mode (in other words, visible) or non-interactive mode.
No script can arbitrarily set up a scheduled task on a remote/guest machine without the cooperation of the guest machine. The details of allowing remote job scheduling vary quite a bit depending upon the operating system being used. In the case of the example shown in Figure 1, the guest machine required four modifications. First, the Windows Firewall on the Windows XP guest must be disabled to allow remote WMI access. Additionally, the Local Security Policy on the guest required three changes in the Security Options area of the Local Policies section of the Security Settings item. The "DCOM: Machine Access Restrictions in Security Descriptor Definition Language syntax" settings must be modified to "Allow for Anonymous Login and Everyone users." The "Network Access: Let Everyone permissions apply to anonymous users" setting must be enabled. And the "Network Access: Sharing and security model for local accounts" setting must be modified to "Classic—local users authenticate as themselves".
In short, if you want to use a Virtual Server host machine to schedule test automation on a guest machine, you need to make several security changes to the underlying .vhd file. Of course, there's no requirement that you schedule tasks from your host machine; an alternative is simply to configure the guest .vhd file with recurring test automation tasks built in.
With our test automation scheduled, we can wait for the tests to run and then shut down our guest machine:
$dummy = read-host -prompt "'nPress <Enter> to continue"
write-host "'nPreparing to turn off guest machine"
write-host "Creating main COM automation object"
$vs = new-object -com "VirtualServer.Application"
if ($vs -eq $null) {
  throw "Failed to create main COM automation object"
}
SetImpersonation $vs
We use the read-host cmdlet to halt our script execution so that we can observe our test automation using the Virtual Server Web-based administration page and determine when all tests are complete. This technique is useful when developing your automation, but in a production environment you will likely want to use some other mechanism to determine when all your tests are finished. For example, one approach is to use a delay loop and use the test-path cmdlet to probe for a test results file.
Our test automation script now has the ability to connect to the virtual machine:
write-host "Looking for virtual machine"
$vm = $vs.FindVirtualMachine("VirtualMachine-Test")
if ($vm -eq $null) {
  throw "Failed to find virtual machine"
}
else {
  write-host "Found virtual machine"
}
SetImpersonation $vm
We use the FindVirtualMachine method to locate the virtual machine, which was created by our createVM.ps1 script and, we assume, is running as a guest. Notice that we do not use the .vmc extension on the name of the target virtual machine.
After setting the impersonation level on the virtual machine object, we can turn off the virtual machine:
write-host "'nTurning virtual machine off"
$offTask = $vm.TurnOff()
SetImpersonation $offTask
$offTask.WaitForCompletion(5000)
write-host "'nDiscarding undo disk"
$vm.DiscardUndoDisks()
The TurnOff method returns a VMTask object that we can use to wait until the virtual machine shuts down or exceeds a timeout value. In this situation, because our virtual hard disks are undoable, we discard the Undo disk that holds changes made to the state of the virtual machine so that the next time we use the machine's .vhd file, we will be starting with a clean state.

Last Word
There are several ways you can adapt and extend the Virtual Server automation techniques we've described in this column. In order to demonstrate as many key principles as possible, we organized our examples into two relatively large, un-parameterized scripts. With the explanations we've presented, you should be able to modify and reorganize our two scripts to suit your own needs and parameterize some or all of the hardcoded values in the script as needed.
Because our scripts are primarily intended for demonstration purposes, we removed some of the error-checking code that you'd normally put into production code. However, our examples have included several error traps, so you shouldn't have too much trouble making your automation robust enough for production scenarios.
Although the new Microsoft Hyper-V virtualization technology may someday replace Virtual Server, Virtual Server is going to be useful for many years to come. Additionally, the Virtual Server automation principles and techniques we've presented here, in particular, the use of Windows PowerShell, will provide you with a nice transition to Hyper-V.

Send your questions and comments to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working for Microsoft in Redmond. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006). James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.

Paul Despe is a Program Manager on the Hyper-V team. Paul has worked as a Software Design Engineer in Test on both the Virtual PC and Virtual Server products. Before joining Microsoft, Paul worked in Japan and at Connectix, a virtualization software company acquired by Microsoft in 2003. Paul can be reached at paulde@microsoft.com.


Foundations
Render Text On A Path With WPF
Charles Petzold
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Uniting Paths and Text
Endpoints and Midpoints
Let's Make It a UserControl!
What Size Is It?
Lowering the Level
The Measuring Conundrum
The World of Visual Children
Warping the Text
Text is more than just its literal meaning. Whether printed or displayed on a computer screen, the choice of font can enhance or detract from the impact of the text. Text can be made inviting or forbidding, soothing or stormy. Computer graphics can also release text from the confines of its customary line-by-line progression. By treating text characters as graphical objects, the programmer can make text dance and fly on the screen.
One desirable technique is positioning text characters along a curved line, as shown in Figure 1. In graphics programming, a collection of straight lines and curves is called a "path," so this task is sometimes described as "text on a path," and that's what I'll tackle in this column.
Figure 1 Text on a Curved Path
When programming for Windows Presentation Foundation (WPF), a significant bonus accompanies the programming of text on a path: you can animate the individual points defining the path and watch the characters bounce around in response.
As usual in WPF, there is more than one way to do this job, and I'll demonstrate several approaches. As is also common in WPF, the difficult part of the job turns out to be something other than what might originally be assumed. In putting text on a path, the big problem really isn't figuring out how to move and rotate the text characters. That's relatively straightforward. The hard part is properly informing the WPF layout system of the correct size of the resultant graphic.

Uniting Paths and Text
A graphics path is a collection of straight lines and curves. Some of these lines and curves might be connected to each other. Some might form enclosed areas. In WPF, the graphics path is encapsulated in both the PathGeometry and StreamGeometry classes. StreamGeometry offers better performance, but the individual points that define the path become fixed and cannot be animated.
In placing text on a path, it's pretty much essential that the lines and curves comprising the path be connected end to end with each other. You probably don't want to deal with a text string jumping across a disconnection in the path. For this reason, the WPF class I'm really interested in is PathFigure, which is a single collection of connected straight lines and curves. A PathGeometry is a collection of one or more PathFigure objects.
The PathFigure itself defines a StartPoint and contains a collection of segments. The first segment begins at the StartPoint, and each successive segment continues where the last one left off. These segments are all derivatives of the abstract PathSegment class: LineSegment, PolyLineSegment, BezierSegment, PolyBezierSegment, QuadraticBezierSegment, PolyQuadraticBezierSegment, and ArcSegment. In other words, a PathFigure is a connected series of straight lines, Bezier curves, quadratic Bezier curves, and arcs, which are curves on the circumference of an ellipse.
A PathFigure has a geometric length. In the source code that accompanies this column, I refer to this length with the variable pathLength. If the PathFigure consists solely of LineSegment and PolyLineSegment objects, calculating the length is trivial: simply use the Pythagorean Theorem to calculate the length of each line and accumulate them. But an ArcSegment requires more complex math, and calculating the length of a Bezier curve is a positively frightening job.
To simplify the calculation of the PathFigure length, it's convenient to convert the path to a collection of polylines that approximate the curves. This is called flattening the path. It's then easy to calculate the length with repeated application of the Pythagorean Theorem. WPF even makes knowledge of the Pythagorean Theorem unnecessary: subtracting one point from another results in a Vector object, and the Vector object contains a Length property that returns the length between the two points.
The PathFigure class contains a method named GetFlattenedPathFigure that returns another PathFigure containing only LineSegment and PolyLineSegment objects. And this is ideal for calculating the length of the entire PathFigure.
A text string rendered with a particular font and font size also has a geometric size. If you use the popular TextBlock element to display text, the width of the displayed text is available from the ActualWidth property. During the layout process prior to the actual display, ActualWidth might not be set yet, but TextBlock will set its DesiredSize property to the size of the text.
TextBlock uses the FormattedText object for calculating the size of its text, and this class is also available to the application programmer. The FormattedText constructor requires the text string itself, a font size, and an object of type Typeface, which is built from the desired FontFamily, FontStyle, FontWeight, and FontStretch objects.
A couple of different approaches come to mind for uniting the PathFigure and the text string. You might want to specify a particular font size for the text and display the text starting at the beginning of the path. But the text might not fit the whole path, and (worse) it might exceed the length of the path. What then?
To avoid those problems, I took a different approach: I decided to scale the size of the text to make it exactly the length of the path from beginning to end. The text is first created with an arbitrary font size, which is 100 in the sample programs. The width of the text is stored in a variable named textLength.
The ratio of pathLength to textLength is a number I call scalingFactor. This scaling factor must be applied as a transform to the individual text characters so they fit on the path from beginning to end.
Each individual character in the text string is subjected not only to this scaling transform but also to two other transforms: a translation transform moves the character to a particular position on the path, and a rotation transform turns the character so its baseline is tangent to the path.
PathGeometry defines a method named GetPointAtFractionLength that is extremely useful for positioning text on the path. The method requires an argument ranging from 0 to 1 indicating a fractional length of the path. The method returns the corresponding point on the path. This fractional length is easy to provide. Each character occupies a fractional length of the path equal to the width of the individual character times the scaling factor divided by pathLength. The point on the path returned by the method is useful for translating the character to the path. As an extra bonus, the GetPointAtFractionLength method returns a second point representing the tangent of the path at that point. Passing the X and Y properties of this second point to the Math.Atan2 method gives you the angle necessary to rotate the text character.
Although GetPointAtFractionLength is defined by PathGeometry and not PathFigure, it's easy to make a PathGeometry from a single PathFigure for purposes of using GetPointAtFractionLength.

Endpoints and Midpoints
Even when the general technique is established, some questions remain: what part of each character should be aligned with the path? Should it be the top of the character, the bottom (below the descenders), or the baseline? I chose the baseline as the most natural approach and figured that a switch to either the top or bottom would then be trivial. FontFamily has a Baseline property suitable for this purpose. It's based on a font size of 1 unit, so the distance from the top of a character to its baseline is simply the product of the Baseline property, the font size (100 in my code), and scalingFactor.
Figure 2 shows two different ways to place text characters on a simple circular path. Look at the h, x, and n characters in the graphic on the left. You'll see that the left and right endpoints of each character's baseline touch the path. In the graphic on the right, look at the e and o characters to see that the midpoint of each character's baseline touches the path.
Figure 2 Two Ways to Align Text on a Path
At first this seems like a trivial distinction. If the circles themselves were absent, you probably wouldn't even see a difference between the two. However, I felt that the approach on the left offers slightly better visual continuity between the characters.
I was disappointed when I realized how much more difficult it is (algorithmically speaking) than the approach on the right. It's not hard to position the left end of the character's baseline on the path, but the right end requires finding a point on the path that is a straight-line distance from the first point equal to the character's scaled width. Moreover, after all the characters have been positioned, you'll discover that the last character goes beyond the end of the path. This happens because the characters have been scaled based on the curved path but positioned based on straight-line shortcuts through the path. This approach requires multiple passes with successive refinements of the scaling factor.
Although I hacked together a simple version of this algorithm for producing Figure 2, all of the downloadable code uses the much simpler midpoint approach. Each character requires a fraction of the path equal to its fraction of the total text width.

Let's Make It a UserControl!
Deriving from UserControl is an extremely popular technique in WPF programming, rather like an erector set for quickly constructing sturdy controls. UserControl derives from ContentControl so the visual tree defining the control is generally defined in XAML as the control's Content property. The codebehind file might define a couple more custom properties and handles events and interactions.
It's possible to implement text on a path using the everyday WPF building blocks of UserControl, Canvas, and TextBlock. UserControl is doubly convenient for this particular job because it already has many of the required properties to define the text: FontFamily, FontStyle, FontWeight, FontStretch, and Foreground. (The FontSize property will be ignored because the text size will be scaled based on the path length.) If TextBlock elements are descendents of the UserControl, the settings of these properties on the UserControl will be inherited by TextBlock. It will be necessary to override the metadata for these properties to supply additional property-changed event handlers, but that's about it. The only required new properties are Text (of type string) and PathFigure (of type PathFigure).
The resultant class is called TextOnPathControl, and Figure 3 shows an excerpt. The rest of the file is available in the code download, of course. The source code for this column consists of a single solution named TextOnPath, which contains a DLL project named Petzold.TextOnPath that contains all the classes that implement the various techniques for arranging text on a path. The other projects in the TextOnPath solution are demonstration programs.
Figure 3 Excerpt from TextOnPathControl.cs
static void OnTextPropertyChanged(DependencyObject obj,
  DependencyPropertyChangedEventArgs args)
{

  TextOnPathControl ctrl = obj as TextOnPathControl;
  ctrl.mainPanel.Children.Clear();

  if (String.IsNullOrEmpty(ctrl.Text))
    return;

  foreach (Char ch in ctrl.Text)
{
    TextBlock textBlock = new TextBlock();
    textBlock.Text = ch.ToString();
    textBlock.FontSize = FONTSIZE;
    ctrl.mainPanel.Children.Add(textBlock);
  }

  ctrl.OrientTextOnPath();
}

void OrientTextOnPath()
{
  double pathLength =
    TextOnPathBase.GetPathFigureLength(PathFigure);
  double textLength = 0;

  foreach (UIElement child in mainPanel.Children)
{
    child.Measure(new Size(Double.PositiveInfinity,
      Double.PositiveInfinity));
    textLength += child.DesiredSize.Width;
  }

  if (pathLength == 0 || textLength == 0)
    return;

  double scalingFactor = pathLength / textLength;
  PathGeometry pathGeometry =
    new PathGeometry(new PathFigure[] { PathFigure });
  double baseline =
    scalingFactor * FONTSIZE * FontFamily.Baseline;
  double progress = 0;

  foreach (UIElement child in mainPanel.Children)
{
    double width = scalingFactor * child.DesiredSize.Width;
    progress += width / 2 / pathLength;
    Point point, tangent;

    pathGeometry.GetPointAtFractionLength(progress,
      out point, out tangent);

    TransformGroup transformGroup = new TransformGroup();

    transformGroup.Children.Add(
      new ScaleTransform(scalingFactor, scalingFactor));
    transformGroup.Children.Add(
      new RotateTransform(Math.Atan2(tangent.Y, tangent.X)
      * 180 / Math.PI, width / 2, baseline));
    transformGroup.Children.Add(
      new TranslateTransform(point.X - width / 2,
        point.Y - baseline));

    child.RenderTransform = transformGroup;
    progress += width / 2 / pathLength;
  }
}
The OnTextPropertyChanged method is the property-changed handler for the Text property; it is responsible for creating all the TextBlock children. The mainPanel variable is a Canvas object set to the Content property of the UserControl. OnTextPropertyChanged concludes by calling OrientTextOnPath, as do the other property-changed handlers for the font-related properties and the PathFigure property.
OrientTextOnPath basically applies transforms to all the TextBlock elements. It obtains the total length of the path (TextOnPathBase is another class in the DLL) and the total width of the TextBlock objects, and calculates the all-important scalingFactor. The method concludes with a large loop through all the TextBlock children of the Canvas. Notice that the variable "progress" passed to the GetPointAtFractionLength method is based on half the scaled width of the character, so the point returned from the method corresponds to the character's midpoint.
The loop concludes with the definition of three transforms: the first one scales the size of the TextBlock based on scalingFactor. The TextBlock is then rotated based on the tangent numbers returned from GetPointAtFractionLength. Notice the center of rotation is the midpoint of the character's baseline. Finally, the TranslateTransform moves the scaled and rotated character to the desired point on the path. This group of three transforms becomes the RenderTransform property of the TextBlock.
The TextOnPathControlDemo1.xaml file shown in Figure 4 was responsible for creating the image in Figure 1. Notice that the TextOnPathControl class doesn't itself draw the PathFigure used to orient the text. If you want it drawn, you'll have to provide for it yourself. However, as the TextOnPathControlDemo1.xaml file shows, you can use the same PathGeometry for both tasks.
Figure 4 TextOnPathControl
<!-- TextOnPathControlDemo1.xaml by Charles Petzold, September 2008 -->
<Page
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:textonpath=
    "clr-namespace:Petzold.TextOnPath;assembly=Petzold.TextOnPath"
  Title="TextOnPathControl Demo #1"
  WindowTitle="TextOnPathControl Demo #1"
  FontFamily="Times New Roman"
  Foreground="Blue">

  <Page.Resources>
    <PathGeometry x:Key="path"
      Figures="M 100 100 C 200 150 300 0 400 100" />
  </Page.Resources>

  <Grid>
    <Path
      Data="{StaticResource path}"
      Stroke="Red" />

    <textonpath:TextOnPathControl
      Text="Hello, Path!"
      PathFigure="{Binding Source={StaticResource path},
      Path=Figures[0]}" />
  </Grid>
</Page>

What Size Is It?
The TextOnPathControl class shows some of the versatility of UserControl, but also its limitations. The control uses a Canvas as parent to all the TextBlock objects, and a Canvas not given an explicit height or width has no dimensions for layout purposes. If you were to put this control in a StackPanel, for example, it wouldn't be visible at all.
You could replace the Canvas in TextOnPathControl with a single-cell Grid. That would give the control a non-zero size for layout purposes, but that size would not be correct. The size of the control would be calculated as if all the individual unscaled TextBlock children were displayed on top of each other—which is precisely what the control does before applying a RenderTransform. As you know, the RenderTransform has no effect on layout.
You might consider replacing the RenderTransform with a LayoutTransform so the transformed TextBlock elements would be recognized by the WPF layout system. But that won't work either, because LayoutTransform ignores translation transforms.
Normally, a UserControl is populated with a visual tree that contains other controls and elements that report their sizes to the WPF layout system. The UserControl then has a composite size encompassing all of its children. But if you set the RenderTransform property of these children, then there's no guarantee they will be inside the element's boundaries.
Let's face it: although the TextOnPathControl class demonstrates the versatility of UserControl, it's not quite the right approach. Creating a TextBlock for each individual character is overkill and only makes sense if you want each character to be associated with its own mouse processing or perhaps its own tooltip.

Lowering the Level
Let's drop down a couple of levels and derive from FrameworkElement, which is the same base class for TextBlock itself and many other elements including Control. A class that derives from FrameworkElement generally draws the element's visual contents in an override of the OnRender method. The OnRender method is called with an object of type DrawingContext, a class that defines a complete set of drawing methods, including DrawText. This is the lowest-level graphics output you can do and still call yourself a full-fledged WPF application.
Unfortunately, FrameworkElement doesn't have any of the FontFamily, FontStyle, FontWeight, FontStretch, or Foreground properties you need, to say nothing of Text and PathFigure. Because other classes in this column will need these same properties, I put all seven of them in a class I called TextOnPathBase. This class also defines four abstract property-changed methods named OnFontPropertyChanged, OnForegroundPropertyChanged, OnTextPropertyChanged, and OnPathPropertyChanged that descendent classes override to implement measuring and drawing code. The TextOnPathElement class derives from TextOnPathBase. Figure 5 shows an excerpt from the class.
Figure 5 Excerpt from TextOnPathElement.cs
protected override void OnTextPropertyChanged(
  DependencyPropertyChangedEventArgs args)
{

  formattedChars.Clear();
  textLength = 0;

  foreach (char ch in Text)
{
    FormattedText formattedText =
      new FormattedText(ch.ToString(),
      CultureInfo.CurrentCulture,
      FlowDirection.LeftToRight, typeface, 100,
      Foreground);

    formattedChars.Add(formattedText);
    textLength +=
      formattedText.WidthIncludingTrailingWhitespace;
  }

  InvalidateMeasure();
  InvalidateVisual();
}

protected override void OnPathPropertyChanged(
  DependencyPropertyChangedEventArgs args)
{

  pathLength = GetPathFigureLength(PathFigure);

  InvalidateMeasure();
  InvalidateVisual();
}

protected override Size MeasureOverride(Size availableSize)
{
  if (PathFigure == null)
    return MeasureOverride(availableSize);

  Rect rect = new PathGeometry(
    new PathFigure[] { PathFigure }).Bounds;
  return (Size)rect.BottomRight;
}

protected override void OnRender(DrawingContext dc)
{
  if (pathLength == 0 || textLength == 0)
    return;

  double scalingFactor = pathLength / textLength;
  double progress = 0;
  PathGeometry pathGeometry =
    new PathGeometry(new PathFigure[] { PathFigure });

  foreach (FormattedText formText in formattedChars)
{
    double width = scalingFactor *
      formText.WidthIncludingTrailingWhitespace;
    double baseline = scalingFactor * formText.Baseline;
    progress += width / 2 / pathLength;
    Point point, tangent;

    pathGeometry.GetPointAtFractionLength(progress,
      out point, out tangent);
    dc.PushTransform(
      new TranslateTransform(point.X - width / 2,
      point.Y - baseline));
    dc.PushTransform(
      new RotateTransform(Math.Atan2(tangent.Y, tangent.X)
      * 180 / Math.PI, width / 2, baseline));
    dc.PushTransform(
      new ScaleTransform(scalingFactor, scalingFactor));

    dc.DrawText(formText, new Point(0, 0));
    dc.Pop();
    dc.Pop();
    dc.Pop();

    progress += width / 2 / pathLength;
  }
}
Whenever the Text property or one of the Font properties change, the class creates FormattedText objects for each character in the string. The width of these characters is accumulated in the textLength field. Whenever the PathFigure property changes, the length of the path is stored in the pathLength field. Most of the real work occurs in the OnRender override.
OnRender makes a call to the DrawText method of DrawingContext for each character in the text string. But notice the three PushTransform calls. These are the same transforms that you saw in the TextOnPathControl class, but they must be pushed on the DrawingContext stack in the opposite order that they are applied. Following the DrawText call, the three transforms are removed from the DrawingContext stack with calls to Pop.

The Measuring Conundrum
When you're working on the level of FrameworkElement, you can also inform WPF how large your element is by overriding the MeasureOverride method. Your responsibility is to return a Size object that is required by the element. MeasureOverride provides an availableSize argument that you can use to scale your graphic or completely ignore.
Here's the catch: the WPF layout system assumes that the element has been rendered in the OnRender method with its upper-left corner at the point (0, 0). If that's not the case, then you'll have a little problem. For example, suppose you want to write a derivative of FrameworkElement devoted solely to drawing a line from the point (-50, 100) to the point (100, 50). The OnRender method probably contains the call:
dc.DrawLine(pen, new Point(-50, 100), new Point(100, 50));
But what does MeasureOverride return? The standard approach is to return a Size object based on the maximum X and Y coordinates of the rendered object, which means the body of MeasureOverride is simply:
return new Size(100, 100);
Actually, MeasureOverride should take the width of the pen used to draw the line into account, but let's ignore refinements like that for this discussion.
If you now put that element in a Grid cell with a Background color of cyan, where both the column and row width are set to GridLength.Auto, the result is shown in the first graphic of Figure 6.
Figure 6 Different Ways to Draw a Line and Specify Its Size
Vertically, the grid cell allows for coordinates going down to zero, but the size does not accommodate the line going beyond the left border of the cell. You might try to compensate for this by returning a value from MeasureOverride that equals the real rendered width and height of the graphic:
return new Size(150, 50);
But doing that only makes it worse! The line still is not within the cell, and the cell is much too large, as illustrated in the second graphic of Figure 6.
You might then decide to be very clever and return a size of (150, 150) from MeasureOverride, but adjust the coordinates in the OnRender method so they have an origin of (0, 0). Just add 50 to the X coordinates and subtract 50 from the Y coordinates:
dc.DrawLine(pen, new Point(0, 50), new Point(150, 0));
Now the graphic fits snugly in the Grid cell, as shown in the third graphic of Figure 6. However, if you want to mix this graphic with other lines in that same Grid cell or on a Canvas, it won't be rendered properly because the DrawLine call has adjusted the coordinate points. If you use one of the text-on-a-path classes to display text on the path, the text and path won't be aligned.
We must reluctantly conclude that the first graphic of Figure 6 shows the correct approach. That's how the regular Line element behaves, for example.
Now let's put some text on the path, as shown in the fourth graphic of Figure 6. How should the size of the element change to accommodate the text? In this particular case, not at all. The text doesn't go beyond the maximum horizontal coordinate of the line or below the maximum vertical coordinate. While there will be other paths where the text will make a difference, it usually won't make much of a difference. That is why the TextOnPathElement method implements the MeasureOverride method based entirely on the dimensions of the PathFigure, as shown in Figure 5. The next class I'll describe will do something just a little more sophisticated.

The World of Visual Children
Any class that derives from FrameworkElement also derives from Visual and thus can maintain a collection of visual children. In a practical sense, these children are usually of type DrawingVisual. And in some measure, these DrawingVisual objects are similar to elements—that is, they have a visual appearance and they can be used for hit testing—but they are a much lighter weight. They do not receive keyboard, mouse, or stylus events, and they do not directly participate in layout.
After first creating an object of type DrawingVisual, you generally call the RenderOpen method to obtain a DrawingContext object. You use the same drawing methods with this DrawingContext object that you use in the OnRender override. Finish up with calling the Close method. The graphic rendered with the drawing commands are now stored in the DrawingVisual.
You don't directly render DrawingVisual objects on the screen. Instead, the class that creates and stores the DrawingVisual objects must override the VisualChildrenCount property to indicate the number of visual children and also override the GetVisualChild method to return a visual child based on an index.
Classes that use visual children should also call AddVisualChild and RemoveVisualChild so the visual children properly participate in event routing. This overhead is handled for you if you store the visual children in a VisualCollection object.
For displaying text on a path, it makes sense to create a DrawingVisual object for each character of the text string. Just as when drawing in the OnRender method, you can call the PushTransform method to apply a transform to the graphic. However, for putting text on a path, there's a better approach: the DrawingVisual itself has a Transform property, which means that the creation of the DrawingVisual objects and the application of the Transform can be separated just as in the earlier TextOnPathControl class. This separation allows the PathFigure to be animated without causing the DrawingVisual objects to be recreated.
The code in Figure 7 shows two methods from the TextOnPathVisuals class. The GenerateVisualChildren method is called whenever the Text property or a font property changes, and it concludes by calling TransformVisualChildren. The TransformVisualChildren method is called whenever the PathFigure changes. Notice how the GenerateVisualChildren method prepares the DrawingVisual by creating a TransformGroup with three transforms and setting that to the DrawingVisual's Transform property. These transforms are filled with the proper values in the TransformVisualChildren method.
Figure 7 Excerpts from TextOnPathVisuals
protected virtual void GenerateVisualChildren() {
  visualChildren.Clear();

  foreach (FormattedText formText in formattedChars)
{
    DrawingVisual drawingVisual = new DrawingVisual();

    TransformGroup transformGroup = new TransformGroup();
    transformGroup.Children.Add(new ScaleTransform());
    transformGroup.Children.Add(new RotateTransform());
    transformGroup.Children.Add(new TranslateTransform());
    drawingVisual.Transform = transformGroup;

    DrawingContext dc = drawingVisual.RenderOpen();
    dc.DrawText(formText, new Point(0, 0));
    dc.Close();

    visualChildren.Add(drawingVisual);
  }

  TransformVisualChildren();
}

protected virtual void TransformVisualChildren()
{
  boundingRect = new Rect();

  if (pathLength == 0 || textLength == 0)
    return;

  if (formattedChars.Count != visualChildren.Count)
    return;

  double scalingFactor = pathLength / textLength;
  PathGeometry pathGeometry =
    new PathGeometry(new PathFigure[] { PathFigure });
  double progress = 0;
  boundingRect = new Rect();

  for (int index = 0; index < visualChildren.Count; index++)
{
    FormattedText formText = formattedChars[index];
    double width = scalingFactor *
      formText.WidthIncludingTrailingWhitespace;
    double baseline = scalingFactor * formText.Baseline;
    progress += width / 2 / pathLength;
    Point point, tangent;
    pathGeometry.GetPointAtFractionLength(progress,
      out point, out tangent);

    DrawingVisual drawingVisual =
      visualChildren[index] as DrawingVisual;
    TransformGroup transformGroup =
      drawingVisual.Transform as TransformGroup;
    ScaleTransform scaleTransform =
      transformGroup.Children[0] as ScaleTransform;
    RotateTransform rotateTransform =
      transformGroup.Children[1] as RotateTransform;
    TranslateTransform translateTransform =
      transformGroup.Children[2] as TranslateTransform;

    scaleTransform.ScaleX = scalingFactor;
    scaleTransform.ScaleY = scalingFactor;
    rotateTransform.Angle =
      Math.Atan2(tangent.Y, tangent.X) * 180 / Math.PI;
    rotateTransform.CenterX = width / 2;
    rotateTransform.CenterY = baseline;
    translateTransform.X = point.X - width / 2;
    translateTransform.Y = point.Y - baseline;

    Rect rect = drawingVisual.ContentBounds;
    rect.Transform(transformGroup.Value);
    boundingRect.Union(rect);

    progress += width / 2 / pathLength;
  }
  InvalidateMeasure();
}
The result is slightly better performance than the TextOnPathElement class and the ability to indicate a layout size that takes account of the characters. The TransformVisualChildren method begins by creating a Rect object named boundingRect. As transforms are applied to each visual child, the ContentBounds property (which represents the untransformed visual) is modified by the composite transform, and then the union is taken of that rectangle and boundingRect:
Rect rect = drawingVisual.ContentBounds;
rect.Transform(transformGroup.Value);
boundingRect.Union(rect);
The result is a rectangle that encompasses all the transformed characters. The body of MeasureOverride is quite simply:
return (Size)boundingRect.BottomRight;
Figure 8 shows the TextOnPathVisualsDemo program running with text following the hills and valleys of an animated sine curve. The pink background indicates the layout size, which changes as the animation moves different characters to the right and bottom. Notice how the rectangle has expanded to accommodate the y on the right and the descender of the g at the bottom.
Figure 8 The TextOnPathVisualsDemo Program
Previously Published
To read more about my adventures with text, animation, and the Microsoft .NET Framework, check out these previous installments of the Foundations column:

Extending the WPF Animation Classes

3D Text in WPF

Vector Graphics and the WPF Shape Class

Warping the Text
A very different approach to putting text on a path is implemented in the TextOnPathWarped class. The baseline of each character is actually curved to lie on the path; each vertical line in the character remains a straight line but becomes perpendicular to the path at that point. If a path is curved, the character will fan out on the convex side but become narrower on the concave side.
This approach doesn't require separating the text into the individual characters; the preliminary step is calling the BuildGeometry method of FormattedText for the entire text string and then flattening the geometry. No transforms are involved, although a little trigonometry is required. As usual, when converting text to a path, the text loses its rendering hints and at small sizes might be difficult to read. You can animate the path, but for long text strings you might see a significant performance hit.
TextOnPathWarped also defines a ShiftToOrigin property. When set to true, the class offsets all the points in the warped geometry so the upper-left corner is (0, 0). This allows the class to report its actual dimensions in the MeasureOverride method, but the result can't be successfully mixed with other graphics. In some cases, such as the screenshot in Figure 9, you can really see the distortion of the characters.
Figure 9 The TextOn-PathWarpedDemo3 Program
While it's always interesting to see several different solutions to the same problem, which one is best? Overall, I think the best solution involves the visual children in the TextOnPathVisuals class, particularly if you will also be animating the path.

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a longtime Contributing Editor to MSDN Magazine and the author of The Annotated Turing: A Guided Tour through Alan Turing's Historic Paper on Computability and the Turing Machine (Wiley, 2008). His Web site is charlespetzold.com.


Windows With C++
X64 Debugging With Pseudo Variables And Format Specifiers
Kenny Kerr
Contents
Processor Architectures
Pseudo Variables
Format Specifiers
Visualizing Calling Conventions
Error Codes
Debugging the Security Context
For many years, Visual C++ has included a set of pseudo variables and format specifiers for use in debugging. By pseudo variables I mean terms that you can enter into a debugger watch window to have it display some value that doesn't necessarily relate to any C++ variable. Unfortunately, pseudo variables have never been well documented. Although I don't have enough insider information to document them myself, I want to share some of the pseudo variables and format specifiers that I've found most useful. I hope the discussion will inspire you to read up on the subject.
Before I introduce pseudo variables (which are sometimes called pseudo registers), I need to say a word about processor architectures and registers, because a little bit of background can make pseudo variables a lot more valuable. It will also help you understand one way the 64-bit versions of your application differ from traditional 32-bit applications. As this column is more about Visual C++ than about the smallest details of processor architecture, I'm going to limit this discussion to the x86 and x64 processor architectures. You can learn more about the differences introduced by the Itanium processor by reading the documentation in the MSDN library.

Processor Architectures
Processor architectures aren't very easy for mere mortals to fully comprehend. That's why the C language and its successors made such an impact on the software industry. At the very least, you should have an idea of what the x86 and x64 processor architectures represent, as they are the processors the vast majority of users own. Windows and Visual C++ also support the Itanium processor, but very few developers have the luxury of targeting it.
For better or worse, x86 has dominated over the last few decades with its roots in the 8-bit Intel 8080 processor. AMD and Intel made a pragmatic choice with x64 in order to retain backward compatibility with x86. Itanium, on the other hand, introduced a powerful new architecture that isn't restricted by the legacy of x86. Of course, far fewer applications are supported as a result, but it is a testament to David Cutler (who led the development of Windows NT) and his vision of a portable operating system that Windows has been able to adopt new architectures so relatively painlessly over the years.
Although many of the differences between the processor architectures are hidden behind the C++ compiler, one that developers can clearly see a benefit from is the move to a single calling convention. The x64 version of the C++ compiler only supports a single calling convention, whereas the x86 compiler supports a handful. This is a welcome change, as it eliminates many of the potential bugs introduced by a mismatch of calling conventions, particularly in managed code interop where the Microsoft .NET Framework compilers, such as C# and Visual Basic .NET, cannot check the original calling conventions in the C++ header files. Instead, they must rely on handcrafted attributes, which can easily be defined incorrectly, leading to various stack corruption bugs.
The ecx register, for example, is used by the native C++ calling convention, known as thiscall, to store the "this" pointer before a member function is called. This knowledge is useful for debugging but is dependent on the calling convention, which may not be obvious just by looking at some assembler code and register values. This is far simpler on x64 as the "this" pointer is always inserted as the first parameter and thus stored in the rcx register.
Naturally, using registers in debugging is very processor specific, but fortunately the relationship between x86 and x64 can be exploited. The x86 architecture introduced the concept of sub-registers where a sub-register consisted of the lower half of its super-register. For example, ax is a 16-bit sub-register consisting of the lower half of the eax register. The x64 architecture extends this by providing 64-bit registers that supersede the 32-bit x86 registers. For example, on x64 eax is a 32-bit sub-register consisting of the lower half of the 64-bit rax register. Both eax and rax are interesting, as they are used by x86 and x64, respectively, for pointers or integers returned by a function. Naturally, for a function running on x86 to return a 64-bit value it needs to use a pair of registers.

Pseudo Variables
Probably the most common pseudo variable is $err, which displays the last error value set with the function SetLastError. The value that is displayed represents what would be returned by the GetLastError function.
Pseudo variables can also be used to display the value of a processor register. For example, $ecx displays the value of the ecx register from the x86 architecture. Looking at the values of processor registers might seem a bit enigmatic to the new generation of developers using .NET, but it can really help in diagnosing obscure bugs or just understanding a program's behavior at run time. You can use $ecx to display the address of the "this" pointer, assuming you're using the native C++ calling convention on x86. You can also use $eax to display a 32-bit return value on both architectures. There are many more registers that may come in handy depending on your debugging needs.
Other useful pseudo variables include $handles, which displays the number of open handles to kernel objects in the process, and $user, which displays very detailed information about the current process and thread tokens. The latter can be useful for debugging impersonation-related code. Figure 1 lists some common pseudo variables.
Figure 1 Pseudo Variables
Pseudo Variable | Description
$handles | Number of handles to kernel objects
$vframe | Current stack frame address
$TID | Current thread identifier
$registername | Contents of specified register
$clk | Time in clock cycles
$user | Process and thread token information

Format Specifiers
Format specifiers come in handy when you want to change the way a watch window displays the value of a variable, pseudo variable, or expression. In most cases, the watch window will figure out the best format for the value based on its type, but there are cases when you might need to override this. The debugger might not be able to deduce the type of variable, or you might just have a memory address that naturally has no type information. For example, the hr format specifier will display the Win32 or HRESULT corresponding to the value. Other examples include su, which will display the value as a null terminated Unicode string.
Of course, you might want to get at the actual value behind some formatted text and ! will do just that. Finally, given a pointer, you can force the watch window to treat it as an array of n elements using a comma followed by the number of elements to display. Figure 2 lists format specifiers.
Figure 2 Format Specifiers
Specifier | Description
D | Decimal
U | Unsigned decimal
O | Octal
X | Hexadecimal
F | Floating point
E | Scientific notation
C | Character
S | Character string
Su | Unicode string
s8 | UTF-8 string
Hr | HRESULT or Win32 error code
wc | Windows class
wm | Windows message
! | Raw format

Visualizing Calling Conventions
With that background information out of the way, let's take a look at a few examples. Consider the snippet below. The x86 compiler defaults to the __thiscall calling convention for member functions. This indicates that the parameters are pushed onto the stack and the "this" pointer is stored in the ecx register:
struct Sample
{
    size_t m;
    Sample() : m(0x11223344) {}

    size_t Method(size_t p1, size_t p2)
    {
        return 0x44556677;
    }
};
Figure 3 illustrates this and more. The watch window displays interesting variables. $vframe provides the address of the current stack frame. This pseudo variable provides a processor-neutral way of getting the address of the stack frame. Next are the addresses of the two parameters. Their values are 0x22334455 and 0x33445566.
Figure 3 32-Bit Stack and Registers
The first memory window also displays the current stack frame; you can see the values of these parameters on the stack. Next is the ecx register; I've cast it as a pointer to a Sample to display the members of the object. The second memory window also displays the object in memory; you can see the member variable farther down in the stack. Next, the $eax variable displays the value of the register that is storing the result of the function call.
The same C++ code compiled by the x64 compiler produces noticeably different results. The compiler takes advantage of the increased number of registers available on x64 and uses them heavily to avoid pushing values on the stack unnecessarily. Specifically, the first four pointer or integer parameters are placed in the rcx, rdx, r8, and r9 registers, respectively. For member functions, the "this" pointer is treated as the first parameter. Space is reserved in case the function needs to save any parameters to the stack temporarily. Finally, any pointer or integer result is returned in the rax register.
Figure 4 illustrates this. Again, $vframe displays the address of the current stack frame. This time you can see that the values are all 64 bits. Next are the addresses of the two parameters. Though the parameters were passed with registers to the called function, these parameters are in the current stack frame. This is a debug build. The function saved the parameters to the stack. The next three variables, $rcx, $r8, and $rdx, provide the values of the three parameters, including the prepended "this" pointer. Finally, $rax displays the value of the register that's storing the result of the function call.
Figure 4 64-Bit Stack and Registers

Error Codes
Pseudo variables and format specifiers are also very useful for looking at error codes in various forms and locations. Figure 5 lists a few examples, such as $err,hr, which provides the value and description for the last error code. This equates to the ERROR_VOLMGR_DISK_NOT_EMPTY error code from the Volume Shadow Copy Service. The $err pseudo variable provides the value, and the hr format specifier tells the watch window to format it as an error code. Note also that hresult is a C++ variable, and the watch window automatically displays the well-known constant representing its value. E_INVALIDARG,x shows how you can override the automatic formatting. In this case the x format specifier instructs the watch window to display it as a hexadecimal value. The last example shows how you can turn almost any address into an array.
Figure 5 Error Codes

Debugging the Security Context
Figure 6 shows the impressive $user pseudo variable. You can see that Bob is impersonating Alice! This pseudo variable provides much detail about both the process and thread tokens and is handy when debugging impersonation issues in client/server applications.
Figure 6 The $user Pseudo Variable
There are more tricks you can use in the Visual Studio watch windows and a handful for debugging C# code. Plus, the Debugging Tools for Windows have an extensive set of pseudo variables. And for more debugging tips, see Advanced Windows Debugging.

Send your questions and comments to mmwincpp@microsoft.com.

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. You can reach Kenny at weblogs.asp.net/kennykerr.


Going Places
How Connection Manager Connects
Marcus Perryman
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
PDP Contexts
Modeling the Network Topology
Making Connections
Connection Requests and Arbitration
Using the SDK Sample Code
I remember it was day three of the conference. I was again sitting in an icy blast of air-conditioning, my mind slightly fuzzy from two previous days of information overload—and maybe being a little too late leaving the bar the night before—listening to the lead PM expound the wonders of the all-new Connection Manager technology.
I came out thinking, what's the point of all that? I mean, who needs a technology that tells me if my device is connected? At the time, connectivity was only possible via umbilical cables or colossal appendages that required a PhD in network protocol analysis and unending patience to operate: if the users didn't know that the device was connected, they weren't qualified to own it!
Thankfully technology has moved on, and today the average mobile phone offers such a plethora of exotic radio technologies it could fry the contents of your lunch box at 40 paces. Windows Mobile devices occupy the more generous end of this scale, and married with the Windows CE preemptive multitasking kernel, the foresight of the Connection Manager team has become very evident.
Without Connection Manager, for all but the simplest network environments, the only way an application could choose a connection would be by presenting the user with a list and asking him to select one—which puts you back to requiring a PhD in network protocol analysis. Connection Manager neatly moves this problem away from the user—who, after all, just wants the application to work—and makes it the mobile operator and OEM's problem, reasoning that if they can't figure it out, the poor user has no hope.
There are two main reasons why the correct use of Connection Manager is vital when a Windows Mobile application requires network data. First, it's not just your code that needs to connect. Many of today's applications are connected or at least offer some form of online ability to download updates or additional features. This increases the likelihood that two applications will contend for connectivity at some point on the device. For enterprise users using push or poll e-mail solutions, this is almost a certainty. Second, Windows Mobile devices are sold and used around the world. An application that successfully appeals to a global market must navigate any network configuration a mobile operator might conceivably dream up—and in my experience mobile operators have active imaginations when it comes to network topology!
I'm a firm believer that better understanding leads to better use. In this column, I want to pull up the hood and shine a light on some of Connection Manager's inner workings, specifically looking at how it adds value and removes complexity when connecting over wide-area cell phone networks. This might appear more relevant for C++ developers who commonly call Connection Manager APIs directly. Managed code developers have many classes, such as HttpWebRequest, that use Connection Manager without the developer being directly involved. However, even for .NET developers, understanding how the managed code is powered by Connection Manager hidden under the surface can be useful for resolving complex connection issues.

PDP Contexts
Much of the network complexity is because mobile operators must tread a narrow path among three competing requirements. First, they have customer demand to match the blistering pace of change in radio network technologies. Second, they provide radio coverage to huge geographical regions through a vast infrastructure. And third, they try not to break the bank for their shareholders.
One area that highlights the challenge and directly affects software developers is related to Packet Data Protocol (PDP) contexts. A PDP context describes an agreement between a phone handset and the Global System for Mobile Communication (GSM) network and contains important information, such as the Access Point Name (APN) and IP address. This agreement is immutable—once created, it can't be changed. (A more complete explanation can be found in the GPRS Core Network article on Wikipedia.)
Before any application can send and receive data, the device must first establish a PDP context by making a request to the base station, sending the name of the desired APN. The request is typically forwarded into the mobile operator's billing network to authorize the request, making sure the user has a valid contract and sufficient available credit, for example. Upon completion, an IP address is supplied by the APN and handed to the base station and on to the handset. At this point, the device can send and receive data under the assigned IP address.
Surely one PDP context is enough? Not always—APNs provide a convenient way to vary the billing or traffic restrictions in different ways. For example, a specific APN might provide a direct bridge to an enterprise's private network, restricting access to the APN via the operator's billing systems, or one APN might be used for per-megabyte billing and another for an "all you can eat" service. This practice is widespread. A real-world example is for a mobile operator to require all push e-mail data to go via a specific APN and other Internet traffic through another.
As I said earlier, the PDP request typically routes through the billing infrastructure, and that can take some time. It varies between operators and networks but can take up to 20 seconds to create the context. Remember, PDP contexts are immutable, so the APN cannot be dynamically swapped. Instead, the whole context must be dropped and a new one created.
For performance alone, multiple concurrent PDP context support would be a good thing. But as you would imagine, this requires extra handset and base-station capabilities. Fortunately, most modern handsets support three or more contexts, but it's a little more interesting when it comes to base stations. Although most new installations support this capability, there are still a good number of first-generation 3G base stations in use that only support single PDP contexts per handset.
That all leads to quite a complex set of rules to ensure each application is connecting in the right way—and the right way may frequently change as the connected base station and radio service level changes.

Modeling the Network Topology
Connection Manager is primarily designed to do three things:
  • Be sufficiently flexible and configurable so mobile operators and OEMs can describe their many and varied network topology and device capabilities.
  • Reduce and simplify the steps a developer must take to select and connect to network resources on any handset in any network.
  • Provide arbitration among applications contending for limited resources.
Considering the complexities of PDP contexts I described earlier and that Connection Manager extends its support beyond General Pocket Radio Service (GPRS) to include WiFi, dial-up, wired, and even desktop pass-through connections, this is not an easy target to aim for!
The first step is to give Connection Manager a complete picture of the available connections, modeled in a sufficiently abstract manner to encompass emerging technologies that become mainstream during the system's lifetime. Connection Manager provides a suite of simple and consistent configuration primitives that allow mobile operators and OEMs to model their hardware and network topology, all of which are configurable via Open Mobile Alliance (OMA) Device Management, OMA Client Provisioning, or, in some cases, through the settings user interface.
Here is a simple OMA Client Provisioning XML sample to give you an idea of what it looks like:
<wap-provisioningdoc>
  <characteristic type="CM_GPRSEntries">
    <characteristic type="Internet GPRS">
      <parm name="DestId" value="{436EF144-B4FB-4863-A041-8F905A62C572}" />
      <characteristic type="DevSpecificCellular">
        <parm name="GPRSInfoValid" value="1" />
        <parm name="GPRSInfoAccessPointName" value="MyInternetAPN" />
      </characteristic>
    </characteristic>
</wap-provisioningdoc>
The basic components start with the meta-network. This is the most abstract concept that Connection Manager uses and can't really be mapped to anything truly tangible, such as a network card or a computer. However, these entries are considered the endpoints or nodes of the network model, often given descriptive names, such as the Internet, DMZ, or Walled Garden, and are identified by GUIDs. There is a predefined list of commonly used destinations, such as "Internet" and "work," published in the MSDN Library and deployed on all devices. However, the list is fully extensible. CM_Networks is the OMA Client Provisioning configuration name when provisioning.
GPRS and other packet data connections are described in their own configuration section that includes relevant information for connecting, such as the APN name, user/password information, and many other settings. These settings must also contain the ID of a meta-network that is available when the connection is established—this is called the destination meta-network. There is no source network for packet data entries because it must always be the device. CM_GPRSEntries is used for OMA Client Provisioning.
There are separate entries for WiFi, dial-up, and other hardware that can be managed by Connection Manager. VPN and proxy entries both have their own configurations but are slightly different because they provide a connection from one meta-network to another—there is a source meta-network and a destination meta-network.
I need to add one more very important configuration setting: the mapping table (CM_Mappings is used for OMA Client Provisioning). Strictly speaking, it doesn't model anything about the network topology; instead, this table is for applications to discover to which meta-network they need to connect. An application is likely to know the URL that it needs but has no clue which meta-network is able to access that URL because it changes from operator to operator, device to device.
The mapping table defers the choice of meta-network to the mobile operator or OEM via an API that converts the provided URL into a meta-network GUID. Without the mapping table, the destination meta-network would either be hardwired or provided by the user, which brings us back to the old issue of relying on the user.
Figure 1 shows an example of how the network topology might be modeled and a corresponding mapping table. The mapping table is an ordered list of URL patterns, each paired with a meta-network GUID and interrogated by the ConnMgrMapURL API. An application passes its target URL to ConnMgrMapURL, which walks through the table starting at the lowest ID looking for the first pattern match with the URL. If a match is found, the GUID is returned and can be handed to ConnMgrEstablishConnection to make the connection.
Figure 1 Example Network Topology
Figure 2 shows an example of what the mapping table might look like in XML form. It shows six mapping entries, 501 being the first. It also shows most of the pattern match possibilities. The pattern isn't quite a regular expression, but it provides a good range of flexibility. * is used as the wild character. So, for example, *://*.operator.*/* will map to any protocol—any address that contains the text pattern .operator. followed by a / and any trailing page name. The following URLs would match this destination network:
http://www.operator.com/
rtsp:// rtsp.operator.media.com/20070707_ABHDD3227DD/today_news1.sd
https://my.long.name.operator.da/index.aspx
Figure 2 Example of XML for Mapping Table
<wap-provisioningdoc>
  <characteristic type="CM_Mappings">
    <characteristic type="501">
      <parm name="Pattern" value="*://*/*.3gp"/>
      <parm name="Network" value="{D3B2D798-9E69-4B65-A75B-6DDFBECEAAAA}"/>
    </characteristic>
    <characteristic type="610">
      <parm name="Pattern" value="*://*.operator.*"/>
      <parm name="Network" value="{7022E968-5A97-4051-BC1C-C578E2FBA5D9}"/>
    </characteristic>
    <characteristic type="536870912">
      <parm name="Pattern" value="wsp://*/*"/>
      <parm name="Network" value="{7022E968-5A97-4051-BC1C-C578E2FBA5D9}"/>
    </characteristic>
    <characteristic type="553648128">
      <parm name="Pattern" value="wsps://*/*"/>
      <parm name="Network" value="{F28D1F74-72BE-4394-A4A7-4E296219390C}"/>
    </characteristic>
    <characteristic type="570425344">
      <parm name="Pattern" value="*://*.*/*"/>
      <parm name="Network" value="{436EF144-B4FB-4863-A041-8F905A62C572}"/>
    </characteristic>
    <characteristic type="587202560">
      <parm name="Pattern" value="*://*/*"/>
      <parm name="Network" value="{A1182988-0D73-439E-87AD-2A5B369F808B}"/>
    </characteristic>
  </characteristic>
</wap-provisioningdoc>
The last four entries are the default entries found on Windows Mobile Standard, and the last two entries identify Internet (with . in the address) and work traffic, respectively.
Sure, this is ideal for general-purpose applications, such as a browser or media player where many and varied URLs are input from the user, but isn't it overkill for an application with just one URL? Even for the one-URL scenario, I would strongly encourage you to make use of the mapping table each time you connect because that's the only way to get the right meta-network in every network configuration the world over.

Making Connections
With a model of the network, Connection Manager can do its magic. There is little value in regurgitating the API documentation here—the Connection Manager API documentation in the MSDN Library does a much better job of explaining. Instead, I think it's more valuable to highlight the normal sequence of APIs and call out key settings.
To make a connection, the sequence is usually this:
  1. Call ConnMgrMapUrl for the meta-network to connect to.
  2. Prepare the CONNMGR_CONNECTIONINFO structure that describes what you want Connection Manager to do.
  3. Call ConnMgrEstablishConnection (there is also a synchronous version of this API) to create the request.
  4. Monitor Connection Manager messages and track any status changes that may occur.
  5. When you are finished with the connection, call ConnMgrReleaseConnection to close the request and allow resources to be reallocated.
Figure 3 shows a simplified example of the API sequence for connecting. There are several bits of the CONNMGR_CONNECTIONINFO structure that are important and often set incorrectly (see the CONNMGR_CONNECTIONINFO article on MSDN Library for details of the full structure). I will explain some of the important pieces here:
DWORD dwFlags There are several flags developers need to be aware of. For example, the Proxy Aware options tell Connection Manager with which types of proxy server your code can communicate. Based on these flags, it will include or exclude certain meta-networks available via those types of proxy server. Wherever possible, you should write proxy-aware code (HTTP proxy is usually enough). It's not difficult! For native C++ code, make use of the InternetOpen or InternetOpenURL WinInet APIs and it's all pretty much done for you. Managed code is even easier, as many of the .NET Compact Framework managed classes (HttpWebRequest, for example) already support proxy servers.
DWORD dwPriority Priority setting is very important because this is the primary way Connection Manager decides how to share scarce connection resources. I'll expand on this later.
BOOL bExclusive This setting can be used to stop Connection Manager from sharing the network connection with other applications, even for the same destination meta-network. It might be useful as security mitigation but should be employed with care.
Figure 3 Example Code for Using Connection Manager
#include <windows.h>
#include <Connmgr.h>
#include <Wininet.h>

HRESULT ConnectionManager_Connect() {
  HRESULT hr = S_OK;
  GUID dest;
  CONNMGR_CONNECTIONINFO connInfo;

  // look up the URL and connect to the specified destination
  hr = ConnMgrMapURL(g_urlString, &dest, NULL);
  if (SUCCEEDED(hr)) {
    memset(&connInfo, 0, sizeof( connInfo));
    connInfo.cbSize = sizeof( connInfo );
    connInfo.bDisabled = FALSE;
    connInfo.bExclusive = FALSE;
    connInfo.hWnd = g_hWnd;
    connInfo.guidDestNet = dest;
    connInfo.dwParams = CONNMGR_PARAM_GUIDDESTNET;
    connInfo.dwFlags = CONNMGR_FLAG_PROXY_HTTP;
    connInfo.dwPriority = CONNMGR_PRIORITY_USERINTERACTIVE;
    connInfo.uMsg = WM_USER+101;    // Message callback.
    hr = ConnMgrEstablishConnection(&connInfo, &g_ConnHandle);

    if (!SUCCEEDED(hr)) {
      LogMessage( TEXT("Failed to Establish connection") );
    }
  }

  return hr;
}

HRESULT ProcessConnectionManagerMessage() {
  HRESULT hr = S_OK;
  DWORD status = 0;

  // Connection Manager status change
  hr = ConnMgrConnectionStatus( g_ConnHandle, &status);

  if (SUCCEEDED(hr)) {
    switch (status) {
      case CONNMGR_STATUS_CONNECTED:
        // Do something useful here…
        // Done with the connection—tear it down.
        ConnMgrReleaseConnection(g_ConnHandle, 1);
        g_ConnHandle = NULL;
        break;

      case CONNMGR_STATUS_NOPATHTODESTINATION:
      case CONNMGR_STATUS_CONNECTIONFAILED:
      case CONNMGR_STATUS_CONNECTIONCANCELED:
      case CONNMGR_STATUS_CONNECTIONLINKFAILED:
      case CONNMGR_STATUS_CONNECTIONDISABLED:
      case CONNMGR_STATUS_AUTHENTICATIONFAILED:
        // Non transient error state.
        ConnMgrReleaseConnection(g_ConnHandle, 1);
        g_ConnHandle = NULL;
        break;

      default:
        // These states will eventually end up at as connected
        // or not. Just ignore for now.
        break;
    }
  }
  return hr;
}

LRESULT CALLBACK WndProc(
  HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam) {

  switch (message) {
    case WM_USER+101:
      ProcessConnectionManagerMessage();
      break;

    case WM_COMMAND:
      if (IDM_GET == LOWORD(wParam)) {
        ConnectionManager_Connect();
      }
      break;

    default:
      return DefWindowProc(hWnd, message, wParam, lParam);
  }
  return 0;
}
Calling ConnMgrEstablishConnection creates an internal connection request (CR) for tracking the life of the connection. The CR is released when ConnMgrReleaseConnection is called.
Here is a summary of the steps Connection Manager runs through when processing a CR. First, Connection Manager verifies that a connection path can be found to the destination meta-network. Connection Manager examines the CR and, using the network model (GPRS, WiFi, VPN, or whatever else might be available, along with the proxy details), it selects the best set of connectors that span to the destination meta-network.
Connection Manager isn't limited to just one connector. For example, in order to get to the Work meta-network it might need a GPRS connection plus a VPN connection. Connection Manager might also choose different paths depending on the current connection state of the device. For example, if the device is not currently connected, then a direct GPRS connector may be best. However, if the device is currently connected to a Work network, then simply using a proxy server may be sufficient. If Connection Manager is unable to find an appropriate path to the destination, the application is notified and no further steps are taken.
Next, the CR's state transition journey begins. Every CR will travel through a number of transient states before ending up eventually at one of the five or so closed states (connected state is also transient). The requesting application will receive windows messages when state changes occur. Once a CR reaches a closed state, it won't change again. So if the application still needs to be connected, it should close the CR and create a new CR.
A picture is worth a thousand words, and the best Connection Manager state transition diagram I've seen so far is on Adam Dyba's blog.

Connection Requests and Arbitration
Just because you created a new CR doesn't mean it will immediately get connected. Arbitration takes place: CRs are ordered by request time within priority order, then resources are allocated from the top of the list down so that the highest priority, most recently requested CR gets first pickings and the lowest priority, oldest CR is last in the list.
Arbitration is triggered not only by new CRs but also by changing a CR priority, releasing a CR, and by events from network resources, such as changing base station.
Voice calls also use Connection Manager and use a reserved highest priority flag, CONNMGR_PRIORITY_VOICE, that overrides all other CRs. When there is a voice connection, connected CRs that share the voice resource can be placed in a suspended state for the duration of the voice call, then automatically reconnected when the voice CR has been released.
Network complexities, such as PDP contexts, can have an impact on arbitration. If only one context is available, then just the top CR in the list will get connected—other CRs will go to the WAITINGFORRESOURCE state. It's actually a little more complex because any lower priority CRs for the same meta-network may also enter the CONNECTED state depending on the exclusive settings in each CR.
With multiple PDP contexts, Connection Manager will connect a number of CRs starting at the top of the list. It's important to remember, as stated previously, that new CRs always take priority over existing CRs of the same priority.
Think about this for a second—there is a very nasty trap here that unwary developers can easily fall into. Let's say for a moment that I have an always-connected application and my application requires connection to a private APN with no Internet access. My application is running in the background, and its CR is connected so data merrily flows back and forth to the server.
Additional Resources

Connection Manager Documentation in MSDN

Connection Manager API

Write an IM App with the .NET Compact Framework

Accessing Data from a Mobile Application
Presently the user fires up a Web browser to see how well her stock portfolio is doing. To access general Internet content, another unrestricted APN connection is required, so the browser issues a CR for a new destination meta-network. With multiple PDP contexts, the second CR would connect and quite happily run side-by-side. However, with a single PDP context, the later CR will supersede my application's request and Connection Manager will close my CR to connect the browser.
Say I wrote my application to be resilient to connection failures, knowing that the user is frequently on the road. When my original CR is closed, my application immediately requests a new one. Because it's at the same priority and newer than the browser CR, it will immediately win back the resource and my application gets connected once again. This all happens very quickly, so the browser won't get to a connected state and is likely to report an error to the user. It's unclear to the user why the browser failed, so she tries again, going round the loop but never actually getting connected.
Now imagine instead of a browser you have another background application. It's more than likely that you just created a denial-of-service situation as the applications battle to stay connected, but neither one can.
What's the application doing wrong? Most likely it's not setting the right connection priority: USERINTERACTIVE is overused and should only be set when the user has directly caused the connection to be established—like a browser page request. Later, if the application is no longer user interactive—for example, a new application window is displayed over yours—ensure the priority of your CR is changed to something lower, such as USERBACKGROUND, using the ConnMgrSetConnectionPriority priority API. This makes race conditions less likely.
Additionally, it may be appropriate to implement back-off logic so that, when the connection is dropped, a new CR isn't immediately raised. Instead, a delay related to the number of consecutive failures is introduced before trying again.
The final step in setting up connections is for Connection Manager to disconnect demoted CRs and connect new ones. Connection Manager first drops released resources before connecting new requests. If no new request is required when a CR is dropped, then Connection Manager has the ability to immediately disconnect or hold the connection cached for a configurable period of time, which can be useful for avoiding reconnect costs.
You may have spotted an interesting challenge that comes about with multiple PDP support—when multiple PDPs are connected, the device has multiple IP addresses for the same adapter, one from each context. How can applications use the right IP address for the right network?
This is a little more magic from Connection Manager. When a CR becomes connected, the IP address from the PDP context is used by the socket library code for all subsequent automatic socket bindings in that process. Notice this is done at process boundary granularity and, as the documentation states, Connection Manager supports only one CR per process. The API doesn't stop you from making more than one CR from a process, but if you do, make very sure all CRs resolve to the same PDP context and subsequent IP address. Otherwise, the behavior of Connection Manager is undetermined.

Using the SDK Sample Code
It's worth pointing out that the SDK sample code for Connection Manager is intended as exactly that—sample code to show the capabilities of the feature. When moving from the sample to production code, be sure to update the priority handling correctly and update the state change handling to process transient states correctly. States like WAITINGFORRESOURCE or WAITINGFORNETWORK are intended as hints so that visual progress or status can be correctly shown to the user.
The sample code treats almost all transient states as failures and destroys the CR. Production code needs to be more resilient to these intermediate states and only consider the connection failed when it's reached a non-transient state.
Connection Manager is a powerful and often underestimated technology. I would encourage you to take another look at how you use it to make sure all applications play well together on even the most obscure network configuration.

Send your questions and comments to goplaces@microsoft.com.

Marcus Perryman has worked at Microsoft for 11 years in various technical roles, including Developer Evangelist and Developer Consultant. At present, Marcus works as a Software Design Engineer in the Windows Mobile product group designing and developing the next generation of mobile operating systems.


Usability in Practice
The Human Face Of Software
Dr. Charles B. Kreitzberg and Ambrose Little
Contents
So, What Is User Experience?
It Ain't Just Paint
The Facets of UX
Synergy with Agile
Welcome to Usability In Practice. This is the first in a series of columns that will focus on the design of the user experience (UX). In the past, user experience was not a high priority for most development projects, but that's changed. Today, end users have a lot of experience with the Web and with software. They want design that's easy to learn and use and that fits their workflow. This column will show you how to deliver such designs.
Although UX design is a well-developed field, its integration with development processes has lagged. This is a bit surprising since UX is, by its nature, a highly iterative and agile process. However, with the increasing involvement of the development community, we expect that development frameworks will pay more attention to UX design and its critical touchpoints.
We, the authors, are both user experience designers, but each has come to the field through a different path. One of us (Charlie) was trained as a cognitive psychologist, and the other (Ambrose) has come up through the ranks as a developer. We both have a passion for UX, but we see it through different lenses. Our mutual goal is a design methodology that fits comfortably into development processes and which treats good UX as essential to the creation of successful interactive products.

So, What Is User Experience?
The term "user experience" describes an individual's subjective experience when she is interacting with a product. Every day we discuss usability informally when we report a great experience at a restaurant or a miserable one at the movies. But when designing or evaluating the user experience of an interactive product, we need a more structured way of looking at the product and discussing it.
To UX designers, success means creating products that are useful, usable, and desirable. To be useful, the product must deliver value to the user. In development terms, this means that the product must implement the right functionality. Usable products are easy to learn and use. Desirable products are products that users crave. The more desirable a product, the more time and energy a user will be willing to devote to it. Passionate users are a product's best advocates.

It Ain't Just Paint
People often mistake UX for visual design, equating it with colors, fonts, and aesthetics. Of course, graphic design is a component of the user experience, but the UX design process goes far deeper. At its core, UX is about how functionality is implemented in terms of user-system interactions.
Functionality specification has always been part of development. In an admittedly oversimplified view of traditional design, a business analyst produces a set of requirements, which are handed off to a developer. The developer then determines how to implement the requirements and produces a product. However, traditional development methods have often ignored an analysis of task flow.
While a business analyst tends to be concerned with what a product should do, the UX designer focuses on how the function should present itself though the UI. Because users have different skills and needs, the UX designer determines what user segments need to be supported. The designer will gather data about the user segments and may create personas—fictional biographies of users based on real user data—that help focus the discussions during the design process. Using interviews and observation, the UX designer will learn what tasks the users perform, how they approach the tasks, and what words they use to describe the tasks. This data may be summarized as scenarios, user stories, or storyboards.
With the user segments and task flow documented, the UX designer can then begin producing designs. These are typically produced in a highly iterative fashion, starting out at a high level of conceptualization and increasingly refined as the design progresses.
Usability testing is a form of design quality assurance (QA) used to identify problems with the design concept. Typically, participants are drawn from the identified user segments and asked to perform specific tasks. In early stages, the UI may be simply drawn on paper and the participant points to the controls to show how the task would be performed. In later stages, high-fidelity prototypes are frequently used.

The Facets of UX
User experience design is comprised of a number of subfields. Most UX designers have skills in all the core areas, but some are more specialized. The key areas are interaction design (IxD), information architecture (IA), user research, visual design, and usability testing.
Interaction Design Defines how the user interacts with the product. It specifies the behavior of the product in response to actions by the user and focuses on the product's navigation as well as the specific controls that are used.
Information Architecture Defines how information is organized and presented. Its goal is to support discoverability and usability.
User Research The process of studying users in order to develop a design that meets their needs, capabilities, and preferences. The methods are varied and typically employ a number of interview techniques along with observation. User logs and secondary sources are also used. Surveys and focus groups can also be used although many UX designers avoid them.
Visual Design The graphic treatment of the UI. Visual designers may be graphic artists rather than UX designers. While visual excitement is important, it is also essential that the visual design does not impair readability or usability.
Usability Testing The process of observing users performing specific tasks on a prototype or mock-up of the UI. Traditionally, usability testing was performed in a lab with observers behind a one-way mirror. However, increasingly, usability tests use webcast technology.

Synergy with Agile
Conceptually speaking, the reasons for designing good UX are to increase effectiveness and efficiency, to reduce development overhead and rework, and to add the maximum possible value, whether for internal operations or for external customers.
Sound familiar? I hope so. These are essentially the same goals behind the Agile approach to software development. In fact, these goals are pretty much central to the building of all business software. Designing for good UX is right in line with basic business strategy and other software development core values. At a high level, it is a matter of putting the focus where it should be for almost all business software—on those for whom the software is being built.
It's not effective to ask users to articulate what a system should do and how. That approach puts the burden on users to translate their own needs into what they think software should do, allowing for little discovery of better alternatives, which a designer or developer might be able to suggest. User-centered design (UCD) adds value by focusing on understanding the people—their goals, tasks, and activities—and using that knowledge as the basis from which to build.
The techniques used (such as personas, scenarios, storyboards, mock-ups, prototyping, and usability testing) combined with an applied knowledge of human factors help increase the probability that the end product serves its audience as best it can. This is not to say that you ignore other concerns (chiefly, business/market and technical) but that the other concerns revolve around the humans who will be using the software.
Moreover, usability testing ideally involves the actual user audience to validate early and often (iteratively) in order to ensure that the software being created has the fewest possible usability bugs. This is a better approach than user acceptance testing (UAT) because UAT typically comes at the end of the development cycle, at which point it's almost always too late or too expensive to fix problems. It's also different in that ideally usability testing is moderated by someone with some training in human factors and who observes how users are interacting with the system, which often reveals usability problems that users are not even consciously aware of.
Practically speaking, there is a lot of synergy with Agile. Both are essentially iterative. Both require regular collaboration with the people (ideally, the actual users) for whom the software is being built.
Although there is particular synergy with Agile, it is worth noting that you can enhance any development process with UCD techniques. In all cases, it offers a similar value of enhancing the opportunity for project success by heightening awareness and knowledge of real users, their needs, goals, tasks, and activities, providing tools for better discovery of the real application requirements, and increasing the likelihood that the underlying goal of all software—to enhance some aspect of human interest—will be met.
In future installments of this column, we'll be exploring more concrete examples for incorporating the knowledge, techniques, and tools of UCD into your software development practice.

Dr. Charles Kreitzberg is CEO of Cognetics Corporation, which offers usability consulting and user experience design services. His passion is creating intuitive interfaces that engage and delight users while supporting the product's business goals. Charles lives in Central New Jersey where he moonlights as a performing musician.
Ambrose Little lives with his wife and four children in central New Jersey. He's been designing and developing software for more than 10 years and is honored to be an INETA speaker and Microsoft MVP. Lately, he's shifted from technical design to designing for people and is now a user experience designer for Infragistics.


{ End Bracket }
What Makes A Good Software Tester?
Dr. James McCaffrey

Recently, in conversations with software engineers and managers at Microsoft and other software companies, I have heard the same message repeatedly: finding really good software testers has always been difficult, but it seems to have become much more difficult in the past few years.
So what makes a great software tester? I narrowed it down to the eight qualities I think are most important.
  1. Passion for Analysis and Testing The key to success in any job is having a true passion for what you do. A great software tester will generally get more satisfaction from writing some test automation that exercises a software application than in writing the application itself. This characteristic is often associated with people who have college degrees in quantitative fields such as mathematics and economics, as well as in computer science. To a large extent, this passion for analysis seems to be an innate, rather than acquired, characteristic.
  2. Technical Skill A great software tester must have significant coding skills in order to understand the system under test, communicate with developers, and write test automation. Most technical skills can be learned through education or experience. There is some disagreement among my colleagues about exactly what level of coding skill is needed to be a great tester, but I believe that a great tester must have at least 7.5-on-a-scale-of-1–10 (applications) development skills.
  3. Raw Intellectual Ability A great software tester has to be smart, with strong analytical and logical ability. Software development is essentially an exercise in logic, and to operate at a high level in this environment a software tester simply has to be smart.
  4. Ability to Prioritize and Organize Although necessary for almost any job, these talents are critically important for software testers. Software development and testing is a highly dynamic and fluid activity; the key variables can change weekly or even daily. The ability to recognize, interpret, and organize around these frequently changing job environment factors is essential.
  5. Ability to Adapt and Learn In software engineering, new technologies appear with incredible speed. For example, the appearance of C# and the Microsoft .NET Framework in early 2003 completely revolutionized software application development. Great software testers must have an ingrained propensity to be ongoing learners and consistently spend part of every week dedicated to upgrading their skill sets. I performed some research a couple of years ago that suggested that a software engineer's "skill set half-life" is about 18 months—meaning that without a significant upgrade in skills, a software engineer will be rendered substantially irrelevant in the job market very quickly.
  6. Ability to Work without Direct Supervision This is another generic job skill but is a characteristic that is particularly important for software testers. The necessity for taking independent action—quickly recognizing a technical or business need related to the software testing effort, and the ability to know what steps to take without the need of managerial approval (and also the ability to know when not to forge ahead without a management OK) is an essential skill.
  7. Ability to Communicate A great software tester must have strong written and verbal communication skills. A great tester must be able to read and analyze product documentation, write test plans, write clear bug reports, write coherent status reports to management (both formal reports and ad hoc e-mail reports), and have the ability to listen critically and speak rationally in person-to-person meetings.
  8. Ability to Understand Business Strategy The characteristics of great software testers I've listed so far are primarily tactical in nature. The final skill on my list is the ability to see the larger picture of a company's overall business strategy. This enables a great software tester to actively participate at a level higher than just an individual contributor—instead of merely finding and reporting a Priority 2 / Severity 2 bug, a great software tester can identify strategic strengths and weaknesses of a software system that can ultimately lead to a business competitive advantage.
As software systems increase in complexity, the value of great software testers is becoming increasingly important. If you think you have what it takes to be a great tester, I encourage you to check out testing careers at Microsoft.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes. He can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.