 MSDN Magazine May 2008

MSDN Magazine May 2008

Office Apps: Integrate VSTO with SharePoint Content Types

See how to build a document-level Visual Studio Tools for Office customization and integrate it with a content type in SharePoint.

Steve Fox MOSS 2007: Automate Web App Deployment with the SharePoint API

Learn how to automate custom SharePoint application deployments, use the SharePoint API, and avoid the hassle of custom site definitions.

E. Wilansky, P. Olszewski, and R. Sneddon C++ Plus: Beef Up Windows Apps with the Visual C++ 2008 Feature Pack

Kenny Kerr sings the praises of the new Visual C++ 2008 Feature Pack, which brings modern conveniences to Visual C++.

Kenny Kerr Security: Safer Authentication with a One-Time Password Solution

One-time passwords offer solutions to dictionary attacks, phishing, interception, and lots of other security breaches. Here's how it all works.

Dan Griffin Alphabet Soup: A Survey of .NET Languages And Paradigms

Here we present a rundown of the various language paradigms of CLR-based languages via short language introductions and code samples.

Joel Pobar Editor's Note: I Want to Believe

Editor-in-Chief Howard Dierking explains why it's wise to consider Microsoft Office a development platform, even though developers typically don't.

Howard Dierking Toolbox: Database Audit Logs, Joel on Software, Code Handouts, and More

This month the Toolbox column takes a look at database logging, Joel Spolsky's blog, printing code projects, and ASP.NET reading.

Scott Mitchell CLR Inside Out: Measure Early and Often for Performance, Part 2

In the second of a two-part series, Vance Morrison delves into the meaning of performance measurements, explaining what the numbers mean to you.

Vance Morrison Data Points: Entity Framework Q&A

The Entity Framework is a hot topic, but how do you know when to use EntityClient, Object Services, Entity SQL, or LINQ? Find out here.

John Papa Cutting Edge: Single-page Interface and AJAX Patterns

This month we begin a look at the Single Page Interface (SPI) model and some design patterns for designing AJAX applications.

Dino Esposito Test Run: Request/Response Testing with Windows PowerShell

Did you know you can use Windows PowerShell to perform lightweight request/response testing for an ASP.NET Web app? Here's how.

Dr. James McCaffrey Office Space: From VBA Macro to Word Add-in

See how to use Word to capture a VBA macro and use Visual Studio Tools for Office to wrap it up into a deployable Word add-in.

Robert Bogue Wicked Code: Silverlight Page Turning Made Simple

As we'll show, with just a few lines of JavaScript you can build a general-purpose framework for incorporating page turns into Silverlight 1.0 apps.

Jeff Prosise Foundations: Loading Workflow Models in WF

When building workflows, you must consider the developer experience, available tools, long-term maintenance, and future versioning.

Matt Milner Security Briefs: Penetration Testing

In this installment of Security Briefs, James Whittaker explains the rules and the pitfalls of penetration testing so you'll know how to avoid them.

James A. Whittaker Team System: Team Foundation Server Event Service

Use the Team Foundation Server EventService to create and manage event subscriptions or create a Web service to receive and process events.

Brian A. Randell { End Bracket }: Rich and Reach Applications

Rich or reach, rich or thin, thick or fat? Thus went the battle over how much functionality a client should have. Here's how it all turned out.

Terry Crowley


Office Apps: Integrate VSTO with SharePoint Content Types

Office Apps

Integrate VSTO with SharePoint Content Types

Steve Fox
This article discusses:
  • VSTO customization basics
  • Building a custom document
  • Integrating a data source
  • Deploying to SharePoint
This article uses the following technologies:
VSTO, SharePoint, SQL Server
Contents
VSTO Customization Basics
Building the VSTO Customization
Creating User Controls
Deployment and Security
Creating a New Content Type
Finishing Up

Office Business Applications (OBAs) are composite applications built using the services, tools, and servers that comprise the Microsoft® 2007 Office system. These applications integrate with line of business (LOB) systems such as SAP, PeopleSoft, and Microsoft Dynamics. The primary goal behind OBAs is to build on existing investments in LOB systems by bridging the gap that exists when employees don't have direct access to business data residing in the LOB system.

As a result of this gap, many employees must work with outdated business data. This leads to potentially misinformed business decisions and causes unnecessary requests to workers who do have access to the LOB systems. There are costs in terms of time and, often, money with this situation.

OBAs have other advantages as well, such as bringing business data into the context of the everyday work environment—typically via familiar Microsoft Office applications. Building on familiar tools helps mitigate training costs for LOB systems that are, to put it kindly, sometimes cumbersome and difficult to navigate.

You can build effective OBAs with four core pieces: the LOB system, a customized Office client application, Microsoft Office SharePoint® Server (MOSS) 2007, and the services that tie the other three pieces together. Other products such as Exchange Server or Office Communications Server (OCS) can also be tied into OBAs to extend their reach and functionality, and Visual Studio® Tools for Office (VSTO) provides built-in features to help you build the solution quickly.

You could, for instance, create an Excel® Services Web Part that points to data within an Excel 2007 spreadsheet, enabling developers to build business intelligence views within SharePoint. Or you could provide rich client-side customizations by integrating Visual Studio Tools for Office (VSTO) customizations into SharePoint so users can invoke the custom templates or documents directly from their document libraries.

This particular type of integration has a significant number of benefits, such as building on the Office client application feature-set (thus not needing to reproduce it in your solutions), enabling efficient management and deployment of document or template assemblies (the VSTO customizations), and allowing for data to be stored and managed centrally while simultaneously allowing it to be manipulated in a collaborative, workflow-driven environment.

This article focuses on the second and third benefits mentioned: building a document-level VSTO customization and integrating it with a content type in SharePoint. The combination of these allows you to leverage the rich user functionality of the Office platform on the client while at the same time seamlessly extends the data captured on the client to the enterprise.

Specifically, I will show you how to create, deploy, and secure a VSTO document-level solution that uses Microsoft Word content controls bound to centrally managed data through a custom actions pane. I will then show you how to create a content type within SharePoint, and I will conclude by showing you how to map the content type to the custom VSTO document-level solution.

VSTO Customization Basics

Note that there are two types of VSTO add-ins you can build. The first, which will be discussed in this article, is a document-level solution—essentially a customization that appears only within a specific document or template. The second, not discussed in this article, is an application-level add-in—essentially a customization that appears every time you open the host application.

Because OBAs are often associated with processes, I'm going to create a fairly simple VSTO customization for Word that is a smaller part of a wider purchase order (PO) process. The process, for the sake of this article, includes three key steps: a PO is created for work requested by a customer, a statement of work (SOW) is generated from the purchase order, and an invoice is generated from the completion of the work. Of course, your PO process may vary, but these are the essential ingredients for the example in this article.

In this scenario, also imagine that the company uses an LOB system to centrally manage data such as work or customer details. For the sake of this article (and to keep things simple), I'll use a SQL Server® connection; however, in a real-world implementation, you'd likely be connecting to the LOB system with a Windows® Communication Foundation (WCF) service or Web service.

This article will focus on the invoicing portion of the process. To do this, I'll create a custom invoice that consumes centrally managed data and then integrates that with SharePoint. Along the way, think about how you could build and integrate the other steps in the process and potentially build a workflow around the custom invoice that will live on SharePoint.

Before you begin coding, however, you will want to use a custom document or template for the invoice. As you can see in Figure 1, I have created a very simple customer invoice with the basic information and not too many frills.

Figure 1 Customer Invoice Document
Figure 1 Customer Invoice Document(Click the image for a smaller view)
Figure 1 Customer Invoice Document
Figure 1 Customer Invoice Document (Click the image for a larger view)

Building the VSTO Customization
Further Reading

The resources listed here represent a good starting point for not only learning about OBA development but also about the individual technologies and options you have for your development.

  • Microsoft Office Business Applications for Office SharePoint Server
  • Programming Microsoft Office Business Applications
  • VSTO Developer Center
  • SharePoint Developer Center
  • Office Developer Center
  • OBA Central

After you've created your document, you can begin the fun part: building your document-level customization. To do this, create a new Word 2007 Document project using Visual Studio 2008 (click File | New | Project. Then, under Visual C#® or Visual Basic®, click Office | 2007 | Word 2007 Document).

Before Visual Studio creates your project shell, it will prompt you to select a document for your application. This is where you link a document (my invoice, in this case) with the project. You can specify an existing file, like I did, or you can create a new file. Visual Studio will then load all of the Word menu options and functionality into the development environment.

VSTO 3.0 is a technology included in Visual Studio Professional Edition and above and is the technology that enables you to build solutions for Office using managed code. One of the things I really like about VSTO document-level solutions is that the document or template you use in the project acts much like a designer in Visual Studio. You can drag controls from the Toolbox and drop them onto your document and even add events to your document just like you would with Windows Forms or Windows Presentation Foundation (WPF) user controls.

Visual Studio creates a project shell for you that uses the custom document as a core part of the project. With the project shell created, I add two main customizations to the invoice document and then add some code behind the document to handle the customer data. Specifically, I add some Word content controls to the document. Then I create a custom actions pane to manage a view of the data loaded into the content controls.

Figure 2 shows a project view with the document open in design mode. In the Toolbox pane, you can see the Word Controls—these are the Word content controls that you can drop onto the document surface. I've added them throughout the document. In Figure 2, the text "Click here to enter text." identifies the content controls I have added to the document. For this project, I've used the Rich Text Content Control, which gives me a little more power over text presentation and formatting. You can also add other types of controls such as images, date pickers, or comboboxes.

Figure 2 Editing a VSTO Custom Document
Figure 2 Editing a VSTO Custom Document(Click the image for a smaller view)
Figure 2 Editing a VSTO Custom Document
Figure 2 Editing a VSTO Custom Document (Click the image for a larger view)

After adding the content controls to the document surface, you'll next want to integrate a data source into your project. In the context of an OBA, this will likely mean integrating a service into your project and then implementing members within that service to read data from or write data to an LOB system. I'm going to keep things simple and use a SQL Server database as an example LOB system. My data source is a table called PODetails—it is defined by columns that include the PO ID (my primary key), customer name, address and contact person, PO details, and a total for the PO.

Enabling the PODetails data for use in my solution is accomplished by using the Add New Data Source Wizard (Data | Add New Data Source | New Connection). The wizard prompts for the tables to integrate into the project and then provides a default connection string for the project.

Creating User Controls

Now I create a user control, which will be hosted inside the actions pane (see Figure 3). The user control includes six fields with labels and four button controls. The fields are each bound to a record in the database table using the data binding property. The purpose of the buttons is to enable you to load the data from SQL Server into the control, to browse back and forth across the data, and to add the selected data to the content controls in the custom invoice document.

Figure 3 Customer PO Information User Control
Figure 3 Customer PO Information User Control(Click the image for a smaller view)
Figure 3 Customer PO Information User Control
Figure 3 Customer PO Information User Control (Click the image for a larger view)

After creating the user control, I add events for each of the button controls. For the load data event, I added the following code:

namespace CustomerInvoice {
  public partial class CustomerDataFilterControl : UserControl {
    private void btnLoadPOData_Click(object sender, EventArgs e) {
      CustomerInvoice.PODataDataSetTableAdapters.
        PurchaseOrderDetailsTableAdapter poAdapter = new
        CustomerInvoice.PODataDataSetTableAdapters.
        PurchaseOrderDetailsTableAdapter();
      poAdapter.Fill(this.pODataDataSet.PurchaseOrderDetails);
    }
  }
}

An alternative design might be loading the data during the core Add-in class's StartUp method, which executes when the host application loads the assembly.

I also added a couple of events to manage browsing back and forth through the data, so a user could move across the customer data in the table by clicking a button element. To do this, I used the MovePrevious and MoveNext methods on the binding source:

namespace CustomerInvoice {
  public partial class CustomerDataFilterControl : UserControl {
    private void btnDataPrevious_Click(object sender, EventArgs e) {
      purchaseOrderDetailsBindingSource.MovePrevious();
    }
    private void btnDataNext_Click(object sender, EventArgs e) {
      purchaseOrderDetailsBindingSource.MoveNext();
    }
  }
}

To add the data to the content controls, I employ the straightforward approach of reading a series of string variables from the Text property on each of my user control's textboxes. I then take those strings as arguments and set each of the Text properties on the corresponding content controls in the document. Figure 4 illustrates this sample technique.

Figure 4 Adding Data to the Word Document (Close Figure 4)
namespace CustomerInvoice {
  public partial class CustomerDataFilterControl : UserControl {
    private void btnAddData_Click(object sender, EventArgs e) {
      string companyName = txtCompanyName.Text;
      string purchaseOrder = txtPONum.Text;
      string workDetails = txtPODetails.Text;
      string companyAddress = txtAddress.Text;
      string contactPerson = txtContact.Text;
      string poTotal = txtTotal.Text;
      addDataToWCCs(companyName, purchaseOrder, workDetails,
        companyAddress, contactPerson, poTotal);
    }
  public void addDataToWCCs(string cmpyName, string poNum,
      string poDetails, string address, string contact, string total) {
      Globals.ThisDocument.wccCustomerName.Text = cmpyName;
      Globals.ThisDocument.wccCustomerAddress.Text = address;
      Globals.ThisDocument.wccCustomerContact.Text = contact;
      Globals.ThisDocument.wccWorkDescription.Text = poDetails;
      Globals.ThisDocument.wccWorkItem.Text = poDetails;
      Globals.ThisDocument.wccPurchaseOrderNum.Text = poNum;
      Globals.ThisDocument.wccPurchaseOrderTotal.Text = total;
      Globals.ThisDocument.wccTotalCharges.Text = total;
    }
  }
}

You may notice that there are a couple of duplicate string variable assignments to the content controls. This is by design, but I should mention that in your solution you may have multiple PO work items for a given invoice, so the second total (assigned to wccTotalCharges in my case) might want to treat the string variables as decimals and call a function to sum them for a total invoice charge. In the case of the second work details content control (wccWorkItem), one approach could be to have an abbreviated description for the table and a longer one for the details. I chose to keep things simple and reused the strings across the two content controls.

I designed my solution with content controls for three reasons. First, content controls are a great way to bind data to a Word document (and are quite stable). Second, I can lock the contents of the controls so users cannot delete the controls or edit the content in the controls. And finally, I can treat the controls in a way that is similar to other objects in a VSTO project. For example, in Figure 4, note the use of Globals to access the Text property of each content control. This is just one of a few properties that can be managed. Figure 5 demonstrates programmatically locking the content controls so users cannot delete or edit the controls.

Figure 5 Locking the Content Controls (Close Figure 5)
public void lockAllContentControls() {
  Globals.ThisDocument.wccCustomerName.LockContentControl = true;
  Globals.ThisDocument.wccCustomerAddress.LockContentControl = true;
  Globals.ThisDocument.wccCustomerContact.LockContentControl = true;
  Globals.ThisDocument.wccWorkDescription.LockContentControl = true;
  Globals.ThisDocument.wccWorkItem.LockContentControl = true;
  Globals.ThisDocument.wccPurchaseOrderNum.LockContentControl = true;
  Globals.ThisDocument.wccPurchaseOrderTotal.LockContentControl = true;
  Globals.ThisDocument.wccTotalCharges.LockContentControl = true;
}

At this point I build and run the solution. The result should look something like Figure 6. I'll also mention that my design does not come close to taking advantage of all the VSTO features available to you. For example, you could also extend the ribbon using the ribbon designer or use WPF controls to enhance the visualization of data in the actions pane. (For an example of using WPF with a VSTO action pane, see msdn2.microsoft.com/magazine/cc163292.aspx.)

Figure 6 VSTO Customization in Debug Mode
Figure 6 VSTO Customization in Debug Mode(Click the image for a smaller view)
Figure 6 VSTO Customization in Debug Mode
Figure 6 VSTO Customization in Debug Mode (Click the image for a larger view)

I'm going to leave the design as is and move on to the more important task of getting this assembly into a shared environment. Ultimately, since the PO scenario involves multiple people, I want to make sure this assembly lives in a shared and secure environment. But first, I'm going to deploy and secure the customization so employees can consume it from within SharePoint.

Deployment and Security

Before you do anything further, right-click your solution and select Clean Solution. This will remove any builds from your development environment. You'll also want to make sure you have created a document library for your VSTO customization—you'll use this later on in the article. To create a document library, simply navigate to your SharePoint site and click Document Center | Hierarchy | Create | Document Library. You'll need to provide some details describing the document library before clicking OK. After you've created your document library, you'll need the URL when deploying and securing the VSTO customization.

Now you need to set up a share. My development environment consists of Windows Server® 2003, MOSS 2007, Visual Studio 2008 with VSTO 3.0, and the 2007 Office system Professional Edition, so I set up the share on my local server (which was called MOSS). You can also select a secure share in your enterprise as the publishing point for this assembly. To do this, you'll need to create a directory for your assembly (for example, C:\CustomerInvoice). Right-click the newly created folder and choose Sharing and Security. Click the Share this folder radio button and give the share a name (for example, CustomerInvoice). Click the Sharing tab and then click the Permissions button. Add Everyone and give them Read permissions, and add Administrators with Full Control permissions. Click the Security tab, and then add Everyone with Read and Execute, List Folder Contents, and Read permissions.

At this point, the share is ready to receive the VSTO customization. (Note that, depending on your security policies, how you delegate permissions against this share may differ from what I've done here.)

The next step is to publish the VSTO assembly files to that particular location. With the VSTO project open in Visual Studio 2008, right-click the project and select Properties. In the Project Properties view, on the Build tab, make sure the output path is set to the share you just created for your project. In my case, this was \\moss\CustomerInvoice. Now click the Publish tab and set the Publishing Folder Location and Installation URL to the share (see Figure 7). Again, for me this is \\moss\CustomerInvoice. Click the Publish Now button. After the VSTO project has successfully published to your share, you can close the project.

Figure 7 Publishing the VSTO Customization
Figure 7 Publishing the VSTO Customization(Click the image for a smaller view)
Figure 7 Publishing the VSTO Customization
Figure 7 Publishing the VSTO Customization (Click the image for a larger view)

At this point, the project has been published to an installation location, but it is not yet a trusted assembly in Word. However, in order to make it a trusted assembly, you need to add the location of the document (stored at your publishing folder location) to the Word Trust Center.

Open Word and click the Office button (the large round button in the top-left-hand corner of the document). Click the Word Options button at the bottom of the menu and select Trust Center from the pane on the left. Then click the Trust Center Settings button and choose Trusted Locations from the pane on the left (see Figure 8). Check the Allow Trusted Locations on my network checkbox, and click the Add New Location button. In the Add New Location Path dialog, add the publish location for your project assemblies (for example, \\moss\CustomerInvoice) and check the Subfolders of this location are also trusted checkbox.

Figure 8 Adding the Customization to Word Trust Center
Figure 8 Adding the Customization to Word Trust Center(Click the image for a smaller view)
Figure 8 Adding the Customization to Word Trust Center
Figure 8 Adding the Customization to Word Trust Center (Click the image for a larger view)

You also need to add the SharePoint document library location where your users will be using this VSTO customization (for example, moss.litware.com/Docs/CustomerInvoice). After you've added your publish and SharePoint locations, click OK twice and then exit Word.

You'll also need to configure the trusted sites for your browser. First, open Internet Explorer® and click Tools | Internet Options, then click the Security tab. Select Trusted Sites and click the Sites button. You'll want to add the sites on which your VSTO assembly is running (for example, moss.litware.com in this scenario) and be sure to uncheck Require server verification.

You can now test the Word document by navigating to the publish share (for example, \\moss\CustomerInvoice) and opening your document (in my case, CustomerInvoice.docx). Whatever customizations you've added within your VSTO project should work fine, but at this point you don't want to save anything, so close the document without saving it. The last thing to do is to create a new SharePoint content type and link it to the customized document.

Creating a New Content Type

A content type is a type definition in Windows SharePoint Services (WSS) that lets you define the behavior of an item in a list or a document within a document library. You can create a custom content type that is mapped to a document library that uses the custom template created earlier in this article.

There are essentially three simple steps to mapping a content type to a custom VSTO assembly. The first is to actually create the content type within SharePoint. The second is to associate that content type with the custom VSTO assembly. And the third is to add the new content type to the SharePoint document library.

In order to create a new content type, you have to navigate to the Document Center and click Site Settings | Site Content Type Gallery | New Site Content Type. Doing this invokes the New Site Content Type page (see Figure 9) on which you can set the core properties of the content type. For my content type, I've called it Customer Invoice, provided a description, and then selected Document as the parent content type. I've also selected the Document Content Types for categorizing my content type. After you've done this, click OK. This creates your new content type.

Figure 9 Creating a New Content Type
Figure 9 Creating a New Content Type(Click the image for a smaller view)
Figure 9 Creating a New Content Type
Figure 9 Creating a New Content Type (Click the image for a larger view)

Now that you've created the content type, you need to associate your custom VSTO assembly (the customized Word document) with the new content type. To do this, in your SharePoint site Document Center click Site Settings | Modify All Site Settings | Site Content Types, then click your newly created content type (in this case, it is Customer Invoice). Now click Advanced Settings. On the Advanced Settings page (see Figure 10), click the Upload a new document template radio button, browse to the publish location where you've stored the custom document, and then click OK.

Figure 10 Customer Invoice Document
Figure 10 Customer Invoice Document(Click the image for a smaller view)
Figure 10 Customer Invoice Document
Figure 10 Customer Invoice Document (Click the image for a larger view)

The last step is to add the custom content type (with the VSTO custom document associated with it) to your document library so that when users click the New button in your document library, the document that is invoked is the custom document. To do this, navigate to your document library (in my case, the Customer Invoice library) and click Settings. This invokes the Settings page for your document library. Under Content Types, click Add from existing site content types. On the Add Content Types page, select your custom content type (in my case, Customer Invoice) from the Available Site Content Types list (see Figure 11) and click Add.

Figure 11 Add Content Types Page
Figure 11 Add Content Types Page(Click the image for a smaller view)
Figure 11 Add Content Types Page
Figure 11 Add Content Types Page (Click the image for a larger view)

Finishing Up

While not mandatory, you might also want to complete one last step, and that step is simply setting the custom document content type button as the top-most option when the user clicks New in your document library. The benefit here is that your customization displays as the top option. Obviously, being at the top of the list makes it much easier for users to find your custom document.

To do this, navigate to your document library and click Settings. Under Content Types, click Change new button order and default content type, and then change your custom content type button to 1 and click OK. Again, not a deal breaker, but it does help place the custom document at the top of the available choices when selecting New in your document library.

You can now navigate to your document library. Click New, and your new custom document (in my case, my Customer Invoice) appears ready for use. This document will appear exactly as it did in Figure 6, the major difference being that it is loaded and managed from a central location: the SharePoint site.

You now have a readily accessible document via SharePoint that integrates a centrally managed data store. You also have a document to which you can attach custom workflows to help manage processes around the wider PO process.

As you begin to design other features into your OBA solutions, you can take advantage of not only additional client-side features within VSTO—such as custom ribbons, WPF, and LINQ—but also the server-side features within SharePoint such as Excel Services, the Business Data Catalog, blogs. And there are many other features that ship with MOSS 2007.

If you're interested in finding out more about the topics discussed in this article, you can check out the helpful resources listed in the "Further Reading" sidebar. These resources include a few books and pertinent Microsoft resource centers.

Steve Fox is a Program Manager with the Developer and Platform Evangelist team at Microsoft. He divides his time between working externally with VSTO customers by providing VSTO and OBA training and delivering VSTO and, more generally, Office Platform Development sessions at conferences and working internally with the development team to help evolve the VSTO product.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


MOSS 2007: Automate Web App Deployment with the SharePoint API

MOSS 2007

Automate Web App Deployment with the SharePoint API

Ethan Wilansky and Paul Olszewski and Rick Sneddon
Code download available at: AutomatingMOSSDeployments2008_05.exe (164KB)
This article discusses:
  • SharePoint deployment basics
  • Deploying custom Web Parts
  • Customizing site branding
  • Deploying lists, forms, and reports
This article uses the following technologies:
SharePoint
Contents
SharePoint Deployment Basics
Web App Provisioning and Settings
Managing Web.config
Custom Web Parts
Simplified Site Branding
Setting Up Lists
Form and Report Deployment
What Next?

At the 2008 Office System Developer's Conference, one presenter metaphorically described a .﻿NET developer's first look at SharePoint® as equivalent to an experienced mountain climber staring at a smooth wall 100 feet tall and trying to figure out exactly how to scale it. Many Microsoft products offer a dizzying array of approaches to completing a task, and deploying custom SharePoint applications is a good example of such a case. Because SharePoint is a complex and sophisticated application platform, deployment can present some puzzles to the uninitiated.

In this article, we will show you how to automate custom SharePoint application deployments by taking advantage of the SharePoint API. Following this approach allows you to avoid having to create and maintain a custom site definition. You benefit from a more modular set of components to deploy and manage in your source control system, which is an essential part of a continuous integration effort. Finally, you are following a model that Microsoft will continue to improve in the coming years.

We will explain how you can leverage the SharePoint object model to automate deployment from Web application creation to Feature activation. The Visual Studio® code project included with this article provides a number of code samples that demonstrate how to automate some essential tasks in a custom deployment.

Along the way, we will explore tools and techniques provided by Microsoft and members of the SharePoint developer community that we have found to be especially valuable in your quest toward a fully automated deployment.

SharePoint Deployment Basics
Solutions and Features

The solution packaging framework bundles all the components used to extend WSS into a new file, called a solution file. The solution file is a .cab file format with a .wsp extension and contains a manifest for deploying the various solution parts within the SharePoint framework.

Solutions are added to a SharePoint Web farm and deployed to one or more Web applications. This task can be accomplished with the Stsadm utility or with the SharePoint object model.

Features are the fundamental components used to extend SharePoint solutions. They provide a way of modifying functionality within SharePoint, from adding Web Parts to a gallery to enabling Reporting Services integration.

Features are activated for a single scope--farm, Web application, site collection, or site. The Feature scope is set by the Scope attribute on the Feature element. Features must be installed before they can be activated, and they must be activated to a scope before they can be used. In addition, Features must be deactivated before they are uninstalled unless they are scoped to the Web application or farm.

An important consideration to bear in mind is how you might scope the automation tasks we have included with this article inside of a Feature. The following figure describes where you might scope the various automation tasks.

From an automated deployment perspective, it is important to know how to programmatically control the state of a Feature. You can use Stsadm to activate and deactivate a Feature, or you can do this programmatically. The following code allows you to enumerate all the Features in a farm and see their display name and scope:

foreach (SPFeatureDefinition definition in
  SPFarm.Local.FeatureDefinitions) {
  Console.WriteLine("Title: {0}, Scope: {1}",
    definition.DisplayName,
    definition.Scope.ToString());
}
To activate a Feature programmatically, add the Feature to the Features collection at the level in which the Feature was scoped when it was created, as the following example demonstrates:
SPSite site = new SPSite(http://msdn.fabrikam.com:45001);
SPFeatureDefinition definition =
  SPFarm.Local.FeatureDefinitions["MyFeature"];

if (definition != null &&
  definition.Scope == SPFeatureScope.Site) {
  site.Features.Add(definition.Id, true);
}

Scoping for Automation Tasks

Automation Task | Switch Name | Create Solution? | Scope
Create Web app and root Web | CreateWebApp | No | N/A
Web.config management | AddVerifyElement, AddVerifyAttrib, RemoveTrackedItem (in a Feature's FeatureDeactivating event ) | Yes | Web application
Apply a theme to site collection | ApplyThemetoSiteCollection | Yes | Site or Web for individual themes
Set master page and css | SetMasterPageandCSS | Yes | Site or Web for individual themes
Exclude items from navigation | ExcludeSiteFromNavigation | Yes | Site or Web
Create a Web | CreatePublishingWeb | Yes | Site
Create a page | CreatePublishingPage | Yes | Web
Add global navigation nodes | AddGlobalNavigationNode | Yes | Web
Add a Web Part to a page | AddListViewWebPart | Yes | Web
Create and modify lists | CreateList, AddItems, CreateView | Yes | Web

Microsoft significantly improved the SharePoint deployment infrastructure with the release of Windows® SharePoint Services (WSS) 3.0 and Microsoft® Office SharePoint Server (MOSS) 2007. The primary components in this deployment infrastructure are the classes in the Microsoft.SharePoint.Administration and Microsoft.SharePoint namespaces and the SharePoint solution deployment framework.

Microsoft also improved the way site definitions and site templates work and how they can be customized. At first glance, it might seem to make sense to use all of these capabilities to support your deployment, but think again. If your goal is to deploy a highly customized portal application in an automated and centralized fashion, you should stay away from site templates and even reconsider a deployment strategy involving custom site definitions.

Most experienced SharePoint developers are likely to agree with our view on site template customization. While they are fine for simple customizations, they are inappropriate for significant customizations because page rendering performance can suffer. Template customizations are stored as differences from the originating site definition and persist as an .stp file in the SharePoint database rather than being served from the file system. When the page renders, the .stp is merged at run time with the underlying site definition in the file system.

What might be more surprising to experienced SharePoint developers is our recommendation to stay away from custom site definitions. Microsoft strongly encourages developers not to customize the out-of-the-box site definitions for a number of reasons. Future service packs may overwrite the existing site definitions and migrations to the next version of SharePoint might not be possible without losing the customizations.

Even if you abide by the recommendation by Microsoft to leave the out-of-the box site definitions alone, there are still more reasons to avoid custom site definitions. For highly customized deployments, site definitions can become huge and unwieldy to manage. A complete site definition can contain multiple Webs. Within each Web there will be one or more pages, and on these pages will be a variety of elements, such as Web Part zones, Web Parts, modules, list definitions, and navigation components. The onet.xml file maintains configuration information about each Web. This file can quickly become a maintenance nightmare as requirements and configurations change. In our experience, it contains between 300 and 600 lines of XML for moderate customizations. If you have five custom Webs in your site definition, suddenly you have between 1,500 and 3,000 lines of XML to maintain.

The SharePoint API and the solution deployment framework provide a better answer. Solutions are deployment packages for SharePoint—they can contain almost anything you need to deploy to SharePoint. An important part of a solution, though technically not required in all instances, is called a Feature. While the solution is the deployable package, the Feature defines aspects of the package that allow it to employ the SharePoint object model and other managed code and to advertise itself within the SharePoint hierarchy (the farm, Web application, site collection, or Web levels).

Microsoft provides a model for automating the initial installation of SharePoint. Note that the task of configuring a SharePoint instance or farm is typically completed by an administrator. The next step is to create a Web application and site for your custom portal application.

While these tasks can be completed by using SharePoint Central Administration, you have the ability to automate them via the SPWebApplicationBuilder and SPWebApplication classes in the Microsoft.SharePoint.Administration namespace. The SharePoint command-line administrative tool (Stsadm) does not support Web application creation by default. By automating these tasks, they will not only get completed faster, but you can ensure consistency across your single server or farm implementations and as your custom solution moves from development to production.

When SharePoint creates a Web application and generates the top-level site collection and root Web, there is a lot happening behind the scenes. When creating a typical Web application, SharePoint does the following:

  • Creates a unique entry in the SharePoint configuration database for the Web app and assigns a GUID to that entry
  • Creates and configures a Web application in IIS
  • Creates a root folder to store the Web application pages and associated resources
  • Creates and configures an IIS application pool
  • Configures authentication protocol and encryption settings
  • Assigns a Default alternate access mapping for the Web app
  • Creates the first content database for the Web application
  • Associates a search service with the Web application
  • Assigns a name to the Web application that appears in the Web application list in SharePoint Central Administration
  • Assigns general settings to the Web application, such as maximum file upload size and default time zone

Optionally, you can add alternate access URLs to the Web application. This is just one of many optional tasks you might want to complete after creating a Web application. And when creating a site collection, SharePoint also creates the top-level site based on a site definition and sets general properties for the site, such as the site title and site owner. Completing these tasks manually can take quite a while—they are tedious and error prone because of the amount of data entry involved. Automating this task speeds up task completion and ensures data consistency from one environment to the next.

As you might expect, there is a fair amount of code involved in automating this task. Rather than show it all here, you should download the code that accompanies this article. You'll find the CreateWebApp method in the SPDeployTests project. (Be sure to read the prerequisites document included in the code sample before attempting to run the console application.) We will emphasize key elements of the code here and call out some specific items that could cause trouble when you automate this part of your custom portal deployment.

One more thing to remember: be sure to run the code from a SharePoint server. It is not designed to run remotely. In our experience, it's best to run an automated SharePoint deployment routine on a SharePoint server in the farm.

Web App Provisioning and Settings
SharePoint Feature and Solution Resources

  • Windows SharePoint Services 3.0 SDK
  • Office Space: Solution Deployment with SharePoint 2007
  • Office Space: Features for SharePoint
  • Site Definitions Demystified—Creating a Custom Site Definition Having Custom Web Parts
  • Windows SharePoint Services 3.0 Tools: Visual Studio 2005 Extensions
  • Implementing Microsoft Office SharePoint Server 2007 and Windows SharePoint Services 3.0 Solutions
  • Heather Solomon's blog on branding

In order to create a new Web application, you start by calling the AdministrationService.Farm property from an SPWebService object. Calling the Farm property returns an SPFarm object. The SPFarm object is at the top of the configuration hierarchy and manages all of the servers, services, and solutions in a SharePoint farm. Once you have the SPFarm object, you can pass it to the SPWebApplicationBuilder constructor:

SPFarm farm =
  SPWebService.AdministrationService.Farm;

SPWebApplicationBuilder webAppBld =
  new SPWebApplicationBuilder(farm);

SPWebApplicationBuilder is the worker class for creating SharePoint Web applications. It can complete all of the steps previously outlined for creating a Web application. To make use of this class, set properties of the SPWebApplicationBuilder class, and then call its Create method, which returns an SPWebApplication object for the new Web application:

SPWebApplication webApp = webAppBld.Create();

Use the SPWebApplication class to set additional properties, and then call the Update method to send any changes back to the farm. Finally, call the SPWebApplication Provision method to create the IIS Web application and application pool.

Calling these methods in the right order and at the right time in your code is important. Therefore, be sure to review the code samples for this article to see a working example.

When setting properties on a Web application, there are a couple of points to consider. First, be clear on what a setting actually means. If you're not sure what the implications are of a setting, such as choosing Windows NT LAN Manager (NTLM) authentication versus Kerberos, work with a SharePoint infrastructure expert to make this decision. Secondly, some properties can be tricky to set or are unclear as to their affect. For instance, here is an example of setting the ID property in the SPWebApplicationBuilder object:

webAppBld.Id = new Guid("3798E478-4E16-4856-A152-F05EF8C2C777");

Like most of the properties in this object, SharePoint will either set the property to a default value or assign a value. In this case, SharePoint will generate a unique GUID and store that along with the Web application in the Objects table of the configuration database. You can run the following query against the SharePoint Configuration database to see all of the Web application objects and their associated ID GUIDs:

SELECT id, name FROM OBJECTS
WHERE Properties
LIKE '%Microsoft.SharePoint.Administration.SPWebApplication%'

You can assign this value or let SharePoint assign one for you.

If you are configuring a new application pool and assigning a user account, which is the preferred method of setting up SharePoint, there are a couple of important caveats. First, in a workgroup, ApplicationPoolUserName is the name of a local user, without the server name prepended. If SharePoint is part of an Active Directory® domain, then you should use an Active Directory user account and prepend the domain name for this property:

@"domainname\username"

For the ApplicationPoolPassword, you must pass in a SecureString data type for the value. Here is the pattern for building a secure password:

// build the password as a secure string
SecureString appPoolPwd = new SecureString();
appPoolPwd.AppendChar('k');
appPoolPwd.AppendChar('E');
// more of the same to build-up the password
appPoolPwd.MakeReadOnly();

Be sure that the user name and password you assign to these values match the values as you've set them in your identity repository before attempting to create the Web application.

The next property, ApplicationPoolId, is simply the name of the application pool as it appears in IIS. It's important to note that the WSS 3.0 SDK help file incorrectly states that ApplicationPoolId is a GUID that identifies the application pool. One could argue that it is a unique identifier within the list of application pools in IIS, but it is neither globally unique nor in the format we typically think of for a GUID.

The next stumbling block is encountered when associating a search service instance with the new Web application. Figure 1 shows how to accomplish this task and is followed by further explanation as to how you find the proper search service instance.

Figure 1 Associating a Search Service Instance (Close Figure 1)
// get the server hosting the search service
SPServer server = new SPServer("myServer");

// get the office SharePoint server search instances
SPSearchService srchService =
  SPFarm.Local.Services.GetValue<SPSearchService>("SPSearch");

// get the proper instance of the search service
SPSearchServiceInstance searchServiceInst =
  (SPSearchServiceInstance)srchService.Instances[new Guid(
    "C22E9545-71B4-471F-81A7-A213D5A7F8B2")];

// set the search service instance to crawl the web application
webAppBld.SearchServiceInstance = searchServiceInst;

There is a service hierarchy in SharePoint where a service contains service instances that actually do the work. Unfortunately, finding a search service instance can be difficult because the instances might have type names, but their Name and possibly DisplayName properties are empty. Therefore, passing the GUID into the Instances collection as in Figure 1 is one way to set this value.

There are two ways you can find the proper GUID of the search service instance. The first option is to query the objects table in the SharePoint Configuration database, as demonstrated in the following code:

SELECT id, name, properties FROM OBJECTS
  WHERE (Properties LIKE
    '<object type=' +
    '"Microsoft.SharePoint.Search.' +
      'Administration.SPSearchServiceInstance%')
  ORDER by ID

Another option is to run the EnumFarmServices method in the code SPDeployTests project contained in the code download. Figure 2 shows output from one of our servers with the proper search service instance shown in red.

Figure 2 Instances of SPSearch (Close Figure 2)
Name: SPSearch
ID: 7fd9c4b0-25c2-4867-b716-cd3b61e7a08e
DisplayName: SPSearch

Instance 1
Name:
ID:c9043683-a431-4cc7-b30a-fa52163f649b
Type name:Windows SharePoint Services Search
DisplayName:

Instance 2
Name:
ID:c22e9545-71b4-471f-81a7-a213d5a7f8b2
Type name:Windows SharePoint Services Search
DisplayName:Search index file on the search server

If you want to configure the search service instance from code and you don't select the proper search service instance, Web application creation will fail with a database foreign key dependency error. There are significantly more settings that you can configure for your Web application. The code download provides some examples to get you started.

Creating the first site collection is straightforward. However, before you create the top-level site collection in a Web farm environment you should reset IIS (using iisreset /noforce). Figure 3 demonstrates how to create a top-level site collection. For brevity, we moved some of the variables into the input parameters for the Add method. The interesting or potentially confusing variables are declared and described above the Add method.

Figure 3 Creating a Site Collection (Close Figure 3)
// for the top-level site
string url = "/";

// this is the MS language Locale ID.
// see http://www.microsoft.com/globaldev/reference/lcid-all.mspx
uint LCID = 1033;

// this is the site template for the
// Publishing Site - Collaboration Portal merged site definition
string template = "SPSPORTAL";

// create a site collection with an out of the box site definition
  SPSite MSDNSiteCollection = webApp.Sites.Add(url,
  "The MSDN site collection", "a description of this site",
  LCID, template, "mydomain\\myLoginName",
  "my owner name", "my email address");

// close the site collection
MSDNSiteCollection.Close();

Managing Web.config

A common task in most SharePoint code projects is updating web.config for each Web application. While updates can be made manually in small implementations, that doesn't scale to large, multi-developer projects. Keeping a copy of a SharePoint Web application's web.config in a source control system and having developers check out the file, update it for their project, and check it in is one obvious possibility. You could then script the web.config deployment to your Web front-end servers on a schedule or based on a particular event, such as check-in. However, this approach does not take advantage of the ability of SharePoint to manage web.config settings for you.

If you use the SharePoint API to update web.config, new front ends added to the SharePoint farm receive the updates to the web.config and SharePoint maintains these settings in existing front ends. There are two common approaches for getting SharePoint managed updates into web.config programmatically. For Web Part SafeControl entries and custom access control policies, rely on the SharePoint solution packaging framework. For unique entries, use the SPWebConfigModification class in the Microsoft.SharePoint.Administration namespace.

Figure 4 shows how to add an XML child element to your web.config file using SPWebConfigModificationType.EnsureChildNode. In this example, the SPWebConfigModification class ensures that the <add key = "key01" value = "Setting01" /> element appears inside of the appSettings element. The comments in the code explain how to achieve this result. Notice the single quotes for the values 'key01' and 'Setting01' in the xmlValue variable. Using single quotes makes the code clearer and the SPWebConfigModification class takes care of changing these to double quotes when the values are written to each front-end Web server.

Figure 4 Adding a Child Node in Web.config (Close Figure 4)
// get the webApp
SPWebApplication webApp =
  new SPSite("http://msdn.fabrikam.com").WebApplication;

// the element to add or verify contains
// the following name and attributes:
string elementName = "add[@key='key01'][@value='Setting01']";

// the location where the new element should be added or verified is
// present in the target web.config
string xPath = "configuration/appSettings";

// the value exactly as it should be written
string xmlValue = @"<add key = 'Key01' value = 'Setting01' />";

SPWebConfigModification modification =
  new SPWebConfigModification(elementName, xPath);

// this name determines who can modify this entry -
// only the owning name (typically a fully qualified program name,
// GUID or possibly the WebConfigModificationFeatureReceiver.OwnerID
// name can modify the entry
modification.Owner = "SPWebConfigTestAddChild";

// for identical entries, this controls what order the entry is made
// typically fine to leave this at 0
modification.Sequence = 0;

// EnsureChildNode means to add or verify that the child node is present
// as specified by the Value property next
modification.Type =
  SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;

// the actual value to write
modification.Value = xmlValue;

// add the modification to the SPWebConfigModification collection
webApp.WebConfigModifications.Add(modification);

// check all web applications on the farm front ends to ensure
// that the web.config modification(s) have been applied
webApp.Farm.Services.GetValue<SPWebService>().ApplyWebConfigModifications();

// propagate changes in web application across the farm
webApp.Update();

You can also use the SPWebConfigModification class to add, change, or ensure that an attribute value within an XML node appears in web.config. This is simple when you're writing or verifying an XML attribute that is unique in the web.config file. For example, you might want to change the autoDetect attribute from true to false in the following unique XML fragment:

<system.net>
  <defaultProxy>
    <proxy autoDetect="true" />
  </defaultProxy>
</system.net>

In this case, the SPWebConfigModification constructor looks like the following:

SPWebConfigModification modification =
  new SPWebConfigModification(
  "autoDetect", "system.net/defaultProxy");

The SPWebConfigModificationType enumeration is then set to EnsureAttribute, like this:

modification.Type =
  SPWebConfigModification.SPWebConfigModificationType.EnsureAttribute;

EnsureChildNode and EnsureAttribute are the two values in this enumeration that you should use for your updates. There are a number of blog entries suggesting that EnsureSection (the third enumeration) works for writing whole sections into web.config. Unfortunately, you cannot remove the entries easily.

Other than the differences mentioned, there is little difference to the pattern you use for writing an attribute versus writing one or more XML nodes to web.config. What's a little trickier is writing or verifying an attribute when there are identically named elements inside of a parent element (for example, multiple add child elements in the appSettings element). In this case, the xPath string gets you to the correct element.

Consider the following appSettings element:

<appSettings>
  <add key="FeedCacheTime" value="300" />
  <add key="FeedPageUrl"
    value="/_layouts/feed.aspx?" />
  <add key="FeedXsl1"
    value="/Style Library/Xsl Style Sheets/Rss.xsl" />
  <add key="key01" value="Setting01" />
</appSettings>

In order to change the value attribute of the fourth add element, use an xPath statement that specifies the child element, like so:

configuration/appSettings/add[4]

xPath syntax is the key to pinpointing exactly what you need to update. One reference for xPath syntax is available at msdn2.microsoft.com/ms256115 .

Figure 5 shows how to modify or verify that the value attribute is set to Setting02 in the fourth "add" child element of the appSettings element. For brevity, we removed comments that appear in Figure 4. Now that these settings have been added, you can inspect the web.config file on any Web front end running the target Web application.

Figure 5 Modifying the Fourth Attribute of a Child Node (Close Figure 5)
SPWebApplication webApp =
  new SPSite("http://msdn.fabrikam.com").WebApplication;

// the attribute's name to which a value will be added,
// changed or verified:
string attributeName = "value";

// the location of the element that should be modified or verified
string xPath = "configuration/appSettings/add[4]";

// the value of the attribute as it should be written
string attributeValue = "Setting02";

SPWebConfigModification modification =
  new SPWebConfigModification(attributeName, xPath);

modification.Owner = "SPWebConfigTestModifyAttribute";

modification.Sequence = 0;

// EnsureAttribute means to add or verify that an attribute value is
// present as specified by the Value property next
modification.Type =
  SPWebConfigModification.SPWebConfigModificationType.EnsureAttribute;

modification.Value = attributeValue;

webApp.WebConfigModifications.Add(modification);

webApp.Farm.Services.GetValue<SPWebService>().ApplyWebConfigModifications();

webApp.Update();

Figure 6 shows how to see what settings are being managed by SharePoint by enumerating the collection of settings contained in the WebConfigModfications property of the Web app. Manually removing or changing the entries in web.config that are managed by SharePoint won't do you any good in the long term. SharePoint tracks these entries and ensures that they are in web.config each time the WSS service restarts.

Figure 6 Enumerating a List of Modifications (Close Figure 6)
SPWebApplication webApp =
 new SPSite("http://msdn.fabrikam.com").WebApplication;

// use the System.Collection collection class to get
// a collection of modifications in the configuration database
Collection<SPWebConfigModification> collection =
  webApp.WebConfigModifications;

// iterate the collection of modifications and return properties
// about the items
foreach (SPWebConfigModification item in collection)
{
 Console.WriteLine("\nitem {0}", collection.IndexOf(item));
 Console.WriteLine("owner is: {0}", item.Owner);
 Console.WriteLine("path is: {0}", item.Path);
 Console.WriteLine("name is: {0}", item.Name);
 Console.WriteLine("value is: {0}", item.Value);
 Console.WriteLine("sequence is: {0}", item.Sequence);
}

To remove entries, you need to know the value of the Owner property for the modification. Then you can use the Remove method of the SPWebApplication's WebConfigModificatons property to delete entries maintained by SharePoint. See Vincent Rothwell's blog post at blog.thekid.me.uk/archive/2007/03/20/removing-web-config-entries-from-sharepoint-using-spwebconfigmodification.aspx for more information about removing entries. There is also a sample in the code download for this column showing how to remove an entry.

Modifying web.config can be tricky using the SPWebConfigModification class. There are a number of caveats that you should be aware of. You can read about the most common ones at Reza Alirezaei's blog (blogs.devhorizon.com/reza/?p=459) and Mark Wagner's blog (www.crsw.com/mark/default.aspx). We highly recommend you take a close look at these articles before getting started using this class. After you've done your research, be sure to carefully test your modifications and your ability to remove these modifications using the SPWebConfigModification class in an isolated environment, in a shared development environment, and in a Web farm development environment.

Even with the ability of SharePoint to manage web.config settings, there are instances when you might want to manage settings without the SharePoint API, for example, managing a unique section of sensitive credential information. In this case, consider using the WebConfigurationManager in the System.Web.Configuration namespace. To encrypt the information, you can use either the aspnet_regiis utility or use the AppSettingsSection class in the System.Configuration namespace. You can read about how to edit and encrypt Web.Config sections using C# 2.0 at c-sharpcorner.com/UploadFile/neo_matrix/EditWebConfig05042007091116AM/EditWebConfig.aspx.

Custom Web Parts

Microsoft built the solution packaging and deployment framework with Web Parts in mind. The key to end-to-end Web Part deployment starts with the Web Part deployment file, officially known as the Web Part Control Description file. There are two Web Part deployment formats: .WebPart and .dwp. Both are XML files with underlying schemas. Web Parts Control Description Files at msdn2.microsoft.com/library/ms227561(VS.80).aspx describes the .WebPart schema.

The .dwp format is available for backward compatibility. It was the SharePoint Web Part control description file format in WSS 2.0 and SharePoint Portal Server 2003. While you can use this format for your Web Part deployment in WSS 3.0 and MOSS 2007, you should avoid .dwp if your goal is to fully automate your Web Part deployments. This is because the .dwp schema does not include some of the essential settings that are required for full automation. For example, if you need to hide the chrome (otherwise known as the styling) of a custom Web Part, you can't do that directly with a .dwp file.

How you begin creating your custom Web Part affects the type of deployment file that is generated and is expected for deploying your component. Surprisingly, there is little guidance explaining the scenarios that automatically generate a .dwp file or a .WebPart file. If you deploy a Web Part to SharePoint using the older .dwp format, the only way we have found to switch to a .WebPart file is to uninstall the solution or, if you're not using a solution, manually remove the Web Part from SharePoint, rebuild the project, and recreate the solution with a .WebPart file.

There are two approaches you can take to ensure that your custom Web Part is deployable using the newer ASP.NET .WebPart format. The first approach is to start a Visual Studio class project. Note that when you declare your Web Part class, instead of inheriting from the SharePoint Web Part class (Microsoft.SharePoint.WebPartPages.WebPart), inherit from the ASP.NET Web Part class, like so:

public class WebPart1 : System.Web.UI.WebControls.WebParts.WebPart

Microsoft recommends inheriting from the ASP.NET Web Part class in most of its documentation chiefly because a Web Part that inherits from this class is potentially deployable both to SharePoint and to a custom ASP.NET portal. We say potentially because a Web Part that interacts with the SharePoint object model might be dependent on running in the context of a SharePoint portal. If you end up requiring one of the handful of features available in the SharePoint Web Part class, you can later switch to inheriting from the SharePoint namespace. However, you should create your SharePoint solution and your .WebPart file before making this switch.

The second approach to ensuring that a .WebPart deployment file is automatically created is by using the Visual Studio Extensions for WSS 3.0 or the new WSPBuilder Extension. When using these tools and the included SharePoint Web Part project template, it doesn't matter which namespace you use when inheriting from the Web Part class.

Once you have a .WebPart file, the next step in full deployment automation has more to do with the solution, which we explore only briefly in this article. For now, the key is to get the .WebPart file prepared for the deployment.

While you can manually add elements to your .WebPart file, the following approach will make this effort significantly easier. In the constructor for your Web Part, add the following code:

// make the web part deployment file exportable
this.ExportMode = WebPartExportMode.All;

When you build and later install and configure your Web Part, this allows you to export the .WebPart file. You can also manually set this in the tool part properties instead of declaring this in code.

Next, install your Web Part into SharePoint or a custom ASP.NET Web application and drop it onto a Web Part zone on the page. (Later in this article, we will describe ways to automate the initial Web Part deployment.) From the Web Part's Tool Part Properties dialog, configure the Web Part exactly as you would like it to appear and function on the page. From the edit menu of the Web Part, click Export. Save the exported .WebPart file into the Visual Studio project created for the Web Part.

Figure 7 shows a basic .WebPart file as prepared by version 1.1 of the Visual Studio Extensions for WSS following the configuration steps outlined previously. The settings for custom toolpart properties and custom toolparts with properties will also appear in the .WebPart export after configuring their settings.

Figure 7 Web Part XML File (Close Figure 7)

Prior to Configuration
<?xml version="1.0" encoding="utf-8"?>
<webParts>
  <webPart xmlns="http://schemas.microsoft.com/WebPart/v3">
    <metaData>
      <!--
        The following Guid is used as a reference to the web part class,
        and it will be automatically replaced with actual type name
        at deployment time.
      -->
      <type name="5ad2f5ec-8c67-4ca9-b2fb-83c4dadd50c0" />
      <importErrorMessage>
        Cannot import WebPart1 Web Part.</importErrorMessage>
    </metaData>
    <data>
      <properties>
        <property name="Title" type="string">WebPart1 Web Part</property>
        <property name="Description" type="string">
          WebPart1 Description</property>
      </properties>
    </data>
  </webPart>
</webParts>
After Configuration
<webParts>
  <webPart xmlns="http://schemas.microsoft.com/WebPart/v3">
    <metaData>
      <type name="WPUsingVSeWSS1dot1.WebPart1, WPUsingVSeWSS1dot1,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=9f4da00116c38ec5" />
      <importErrorMessage>
        Cannot import WebPart1 Web Part.</importErrorMessage>
    </metaData>
    <data>
      <properties>
        <property name="AllowClose" type="bool">False</property>
        <property name="Width" type="unit">170px</property>
        <property name="AllowMinimize" type="bool">False</property>
        <property name="AllowConnect" type="bool">True</property>
        <property name="ChromeType" type="chrometype">None</property>
        <property name="TitleIconImageUrl" type="string" />
        <property name="Description" type="string">
          WebPart1 Description</property>
        <property name="Hidden" type="bool">False</property>
        <property name="TitleUrl" type="string" />
        <property name="AllowEdit" type="bool">True</property>
        <property name="Height" type="unit">25px</property>
        <property name="HelpUrl" type="string">
          /pages/SimpleHelp.htm</property>
        <property name="Title" type="string">WebPart1 Web Part</property>
        <property name="CatalogIconImageUrl" type="string" />
        <property name="Direction" type="direction">NotSet</property>
        <property name="ChromeState" type="chromestate">Normal</property>
        <property name="AllowZoneChange" type="bool">False</property>
        <property name="AllowHide" type="bool">False</property>
        <property name="HelpMode" type="helpmode">Modal</property>
        <property name="ExportMode" type="exportmode">All</property>
      </properties>
    </data>
  </webPart>
</webParts>

Unfortunately, if your Web Part uses the connection provider or connection consumer interface, you have to either programmatically wire up your Web Parts or, more commonly, allow the connection to be made at run time via the Web Part edit pulldown menu. The tool part connection settings aren't saved in the export file. If you don't want to wire up your Web Parts in the Web Part code, use the SharePoint object model to automate this part of the deployment. Figure 8 shows how to programmatically connect the provider and consumer connection points of two Web Parts.

Figure 8 Connecting Web Parts (Close Figure 8)
// webPartPage is an SPFile object
// page check out is required before modifying publishing pages
webPartPage.CheckOut();

// get the web part manager for the page
SPLimitedWebPartManager webPartMgr =
  webPartPage.GetLimitedWebPartManager(PersonalizationScope.Shared);

// providerWebPart and consumerWebPart are of type
// System.Web.UI.WebControls.WebParts.WebPart

// providerConnection is type
// System.Web.UI.WebControls.WebParts.ProviderConnectionPoint

// consumerConnection is type
// System.Web.UI.WebControls.WebParts.ConsumerConnectionPoint

// connect the web parts
  webPartMgr.SPConnectWebParts(providerWebPart,
  providerConnection,   consumerWebPart, consumerConnection);

// update, check in, and publish the page
webPartPage.Update();
webPartPage.CheckIn("");
webPartPage.Publish("");

You obtain the consumer and provider connection points for the corresponding Web Parts from the collections returned by the GetConsumerConnectionPoints and GetProviderConnectionPoints methods of SPLimitedWebPartManager. This approach works if the connection types are compatible. If they are not, such as when you need to pass filter values from a QueryStringFilter Web Part to a consumer Web Part, you add a transformer to the SPConnectWebParts call, like so:

TransformableFilterValuesToFilterValuesTransformer
  transformer = new
    TransformableFilterValuesToFilterValuesTransformer();

webPartMgr.SPConnectWebParts(providerWebPart,
  providerConnection, consumerWebPart,
  consumerConnection, transformer);

Simplified Site Branding

We have already discussed creating a Web application and site collection. We can now turn our attention to modifying the default site appearance. There are many different ways to brand a MOSS 2007 site, but we will address one approach to provisioning and branding that builds on the out-of-the-box SharePoint collaboration portal, publishing pages, a custom master page, style sheets, page layouts, and themes. While the focus is on publishing sites, keep in mind that many of these techniques are applicable to non-publishing sites as well.

Let's start by saying the following: do not directly modify any of the out-of-the-box SharePoint components. Always make a copy and move it to a separate work space and use your favorite code editor for this task.

SharePoint master pages are implemented in the same way as any ASP.NET 2.0 master page. Once you've created the master page in your favorite tool, upload it to a virtual folder called /_catalogs/masterpage within the top-level SharePoint site. You can then apply the master page to an individual page, site, or entire site collection using the SPWeb class in the Microsoft.SharePoint namespace. MasterUrl and CustomMasterUrl are the two properties of the SPWeb class that identify site master pages. These two properties refer to ~masterurl/default.master and ~masterurl/custom.master, respectively, in the MasterPageFile attribute of the page layout. Setting MasterUrl changes the system master page for forms and views within the site, and setting CustomMasterUrl affects publishing pages within the site. As you might imagine, changing the system master pages does not affect pages below /_layouts in the address, such as the Administrative Site Settings pages and Access Denied page. These inherit from the application.master or simple.master pages.

In MOSS 2007, you can use the AlternateCssUrl property of the SPWeb class to specify an overriding style sheet for a site. WSS sites do not support this option, so your alternative in that environment is to specify the CSS link in the master or content pages. A good location for the CSS file, if it is to be maintained on the site, is in /Style Library. However, any location that is accessible via a URL path in SharePoint is fine, too.

The next step is to apply a theme. A theme presents a way to skin the SharePoint user interface with new colors, styles, and images using CSS. Themes are stored in the \Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\TEMPLATE\THEMES folder on the MOSS server.

A theme is applied at the site scope and is not inherited by child Webs within the site collection. Because of their scoping, themes modify the appearance of all pages within the root Web, including the Site Settings pages. This code sets the Theme for all Webs within a site collection:

// iterate all sites in the site collection
foreach (SPWeb webSite in MSDNSiteCollection.AllWebs)
{
  // apply the Lacquer theme
  webSite.ApplyTheme("Lacquer");

  // update the site
  webSite.Update();
}

You can extend this example to also set the master page and style sheet for Webs that do not inherit settings from the parent site. By default, our approach causes all new child Webs to have the same style settings as the top-level Web.

Creating customized page layouts for your publishing content pages is another option for site branding and goes hand-in-hand with master pages. For information on branding SharePoint, be sure to review Heather Solomon's blog posting on branding starting at the landing page named at heathersolomon.com/blog/articles/sp2007.aspx.

Once you have finished creating your page layouts, you must upload them to the /_catalogs/masterpage gallery, along with the master page. Figure 9 shows how to create a publishing page by selecting a desired page layout from the list of available layouts for the publishing Web site.

Figure 9 Creating a New Page from a Page Layout (Close Figure 9)
PageLayout myLayout = null;

// open a web site
SPWeb webSite = MSDNSiteCollection.OpenWeb();

if (PublishingWeb.IsPublishingWeb(webSite)) {
  // get the publishing web instance for this site
  PublishingWeb publishingWeb = PublishingWeb.GetPublishingWeb(webSite);

  // get the collection of publishing pages
  PublishingPageCollection ppages = publishingWeb.GetPublishingPages();

  // iterate the available page layouts until we find the one we want
  PageLayout[] layouts = publishingWeb.GetAvailablePageLayouts();
  for (int index = 0; index < layouts.Length; index++) {
    if (layouts[index].Name.Equals("my_page_layout.aspx",
      StringComparison.OrdinalIgnoreCase)) {
      myLayout = layouts[index];
      break;
    }
   }

  // add the page to the publishing pages collection
  newPage = ppages.Add("sample_page.aspx", myLayout);

  // set the page title
  newPage.Title = "Test Page";

  // update the page, check in and publish
  newPage.Update();
  newPage.CheckIn("");
  newPage.ListItem.File.Publish("");

  webSite.Update();
}

Modifying navigation is another common branding task since it affects what users can see and how they can proceed through a site hierarchy. The Microsoft.SharePoint.Publishing namespace exposes several classes that target the Publishing site infrastructure, such as PublishingWeb and PublishingPage. Using these classes, we can easily modify navigation for each site.

If you want a child Web to display as a root level site in global navigation, first turn off inheritance from the parent site, like so:

publishingWeb.InheritGlobalNavigation = false;

You might also want to hide all site pages from global navigation. Setting IncludePagesInNavigation to false hides all pages in the site, regardless of whether the PublishingPage.IncludeInGlobalNavigation property is set to true:

// do not show pages in navigation
publishingWeb.IncludePagesInNavigation = false;

If you are dealing with default sites that don't inherit from PublishingWeb, it's still possible to hide these sites from the global navigation bar. For example, if you create a site collection using the collaboration portal template and want to exclude the News site from global navigation, add that site to the __GlobalNavigationExcludes property of the site:

string globalNavExcludes = String.Empty;
SPWeb webSite = MSDNSiteCollection.RootWeb;
// _GlobalNavigationExcludes property contains a delimited string of
// GUIDs identifying the Id of each site to be excluded from global
// navigation

if (webSite.AllProperties.ContainsKey("__GlobalNavigationExcludes")) {
  globalNavExcludes =
    webSite.AllProperties["__GlobalNavigationExcludes"].ToString();
}

SPWeb newsSite = MSDNSiteCollection.AllWebs["News"];
// string is delimited "{GUID};{GUID};",
// use format code B to convert to string
globalNavExcludes += String.Concat(currentWeb.ID.ToString("B"), ";");

webSite.AllProperties["__GlobalNavigationExcludes"] = globalNavExcludes;
webSite.Update();

Adding navigation nodes directly to an SPNavigationNodeCollection is a good way to display only the nodes you want as well as to group nodes and links to external sites. Figure 10 shows how to add an internal link, external link, and a heading to the global navigation bar. This example addresses some of the properties of the SPNavigation class that affect whether the link opens in a new window and how to handle empty URLs.

Figure 10 Creating Navigation Nodes (Close Figure 10)
SPWeb webSite = MSDNSiteCollection.OpenWeb("test", false);

if (PublishingWeb.IsPublishingWeb(webSite)) {
  // get the publishing web instance for this site
  PublishingWeb publishingWeb = PublishingWeb.GetPublishingWeb(webSite);

  // get the collection of global navigation nodes
  SPNavigationNodeCollection navNodes =
    publishingWeb.GlobalNavigationNodes;

  // add an internal link as the first node
  SPNavigationNode internalLink = new SPNavigationNode("Help",
    "Pages/Help.aspx", false);
  navNodes.AddAsFirst(internalLink);

  // add an external link after the internal link node
  SPNavigationNode externalLink = new SPNavigationNode("MSDN",
    "http://msdn2.microsoft.com/en-us/default.aspx", true);
  navNodes.AddAsLast(externalLink, internalLink);

  // set the node to open in a new window
  navNode.Properties["Target"] = "_blank";
  navNode.Update();
  // add a header node with some children
  SPNavigationNode headerNode = new SPNavigationNode(
    "Header", "", false);
  navNodes.AddAsLast(headerNode);

  // set the node to be a header element
  headerNode.Properties["UrlFragment"] = "";
  headerNode.Properties["NodeType"] = "Heading";
  headerNode.Properties["BlankUrl"] = "True";
  headerNode.Update();

  // create the child nodes
  SPNavigationNode child1Node = new SPNavigationNode("Test1",
    "Pages/Test1.aspx", false);
  headerNode.Children.AddAsLast(child1Node);

  SPNavigationNode child2Node = new SPNavigationNode("Test2",
    "Pages/Test2.aspx", false);
  headerNode.Children.AddAsLast(child2Node);

  headerNode.Update();
  publishingWeb.Update();
}

There is a lot more to branding a site than what we've covered here. However, these code samples should help you get started with deploying customizations and configuring a site collection to apply your brand.

Setting Up Lists

Because lists are a fundamental feature of any SharePoint site, getting them deployed in an automated way is important. There are two common approaches for automating list deployment: deploying list definitions and deploying individual populated list instances. The approach you take really depends on exactly what you want to achieve. If you want list structures that can be created from list templates, then list definitions are the way to go. If your goal is to deploy a single instance of a list with values, then use the object model to generate a deployable list instance.

If you decide list definitions are what you need, you can use the SharePoint Solution Generator utility in Visual Studio Extensions for WSS to create a list definition Visual Studio project. Using the generator, you choose a Web application or specify a Web application, as shown in Figure 11.

Figure 11 Generate List Definition Wizard
Figure 11 Generate List Definition Wizard(Click the image for a smaller view)
Figure 11 Generate List Definition Wizard
Figure 11 Generate List Definition Wizard (Click the image for a larger view)

The wizard then has you select a target site collection or a specific Web within a site collection as the target of the extraction process. If you choose the List Definition option in Visual Studio Extensions for WSS, you can pick specific lists to include in the Visual Studio project. Before extracting your lists to a project, be sure that you've configured the source lists exactly as you want them to appear when they are deployed elsewhere.

If you're creating list instances in your custom deployment that you don't want templated, you can use the object model to create lists and configure list settings, such as adding a view. Note that the key classes for this operation are SPListCollection, SPList, SPListItemCollection, SPListItem, and SPViewCollection in the Microsoft.SharePoint namespace. The WSS 3.0 SDK does a good job of covering these classes.

The pattern for creating a list, creating items in a list, and creating a view all involve getting a collection, creating an object, and adding the object to the collection. For creating a list and adding items to it, you also must call the Update method to submit the update to the server. Review the SPListConfig class in the code download for simple methods that demonstrate how to complete all of these tasks with code. Note, though, that to keep the code download as simple as possible, we have left out object disposal and error handling.

The trickiest part of the effort will be creating the view in your code because programmatic view configuration is achieved using the Collaborative Application Markup Language (CAML) XML query syntax. One way to simplify the process is to create a view for a list through the SharePoint Web interface, and then extract your list using the SharePoint Solution Generator. Finally, open the schema.xml file and find the View tag containing the name of your custom view. Within that tag, look for the Query element and copy the XML within it to a string variable in your code. This is the CAML that gets passed in as the third parameter of the Add method in the SPViewCollection class (views object), as shown here:

StringCollection fieldsInView = new StringCollection();

string query =
  "<GroupBy Collapse=\"TRUE\" GroupLimit=\"100\">" +
    "<FieldRef Name=\"" + _fieldBool + "\" />" +
  "</GroupBy>" +
  "<OrderBy>" +
    "<FieldRef Name=\"Title\" />" +
  "</OrderBy>";

views.Add("myView", fieldsInView, query, 10, true, true);

You can also use the Lists class in the Lists Web service to complete these tasks. However, for an automated deployment, working with the object model directly is faster.

Form and Report Deployment

InfoPath® form development is a large topic unto itself, and there are many resources to help you understand how to develop InfoPath form applications. Using the InfoPath form templates from Visual Studio 2008 or Visual Studio 2005 is our preferred approach.

Once you have your InfoPath form application completed, the next step is to publish it to either a SharePoint farm or to a network location from Visual Studio. When you click Publish on the Build menu in Visual Studio, the Publishing Wizard deploys an .xsn file to the target location, which is an InfoPath form package containing your form project components, such as an .xsf InfoPath form, an assembly, and various XML and schema files. The process of adding the InfoPath package into the Central Administration—Application Management—Manage Form Templates library causes SharePoint to build a solution file (.wsp package) for you, which is automatically added to the target SharePoint farm.

We haven't given any detail on SharePoint solutions or methods for building them up to this point. For the purpose of automating deployment of your InfoPath form application, the key is to recognize that SharePoint builds the solution file and adds it to the farm for you. After that's accomplished, you can use Mark Wagner's simple-but-powerful solutionExport tool to extract the .wsp file for deployment to other SharePoint farms. You can read about this tool and download it from Mark Wagner's blog (mentioned previously)﻿﻿.

There are two very important caveats to keep in mind here. The first is that the solution is only updated when you deploy a new .xsn file to the farm. So if you are planning to deploy the solution package to a different farm environment, you need to make sure that the data connection used by your InfoPath form points to the correctly configured data connections. In addition, the .udcx data connection files do not get packaged with the InfoPath form .xsn package or the resulting .wsp solution package.

One way to automate the deployment of the data connection files is to create another solution package that contains a Feature to deploy the .udcx files to a target data connection library when the Feature is activated. As you move your InfoPath Solution files from one SharePoint instance to the next, you will have a much easier time deploying them if you keep the relative path to your form library and data connection library the same. This is a perfect example of convention over configuration.

With the release of SQL Server 2005 SP2, Microsoft created a new mode in SQL Server Reporting Services that integrates reports into SharePoint. This new mode is called SharePoint Integrated Mode. Unfortunately, Microsoft didn't address how to automate the deployment of reports and their associated data connections into SharePoint.

Fortunately, Shawn Feldman has come to the rescue with his brilliant blog entry, Automating Report Deployment with Reporting Services in SharePoint Integration Mode, at feldrox.spaces.live.com/blog/cns!C46024CE04ED4278!469.entry﻿. ﻿In this blog, he explains exactly how you can automate report deployment. In fact, this entry explains related tasks and how to automate them using the SharePoint object model and the ﻿ReportingServices2006 Web service. Here are the tasks he demonstrates using these APIs:

  • Creating a reports library in SharePoint
  • Adding a Content Type to the library
  • Creating a data source and adding it to a SharePoint data connection library
  • Adding a Reporting Services report to the reports library
  • Associating a Reporting Services report with the data source

Shawn doesn't cover programmatic Feature activation; however, we demonstrate how you can accomplish this step in the "Solutions and Features" sidebar.

There are many other deployment tasks we couldn't cover here, such as configuring SharePoint permissions, integrating unique IIS Web applications that run in the context of a SharePoint instance, and deploying HTTP handlers and updating web.config following their deployment. The tasks we did cover, however, should get you well on your way to fully automating your deployment.

What Next?

We have given you a whirlwind tour of how to automate the most common tasks involved in getting a custom SharePoint application deployed. The next steps to further your automation efforts are to continue your exploration of the SharePoint object model, work with one or more solution packaging tools, and move appropriate automation steps into your solution packaging framework.

The "SharePoint Feature and Solution Resources" sidebar provides links to some additional helpful articles and blog posts on developing and deploying elements of your SharePoint application. In addition, we've provided more information about creating solutions in the "Some Approaches for Creating Solutions" sidebar.

Some Approaches for Creating Solutions

While you can create solutions manually, there are a number of tools released by both Microsoft and the SharePoint community to ease the pain of building and deploying such solutions.

Visual Studio Extensions for WSS 3.0

Visual Studio Extensions for WSS 3.0 is an effective tool for automated SharePoint component build and deployment activities and it's fully integrated into Visual Studio 2005. A distinct advantage of the Extensions for WSS is the deployment capability it adds to Visual Studio. But while this deployment capability works really well for individual solutions, it is not ideal for large teams where full control and automation of a centralized deployment is important.

The biggest limitation with the Extensions for WSS is that it is designed to run in a Visual Studio installation on a SharePoint server. You can get the extensions installed on Windows XP or Windows Vista and use a limited set of capabilities, such as creating a Web Part project, by adding the following registry entry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server
Extensions\12.0]
"SharePoint"="Installed"
Other developers who want to work on your code must have the Extensions for WSS installed or they won't be able to open your project.

Some encouraging developments with Visual Studio Extensions for WSS version 1.1 include the ability to edit your .wsp solution and build it in real-time in the WSPView pane. This provides a powerful way to work with the files and structure of your solution package. In addition, when you deploy a solution, a copy of the .wsp file is dropped into the build folder for your project so that you can deploy it independently of the Extensions for WSS deployment mechanism.

Visual Studio Extensions for WSS version 1.2, which is planned to support Visual Studio 2008, is expected to ship this year. Read about the new version on the Microsoft SharePoint Products and Technologies Team Blog at blogs.msdn.com/sharepoint/archive/2008/02/11/announcing-the-final-release-of-vsewss-1-1-and-the-upcoming-version-1-2.aspx.

WSP Builder

WSP Builder, by Carsten Keutmann, is an outstanding tool worth considering for your SharePoint solution build efforts. WSP Builder is particularly useful for continuous integration scenarios and in situations where developers hand off the packaging effort.

The tool creates solution files based on a folder structure that mimics the SharePoint \12 folder, as well as a few other specific folders. In Visual Studio, you simply create the folder structure in your project, add your files, and run WSPBuilder as a post-build process or independently of your build process. WSPBuilder automatically creates the manifest.xml file and bundles it into a solution file to deploy the content of the \12 folder.

If your Visual Studio solution contains multiple projects containing Features and assemblies, you simply point the SolutionPath option to your Visual Studio solution folder and WSPBuilder will package all the components into a single SharePoint solution. The need to create a specific folder structure within your project may require extensive modification to existing projects and may be confusing to developers not familiar with Features or solutions.

Bear in mind that open source tools like WSPBuilder are constantly evolving and regression testing may not be as robust as that of commercial tools. Consequently, an update to the tool could impact features that worked in an earlier release. For instance, the addition of the feature to build solutions in WSPBuilder changed the behavior of the existing SolutionPath option and required changing it to ProjectPath; otherwise an empty solution would be created.

A new version of WSPBuilder, WSPBuilder Extensions, is available as an add-in to Visual Studio 2005 and Visual Studio 2008. The extensions add two WSPBuilder project types and several item types, such as Web Part with Feature and Feature with Receiver. Adding an item creates the necessary folder structure and files needed for packaging and deployment, including the .WebPart file for Web Part items. The extensions also add deployment options to the project menu, with the limitation that you can only deploy locally.

Currently, there is limited documentation available for this release; however, the author walks through using the extensions in his blog. The project is located at codeplex.com/wspbuilder.

SPDeploy

The SPDeploy tool is an appealing blend of what SharePoint development teams are looking for in a solution packaging and deployment tool. What's really appealing about this tool is its focus on remote deployment. With it, you create your .wsp file from a standard C# class library project inside Visual Studio 2005 and it automatically reads the tree of files in Visual Studio and builds your .wsp package. The tool supplies an extension to MSBuild and incorporates .targets files to provide a reliable incremental build and deployment process.

Deployment targets are executed at the command prompt and can also be launched from Visual Studio 2005 with some minor modifications. Targets include CompileWsp, AddWsp, DeployWsp, UpgradeWsp, RetractWsp, DeleteWsp, UpgradeIncremental, UpgradeIncrementalFiles, UpgradeIncrementalAssembly, CreateSite, DeleteSite, CreateWebApplication, and DeleteWebApplication.

The author of the tool, Clint Simon, provides some useful tips on remote debugging. Also, be sure to read the blog post titled "SharePoint Custom Application Development Methodologies" at ascentium.com/blog/sp/default.aspx. It's the most accurate view we have seen for development teams of more than one.

STSDev

Another promising deployment tool emerging from the CodePlex community is STSDev (Simple Tools for SharePoint 2007 Development). The project is coordinated by SharePoint guru Ted Pattison, who wrote about it in the March 2008 issue of MSDN Magazine (msdn2.microsoft.com/magazine/cc337895.aspx). This tool provides a good foundation for building, testing, and debugging .wsp files and is very useful for development teams who need to settle on a common process for their packaging and deployment efforts.

STSDev is a console application that initiates different dialog boxes and uses a custom .targets file to build .wsp files. While the tool is not integrated into Visual Studio, it can be shelled out to Visual Studio by pointing to the .exe file through the external tools interface. Once you make the Visual Studio connection, the tool automates the process of creating and updating the manifest.xml and .ddf files and adds common SharePoint developer commands to the Visual Studio environment.

STSDev creates the basic Visual Studio solution project structure or deployment files directory that populates the requisite SharePoint files needed to generate the .wsp file. It also creates the solutionconfig.xml file used to modify the input parameters for the manifest.xml file. The Microsoft.SharePoint.targets file provides a number of different commands that allow you to choose and switch between different deployment configurations.

This is a great tool for developers who are new to SharePoint development and the CodePlex project site contains three excellent tutorials worth viewing. You can access the project at codeplex.com/stsdev/Release/ProjectReleases.aspx?ReleaseId=10119.

Ethan Wilansky is a Microsoft Directory Services MVP and an enterprise architect. In his day job, he leads a development team currently focused on building custom SharePoint applications, and he serves as a directory services programming solutions SME.

Paul Olszewski is an Information Specialist on a .NET Capability team. He specializes in creating and deploying reusable component applications leveraging Microsoft technologies.

Rick Sneddon is a Senior Infrastructure Specialist for EDS in its Integration Engineering practice and an MCSE. He is focused on creating portal architectures, developing technology strategies, and advising customers on the operational aspects of deploying and supporting portal composite application solutions.


C++ Plus: Beef Up Windows Apps with the Visual C++ 2008 Feature Pack

C++ Plus

Beef Up Windows Apps with the Visual C++ 2008 Feature Pack

Kenny Kerr
This article is based on a prerelease version of the Visual C++ Feature Pack. All information herein is subject to change.
This article discusses:
  • Microsoft Foundation Classes updates
  • Programming the Ribbon in C++
  • Tabbed MDIs in C++
  • Polymorphic functions and smart pointers
This article uses the following technologies:
Visual Studio 2008, MFC
Contents
Office Ribbon User Interface
Tabbed Multiple Document Interface
What's New in the Standard C++ Library
Polymorphic Function Objects
Smart Pointers

As a developer using Visual C++ , you may have felt a bit left behind in recent years as it seems like Microsoft has added more new features and functionality to Visual C#® than to Visual C++®. The truth is that although the Visual C++ compiler has continued to improve in a variety of areas including performance, security, and standards conformance, little has been done in the way of new library and productivity features for quite some time. And while MFC was updated to better support Windows Vista®, more could have been done.

Now, however, to better support developers who use native code and MFC in particular, Microsoft has released the Visual C++ 2008 Feature Pack. Here's your evidence of a renewed commitment to Visual C++.

The feature pack includes a large set of new MFC classes for building modern user interfaces. It also includes a significant amount of the functionality being added to the Standard C++ Library as part of Technical Report 1 (TR1). TR1 is the first major update and addition to the Standard C++ Library adopted by the C++ committee.

The traditional paradigms of single and multiple document/view applications, menus, toolbars and dialogs have been a staple of MFC development for years. If you wanted your MFC application to look a little more modern, you'd be on your own.

That's all changed. MFC now includes many new user interface paradigms including dockable panes resembling those found in Microsoft® Office and Visual Studio®. It also includes full support for the Microsoft Office Ribbon user interface as well as many other new controls, dialog boxes and windows.

Next, I'm going to demonstrate two of the new user interface features in MFC: the Office Ribbon and the Tabbed Multiple Document Interface (MDI).

Office Ribbon User Interface

By now, I'm certain you've seen the new 2007 Microsoft Office system Ribbon element, and you may have wondered how to create that look in your own applications. The good news is that it is surprisingly easy to add a Ribbon bar to an MFC frame window.

Much of the new functionality relies on new versions of the CWinApp, CFrameWnd, and CMDIFrameWnd classes; these classes represent the foundation for most MFC applications. CWinAppEx derives from CWinApp and should be used as the base class for your application object. CFrameWndEx derives from CFrameWnd and should be used as the base class for your single document interface (SDI) frame window. Similarly, CMDIFrameWndEx derives from CMDIFrameWnd and should be used as the base class for your MDI frame windows. These new base classes provide all of the plumbing necessary to support many of the new user interface facilities such as dockable and resizable window panes as well as workspace persistence.

Figure 1 shows a minimal application object that can support a Ribbon bar. As you can see, the Application class derives from CWinAppEx and implements the familiar InitInstance member function that's traditionally used to create the application's main window. You must remember to call the SetRegistryKey member function to set the registry location for application settings since the frame classes rely on it. InitInstance then proceeds to create the main window in the usual manner.

Figure 1 Ribbon Application Object (Close Figure 1)
class Application : public CWinAppEx
{
public:

    virtual BOOL InitInstance();

};

BOOL Application::InitInstance()
{
    SetRegistryKey(L"SampleCompany\\SampleProduct");

    m_pMainWnd = new MainWindow;
    m_pMainWnd->ShowWindow(m_nCmdShow);
    m_pMainWnd->UpdateWindow();

    return TRUE;
}

The code in Figure 2 shows a minimal SDI frame window with a Ribbon bar and application button. The application button is not required but is typically used in conjunction with a Ribbon bar to provide a main menu of sorts for the application in lieu of the traditional File menu.

Figure 2 Ribbon Frame Window (Close Figure 2)
class MainWindow : public CFrameWndEx
{
    DECLARE_MESSAGE_MAP()

public:

    MainWindow();

private:

    int OnCreate(CREATESTRUCT* createStruct);

    CMFCRibbonBar m_ribbon;
    CMFCRibbonApplicationButton m_appButton;
};

BEGIN_MESSAGE_MAP(MainWindow, CFrameWndEx)
    ON_WM_CREATE()
END_MESSAGE_MAP()

MainWindow::MainWindow()
{
    Create(0, // class name
        L"MFC Ribbon Sample Application");
}

int MainWindow::OnCreate(CREATESTRUCT* createStruct)
{
    if (-1 == __super::OnCreate(createStruct))
    {
        return -1;
    }

    if (-1 == m_ribbon.Create(this))
    {
        return -1;
    }

    m_appButton.SetImage(IDB_APP_BUTTON);

    m_ribbon.SetApplicationButton(&m_appButton,
        CSize(45, 45));

    CMFCRibbonMainPanel* appButtonMenu =
        m_ribbon.AddMainCategory(L"Menu",
        IDB_APP_BUTTON_MENU_SMALL,
        IDB_APP_BUTTON_MENU_LARGE);

    appButtonMenu->Add(new CMFCRibbonButton(ID_FILE_NEW,
        L"&New",
        0, // small image index
        0)); // large image index

    appButtonMenu->Add(new CMFCRibbonButton(ID_FILE_OPEN,
        L"&Open...",
        1, // small image index
        1)); // large image index

    appButtonMenu->AddToBottom(new CMFCRibbonMainPanelButton(ID_APP_EXIT,
        L"E&xit",
        15));
        //small image index

    CMFCRibbonCategory* category = m_ribbon.AddCategory(L"Home",
        IDB_RIBBON_CAT_HOME_SMALL,
        IDB_RIBBON_CAT_HOME_LARGE);

    CMFCRibbonPanel* panel = category->AddPanel(L"Clipboard");

    panel->Add(new CMFCRibbonButton(ID_EDIT_PASTE,
        L"Paste",
        0, // small image index
        0)); // large image index

    panel->Add(new CMFCRibbonButton(ID_EDIT_CUT, L"Cut", 1));
    panel->Add(new CMFCRibbonButton(ID_EDIT_COPY, L"Copy", 2));
    panel->Add(new CMFCRibbonButton(ID_EDIT_SELECT_ALL,
        L"Select All", -1));

    m_ribbon.AddCategory(L"Insert",
        IDB_RIBBON_CAT_HOME_SMALL,
        IDB_RIBBON_CAT_HOME_LARGE);

    CMFCVisualManager::SetDefaultManager(
        RUNTIME_CLASS(CMFCVisualManagerOffice
        2007));
    CMFCVisualManagerOffice2007::SetStyle
        (CMFCVisualManagerOffice2007::Office
        2007_LunaBlue);

    return 0;
}

Conceptually, a Ribbon consists of a number of tabs known as categories that each host a group of panels. These panels, in turn, host the Ribbon elements or controls that represent the various operations specific to your application. If a Ribbon hosts an application button (the large circular button in the top-left corner), then the popup window displayed when the user clicks on the application button also displays a panel that is considered the Ribbon's main category.

The CMFCRibbonBar class implements the Ribbon bar itself and the CMFCRibbonApplicationButton class represents the application button hosted by the Ribbon bar and displayed in the top-left corner of the window frame. The Ribbon bar is typically created and prepared inside your WM_CREATE message handler. To create the Ribbon bar, simply call CMFCRibbonBar's Create member function providing the address of the window frame to attach it to. You can then populate it as you wish. The AddMainCategory member function adds the main category to a Ribbon and returns a pointer to the CMFCRibbonMainPanel to which you can add the Ribbon elements that will be featured on this panel.

More categories can be added to represent the Ribbon's tabs by calling the AddCategory member function. AddCategory returns a pointer to a CMFCRibbonCategory object to which you can add panels using its AddPanel member function. AddPanel returns a pointer to a CMFCRibbonPanel object to which you can add Ribbon elements just like you can with the Ribbon's main panel. Finally, you can set the visual manager that is responsible for the style and appearance of the frame window using the CMFCVisualManager::SetDefaultManager static member function. Figure 3 shows you what the Ribbon application might look like, assuming you've added the necessary event handlers for the buttons on the Ribbon bar.

Figure 3 Ribbon Sample Application
Figure 3 Ribbon Sample Application

Tabbed Multiple Document Interface

MFC has long supported an MDI implementation with its document/view architecture, but the traditional MDI you see in Figure 4 looks so outdated, your users may think your application hasn't been updated since Windows® 95. Today, the majority of users expect multiple documents to be accessible through tabs along the edge of a window and this is exactly what the new CMDIFrameWndEx MDI frame window provides.

Figure 4 The Windows Stone Age
Figure 4 The Windows Stone Age(Click the image for a smaller view)
Figure 4 The Windows Stone Age
Figure 4 The Windows Stone Age (Click the image for a larger view)

You'll need to update your multiple document/view application object to support the new frame window. Figure 5 shows a minimal application object that will suffice. It is similar to a traditional MDI application object, but a few points are worth mentioning.

Figure 5 Tabbed MDI Application Object (Close Figure 5)
class Application : public CWinAppEx
{
    DECLARE_MESSAGE_MAP()

public:

    virtual BOOL InitInstance();

};

BEGIN_MESSAGE_MAP(Application, CWinAppEx)
    ON_COMMAND(ID_FILE_NEW, &CWinAppEx::OnFileNew)
END_MESSAGE_MAP()

BOOL Application::InitInstance()
{
    SetRegistryKey(L"SampleCompany\\SampleProduct");

    VERIFY(InitContextMenuManager());

    AddDocTemplate(new CMultiDocTemplate(IDR_CHILDFRAME,
                                         RUNTIME_CLASS(Document),
                                         RUNTIME_CLASS(CMDIChildWndEx),
                                         RUNTIME_CLASS(View)));

    MainWindow* mainWindow = new MainWindow();
    VERIFY(mainWindow->LoadFrame(IDR_MAINFRAME));

    m_pMainWnd = mainWindow;
    m_pMainWnd->ShowWindow(m_nCmdShow);
    m_pMainWnd->UpdateWindow();

    return TRUE;
}

First, the runtime class for the child window frame is CMDIChildWndEx instead of the traditional CMDIChildWnd class. The InitContextMenuManager function is also called to prepare the menu manager used when switching between the tabbed views.

Figure 6 shows a minimal MDI frame window. Again, you should be pleased to see how little is involved in turning on this functionality. All that's really needed is a call to the EnableMDITabbedGroups member function to enable the MDI tabbed groups feature. The CMDITabInfo class provides a variety of member variables that you can use to customize the appearance and behavior of the tabbed groups. And as its name implies, it even allows the user to drag different views to create groups of tabs either vertically or horizontally aligned. Figure 7 shows what it might look like.

Figure 6 Tabbed MDI Frame Window (Close Figure 6)
class MainWindow : public CMDIFrameWndEx
{
    DECLARE_DYNCREATE(MainWindow)
    DECLARE_MESSAGE_MAP()

private:

    int OnCreate(CREATESTRUCT* createStruct);

};

IMPLEMENT_DYNCREATE(MainWindow, CMDIFrameWndEx)

BEGIN_MESSAGE_MAP(MainWindow, CMDIFrameWndEx)
    ON_WM_CREATE()
END_MESSAGE_MAP()

int MainWindow::OnCreate(CREATESTRUCT* createStruct)
{
    if (-1 == __super::OnCreate(createStruct))
    {
        return -1;
    }

    CMDITabInfo tabInfo;
    tabInfo.m_bAutoColor = true;
    tabInfo.m_bDocumentMenu = true;
    EnableMDITabbedGroups(true, tabInfo);

    return 0;
}
Figure 7 Modern Tabbed MDI Application
Figure 7 Modern Tabbed MDI Application(Click the image for a smaller view)
Figure 7 Modern Tabbed MDI Application
Figure 7 Modern Tabbed MDI Application (Click the image for a larger view)

What's New in the Standard C++ Library

As I've mentioned, the feature pack also includes significant additions to the Standard C++ Library as a part of TR1. This includes support for reference-counted smart pointers, polymorphic function wrappers, hashtable-based containers, regular expressions, and much more. In the following scenarios, I'll take a look at some of these new TR1 features.

Polymorphic Function Objects

The ability to refer to a function as a value that can be passed as a parameter or stored for later use is a vital part of many applications. This concept can be used to implement various common constructs including callback functions, event handlers and asynchronous programming facilities. Functions, however, can be quite unwieldy in C++. The design of functions was primarily driven by the need for compatibility with C and for great performance. Although these goals were achieved, it didn't make it any easier to treat functions as objects that can be stored, passed around and ultimately called asynchronously. Let's take a look at a few common function-like constructs in C++.

First, there is the good old non-member function:

int Add(int x, int y)
{
    return x + y;
}

Naturally, it can be called as follows:

int result = Add(4, 5);
ASSERT(4 + 5 == result);

Another common function-like construct is the function object, or functor:

class AddFunctor
{
public:
    int operator()(int x, int y) const
    {
        return x + y;
    }
};

Since it implements the call operator, a function object can be used as if it were a function:

   AddFunctor fo;

   int result = fo(4, 5);
   ASSERT(4 + 5 == result);

Then there is the non-static member function:

   class Adder
   {
   public:
       int Add(int x, int y) const
       {
           return x + y;
       }
   };

Of course, calling the member function requires an object:

Adder adder;

int result = adder.Add(4, 5);
ASSERT(4 + 5 == result);

So far, so good. Now let's say you need to store these function-like constructs for later use. You can define a type that is capable of storing a pointer to a non-member function as follows:

typedef int (*FunctionPointerType)(int x, int y);

A function pointer can also be used as if it were a function:

FunctionPointerType fp = &Add;

int result = fp(4, 5);
ASSERT(4 + 5 == result);

Although the function object can also be stored, it cannot be stored polymorphically along with a function pointer.

The member function can be stored in a pointer-to-member-function:

Adder adder;

typedef int (Adder::*MemberFunctionPointerType)(int x, int y);
MemberFunctionPointerType mfp = &Adder::Add;

But the pointer-to-member-function type is not compatible with the pointer-to-non-member-function type and thus cannot be stored polymorphically alongside its non-member function rival. Even if that were possible, the member function still would require an object to provide the context for the member function call:

int result = (adder.*mfp)(4, 5);
ASSERT(4 + 5 == result);

I could go on, but I think you get the idea. Fortunately, a solution is provided by the new tr1::function class template. The tr1::function class template holds a callable object for the function type defined in its template parameter. Here I'm initializing it with the non-member function:

function<int (int x, int y)> f = &Add;

int result = f(4, 5);
ASSERT(4 + 5 == result);

You can just as easily initialize it with the function object:

function<int (int x, int y)> f = AddFunctor();

You can even use the new function binding facility to initialize it with the member function:

function<int (int x, int y)> f = bind(&Adder::Add, &adder, _1, _2);

I'll talk about the bind function in a moment but the beauty here is that now there is a single function wrapper that you can bind to non-member functions, function objects, or even member functions. You can store it and call it at some point in the future, and it's all done polymorphically.

The function wrapper is also rebindable and can be set to nothing just like normal function pointers:

function<int (int x, int y)> f;
ASSERT(0 == f);

f = &Add;
ASSERT(0 != f);

f = bind(&Adder::Add, &adder, _1, _2);

The bind function template provides a much more powerful alternative to the function object adapters in the Standard C++ Library—specifically std::bind1st() and std::bind2nd(). In this example, the first parameter of bind is the address of the member function. The second parameter is the address of the object on which the member is to be called. The last two parameters in this example define placeholders that will be resolved when the function is called.

Of course, bind is not limited to member functions. You can create a squaring function by binding the Standard C++ Library's multiplies function object to produce a function with a single parameter that will produce the result of the parameter squared:

function<int (int)> square = bind(multiplies<int>(), _1, _1);

int result = square(3);
ASSERT(9 == result);

Note that the tr1::function class template works great with Standard C++ Library algorithms. Given a container of integers, you can produce the sum of all the values using the member function, as follows:

function<int (int x, int y)> f = // initialize

int result = accumulate(numbers.begin(),
                        numbers.end(),
                        0, // initial value
                        f);

Keep in mind that the tr1::function class template can inhibit compiler optimizations (such as inlining) that may be possible if you simply use a function pointer or function object directly. Therefore, you should only use the tr1::function class template when necessary, such as in the case of algorithms like accumulate where it may be called repeatedly. When possible, you should pass function pointers, member function pointers (adapted with mem_fn from TR1), and function objects (such as those returned by bind) directly to Standard C++ Library algorithms and other templated algorithms.

Let's not stop there. Here's a more interesting problem. Say you have a Surface class representing some drawing surface and a Shape class that is capable of drawing itself onto the surface:

class Surface
{
    //...
};

class Shape
{
public:
    void Draw(Surface& surface) const;
};

Now consider how you might draw each shape in a container onto a given surface. You might want to use the for_each algorithm, as shown in the following:

Surface surface = // initialize

for_each(shapes.begin(),
         shapes.end(),
         bind(&Shape::Draw, _1, surface)); // wrong

Here I'm taking advantage of the bind function template to invoke the member function on each element of the shapes container, binding the surface as the parameter to the Draw member function. Unfortunately, depending on how Surface is defined, it may not work as expected or even compile. The trouble is that the bind function template will try to make a copy of surface when what you really want is a reference. Fortunately, TR1 also introduces the reference_wrapper class template that allows you to treat a reference as a value that may be freely copied. The ref and cref function templates simplify the creation of reference_wrapper objects thanks to their type inference.

With the help of reference_wrapper, the for_each algorithm is now able to successfully draw the shapes to the surface both simply and efficiently:

for_each(shapes.begin(),
         shapes.end(),
         bind(&Shape::Draw, _1, ref(surface)));

As you can imagine, the new function wrappers, binding features, and reference wrappers can be combined in a myriad of ways to solve all kinds of problems elegantly.

Smart Pointers

Smart pointers are an essential tool for C++ developers. I routinely use ATL's CComPtr for handling COM interface pointers and the Standard C++ Library's auto_ptr for handling raw C++ pointers. The latter is useful for scenarios where you need to dynamically create C++ objects and ensure the object is safely deleted when the auto_ptr object goes out of scope.

As useful as auto_ptr is, it can only be used safely in a few scenarios. This is because of the transfer-of-ownership semantics that it implements. That is, if you copy or assign an auto_ptr object, the ownership of the underlying resource is transferred and lost by the originating auto_ptr object. Although this is clearly useful in scenarios where you have fine-grained control over resource allocations, there are many scenarios where you might need to share objects, and a smart pointer implementing shared-ownership semantics would be very useful indeed. More importantly, auto_ptr cannot be used with Standard C++ Library containers.

TR1 introduces two new smart pointers that work together to provide a multitude of uses. The shared_ptr class template works in much the same way as auto_ptr but instead of transferring ownership of a resource it merely increases a reference count on the resource. When the last shared_ptr object that holds a reference to the object is destroyed or reset, the resource is automatically deleted. The weak_ptr class template works with shared_ptr to allow callers to reference a resource without affecting its reference count. This is useful for object models where you might have cyclical relationships or perhaps if you're implementing a caching service. It also works great with Standard C++ Library containers!

As a comparison, consider the following use of auto_ptr:

auto_ptr<int> ap(new int(123));

ASSERT(0 != ap.get());

// transfer ownership from ap to ap2
auto_ptr<int> ap2(ap);

ASSERT(0 != ap2.get());
ASSERT(0 == ap.get());

The auto_ptr copy constructor transfers the ownership from ap to ap2. The behavior of shared_ptr is equally predictable:

shared_ptr<int> sp(new int(123));

ASSERT(0 != sp);

// increase reference count of shared object
shared_ptr<int> sp2(sp);

ASSERT(0 != sp2);
ASSERT(0 != sp);

Internally, all shared_ptr objects that refer to the same resource share a control block that tracks the number of shared_ptr objects that jointly own the resource as well as the number of weak_ptr objects that refer to it. I'll show you how to use the weak_ptr class template in a moment.

Member functions similar to auto_ptr are provided by shared_ptr. They include the dereference and arrow operators, the reset member function to replace the resource, and the get member function that returns the address of the resource. A few unique member functions, including one coincidentally named unique, are also provided. The unique member function tests whether the shared_ptr object is the only smart pointer to hold a reference to the resource. Here's an example:

shared_ptr<int> sp(new int(123));
ASSERT(sp.unique());

shared_ptr<int> sp2(sp);
ASSERT(!sp.unique());
ASSERT(!sp2.unique());

You can also get the number of shared_ptr objects that own the resource with the use_count member function:

shared_ptr<int> sp;
ASSERT(0 == sp.use_count());

sp.reset(new int(123));
ASSERT(1 == sp.use_count());

shared_ptr<int> sp2(sp);
ASSERT(2 == sp.use_count());
ASSERT(2 == sp2.use_count());

You should, however, limit the use of use_count to debugging only as it is not guaranteed to be a constant time operation on all implementations. Note that you can rely on the provided operator unspecified-bool-type to determine whether the shared_ptr owns anything, and you can use the unique function to determine whether the shared_ptr is the unique owner of something.

The weak_ptr class template stores a weak reference to a resource owned by a shared_ptr object. Once all of the shared_ptr objects owning the resource are destroyed or reset, the resource is deleted regardless of whether any weak_ptr objects are referring to it. To ensure that you cannot use a resource referred to only by a weak_ptr object, the weak_ptr class template does not provide the familiar get member function to return the resource's address or the member access operator. Instead, you need to first convert the weak reference into a strong reference in order to access the resource. This functionality is provided by the lock member function, as shown in Figure 8.

Figure 8 Convert to Strong Reference (Close Figure 8)
Surface surface;

shared_ptr<Shape> sp(new Shape);
ASSERT(1 == sp.use_count());

weak_ptr<Shape> wp(sp);
ASSERT(1 == sp.use_count()); // still 1

// arbitrary application logic...

if (shared_ptr<Shape> sp2 = wp.lock())
{
    sp2->Draw(surface);
}

If in the interim the resource is released, the weak_ptr object's lock member function will return a shared_ptr object that does not own a resource. As you can probably imagine, shared_ptr and weak_ptr can dramatically simplify resource management in many applications.

It's pretty clear that Visual C++ 2008 Feature Pack is a welcome upgrade to the Visual C++ libraries and is sure to come in handy! There's so much more to explore but I hope that this introduction has piqued your interest and that you'll set aside a few moments to take a closer look yourself.

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. You can reach Kenny at his blog, weblogs.asp.net/kennykerr.


Security: Safer Authentication with a One-Time Password Solution

Security

Safer Authentication with a One-Time Password Solution

Dan Griffin
Code download available at: OTPAuthentication2008_05.exe (496KB)
This article discusses:
  • The problems with passwords
  • One-time password generation
  • Building a Web service-based OTP solution
  • Testing and deploying OTP
This article uses the following technologies:
IIS 7.0, SQL Server
Contents
One-Time Passwords
A Complete OTP Solution
Test OTP Generator Client
The Sample Web Site
The Authentication Web Service
The Complete Architecture
Running the Code Sample
Deployment Considerations
Give It a Try

Passwords can be a big security and manageability headache for enterprise IT administrators. Users often create simple passwords or write their passwords down to make sure that they'll remember them. In addition, there are few secure and efficient procedures for resetting passwords.

Knowing these limitations, how can you mitigate these types of security problems when remote users access your network? Seeing that many users write down their passwords, how would you make your company's password solution more robust? I'll show you how to use standards-based technologies with C# and C to develop a one-time password (OTP) proof of concept. First, though, I want to briefly take a broader look at password-replacement technologies.

There are a few ways that you could go about eliminating standard passwords for your remote users. You could use certificate authorities to issue certificates to your users, but this requires a public key infrastructure (PKI) and is expensive to set up and maintain. It can also be difficult to manage certificates for remote users, especially if you are using a hardware-based token, such as a smart card. This kind of trade-off of high cost for high security is a common theme.

Alternatively, you could use SecureID, which is the one-time password solution from RSA. However, you should note that SecureID is not based on a standard, which can cause incompatibilities and licensing overhead.

A third option is to use a standards-based OTP solution. But what kind of one-time password options are out there, and why is OTP better than traditional passwords anyway? Well, let's see.

One-Time Passwords

A traditional, static password is usually only changed when necessary: either when it has expired or when the user has forgotten it and needs to reset it. Because passwords are cached on computer hard drives and stored on servers, they are susceptible to cracking. This is especially a concern for laptops since they can be easily stolen.

Many businesses give employees laptops and open their networks to remote access. They also hire temporary employees and vendors. In this environment, a simple static password solution can become a liability.

Unlike a static password, a one-time password changes each time the user logs in. The passwords themselves are generated in one of two ways: either as time-synchronized or counter-synchronized. Both approaches typically require the user to carry a small hardware device (often on a key chain) that is synchronized with a server, and both typically use some algorithm to generate the password.

Time-synchronized OTPs are widely deployed but are subject to problems caused by clock skew. That is, if the authentication server and the user token don't keep the same time, then the expected OTP value won't be produced and the user authentication will fail. With time-synchronized OTPs, the user typically must enter the password within a certain period of time before it's considered expired and another one must be generated.

A counter-synchronized OTP solution synchronizes a counter between the client device and the server. The counter is advanced each time an OTP value is requested of the device. Just like with time-synchronized OTPs, when the user wants to log on, he enters the OTP that is currently displayed on the device.

Challenge-based OTPs are a special case and also often use a hardware device. However, the user must provide a known value, such as a personal identification number (PIN), to cause the OTP to be generated. This type of OTP is currently being rolled out in Europe to add authentication to credit and debit cards. The OTP solutions in use today are all built on some sort of cryptographic processing to generate the current password from a synchronization parameter (that is, the time or counter value), a secret key, and possibly a PIN.

For example, hash-based OTPs use cryptographic hashing algorithms to compute the password. As you know, a cryptographic hash is a one-way function that maps an arbitrary length message to a fixed-length digest. Thus, a hash-based OTP starts with the inputs (synchronization parameter, secret key, PIN), runs them through the one-way function, and produces the fixed-length password.

So, which method should you choose? I wanted to know how to do this too, so I created and tested a solution. Next, I'll show you how you can create counter-synchronized OTPs using IIS 7.0 and keyed-hash message authentication, as described in the RFC 2104 and RFC 4426 standards (RFC 2104, "HMAC: Keyed-Hashing for Message Authentication" at go.microsoft.com/fwlink/?LinkID=112151, and RFC 4226, "HOTP: An HMAC-Based One-Time Password Algorithm" at go.microsoft.com/fwlink/?LinkID=112153).

Because this is a test deployment, I will use a simple client application to create the OTPs. As mentioned previously, in the real world, you would want to integrate this with a tamper-resistant hardware device. I'll go over the basics of what that would require and point you to some resources to get you started.

A Complete OTP Solution

In order to build my OTP solution, I'll need to create a standards-based OTP authentication Web service that is backed by SQL Server® and integrated into ASP.NET. I'll create an OTP generator that is installed on each client computer, which users will run to generate a new OTP.

The user will type the OTP value when prompted by the Web browser and click Submit to authenticate. The OTP plug-in module is notified by IIS and subsequently calls the Web service to verify authentication attempts. The Web service looks up the user's key and counter value in the SQL Server table, verifies the OTP computation, and responds with authentication success or failure.

Figure 1 shows the architecture of the sample solution. Keep in mind that in a production environment, this architecture should be further hardened against denial of service (DoS) attacks by actions such as configuring a trust relationship between the client and server and throttling invalid logon attempts.

Figure 1 One-Time Password Solution Components
Figure 1 One-Time Password Solution Components

The sample code that accompanies this article, in the download section of the MSDN® Magazine Web site, consists of a Visual Studio® 2005 solution that includes a C++ DLL to generate the OTP (HmacOtpDll). Because this DLL is used by the OtpClient and the Web service, I put it into the system32 folder. (I use a post-build event to automatically copy it there.) The sample also includes a console application called OtpClient that generates the OTP value. OtpClient uses an XML file to store the secret code and counter. Every time the application is rebuilt, it copies the XML from the project root to the target directory, causing the counter to reset to 0.

An IIS module called IIS7Module provides OTP authentication services, and a Web service (called WebService) is used by the module to verify the OTP value. The Web service contains a SQL Server Express database, which is located in App_Data. Finally, I've included Web pages in the TestWebsite project, which I use for testing the solution.

Test OTP Generator Client

The test OTP generator client application is a self-contained tool that allows users to obtain OTP authentication values. It takes the place of a hardware device plus the challenge (such as a PIN request) that would typically be required in a real deployment. To compute OTPs, this client component uses a DLL that is shared with the authentication Web service. For the purpose of this sample application, the user will run the tool to create the next OTP and then manually type that value into a form in a Web browser. I'll do all of this using C# and some C. (I chose C for the low-level implementation of the OTP cryptography.)

You've seen how OTPs work at a user level, but how does this solution work at a functional level? This hash-based OTP solution takes two values as input: a key and a count. However, an OTP solution also has metadata relevant to the implementer, including the length of the key and the length of the expected OTP value that the user will have to type in when authenticating.

My sample produces OTPs that are six characters in length and can support up to eight characters. For the sake of simplicity, this implementation uses some fixed-length buffers that limit the key length to 64 bytes. However, assuming that the key is a high-quality, cryptographically random number, that's a huge key space. Such a key would not be the weak link in a production deployment. (A typical size for a random key today is 256 bits, or 32 bytes.)

The count value increments each time an authentication attempt is made by a given user (or technically, with a given key). The security of the OTP solution depends upon the count value never being reused; this is enforced by the OTP server. In this implementation, the count is a 64-bit unsigned integer. As I discussed earlier, another way to deploy this would be to use time synchronization with a server.

A keyed hash message authentication code (HMAC) is a key-based cryptographic hash. Or to put it another way, an HMAC takes an arbitrary message and a key and maps the message to a fixed-size digest value (20 bytes, for example), ensuring that only someone with the same key can produce the same digest value from the same message.

The first computational step for HMAC-OTP is to take the count value and encode it as the input message for an HMAC computation. In this implementation, the message is an 8-byte buffer set to the counter value. Figure 2 depicts this and the following two steps.The next computational step is to compute the HMAC of the above message with the user's key. Note that I addressed byte ordering in this implementation in order ensure that it is compliant with the RFCs.

Figure 2 One-Time Password Flow
Figure 2 One-Time Password Flow(Click the image for a smaller view)
Figure 2 One-Time Password Flow
Figure 2 One-Time Password Flow (Click the image for a larger view)

The 20-byte HMAC result is then turned into the OTP value, which is accomplished by making a decimal encoding of the HMAC result. There are two practical requirements here: first, I need to preserve as many bits of the HMAC computation as possible, up to the length of the OTP result (six numbers in this case), since losing bits exposes the computation to cryptographic attack. Second, I need to create an OTP that is compatible with as many varieties of input devices as possible. This compatibility requirement is why I implement decimal encoding. (This strong-authentication implementation is even compatible with a rotary phone!)

The Sample Web Site

My goal is to ensure that I have designed the site properly to show whether users have successfully or unsuccessfully logged in using OTPs. In order to facilitate testing, the OTP solution includes a sample Web site. The first page in the site is Default.htm, a landing page for authenticated users that demonstrates how even non-ASP.NET pages are protected by the OTP module. Test.aspx, a file that shows the currently authenticated user name, is also included, as is Test.aspx.cs, the file that shows how the Test.aspx page retrieves the authenticated user name from the System.Web.UI.Page.User property. Also included in the solution are a web.config file that includes a reference to OtpModule and a Visual Studio project solution with a reference to the IISModule.dll file.

The IIS HTTP OTP plug-in module is a component of the Web site. The module interfaces with IIS and redirects users to a Web form where they can enter their user names and OTPs. When a user submits a user name and OTP, the module validates the input and redirects the user to the appropriate success or failure page. The module also pairs the authentication status of the user with the user's session.

For maintainability and supportability, I wanted the module to be managed, so I wrote it in C#. The plug-in module is a client of the authentication Web service, which I discuss next.

The OTP module implements the IHttpModule interface. In fact, the module is remarkably simple and consists of only three public methods. The first is Init, as shown here:

public void Init(HttpApplication application)
{
  application.BeginRequest += new EventHandler(application_BeginRequest);
}

As you can see, the module uses this method to register its BeginRequest handler, application_BeginRequest. The purpose of the OTP BeginRequest handler is to ensure that all HTTP requests are made by an authenticated user. This is accomplished by using a few helper functions: one to determine whether the caller is authenticated and a few others to perform the authentication if the caller is not already authenticated. It is also worth mentioning here that intercepting the request during the BeginRequest event does not follow the standard authentication pattern used by ASP.NET applications. This approach is recommended, however, if you don't want other modules to see the request, even those that, by design, want to intercept requests prior to authentication.

The IsAuthenticated helper function determines if the request indicates an authenticated user. This is done by checking the application context for a properly encrypted authentication cookie using the classes HttpContext, HttpCookie, and Security.FormsAuthenticationTicket within the System.Web namespace. If the cookie is present and decrypts without error, the caller is deemed to be authenticated. Otherwise, there are two possible states: either this is an OTP authentication request that should be processed, or the Web client request is invalid (unauthenticated) and a login form is displayed.

The OTP module includes a built-in login form known as LoginPage.htm. It consists of five HTML elements: an initially empty error message field, a user name field, a password field, a submit button, and a hidden input field named hdLoginForm.

If the caller is already authenticated, the module takes no further action. Accordingly, the request continues to process. In this demonstration, the Default.htm page is loaded.

If the caller isn't authenticated, the IsAuthenticationPost helper function is called. It checks whether the request type is POST and if the request form has the hdLoginForm input field. The method returns true if both conditions are confirmed.

If the request is an authentication request, the TryAuthenticate helper function is called. The user name and OTP values are retrieved from the request context and passed to the authentication Web service's VerifyOtpCode method. If the verification is successful, a new encrypted authentication cookie is attached to the response. The response is then redirected to the default page (Default.htm). One way to extend this demo would be to save the original page requested by the user and redirect the user to that page following a successful authentication.

If the VerifyOtpCode Web service call fails, the request is redirected back to the login form, which now displays an error message, and the helper function ShowLoginForm is called. The helper function loads the login page from the resource section of the module, sets the error message string in the page (if appropriate), and sets the login page as the response to the current request. It then signals the request as complete (this happens regardless of the nature of the request).

The Authentication Web Service

The authentication Web service is responsible for performing the actual OTP authentication by determining whether the provided OTP value demonstrates that the named user has knowledge of the secret key.

The implementation of the OTP authentication Web service is quite simple since it reuses the low-level cryptographic library, which implements the OTP computation discussed in the earlier section on the test OTP generator client. This reuse comes in the form of a P/Invoke call to the GenerateOTP export of the native HmacOtpDll.dll.

The Web service exposes a single Web method, VerifyOtpCode, which returns true for a successful authentication. The first step in that method is to load the SQL Server row corresponding to the user name indicated in the authentication request. If SQL Server cannot find a matching row, the method returns false.

If the user name is found in the SQL Server database, the following data items are passed to the native GenerateOTP: the OTP value that is specified in the request, the secret key for the user (retrieved from SQL Server), and a counter value (also retrieved from SQL Server).

GenerateOTP is retried with sequential counter values until either it returns a matching OTP value or until at most 1,000 sequential counter values have been checked. That allows for the user to have accidentally advanced the client counter offline, although that's admittedly unlikely to have happened a thousand times since the last successful authentication!

Reducing this range would also reduce the already unlikely chance that an attacker might actually guess an OTP value that happens to be in the sequence, but it increases the possibility that the user may accidentally advance the client counter beyond the range attempted by the server. In the latter case, administrator intervention would be required in order for the user to be able to authenticate again.

If a matching OTP value is found within the counter range, the new counter value is written back to the database. While that unfortunately requires the authentication Web service to have write access to the database, it's also critical to the security of the OTP since, as I described previously, a counter value must never be reused. (For a one-time password to be unique, it really must only be used one time.)

A SQL Server database with a simple schema is used to store user names and their corresponding OTP secret key or seed value. You can also extend the schema to include logging information, such as the number and times of successful and unsuccessful logons using an OTP. The database consists of the Username, SecretCode, and Counter columns. For an explanation of each of these elements, see the Authentication Web service section. The copy of the database that accompanies the sample code has only one row for the user name "testuser."

The Complete Architecture

Figure 3 shows what the solution looks like when it's complete. As you can see in the figure, the user launches the client application, generates an OTP, then navigates to the authentication Web application, and pastes the OTP into the Web browser form. The form is generated by the OTP module when it detects that the request isn't authenticated. When the user clicks Submit, the Web browser form sends the request to the server where it's again intercepted by the OTP module. Next, the OTP module calls the OTP Web service to verify the user authentication data. Finally, if successful, the Web server calls the handler for the requested page, which could be of any type (HTML, ASP.NET, PHP, and so on).

Figure 3 Detailed OTP Solution Architecture
Figure 3 Detailed OTP Solution Architecture(Click the image for a smaller view)
Figure 3 Detailed OTP Solution Architecture
Figure 3 Detailed OTP Solution Architecture (Click the image for a larger view)

Running the Code Sample

To run the code sample, you'll need Windows Vista® or Windows Server® 2008 with IIS 7.0 (with the Application Server role enabled for Windows Server 2008), Visual Studio 2005, SQL Server 2005 or SQL Server Express, and the OTP sample code. For experimenting with the demo, I recommend using Windows Server 2008 and SQL Server Express, since that's what I used. In the following description, I will assume that the OTP sample code solution file is located at C:\Test\OTP\Otp.sln.

To prepare your environment, install the IIS module using either the IIS Manager or by editing the web.config file. Enable read and write access to C:\Test\OTP\webservice\app_data for either the IIS_IUSRS account or the account configured as the identity for a custom application pool. Then add a Web site for OtpTest and OtpService.

Registering the IIS module using the web.config file (this step has already been completed in the sample—see TestWebsite\Web.Config) requires the following configuration markup:

<system.webServer>
   modules>
    <add name="OtpModule" type="OtpModule" />
  </modules>
</system.webServer>

You should also add the module DLL to the bin folder or the global assembly cache (GAC). To register the OTP module DLL with the GAC, use a command such as the following:

gacutil.exe /i iis7module.dll

To use IIS Manager, open it and click on the name of your computer in the console tree. In the middle pane, double-click the Modules icon, and then click Add Managed Module in the actions pane. Next, select OtpModule from the dropdown listbox. If you put the DLL into the GAC and you're using IIS Manager to add the module, then you might need to restart IIS in order to refresh the modules list.

In order to properly increment the number-of-attempts value in SQL Server Express, the NETWORK SERVICE account requires read and write access to the C:\Test\Otp\WebService\App_Data directory. The required permissions for this object are read and execute, list folder contents, read, and write, as you see in Figure 4.

Figure 4 Required Permissions for the App_Data Directory
Figure 4 Required Permissions for the App_Data Directory

You must also set up a site in IIS for the test Web site. To do this, open the IIS Manager console, and in the ISS Manager console tree, expand the node with your computer's name, right-click Sites, and click Add Web Site. Use the following settings for your new site, and then click OK (Figure 5 shows these settings):

Figure 5 OtpTest Web Site Settings
Figure 5 OtpTest Web Site Settings(Click the image for a smaller view)
Figure 5 OtpTest Web Site Settings
Figure 5 OtpTest Web Site Settings (Click the image for a larger view)

  • Site name: OtpTest
  • Physical path: C:\test\Otp\TestWebsite
  • Port: 8000

You must also create a site for the Web service. Use the following settings for this site: site name should be OtpService, physical path should be C:\Test\Otp\WebService, and the port should be 8080.

Next, open the Otp.sln solution in Visual Studio, expand IIS7Module in the solution explorer, expand Web References, right-click OtpService, click Properties, and confirm that the Web Reference URL is set to http://localhost:8080/service.asmx. In the Build menu, click Build Solution and confirm that no build errors have occurred.

Now, you should test to ensure that the OTP module has been properly registered and loads correctly. To do so, navigate to http://localhost:8000 and verify that you see a page that looks like the logon page in Figure 6. To test the implementation, type in the test user name of testuser.

Figure 6 First Page of Test Web Site
Figure 6 First Page of Test Web Site

To get the value for the OTP Code textbox, open a Command Prompt Window as an administrator and navigate to the build directory of the OTP client program (C:\Test\Otp\otpclient\bin\debug). Run OtpClient.exe to get the next OTP value. You should get something like Figure 7.

Figure 7 Running OtpClient.exe to Get OTP Value
Figure 7 Running OtpClient.exe to Get OTP Value(Click the image for a smaller view)
Figure 7 Running OtpClient.exe to Get OTP Value
Figure 7 Running OtpClient.exe to Get OTP Value (Click the image for a larger view)

Type the OTP value into the OTP Code textbox, and then click Submit. Figure 8 shows the sample OTP code in the page. If the authentication is successful, you should be directed to the Default.htm page.

Figure 8 Test Page with Sample OTP Value
Figure 8 Test Page with Sample OTP Value

Click the Text.aspx link on the default page to go to a demo page, which displays the name of the currently authenticated user (if any). If the authentication credentials were incorrect, you should see the page shown in Figure 9.

Figure 9 Authentication was Unsuccessful
Figure 9 Authentication was Unsuccessful

Deployment Considerations

If you were to deploy this sample solution in a real-world environment, there are a number of items that you would want to consider. Real deployments store the user key/seed in a tamper-resistant device, such as hardware tokens. I would recommend that you modify the solution to lock the account in response to too many consecutive failed authentication attempts. You can do this by adding SQL Server columns and a feature to the Web service.

If interoperability is a concern, I recommend that you perform interoperability testing on the low-level HMAC code. Also realize that the MD5 hash algorithm is no longer considered secure. I used it in this sample solution in order to run some basic Known-Vector Tests with what's available in the RFC documents. But a deployable solution should use one of the SHA-2 algorithms to create the hashes.

When developing more user-friendly authentication pages, be sure not to distinguish between bad user name and bad password. Otherwise, you're letting attackers learn valid user names. You'll also need some sort of provisioning solution for adding and removing users or perhaps synchronizing the authentication database with another repository such as Active Directory®.

If you are just targeting ASP.NET, the module could have been implemented as a standard ASP.NET HTTP module. However, such a solution would only secure .aspx files. Note that the IHttpModule interface is the same whether you are configuring it with the ASP.NET runtime or IIS 7.0, so if an original solution was an ASP.NET-specific module, then enabling it to support all file types is a simple change in configuration.

I used Visual Studio for editing the database, a cool feature of the IDE of which I had not previously been aware. To edit the database, go to the Solution Explorer tree view, expand WebService, expand App_Data, and select the Open option from the context menu. In the Server Explorer panel that appears, expand Data Connections, expand otp.mdf, expand Tables, right-click Users, and then click Show Table Data. Figure 10 shows the table data in the Visual Studio IDE. Just be sure not to leave the database open in the IDE when testing; the Web service won't be able to open it and, thus, the authentication will fail.

Figure 10 One-Time Password Solution Components
Figure 10 One-Time Password Solution Components

Give It a Try

One-time passwords are an excellent alternative to standard passwords. Because users do silly things with their passwords, such as writing them down and falling victim to phishing attacks, you must strengthen the authentication process. Using the IIS 7.0 plug-in module model, you can create a standards-based OTP solution and more robust, production-ready OTP solutions using hardware tokens and challenges.

You should follow the sample here and give it a try for yourself. Since many employers, as well as consumer-facing interfaces such as online banking, are adapting OTP, you should familiarize yourself with the technology.

For further reading, see Mike Volodarsky's January 2008 MSDN Magazine article called "IIS 7.0: Enhance Your Apps with the Integrated ASP.NET Pipeline" at msdn.microsoft.com/msdnmag/issues/08/01/PHPandIIS7. This article will provide some helpful background information.

Dan Griffin is a software security consultant in Seattle, Wash. He previously spent seven years at Microsoft on the Windows Security development team. Dan can be contacted at www.jwsecure.com.


Alphabet Soup: A Survey of .NET Languages And Paradigms

Alphabet Soup

A Survey of .NET Languages And Paradigms

Joel Pobar
This article discusses:
  • Object-oriented programming
  • Functional programming
  • Dynamic programming
  • New paradigms for .NET languages
This article uses the following technologies:
C#, C++, F#, IronPython, IronRuby, Visual Basic
Contents
Object Orientation
Functional Programming
Dynamic Languages
Safe and Useful
LINQ
Inline XML in Visual Basic 9.0
More to Come

The Windows operating system has never been a better platform for programmers. Hundreds of languages target Windows® either directly through the Win32® API or through the CLR, and more are being built as you read.

One of the goals of the CLR was to enable a thousand flowers to bloom in one compatible ecosystem, where languages and APIs seamlessly integrate within the same runtime. It thus far has succeeded fantastically—new languages are popping up everywhere. Dynamic languages like the Microsoft implementations of Ruby (IronRuby), Python (IronPython), and PHP (Phalanger) are now first-class citizens among Microsoft® .NET Framework languages. A functional language called F# was recently inducted, too. And while you've probably heard a lot of talk about all of these new languages and language paradigms, you may be wondering what it all means.

A tour of these language paradigms and a look at some of the important implementation aspects will answer your questions and will also help to illustrate how some of these new languages and old paradigms are influencing the design and implementation of future iterations of the C# and Visual Basic® languages.

To understand the kinds of changes the new designs represent, you need to understand the differences between traditional languages (such as C# and Visual Basic) and new languages (such as F#, IronPython, and IronRuby). There are three main themes in this space: object-oriented programming (a model that C# and Visual Basic both leverage), functional programming (F#), and dynamic programming (IronPython and IronRuby). Let's take a look at these paradigms and explore their distinguishing features.

Object Orientation

Object orientation (OO) is the paradigm with which you're probably most familiar. It enables you to describe the world in terms of objects and the contracts that bind the interactions between them. OO leverages features such as type contracts, polymorphism, and granular visibility to provide excellent reuse and encapsulation.

Typically, object-oriented languages employ a static type system and, as a result, are referred to as statically typed languages. This means that all types created and used by a program are checked at compile time; this prevents you from calling a method, Moo, on an object of type Duck if the method doesn't exist. Broken and misused contracts between types will be detected by the compiler before the code runs, which in theory leads to fewer runtime errors.

OO has its drawbacks, however. Type safety can lead programmers to rely on the compiler to catch errors instead of creating the necessary test infrastructure. OO also persuades programmers to define their contracts up front, which tends to be the antithesis of rapid prototyping and multi-person programming environments. In large software projects, contracts between components (usually owned by separate teams) tend to evolve throughout the development cycle, requiring the contract consumers to constantly update their code. OO languages can be complicated and verbose as a result of these problems.

Functional Programming

Functional programming treats program computation as the evaluation of mathematical functions. Adding two numbers together is a function. Given two input values, say 5 and 10, the value 15 is output. To solve a problem, a functional programmer breaks the problem down into smaller chunks that can be represented by functions and then combines these functions to produce the intended output.

Functional programming generally avoids state (things like variables and objects) and the mutation of state. This is practically the opposite of OO, whose main purpose is to create and manipulate state (objects). By avoiding state, functional programs tend to be more precise, more accurate, and provable. This is due to the lack of unpredicted side effects that can occur when the internal state of a program (such as a variable) changes in between operations, thus producing an unexpected result in one or more of the operations. Object-oriented and dynamic languages rely on the programmer to encapsulate and protect state to reduce the unpredicted side effects that lead inevitably to more errors.

Functional languages can be pure, meaning they have no side effects or state. However, most popular functional languages have state manipulation features to facilitate interoperability with external APIs, the operating system, and other languages. They are also mindful of the fact that some programs must represent problems with some amount of state information.

There are a number of useful features that are common to functional languages. A personal favorite of mine is the higher-order function, which is a function that can take another function as an argument and can return a function as a result, leading to extremely powerful code reuse. And while most languages have this mechanism built-in, functional languages tend to elevate it as a first-class feature.

A nice example of the power of higher-order functions is the classic Map API, which executes (or maps) a function over every element in an array or list of data. To start with, let's write this in JavaScript, a commonly used Web scripting language. Given this array

var data = [1, 2, 3, 4, 5];

let's write a function that will execute code over every element in the array, like this:

function map (func, array)
{
    var returnData = new Array(array.length);
    for (i = 0; i< array.length; i++)
    {
        returnData[i] = func(array[i]);
    }

    return returnData;
}

Here's a function that increments a number by one:

function increment(element)
{
    return element++;
}

And now let's use it:

print (map (incremenent, data));

output: [2,3,4,5,6]

This example of the Map API is simplistic, but it shows the power of higher-order functions. Map takes a function and an array and executes that function over each array element, returning the result in a new array.

Given that there's no mutated state (you create a new array for the return result), and that no function result depends on a previous result, you can now start to think about extending the map example across multiple processor cores (by splitting the array in half and joining the two resulting arrays), and even across multiple machines (split the serialized data across n machines, pass the code function to the machine, execute and join the serialized results back at a single host) without having to worry about concurrency issues like state management. Extremely powerful stuff for something so simple!

Initially developed by Don Syme in Microsoft Research, the F# programming language is the Microsoft answer to functional programming on the .NET Framework. When released, it will be a fully supported language within Visual Studio®. The F# and the Visual Studio integration package are both available for download at go.microsoft.com/fwlink/?LinkId=112376. In addition, Foundations of F# by Robert Pickering is a great introduction to the language.

OK, let's use F# to explore some of the functional language world. Take the Map example again, this time in F#:

// increment function
let increment x = x + 1

// data
let data = [1 .. 5]

// map (func, myList)
let map func myList =
  { for x in myList -> func x }

print_any (map increment data)

Yes, it's beautifully succinct. The very first line defines a simple function, increment, which takes an argument, x, which evaluates to x + 1. I then define data as an immutable list that contains the integers 1 through 5. The map function takes a function and a list as arguments and uses a little sprinkling of traditional programming to loop through the list and execute the func function argument, passing to it the current element in the list. The print_any function then executes the map function using the increment function and the data list as arguments, and it then prints the result to the screen.

Where are the types? Nowhere in this example. Under the hood, the variable, data, is actually typed as a list of System.Int32, but I didn't have to tell the compiler that—it simply infers it using type inference. The compiler works a little harder so that you can type a little less. Sounds good to me. You could easily rewrite some of the previous code in just one line, using a feature called list comprehension:

print_any { for x in 1..5 -> x + 1 }

Another cool F# feature is pattern matching:

let booleanToString x =
   match x with false -> "False" | _ -> "True"

This function takes a Boolean type and matches it against false or everything else, returning the appropriate string. What happens if I pass a string to the booleanToString function? Again, the compiler has done the heavy lifting and defined the type of the x argument as type bool. It infers this from the way x is used in the function (it's only matched against a bool type in this case).

Pattern matching can also be used to build powerful function dispatch mechanisms, easily reproducing virtual method dispatch in the OO world and more. Virtual methods really start to strain when you need to vary behavior based on the receiver (the object on which the virtual method is invoked) and the methods parameters. The Visitor Pattern was designed to help out in this situation but is basically unnecessary in F#—pattern matching subsumes this.

User-defined types are supported and usually exposed via records (similar to classes in the OO world) or tuples (usually a type that is an ordered sequence). Here is a user-defined player and soccer team record:

type player =
  { firstName : string;
    lastName : string;
  }

type soccerTeam =
  { name : string;
    members : player list;
    location : string;
  }

Lazy evaluation is another common functional language feature whose power is derived from the theory that there are no explicit side effects in functional programming. Lazy evaluation relies on both the compiler and programmer's ability to choose the evaluation order of expressions, which allows the computation to be delayed until needed. Both the compiler and programmer can use lazy evaluation techniques as a very precise performance optimization that works by avoiding unnecessary calculations. It's also useful as a technique to deal with computation over infinite (or extremely large) datasets. In fact, the Applied Games research group at Microsoft Research uses this technique in F# to parse terabytes of Xbox LIVE® log data.

Here's lazy evaluation in F#:

    let lazyTwoTimesTwo = lazy (2 * 2)

    let actualValue = Lazy.force lazyTwoTimesTwo

When this code executes, the lazyTwoTimesTwo simply sits in the runtime as a lightweight pointer to a function that executes 2 * 2. Only when you actually force the function to execute do you get a result.

While the functional programming model can be hard to get your head around (expect around 30-40 hours of ramp up), it rewards you as a programmer graciously. You can solve problems with less code and fewer errors. The paradigm keeps your code safe by minimizing unintended side effects and keeps it running fast by performing aggressive optimizations. Besides, simplicity tends to scale well—you just need to peek at the Map code and think about how that code could easily be distributed across thousands of machines to get a feeling for that.

So with that, you've seen some of the nicer features of functional programming such as type inference, higher-order functions, pattern matching, and user-defined types. Now, let's take a tour of dynamic languages.

Dynamic Languages

Dynamic programming became popular back in the mid 1990s when Web-based applications became all the rage. Perl and PHP were two dynamic languages used heavily on the server to add interactive, dynamic elements to the Web sites they drove. Since then, dynamic languages have gained even more traction, providing a technical answer to new software development methodologies like agile.

A dynamic programming language differs from a static language (C# and Visual Basic .NET are both static object-oriented languages) in the way it compiles and executes program code. With dynamic languages, code compilation is typically delayed until run time, when the program is actually running. In other cases, the code is simply interpreted. These delays in compilation enable the program code to include and execute behaviors such as extending objects on the fly and allowing the programmer to bend the type system at will.

In fact, the just-in-time (JIT) compilation and execution technique that delays the process until the last second is what gives dynamic languages their power and rich feature set. As a result, dynamic languages are often referred to as late-bound because all action binding (such as calling a method or getting a property) is done at run time instead of at compile time. To illustrate some of the features of dynamic languages, let's take a closer look at IronPython and IronRuby.

IronPython is the Microsoft implementation of the Python programming language for the .NET Framework. It is the brainchild of Jim Hugunin, who started working on the original prototype in his garage. Six months later, he joined Microsoft and built the IronPython compiler. You can download the full source and installer for IronPython at go.microsoft.com/fwlink/?LinkId=112377.

IronRuby is the name given to the Microsoft implementation of the Ruby programming language. John Lam is the active Ruby hacker who wrote the original Ruby-to-CLR bridge, called RubyCLR. He later joined Microsoft to work on the IronRuby team. You can download the full IronRuby source at ironruby.net.

Now let's start IronPython and take a peek at a nice feature called a Read Eval Print Loop (REPL). A REPL loop is designed to take code input one line at a time and execute. It really shows off the late-bound nature of dynamic languages like Python. Figure 1 shows the IronPython command-line REPL loop.

Figure 1 IronPython REPL Loop
Figure 1 IronPython REPL Loop(Click the image for a smaller view)
Figure 1 IronPython REPL Loop
Figure 1 IronPython REPL Loop (Click the image for a larger view)

The REPL loop accepts Python code as input and executes it immediately. In this case, I ask it to print "Hello, World!" to the screen. As soon as the programmer hits the enter key, the IronPython compiler compiles the statement into intermediate language (IL) and executes it—all at run time.

The REPL loop also adheres to the full scoping rules of the language—in this case, I assigned the expression 1 + 2 to the variable "a" and was able to reference that variable in the very next line. The compilation of REPL input happens in-process. There's no need to emit a .dll or .exe file; the IronPython compiler simply leverages an in-memory code generation feature of the CLR called lightweight code generation (LCG).

The type systems of dynamic languages are very flexible and relaxed. Object-oriented concepts such as classes and interfaces are usually supported, but with a twist: strict contract enforcement such as that found in static languages isn't generally seen. A classic example of this more relaxed type system is found in the concept of duck typing, a commonly supported dynamic language type system feature. With duck typing, if a type looks like a duck and quacks like a duck, the compiler assumes it must be a duck. Figure 2 shows this concept in action using Ruby—you can run this code in IronRuby if you like.

Figure 2 Duck Typing in Ruby (Close Figure 2)
class Duck
  def Quack()
     puts "Quack!"
  end
end

class Cow
  def Quack()
    puts "Cow's dont quack, they Mooo!"
  end
end

def QuackQuack(duck)
  duck.Quack()
end

animal = Duck.new()
QuackQuack(animal)
animal = Cow.new()
QuackQuack(animal)

Figure 2 shows two class definitions (Duck and Cow), a method called QuackQuack, and code to instantiate those objects, passing them to the QuackQuack method. The code once loaded and run outputs the following:

Quack!
Cow's don't quack, they Mooo!

The QuackQuack method invokes the method Quack on the method argument, not caring what the type of that argument is—it only cares whether the argument has the method Quack. The lookup and invocation of the Quack method occurs on the argument at run time instead of compile time, allowing the language to make sure the argument looks like a duck before it makes it quack.

The simile to duck typing in the static typing world is the interface. If a type implements an interface in the static world, the static type system enforces its adherence to the full and complete interface structure, so invocation of an interface method on any implementing type is guaranteed. Duck typing only concerns itself with part of the type's structure—the method name being addressed. It achieves this through late binding—the compiler simply emits code that will perform a method lookup on the object (usually using the string-based name of the method) and then a call to that method if the lookup was successful.

Language features aren't the only cool features in dynamic languages such as IronPython and IronRuby. Dynamic languages even allow you to host APIs, which means you can host the language, compiler, and even the REPL interactive feature in your own applications. Say you're building a large, complicated application, maybe a photo manipulation tool, and you want to expose an automation or scripting facility to your users. You could host IronPython or IronRuby and allow users to create Python or Ruby scripts that manipulate your application programmatically. Simply expose your application's internal objects and APIs to the language hosting API, create a text window to host the REPL loop, and you'll be enabling scripting in your application.

Having features like command-line-driven REPL loops has a really nice productivity side effect: you can easily cook up a working prototype without the save-compile-run cycle. Modifying and extending types on the fly is encouraged, allowing you to get the solution from your head to code quickly without worrying about what interface or class definition contracts look like. Once you're in a good place, you can move from your dynamic language prototype to the stricter and safer world of static languages like C#.

Flexible type systems are also really good at consuming unstructured data (like HTML on the Web) and can easily be used as change-resistant glue to strict interfaces that are likely to change versions over time. With new application paradigms like Web-based computing, dynamic languages provide a great solution for dealing with uncertainty.

Hosting APIs give your applications an extensibility point that's second to none; you can offer an application experience similar to those of Microsoft Excel® and VBScript Macros.

Safe and Useful

Safety and usefulness are also important considerations in language selection. The usefulness of a language is measured by its ease of use, productivity benefits, and so forth. The safety of a language is measured by its type safety (whether it allows invalid casts), programmatic barriers (whether you can walk over an array boundary and trample on other memory in the stack), security features, and more. Sometimes you trade usefulness for safety and vice versa.

Interestingly, you find C#, C++, Visual Basic .NET, Python, and the like in the useful but less safe category. While it's true that managed code languages like C# are safer than traditional native code, you can still get in trouble. Because these languages focus on state manipulation, they generally have unknown side effects—and when working in a concurrent world, this becomes a problem.

A C#/Visual Basic .NET multithreaded program that runs fine on a single-core machine might crash on a multicore due to a race condition that exposes a side effect that the programmer didn't know about. Also, while these languages have a good general level of type safety, they still can't prevent an unknowing programmer from casting an object to void*.

Haskell (a pure functional language used in academia) is in the safe-but-not-as-useful category. Because Haskell has no side effects (code blocks are functions and variables never change), and because it forces the programmer to define his types and intentions up front, it is considered a safe language. Unfortunately, Haskell is incredibly hard to get your head around, even for ninja programmers, and it's generally hard to read Haskell code you haven't written yourself.

In my opinion, F# is more useful than Haskell with its .NET Framework integration, opt-in procedural syntax, and features to deal with Objects that come from the object-oriented world. Clearly, the next-generation languages would do well to borrow the best features of all the languages I've discussed. Who wouldn't like a language that could move from a rigid, enforceable static type system to more dynamic and flexible types on the fly, had the power of functional programming, and avoided concurrency problems?

Thankfully, it's easy to go from C# to F# to IronPython and back again thanks to the common type system (CTS). And the designers of C# and Visual Basic .NET are borrowing from the best features of the dynamic and functional paradigms and integrating them straight in to the languages as first-class features. So now it's time to take a look at a few of these features and see how they relate back to what you've learned about the various languages and their paradigms.

LINQ

LINQ is a new feature in the .NET Framework that can be used directly as an API from any language in .NET or actually exposed as a language feature through the set of LINQ language extensions. LINQ can be used to query data in a storage-agnostic manner by exposing a set of query operators similar to that of the T-SQL language. Visual C#® 2008 and Visual Basic 2008 in the .NET Framework 3.5 both support LINQ as a first-class citizen.

Your classic LINQ query using Visual C# 2008 looks like this:

List<Customer> customers = new List<Customer>();
// ... add some customers
var q = from c in customers
        where c.Country == "Australia"
        select c;

This code walks the list of customers, looking for Customer objects that contain a Country property that equals "Australia", and then adds them to a new list. The first thing you'll notice is the "var" keyword, a new keyword added to Visual C# 2008 that tells the compiler to perform type inference. Then you see the C# LINQ extensions in action, with the from, where, and select keywords. These form a comprehension of the list of customers (list comprehension), just as you saw in the F# language and as is commonly featured in dynamic languages.

Because you create a new list on execution of this LINQ query, there are effectively no side effects—the original list of customers is untouched. Side effects are the thorn in the side of every concurrent program, but this style of programming gives you safety from side effects while maintaining readability and overall usefulness. And given there are no side effects, you can scale out this query as wide as you like.

In fact, that's the very purpose of Parallel LINQ (PLINQ), an extension to LINQ that is available as a community technology preview through the Parallel FX library (go.microsoft.com/fwlink/?LinkId=112368)—parallel execution of LINQ queries across multiple CPU cores—an almost linear increase in speed with no added programming cost! This looks a lot like the Map API example you saw in the functional programming section. Overall, the integration of LINQ into C# and Visual Basic has added a functional flavor, with a touch of dynamism, making these languages much more expressive, and a lot less verbose.

Inline XML in Visual Basic 9.0

Some call inline XML the best feature to be added to Visual Basic—I tend to agree. And while it doesn't look like something out of the functional or dynamic paradigms, its implementation comes directly from the dynamic language world. Figure 3 shows XML inline in Visual Basic.

Figure 3 Inline XML in Action (Close Figure 3)
Dim rss = From cust In customers _
          Where cust.Country = "USA" _
          Select <item>
                     <title><%= cust.Name %></title>
                     <link><%= cust.WebSite %></link>
                     <pubDate><%= cust.BirthDate %></pubDate>
                 </item>

Dim rssFeed = <?xml version="1.0" encoding="utf-8"?>
    <rss version="2.0"><channel></channel></rss>

rssFeed...<channel>(0).ReplaceAll(rss)
rssFeed.Save("customers.xml")

This code uses the Visual Basic LINQ to XML implementation to query a list of customers for those who live in the USA and returns a list of XML elements that form the basis of an RSS Feed. The code then defines the RSS Feed schema and uses the rather curious Visual Basic "..." operator to traverse the XML document to find the "<channel>" element. Then the code calls ReplaceAll on that <channel> element to include all of the XML elements created above. Phew.

The ... operator plus the use of an inline XML element is what is interesting here. You see, Visual Basic doesn't know about the XML <channel> element—in fact, you can change that to <foo> and it will compile fine—so if it doesn't know about it, it can't possibly be statically checking that call site for correctness. Instead, it's compiling code that will do a dynamic check at run time—the kind of check you saw in the dynamic paradigm. It's leveraging the dynamic language concept of late binding.

For those who eat IL for breakfast, let's crack open the assembly and take a look at the code for the "...<channel>" statement:

ldloc.1
ldstr      "channel"
ldstr      ""
call       class [System.Xml.Linq]System.Xml.Linq.XName
     [System.Xml.Linq]System.Xml.Linq.XName::Get(string,
                                             string)
callvirt   instance class Generic.IEnumerable'1<XElement>
           XContainer::Descendants(System.Xml.Linq.XName)

Ldloc.1 loads the rssFeed variable, loads the channel string, then calls the XName.Get method, which returns an XName. This XName result gets passed to the Descendants method, which then returns all the descendents of the XML Document that match the <channel> name. This mechanism is classic, late-bound dynamic dispatch, borrowed straight from the dynamic languages playbook.

But wait, there's more. If you drop the RSS schema definition into Visual Studio and add an "Imports" reference to the schema, the Visual Basic .NET compiler will pick that up, parse it, and then enable full IntelliSense® over any XML that adheres to the schema. So if you dropped in the RSS XML Schema, the "...<channel>" statement would immediately show an IntelliSense window with the possible descendent options available as per the schema definition. Very cool!

More to Come

I've only touched on a few examples of how languages like C# and Visual Basic are combining the best elements from other paradigms and languages. Right now, the language design teams are already adding new flavor to their languages to cope with tomorrow's programming challenges.

While I have looked at a few high-level categories of programming paradigms, there are others. Declarative programming (the art of describing what something is) using XML as the descriptor has had a huge impact on development through frameworks such as Windows Presentation Foundation and Windows Workflow Foundation. They're also excellent at describing and manipulating data. And with the ever-growing data assets that organizations have created over the years, I suspect that declarative languages will be a future driving force in trying to programmatically harness those assets. There's also logic programming (leveraging logic for computer programming), which I haven't touched on but is used widely in the artificial intelligence and data-mining spaces with languages like Prolog.

As new features are added from all these languages and paradigms, you benefit. Your favorite language gets better through cross-pollination, and you get more flexibility and productivity. And if you think there are some cool features in another language that you want to use, no problem; your favorite runtime and framework supports that interoperability scenario, too. It's great to be a .NET programmer.

Joel Pobar is a former Program Manager on the CLR team at Microsoft. He now hangs out at the Gold Coast in Australia, hacking away on compilers, languages, and other fun stuff. Check out his latest .NET ramblings at callvirt.net/blog.


Editor's Note: I Want to Believe

Editor's Note
I Want to Believe
Howard Dierking

When I was in college, one of my favorite television shows was "The X-Files." One of the show's visual landmarks was a poster hanging behind the desk of agent Fox Mulder. It featured a rendering of an alien spacecraft accompanied by the text "I Want to Believe." This simple declaration served to define the character of the man who hung the poster and was a major theme of the show.

In many ways, this statement describes my feelings about Office Business Applications (OBA). I absolutely buy into the vision. It makes too much sense to not buy in. I mean, why would you not want your application to seamlessly integrate into a user experience that is already familiar? Why spend inordinate amounts of energy configuring and extending a grid control so that it can still have just a fraction of the power of Excel? And yet so many individuals in today's developer community completely ignore Office as a viable application platform. As always, I have a few theories as to why.

First, there's the fact that Office client applications don't have the luxury of an API written from the ground up in the Microsoft .NET Framework. Many, if not all, of the client application APIs began as procedural C programs. Over the years, these APIs have been encapsulated by multiple wrapping layers of technology. The latest version of Visual Studio Tools for Office (VSTO) takes great strides toward aligning the Office development paradigm with how programmers using .NET are used to thinking about application development. Even so, there are still aspects that remind you of exactly what lurks just beneath the latest friendly, top-level wrapper.

Second, the OBA development world still needs some best practices that relate to process maturity. I'm referring to practices like testing, deploying, and lifecycle management. With specific regard to testing, I have not yet found a good way to unit test an Office customization. That said, with regard to deployment, I am elated that support for click-once deployment of customizations was added to the latest release of VSTO. Again, a huge step forward.

Finally, there are a lot of very different technologies that fall under the OBA umbrella—client technologies such as VSTO, server technologies such as SharePoint, and peer-to-peer technologies such as Groove. As a developer, it's hard to know where to jump in. If I want to build collaboration into my system, do I build on top of SharePoint or Groove? And many of these technologies challenge what we even consider to be development. If you can create a feature-rich SharePoint site without even opening Visual Studio, does that mean that you're not developing?

The release of the 2007 Office system offers huge steps forward in the capabilities offered to OBA developers, from the client to the server. And we will see the development experience continue to mature, best practices emerge, and product integration continue to become more seamless. Like I said, I want to believe. It just makes too much sense. After learning VSTO, it's amazing to look back and see how many UIs I have written needlessly in the past.

On another note, I want to tell you about another recent and very exciting event that has transpired recently. MSDN Magazine is now a part of the Microsoft Developer Division. This is absolutely fantastic news, as it means we will now be even more closely aligned with the product teams. To celebrate this new alignment, we're now adding special "Insights" sidebars to certain articles. These sidebars give the various product teams a voice to provide interesting behind-the-scenes explanations. Go to John Papa's Data Points column in this issue for the very first "Insights" sidebar. We hope you enjoy this new feature!

Thanks to the following Microsoft technical experts for their help with this issue: Sarita Bafna, John Boylan, Bill Chiles, Barak Cohen, Ale Contenti, Bill Dunlap, John Durant, Bill Essary, Elisa Flasko, Matt Gibbs, Gordon Hogenson, Stephan T. Lavavej, Bertrand Le Roy, Bradley Millington, Laurence Moroney, Keith Pijanowski, Dave Reed, Mark Rideout, Paul Stubbs, Diego Vega, Mike Volodarsky, and Doug Walter.


Toolbox: Database Audit Logs, Joel on Software, Code Handouts, and More

Toolbox
Database Audit Logs, Joel on Software, Code Handouts, and More
Scott Mitchell

Contents
Audit Database Changes
Blogs of Note
Print and Export Your Code
The Bookshelf

Audit Database Changes

Database systems like Microsoft® SQL Server® have long supported triggers—developer-created scripts that automatically execute after or in place of INSERT, UPDATE, and DELETE statements. Because triggers can access the data modified by INSERT, UPDATE, or DELETE, they are an excellent tool for recording data changes to an audit log. Audit logs are essential for applications working with sensitive or critical data, as they keep track of the date and time a data change was made, along with the user who made it. In fact, many financial and healthcare applications must by law employ audit logs. Plus, audit logs are helpful for debugging because they provide a window into the database state at the time that an exception occurs.

While you can certainly design, build, and maintain a custom audit log, doing so requires a substantial amount of time and effort. It is usually more cost-effective to use a third-party product, such as OmniAudit version 1.11, by Krell Software.

OmniAudit makes implementing an audit log as simple as point and click. From the manager interface, select the database to work with and then choose the tables and columns to audit. For each audited column, you can specify that the column value be recorded in the audit log either when that value changes or when any other column value in the associated row changes. What's more, you can configure OmniAudit to log data from related tables as well.

After configuring the audit log, OmniAudit builds the creation scripts for the necessary log tables and triggers. Once these tables and triggers have been created on the database, any modifications to the specified tables or columns are automatically and transparently recorded to the audit log! Each audit log entry includes all of the essential information, including the date and time of the modification along with the names of the user, workstation, and application that made the modification.

Because the audit log can quickly grow to an unwieldy size when there are frequent changes, OmniAudit can be configured to periodically purge audit log entries older than a certain number of days or weeks. To view the log, use the OmniAudit Log Viewer, which provides a variety of filtering mechanisms. You can narrow the displayed log entries by a particular table or column, a specific user, a date range, or the type of operation performed. The filtering choices are a bit limiting—you can't filter on more than one table, column, or user, for instance—but they will probably be quite sufficient for most users' needs. Best of all, the Log Viewer requires no knowledge of SQL, so managers and other non-technical resources can review the audit log.

Price: $479 for a single-user license.

krell-software.com/omniaudit

Viewing a Log in OmniAudit Log Viewer
Viewing a Log in OmniAudit Log Viewer (Click the image for a smaller view)
Viewing a Log in OmniAudit Log Viewer
Viewing a Log in OmniAudit Log Viewer (Click the image for a larger view)

Blogs of Note

Why do so many software projects fail? What 12 factors can you use to quickly rate the likelihood of success for a team of developers? What are the best ways to educate and train people in the art of software development, and how does that compare to what is being taught in university programs today? These are the sorts of topics addressed by Joel Spolsky on his blog, Joel on Software. Joel has programmed, architected, and managed in a variety of settings; he began his software career as a program manager for the Excel® team at Microsoft. In 2000, he cofounded FogCreek Software, which makes FogBugz, a popular project management application.

As Joel explains it, "Management's first responsibility is to create an abstraction layer for developers: to create the infrastructure so that programmers really just have to program." Developers work best when they can focus on programming and are given a quiet, well-lit and well-equipped work environment. Management's task is to find smart and motivated developers, keep distractions to a minimum, and place the team in an environment to succeed.

Of course, a team of happy and focused programmers does not guarantee a successful software project, but it is an important ingredient. There are other operational and personnel factors that contribute to the ultimate success or failure of a project, which Joel has distilled to 12 checkpoints. How does your company's policies and practices compare to those outlined in The Joel Test: 12 Steps to Better Code?

While much of Joel on Software is a high-level look at the software industry at large, there is plenty of content for developers, too. Joel has lots to say about user interface design, testing, and deployment. And in addition to the regularly updated blog, which is translated into more than 30 languages, there are discussion boards on topics ranging from the business of software to Microsoft .NET Framework development.

JoelOnSoftware.com

Joel Spolsky's Blog
Joel Spolsky's Blog (Click the image for a smaller view)
Joel Spolsky's Blog
Joel Spolsky's Blog (Click the image for a larger view)

Print and Export Your Code

I recently gave a talk to a user group on an open-source ASP.NET application and wanted to provide the application's source code as a handout to the attendees. Visual Studio® makes it easy enough to print the contents of a single file—just open the file and select the Print option from the File menu. However, printing one file at a time is a tedious process when there are dozens of files to print. Furthermore, Visual Studio offers few print-related options.

A better option is to use StarPrint's VSNETcodePrint. VSNETcodePrint is a Visual Studio add-in that can print Visual Basic®, C#, J#, and ASP.NET code and markup. Once installed, it is accessible through a menu item in the Visual Studio IDE. In addition to printing, VSNETcodePrint can export code to an RTF or PDF document, or to a Web page.

With the click of a button you can print or export the entire contents of a solution. There's no need to open each file. Similarly, one click is all it takes to print or export the contents of an entire project, namespace, class, or selected text.

VSNETCodePrint offers a variety of print-related options, including: portrait and landscape orientations, multiple columns, custom page headers and footers, line numbering, syntax formatting, connection lines, and line spacing.

Connection lines clearly illustrate what lines of code belong to the same statement block. And line numbering can be configured to reset at certain points, such as the start of each new procedure or the start of each new page. There are also options to add a cover sheet, to watermark the printed or exported content, and to include screen shots or other image files in the printout.

In addition to VSNETCodePrint, StarPrint also offers similar products for printing and exporting database objects from Microsoft SQL Server.

Price: $59.

starprint2000.com

The Bookshelf

The Microsoft ASP.NET AJAX framework simplifies the process of building AJAX-enabled Web applications by providing a rich and powerful, high-level API over the multitude of low-level interactions that must seamlessly occur between the client and server. Any book about the ASP.NET AJAX framework must balance the low- and high-level details. The space devoted to client-side issues like the XMLHttpRequest object and JSON must be weighed against coverage of the framework's Web controls and client- and server-side APIs. One book that does an outstanding job in presenting the framework in a balanced and accessible context is ASP.NET AJAX In Action by Alessandro Gallo, David Barkol, and Rama Krishna Vavilala.

ASP.NET AJAX In Action starts from the low-level details and works its way up. The book is divided into four sections. The first covers AJAX fundamentals, the ASP.NET AJAX framework, and the UpdatePanel control. There are plenty of simple yet interesting examples using the framework's client- and server-side capabilities. There's also an excellent chapter in this section that covers JavaScript concepts, syntax, and functionality that are paramount for AJAX developers.

The second section delves into some more advanced AJAX scenarios, exploring the UpdatePanel and client-side functionality in much greater depth. This part of the book also introduces the AJAX Control Toolkit, which is an additional set of Web controls for accomplishing many common AJAX scenarios.

The third section looks at ASP.NET AJAX Futures, which is a preview of upcoming AJAX and client-side functionality. And the final section looks at how to implement common AJAX patterns.

ASP.NET AJAX In Action is not intended for beginning Web developers, as it assumes a certain level of familiarity with how to build ASP.NET applications, but this book is accessible regardless of past client-side or AJAX development experience.

Price: $44.99.

manning.com

(Click the image for a smaller view)

(Click the image for a larger view)

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out: Measure Early and Often for Performance, Part 2

CLR Inside Out
Measure Early and Often for Performance, Part 2
Vance Morrison

Code download available at: CLRInsideOut2008_05.exe (205KB)
Contents
MeasureIt Recap
Some Quick Observations
Validating the Data
Caveats
The Uneven Performance Landscape

In the last installment of CLR Inside Out, I stressed that to reliably create high-performance programs, you need to understand the performance of the individual components you use early in the design process (see msdn2.microsoft.com/magazine/cc424899). To do this you need performance data. Thus, measurement has to be an integral part of the design process.

At that time I also introduced a tool called MeasureIt that makes it very easy to create new benchmarks so you can quickly gather the data you need to make good design decisions. The raw numbers that a tool like MeasureIt provides are critically important, but it is also important to develop a general understanding of what the numbers mean. This understanding allows you to predict the performance of something before you actually measure it. That is what I'll discuss here.

MeasureIt Recap

If you have not already downloaded the MeasureIt tool, I strongly encourage you to do so now. It's in the code download for this column, found on the MSDN® Magazine Web site, and it consists of a single EXE file. Running it will generate a Web page displaying the results of running a suite of benchmarks. Additional documentation can be accessed after installation by running the following command:

measureIt /usersGuide

MeasureIt comes complete with its source code, which can be conveniently unpacked using the /edit qualifier. This makes adding a new benchmark as easy as writing one or two lines of code (and supplying the code to be timed). See the user's guide for more on exactly how to do this.

MeasureIt benchmarks are associated with different performance areas, which can be indicated on the command line when the tool is launched. By default (with no command-line parameters), MeasureIt runs a suite of 50 or so benchmarks covering a broad variety of basic Microsoft® .NET Framework runtime operations. Abbreviated sample output is show in Figure 1.

Figure 1 Sample Benchmarks (Close Figure 1)

Name | Median | Mean | StdDev | Min | Max | Samples
NOTHING [count=1,000] | 0.000 | 0.037 | 0.110 | 0.000 | 0.366 | 10
MethodCalls: EmptyStaticFunction() [count=1000 scale=10.0] | 1.000 | 1.103 | 0.496 | 0.857 | 2.577 | 10
ObjectOps: new Class() [count=1000 scale=10.0] | 5.060 | 10.223 | 13.927 | 3.340 | 51.215 | 10
ObjectOps: new FinalizableClass() [count=1000 scale=10.0] | 78.552 | 155.408 | 168.595 | 64.997 | 629.243 | 10
ObjectOps: (Class) Activator.CreateInstance(classType)] | 102.510 | 102.949 | 4.076 | 96.876 | 109.819 | 10
Arrays: localIntPtr[i] = 1 [count=1,000 scale=10.0] | 0.713 | 0.664 | 0.076 | 0.574 | 0.773 | 10
Arrays: string[i] = aString [count=1,000 scale=10.0] | 3.402 | 3.405 | 0.012 | 3.397 | 3.442 | 10
Delegates: aInstanceDelegate() [count=1,000 scale=10.0] | 1.235 | 1.205 | 0.111 | 1.094 | 1.475 | 10
MethodReflection: Method.Invoke EmptyStaticFunction() | 472.283 | 472.744 | 5.409 | 466.291 | 482.094 | 10
P/Invoke: FullTrustCall() [count=1,000] | 6.184 | 6.254 | 0.793 | 5.469 | 7.599 | 10
P/Invoke: 10 FullTrustCall() (10 call average) | 2.669 | 2.688 | 0.061 | 2.665 | 2.870 | 10
P/Invoke: 1 PartialTrustCall [count=1,000] | 27.806 | 30.440 | 8.735 | 26.343 | 56.582 | 10

MeasureIt runs each benchmark 10 times and computes statistics based on the results. The reported values are scaled so that a single call to an empty method takes one unit of time. For example, because Figure 1 shows the median time of allocating a small object to be 5.06, you know it typically takes only a bit more than five times as long to allocate a small object as it does to call a method. But that is not the whole story. Notice that the maximum time for object allocation is more than 51 units. Thus there are times when it can take much longer than the average case. In fact, if the benchmark was unlucky enough to force a full garbage collection of the heap, it might spend considerably more time in the method than the maximum value that is reported here.

Already, however, you should be able to see the value of the MeasureIt harness. With almost no work at all, you can immediately get a rough idea of how expensive small allocations are. Just as important, because the harness collects multiple samples and computes statistics, you can also understand that some operations, such as object allocation, can have considerable variation. Having minimum, maximum, and standard deviation information lets you determine whether or not the measurements are stable enough to be relied upon.

Some Quick Observations

Figure 1 provides you with a rough idea of the cost of object allocation of small objects in the .NET Framework. However, you can quickly deduce quite a bit more. For example, you can surmise that objects with finalizers—declared in C# as ~ClassName()—are more than 10 times as expensive as objects that don't have finalizers. Worse, finalizers are inherited, which means that allocations of instances of any subclass will be similarly penalized. Thus, if at all possible, finalizers should generally be avoided on a class that will have many instances.

You can also infer that allocating an object using reflection (as in Activator.CreateInstance) is more than 10 times as expensive as allocating an object normally. This will be a common theme with reflection APIs: they are expensive compared with their static equivalents, and you should not use them on performance-critical paths.

Furthermore, you can also conclude from the data that calling a method using reflection (as in MethodInfo.Invoke) is more than 450 times slower than calling the method statically (the ratio is so high in part because calling a method normally is very cheap). Likewise, you are also able to deduce that array access is cheap (less than the cost of a method call), but setting an element in an array of object references (as in string[]) is more than 4 times as expensive as a normal array set. You can also see from Figure 1 that invoking a delegate (pointer to a method) is only about 20 percent slower than invoking a method whose target is known at compile time.

Finally, you can tell that when security checks are suppressed, invoking unmanaged code (P/Invoke) is not too expensive (6 times a normal method call), and the average cost can get even cheaper (2.6 times) if many calls are made from the same method. However, if security checks are used (which is the default), the cost is significantly more (27 to 30 times a normal method call).

Already, some useful observations have been made from the built-in MeasureIt benchmarks. Moreover, since the MeasureIt download includes source code, you can get all the necessary details to drill deeper into the data. For example, you may want to know exactly how to suppress the security checks for P/Invoke calls (or even how to make a call to native code in the first place). To do so, just unpack the source using the following command:

measureIt /edit

Then search for the P/Invoke benchmark. You'll get the exact code you need, which you can examine, debug, and even modify for your own purposes.

Validating the Data

While MeasureIt quickly tells you the costs of some of the basic operations performed by the .NET runtime, it will not tell you why the numbers are what they are. It is very possible that you are not measuring what you think you are measuring.

In last month's column I highlighted how easy it is to create misleading benchmarks (especially micro-benchmarks), underscoring the importance of validating performance measurements before relying on them. Remember that you must also validate the data from the built-in benchmarks.

Here is where some expertise in the internals of .NET runtime is helpful. By knowing exactly how operations within the runtime are translated into machine instructions, I can make estimates on how long various operations should take. This is summarized in Figure 2. Because of the optimizations that the hardware itself performs, instruction counts do not map precisely to execution time; however, they do provide a good ballpark estimate. If you ever find that the instruction execution time far exceeds your expectations based on the instruction count, it's more than likely that the data is incorrect. The data shown in Figure 2 is best thought of as a qualitative view of the operations, which nicely compliments MeasureIt's quantitative view.

Figure 2 Performance of .NET Runtime Operations from MeasureIt (Close Figure 2)

Operation | Instruction Count | Comments
Integer Arithmetic | 1 | Simple unchecked arithmetic compiles to its corresponding machine instruction and often has a cost of less than one machine cycle.
Floating Point Arithmetic | 1 | Compiles to x87 instructions (for 32-bit). Currently, the runtime does not do vectorization or strongly leverage newer hardware instructions (SSE2), so typically the best unmanaged compilers will do better on floating point intensive apps.
Instance Field Fetch or Set | 1-10 | Most operations take 1 instruction. However, setting a field that is an object reference (not a primitive type) requires the use of a write barrier routine that takes 6-10 instructions.
Static Field Fetch | 1-12 | Only 1 instruction is required in the common case of just-in-time (JIT) compiled code that runs in one AppDomain. However, hosts like ASP.NET can ask the runtime to generate code that can be shared across all AppDomains (saving a lot of memory and JIT time) at the expense of making a static field fetch about 12 instructions. The JIT compiler does hoist common operations like these out of loops, partially mitigating this cost. Code precompiled with native image generation (Ngen) is generated in such a way that it will work regardless of the code sharing requested, which means that even in the best case, Ngen-generated code takes 6 instructions to fetch a static field.
Non-Virtual or Static Method Call | 1 | Compiles to a single call instruction and is the fastest kind of call.
Virtual Method Call | 2 | Uses dispatch tables just like C++. This is the fastest indirect kind of call.
Interface Method Call | 4-20 | Interface methods are dispatched through stubs that are very fast (4 total instructions) when the particular call site almost always has the same target. When a single call site dispatches to many targets, it takes 10-20 instructions to do a hash table lookup to dispatch.
Delegate Dispatch | 4-15 | If there is a single target method and it is an instance (non-static) method, which is the common case, it takes 4 instructions and is comparable in speed to any other type of indirect dispatch. When the target is static, the arguments need to be shuffled to remove the unneeded "this" pointer that was passed, which can take several instructions. When a delegate has several subscribers (events can do this), a dispatch must have a loop and is more expensive, but it is a less common scenario.
Object Allocation | 10-1,000+ | The code path in "new" for most objects takes 10-15 instructions; however, some object types (such as finalizable objects) cost much more. In all cases, the allocation incurs additional costs that are paid later in bulk. These include the cost of clearing the memory (proportional to its size) and scanning costs during some garbage collections (GCs) for as long as the object lives. Objects that live a short time incur significantly less GC overhead than objects that live longer; however, the cost is still high enough that allocation should be minimized on performance-critical code paths.
Array Access | 1-25 | Array access is typically two instructions: bounds check and fetch. On simple loops where you iterate over all elements in an array, the JIT compiler can eliminate the bounds check, making it just one instruction. In fact, as a result of this optimization, using unsafe pointer access instead of array access will often not improve performance significantly. If the element being set is an object reference, the set requires a write barrier and type check, which increases the size of the code path to about 25 instructions.
Casting | 4-100+ | Successfully casting an object to its exact type is fast (4 instructions); however, casting to a superclass is slower, and casting to an interface is slower still. Casting arrays is even slower. Failed casts (often the case when using the C# "is" or "as" operators) are also relatively slow.
Locking | 20-1,000+ | Entering and exiting a lock (System.Monitor.Enter or the c# lock statement) is moderately expensive (10 to 15 times a call-ret), even in the best case. If there is contention for the lock, it can be significantly slower.
P/Invoke Call | 15-1,000+ | Calling native code (P/Invoke) from managed code, in the best case when security checks are suppressed and no argument conversions are needed, is 15-20 instructions. It is much worse when strings are passed (requires conversion) and/or COM interop is involved.
Reflection | 1,000-10,000+ | Calling simple reflection APIs on types (such as System.Type.GetType) can be cheap (< 10 instructions); however, using reflection APIs to do almost anything else, such as call methods, set fields, or create objects, is significantly more expensive (10 to 100 times) than its non-reflection alternative. As a rule, reflection should not be used in performance-critical code paths.
Generics | any | Generic types can have performance similar to their non-generic counterpart. When the type argument of the generic type is a class (rather than a struct), the code is shared among all instances of the type. This sharing saves space, but it can mean that operations that depend on the type parameter may be slower than expected. If you use generics on a performance-critical path, you should measure.

The information in Figure 2 is useful for validating the performance numbers that MeasureIt provides, but it also allows you to make rough predictions about the performance of your programs and the costs of various design trade-offs. For example, you can predict the cost trade-off between two code factoring designs that differ in how often they cross a managed-to-unmanaged (P/Invoke) boundary. You can also make predictions about how much would be saved by avoiding reflection methods or the cost of adding locks to a program to make it thread safe.

Caveats

Personally, I have found the data from MeasureIt to be very helpful in making many performance decisions. However, the built-in benchmarks focus on CPU-intensive operations. If your application's performance is not limited by the CPU, a focus on CPU optimization is misguided. Three common scenarios where CPU is not the critical performance factor are when the application response time is dominated by I/O costs or network latency, when the application response time is dominated by memory caching costs, and when, in a multithreaded application, the application response time is impacted by thread serialization delays.

The performance of some applications is limited by the speed at which they can execute disk and network I/O, which is many times slower than actual instruction execution. This can happen on application startup when the application requests files that are not yet cached in memory (cold startup). It can also happen when your application consumes enough memory to cause many page faults. In general, if your application is not consuming 100 percent of a processor, then these other latencies are important. Most of the information I present here is not relevant to this case since this column focuses on identifying CPU-bound performance bottlenecks.

In other applications, response time is dominated by memory caching costs. This typically happens when you have large, in-memory data structures. The application looks CPU bound (it consumes 100 percent of a processor), but the CPU is actually waiting on the memory subsystem most of the time. You need a profiler with access to statistics on the memory subsystem to diagnosis this properly, but large memory consumption (> 50MB) and a large number of page faults are usually an indication that memory issues are a factor. Generally, your design strategy in this case should be to make things small (even if it increases the number of instructions executed).

In multithreaded applications operating on multiprocessor hardware, it is not uncommon for the threads to be delayed as they wait to enter critical sections of code—code that must be accessed by only one thread at a time. These serialization delays tend to grow worse as the number of concurrently executing threads grows. This lock contention can show up either as being non-CPU bound (when the wait time is long and the waiting thread goes to sleep) or CPU bound (when the wait time is very short but happens very frequently).

Sadly, explaining these non-CPU issues well enough to impart good design intuition would require another full column or two. Just remember that memory consumption does matter (especially for large applications), and you should try to decrease the size of data structures that are larger than 1MB if the trade-off in instruction count is modest.

However, if your application consumes 100 percent of one CPU when it is running (with very few or no page faults), consumes a modest amount of memory when CPU bound (hot data structures are less than 1MB), and is not intended to be a highly parallelized application that relies on shared data, then your performance issue is likely related to the number of instructions that execute. Therefore, the information in Figure 2 and the data MeasureIt provides will be useful in predicting performance.

The Uneven Performance Landscape

The .NET runtime team goes to some trouble to optimize operations whenever possible. Generally speaking, this is a great thing, but it does create an uneven performance landscape because there are cases where the optimization can't be applied. We have already seen an example of this: finalizable allocation. Here are some more examples.

First of all, interface calls are highly tuned for the case in which a particular call site tends to dispatch to the same target. For example, you could have a routine that takes a List<object> and calls ICompareable.Compare on each element. If this was passed a List of strings, the interface call would always go to String.Compare and would be fast. However, if the list contained many different types, then that same call site must dispatch to many different targets. This latter case is significantly slower than the homogenous case.

Second, when primitive types (such as int) need to be passed to operations expecting an object, the value must be converted to an object (this operation is known as boxing). This requires an object allocation and thus is more expensive than expected. Often this boxing is inserted by the compiler automatically, making it all too easy to fall into this trap.

Third, methods that use the variable argument params feature of C# (such as Console.WriteLine) need to allocate an array—and possibly box (allocate) objects for some parameters—for each call. This makes the call much more expensive than normal calls (by a factor of 10). There are dozens of more examples that I could easily list, but these will suffice.

While people who know about the internals of the runtime can warn you about some of these, realistically the list will never be complete. This is why the information in Figure 2 and the data gathered from MeasureIt should only be used as a rough approximation. You never know when you might fall into a bad performance code path. Thus it pays to be cautious: if there is even a possibility that your hot code path might not be optimized, you should not rely solely on deductions made from Figure 2 but instead write a quick micro-benchmark that very accurately represents your hot code path and measure it yourself.

Send your questions and comments to clrinout@microsoft.com.

Vance Morrison is the Compiler Architect for the .NET runtime at Microsoft, where he has been involved in the design of .NET since its inception. Vance drove the design for the .NET Intermediate Language and was lead for the just-in-time compiler team.


Data Points: Entity Framework Q&amp;A

Data Points
Entity Framework Q&A
John Papa

Code download available at: DataPoints2008_05.exe (962KB)
Contents
EntityClient + Entity SQL
Object Services + Entity SQL
Object Services + LINQ
Wrapping Up

Judging by the hundreds of questions I've received on the topic, developers are very interested in the Entity Framework and its data access and modeling implications. While I cannot possibly respond to every one of them, in this month's column I will answer some of the more common questions I have been asked.

Understanding entity modeling, the mapping of entity models to relational databases, and Entity Data Model (EDM) design are the first steps in grasping the Entity Framework. I'll start this Q & A off by answering some questions about the foundations of the Entity Framework, including the ObjectContext, and then discuss when and where it is appropriate to use Entity Client with Entity SQL. I will also explain the differences between EntityClient and Object Services and cover the implications of using LINQ and Entity SQL with these services.

Analysis of the queries created both in code targeting the Microsoft® .NET Framework and in the native SQL code is also an important part of the Entity Framework, so I'll examine explicit loading and eager loading by reviewing the generated SQL. All code examples and the sample NorthwindEF database are available for download with this column from the MSDN® Magazine Web site.

Why use Entity SQL when I can use LINQ to get to entities? Every time I give a presentation about using Entity SQL with EntityClient or Object Services, someone asks this question. (I can't blame them. It's one of the first questions I had when I began diving into the Entity Framework!) The strong typing and query syntax of LINQ is so compelling that developers cannot help but wonder why they need a new language to interact with entities.

To fully answer this question, I first must discuss the three major techniques that you can use to interact with an EDM:

  • Writing Entity SQL queries using the EntityClient provider
  • Writing Entity SQL queries using Object Services
  • Writing LINQ queries using Object Services

Each of these techniques has common characteristics; for example, all use the EntityClient provider either directly or indirectly. But they differ in the results they produce and how they go about getting those results.

The EntityClient provider has a series of objects that should be familiar to you if you know the ADO.NET object model. EntityConnection is used to connect to the EDM, EntityCommand is used to issue a query against the EDM, and the results of the command are returned via DbDataReader. Whether you are using EntityClient directly or indirectly via Object Services, EntityClient ultimately issues the query and returns the results.

So, restating the question, why use Entity SQL when you have LINQ? The answer lies in the pros and cons of each technique.

EntityClient + Entity SQL
Insights: Entity Framework Data Loading

Many object-relational mapping frameworks support two well-known behaviors that help in writing less persistence "plumbing" code and in improving how many times the application hits the database and how much data is transferred:

Eager Loading Each time the program needs to work with a set of related objects, the whole graph is hydrated from the database in a single upfront request.

Lazy Loading Data is loaded from the database automatically as the program navigates the properties that connect the object graph, potentially hitting the store multiple times but loading only what is necessary.

A question that the Data Programmability team at Microsoft often gets (and one I used to ask myself—see forums.microsoft.com/MSDN/ShowPost.aspx?PostID=644979 and forums.microsoft.com/MSDN/ShowPost.aspx?PostID=644808) is why the Entity Framework comes out of the box without automatic lazy loading support. The answer has its roots in the past, but it also has some implications for the future.

Some History

In 2000, when I first learned about the ADO.NET object model (called ADO+ at the time), my impression was that DataSets were very cool, but I could not understand why the connected and the disconnected APIs were so different from each other. I thought I would always miss the "location transparency" of .NET ADO Recordsets before the existence of the Microsoft .NET Framework.

By that time, one of the best ways to learn about .NET was to spend time in the DOTNET discussion list, so I posted my question there (discuss.develop.com/archives/wa.exe?A2=ind0009B&L=DOTNET&D=0&I=-3&P=55148). Omri Gazitt gave me the answer: network roundtrips should not be hidden from the developer. This was a novel idea for me at the time. But, in fact, this is one of the foundational tenets of ADO.NET. No wonder the Entity Framework builds on that!

Back to the Present

Now, eight years later, the Entity Framework Team is readying the first version of a very ambitious data access platform on top of the now very successful ADO.NET framework. Many of the same principles still apply, and, once more, we want it to be a solid foundation for the future.

Following the "no hidden network roundtrips" principle, Entity Framework avoids automatic lazy loading. The sole action of traversing a graph composed of our code-generated classes will never trigger a query against the store. Instead, Entity Framework requires user code to explicitly hydrate any object before use.

It is possible to load as you go, but you need to invoke a method called Load on references or collection every time you want to do that. In fact, by interfering with the code generation process or by writing your own classes, it is still possible to get implicit lazy loading working.

However, there is more to lazy loading than just hidden roundtrips. It is important for you to know that every time your program uses lazy loading, it may easily end up with a graph that contains inconsistent data. Also, if a program keeps loaded things around without refreshing them for a long time, it runs into the risk of having stale data in the graph.

The opposite strategy, eager loading, is fully supported on a query-by-query basis, explicitly using a query builder method named Include that you can use to "span" your query across additional related entities. The parameter for Include is a string containing a dot-separated path of navigation properties.

Include is generally translated to a JOIN query between the source tables, so it will usually cause data redundancy over the wire. Because of this, remember that you should also take great care when you use Include.

—Diego Vega
Program Manager, Microsoft Data Programmability Team

Writing code with the EntityClient API provides you the most granular control of the three techniques. You can create an EntityConnection to connect to the EDM, write a query in Entity SQL and execute it with an EntityCommand, and return the results via a DbDataReader. This technique is also slightly more lightweight, cutting through a little bit of the syntactic sugar that LINQ and Object Services provide.

The greatest advantage of Entity SQL is its flexibility. Its string-based syntax lends itself easily to building dynamic queries. This can be very useful when you need to create an ad hoc query.

One effect of this flexibility and light weight, however, is that you can only return results via the DbDataReader. It is not possible to return pure entities from the EDM using EntityClient with Entity SQL. The DbDataReader can be retrieved and used to iterate through the collection of rows that satisfy the Entity SQL query. The code in Figure 1 allows you to iterate through customer records via a DbDataReader, not Customers entities.

Figure 1 Iterating through Rows Via DbDataReader (Close Figure 1)
string city = "London";
using (EntityConnection cn = new EntityConnection("Name=Entities"))
{
  cn.Open();
  EntityCommand cmd = cn.CreateCommand();
  cmd.CommandText = @"SELECT VALUE c FROM Entities.Customers AS c WHERE
                     c.Address.City = @city";
  cmd.Parameters.AddWithValue("city", city);
  DbDataReader rdr = cmd.ExecuteReader(CommandBehavior.SequentialAccess);
  while (rdr.Read())
    Console.WriteLine(rdr["CompanyName"].ToString());
  rdr.Close();
}

Here's a good tip: when in debug mode, you may receive an error that states you cannot read from column ordinal X. This error only occurs in debug mode and can be avoided if you close the autos window in debug mode. This is a known issue with the Entity Framework beta 3.

There is currently no comparable Data Manipulation Language (DML) for Entity SQL. This means you cannot directly express Insert, Update, or Delete statements against the EDM (see Figure 2).

Figure 2 Entity Framework APIs (Close Figure 2)

 | EntityClient with Entity SQL | Object Services with Entity SQL | Object Services with LINQ
Direct to EntityClient Provider | Yes | No | No
Good for Ad Hoc | Yes | Yes | No
Can Issue DML Directly | No | No | No
Strongly Typed | No | No | Yes
Can Return Entities as Results | No | Yes | Yes

Object Services + Entity SQL

The next technique is using Object Services to execute queries with Entity SQL. This moves away from direct interaction with the EntityClient provider (although it still communicates with the provider under the covers). You would use ObjectContext and ObjectQuery<T> to issue queries against the EDM.

This technique is good for issuing ad hoc queries, as in the first technique. However, rather than returning data via the DbDataReader, using Object Services with Entity SQL allows you to return entities from the EDM. This gives you a solid combination: flexible queries that return first-class entities.

Since Entity SQL currently lacks any DML constructs, it is not possible to issue Insert, Update, or Delete commands using Entity SQL and Object Services. You could, however, retrieve an entity from the EDM using this technique and then update the entity using the SaveChanges method on the ObjectContext. As you can see, the following code sample iterates through a collection of Customers:

string city = "London";
using (Entities entities = new Entities())
{
  ObjectQuery<Customers> query = entities.CreateQuery<Customers>(
    "SELECT VALUE c FROM Customers AS c WHERE c.Address.City = @city",
    new ObjectParameter("city", city)
  );

  foreach (Customers c in query)
    Console.WriteLine(c.CompanyName);
}

Object Services + LINQ

Using Object Services with LINQ is not as good for ad hoc queries as the other techniques. The following code sample returns a collection of Customers from the EDM:

string city = "London";
using (Entities entities = new Entities())
{
  var query = from c in entities.Customers
              where c.Address.City == city
              select c;

  foreach (Customers c in query)
﻿    Console.WriteLine(c.CompanyName);
}

Just like Entity SQL, LINQ also does not support direct syntax for DML statements. Currently, it is only possible to update entities against a database when using Object Services (by way of the SaveChanges method). This is done by returning entities from the EDM, whose changes are tracked by the Entity Framework. In short, neither LINQ nor Entity SQL perform update operations; it's the ObjectContext of the EDM that does this.

I summarized the differences in these techniques in Figure 2. So why would you want to use Entity SQL when you already have LINQ? Entity SQL is a solid choice when you have a need for ad hoc queries or want to create a more flexible query than what can be done with LINQ. Otherwise, I suggest using LINQ with Object Services so you can benefit from its strong typing and ability to return entities and projections.

Along with the strongly typed syntax of LINQ, you also get design time views of many errors that you would not otherwise catch until running the application. I really appreciate this feature—it allows me to keep writing code instead of having to build and run to catch errors.

What is the role of ObjectContext? ObjectContext is the gateway to the EntityConnection for Object Services. It provides access to the EDM via the underlying EntityConnection. For example, you can access the entities via the ObjectContext, interrogate ObjectContext to find out information about the state of objects, and create ObjectQuery<T> queries using the CreateQuery method.

Another purpose of ObjectContext is to provide a way for objects to get update information about database entries. For instance, you can use the methods of ObjectContext to add entities to the ObjectContext, delete entities, manipulate entities, and ultimately save the changes of the entities to the database (via the SaveChanges method).

How do explicit and eager loading work in the Entity Framework? Explicit loading is the default behavior in LINQ to Entities and the Entity Framework. When a query is executed within the Entity Framework, the entities returned by the query are fully accessible, but any associated entities are not immediately loaded. For example, if you write a query that retrieves all Orders in the EDM, a SQL query will be executed under the hood that grabs the order records and returns a collection of Orders entities. However, the customer record associated with an order is not fetched as a part of this query, and as a result of this, an Orders entity's associated Customers entity is not loaded. Therefore, the following code sample will throw an exception when it tries to access the Order's Customers since it is not loaded:

using (Entities entities = new Entities())
{
  var query = (from o in entities.Orders
               where o.Customers.CustomerID == "ALFKI"
               select o).First<Orders>();

  Orders order = query as Orders;
  Console.WriteLine(order.OrderID);

  Console.WriteLine(order.Customers.CompanyName);
}

The Entity Framework provides a Load method on each instance of the EntityReference class. This method can be used to explicitly load a collection that is associated with another entity. For example, you could change the previous code example by telling the Entity Framework that it needs to go get the Customers record for the Order. The revised code that performs the explicit load is shown in Figure 3. It first checks to see if the Customers entity is loaded. If not, it loads the Customers for that Order. This technique is known as explicit loading.

Figure 3 Explicit Loading (Close Figure 3)
using (Entities entities = new Entities())
{
  var query = (from o in entities.Orders
               where o.Customers.CustomerID == "ALFKI"
               select o);

  foreach (Orders order in query)
  {
    if (!order.CustomersReference.IsLoaded)
      order.CustomersReference.Load();
    Console.WriteLine(order.OrderID + " --- " +
      order.Customers.CompanyName);
  }
}

The example in Figure 3 will execute a SQL query to get the Customer record for each order that is processed. If iterating over hundreds or even just dozens of orders, this would cause a lot of individual queries against the database. If you know in advance that you will need more data than what will be returned by your query (such as customer information for an order), you can load this information ahead of time.

For some further thoughts on this topic, direct from the Microsoft team responsible for the Entity Framework, see the sidebar, "Insights: Entity Framework and Date Loading."

Figure 4 demonstrates a technique known as eager loading. The Include method being invoked on the Orders entity in the LINQ query accepts an argument that in this case directs the query to retrieve not only the Orders but their related Customers as well. This technique will generate a single SQL statement that loads all Orders and Customers meeting the criteria in the LINQ query.

Figure 4 Eager Loading (Close Figure 4)
using (Entities entities = new Entities())
{
  var query = (from o in entities.Orders.Include("Customers")
               where o.ShipCountry == "USA"
               select o);

  foreach (Orders order in query)
    Console.WriteLine(order.OrderID + " --- " +
      order.Customers.CompanyName);
}

You must understand that the type of loading is important for you to consider. If you use the Load method (as shown in Figure 3) to explicitly load entities during an iteration of a collection, this will cause several queries to hit the database, one for each call to the Load method.

This is a perfectly good technique if all you need to do is access the data once or twice. However, eager loading using the Include method (as shown in Figure 4) is a much better choice if you know that for a given entity, you will always need to have access to the data in related entities.

The best course of action is to profile the performance of the different options at your disposal and test them to see which best suits your scenarios. As with most decisions, the decision to use eager loading or explicit loading depends on your situation.

How do I see the SQL that will execute? I have often wanted to see what SQL would execute when I created a query with ObjectQuery and I've found two helpful techniques. The first is to use the SQL Server® Profiler tool (or whatever database engine's profiling tool is available to you). The second technique is to use the ObjectQuery class's ToTraceString method.

Figure 5 demonstrates how to invoke the ToTraceString method on an ObjectQuery<T>. Notice that the ObjectContext's connection is open (the EntityConnection). The ToTraceString method requires that the connection be open so it can determine the query that will execute. The ToTraceString method is available on both the ObjectQuery<T> and the EntityCommand classes.

Figure 5 Using ToTraceString (Close Figure 5)
string city = "London";

using (Entities entities = new Entities())
{
  ObjectQuery<Customers> query = entities.CreateQuery<Customers>(
    "SELECT VALUE c FROM Customers AS c WHERE c.City = @city",
    new ObjectParameter("city", city)
  );
  entities.Connection.Open();

  Console.WriteLine(query.ToTraceString());

  foreach (Customers c in query)
    Console.WriteLine(c.CompanyName);
}

What can I do with complex types? The Entity Framework offers a data structure known as a complex type for representing a set of properties, usually closely related to each other. Consider an Address type. A complex type could be used to create an Address type for a customer so that the Customer entity's address-related properties (such as City, Region, and Phone) would now fall under the Address type rather than directly under the Customer type. Complex types provide for a logical grouping of similar scalar properties, which makes it easier to find closely related properties on an entity and keeps them logically grouped in the EDM. Like entities, complex types contain scalar properties; unlike them, they do not have identity (key values), and they cannot be persisted to the database via the ObjectContext. Complex types are an aspect of entities, not entities themselves. They are a good tool when grouping logically related properties of an entity.

How do I create a complex type? Since the EDM designer does not support visually creating complex types right now, creating them involves cracking open the EDMX file in an XML editor. The first step is to create the ComplexType in the Conceptual Scheme Definition Language (CSDL); then, modify the EntityType that will reference the complex type. Figure 6 shows the Customers EntityType in the CSDL with the newly created AddressType complex type. Notice that the Customers EntityType has had its Address, City, Region, and other address-related properties removed and replaced with a new property called Address. This new Address property refers to the AddressType complex type.

Figure 6 Creating a ComplexType (Close Figure 6)
<EntityType Name="Customers">
  <Key>
    <PropertyRef Name="CustomerID" />
  </Key>
  <Property Name="CustomerID" Type="String" Nullable="false"
    MaxLength="5" FixedLength="true" />
  <Property Name="CompanyName" Type="String" Nullable="false"
    MaxLength="40" />
  <Property Name="ContactName" Type="String" MaxLength="30" />
  <Property Name="ContactTitle" Type="String" MaxLength="30" />
  <Property Name="Address" Type="Self.AddressType"
    Nullable="false"/>
  <NavigationProperty Name="Orders"
     Relationship="NorthwindEFModel.FK_Orders_Customers"
      FromRole="Customers" ToRole="Orders" />
</EntityType>
<ComplexType Name="AddressType">
  <Property Name="Address" Type="String" MaxLength="60" />
  <Property Name="City" Type="String" MaxLength="15" />
  <Property Name="Region" Type="String" MaxLength="15" />
  <Property Name="PostalCode" Type="String" MaxLength="10" />
  <Property Name="Country" Type="String" MaxLength="15" />
  <Property Name="Phone" Type="String" MaxLength="24" />
  <Property Name="Fax" Type="String" MaxLength="24" />
</ComplexType>

The next step is to modify the mapping in the mapping specification language (MSL) to take into consideration the new complex type. Figure 7 shows the mapping of the Customers EntityType with the new AddressType complex type. The project will build and you can now reference the complex types in code. For example, the following LINQ query can be executed to filter only the customers in the city of London:

Figure 7 Mapping a Complex Type (Close Figure 7)
<EntitySetMapping Name="Customers">
  <EntityTypeMapping
    TypeName="IsTypeOf(NorthwindEFModel.Customers)">
    <MappingFragment StoreEntitySet="Customers">
      <ScalarProperty Name="CustomerID" ColumnName="CustomerID" />
      <ScalarProperty Name="CompanyName" ColumnName="CompanyName" />
      <ScalarProperty Name="ContactName" ColumnName="ContactName" />
      <ScalarProperty Name="ContactTitle" ColumnName="ContactTitle" />
      <ComplexProperty Name="Address"
        TypeName="NorthwindEFModel.AddressType">
        <ScalarProperty Name="Address" ColumnName="Address" />
        <ScalarProperty Name="City" ColumnName="City" />
        <ScalarProperty Name="Region" ColumnName="Region" />
        <ScalarProperty Name="PostalCode" ColumnName="PostalCode" />
        <ScalarProperty Name="Country" ColumnName="Country" />
        <ScalarProperty Name="Phone" ColumnName="Phone" />
        <ScalarProperty Name="Fax" ColumnName="Fax" />
      </ComplexProperty>
    </MappingFragment>
  </EntityTypeMapping>
</EntitySetMapping>

var query = from c in entities.Customers
            where c.Address.City == "London"
            select c;

Wrapping Up

In this month's column I compared and contrasted the use of EntityClient and Object Services along with Entity SQL and LINQ. I also touched on how and why you create complex types, and I demonstrated how eager and explicit loading works. I've received so many questions from interested developers regarding the Entity Framework that I intend to discuss similar topics and practical tips in future installments of Data Points.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a Senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP and INETA speaker, has authored several books about ADO, XML, and SQL Server. He can often be found speaking at industry conferences such as DevConnections and VSLive or blogging at johnpapa.net.


Cutting Edge: Single-page Interface and AJAX Patterns

Cutting Edge
Single-page Interface and AJAX Patterns
Dino Esposito

Code download available at: CuttingEdge2008_05.exe (203KB)
Contents
Impact of the AJAX Paradigm
Single-Page Interface Model
Drawbacks of the Single-Page Interface Model
Accessibile Rich Internet Apps
A Brief Tour of AJAX Patterns
The Unique URL Pattern
The Timeout Pattern

Compared to the development paradigm under which the vast majority of today's Web applications have been built, AJAX represents a paradigm shift for architects of Web solutions. It leans on a few new principles and rules to explain the behavior of a Web-based system and requires some new algorithms to implement them.

The main principle behind AJAX is that you send plain data to the Web server and receive more plain data.

The second AJAX principle is that you orchestrate operations on your own, thus bypassing the host browser and its single page request/response mechanics.

The third AJAX principle is that the client code is fully responsible for updating the user interface using the plain data it receives from the server.

This column will lay out the foundation for developers ready to completely move away from the defensive AJAX implementation represented by partial rendering. Partial rendering is a way to have some AJAX functionality while still staying rooted in a Web Forms architecture. The AJAX paradigm is based on new principles that require new design patterns.

Impact of the AJAX Paradigm

ASP.NET partial rendering is a clever addition that sits on top of the classic Web Forms postback model. In a nutshell, a page using partial rendering has the same postback architecture and page lifecycle as that which occurs in a non-AJAX page (see Figure 1). The difference, then, lies in a client-side interceptor that simply prevents the browser's default action—the form submission—and replaces it with an XMLHttpRequest-led HTTP request (see Figure 2). This trick saves the user a full page refresh while saving developers hours of training on a new architecture and new patterns.

Figure 1 Traditional Full-Page Postback Operation
Figure 1 Traditional Full-Page Postback Operation(Click the image for a smaller view)
Figure 1 Traditional Full-Page Postback Operation
Figure 1 Traditional Full-Page Postback Operation (Click the image for a larger view)
Figure 2 AJAX XMLHttpRequest Partial Rendering
Figure 2 AJAX XMLHttpRequest Partial Rendering(Click the image for a smaller view)
Figure 2 AJAX XMLHttpRequest Partial Rendering
Figure 2 AJAX XMLHttpRequest Partial Rendering (Click the image for a larger view)

It goes without saying that partial rendering offers graphical benefits within the context of the current page. If your application is designed around a collection of individual pages, however, a transition from one page to the next still requires a full page load. Partial rendering is excellent at replacing intra-page postbacks, such as paging a grid, adjusting the user interface after a selection change, or editing a record in-place.

Because partial rendering is just a smarter form of a page postback, it has the same architectural limitations of the Web Forms model. For example, you can have just one pending request per browser window. Under the current model for Web apps—submit a form, get a new page—this limitation may not be much of an issue.

However, once you broaden your horizon with AJAX, it starts becoming normal to expect that a user can execute multiple activities at the same time. With partial rendering alone, it's impossible for a user to have two asynchronous operations execute simultaneously and have both complete.

Partial rendering is limited to one request operation at a time in order to preserve the consistency of the view state—which remains an integral part of the model. And this goes against the first A in AJAX: asynchronous. With partial rendering, out-of-band, JavaScript-driven postbacks are possible, but only one at a time. If a second operation is triggered before the previous one has completed, the pending operation is aborted so that the new operation can proceed. This last-win discipline can be altered programmatically to a first-win model where the current operation is kept active and the new one swallowed by the system—but still, the fact remains that only one operation is possible at a time.

Single-Page Interface Model

To take full advantage of AJAX, you need to have all of your features, or at least most of them, in a single page. This is known as the Single-Page Interface (SPI) model. In the SPI model, all browser interactions with a Web application take place within the boundaries of just one page. This approach is revolutionary for the Web, but it is nothing new for Windows® and desktop development. In the end, the SPI model is just like having a Windows application with a main (and unique) window.

In the SPI model, the main page is a combination of visual elements that can be loaded, updated, and replaced independently (see Figure 3). In this way, the entire page doesn't have to be reloaded following a user action. At any time, only visual elements and content that is relevant to the current stage of the application is displayed. Everything else is hidden; it will show up as soon as there's any need for it in the application's flow.

Figure 3 Single-Page Interface Elements within a Page
Figure 3 Single-Page Interface Elements within a Page(Click the image for a smaller view)
Figure 3 Single-Page Interface Elements within a Page
Figure 3 Single-Page Interface Elements within a Page (Click the image for a larger view)

The SPI model naturally enables a number of highly interactive features, which include in-place editing, context-sensitive user interface, immediate user feedback prompts, and asynchronous operations. But more than performance or responsiveness, the primary benefit of the SPI model is the significant improvement it offers in the user experience.

Even with these many benefits, though, you should know that designing a new application with the SPI model in mind is challenging, as no established set of patterns and best practices exists. In the end, there's an easy way to employ AJAX that fits a large number of companies and situations. And there's a not-so-easy way to employ AJAX that is taking root, too.

For building pure AJAX apps, you need a good library of user interface widgets to provide effects and special behaviors. You need a rich and customizable document object model (DOM) that uses the standard World Wide Web Consortium (W3C) DOM as its underlying engine but allows you to define your own application-specific model. And finally, you need a server and client framework for developing user interface and codebehind scripts easily and effectively. Ideally, you'll also have tools to debug and test all of this.

Microsoft and other vendors are providing some of these bits and pieces. You have a number of UI widget libraries to choose from and you have vendors that are producing controls that define their own client-side object models. What would really help is an improved AJAX framework specifically designed as an alternative to Web Forms and inspired by the SPI model.

One such alternate programming model is the Model View Controller (MVC) Framework found in the ASP.NET 3.5 Extensions library. In its current form, though, it doesn't have too much to share with AJAX, and while it doesn't prevent the implementation of any AJAX feature, it doesn't seem to have been designed to evolve into an SPI model, either. Time will tell, however.

In the SPI model, the main page points to HTTP endpoints in the same application. It executes remote code but doesn't reload the full page. And it updates the user interface using controls that emit both HTML and script. And these controls could be intelligent enough to generate much of the JavaScript code they require. For example, imagine a form used to book a flight, with two possible parameters for the search: time or cost. If the user is more interested in the cheapest flight, you have no need to display the dropdown list with hours. If the user needs to fly at an exact time, you must bring up the HTML that displays hours.

Clearly, this show/hide trick can be easily accomplished with a bit of JavaScript. Today, though, the page developer is responsible for writing this code. Controls such as the CollapsiblePanel control from the ASP.NET AJAX Control Toolkit (asp.net/AJAX/AjaxControlToolkit/Samples/CollapsiblePanel/CollapsiblePanel.aspx) can also aid in accomplishing this.

Drawbacks of the Single-Page Interface Model

While it yields a more interactive user experience, the SPI model, and the whole AJAX paradigm it embodies, raises a number of issues such as searchability, history management, accessibility, and offline support. For years, Web pages have been tracked using permanent links. Search engines built their business by just mapping a set of keywords to one or more URLs. This model worked based on the assumption that each state in a Web application corresponds to a page and a distinct URL.

With the SPI model in AJAX, this assumption is no longer valid. If everything (or most of it) happens within the same page, there's no URL transition to mark a different state and different site content. Subsequently, there's no easy way to associate content (and keywords) with unique URLs. This aspect of the SPI model affects searchability as well as history management (such as the ability to use Back and Forward buttons).

Browser history may ultimately become an outmoded concept, tightly bound to the classic, static Web model. Nonetheless, you must keep in mind that users are so familiar with these buttons that simply disabling them is not a viable option.

Accessibility is another big issue with AJAX applications. Most popular screen-readers have great difficulty with any content generated via DOM scripting. By design, all forms of AJAX (including partial rendering) are heavily based on DOM scripting. Subsequently, you will need to address accessibility through other avenues.

Section 508 of the paper on the Web Content Accessibility Guidelines (WCAG) recommends that pages provide alternate functional text that can be read by assistive technology whenever they utilize scripting languages to display content or to create visual elements. The <noscript> tag exists to comply with this recommendation. Currently, the majority of AJAX frameworks make their dynamic updates within the page without updating static information contained in <noscript> tags.

Accessibile Rich Internet Apps

The problem seems to be twofold and involves both AJAX applications and the technology behind screen readers. On the one hand, you have screen readers that understand some client events such as onclick, keypress, and readystatechange. Awareness of these events in a screen reader might be sufficient to add at least a first layer of accessibility to a large number of AJAX-based applications.

On the other hand, screen readers don't usually speak any new information if they have proceeded beyond the initial page load event in the DOM. But there are some tricks you can use to let the screen reader know that something has changed. One of the most effective involves setting tabindex to -1 on the root of the updated DOM tree. It's a mere trick, though, and accessibility deserves a broader AJAX solution.

A possible solution is the new standard being developed at W3C, called Accessible Rich Internet Applications (ARIA), that mainly consists of reader-specific extensions to HTML tags. A few popular client-side AJAX libraries already support some ARIA features. More generally, though, the problem is not just with the availability of an AJAX-compliant standard for accessibility. The problem is also, if not especially, with the contents (both markup and script) that server controls actually emit.

And what about offline applications? Many developers think that offline AJAX applications are not possible because AJAX applications are still strictly Internet-based. In my opinion, this statement is a bit too simplistic, but it's not entirely false. Today nearly all Web applications—AJAX and non-AJAX—are Internet- or intranet-based.

But non-AJAX applications are definitely able to work offline. In the case of history management, for instance, the magic of enabling offline navigation is all in the browser. In a classic Web application, each HTTP request is managed by the browser. When no connection is available (and before firing an HTTP 404 error), the browser looks in its local cache of pages.

An AJAX application differs from a classic Web application in that it uses the XMLHttpRequest object instead of the browser engine to send HTTP requests. For AJAX applications to support offline scenarios, you only need to endow the XMLHttpRequest object with either access to the browser's cache or the ability to create and manage its own cache of visited pages. Some AJAX frameworks are starting to include this capability. It's not trivial, though, because it involves disk access from within JavaScript code.

A Brief Tour of AJAX Patterns

The next evolution of Web-based applications leads straight to AJAX and, more generally, Rich Internet Applications (RIA). Initially leading this next evolution are three main categories of applications: classic HTML-based sites, mashups built to integrate multiple systems in a single Web-based front end, and thick clients.

For the first category, partial user interface loading (partial rendering) represents the easy way to implement AJAX with limited impact on existing code and skills.

The concept of a mashup, on the other hand, does not necessarily lend itself to AJAX and improved user interaction. It is merely a way to collect data from a variety of sources and put that together into a coherent and cohesive user interface. This can also happen in a classic server-to-server scenario. However, AJAX makes it simpler and, frankly, cooler. From an AJAX perspective, a mashup application needs standard data serialization formats (syndication), a lightweight framework for calling remote services via script, an updatable DOM, and maybe a few rich visual controls with a convenient programming model.

Building thick clients with AJAX is the biggest challenge. A thick client can be the front end of a distributed enterprise system or perhaps the presentation layer of a line-of-business application. It can also be a standalone application that the IT department decided to expose as a Web application. These applications, whether published to the Internet or restricted to an intranet, need the richness and speed of a regular desktop UI.

Compared to Windows development, the Web is a step backward in terms of interactivity and responsiveness. With AJAX, you finally have the tools (and environmental conditions) to evolve toward a substantially different model. But there is a significant trade-off. On the one hand, you have millions of users and developers accustomed to the old Web and its paradigm of history, offline navigation, favorites, single operations, page transitions, and permanent links. On the other hand, you have AJAX and its paradigm of simultaneous operations and a single auto-updating user interface. In this case, for user tasks, the AJAX model requires a true undo model as opposed to simply using the browser's history and page navigation features.

To code against the SPI model, a new set of design patterns is necessary. Figure 4 lists some of the most popular AJAX patterns. As you can see, most of them are focused on user interface techniques and arrangements, though you should note that the list doesn't include some popular AJAX patterns and practices that are already implemented in the Microsoft® AJAX Client Library. For example, you don't need an AJAX stub, a cross-browser JavaScript model, or a call-tracking model if you use ASP.NET AJAX—all these features are available out of the box.

Figure 4 Some AJAX Patterns (Close Figure 4)

Pattern | Goal
Browser-Side Templating | The pattern suggests using HTML templates that will be fleshed out dynamically with data retrieved from remote HTTP endpoints. Instead of regenerating the HTML layout for the data on the fly and on a per-request basis, the pattern suggests that you set up your own template layer. This pattern represents an alternative to the HTML Message pattern.
Cross-Domain Proxy | The pattern operates a server-to-server connection to a reachable and publicly exposed service and carries data back to the client. From the client browser, an AJAX application is not allowed to connect to any URL outside the domain of the page. However, a local proxy located in the same domain can easily grab data from anywhere and flow it back to the caller.
Heartbeat | Because most AJAX applications may be doing a lot of work on the client without ever posting back, there might be the need to inform the server that a given client is still active. The pattern suggests that the client application periodically uploads a "heartbeat" message to indicate the application is still loaded and working in the browser.
HTML Message | Remote HTTP endpoints normally return JavaScript Object Notation (JSON) data on the client to be integrated in the existing DOM. This task can only be accomplished via JavaScript. However, if the client code is particularly complex, or for performance reasons, you might want to return HTML (data and layout) from the server rather than plain data.
Microlink | AJAX is primarily used for doing a lot of activity within the same page. How would you reference external contents, namely the contents that would go to a different page in a classic Web application? You need a sort of in-page hyperlink, or a microlink. The microlink is a reference to a chunk of markup that gets retrieved via server calls and then inserted in the page. A microlink can be an HTTP endpoint or perhaps a method on a JavaScript command object.
On-Demand JavaScript | This is the JavaScript counterpart of the popular lazy loading pattern often used in a data access layer. Downloading all the required JavaScript during page initialization may have a performance impact and slows down the whole process. By loading JavaScript files on a per-request basis, you give your pages a much faster load without affecting functionality.
Page Arrangement | Because most of the application's activity takes place in the same page, you need to update the page's content and present new information as the context changes. This pattern simply suggests that you use the DOM to add/remove or show/hide elements to reflect state transitions.
Periodic Refresh | The browser periodically schedules a request to gain new and up-to-date information to refresh the user interface.
Popup | The pattern represents the Web version of modal/modeless Windows dialog boxes. A popup consists of a some HTML content displayed in front of the existing content for a relatively short period of time or until the user dismisses it.
Predictive Fetch | The pattern suggests anticipating the most likely user actions and fetching the required data beforehand. Implementation of this pattern comes at a cost: it's ultimately a guess, and you may be wrong. While effective for increasing perceived performance, this pattern may also result in a loss of performance if poorly implemented or implemented in a less than ideal scenario due to heavy server bandwidth consumption.
Progress Indicator | The pattern used to monitor the progress of server operations. The idea is that the server operation writes its own progress to a shared location that a client-side monitoring service can read in the context of a progress refresh.
Submission Throttling | One of the potential drawbacks of AJAX is that too many requests can be generated to the server in a unit of time. If this is the case, there's a clear scalability problem. This pattern suggests that you use a timer to periodically upload data to the server and a local cache or queue to accumulate requests.
Timeout | In case of heavyweight operations conducted from the client such as streaming or periodic refresh, there's the problem of ensuring that each connected client is really using the application. This pattern suggests that you time out the heavy operation and resume only if the user explicitly requires that.
Unique URLs | The pattern lets you assign distinct URLs to different parts of your application that typically reflect different states. This pattern is commonly used to support history in AJAX applications.
Virtual Workspace | The server needs to respond to requests as quickly as possible, but it can't necessarily return all of the data available for bandwidth reasons. This pattern suggests that you build a virtual user interface that transmits the idea that all data is available while only a small fraction of it is really on the client. It is the application's responsibility to download data on demand and cache it locally.

You should also notice that patterns listed in Figure 4, and AJAX patterns in general, are more reference patterns than design patterns. They point out common ways of doing things, but not all of them are design issues.

In Figure 4, I briefly outlined the gist of each pattern. For the rest of this column, I'm going to drill down into a couple of these patters. In future columns, I'll return to other significant patterns to discuss their driving forces and demonstrate some practical implementations. (A great Web site to learn more about AJAX patterns is ajaxpatterns.org.)

The Unique URL Pattern

URLs are the essence of the Web. Users save URL favorites for future reference, follow URLs to new content experiences, and use URLs to move back to a previous state. With AJAX and the SPI model, an application may accomplish a number of tasks from within a single URL. And this threatens to invalidate a central pillar of the Web experience: that a discrete state of an application is identified by a distinct URL.

Browsers build their own caches of URLs as users navigate. But with AJAX, many operations don't go through the browser and aren't cached in the list of visited URLs from which the Back and Forward menus are driven. On the other hand, client browsers don't expose a programming model for JavaScript code to add URLs to the list. The browser object model only supplies methods to navigate back and forward within the existing list.

The Unique URLs pattern assigns a unique and expressive URL to each significant application state. For example, if the user clicks to edit a value in an AJAX page, a new URL should be added to the browser cache, even though the operation occurred via XMLHttpRequest from within the same page.

You change the URL without reloading the page by using the following JavaScript:

window.location.hash = stateInfo;

The effect of this code is adding a #-prefixed fragment to the URL, as shown here:

http://www.contoso.com/shopping.aspx#edit-1234

Using this pattern, the URL actually changes when you start any given AJAX operation and thus enables changes in the application state to be tracked by the browser.

However, there's more to it than just capturing the URL. When a browser is directed to a hash-based URL, it first loads the main URL and then looks up for a page fragment with the hash name. In an AJAX context, the hash name doesn't point to a real page fragment but instead to application-specific information that represents the current state. For example, edit-1234 may indicate that you were editing an item with an ID of 1234. The actual format is entirely up to you.

If it can't find the proper fragment, the browser just ignores the URL hash. As a result, the user has the page loaded but not necessarily in the expected application state. A second trick is required. You should intercept the page's onload event, parse the URL, extract the hash, and run any JavaScript code that is required to put the page into the desired state, like so:

window.onload = function() {
    checkAndParseURL();
}

checkAndParseURL() {
     var state = window.location.hash;
     restorePage(state);
}

A similar approach is implemented in the history support available with ASP.NET 3.5 Extensions. You can learn more about this at quickstarts.asp.net/3-5-extensions/ajax. The solution in ASP.NET 3.5 Extensions is fully integrated in the framework. You'll see it in the form of new properties and events added to the ScriptManager control. But in the end, it is an implementation of the Unique URLs pattern.

In addition, it should be noted that the trick based on the URL's hash does not work in Internet Explorer® because Internet Explorer doesn't recognized hash changes for URLs, except for inline frames. In fact, all browsers have different peculiarities in the way that they handle fragment navigation (see weblogs.asp.net/bleroy/archive/2007/09/07/how-to-build-a-cross-browser-history-management-system.aspx for more on this topic). The solution in ASP.NET 3.5 Extensions takes care of this difference and qualifies it as a true cross-browser trick.

The Timeout Pattern

One of the biggest benefits of AJAX is the possibility of implementing real-time page updates. However, live updates, if misused, can become a serious danger for the application. Imagine a scenario where the user displays a live page that polls the server every few seconds to update some contents. Assume that the user leaves for hours without shutting down the browser. As a result, the page keeps on sending requests, generating a significant—and useless—workload for the server.

How can you determine whether a client session is timed out? On the server, there's session timeout, but in AJAX there's also the client session that matters. To detect the end of a client session, you need to check whether there was user activity such as clicking and tapping for a given amount of time. Monitoring keyboard and mouse activity might be heavy; a simpler and lighter approach based on timers is generally more effective.

To detect the end of a session based on a timer, set up a client timer that expires after a specified number of seconds (or, more likely, minutes), stops the ongoing task, and pops up an alert box. If the user responds to the prompt, processing resumes as usual.

Figure 5 shows a piece of JavaScript that illustrates the essence of the Timeout pattern. The sample page embeds a clock. The clock is obtained using a Label control in an UpdatePanel that is periodically updated by a Timer control, like so:

Figure 5 Implementing a Client Session Timeout (Close Figure 5)
<script type="text/javascript">
    var timer = null;
    function pageLoad()
    {
        if (timer === null)
        {
            timer = new Samples.TaskTimer(5000, stopTask);
            timer.start();
        }
    }

    function pageUnload()
    {
        if (timer != null)
            timer.stop();
    }

    function stopTask()
    {
        // Stop the clock
        var clock = $find("<%= Timer1.ClientID%>");
        clock._stopTimer();

        AskIfTheUserWantsToContinue();
    }

    function AskIfTheUserWantsToContinue()
    {
        // Ask if the user wants to continue
        var answer = window.confirm(
          "Is it OK to continue with the clock?");
        if (answer)
        {
            // Restart the task
            var clock = $find("<%= Timer1.ClientID%>");
            clock._startTimer();

            // Restart our own timeout engine
            if (timer !== null)
                timer.start();
            return;
        }
    }
</script>

protected void Timer1_Tick(object sender, EventArgs e)
{
    Label1.Text = DateTime.Now.ToLongTimeString();
}

The clock represents a heavy task that could flood the server with requests. The idea is to set up a timer that periodically asks the user if he really wants to continue with the clock. The timeout code first stops the clock and then displays the message box. Based on the user's response, the clock is restarted.

The code utilizes the $find function to locate an ASP.NET AJAX component—in this case, it happens to be the client object model of the ASP.NET Timer server control. Figure 6 shows the page in action with a popup asking the user if it's OK to continue.

Figure 6 Asking the User Whether to Continue
Figure 6 Asking the User Whether to Continue(Click the image for a smaller view)
Figure 6 Asking the User Whether to Continue
Figure 6 Asking the User Whether to Continue (Click the image for a larger view)

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is the author of Programming ASP.NET 3.5 Core References. Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Test Run: Request/Response Testing with Windows PowerShell

Test Run
Request/Response Testing with Windows PowerShell
Dr. James McCaffrey

Code download available at: TestRun2008_05.exe (152KB)
Contents
The Application under Test
Fundamental ASP.NET Request/Response Testing
Inside the Helper Function
Extracting ViewState and EventValidation
Working with the ViewState and EventValidation Values
Extending the Request/Response Automation

The most fundamental form of Web testing is HTTP request/response testing. This involves programmatically sending an HTTP request to the Web application, fetching the HTTP response, and examining the response for an expected value.

When you need to perform request/response testing for ASP.NET applications, there are a number of approaches from which you can choose. You can write a C# or Visual Basic® console application. You can write JavaScript or Perl scripts. If you're a glutton for punishment you can write a C/C++ program. And nowadays, you can use Windows PowerShell™.

I've used many of these techniques, and Windows PowerShell has become my preferred method for lightweight HTTP request/response testing. I can simply write a small Windows PowerShell script that programmatically posts information to an ASP.NET application on a Web server. The script can then retrieve the HTTP response and examine it for an expected value to determine a pass/fail result.

In this month's column, I will show you how to use Windows PowerShell to perform request/response testing for an ASP.NET application. I'll assume you have at least some basic familiarity with ASP.NET and with Windows PowerShell. But even if these topics are new to you, you should still be able to follow this discussion without too much trouble.

Figure 1 shows the simple but representative ASP.NET Web application named MiniCalc. As you can see, the MiniCalc application accepts two integers, computes either a sum or a product, and displays the result out to four decimals places. Real Web applications are significantly more complex than this simple sample app, but the techniques I'm discussing will easily scale to more complex applications.

Figure 1 Simple ASP.NET Web Application under Test
Figure 1 Simple ASP.NET Web Application under Test(Click the image for a smaller view)
Figure 1 Simple ASP.NET Web Application under Test
Figure 1 Simple ASP.NET Web Application under Test (Click the image for a larger view)

Figure 2 shows my Windows PowerShell test script in action. For simplicity, I have hard-coded all of the information regarding the application being tested. I will explain how you can parameterize the script later. The script first echoes the target URL of http://localhost/MiniCalc/Default.aspx and then performs a probing request for the Web app's initial ViewState and EventValidation values. These are key values, which I'll explain shortly.

Figure 2 Windows PowerShell Test Script in Action
Figure 2 Windows PowerShell Test Script in Action(Click the image for a smaller view)
Figure 2 Windows PowerShell Test Script in Action
Figure 2 Windows PowerShell Test Script in Action (Click the image for a larger view)

Next my script programmatically posts information that simulates a user typing 5 into TextBox1, 3 into TextBox2, selecting RadioButton1 (to indicate an addition operation), then clicking on Button1 (to calculate). The script captures the HTTP response from the Web server, displays the entire response, and searches the response to see if 8.0000 (the result of 5 + 3) is in TextBox3, the control created to display the calculation result.

The Application under Test

The target of my test automation is a basic Web application called MiniCalc, which I first introduced in my March 2008 column (see msdn2.microsoft.com/magazine/cc337896.aspx). The complete code for my MiniCalc Web application is shown in Figure 3.

Figure 3 MiniCalc Source Code (Close Figure 3)
<%@ Page Language="C#" %>
<script language="C#" runat="server">
  private void Button1_Click(object sender, System.EventArgs e)
  {
    int alpha = int.Parse(TextBox1.Text.Trim());
    int beta = int.Parse(TextBox2.Text.Trim());

    if (RadioButton1.Checked) {
      TextBox3.Text = Sum(alpha, beta).ToString("F4");
    }
    else if (RadioButton2.Checked) {
      TextBox3.Text = Product(alpha, beta).ToString("F4");
    }
    else
      TextBox3.Text = "Select method";
  }
  private static double Sum(int a, int b) {
    double ans = a + b;
    return ans;
  }
  private static double Product(int a, int b) {
    double ans = a * b;
    return ans;
  }
</script>

<html>
  <head>
    <style type="text/css">
      fieldset { width: 16em }
      body { font-size: 10pt; font-family: Arial }
    </style>
    <title>Default.aspx</title>
  </head>
  <body bgColor="#ccffff">
    <h3>MiniCalc by ASP.NET</h3>
    <form method="post" name="theForm" id="theForm" runat="server"
       action="Default.aspx">
      <p><asp:Label id="Label1" runat="server">
         Enter integer:&nbsp&nbsp</asp:Label>
        <asp:TextBox id="TextBox1" width="100" runat="server" />
      </p>
      <p><asp:Label id="Label2" runat="server">
          Enter another:&nbsp</asp:Label>
         <asp:TextBox id="TextBox2" width="100"  runat="server" />
      </p>
      <fieldset>
        <legend>Arithmentic Operation</legend>
        <p><asp:RadioButton id="RadioButton1" GroupName="Operation"
            runat="server"/>Addition</p>
        <p><asp:RadioButton id="RadioButton2" GroupName="Operation"
            runat="server"/>Multiplication</p>
        <p></p>
      </fieldset>
      <p><asp:Button id="Button1" runat="server" text=" Calculate "
        onclick="Button1_Click" /> </p>
      <p><asp:TextBox id="TextBox3" width="120"  runat="server" /></p>
    </form>
  </body>
</html>

The most important parts of this source code for you to note are the IDs of my ASP.NET server controls. I use default IDs: Label1 (user prompt), TextBox1 and TextBox2 (input for two integers), RadioButton1 and RadioButton2 (to choose addition or multiplication), Button1 (calculate), and TextBox3 (result). To perform automated HTTP request/response testing on an ASP.NET application, you need to know the IDs of the application's controls. And while you can view the IDs of my test application via the downloadable source code, even if you are testing a Web application you didn't write, you can examine the IDs by viewing the application's source using the Internet Explorer® View | Source command.

The action attribute of my <form> element is set to Default.aspx. Every time a user submits a request, the same Default.aspx code is executed. Because HTTP is a stateless protocol, ASP.NET gives the effect of being a stateful application by maintaining the application's state in a hidden value called ViewState.

A ViewState value is a Base64-encoded string that maintains the state of an ASP.NET page through multiple request/response round-trips. ASP.NET uses hidden input controls to maintain the ViewState. Similarly, an EventValidation value has been added in ASP.NET 2.0 that is used for security purposes to help prevent script-insertion attacks. These two mechanisms are key to programmatically posting data to an ASP.NET 2.0 Web application.

Fundamental ASP.NET Request/Response Testing

The Windows PowerShell script that produced the screenshot in Figure 2 illustrates all the fundamental techniques you need to perform basic HTTP request/response testing for ASP.NET Web applications. And this script can serve as a foundation for more sophisticated testing.

Notice that even though my script is in the current directory, for security reasons I must still specify the script's location (.\test.ps1 rather than simply test.ps1). Also note that, by default, Windows PowerShell does not allow script execution at all. To enable script execution, I entered the command Set-ExecutionPolicy unrestricted in a previous Windows PowerShell session. The PowerShell execution policy stays in effect for a particular user across all new shell instances and user sessions, until the policy is explicitly changed.

The overall structure for my test.ps1 script is:

# file: test.ps1
function main()
{
  # send request, get response, check response
}
function getVSandEV($url)
{
  # get initial ViewState and EventValidation values
}
main
# end script

The first line is a Windows PowerShell comment. Next I have a main function that does most of the work, followed by a helper function named getVSandEV. (I'll dig into this helper function in a moment.) The use of a "main" function is optional. And note that main is not a reserved word, so I could have used some other identifier if I had wanted.

In terms of script organization, Windows PowerShell is quite flexible—I could have placed all my code in test.ps1, without any program-defined functions, or I could have broken my main function down a bit further by writing additional helper functions.

Notice the second-to-last line of my script. Since main is not a predefined program entry point, I must explicitly invoke the main function.

Inside my main function, I begin with this statement:

$url = "http://localhost/MiniCalc/Default.aspx"

This assigns a string value to a variable named $url. (Technically speaking, $url is an object, but, even so, it is considered acceptable to call simple Windows PowerShell objects variables.) Windows PowerShell variables begin with the $ character, making them quite easy to identify in a script.

The Windows PowerShell scripting language is an explicitly typed language that supports type inference. If a Windows PowerShell variable is not explicitly typed, the Windows PowerShell execution engine will do its best to infer a type. In this case, variable $url will be interpreted as a string variable because of the double-quote characters delimiting its value. But I could have supplied an explicit type qualifier like this:

   [string] $url = `
     "http://localhost/MiniCalc/Default.aspx"

Since Windows PowerShell is built on the Microsoft® .NET Framework, I can use any .NET type (such as Int32 or String) or equivalent C# mapping alias (such as int or string). Adding an explicit type to a variable provides some runtime error checking and improved readability, but at the expense of some extra typing.

My second statement prints a message to the shell, shown here:

   write-host `
      "`nWeb Application under test is: " $url

Here I use the Write-Host cmdlet to display a message. Windows PowerShell uses both single quotes and double quotes for strings. I have used double quotes here so I can embed the `n sequence, which is the Windows PowerShell method of placing a "new line" character in a string. (Strings that are delimited by single quotes are considered literals.)

Inside the Helper Function

Next I call my program-defined function to determine the initial ViewState and EventValidation values for the MiniCalc Web app:

$vs,$ev = getVSandEV($url)

This indicates that I am assigning two return values from the getVSandEV function to two variables, $vs and $ev. (Actually, as I'll explain in a moment, the function's return value is a single array with two values.)

The entire getVSandEV helper function is shown in Figure 4. The getVSandEV function returns initial ViewState and EventValidation values for the specified ASP.NET Web page. I have not specified a type for my input parameter, $url, but I could have explicitly typed the parameter for a measure of runtime error checking.

Figure 4 Code for getVSandEV Helper Function (Close Figure 4)
function getVSandEV($url)
{
  $wc = new-object net.WebClient
  $probe = $wc.downloadData($url)
  $s = [text.encoding]::ascii.getString($probe)

  $start = $s.indexOf('id="__VIEWSTATE"', 0) + 24
  $end = $s.indexOf('"', $start)
  $vs = $s.substring($start, $end-$start)

  write-host "`nInitial ViewState is: " $vs

  $start = $s.indexOf('id="__EVENTVALIDATION"', 0) + 30
  $end = $s.indexOf('"', $start)
  $ev = $s.substring($start, $end-$start)

  write-host "`nInitial EventValidation is: " $ev

  return ($vs,$ev)
}

The first line in the body of getVSandEV instantiates a WebClient object, shown here:

$wc = new-object net.WebClient

I use the new-object cmdlet to create a WebClient object (the WebClient class is part of the System.Net namespace). I could also have explicitly specified a type for my $wc object, like so:

[net.WebClient] $wc = new-object net.WebClient

The WebClient class provides an easy way to send an HTTP request and retrieve the resulting HTTP response. I can accomplish this with a single statement:

$probe = $wc.downloadData($url)

I call the DownloadData method of the WebClient class, and the return value is a byte array that holds the entire HTTP response from the application at the given URL. I store this response into a byte array named $probe, which I could have also explicitly typed as [byte[]] $probe. Next I convert the HTTP response from an array of bytes to a string:

$s = [text.encoding]::ascii.getString($probe)

The syntax here is a bit tricky at first glance: text.encoding is a class—the Encoding class of the System.Text namespace. The ascii part is a static property of the Encoding class and the getString part is a method. I could have converted my byte array to a string in two steps instead of one, like this:

$e = [text.encoding]::ascii
$s = $e.getString($probe)

Extracting ViewState and EventValidation

At this point, variable $s holds the entire HTTP response as a string. This string includes the ViewState value and looks something like the following:

<input type="hidden" name="__VIEWSTATE"
 id="__VIEWSTATE" value="/wEPDwUK ... Bbzmv" />

Now I need to extract the ViewState value from $s. I identify the index position within $s where the ViewState value begins:

$start = $s.indexOf('id="__VIEWSTATE"', 0) + 24

As shown above, I use the IndexOf method to search for the string id="__VIEWSTATE". The 0 argument means begin searching $s at index position 0—the beginning of the response string. Notice that since my target string contains double quote characters, I delimit the target string using single quote characters. Once I find where my target string begins, the actual ViewState value starts 24 characters from that index. (Note that "__VIEWSTATE" has two leading underscore characters, not just one.)

This is a pretty crude way to parse for the initial ViewState value, and it makes my code more brittle. However, I'm just performing quick and easy test automation, and because I can quickly and easily modify my script, I'm willing to accept the risk that my code may break if the storage format for the view state changes.

Now I find where within $s the ViewState value ends:

$end = $s.indexOf('"', $start)

I search for the first occurrence of any double quote character found after the beginning of the ViewState value and store that location into variable $end. Now that I know where the ViewState value begins and ends within the response string $s, I can extract the value using the SubString method:

$vs = $s.substring($start, $end-$start)
write-host "`nInitial ViewState is: " $vs

The two arguments to the SubString method are the index within $s from which to begin extracting and the number of characters to extract (not the index to end extracting, which is a common mistake). At this point, variable $vs holds the initial ViewState value for the target page. Extracting the EventValidation value follows the same pattern:

$start = $s.indexOf('id="__EVENTVALIDATION"', 0) + 30
$end = $s.indexOf('"', $start)
$ev = $s.substring($start, $end-$start)
write-host "`nInitial EventValidation is: " $ev

Here I reuse my $start and $end variables. The only difference between the two is that the EventValidation value in the response string $s begins 30 characters after the start of the id= "__EVENTVALIDATION" string.

Now I have two values, $vs and $ev, that I want to return from my getVSandEV function. To do this, I use a simple but powerful Windows PowerShell function:

return ($vs,$ev)

The parentheses specify a Windows PowerShell array. And since Windows PowerShell is based on a pipeline architecture, I can omit the return keyword if I want. But I like to use "return" for clarity since it is a familiar programming idiom.

Working with the ViewState and EventValidation Values

Now back to the main function in my test.ps1 script. The next three statements are:

[Reflection.Assembly]::LoadFile( `
  'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\System.Web.dll')`
  | out-null

$vs = [System.Web.HttpUtility]::UrlEncode($vs)
$ev = [System.Web.HttpUtility]::UrlEncode($ev)

Both my ViewState and EventValidation values are base64-encoded strings. Base64 encoding uses 64 characters, including uppercase A-Z, lowercase a-z, digits 0-9, the + character, and the / character. (The = character is used in Base64 for padding.) Because some of these characters are not permitted in an HTTP request stream, I therefore need to perform a UrlEncode operation on my ViewState and EventValidation values.

UrlEncoding converts potentially troublesome characters into a three-character sequence beginning with the % character. For example, a raw > character is encoded as %3D and a blank space is encoded as %20. When a Web server receives an HTTP request containing any of these special three-character sequences, the server decodes the sequences back to raw input.

The .NET Framework has a static UrlEncode method in the HttpUtility class, which is in the System.Web assembly. However, unlike many .NET namespaces, System.Web is not accessible to Windows PowerShell by default. So, before I can use UrlEncode, I must load the System.Web assembly. To do this, I use the static LoadFile method in the System.Reflection namespace, and pipe that command to the out-host cmdlet to suppress noisy progress messages as System.Web is loaded and registered.

Now I can prepare my post data. HTTP post data is a set of name-value pairs connected by the ampersand character, like so:

$postData = `
  'TextBox1=5&TextBox2=3&Operation=RadioButton1&Button1=clicked'
write-host "`nPosting: " $postData

The first two name-value pairs of the post string are fairly self-explanatory and simply set TextBox1 and TextBox2 to 5 and 3, respectively. The third name-value pair, Operation=RadioButton1, is how I simulate a user selecting a RadioButton control—in this case, the RadioButton that corresponds to addition. You might have guessed incorrectly (as I originally did) that the way to set the radio button would be to use a syntax like RadioButton1=checked. But RadioButton1 is a value of the Operation control, not a control itself.

The fourth name-value pair, Button1=clicked, is somewhat misleading. I need to supply a value for Button1 to simulate that it is being clicked, but any value will work. I could have used Button1=foo (or even just Button1=), but Button1=clicked provides more clarity.

Next I add my Url-encoded ViewState and EventValidation values to the post data and convert to a byte array:

$postData += '&__VIEWSTATE=' + $vs
$postData += '&__EVENTVALIDATION=' + $ev
$buffer = [text.encoding]::ascii.getbytes($postData)

Windows PowerShell supports the familiar += operator for string concatenation. Note that both of these strings have two leading underscore characters. The GetBytes method accepts a string and returns that string as an array of bytes.

Now I create an HttpWebRequest object, as shown here:

[net.httpWebRequest] $req = [net.webRequest]::create($url)
$req.method = "POST"
$req.ContentType = "application/x-www-form-urlencoded"
$req.ContentLength = $buffer.length
$req.TimeOut = 5000

To create an HttpWebRequest object, use an explicit Create method. Note that the Create method belongs to the System.Net.WebRequest class. Since this is potentially confusing, I explicitly supply a type ([Net.HttpWebRequest]) for my $req object.

The next four lines of code prepare the HttpWebRequest object by specifying the type (POST), content type (application/ x-www-form-urlencoded), content length (calculated on the fly), and a server time-out value (5000, which is 5,000 milliseconds or 5 seconds). Now I send the HTTP request to my MiniCalc Web application, like this:

$reqst = $req.getRequestStream()
$reqst.write($buffer, 0, $buffer.length)
$reqst.flush()
$reqst.close()

The HttpWebRequest class does not have a Send method. So I create a request stream object from my HttpWebRequest object and then write to the stream. The Write method prepares the request; the Flush method, then, actually causes the HTTP request to be sent. Because the Close method performs a Flush automatically, I could have also omitted the $reqst.flush() statement.

Now I need to capture the HTTP response, like this:

[net.httpWebResponse] $res = $req.getResponse()
$resst = $res.getResponseStream()
$sr = new-object IO.StreamReader($resst)
$result = $sr.ReadToEnd()
write-host "`nHTTP Response is " $result

In these lines, I first create an HttpWebResponse object ($res) from my HttpWebRequest object ($req). Here I explicitly supply a type for clarity. Then I create a System.IO.Stream object ($resst). I then create a StreamReader object to process the response stream and use the ReadToEnd method to capture the entire HTTP response as a string into variable $result. This programming paradigm may appear a bit awkward at first, but the pattern quickly becomes familiar after you've used it a few times.

Now that I have my HTTP response, I can examine it for an expected value:

write-host "`nLooking for 8.000"
if ($result.indexOf('value="8.0000"') -ge 0) {
  write-host "`nFound 8.0000 -- Test scenario Pass`n" `
    -foregroundcolor green
}
else {
  write-host "`nDid not find 8.0000 -- Test scenario FAIL`n" `
    -foregroundcolor red
}

I use the String.IndexOf method to look for value=8.0000 in TextBox3 (since I posted data that equals 5 + 3). Note that Windows PowerShell uses Boolean comparison operators such as -ge (rather than >=), -eq (rather than ==), and so on. The last two lines of my test automation script clean up the resources:

$sr.close()
$resst.close()

I simply close the StreamReader object and the associated request Stream. Of course, I could have done this earlier (before determining my pass/fail result).

Extending the Request/Response Automation

The techniques I've discussed provide all the basics you need to perform lightweight HTTP request/response testing of an ASP.NET Web application. However, there are many ways you can extend this code to create a simple test harness (see Figure 5).

Figure 5 A Simple Test Harness
Figure 5 A Simple Test Harness(Click the image for a smaller view)
Figure 5 A Simple Test Harness
Figure 5 A Simple Test Harness (Click the image for a larger view)

You may want to parameterize your Windows PowerShell scripts. In my example, the URL of the target app is hardcoded into the script. However, you can write your script to accept a URL from the command line:

   #file: harness.ps1
   param($url = $(throw "URL required"))
   (etc.)

You could then call your script, like this:

   PS C:\MyScripts> .\harness.ps1 `
    'http://localhost/MiniCalc/Default.aspx'

If you use the param statement, it must be the first non-comment statement in a Windows PowerShell script. The throw statement makes a command-line argument required—if an argument is not given, the script will throw an exception.

You might also want to parameterize your test case data. One simple approach would be to place the test case data inside your script in an array like the following:

$cases = (
  ('0001,TextBox1=5&TextBox2=3&Operation=' +
    'RadioButton1&Button1=clicked,value="8.0000"'),
  ('0002,TextBox1=5&TextBox2=3&Operation=' +
    'RadioButton2&Button1=clicked,value="15.0000"'),
  ('0003,TextBox1=0&TextBox2=0&Operation=' +
    'RadioButton1&Button1=clicked,value="0.0000"')
)

The @ character forces variable $cases to be an array type. But it really is optional here since there is no potential ambiguity in this scenario. Then you can iterate through each test case, like this:

foreach ($case in $cases) {
  $caseID,$input,$expected = $case.split(",")
  # get ViewState and EventValidation values
  # construct post data
  # send HTTP request, get response
  # determine pass/fail
}

The foreach loop will extract each string in turn from array $cases and store the current string into $case. Then the split method breaks the current string at each comma character and stores the results into variables $caseID, $input, and $expected.

Windows PowerShell has several error-handling features, including a trap statement to deal with exceptions. For example, in test automation, you typically want your harness to continue executing even if one test case throws an exception. To do this in Windows PowerShell, you can use the trap and continue statements:

function main(){
  # test cases here
  foreach ($case in $cases) {
    # run test case
    trap {
      write-host "Fatal exception caught!"
      continue
    }
  }
}

Of course, there are many other modifications you can employ, such as keeping track of the number of test cases that pass and fail, writing test results to an external file or database, and so on.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products, including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes, and he can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Office Space: From VBA Macro to Word Add-in

Office Space
From VBA Macro to Word Add-in
Robert Bogue

Code download available at: OfficeSpace2008_05.exe (166KB)
Contents
Book Learning
Macro Basics
Moving Macros to VSTO
Writing New Add-In Code
More Buttons

Document automation isn't a new functionality by any means. It has been going on since the invention of macros, and there has been a full-fledged programming model within Microsoft® Office applications since the early 1990s. For many years, the Office tools have featured macro recorders that weren't limited to simply replaying keystrokes but have been capable of writing code on the fly as well. Macros have even demonstrated that they are functional enough to write viruses. And while writing a virus using a Word macro may be a dubious feat, it is a feat nonetheless.

However, even with all of the capabilities of Visual Basic® for Applications (VBA), there are some things it just doesn't do very well. In the early days of VBA, XML hadn't yet been invented, the Internet was in its infancy, and the first HTML pages were just beginning to appear. As such, it's no wonder that making a Web service call isn't well handled in the context of Visual Basic for Applications.

The Microsoft .NET Framework CLR, on the other hand, understands these technologies quite well. The ability to call Web services is just one among a number of reasons for wanting to write .NET-targeted code rather than VBA—and doing so means using Visual Studio® Tools for Office (VSTO). But switching from VBA to VSTO doesn't have to be a case of throwing the baby out with the bath water; rather, it can be just a natural way to extend how you are already developing solutions in Office.

In this column you'll see how to use Word to capture some VBA code that solves a basic problem. Then I will use the latest version of VSTO included with Visual Studio 2008 to wrap this code into a deployable Word add-in. I'll also write some simple code for some of the tasks that the macro recorder either just can't record or doesn't record in an optimal way.

Book Learning

Recently, I found myself in a situation where this kind of VBA-to-VSTO conversion was perfect for the problem at hand. I was finishing my latest book, The SharePoint Shepherd's Guide for End Users. The book is self-published, and I needed to output the manuscript from Word into a PDF that the printer could use. For that to work, there were several steps I first needed to complete to get the manuscript ready.

First, I had to assemble multiple files into one large document. With more than 140 individual files representing 13 sections and 116 tasks, munging them together wasn't something I wanted to do by hand. Having so many individual files was great when working with Microsoft SharePoint® because it allowed for individual tracking of each task through the editorial workflow, but the sheer number made assembly a task best handled by automation.

Second, I wanted to make sure that all of the tracked changes in all of the documents had been accepted. During the editing process, I used revision marks (via the Track Changes feature in Word) to keep track of editing and other changes to the manuscript. The revision marks are supposed to all be accepted as part of the final checkout of the content, but if I missed some, the formatting of the revision marks would stand out in the final book, which wouldn't be very professional looking.

Third, I needed to remove any remaining comments in the documents. There were comments between the various editors about how things worked and how to standardize some of the terminology. Again, these should all have been removed during an earlier review, but a bit of automation to verify that is prudent. Like the revision marks, the comments were not something I wanted to see in the final product.

I am going to illuminate the process of creating each of these pieces and the relative ability of the macro recorder to capture the code needed for each function. I'll start by using the recorder to generate the basic automation code. From there I'll take a closer look at workable but sub-optimal generated code. Finally, I'll look at the recorder not generating code at all. I'll convert the VBA code into VSTO code and put it into a Word add-in that I can use to assemble the final book. Just to make the process more challenging, I'll convert the code from VBA to C# in the course of bringing it into VSTO.

Macro Basics

The first step in this process is to make sure the Developer tab is visible in Word so I am able to record the macros. Click the Office button, then click the Word Options button at the bottom of the menu. In the Word Options dialog box, check the box next to Show Developer tab in the Ribbon (see Figure 1) and click OK. The developer tab will appear in the Ribbon.

Figure 1 Enabling the Developer Tab in the Ribbon
Figure 1 Enabling the Developer Tab in the Ribbon(Click the image for a smaller view)
Figure 1 Enabling the Developer Tab in the Ribbon
Figure 1 Enabling the Developer Tab in the Ribbon (Click the image for a larger view)

Let's record opening a file and adding the contents of that document into a new file. To do this, click the Record Macro button on the Developer tab (see Figure 2). In the dialog that appears (as shown in Figure 3), enter the name AddFiles and click OK. The cursor icon changes to a pointer with a little cassette tape next to it.

Figure 2 Record a Macro from the Developer Tab
Figure 2 Record a Macro from the Developer Tab(Click the image for a smaller view)
Figure 2 Record a Macro from the Developer Tab
Figure 2 Record a Macro from the Developer Tab (Click the image for a larger view)
Figure 3 Naming the Macro
Figure 3 Naming the Macro

Now, just open a file as you normally would and copy the contents of the document. Then create a new blank document and paste the copied text into the new document. Click the Stop Recording button that appears where the Record Macro button was on the Developer tab. When you're finished you can click the Visual Basic button on the far left of the Developer tab to launch the Visual Basic (for Applications) editor. What you should see is a new AddFiles function that looks similar to Figure 4.

Figure 4 AddFiles VBA Macro (Close Figure 4)
Sub AddFiles()
'
' AddFiles Macro
'
'
ChangeFileOpenDirectory _
  "https://sharepoint.contoso.com/sites/sharepoint/" & _
    "Shared%20Documents/SharePoint%20Tasks/"

  Documents.Open fileName:= _
    "https://sharepoint.contoso.com/sites/SharePoint/" & _
      "Shared%20Documents/SharePoint%20Tasks/" & _
      "Task001%20-%20Create%20a%20Team%20Web%20Site.docx", _
    ConfirmConversions:=False, _
    ReadOnly:=False, _
    AddToRecentFiles:=False, _
    PasswordDocument:="", _
    PasswordTemplate:="", _
    Revert:=False, _
    WritePasswordDocument:="", _
    WritePasswordTemplate:="", _
    Format:= wdOpenFormatAuto, _
    XMLTransform:=""

    Selection.WholeStory
    Selection.Copy
    Documents.Add Template:="Normal", NewTemplate:=False, DocumentType:=0
    Selection.PasteAndFormat (wdPasteDefault)
End Sub

Notice that there is an extraneous line that resets the next File Open directory. After that you see the Open command, and then there's the series of commands to copy the text, create a new document, and paste the text into the new document. The code that the macro recording produced isn't perfect, but it isn't bad either. So I'll take that, enhance it, and put it into VSTO.

Moving Macros to VSTO

To get started with VSTO, I first create a Word 2007 ﻿Add-in project. I open Visual Studio, start a new project, use the Word 2007 ﻿Add-in template for the project, and name the project PublishPrep. Upon successful creation of the new VSTO project, your Visual Studio 2008 environment should look similar to Figure 5.

Figure 5 New Word Add-In Project
Figure 5 New Word Add-In Project(Click the image for a smaller view)
Figure 5 New Word Add-In Project
Figure 5 New Word Add-In Project (Click the image for a larger view)

With the project created, I need to create a way for users to access the functionality in the add-in. For 2007 Office system applications, that means creating a Ribbon tab and buttons. First, add a new item to your project and select the Ribbon (Visual Designer) template from the Add New Item dialog. Name your new ribbon PublishPrep.

The next step is to customize the Ribbon. In this case, I'm just going to create a group that will live on the Add-ins tab instead of adding my own tab to the Ribbon. My group will contain three buttons.

Click on the group1 group in the ribbon, then right-click and select properties. In the properties pane, change the name of the group to grpPublishPrep, and change the label to Publishing Preparation, as shown in Figure 6.

Figure 6 Configuring the PublishPrep Ribbon
Figure 6 Configuring the PublishPrep Ribbon(Click the image for a smaller view)
Figure 6 Configuring the PublishPrep Ribbon
Figure 6 Configuring the PublishPrep Ribbon (Click the image for a larger view)

Open the toolbox, scroll down to the Office Ribbon Controls group, and drag three button controls over onto the Publishing Preparation group. Note that the buttons will stack vertically by default.

Right-click the first button and select Properties. Set the name property to btnCreateMaster and the label to Create Master. For the second button, set the name to btnAcceptRevisions and the label to Accept Revisions. Finally, for the third button, set the name property to btnRemoveComments and the label to Remove Comments. The group should look like what's shown in Figure 7.

Figure 7 Buttons and Group Configured for the Add-In
Figure 7 Buttons and Group Configured for the Add-In(Click the image for a smaller view)
Figure 7 Buttons and Group Configured for the Add-In
Figure 7 Buttons and Group Configured for the Add-In (Click the image for a larger view)

Writing New Add-In Code

The macro recorder did a pretty good job of recording the steps necessary to assemble the book from its individual files. However, it didn't include the code necessary to select individual files. For that, I'll take advantage of the standard File Open dialog box. I'll allow the users to select a single file that is a text file containing the names of all of the files to be assembled. There are two major parts to the code. The first part actually appends the file. The second part gets the list of files to append.

The first part of the code, a function called AppendFile (see Figure 8), takes a single parameter, the file name. At a quick glance the code doesn't resemble the code the macro recorder wrote for me, but that's mostly an illusion.

Figure 8 AppendFile (Close Figure 8)
void AppendFile(string file)
{
    if (string.IsNullOrEmpty(file)) return;

    Application app = Globals.ThisAddIn.Application;
    Document activeDoc = app.ActiveDocument;
    if (activeDoc == null) return;

    object fileObj = file;
    object confirmConversions = false;
    object readOnly = true;
    object addToRecentFiles = false;
    object passwordDocument = Missing.Value;
    object passwordTemplate = Missing.Value;
    object revert = true;
    object writePasswordDocument = Missing.Value;
    object writePasswordTemplate = Missing.Value;
    object format = Missing.Value;
    object encoding = Missing.Value;
    object visible = false;
    object openAndRepair = false;
    object documentDirection = Missing.Value;
    object noEncodingDialog = Missing.Value;
    object xMLTransform = Missing.Value;

    Document newDoc = app.Documents.Open(ref fileObj,
        ref confirmConversions, ref readOnly,
        ref addToRecentFiles, ref passwordDocument,
        ref passwordTemplate, ref revert,
        ref writePasswordDocument, ref writePasswordTemplate,
        ref format, ref encoding, ref visible,
        ref openAndRepair, ref documentDirection,
        ref noEncodingDialog, ref xMLTransform);

    app.Selection.WholeStory();
    app.Selection.Copy();
    activeDoc.Select();
    object collapseEnd = WdCollapseDirection.wdCollapseEnd;
    app.Selection.Collapse(ref collapseEnd);
    app.Selection.Paste();

    object saveChanges = WdSaveOptions.wdDoNotSaveChanges;
    object originalFormat = WdOpenFormat.wdOpenFormatAuto;
    object routeDocument = Missing.Value;
    newDoc.Close(ref saveChanges, ref originalFormat,
        ref routeDocument);
    object breakType = WdBreakType.wdPageBreak;
    app.Selection.InsertBreak(ref breakType);
}

The first four lines just get the active document and perform error checking. If there's no active document, you can't exactly append to it, and if you don't have a file name to append, there's not much more you can do. The other thing that these lines are doing is getting references to the application—which is assumed in VBA—and the active document when the button method was called, something the recorded version of the macro didn't need to do.

The next set of lines—the object variable declarations—are necessary because of the way C# makes the call to the COM components that Word exposes. I need to specify missing values, and since the values are all passed by reference, I need a variable to hold them.

The code then performs the same copy operation that the macro recorder generated. The real difference here is that my add-in code is responsible for the management of the active document, including making sure that the cursor position is set to the end of the active document.

The final block of code closes the document I opened, making sure not to save any changes. It also adds a page break after the inserted content because I want to make sure that the contents of the individual files don't run together.

The second part of the code actually gets the list of files to assemble and calls AppendFile (see Figure 9). This isn't code that the macro recorder captured, but it is where you see the power of VSTO because it allows you to leverage all of the .NET constructs. In this case, I'll leverage the OpenFileDialog control, the ability to open and read from a text file, the use of generics to create a list of files, and calling another smaller method that iterates through the list.

Figure 9 VSTO Power Behind the Button (Close Figure 9)
private void btnCreateMaster_Click(object sender,
    RibbonControlEventArgs e)
{
    Document activeDoc = Globals.ThisAddIn.Application.ActiveDocument;
    if ((activeDoc == null)) return;
    didSelect = false;
    dlgOpenFile.DefaultExt = ".TXT";
    dlgOpenFile.FileOk +=
        new System.ComponentModel.CancelEventHandler(
        dlgOpenFile_FileOk);
    dlgOpenFile.ShowDialog();
    if ((didSelect))
    {
        string selectedFile = dlgOpenFile.FileName;
        List<string> files = new List<string>();
        using (StreamReader inFile = File.OpenText(selectedFile))
        {
            string fileLine = inFile.ReadLine();
            while (fileLine != null && !inFile.EndOfStream)
            {
                if ((fileLine.Length > 0))
                {
                    // Process non-blank lines
                    files.Add(fileLine);
                }
                fileLine = inFile.ReadLine();
            }
        }

        if ((files.Count > 0))
        {
            AppendFiles(files);
        }
    }
}

private bool didSelect = false;
void dlgOpenFile_FileOk(object sender,
    System.ComponentModel.CancelEventArgs e)
{
    didSelect = true;
}

void AppendFiles(List<string> files)
{
    foreach (string file in files)
    {
        AppendFile(file);
    }
}

One fairly obvious question about the structure of this code is: why do I bother to create a list of files when they're listed sequentially in the file and it would be nearly as easy to call AppendFile directly from the loop reading the file? I do this to provide future flexibility. One easy-to-see enhancement to this tool would be to allow the user to select the files to append, rather than having a control file that lists the files to append. This is one way that VSTO excels over VBA—because VSTO is built upon the full .NET Framework, you can take advantage of features like generics and create strongly typed parameters.

There are a few more steps needed to take advantage of the code you've just seen: adding the OpenFileDialog and adding the using statements for all of the various namespaces the solution requires. In design mode for the PublishPrep.cs file, go to the Toolbox and, from the Dialogs group, drag the OpenFileDialog onto the Ribbon. Visual Studio will move the OpenFileDialog to a dock at the bottom. Change the name of the dialog to dlgOpenFile, then clear the FileName. Double-click the Create Master button you created earlier. At the top of the file, add using statements, as follows:

using Microsoft.Office.Interop.Word;
using System.IO;
using System.Reflection;

Finally, replace the method stub created by Visual Studio with the code in Figure 9.

At this point I can press F5 and Visual Studio will fire up a copy of Word with the add-in installed. I can click the Create Master button and select a text file from which the code will get the files to add. Once I'm finished testing, I can either exit Word or go into Visual Studio and press Shift-F5 to stop debugging.

More Buttons

With most of the heavy lifting complete, it's time to quickly add the code for the Accept Revisions button. I can go back to the Developer tab in the Word Ribbon and start the macro recorder again to record the step of accepting all revisions. The actual task is accomplished by going to the Review tab, clicking the Accept button menu, and clicking Accept All Changes in Document (see Figure 10). This will result in a recorded macro that looks something like this:

Figure 10 Enabling the Developer Tab in the Ribbon
Figure 10 Enabling the Developer Tab in the Ribbon(Click the image for a smaller view)
Figure 10 Enabling the Developer Tab in the Ribbon
Figure 10 Enabling the Developer Tab in the Ribbon (Click the image for a larger view)

Sub AcceptAllChanges()
'
' AcceptAllChanges Macro
'
    WordBasic.AcceptAllChangesInDoc
End Sub

The macro recorder recorded this action to use the WordBasic object. The WordBasic object is a holdover from before Word macros were modified to use VBA. It's not the best way to approach the problem. It's much better to use ActiveDocument. You can either employ the AcceptAllRevisions method or the Revision property's AcceptAll method. I slightly prefer the Revisions.AcceptAll nomenclature, but either method works.

Here is the code necessary to accept revisions from within VSTO code:

private void btnAcceptRevisions_Click(
    object sender, RibbonControlEventArgs e)
{
    Application app = Globals.ThisAddIn.Application;
    Document activeDoc = app.ActiveDocument;
    if (activeDoc == null) return;

    // Accept all Changes
    activeDoc.Revisions.AcceptAll();
}

The method could consist of one line, but it's a good idea to check for an active document and to provide intervening variables should you need to debug. To plug in this code, simply open Publish Prep.cs in Design mode in Visual Studio. Double-click the Accept Revisions button, then replace the btnAcceptRevisions_Click method prototype that Visual Studio added for you with this new code. Press F5 to start debugging and navigate to the Add-ins tab. You should be able to click the button and have all of the revisions in the document accepted.

The Remove Comments button is a bit trickier because it's not something that the macro recorder will record for you. If I start the macro recorder and then try to right-click on a comment to select the delete option for it—well, it just doesn't work. That means I can't select a comment to remove it, so I've got to write my own code from scratch for VSTO. Luckily the code isn't that difficult.

I already have a reference to the active document in Word. That object has a comments collection property. From that collection, I can keep deleting the first comment until I have no more comments left:

private void btnRemoveComments_Click(
    object sender, RibbonControlEventArgs e)
{
    Application app = Globals.ThisAddIn.Application;
    Document activeDoc = app.ActiveDocument;
    if ((activeDoc == null)) return;
    // Clear Comments
    while ((activeDoc.Comments.Count > 0))
    {
        app.Selection.Comments[1].Delete();
    }

}

One interesting thing to note is that the indexer here is 1-based, not zero-based as you would expect in C#.

Adding this feature to the solution is as simple as double-clicking the Remove Comments button and replacing the btnRemoveComments_Click method prototype that Visual Studio added with the code just shown.

At this point I can press F5 again and complete all of the tasks that I set out to do—compiling files, accepting revisions, and removing comments. This isn't world-shaking functionality, but the completed add-in demonstrates handily how the macro recorder can quickly generate some prototype VBA code that can put you on the right track for writing more complete and sophisticated solutions in Visual Basic or C# with VSTO. The macro recorder wasn't able to handle every scenario, but it sure makes the tasks it does support a lot easier to tackle.

Send your questions and comments to mmoffice@microsoft.com.

Robert Bogue, Microsoft MVP for Microsoft Office Sharepoint Server, MCSE, and MCSA:Security, has contributed to more than 100 book projects and numerous other publishing projects. Robert blogs at www.thorprojects.com/blog; you can reach him at Rob.Bogue@thorprojects.com. Find out more about his latest book project, The SharePoint Shepherd's Guide for End Users, at sharepointshepherd.com.


Wicked Code: Silverlight Page Turning Made Simple

Wicked Code
Silverlight Page Turning Made Simple
Jeff Prosise

Code download available at: WickedCode2008_05.exe (1110KB)
Contents
The PageTurnDemo Application
Using the Page-Turn Framework
XAML Structure
Framework Internals
Conclusion

Two years ago, I was cruising down a hallway in Redmond when a friend stopped me. "I have something you need to see," he said. He popped open his laptop and showed me a demo that changed my life. That demo was an early version of the SilverlightTM Page Turn Sample, which you can see now at silverlight.net/samples/1.0/Page-Turn/default.html.

I couldn't believe what I was seeing because the demo was running—gasp—in a browser! More surprisingly, it didn't require the Microsoft® .NET Framework, it didn't require Internet Explorer®, and although no one knew it at the time, one day it wouldn't even require Windows®.

PageTurn is the quintessential Silverlight 1.0 demo. I use it all the time when I want to impress upon first-timers what Silverlight is all about. But look inside PageTurn and you'll find that building a page-turning application of your own is no small chore. PageTurn relies on transforms, clipping regions, dynamically created XAML objects, and more; the source code requires time and effort (and no small knowledge of Silverlight) to understand. It deftly demonstrates some of the most prolific capabilities of Silverlight, but it was not necessarily designed for general reuse.

That's why I built a general-purpose page-turning framework that makes it incredibly simple to incorporate page turns into your Silverlight 1.0 applications. With my framework to help out, an entire application can be built with just a few lines of JavaScript. It requires minimal knowledge of Silverlight itself, and since the entire framework consists of about 500 lines of JavaScript, you can dig in and understand how it works without having to wrap your head around thousands of lines of code. And, of course, you can modify it to your heart's content.

The PageTurnDemo Application

Before I introduce the framework, let's examine an application built around it. The PageTurnDemo application pictured in Figure 1 lets you page through the first several pages of the November 1988 issue of Microsoft Systems Journal, now known as MSDN® Magazine. (How do you avoid copyright issues when reproducing pages from a magazine? Use pages from the magazine you work with, and use an article you wrote yourself, no less!) Each of the pages is a scanned image, but one of them—the final page—overlays the image with Extensible Application Markup Language (XAML) text. I included the text to reinforce a key point: pages fed to the framework aren't limited to just images; with few constraints, they can contain arbitrary XAML, including Images, TextBlocks, MediaElements, and more.

Figure 1 PageTurnDemo Shows a Partially Turned Page
Figure 1 PageTurnDemo Shows a Partially Turned Page(Click the image for a smaller view)
Figure 1 PageTurnDemo Shows a Partially Turned Page
Figure 1 PageTurnDemo Shows a Partially Turned Page (Click the image for a larger view)

You can run the demo by downloading the source code and launching it from Visual Studio® 2008, or you can view it at wintellect.com/silverlight/pageturndemo. Once the magazine cover appears (a progress bar apprises you of the progress of the download that retrieves all the images used in the app), use the left mouse button to drag the mouse over the cover from right to left to page forward. You can continue dragging left over right-hand pages to reveal more pages, or drag from left to right over left-hand pages to page backward.

How difficult was it to create this sample? Aside from scanning, cropping, sizing the images, and packaging them in a ZIP file, not difficult at all. The three key source code files (which you can find in the download accompanying this column) are the HTML file, the associated JavaScript file, and the XAML file. Take away the code that uses the Silverlight downloader object to download the images, and there's very little source code at all. In fact, only about 10 lines are devoted to the page-turning functionality.

Using the Page-Turn Framework

PageTurnDemo demonstrates the four basic steps you'll need to complete to use the page-turn framework. The first step is to include PageTurn.js—the script file containing the framework implementation—in the HTML file. Here's the relevant line in Default.html:

<script ... src="PageTurn.js"></script>

The second step is to instantiate the framework. Because the page-turn framework is encapsulated in a JavaScript class named PageTurnFramework, the following statement in Default.html.js instantiates the framework:

_ptf = new PageTurnFramework(_control,
    _control.content.findName('PageTurnCanvas'));

The first parameter passed to the PageTurnFramework constructor is a reference to the Silverlight control. The second parameter is a reference to the canvas containing your pages. In PageTurnDemo's XAML document, that canvas is named "PageTurnCanvas."

The third step is to register your pages with the framework. Each page is represented by a canvas, and PageTurnFramework exposes an addPage method that you can call to register pages. AddPage accepts two parameters:

_ptf.addPage(_control.content.findName('EvenPage0'),
    _control.content.findName('OddPage0'));

The first parameter is a reference to the canvas representing the left-hand page in a pair of facing pages; the second is a reference to the canvas representing the right-hand page. You can call addPage as many times as you need to register all your page pairs. And then, the final call to addPage must be followed by a call to initializeFramework:

_ptf.initializeFramework();

The initializeFramework method initializes the internal state of the framework. Among other things, it creates several XAML objects used to clip and rotate pages, and it also registers handlers for key events.

The fourth and final requirement for using the framework is to be sure to clean up properly. To avoid memory leaks, browser-based apps that programmatically register event handlers should unregister them as well. PageTurnFramework includes a dispose method that deregisters the event handlers registered by addPage and initializeFramework. In PageTurnDemo, an onunload attribute in the <body> element calls a local dispose function when the page unloads:

<body ... onunload="dispose()">

This local dispose function, which lives in Default.html.js, calls the framework's dispose method:

if (_ptf != null)
    _ptf.dispose();

I used a DOM event to trigger calls to dispose because Silverlight doesn't fire unload events. You could just as easily use window.unload events if you'd prefer.

The PageTurnFramework class exposes six public methods that you can call to add page-turning functionality to your apps (see Figure 2). PageTurnDemo uses three of them: addPage, initializeFramework, and dispose. The other methods can be used to add extra functionality. For example, if you wanted to include a navigation bar consisting of page thumbnails in your app (as the Silverlight PageTurn demo does), you could call PageTurnFrame work.goToPage when a thumbnail is clicked to navigate directly to the corresponding page.

Figure 2 Page-TurnFramework API (Close Figure 2)

Method | Description
addPage | Registers a page pair with the framework.
Dispose | Releases resources held by the framework for proper cleanup.
getCurrentPageIndex | Returns the 0-based index of the page pair currently displayed.
getPageCount | Returns the number of page pairs added with addPage.
goToPage | Displays the specified page pair.
initializeFramework | Initializes the page-turn framework.

XAML Structure

The page-turn framework imposes a few basic requirements on the structure of XAML documents. Those requirements are:

  1. Represent each page in a page pair with a XAML canvas (a "page canvas").
  2. Include a "page-turn" canvas that is a container for all of the page canvases.
  3. Assign a width, height, and background color (even if it's Transparent) to the root canvas.

The reason for the third requirement is that at initialization, the framework registers a handler for MouseLeave events fired by the root canvas. These events are used to complete page turns if the cursor leaves the Silverlight control with a page partially turned.

As a matter of best practice, all Silverlight 1.0 apps that capture the mouse, as the page-turn framework does when a turn begins, should process MouseLeave events emanating from the root canvas and take that opportunity to release the mouse. And in order for an element to receive mouse events such as MouseEnter and MouseLeave, the element must be "hit testable" within Silverlight. This is done by ensuring that the element (in this case, a Canvas) has a size and that it has a "non null" background.

With these requirements in mind, Figure 3 shows the general structure of a XAML document used with the page-turn framework. The page-turn canvas is the one you pass to PageTurnFramework's class constructor. Page canvases are passed to PageTurnFramework.addPage. The page-turn canvas and the page canvases should generally be tagged with explicit widths and heights in order to ensure that the mouse events used by the framework fire properly no matter what kind of XAML content the canvases contain. The rest is just XAML.

Figure 3 XAML Structure (Close Figure 3)
<Canvas
    xmlns="http://schemas.microsoft.com/client/2007"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Background="color" Width="width" Height="height">

    <!-- Other XAML content goes here -->

    <!-- Page turn canvas -->
    <Canvas>

        <!-- Canvases representing first page pair -->
        <Canvas>
            <!-- Content for left-hand page goes here -->
        </Canvas>
        <Canvas>
            <!-- Content for right-hand page goes here -->
        </Canvas>

       <!-- Canvases representing additional page pairs go here -->

    </Canvas>

    <!-- Other XAML content goes here -->

</Canvas>

The page-turn canvas can happily exist alongside any other rich content in your XAML document. And individual pages can consist of simple XAML images or complex XAML renderings. You should generally avoid using the Clip property of the page canvases because the framework itself uses that property. However, you could always declare a subcanvas inside a page canvas and use the subcanvas's Clip property to define clipping regions.

If you want the first left-hand page to be blank so that the first right-hand page looks like the cover of an unopened book, simply declare an opaque rectangle in the first left-hand page. Similarly, you can use an opaque rectangle for the final right-hand page to create the appearance of a closed book when the user turns to the last page. PageTurnDemo does both to perpetuate the illusion that you're flipping through the pages of a magazine.

When you scan pages from a book or magazine to use with the page-turn framework, as I did for PageTurnDemo, the scans often have shadows at the inside edges, which give the pages a more realistic appearance. If you don't use scanned images, a bit of XAML can simulate shadows. For the application pictured in Figure 4 (which depicts some of my radio-controlled airplanes and jets and is viewable at wintellect.com/silverlight/mymodels), I used the XAML rectangles in Figure 5 to create shadows around the vertical divide at the center of each page pair. Alpha values in the GradientStop colors cause the rectangles to fade from right to left on left-hand pages and from left to right on right-hand pages.

Figure 5 XAML for Creating Shadows on Page Interiors (Close Figure 5)
<!-- Shadow on left-hand page -->
<Rectangle Canvas.Left="380" Canvas.Top="0" Width="20" Height="600">
    <Rectangle.Fill>
        <LinearGradientBrush StartPoint="0,0" EndPoint="1,0">
            <GradientStop Color="#00000000" Offset="0.0" />
            <GradientStop Color="#40000000" Offset="1.0" />
        </LinearGradientBrush>
    </Rectangle.Fill>
</Rectangle>

<!-- Shadow on right-hand page -->
<Rectangle Canvas.Left="0" Canvas.Top="0" Width="20" Height="600">
    <Rectangle.Fill>
        <LinearGradientBrush StartPoint="0,0" EndPoint="1,0">
            <GradientStop Color="#40000000" Offset="0.0" />
            <GradientStop Color="#00000000" Offset="1.0" />
        </LinearGradientBrush>
    </Rectangle.Fill>
</Rectangle>
Figure 4 Page-Turn Sample with XAML Shadows on Page Interiors
Figure 4 Page-Turn Sample with XAML Shadows on Page Interiors(Click the image for a smaller view)
Figure 4 Page-Turn Sample with XAML Shadows on Page Interiors
Figure 4 Page-Turn Sample with XAML Shadows on Page Interiors (Click the image for a larger view)

If any of this is unclear, you can start an application by building from PageTurnDemo. Simply replace my XAML for individual pages with XAML of your own. My page canvases are dimensioned with a width and height of 400 and 544, but you can change those dimensions to whatever you prefer.

Framework Internals

The page-turn framework is implemented in PageTurn.js, which you can see in the download for this issue. The PageTurnFramework class begins just a few lines from the top. JavaScript doesn't support classes, but this framework uses the same prototype pattern used by ASP.NET AJAX and many Silverlight 1.0 apps to simulate classes in JavaScript. It's just an illusion, but it's an effective one.

The class constructor defines all the instance variables—the equivalent of fields in C#—needed to store internal state. For example, the statement

   this._control = control;
declares a "field" named _control and initializes it with the Silverlight control reference passed to the constructor. In all, approximately 40 fields are declared and used to hold everything from the percent-complete figure for an in-progress page turn (_percent) to tokens representing registered event handlers (which dispose uses to de-register the handlers). Fields also store references to several XAML objects that are created dynamically in initializeFramework and to XAML objects registered with calls to addPage.

PageTurnFramework.prototype contains all the PageTurnFramework methods. The methods fall into three broad categories: public methods such as addPage and initializeFramework; event handlers, which act in response to the mouse events and Storyboard.Completed events that the framework uses internally; and private methods, which are used internally by the framework but aren't intended to be called from the outside. There's plenty of interesting code, but due to space constraints, you'll have to download PageTurn.js to see them.

One of the interesting aspects of the framework architecture is how it completes page turns if the mouse button is released (or the cursor leaves the control) while a page turn is ongoing. At initialization time, the framework uses createFromXaml to create a Storyboard object to use as a timer. Then it adds the Storyboard to the page-turn canvas (this is one of the reasons you need to pass a reference to the page-turn canvas to the class constructor) and registers a handler for Storyboard.Completed events.

To finish an incomplete page turn, the framework calls Storyboard.begin to start the timer. On each timer tick, it advances the page another increment (using the step size stored in the _step field) and calls Storyboard.begin again if the page turn still isn't complete. You can see this in action by turning a page halfway and then releasing the mouse button. Depending on how far the page was turned when you released it, it will move back to the fully closed or fully opened position.

The Storyboard's XAML definition is stored in the variable named _sb in the initializeFramework method. You may wonder why that definition includes an x:Name attribute whose value is a GUID. In Silverlight 1.0, Storyboards created with createFromXaml must be named or createFromXaml will fail. I had to give the Storyboard a name, but I wanted to make sure the name didn't conflict with other Storyboards used in the application. Therefore, I named it after a GUID.

Another interesting part of the architecture is how the framework uses mouse events. For canvases representing pages, addPage registers handlers for MouseLeftButtonDown, MouseMove, and MouseLeftButtonUp events. A left turn begins when a right-hand page is clicked and progresses as the mouse moves to the left with the left button held down. Similarly, a right turn begins when a left-hand canvas is clicked and progresses as the mouse moves to the right. The key method used by the event handlers is _turnTo, which advances a partially turned page to the position denoted by the percent-complete parameter.

A final aspect of the framework that you may care to examine more closely is how it uses transforms and clipping regions to depict page turns. initializeFramework creates two PathGeometry objects: one to serve as a clipping region for right-hand pages and another for left-hand pages. It also creates a TransformGroup object containing a RotateTransform and a TranslateTransform.

Figure 6 illustrates how the clipping regions and transforms are used to depict a partially turned page. The red triangle is the clipping region used on the right-hand page that's on top. As the mouse moves to the left, the clipping region grows smaller so that less of the top page is visible and more of the page underneath is revealed. The blue triangle represents the clipping region used on the left-hand page that is revealed as the turn progresses, and the yellow rectangle represents the portion of that page that is clipped out.

Figure 6 Clipping Regions and Transforms Used to Depict Page Turns
Figure 6 Clipping Regions and Transforms Used to Depict Page Turns(Click the image for a smaller view)
Figure 6 Clipping Regions and Transforms Used to Depict Page Turns
Figure 6 Clipping Regions and Transforms Used to Depict Page Turns (Click the image for a larger view)

The RotateTransform and TranslateTransform are combined in order to position and orient the page. (Yes, there is a lot of trigonometry involved!) As the mouse moves to the left, imagine the yellow rectangle sliding to the left and rotating into an upright position. Couple that with a mental image of the blue triangle increasing in size while the red one grows progressively narrower and you'll have a pretty good idea of how page turns work.

Conclusion

Before the ink was dry on this column, I thought of other useful additions to the PageTurnFramework API. For example, it might be useful if the framework fired an event every time a page turns so you could write handlers that update other content on the page. And it could be useful to expose properties to control key page-turn parameters, such as the width of the shadow that follows turning pages and the step size used to animate incomplete turns.

Feel free to use this framework in your own projects and to modify it. Let me know your feedback, and if you think of useful additions to the feature set and API, let me know about that, too. I'll fold in your suggestions with my own and make sure that version 2.0 of the page-turn framework is even better than version 1.0.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET. He's also cofounder of Wintellect (www.wintellect.com), a software consulting and education firm that specializes in the .NET Framework. Have a comment about this column? Contact Jeff at wicked@microsoft.com.


Foundations: Loading Workflow Models in WF

Foundations
Loading Workflow Models in WF
Matt Milner

Code download available at: Foundations2008_05.exe (161KB)
Contents
CLR Types for Models
XAML for Workflows
Workflow Loading
Comparing Modeling Methods
Conclusion

Workflows in Windows® Workflow Foundation (WF) can be modeled in several ways. Developers model a hierarchy of activities that defines the control flow and program statements to be executed in the workflow. (For more on the declarative nature of Windows WF, see "WinFX Workflow: Simplify Development with the Declarative Model of Windows Workflow Foundation," by Don Box and Dharma Shukla, at msdn2.microsoft.com/magazine/cc163661.aspx.)

The workflow model or definition is the artifact that tools will create and modify and provides the template for runtime instances of workflows. The flexibility to model workflows in different ways is a powerful feature and allows for modeling tools that target different audiences. But what format should you use to model your workflows? The answer, as usual, is not simple. In this column I will explain how two common models are applied and how the runtime handles them. In addition, I will discuss how the runtime creates new instances of workflows for both Microsoft® .NET Framework types and eXtensible Application Markup Language (XAML), and how you can customize that process to suit your needs.

CLR Types for Models

One way to model workflows is in .NET-targeted code. You create a class that will represent the workflow, and you build the hierarchy of activities in the constructor for that class. Typically, the class you write will derive from some composite activity, such as SequenceActivity, and then in the constructor you add activities as children. Figure 1 shows a workflow model using SequenceActivity as the base class and a simple WriteLine activity as the children.

Figure 1 Defining a Workflow in C# (Close Figure 1)
public class WorkflowInCode : SequenceActivity
{
    public WorkflowInCode()
    {
        WriteLineActivity writeLine1 = new WriteLineActivity();
        writeLine1.Name = "writeLine1";
        writeLine1.OutputText = "Hello";

        WriteLineActivity writeLine2 = new WriteLineActivity();
        writeLine2.Name = "writeLine2";
        writeLine2.OutputText = "MSDN Reader";

        this.Activities.Add(writeLine1);
        this.Activities.Add(writeLine2);

    }
}

This model's mechanics and tool use are similar to the way Windows Forms are designed. A visual designer is used to model the workflow hierarchy, dragging and dropping activities into the model; behind the scenes the designer generates code to recreate the structure. This is one of the primary models supported by the Visual Studio® tools for Windows WF. The constructor for the workflow calls the InitializeComponent method, which is located in the designer file generated by the visual design surface.

Figure 1 models a very simple activity hierarchy; the class I created does not have any code outside of the hierarchy creation. It is also possible to use this .NET modeling mechanism to model activity binding, event handlers, and more complex interactions between activities. For example, if the workflow contained a ReadLine activity and a WriteLine activity, I could bind properties between them and also create event handlers for events raised by the activities. Figure 2 shows an updated workflow definition that includes these changes.

Figure 2 More Complex Workflow Code Model (Close Figure 2)
public class WorkflowInCode : SequenceActivity
{
  public WorkflowInCode()
  {
    ReadLineActivity readLine1 = new ReadLineActivity();
    readLine1.Name = "readLine1";
    readLine1.Invoked += new EventHandler(readLine1_Invoked);

    WriteLineActivity writeLine1 = new WriteLineActivity();
    ActivityBind textBind = new ActivityBind("readLine1", "InputText");

    writeLine1.Name = "writeLine1";
    writeLine1.SetBinding(WriteLineActivity.OutputTextProperty,
       textBind);

    this.Activities.Add(readLine1);
    this.Activities.Add(writeLine1);
  }

  void readLine1_Invoked(object sender, EventArgs e)
  {
    ReadLineActivity readLine = sender as ReadLineActivity;
    Console.WriteLine("Event handler in workflow read \"{0}\"
       from readline.",
      readLine.InputText);
  }
}

In the code shown in Figure 2, the workflow class has code that represents part of the workflow but is not hierarchy related. When I compile the workflow definition, the event handler for the Invoked event on the ReadLine activity will also be included in the assembly.

When modeling workflows in code, you compile the model into a .NET assembly and instantiate a CLR class at run time to create the workflow hierarchy. When you want to start a workflow modeled in this way, you request the runtime to create an instance for you based on the class, as shown here:

WorkflowRuntime runtime = new WorkflowRuntime();
WorkflowInstance instance = runtime.CreateWorkflow(typeof(WorkflowInCode));
instance.Start();

Because I'm compiling all of this information into CLR classes, when the type is loaded, the runtime can easily resolve references to event handlers and other code that is part of the workflow definition. Later I'll go into how these types are actually loaded and the implications of using this modeling technique as opposed to XML.

XAML for Workflows

A more compelling way to model workflows is using XAML, an XML dialect for representing hierarchies of CLR objects. XAML is most often associated with Windows Presentation Foundation (WPF), where it is used to represent hierarchies of visual objects drawn on the screen: windows, buttons, menus, and so on. In Windows WF, XAML represents a hierarchy of activities that form a workflow. The following XML describes the simple workflow that I previously modeled in code:

<SequenceActivity xmlns="http://schemas.microsoft.com/winfx/2006
                         /xaml/workflow"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:msdn="clr-namespace:MSDN.CustomActivities;Assembly=
              CustomActivities">
  <msdn:WriteLineActivity x:Name="writeLine1" OutputText="Hello" />
  <msdn:WriteLineActivity x:Name="writeLine2" OutputText="MSDN Reader" />
</SequenceActivity>

The first thing to notice about this XAML is that because I am defining a structure of CLR types in XML, I need to provide a bit of mapping between the two worlds. It is necessary that the WorkflowRuntime be able to reconcile my custom WriteLine activity, which is represented in XAML, to a .NET type when building the hierarchy.

In the previous XML, an XML namespace declaration maps the msdn prefix to the MSDN.CustomActivities CLR namespace in the CustomActivities assembly. When the XAML is processed and deserialized to create the object hierarchy, this information will be used to resolve the .NET type. The attributes on the XAML element translate into the property values on the .NET object when it is created, thus fully initializing the object.

Via XAML, I also have the ability to manage activity binding, hook up event handlers, and do more complex modeling of relationships or interactions. Using something called Markup Extensions, I can bind properties or event handlers. The following code represents the workflow containing both ReadLine and WriteLine activities with a binding between them:

<SequenceActivity xmlns="http://schemas.microsoft.com/winfx/2006/
                         xaml/workflow"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:msdn="clr-namespace:MSDN.CustomActivities;Assembly=
              CustomActivities">
  <msdn:ReadLineActivity x:Name="readLine1" InputText="" />
  <msdn:WriteLineActivity x:Name="writeLine2"
    OutputText="{ActivityBind readLine1 Path=InputText}" />
</SequenceActivity>

The OutputText property of the WriteLine activity has an ActivityBind extension. This turns the supplied text into an ActivityBind class instance when the workflow is deserialized into objects.

When defining workflows in XAML, you can either translate the XML to a CLR activity hierarchy at compilation time or at run time. If you choose to compile a XAML workflow definition, you need to use the WorkflowCompiler class or the wfc.exe command-line compiler, either of which will allow you to generate a .NET assembly containing the workflow definition in a CLR class. You also need to add the Class attribute to the root activity to indicate the class to be created when the workflow is compiled.

The result of compiling a XAML definition is the same as if you had created the definition in code: a .NET type that can be used to create workflows. The real difference is in how tools create the model and how the model is used at run time. I will talk about the trade-offs in both cases shortly.

If you build the hierarchy at run time, you load the XAML into an XmlReader and then ask the WorkflowRuntime to create a WorkflowInstance from the XAML definition. Earlier, I created the instance by its CLR type using the CreateWorkflow method on the WorkflowRuntime class. This method has several overloaded versions, including one that accepts an XmlReader containing the workflow definition. The code below shows this simple but powerful change, which loads the definition directly from a XAML file:

WorkflowInstance instance = runtime.CreateWorkflow(
XmlTextReader.Create("..\\..\\WorkflowInXML.xml"));
instance.Start();

It is CreateWorkflow that translates the model or definition into an instance that can be used by the runtime. I will discuss the details of this transition shortly, but first I want to elaborate on some more details of creating workflow models in XAML.

Looking at the sample XAML workflow definition, you will notice that there is no mapping for the root SequenceActivity element to the System.Workflow.Activities CLR namespace. This activity is included as part of the .NET Framework 3.0 and is used here as the root activity in the hierarchy.

One of the benefits of this sort of model is that the model itself does not have to be tied to a single interpretation or implementation of a type. Because I have included CLR type information in this XAML model for the WriteLine activity via the namespace mapping, I'm tied to a specific implementation of the activity class. Further, if I have different versions of the activity, I'm tied to a specific version of that activity for the workflow model by the strong name used to reference the assembly. What I need is an abstraction to decouple the model from the CLR type information but still allow me to deserialize this markup into an activity hierarchy.

This decoupling is done using a mapping layer between XML namespaces and CLR namespaces. Windows WF allows the hosting code to define the assemblies used to fulfill the mapping. In the previous example, the SequenceActivity element in the XAML is in the default XML namespace. The default namespace is called out as "http://schemas.microsoft.com/winfx/2006/xaml/workflow" in the first line of the XAML.

The other side of the equation is somehow mapping this XML namespace to a CLR namespace, and that is accomplished by adding custom attributes into the assembly where the activities are defined. The System.Workflow.Activities.dll assembly contains an attribute that maps the XML namespace seen in the XAML to the CLR namespace System.Workflow.Activities. I can do the same with my custom activities. I add the following attribute to the assemblyinfo.cs file in the activities project:

using System.Workflow.ComponentModel.Serialization;
[assembly:XmlnsDefinition("http://msdnmagazine/foundations/workflow",
  "MSDN.CustomActivities")]

Then I change the XML namespace in my XAML file to match this attribute, as shown here:

xmlns:msdn="http://msdnmagazine/foundations/workflow"

At this point, I have the XML namespace neatly in my XAML file with no direct ties to the CLR assembly or namespace. I also have the mapping from XML to CLR namespaces defined via the CLR attribute on my assembly. When the workflow runtime attempts to deserialize the XAML, it needs to know which assemblies to inspect for the custom attribute. It uses the TypeProvider class to do so; this class implements the ITypeProvider interface and is a container for assemblies that need to be available when types are resolved. In my example, I create an instance of the TypeProvider class in my hosting application, add my activity assembly to the TypeProvider, and then add the TypeProvider to the WorkflowRuntime as a service:

using (WorkflowRuntime runtime  = new WorkflowRuntime())
{
  TypeProvider provider = new TypeProvider(runtime);
  provider.AddAssemblyReference("CustomActivities.dll");
  runtime.AddService(provider);

  //start workflow
  ...
}

I create a model in XML and then defer the mapping to CLR types until run time by configuring the TypeProvider with the assemblies I want for my host. This gives me a flexible model and control over the CLR activity types used at run time to create the instance. When the workflow is deserialized, the WorkflowMarkupSerializer class is used, which uses the TypeProvider to resolve XML names to CLR types. Figure 3 shows the interaction between the loader (described next), the WorkflowMarkupSerializer, and the TypeProvider.

Figure 3 XAML Workflow Creation
Figure 3 XAML Workflow Creation(Click the image for a smaller view)
Figure 3 XAML Workflow Creation
Figure 3 XAML Workflow Creation (Click the image for a larger view)

Workflow Loading

Once a workflow definition has been created, either as CLR classes or as XAML markup, the runtime needs to be able to take that definition and create a WorkflowInstance. Many developers assume that WorkflowRuntime is responsible for loading the workflow instance because that is the class they use to create the instance. But WorkflowRuntime uses a runtime service to do the actual loading of the workflow. A base class, WorkflowLoaderService, defines the abstract methods that a workflow loader service must implement. The definition of this class is shown here:

public abstract class WorkflowLoaderService
{
  protected internal abstract Activity CreateInstance(Type workflowType);
  protected internal abstract Activity CreateInstance(XmlReader
    workflowDefinitionReader, XmlReader rulesReader);
}

As you can see, WorkflowLoaderService takes either a CLR Type or an XmlReader containing the workflow definition and creates an Activity hierarchy from it. The runtime also contains a DefaultWorkflowLoaderService that it uses if no other loader service is added to the runtime. In this default loader, when a workflow is to be created from type information, the service simply calls Activator.CreateInstance on the type and returns that object.

When DefaultWorkflowLoaderService is asked to create a workflow from an XmlReader, it attempts to deserialize the XAML into an activity hierarchy using the WorkflowMarkupSerializer class. Any validation errors are captured and an exception is thrown if the deserialization fails.

The default loader works in many situations, especially when using XAML or .NET workflow definitions, but there may be times when you want to create a custom workflow loader to augment or replace the default behavior. When creating a custom loader, you can either derive from the WorkflowLoaderService abstract class or DefaultWorkflowLoaderService, depending on how much functionality you want to reuse.

Many developers want to store their XAML workflows in a database. This is accomplished by updating the authoring or deployment tools to store the XAML in a database table. Once the XAML is stored, you could load the XAML into an XmlReader in the host application and then call CreateWorkflow, but a custom loader service centralizes that logic and makes it reusable across hosts.

There are some constraints when creating a custom workflow loader service. First, because both the CreateWorkflow method and the WorkflowLoaderService only have methods for creating workflows from a CLR Type or an XmlReader, the custom loader must be able to get the information it needs from either a CLR type or an XML file. For security reasons, the runtime validates that the loader service does not create a workflow that is a different type than specified. Suppose I defined an interface, and then wanted to implement logic such that when a type implementing that interface was passed to the loader, I would call a method on that interface to get the workflow type. This would allow for more dynamic code-driven workflows. But this would not work, as the type passed to the CreateWorkflow call would not match the type of the returned workflow.

This leaves the XmlReader route, and here I have a lot more flexibility. Since the runtime doesn't know anything about the workflow type itself until after the loader creates it, the workflow loader has the ability to interpret the XML as it sees fit and return a root activity representing the workflow. What I generally do is create an XML file format that includes all of the information I need to create a workflow. As an example, I will show you how to create a custom loader service that loads workflows from a SQL Server® database. The information about the workflow will be stored in a simple XML file that can be wrapped in an XmlReader and passed to the CreateWorkflow method. Figure 4 shows the interactions among the runtime, my loader service, the database, and the WorkflowMarkupSerializer.

Figure 4 Custom Workflow Loading from a Database
Figure 4 Custom Workflow Loading from a Database(Click the image for a smaller view)
Figure 4 Custom Workflow Loading from a Database
Figure 4 Custom Workflow Loading from a Database (Click the image for a larger view)

The first decision was how I would store my workflows in the database. One obvious way to store the model is as a XAML document in a column. In that case, the first question is whether I store the XAML as one of the SQL text types or as an XML type in SQL Server 2005. If I wanted to be able to query or update the XAML in the database, I would use the XML type, but for this example, I just want a place to store the XAML, so I used the ntext data type. I also added some metadata columns and an ntext column to store the serialized representation of the rules for my workflow. Most workflows will contain some rules, if only to control composite activity execution like the IfElseActivity, so a custom loader solution should also include rule management capabilities.

I chose to store the entire workflow model in the database as a XAML file, but there are many alternatives. One simple example would be to store workflow fragments in the database as XAML and then have the complete workflow definition consist of a collection of those fragments. This would create reusable workflow logic in XML, unlike creating custom composite activities, which are compiled into binary form. Another idea would be to store the workflow definition as relational data. For example, I might have a table that consists of individual steps in a workflow, which defines the basic activities, and then have a workflow table that would link these activities together.

The storage model for the workflow depends on your requirements for managing the model and its various parts. When you create a custom loader, you simply need to be able to correctly combine that data to create the activity hierarchy that represents the modeled workflow.

The next step is to determine the format of the XML document that references the workflow from the database. I chose to select workflows based on a name and version number, and I created a very simple XML model. The sample XML document that follows is what I load into an XmlReader and then pass to the CreateWorkflow call:

<Workflow xmlns="http://msdnmagazine/foundations/workflow"
  Name="HelloWorld" Version="1.0" />

You might choose a variety of identifiers to select the correct workflow, but I find that using a common name and version number can make it easier to select or identify workflows.

I wrote my custom loader service to consume the XML document and return the workflow definition from the database. I had two design considerations: to support CLR-based workflow models and to support someone passing XAML directly to the CreateWorkflow method instead of using my custom XML format. Since the WorkflowRuntime can only use one WorkflowLoaderService, my loader is derived from the DefaultWorkflowLoaderService. It depends on functionality in the base to support the existing models as well as my own database model.

I define a connection string property on my service and provide constructors that allow this value to be supplied when the service is created in code or in configuration. This eliminates the need to have connection information in the XML files.

For the CreateInstance method that takes a CLR type as a parameter, I simply rely on the base class implementation to return the instance. To overload the CreateInstance method that takes an XmlReader, I first check to see if the XML matches my custom format and extract the name and version of the workflow to look up.

Once I have a valid workflow name, I call a stored procedure in the database to return the XAML and serialized rules for my workflow. Once I have the XAML, I call down to the base class to deserialize the XAML into an Activity hierarchy and return the result. In the case where the XML does not meet my format, I simply pass a clone of the XmlReader to the base class and let the default loading behavior and validation take over. Figure 5 shows the CreateInstance method that handles the XmlReader logic.

Figure 5 Custom CreateInstance (Close Figure 5)
protected overrideActivity CreateInstance(XmlReader workflowDefinitionReader,
  XmlReader rulesReader)
{
  string workflowName = default(string);
  string workflowVersion = default(string);
  XmlReader dbWfDefinitionReader = null;
  XmlReader dbWfRulesReader = null;

  //get the workflow name and version from the xml
  workflowDefinitionReader.MoveToContent();

  //if we have the correct root node, read the attributes
  if (String.Compare(
    workflowDefinitionReader.LocalName, "Workflow", true,
    CultureInfo.InvariantCulture) == 0 &&
    String.Compare(workflowDefinitionReader.NamespaceURI,
    "http://msdnmagazine/foundations/workflow", true,
    CultureInfo.InvariantCulture) == 0)
  {
    workflowName = workflowDefinitionReader.GetAttribute("Name");
    workflowVersion = workflowDefinitionReader.GetAttribute("Version");

    if (String.IsNullOrEmpty(workflowName))
      throw new ArgumentNullException("WorkflowName");
  }
  else
  {
    //try to load the workflow as normal
    XmlReader wfclone = workflowDefinitionReader.ReadSubtree();
    return base.CreateInstance(wfclone, rulesReader);
  }

  //get the workflow definition and rules from the database
  GetWorkflowFromDB(workflowName, workflowVersion,
    ref dbWfDefinitionReader, ref dbWfRulesReader);

  //use the base class to create the workflow instance
  if(dbWfDefinitionReader != null)
    return base.CreateInstance(dbWfDefinitionReader, dbWfRulesReader);
  else
    throw new ArgumentException(
      "No workflow found with that name and version", "WorkflowName");
}

Comparing Modeling Methods

Now let's compare modeling types. There are three broad dimensions of comparison: tooling, versioning, and deployment. Tooling actually comes in two flavors: the tools you get from Microsoft and the tools you might create on your own. The project and item templates in the Microsoft-provided extensions to Visual Studio allow you to create either CLR-based workflows in Visual Basic® or C#, or XAML-based workflows that are compiled into assemblies. While there are no templates included for XAML workflows that are not compiled, you can find some third-party templates, including some that I have posted on my blog at pluralsight.com/blogs/matt. With these tools and Visual Studio, a developer can create workflows in the two predominant formats, manage them with source control, and end up with a compiled assembly or XAML files. For a custom loader solution such as the one described here, you would still have to create a mechanism for loading workflow definitions into the database outside of the tools.

When creating custom tools for authoring workflows, the approach with the lowest barrier to entry is using the workflow designer control included in the .NET Framework 3.0 runtime components and storing the definition as XAML with no codebehind file. A development team could quickly get a custom designer up and running with the XAML being stored in SQL Server. Team members could open the workflow model directly from the database, work on it, and save the changes back to the database when finished.

From a tooling perspective, the XAML model is the easiest way to manage the workflow model, as it requires a single XML file, with no worries about .NET code files and compilation. Since XAML is supported in Visual Studio, developers can use the visual designer to create and edit workflows that are not going to be compiled.

Managing workflows after they are developed is another consideration. If you model your workflows as CLR types, then you will be managing .NET assemblies for deployment; if you use XAML, you will be managing XML files and .NET assemblies. Remember, most workflows will use custom activities, which will have to be compiled and deployed to be usable by the workflows. XML files are more easily edited, scanned, differenced, and otherwise processed than assemblies. Because they are text files, their contents are more transparent.

Finally, but perhaps most importantly, how you version your workflows depends heavily on the model you choose. If you want to be able to have two different versions of a workflow model in use at the same time, you have different considerations when using CLR types or XAML as your model.

When you use CLR types to model your workflow, your versioning is tied to the CLR versioning system. This means your workflow assemblies must be signed with a strong name and you must increment the version number to indicate that you are creating a new version of the assembly. When your host application creates a workflow, it will be referencing a particular version of the CLR type that models the workflow.

Running instances of your workflow that were created from an earlier version of the model must be able to find that same type information so that they are able to resume. This means that you have to keep all versions of your .NET assembly around until all workflows are finished executing. In addition, if you use tools to visualize workflows or otherwise need to reconstruct the activity graph, you will need that .NET type information around. If you want to look at historical analyses of a workflow, you might have to keep that .NET assembly around for a long time.

When you have XAML-based workflows, the potential version dependencies are from the XML to the CLR types for the activities. If you do not use the namespace mapping discussed earlier, your XAML will be version-dependent and must be able to find that version when it is created. If you have mapped the namespaces, then the XAML is not bound to a specific version or even a specific implementation. The host environment, through the use of the TypeProvider, can provide the appropriate activity assemblies to be used by the runtime when resolving references from XAML to CLR. If you are building XAML workflows, I suggest you create and use the namespace mapping to simplify your versioning story.

Conclusion

Workflow models and their transformation into workflow instances is extremely flexible in Windows WF. When building workflows for your environment, it is important that you understand the choices and implications of choosing a particular model. Consider not only the developer experience and tooling, but also long-term maintenance and subsequent versions of your workflows.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Windows WF, WCF, and BizTalk Server. As an instructor for Pluralsight, Matt authors courses on Workflow and BizTalk Server, and teaches WCF. Matt lives in Minnesota with his wife, Kristen, and his two sons. Contact Matt through his blog, pluralsight.com/blogs/matt.


Security Briefs: Penetration Testing

Security Briefs
Penetration Testing
James A. Whittaker

Contents
Planning
Types of Penetration Tests
Environment Attacks
Input Attacks
Data and Logic Attacks
Don't Be Deterred

When you hear the term penetration testing, you probably envision a lone genius performing arcane tests against some hapless piece of software. And before the renaissance in penetration testing, that was probably a realistic image. Today penetration testing is performed in a far more methodical manner. This is necessary because the Security Development Lifecycle (SDL), and its front-loaded secure design and development focus, is reducing the number of latent defects, making the task of finding vulnerabilities during testing much more difficult. Software security testing is too important to leave to a small group of virtuosos. It must be teachable, methodical, and repeatable so that it can be applied in a wide variety of circumstances.

This is not to say that penetration testing is a science. All testing has exploratory aspects to it; testers apply input, which changes data inside the software causing it to react in any number of ways. The complexity is too great to accurately predict; however, there is much that can be done to plan ahead, and this column will describe how we do that planning at Microsoft.

Planning

With traditional testing, specifications, user documentation, use cases, and other design documentation are abundantly available. This information can be used to design a set of test cases that will verify the specified functionality. But the available sources of information on planning are severely limited. Penetration testing is not about verifying functionality; it's about verifying the absence of insecure functionality. Unfortunately, no one has defined any software development artifacts for these behaviors, and testers are forced to fend for themselves.

The first place to harvest penetration testing information is in the interfaces between the software and its external environment. User interfaces, network interfaces, APIs, and any other place where input is processed are clear attack vectors for hackers. If any of these interfaces are poorly designed or implemented, they may allow maliciously crafted input to enter and wreak havoc. Identifying and documenting these interfaces is a good place to start penetration testing.

The second area requiring special attention is error messages and user alert dialogs, which communicate information from the software to external users. Since such users may have malicious intent, it is important to understand what information is revealed to them and how that information is communicated.

Finally, penetration testers often define disaster scenarios that specify what a successful attack might look like. These misuse cases (or abuse cases if you prefer) often spring from a threat model or from prior known attacks.

Collecting information from all three of these sources is crucial prep work for a penetration test and will help guide you through the actual testing.

Types of Penetration Tests

Testing is about variation—finding the things in the software and its environment that can be varied, varying them, and seeing how the software responds. The goal is to ensure that the software performs reliably and securely under reasonable and even unreasonable production scenarios. So the most fundamental planning a tester can do is to understand what can be varied and what ways that variation needs to be staged for testing.

From a security standpoint, the environment, user input, and internal data and logic are the primary places where such variation can reveal security issues. The environment consists of the files, applications, system resources, and other local or network resources used by the application. Any of these could be the entry point of attack. User input is the data that originates with external (usually untrusted) entities that is parsed and used by the software. Internal data and logic are the internally stored variables and logic paths, which have any number of potential enumerations.

By varying the information in the software's environment, input domain and data/logic paths, you can perform attacks. I'll explore each of these three categories in more detail.

Environment Attacks

Software does not execute in isolation. It relies on any number of binaries and code-equivalent modules, such as scripts and plug-ins. It may also use configuration information from the registry or file system as well as databases and services that may reside anywhere. Each of these environmental interactions may be the source of a security breach and therefore must be tested.

There are also a number of important questions you must ask about the degree of trust that your application has in these interactions, including the following: How much does the application trust its local environment and remote resources? Does the application put sensitive information in a resource (for instance, the registry) that can be read by other applications? Does it trust every file or library it loads without verifying the contents? Can an attacker exploit this trust to force the application to do his bidding?

In addition to the trust questions, penetration testers should watch for DLLs that might be faulty or have been replaced (or modified) by an attacker, binaries, or files with which the application interacts that are not fully protected by access control lists (ACLs) or are otherwise unprotected. Testers must also be on the lookout for other applications that access shared memory resources or store sensitive data in the registry or in temporary files. Finally, testers must consider factors that create system stress, such as a slow network, low memory, and so forth, and determine the impact of these factors on security features.

Environment attacks are often conducted by rigging an insecure environment and then executing the application within that environment to see how it responds. This is an indirect form of testing; the attacks are waged against the environment in which the application is operating. Now let's look at direct testing.

Input Attacks

In penetration testing, the subset of inputs that come from untrusted sources are the most important. These include communication paths such as network protocols and sockets, exposed remote functionality such as DCOM, remote procedure calls (RPCs) and Web services, data files (binary or text), temporary files created during execution, and control files such as scripts and XML, all of which are subject to tampering. Finally, UI controls allowing direct user input, including logon screens, Web front ends, and the like, must also be checked.

Specifically, you'll want to determine whether input is properly controlled: are good inputs allowed in and bad ones (such as long strings, malformed packets, and so forth) kept out? Suitable input checking and file parsing are critical.

You'll need to test to see whether dangerous input can be entered into UI controls, and find out what happens when it is. This includes special characters, encoded input, script fragments, format strings, escape sequences, and so forth. You'll need to determine whether long strings that are embedded in packets fields or in files and are capable of causing memory overflow will get through. Corrupt packets in protocol streams are also a concern. You must watch for crashes and hangs and check the stack for exploitable memory corruption. Finally, you must ensure that such things as validation and error messages happen in the right place (client-side rather than server side) as a proper defense against bad input.

Input attacks really are like lobbing grenades against an application. Some of them will be properly parried and some will cause the software to explode. It's up to the penetration team to determine which are which and initiate appropriate fixes.

Data and Logic Attacks

Some faults are embedded in an application's internal data storage mechanisms and algorithm logic. In such cases, there seem to be design and coding errors where the developer was assuming either a benevolent user or failed to consider some code paths where a user might tread.

Denial of service is the primary example of this category but certainly not the most dangerous. Denial of service attacks can be successful when developers have failed to plan for a large number of users (or connections, files, or whatever inputs cause some resource to be taxed to its limit). However, there are far more insidious logical defects that need to be tested. For example, information disclosure can happen when inputs that drive error messages and other generated outputs reveal exploitable information to an attacker. One practical example of such data that you should always remove is any hardcoded test accounts or test APIs (which are often included in internal builds to aid test automation). These can provide easy access to an attacker. Two more tests you should run are to input false credentials to determine if the internal authorization mechanisms are robust, and choose inputs that vary the code paths. Often one code path is secure but the same functionality can be accessed in a different way, which could inadvertently bypass some crucial check.

Don't Be Deterred

Penetration testing is very different from traditional functional testing; not only do penetration testers lack appropriate documentation, but they also must be able to think like users who intend to do harm. This point is very important—developers often operate under the assumption that no reasonable user would execute a particular scenario, and therefore decline a bug fix. But you really can't take chances like that. Hackers will go to great lengths to find vulnerabilities and no trick, cheat, or off-the-wall test case is out of bounds. The same must be true for penetration testers as well.

Send your questions and comments to briefs@microsoft.com.

James A. Whittaker is a Principal Architect for Visual Studio Team System and author of the How to Break Software series.


Team System: Team Foundation Server Event Service

Team System
Team Foundation Server Event Service
Brian A. Randell

Code download available at: TeamSystem2008_05.exe (177KB)
Contents
Project Alerts
Controlling E-Mail Message Formatting
Available Events
Creating Subscriptions
Creating a Custom Event Listener
Final Details

Microsoft built Team Foundation Server (TFS) as a collection of major and minor services including version control, work item tracking, and the EventService service. I classify EventService as a minor or, better yet, supporting service. EventService exposes a set of events that, when fired, can perform actions such as sending e-mail or making a SOAP-based Web service call.

In this column, I'll examine what's available out of the box via the Visual Studio® user interface, what events are exposed by EventService, how you can create and manage subscriptions, and how you can create your own Web service to receive and process events. While I will be using Visual Studio 2008 and TFS 2008, most of this column applies equally to TFS 2005.

Project Alerts

If you start up Visual Studio 2008 with Team Foundation Client (TFC) installed and connect to an existing team project, you'll be able to access the Team menu on the main menu bar. Once you do, you will see a menu item labeled Project Alerts. If you choose this menu item, Visual Studio will open the Project Alerts dialog, which allows you to create e-mail subscriptions for up to four events (see Figure 1). TFS will format the body of the message as either plain text or HTML depending upon your preferences.

Figure 1 Project Alerts for TFC
Figure 1 Project Alerts for TFC(Click the image for a smaller view)
Figure 1 Project Alerts for TFC
Figure 1 Project Alerts for TFC (Click the image for a larger view)

Visual Studio exposes four events: artifacts are checked-in, a build completes, a build status changes, and someone else changes work items assigned to you. For each event, you can enter one or more e-mail addresses separated by semicolons. You choose the format of the e-mail message (HTML or plain text) on an event-by-event basis. Note that there are more events raised by TFS than those listed in the Project Alerts dialog. (More on this a bit later.)

Controlling E-Mail Message Formatting

TFS controls the formatting of the alert e-mail messages via a set of XSL transforms. You'll find 28 files stored in the folder C:\Program Files\Microsoft Visual Studio 2008 Team Foundation Server\Web Services\Services\v1.0\Transforms. There are four categories of files: eMailTemplate, plantextXsl, XSD, and XSL. Microsoft defines the data layout for its exposed events using the XSD files. The event service uses the XSL and the plaintextXsl files to transform the XML data into the e-mail message body. Microsoft created XSL files for the four events surfaced by the Project Alerts dialog. Note that, in TFS 2008, Microsoft enhanced the Build Completion event and thus created new XSL files suffixed with the numeral 2.

Finally, the eMailTemplate files refer to a set of five additional events that are available via EventService. However, Microsoft is no longer using or supporting these files. Instead, you'll use the DataChangedEvent.xsl file. The Project Alerts dialog doesn't expose these events, so in order to subscribe to them and receive an e-mail notification, you'll need to either use a command-line tool provided by Microsoft (BisSubscribe.exe) or write some code yourself.

To control the format of the notification e-mail messages, you need to edit the appropriate transform files. For the events exposed via the Project Alerts dialog box, you'll want to edit one or more of the respective XSL files. For HTML-formatted e-mail messages, you'll find the core layout for all events stored in TeamFoundation.xsl. You should make a backup copy of any files you intend to change first, and then test your changes. Once you've made your changes, TFS will use the modified transform the next time it raises the event.

Available Events

Microsoft designed the EventService service to be flexible and extensible. There are a couple of techniques you can use to figure out the list of exposed events. I discussed the first mechanism already: just examine the XSD files stored in the Transforms folder.

Programmatically, you can access another support service, the Registration Service, to get a list of services and the events they expose. Figure 2 shows you the code needed to do that and offers an example of the output.

Figure 2 Events Defined by Registered Services (Close Figure 2)

Code
If mtfs IsNot Nothing Then
  Dim rs As IRegistration = _
  DirectCast(mtfs.GetService(GetType(IRegistration)), IRegistration)

  Dim res() As RegistrationEntry = _
    rs.GetRegistrationEntries(String.Empty)

  Debug.WriteLine("Registration Entries and their Event Types.")
  For Each re As RegistrationEntry In res
    Debug.WriteLine(re.Type)
    If re.EventTypes.Length > 0 Then
      For Each et As EventType In re.EventTypes
        Debug.WriteLine("-->" & et.Name)
      Next
    Else
      Debug.WriteLine("-->No Events Found")
    End If
  Next
End If
Output
Registration Entries and their Event Types.
Build
-->Build Completion Event
-->Build Completion Event 2
-->Build Status Changed Event
vstfs
-->BranchMovedEvent
-->NodeCreatedEvent
-->NodePropertiesChangedEvent
-->NodeRenamedEvent
-->NodesDeletedEvent
-->ProjectCreatedEvent
-->ProjectDeletedEvent
Reports
-->No Events Found
Wss
-->No Events Found
WorkItemTracking
-->No Events Found
VersionControl
-->CheckinEvent
TestTools
-->No Events Found

Note that the WorkItemTracking system doesn't expose an event. Yet, if you look in the Transforms folder, you see there is, in fact, a WorkItemChangedEvent.xsd file. In addition, between Visual Studio 2005 and Visual Studio 2008, Microsoft modified how it raises certain events related to Group Security Service (GSS), Authorization, and the Common Structure Service (CSS). Figure 3 provides the current list of events you'll find exposed by EventService. You should consider those events not listed in Figure 3 to be obsolete.

Figure 3 Services and Supported Events (Close Figure 3)

Service | Event | Listed in Registration Service?
Work Item Tracking | WorkItemChangedEvent | No
Version Control | CheckInEvent | Yes
Team Build | BuildCompletionEvent2 | Yes
Team Build | BuildStatusChangeEvent | Yes
Team Build | BuildCompletionEvent | Yes
Common Structure Service | ProjectCreatedEvent | Yes
Common Structure Service | ProjectDeletedEvent | Yes
GSS/Authorization/CSS | DataChangedEvent | No

The new DataChangedEvent, defined in DataChangedEvent.xsd, replaces the events exposed via the vstfs service listed in Figure 2. When TFS fires a DataChangedEvent, you'll find it's one of three subtypes: Structure, Security, or Identity. You'll then need to call the appropriate Web service to get the details related to those events. TFS 2008 fires the BuildCompletionEvent2 event for all new subscriptions that you create for build completion. If you've upgraded your server, TFS will continue to fire the BuildCompleteEvent for any subscriptions that existed at the time of your upgrade.

Creating Subscriptions

If you want to create a subscription for an event not available in the Visual Studio Project Alerts dialog, you have two choices. The first choice is to use the BisSubscribe.exe command-line tool installed with TFS at C:\Program Files\Microsoft Visual Studio 2008 Team Foundation Server\TF Setup. Note that this tool is also available as a part of the Visual Studio Team Server (VSTS ) SDK, which is, in turn, a part of the larger Visual Studio SDK. However, the version in the SDK is out of date, and so you should verify that the version you are using is the same as the one provided with your TFS installation.

BisSubscribe.exe supports two commands: subscribe and unsubscribe. Subscribe is implicit. It supports up to six parameters for subscribe and two for unsubscribe. Figure 4 lists the parameters as provided by the tool's help command.

Figure 4 Command-Line Arguments for BisSubscribe.exe (Close Figure 4)

Parameter | Command | Notes
eventType | Subscribe | The name of the event. Case sensitive.
filter | Subscribe | A filter expression. Default is none.
address | Subscribe | The e-mail address or Web method URL for the subscriber.
server | Subscribe/Unsubscribe | The Team Foundation Server name.
tag | Subscribe | A field to use later to identify this subscription. Default is none.
deliveryType | Subscribe | Indicates the preferred message delivery type: EmailHtml, EmailPlaintext, or SOAP. Default is SOAP.
id | Unsubscribe | The integer ID for the subscription to be deleted when unsubscribing.

For example, if you want to subscribe to the check-in event for the entire server, you could issue this command (all on one line):

bissubscribe /eventType CheckinEvent
/address brianr@tfs.local /deliveryType EmailHtml
/server tfsrtm08

When you execute this command, BisSubscribe returns a message similar to the following:

TF50001:  Created or found an existing subscription.
The subscription ID is 7.

Using this subscription ID, you can unsubscribe:

bissubscribe unsubscribe /id 7 /server tfsrtm08

Note that you need to manually track the ID returned from your subscribe commands if you don't plan on writing code. There's no way to get a list of subscriptions from the command line. While you can poke around inside the SQL Server® 2005 databases, that is neither recommended nor supported.

Of course, you might not want e-mail for every event. Thus, you can add a bit of filtering:

bissubscribe /eventType CheckinEvent /address brianr@tfs.local
  /deliveryType EmailHtml /server tfsrtm08
  /filter "'Artifacts/Artifact[@TeamProject = MSFAgile]' <> null"

What this filter will do is tell TFS that you're only interested in getting an e-mail notification whenever a check-in occurs in the MSFAgile team project.

While BisSubscribe works, it isn't the most intuitive tool. It also does not provide a way to get a list of your existing subscriptions. Therefore, your second choice—to write some of your own code—looks more promising.

I've created a simple app that you can download. It builds on code from earlier columns on work items that provide the UI and framework for connecting to a TFS 2008 installation. To get the list of subscriptions for your user account, you need to connect to EventService by using the TeamFoundationServer object's GetService method and retrieve an IEventService reference. IEventService exposes an EventSubscriptions method that accepts a user ID in the form of User Domain Name and User ID and returns an array of zero or more Subscription instances:

Dim es As IEventService = _
  DirectCast(mtfs.GetService( _
    GetType(IEventService)), IEventService)

Dim userSubs As Subscription() = _
  es.EventSubscriptions( _
  Environment.UserDomainName & _
  "\" & Environment.UserName)

For Each usersub As Subscription In userSubs
  ' Do something
Next

The Subscription object provides a number of properties, including the all-important ID necessary to unsubscribe a user from an event. Using the ID property, you simply execute the UnsubscribeEvent method on a valid IEventService reference and you're finished.

Creating a new subscription is just as easy, with one caveat. IEventService exposes a SubscribeEvent method. It accepts a number of parameters just like BisSubscribe.exe. The hard work is that you need to create a good user experience for defining the filter expression and you also have to write code to ensure expression correctness. You'll find a community project available on the CodePlex site (codeplex.com/tfseventsubscription) that provides sample code for defining event filters.

Creating a Custom Event Listener

While e-mail notifications are useful, you might also want to execute code in response to a particular event. The EventService service supports SOAP-based notifications, allowing you to catch and process events from just about any type of code you prefer. An easy way to get started is by looking at existing code.

For Team Foundation Server 2005, Microsoft released a sample showing how you could implement Continuous Integration using EventService and some code. There's an article at msdn2.microsoft.com/ms364045 and a code download available. This sample is no longer necessary (and it won't work correctly out of the box) with TFS 2008. However, it provides an example of how you can create your own ASMX-based Web service using just a little bit of C#. Using the Microsoft example for inspiration, I've created a simple Web service that catches the WorkItemChanged event.

The first step requires creating a Web service project. To test this code, I used the Virtual PC image provided by Microsoft—you really want to try this stuff out first before you mess with a production server. You can download the image from go.microsoft.com/fwlink/?LinkId=114644.

You'll want to use the ASP.NET Web Service Application template for Visual Basic®. Set the name to MSDNMagTFSEvents and place it at C:\Program Files\Microsoft Visual Studio 2008 Team Foundation Server\Web Services\. Also, make sure to uncheck the "Create directory for solution" option.

After Visual Studio has created the project, adjust the project's properties so that the project URL uses port 8080 (do this via the Web tab in the Project designer). Then, using the IIS management tools, create an IIS application for your project under the Team Foundation Server Web. Ensure that you put your IIS application in the Microsoft® Team Foundation Server Application Pool. After you've done this, try to debug your service to make sure you got the configuration correct.

Now, rename Service1.asmx to ProcessEvents.asmx. Change the backing class name from Service1 to ProcessEvents. Finally, if you feel so inclined, update the Web service namespace to something appropriate. Replace the existing HelloWorld WebMethod with the prototype shown in Figure 5.

Figure 5 ProcessEvents WebMethod (Close Figure 5)
<SoapDocumentMethod( _
  "http://schemas.microsoft.com/TeamFoundation/2005" & _
  "/06/Services/Notification/03/Notify", _
  RequestNamespace:="http://schemas.microsoft.com" & _
  "/TeamFoundation/2005/06/Services/Notification/03")> _
  <WebMethod()> _
Public Sub Notify(ByVal eventXml As String, _
                 ByVal tfsIdentityXml As String)

End Sub

The first parameter represents the XML document containing the event information. Think of it as being equivalent to the EventArgs parameter in regular Microsoft .NET Framework events. The second parameter identifies the TFS server raising the event.

At this point, you have all you need to get going. When TFS raises an event, EventService will call your Web service. What you do in the method is up to you and your creativity. Figure 6 provides a simple example that writes to the Windows® event log. The code first checks to verify that the event Xml parameter has data. If not, it exits. Assuming there is data to parse, the code uses LINQ to XML to see whether the collection of changed fields includes the Title. If it does, the code grabs the old and new values, as well as the work item's ID, and writes the data out.

Figure 6 Writing to the Event Log (Close Figure 6)
<SoapDocumentMethod( _
  "http://schemas.microsoft.com/TeamFoundation/2005/06" & _
  "/Services/Notification/03/Notify", _
  RequestNamespace:="http://schemas.microsoft.com/TeamFoundation/" & _
  "2005/06/Services/Notification/03")> _
<WebMethod()> _
Public Sub Notify(ByVal eventXml As String, _
  ByVal tfsIdentityXml As String)

  Const logSource As String = "MSDNMagTFSEvents"
  Const logApp As String = "ProcessEvents"
  Dim el As New EventLog(logApp)
  el.Source = logSource

  Try
    el.WriteEntry("Notify event called.")

    If eventXml Is Nothing OrElse eventXml = String.Empty Then
      el.WriteEntry("eventXml data is empty.")
      Return
    Else
      Dim x As XElement = XElement.Load(New StringReader(eventXml))
      Dim referenceName = (From element _
        In x...<ChangedFields>...<ReferenceName> _
        Where element.Value = "System.Title").SingleOrDefault
      If referenceName IsNot Nothing Then
        Dim oldValue = referenceName.Parent.<OldValue>.Value
        Dim newvalue = referenceName.Parent.<NewValue>.Value
        Dim refNameId = (From element _
          In x...<CoreFields>...<ReferenceName> _
          Where element.Value = "System.Id").SingleOrDefault

        ' the Work Item's ID is always included.
        Dim id = refNameId.Parent.<OldValue>.Value
        ' Do something with the data
        Dim sw As New StringWriter()

        sw.WriteLine("Work Item {0} was changed.", id)
        sw.WriteLine("Old Title: " & oldValue)
        sw.WriteLine("New Title: " & newvalue)

        el.WriteEntry(sw.ToString())
      Else
        ' The title wasn't changed.
        ' Do something else
      End If
    End If

  Catch ex As Exception
    el.WriteEntry(ex.Message, EventLogEntryType.Error)
  End Try
End Sub

Note that the code to create the event log's event source is defined in the sample client application and needs to run first; otherwise, calls to write to the event log will fail. In a real application, you would have your installer define the event source.

To have TFS call your Web service, you will need to create an event subscription for it. You can do that using BisSubscribe with the following command line:

bissubscribe /eventType WorkItemChangedEvent
  /address http://localhost:8080/MSDNMagTFSEvents/ProcessEvents.asmx
  /deliveryType Soap
  /server tfsrtm08

To create an event subscription in code, you would need to write something like the example in Figure 7, also defined in the sample client application.

Figure 7 Subscribing to Events (Close Figure 7)
If mtfs IsNot Nothing Then
  UpdateStatus(stSubscribeforWebService, True)

  Dim es As IEventService = _
   DirectCast(mtfs.GetService(GetType(IEventService)), IEventService)

  Dim dp As New DeliveryPreference()
  dp.Address = String.Format("http://{0}{1}", mtfs.Uri.Authority, _
    "/MSDNMagTFSEvents/ProcessEvents.asmx")
  dp.Schedule = DeliverySchedule.Immediate
  dp.Type = DeliveryType.Soap

  Dim filter As String = Nothing
  If Me.chkOnlySelectedProject.Checked Then
    filter = String.Format("PortfolioProject = '{0}'", _
      Me.cboTeamProjects.SelectedItem.ToString())
  End If

  Try
    Dim subId As Integer = es.SubscribeEvent( _
      mtfs.AuthenticatedUserName, "WorkItemChangedEvent", filter, dp)

    MsgBox(String.Format(otSubscriptionAdded, subId), _
           MsgBoxStyle.Information, otSuccess)

    Me.btnListMySubs_Click(Me.btnListMySubs, EventArgs.Empty)
  Catch ex As Exception
    MsgBox(ex.Message, MsgBoxStyle.Critical, ex.Source)
  End Try

  UpdateStatus(stStatus, False)
End If

Final Details

As with other event-related code, you need to connect to the EventService service using a valid TFS reference. You then create a DeliveryPreference instance and define its properties. The sample code for this column uses the TFS reference to acquire the name and port for the TFS application tier where it assumes you installed the sample Web service. If this is not the case, you'll need to change this in your deployment.

The code sets the delivery schedule to be immediate; however, you can choose either daily or weekly as alternative intervals. You'll want to ensure you use the DeliveryType.Soap option when configuring Web service subscriptions.

Next, you can define a filter string. The sample code provides an option to restrict the notifications to a particular team project, reading the names from the form's combobox of team project names. After you've defined the parameters, you call the Subscribe method. If successful, EventService returns the ID for the new subscription.

There are a few things you need to consider when building your own Web services. If you followed the instructions in this column to configure the service on the application tier, the service runs using the tfsservice account's security context. You did this so that, if necessary, the service can access additional privileged TFS services. If this is not necessary, feel free to use a different deployment model (machine or account) for your own solution.

A second issue you need to be aware of relates to creating event subscriptions. TFS only allows users who are members of the TFS Administrators or the TFS Services security groups to create subscriptions that call Web services.

The TFS EventService service provides lots of hidden power and flexibility just waiting for you to employ in your own applications. Using the code provided here, as well as additional code you can access on Internet sites such as CodePlex, you can mold TFS to fit perfectly in your organization.

Special thanks, as always, to the VSTS Team for answering questions. In particular, Bill Essary from Microsoft provided invaluable information.

Send your questions and comments to mmvsts@microsoft.com.

Brian A. Randell is a Senior Consultant with MCW Technologies LLC. He spends his time speaking, teaching, and writing about Microsoft technologies. Brian is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


{ End Bracket }: Rich and Reach Applications

{ End Bracket }
Rich and Reach Applications
Terry Crowley

The difference between rich and reach applications used to be clear. Rich applications ran natively on a PC, leveraging local graphics, CPU, memory, and storage capabilities. Reach applications were written in HTML and deployed in the browser, minimizing system dependencies. Now we're seeing a blurring of the boundaries. AJAX applications download significant amounts of code and can deliver polished, responsive applications. Runtimes such as Flash, Silverlight™, and Google Gears allow you to create applications that can leverage local CPU and graphics capabilities, local storage, or that can run offline.

On the rich applications side, features, such as ClickOnce deployment for managed applications, or application virtualization and streaming tools, such as Microsoft® SoftGrid®, greatly reduce the deployment hassles for native rich applications. Even more important is the increasing capabilities and sophistication of rich clients in interacting with remote data. Microsoft OneNote® is a great example of this. While often thought of as a personal note-taking application, OneNote provides great "it just works" sharing of notebooks stored on a server or service. Transparent caching of remote data in the background allows the app to stay always responsive, even in the face of varying network connectivity. Local caching supports offline access. Seamless merging of changes provides wiki-like collaboration without having any awkward manual overhead.

HTML reach applications initially realized benefits in speed of development by "walking away" from complexity and focusing on key business problems. As you add back features, you add complexity. The asynchronous nature of AJAX applications is critical for a responsive user experience, but managing the state of multiple outstanding asynchronous interactions and their impact on the user experience can be complex.

Check it Out

See Terry Crowley's "Behind the Code" interview at channel9.msdn.com/Shows/Behind_The_Code.

The Model View Controller design pattern is great for providing a robust programming model, but the boundary between model and view gets stretched when it gets distributed between server and browser. Providing local responsiveness and offline capabilities argues for moving more of the model into the local client, but that complicates the client/server boundary and how state and the underlying application model is managed across this boundary. There is inherent stress when attempting to minimize the amount of data that needs to be downloaded to the client and then managing how that partial model is manipulated locally.

While scripting languages like JavaScript can be incredibly productive, as the code running in these reach applications becomes larger and more complex, languages like ECMAScript 4 or C# in Silverlight that support "programming in the large" become critical for managing this complexity. I have great respect for teams that have built applications with tens of thousands of lines of JavaScript, but JavaScript simply does not have the critical language features for structuring large applications that we have dealt with for decades.

As we consider providing offline capabilities to reach applications, I am struck by how trivial most commentators assume this is. Give me access to a local key-value store or a simple database and I'm done. Not likely. Most interesting applications have a rich semantic model. The richer the model, the more complicated the local state and the more complicated the process (including the user model) of what merging that local state back to the server means.

For rich applications interacting with remote data, I always keep in mind a quote from Pat Helland of Microsoft: "We live in an Einsteinian universe; there is no such thing as simultaneity." Database techniques like two-phase commit essentially try to provide the illusion of simultaneity, but at the cost of responsiveness and the ability to scale out gracefully.

A better approach is to accept that distributed components in a system are always going to have different views of the "world" and to shoot for a looser target of eventual consistency. This means that instead of struggling to keep the underlying model of distributed components of the system consistent at all times (and building fragile, unresponsive systems as a result), you should accept the inconsistency and focus your effort on how to robustly resolve those inevitable inconsistencies. This model of loose coupling ends up being a good architectural strategy in many places; it's useful to recognize that it has a core basis in the physics of our universe.

Ultimately, as the boundaries and capabilities of reach and rich applications converge, the problems that developers building these apps face will converge as well. If you're solving a difficult problem, you need a great design strategy and great tools as well.

Terry Crowley is a Microsoft Technical Fellow and Director of Development for the Office Division. He has been building rich Internet-enabled applications for the past 25 years at Microsoft, Vermeer, Beyond, and BBN.