 MSDN Magazine March 2008

MSDN Magazine March 2008

ASP.NET MVC: Building Web Apps without Web Forms

Chris Tavares explains how the ASP.NET MVC Framework's Model View Controller pattern helps you build flexible, easily tested Web applications.

Chris Tavares Loosen Up: Tame Your Software Dependencies for More Flexible Apps

James Kovacs explains the dark side of tightly coupled architectures, why they're hard to test and how they limit adaptation. He then proposes a number of solutions.

James Kovacs CI Server: Redefine Your Build Process with Continuous Integration

Jay Flowers demonstrates how to set up and use a Continuous Integration server using both discrete tools and the more comprehensive CI Factory solution.

Jay Flowers Performance: Find Application Bottlenecks with Visual Studio Profiler

We will introduce you to the Visual Studio Profiler by walking through a sample performance investigation, pinpointing code inefficiencies in some sample applications.

Hari Pulapaka and Boris Vidolov Office Development: OBA Solution Patterns In The Real World

OBA solution patterns help architects and developers build Office Business Applications (OBAs). This article introduces the seven core OBA solution patterns and applies one to a real-world problem.

Steve Fox Editor's Note: Too Many Swimming Pools

This month Howard Dierking cautions against designing applications in a vacuum with no connection to reality.

Howard Dierking Toolbox: Easy Web Debugging, Extending Reflector, and More

This month, a better design-time experience for Web apps, Scott Guthrie's blog, extending Extend Lutz Roeder's Reflector, and more.

Scott Mitchell CLR Inside Out: Improving Application Startup Performance

Application startup performance matters to users, and there's plenty you can do to improve it. Here's a look at where to begin.

Claudio Caldato Data Points: Standard Query Operators with LINQ

LINQ to Objects and LINQ to Entities have an arsenal of Standard Query Operators that operate on sequences to perform a wide variety of operations.

John Papa Advanced Basics: Office 2007 Files and LINQ

LINQ to XML and the Microsoft SDK for Open XML Formats simplify access to the parts of a 2007 Office system Open XML document when retrieving or modifying data, resulting in shorter, less complex code.

Ken Getz Office Space: Simplify SharePoint Development with STSDEV

Ted Pattison shows how to use a new STSDEV utility to set up and deploy SharePoint development projects in Visual Studio in an easy and repeatable manner.

Ted Pattison Test Run: Web UI Automation with Windows PowerShell

Here we show you how to use Windows PowerShell to create quick and easy UI test automation for ASP.NET and classic ASP Web applications.

Dr. James McCaffrey Security Briefs: Protecting Your Code with Visual C++ Defenses

Michael Howard outlines some of the buffer overrun defenses available in Visual C++ 2005 and beyond.

Michael Howard Extreme ASP.NET: The Only Data-binding Control You'll Ever Need

Fritz Onion demonstrates how the ListView control in ASP.NET 3.5 makes data-binding tasks easier with support for styling with CSS, flexible pagination, and a full complement of sorting, inserting, deleting, and updating features.

Fritz Onion Foundations: Vector Graphics and the WPF Shape Class

The System.Windows.Shapes namespace is Charles Petzold's namespace of choice for rendering two-dimensional vector graphics in WPF. Here he explains why.

Charles Petzold .NET Matters: Asynchronous Stream Processing

This month Stephen Toub discusses asynchronous stream processing.

Stephen Toub { End Bracket }: What is ALT .NET?

Jeremy D. Miller explains the motivation behind the ALT .NET movement.

Jeremy D. Miller


ASP.NET MVC: Building Web Apps without Web Forms

ASP.NET MVC

Building Web Apps without Web Forms

Chris Tavares
This article is based on a prerelease version of the ASP.NET MVC Framework. Details herein are subject to change.
This article discusses:
  • Model View Controller pattern
  • Creating controllers and views
  • Building forms and posting back
  • Controller factories and other extensibility points
This article uses the following technologies:
ASP.NET
Code download available at: MVCFramework2008_03.exe (189KB)
Contents
Model View Controller Pattern
Creating a Controller
Creating a View
A Bigger Example
Creating Forms and Posting Back
Controller Creation
Other Extensibility Points
Farewell Web Forms?

I
have been a professional developer for about 15 years now and a hobbyist for at least 10 years before that. Like most of my generation, I started out on 8-bit machines and then moved on to the PC platform. Working my way up through machines of increasing complexity, I wrote apps that did everything from small games to personal data management to controlling external hardware.

For the first half of my career, however, all the software I wrote had one thing in common: it was always local applications running on a user's desktop. In the early '90s I started hearing about this new thing called the World Wide Web. I saw an opportunity here to build a Web application that let me enter my timecard information without actually having to schlep back to my office from the work site.

The experience was, in a word, baffling. Coming to grips with the stateless Web just didn't click with my desktop-oriented mindset. Add lousy debugging, a UNIX server I didn't have root access to, and that weird angle bracket stuff, and my younger self shamefully dove back into desktop development for several more years.

I stayed away from Web development; it was obviously important, but I didn't really understand the programming model. Then the Microsoft® .NET Framework and ASP.NET were released. Finally, here was a framework that let me work on Web applications, yet it was almost like programming desktop applications. I could build windows (pages), hook up controls to events, and the designer kept me from having to deal with those darn angle brackets. And best of all, ASP.NET automatically handled the stateless nature of the Web for me with view state! I was a happy programmer again ... at least for a while.

As my experience grew, so did my choices in design. I'd learned several best practices that I applied when working on my desktop apps. Two of them were:

  • Separation of concerns: don't mix UI logic with underlying behavior.
  • Automated unit testing: write automated tests that verify whether your code does what you think it does.

The underlying principles here apply regardless of technology. Separation of concerns is a fundamental principle that exists to help you deal with complexity. Mixing Different responsibilities within the same object—like calculating remaining work hours, formatting data, and drawing a graph—is just asking for maintenance problems. And automated testing is crucial to getting production-quality code while still maintaining your sanity, particularly when you're updating an existing project.

ASP.NET Web Forms made it very easy to get started, but, in other ways, trying to apply my design principles to Web apps was a struggle. Web Forms are relentlessly UI focused; the fundamental atom is the page. You start by designing your UI and dragging controls. It's very seductive to just start slapping your application logic into the page's event handlers (much like Visual Basic® enabled for Windows® apps).

On top of that, unit testing of pages is often difficult. You can't run a Page object through its lifecycle without spinning up all of ASP.NET. While it is possible to test Web apps by sending HTTP requests to a server or automating a browser, that kind of testing is fragile (change one control ID and the test breaks), hard to set up (you have to set up the server on every developer's machine exactly the same way), and slow to run.

As I started building more sophisticated Web apps, the abstractions that Web Forms provides, like controls, view state, and the page lifecycle, started to chafe rather than help. I was spending more and more time configuring data binding (and writing tons of event handlers to get it configured correctly). I had to figure out how to reduce the view state size to get my pages to load faster. Web Forms require that a physical file exist at every URL, something that dynamic sites (like a wiki, for example) make difficult. And successfully writing a custom WebControl is a remarkably complex process that requires a comprehensive understanding of both page lifecycle and the Visual Studio® designer.

Since coming to work at Microsoft, I've had the opportunity to share my learning about various .NET pain points and to hopefully alleviate some of that pain. Most recently, such an opportunity arose through my participation as a developer on the patterns & practices Web Client Software Factory project (codeplex.com/websf). In particular, one of the things that patterns & practices builds into its deliverables is automated unit testing. In the Web Client Software Factory, we proposed the use of the Model View Presenter (MVP) pattern for building testable Web Forms.

In a nutshell, instead of putting your logic in the page, MVP has you build your pages so that the page (View) simply makes calls into a separate object, the Presenter. The Presenter object then performs any logic necessary to respond to the activity on the view, typically by using other objects (the Model) to access databases, perform business logic, and so on. Once those steps are complete, the Presenter updates the view. This approach gives you testability because the presenter is isolated from the ASP.NET pipeline; it communicates with the view through an interface and can be tested in isolation from the page.

MVP works, but the implementation can be a bit awkward; you need a separate view interface, and you have to write lots of event forwarding functions in your codebehind files. But if you want a testable UI in your Web Forms applications, it's about the best you're going to get. Any improvements would require a change in the underlying platform.

Model View Controller Pattern

Luckily, the ASP.NET team has been listening to developers like me and has started development of a new Web application framework that sits side-by-side with the Web Forms you know and love but has a distinctly different set of design goals:

  • Embrace HTTP and HTML—don't hide it.
  • Testability is built-in from the ground up.
  • Extensible at almost every point.
  • Total control over your output.

This new framework is based around the Model View Controller (MVC) pattern, thus the name, ASP.NET MVC. The MVC pattern was originally invented back in the '70s as part of Smalltalk. As I'll show in this article, it actually fits into the nature of the Web quite well. MVC divides your UI into three distinct objects: the controller, which receives and handles input; the model, which contains your domain logic; and the view, which generates your output. In the context of the Web, the input is an HTTP request, and the request flow looks like Figure 1.

Figure 1 MVC Pattern Request Flow
Figure 1 MVC Pattern Request Flow(Click the image for a smaller view)
Figure 1 MVC Pattern Request Flow
Figure 1 MVC Pattern Request Flow (Click the image for a larger view)

This is actually quite different from the process in Web Forms. In the Web Forms model, the input goes into the page (the View), and the view is responsible for both handling the input and generating the output. When it comes to MVC, on the other hand, the responsibilities are separated.

So, right now there's probably one of two things running through your head. Either, "Hey, this is great. How do I use it?" or "Why would I write three objects when I only had to write one before?" Both are excellent questions and best explained by looking at an example. So I'm going to write a small Web app using the MVC Framework to demonstrate its advantages.

Creating a Controller

To follow along, you'll need to install Visual Studio 2008 and get a copy of the MVC Framework. At the time of this writing, it's available as part of the December 2007 Community Technology Preview (CTP) of the ASP.NET Extensions (asp.net/downloads/3.5-extensions). You'll want to grab both the extensions CTP and the MVC Toolkit, which includes some very useful helper objects. Once you download and install the CTP, you'll get a new project type in your New Project dialog called ASP.NET MVC Web Application.

Selecting the MVC Web Application project gives you a solution that looks a little bit different from the usual Web site or application. The solution template creates a Web application with some new directories (as shown in Figure 2). In particular, the Controllers directory contains the controller classes, and the Views directory (and all of its subdirectories) contains the views.

Figure 2 The MVC Project Structure
Figure 2 The MVC Project Structure

I'm going to write a very simple controller that returns a name passed in on the URL. Right-clicking on the Controllers folder and choosing Add Item displays the usual Add New Item dialog, with some new additions, including an MVC Controller Class and several MVC View components. In this case, I'm adding an ever-so-imaginatively named HelloController class:

using System;
using System.Web;
using System.Web.Mvc;

namespace HelloFromMVC.Controllers
{
    public class HelloController : Controller
    {
        [ControllerAction]
        public void Index()
        {
            ...
        }
    }
}

A controller class is a lot lighter weight than a page. As a matter of fact, the only things that are truly necessary are to derive from System.Web.Mvc.Controller and to put the [ControllerAction] attribute on your action methods. An action is a method that's called in response to a request to a particular URL. Actions are responsible for doing whatever processing is required and then rendering a view. I'll start by writing a simple action that passes the name along to the view, as you can see here:

[ControllerAction]
 public void HiThere(string id)
 {
     ViewData["Name"] = id;
     RenderView("HiThere");
 }

The action method receives the name from the URL via the id parameter (more about how in a moment), stores it in the ViewData collection, and then renders a view named HiThere.

Before discussing how this method gets called, or what the view looks like, I'd like to talk about testability. Remember my previous comments about how hard it is to test Web Forms page classes? Well, controllers are a lot easier to test. In fact, a controller can be instantiated directly, and action methods called, without any additional infrastructure. You don't need an HTTP context, and you don't need a server, just a test harness. As an example, I've included a Visual Studio Team System (VSTS) unit test for this class in Figure 3.

Figure 3 Controller Unit Test (Close Figure 3)
namespace HelloFromMVC.Tests
{
    [TestClass]
    public class HelloControllerFixture
    {
        [TestMethod]
        public void HiThereShouldRenderCorrectView()
        {
            TestableHelloController controller = new
              TestableHelloController();
            controller.HiThere("Chris");

            Assert.AreEqual("Chris", controller.Name);
            Assert.AreEqual("HiThere", controller.ViewName);
        }

    }

    class TestableHelloController : HelloController
    {
        public string Name;
        public string ViewName;

        protected override void RenderView(
            string viewName, string master, object data)
        {
            this.ViewName = viewName;
            this.Name = (string)ViewData["Name"];
        }
    }

}

There are several things going on here. The actual test is very straightforward: instantiate the controller, call the method with the expected data, then verify that the correct view was rendered. I do the check by creating a Test-specific subclass that overrides the RenderView method. This allows me to short-circuit the actual creation of HTML. I just care that the right data was sent to the view and that the correct view was rendered. I don't care about the underlying details of the view itself for this test.

Creating a View

Of course, ultimately it is necessary for me to generate some HTML, so let's create that HiThere view. To do this, first I create a new folder in the solution named Hello under the Views folder. By default, the controller will look for a view in the Views\<ControllerPrefix> folder (the controller prefix is the name of the controller class minus the word "Controller"). So for views rendered by the HelloController, it looks in Views\Hello. The solution ends up looking like Figure 4.

Figure 4 Adding a View to the Project
Figure 4 Adding a View to the Project(Click the image for a smaller view)
Figure 4 Adding a View to the Project
Figure 4 Adding a View to the Project (Click the image for a larger view)

The HTML for the view looks like this:

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Hi There!</title>
</head>
<body>
    <div>
        <h1>Hello, <%= ViewData["Name"] %></h1>
    </div>
</body>
</html>

Several things should leap out at you. There are no runat="server" tags. There's no form tag. There are no control declarations. In fact, this looks a lot more like classic ASP than ASP.NET. Note that MVC views are only responsible for generating output, so they don't need any of the event handling or complex controls that Web Forms pages do.

The MVC Framework does borrow the .aspx file format as a useful text templating language. You can even use codebehind if you want, but by default the codebehind file looks like this:

using System;
using System.Web;
using System.Web.Mvc;

namespace HelloFromMVC.Views.Hello
{
    public partial class HiThere : ViewPage
    {
    }
}

No page Init or load methods, no event handlers, nothing except the declaration of the base class, which is not Page but instead ViewPage. This is all you need to be an MVC view. Run the app, navigate to http://localhost:<port>/Hello/HiThere/Chris, and you'll see something like Figure 5.

Figure 5 Successful MVC View
Figure 5 Successful MVC View(Click the image for a smaller view)
Figure 5 Successful MVC View
Figure 5 Successful MVC View (Click the image for a larger view)

If, instead of Figure 5, you see a nasty looking exception, don't panic. If you have the HiThere.aspx file set as the active document in Visual Studio when you press F5, Visual Studio will attempt to access the .aspx file directly. Since MVC views require the controller to have run before displaying, trying to navigate directly to the page will not work. Just edit the URL to match what you see in Figure 5, and it should work fine.

How'd the MVC Framework know to call my action method? There wasn't even a file extension for that URL. The answer is URL routing. If you look inside the global.asax.cs file, you'll see the chunk of code in Figure 6. The global RouteTable stores a collection of Route objects. Each Route describes a URL form and what to do with it. By default, two routes get added to the table. The first is what does the magic. It says that for each URL that consists of three parts after the server name, the first part should be taken as a controller name, the second as an action name, and the third as the ID parameter:

Figure 6 Route Table (Close Figure 6)
public class Global : System.Web.HttpApplication
{
    protected void Application_Start(object sender, EventArgs e)
    {
        // Change Url= to Url="[controller].mvc/[action]/[id]"
        // to enable automatic support on IIS6

        RouteTable.Routes.Add(new Route
        {
            Url = "[controller]/[action]/[id]",
            Defaults = new { action = "Index", id = (string)null },
            RouteHandler = typeof(MvcRouteHandler)
        });

        RouteTable.Routes.Add(new Route
        {
            Url = "Default.aspx",
            Defaults = new {
                controller = "Home",
                action = "Index",
                id = (string)null },
            RouteHandler = typeof(MvcRouteHandler)
        });
    }
}

Url = "[controller]/[action]/[id]"

This default route is what enabled my HiThere method to get invoked. Remember this URL: http://localhost/Hello/HiThere/Chris? This route mapped Hello to the controller, HiThere to the action, and Chris to the ID. The MVC Framework then created a HelloController instance, called the HiThere method, and passed Chris as the value of the ID parameter.

This default route gives you a lot, but you can add your own routes as well. For example, I want a really friendly site where folks should only have to enter their name for a personalized greeting. If I add this route to the top of the routing table

  RouteTable.Routes.Add(new Route
  {
    Url = "[id]",
    Defaults = new {
        controller = "Hello",
        action = "HiThere" },
    RouteHandler = typeof(MvcRouteHandler)
  });

I can then simply go to http://localhost/Chris and my action is still invoked, and I see my familiar friendly greeting.

How did the system know which controller and action to invoke? The answer is in the Defaults parameter. This makes use of the new C# 3.0 anonymous type syntax to create a pseudo dictionary. The Defaults object on the Route can contain arbitrary extra information, but for MVC it can also contain some well-known entries: controller and action. If there's no controller or action specified in the URL, then it'll use the name in the Defaults. That's why I can leave them out of the URL and still have my request mapped to the correct controller and action.

One more thing to note: remember that I said "add to the top of the table"? If you put it at the bottom, you'll instead get an error. Routing works on a first come, first served basis. When processing URLs, the routing system walks the table from top to bottom, and the first route to match wins. In this case, the default route "[controller]/[action]/[id]" matches because there are default values for the action and ID. Thus, it goes looking for ChrisController, and since I don't have a controller I get an error.

A Bigger Example

Now that I've demonstrated the basics of the MVC Framework, I'd like to show you a larger example that does more than just display a string. A wiki is a Web site that can be edited in the browser. Pages can be added or edited easily. I've written a small sample wiki using the MVC Framework. The "Edit this page" screen is shown in Figure 7.

Figure 7 Editing the Home Page
Figure 7 Editing the Home Page(Click the image for a smaller view)
Figure 7 Editing the Home Page
Figure 7 Editing the Home Page (Click the image for a larger view)

You can check out the code download for this article to see how the underlying wiki logic is implemented. Right now I want to concentrate on how the MVC Framework made it easy to get the wiki onto the Web. I started out by designing my URL structure. I wanted the following:

  • /[pagename] displays the page with that name.
  • /[pagename]?version=n shows the requested version of the page, where 0 = the current version, 1 = the previous one, and so on.
  • /Edit/[pagename] opens the editing screen for that page.
  • /CreateNewVersion/[pagename] is the URL that is posted to in order to submit an edit.

Let's start off with the basic display of a wiki page. To do this, I created a new class named WikiPageController. I then added an action named ShowPage. The WikiPageController started out looking like Figure 8. The ShowPage method is pretty straightforward. The WikiSpace and WikiPage classes represent a set of wiki pages and a specific page (and its revisions), respectively. This action just loads up the model and calls RenderView. But what's that "new WikiPageViewData" line in there?

Figure 8 WikiPageController Implementation of ShowPage (Close Figure 8)
public class WikiPageController : Controller
{
  ISpaceRepository repository;

  public ISpaceRepository Repository
  {
    get {
      if (repository == null)
      {
        repository = new FileBasedSpaceRepository(
            Request.MapPath("~/WikiPages"));
      }
      return repository;
    }

    set { repository = value; }
  }

  [ControllerAction]
  public void ShowPage(string pageName, int? version)
  {
    WikiSpace space = new WikiSpace(Repository);
    WikiPage page = space.GetPage(pageName);

    RenderView("showpage",
      new WikiPageViewData
      {
        Name = pageName,
        Page = page,
        Version = version ?? 0
      });
  }
}

My previous example demonstrated one way to pass data from the controller to the view: the ViewData dictionary. Dictionaries are convenient, but they're also dangerous. They can contain absolutely anything, you don't get any IntelliSense® for the contents, and because the ViewData dictionary is of type Dictionary<string, object> to consume its contents you have to cast everything.

When you know what data you will need in the view, you can instead pass strongly typed ViewData object. In my case, I created a simple object, WikiPageViewData, as shown in Figure 9. This object carries the wiki page information to the view along with a couple of utility methods to do things like get the HTML version of the wiki markup.

Figure 9 WikiPageViewData Object (Close Figure 9)
public class WikiPageViewData {

    public string Name { get; set; }
    public WikiPage Page { get; set; }
    public int Version { get; set; }

    public WikiPageViewData() {
        Version = 0;
    }

    public string NewVersionUrl {
        get {
            return string.Format("/CreateNewVersion/{0}", Name);
        }
    }

    public string Body {
        get { return Page.Versions[Version].Body; }
    }

    public string HtmlBody {
        get { return Page.Versions[Version].BodyAsHtml(); }
    }

    public string Creator {
        get { return Page.Versions[Version].Creator; }
    }

    public string Tags {
        get { return string.Join(",", Page.Versions[Version].Tags); }
    }
}

Now that I've got the view data defined, how do I use it? In ShowPage.aspx.cs you'll see this:

namespace MiniWiki.Views.WikiPage {
    public partial class ShowPage : ViewPage<WikiPageViewData>
    {
    }
}

Notice that I have defined the base class to be of type ViewPage<WikiPageViewData>. This means that the page's ViewData property is of type WikiPageViewData and not a Dictionary like in the previous example.

The actual markup in the .aspx file is pretty straightforward:

<%@ Page Language="C#" MasterPageFile="~/Views/Shared/Site.Master"
  AutoEventWireup="true" CodeBehind="ShowPage.aspx.cs"
  Inherits="MiniWiki.Views.WikiPage.ShowPage" %>
<asp:Content
  ID="Content1"
  ContentPlaceHolderID="MainContentPlaceHolder"
  runat="server">
  <h1><%= ViewData.Name %></h1>
  <div id="content" class="wikiContent">
    <%= ViewData.HtmlBody %>
  </div>
</asp:Content>

Notice that I'm not using the indexing operator [] when referencing ViewData. Instead, since I now have a strongly typed ViewData, I can just access the property directly. No casts are required, and Visual Studio gives you IntelliSense.

The astute observer will have noticed the <asp:Content> tag in this file. Yes, Master Pages do work with MVC views. And Master Pages can also be views. Let's look at the Master Page codebehind:

namespace MiniWiki.Views.Layouts
{
    public partial class Site :
        System.Web.Mvc.ViewMasterPage<WikiPageViewData>
    {
    }
}

The associated markup is shown in Figure 10. Right now, the Master Page gets the exact same ViewData object as the view does. I've declared my Master Page's base class to be ViewMasterPage<WikiPageViewData> so that I have the proper type of ViewData. From there, I set up the various DIV tags to lay out my page, fill out the version list, and finish up with the usual content placeholder.

Figure 10 Site.Master (Close Figure 10)
<%@ Master Language="C#"
  AutoEventWireup="true"
  CodeBehind="Site.master.cs"
  Inherits="MiniWiki.Views.Layouts.Site" %>
<%@ Import Namespace="MiniWiki.Controllers" %>
<%@ Import Namespace="MiniWiki.DomainModel" %>
<%@ Import Namespace="System.Web.Mvc" %>
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
  <title><%= ViewData.Name %></title>
  <link href="http://../../Content/Site.css" rel="stylesheet" type="text/css" />
</head>
<body>
  <div id="inner">
    <div id="top">
      <div id="header">
        <h1><%= ViewData.Name %></h1>
      </div>
      <div id="menu">
        <ul>
          <li><a href="http://Home">Home</a></li>
          <li>
            <%= Html.ActionLink("Edit this page",
                  new { controller = "WikiPage",
                        action = "EditPage",
                        pageName = ViewData.Name })%>
        </ul>
      </div>
    </div>
    <div id="main">
      <div id="revisions">
        Revision history:
        <ul>
          <%
            int i = 0;
            foreach (WikiPageVersion version in ViewData.Page.Versions)
            { %>
              <li>
                <a href="http://<%= ViewData.Name %>?version=<%= i %>">
                  <%= version.CreatedOn %>
                  by
                  <%= version.Creator %>
                </a>
              </li>
          <%  ++i;
          } %>
        </ul>
      </div>
      <div id="maincontent">
        <asp:ContentPlaceHolder
          ID="MainContentPlaceHolder"
          runat="server">
        </asp:ContentPlaceHolder>
      </div>
    </div>
  </div>
</body>
</html>

One other thing to notice is the call to Html.ActionLink. This is an example of a rendering helper. The various view classes have two properties, Html and Url. Each has useful methods to output chunks of HTML. In this case, Html.ActionLink takes an object (here, of an anonymous type) and runs it back through the routing system. This produces a URL that will route to the controller and action I've specified. This way, no matter how I change my routes, the "Edit this page" link will always point to the right place.

You may also notice that I have also had to resort to manually building a link (the links to previous page versions). Unfortunately, the current routing system doesn't work that well for generating URLs where query strings are involved. This should be fixed in later versions of the framework.

Creating Forms and Posting Back

Now, let's look at the EditPage action on the controller:

[ControllerAction]
public void EditPage(string pageName)
{
  WikiSpace space = new WikiSpace(Repository);
  WikiPage page = space.GetPage(pageName);

  RenderView("editpage",
    new WikiPageViewData {
      Name = pageName,
      Page = page });
}

Again, the action doesn't do much—it just renders the view with the given page. Things get more interesting in the view, shown in Figure 11. This file is building an HTML form, but there's no Runat="server" to be seen. The Url.Action helper is used to generate the URL that the form posts back to. There are also several uses of various HTML helpers like TextBox, TextArea, and SubmitButton. They do pretty much what you'd expect: generate HTML for various input fields.

Figure 11 EditPage.aspx (Close Figure 11)
<%@ Page Language="C#"
  MasterPageFile="~/Views/Shared/Site.Master"
  AutoEventWireup="true"
  CodeBehind="EditPage.aspx.cs"
  Inherits="MiniWiki.Views.WikiPage.EditPage" %>
<%@ Import Namespace="System.Web.Mvc" %>
<%@ Import Namespace="MiniWiki.Controllers" %>
<asp:Content ID="Content1"
  ContentPlaceHolderID="MainContentPlaceHolder"
  runat="server">
  <form action="<%= Url.Action(
    new { controller = "WikiPage",
    action = "NewVersion",
    pageName = ViewData.Name })%>" method=post>
    <%
      if (ViewContext.TempData.ContainsKey("errors"))
      {
    %>
    <div id="errorlist">
      <ul>
      <%
        foreach (string error in
          (string[])ViewContext.TempData["errors"])
        {
      %>
        <li><%= error%></li>
      <% } %>
      </ul>
    </div>
    <% } %>
    Your name: <%= Html.TextBox("Creator",
                   ViewContext.TempData.ContainsKey("creator") ?
                   (string)ViewContext.TempData["creator"] :
                   ViewData.Creator)%>
    <br />
    Please enter your updates here:<br />
    <%= Html.TextArea("Body", ViewContext.TempData.ContainsKey("body") ?
        (string)ViewContext.TempData["body"] :
        ViewData.Body, 30, 65)%>
    <br />
    Tags: <%= Html.TextBox(
              "Tags", ViewContext.TempData.ContainsKey("tags") ?
              (string)ViewContext.TempData["tags"] :
              ViewData.Tags)%>
    <br />
    <%= Html.SubmitButton("SubmitAction", "OK")%>
    <%= Html.SubmitButton("SubmitAction", "Cancel")%>
  </form>
</asp:Content>

One of the more annoying things to deal with in Web programming is errors on a form. More specifically, you want to display error messages, but you want to keep the previously entered data. We've all had the experience of making a mistake on a form that has 35 fields, only to be presented with a bunch of error messages and a new, blank form. The MVC Framework offers TempData as a place to store the previously entered information so that the form can be repopulated. This is something that ViewState actually made very easy in Web Forms, since saving the contents of controls was pretty much automatic.

I'd like to do this in MVC as well, and that's where TempData comes in. TempData is a dictionary, much like the untyped ViewData. However, the contents of TempData only live for a single request and then they're deleted. To see how this is used, look at Figure 12, the NewVersion action.

Figure 12 NewVersion Action (Close Figure 12)
[ControllerAction]
public void NewVersion(string pageName) {
  NewVersionPostData postData = new NewVersionPostData();
  postData.UpdateFrom(Request.Form);

  if (postData.SubmitAction == "OK") {
    if (postData.Errors.Length == 0) {
      WikiSpace space = new WikiSpace(Repository);
      WikiPage page = space.GetPage(pageName);
      WikiPageVersion newVersion = new WikiPageVersion(
        postData.Body, postData.Creator, postData.TagList);
      page.Add(newVersion);
    } else {
      TempData["creator"] = postData.Creator;
      TempData["body"] = postData.Body;
      TempData["tags"] = postData.Tags;
      TempData["errors"] = postData.Errors;

      RedirectToAction(new {
        controller = "WikiPage",
        action = "EditPage",
        pageName = pageName });
      return;
    }
  }

  RedirectToAction(new {
    controller = "WikiPage",
    action = "ShowPage",
    pageName = pageName });
}

First, it creates a NewVersionPostData object. This is another helper object that has properties and methods that store the contents of the post and some validation. To load up the postData object, I'm using a helper function from the MVC Toolkit. UpdateFrom is actually an extension method provided by the toolkit, and it uses reflection to match up the names of form fields to names of properties on my object. The net result is that all the field values are loaded into my postData object. Use of UpdateFrom does have the disadvantage, however, of getting form data directly from HttpRequest, making unit testing more difficult.

The first thing the NewVersion checks is SubmitAction. This will be OK if the user clicked the OK button and actually wants to post the edited page. If any other value is here, the action ends up redirecting back to ShowPage, which just redisplays the original page.

If the user did click OK, then I check the postData.Errors property. This runs some simple validations on the post contents. If there aren't any errors, I do the processing to write the new version of the page back into the wiki. However, if there are errors, things get interesting.

If there are errors, I set the various fields of the TempData dictionary so that it contains the contents of the PostData. Then I redirect back to the Edit page. Now, since the TempData is set, the page will redisplay with the form initialized with the values the user posted the last time.

This process of dealing with posts, validation, and TempData is a little aggravating right now and requires a bit more manual work than is really necessary. Future releases should include helper methods that automate at least some of the checking of TempData. One last note on TempData: the contents of TempData are stored in the user's server-side session. If you turn session off, TempData won't work.

Controller Creation

The basics of the wiki are now working, but there are a few rough spots in the implementation I'd like to clean up before going further. For example, the Repository property is used to decouple the logic of the wiki with the physical storage. You can provide repositories that store content on the file system (just as I've done here), a database, or wherever else you want. Unfortunately, I've got two problems to resolve.

First, my controller class is tightly coupled to the concrete FileBasedSpaceRepository class. I need to have a default, so that if the property's not set, I have something reasonable to use. Even worse, the path to the files on disk is hardcoded here as well. At the very least this stuff should come from configuration.

Second, the Repository is really a required dependency; my object won't run without it. Good design indicates that the repository should really be a constructor parameter, not a property. But I can't add it to the constructor because the MVC Framework requires a no-argument constructor on controllers.

Luckily, there's an extensibility hook that can get me out of this bind: the controller factory. A controller factory does much like the name says: it creates Controller instances. You just need to create a class that implements the IControllerFactory interface and register it with the MVC system. You can register controller factories for all controllers or just for specific types. Figure 13 shows a controller factory for WikiPageController, passing the repository now as a constructor parameter.

Figure 13 Controller Factory (Close Figure 13)
public class WikiPageControllerFactory : IControllerFactory {

  public IController CreateController(RequestContext context,
    Type controllerType)
  {
    return new WikiPageController(
      GetConfiguredRepository(context.HttpContext.Request));
  }

  private ISpaceRepository GetConfiguredRepository(IHttpRequest request)
  {
    return new FileBasedSpaceRepository(request.MapPath("~/WikiPages"));
  }
}

In this case, the implementation is pretty trivial, but this can enable the creation controllers that use much more powerful tools (in particular dependency injection containers). In any event, now I have all the details of getting the dependencies for the controller separated into an object that's easier to manage and maintain.

The last step in making this work is to register the factory with the framework. I do this via the ControllerBuilder class by adding the following line to Global.asax.cs in the Application_Start method (either before or after the routes):

ControllerBuilder.Current.SetControllerFactory(
  typeof(WikiPageController), typeof(WiliPageControllerFactory));

This will register a factory for the WikiPageController. If I had other controllers in this project, they wouldn't use this factory, since it's registered only for the WikiPageController type. You can also call SetDefaultControllerFactory if you want to set a factory to be used for every controller.

Other Extensibility Points

The controller factory is just the start of the framework extensibility. I don't have space in this article to get into details about all of them, so I'll just hit the highlights. First, if you want to output something other than HTML or if you want to use a templating engine other than Web Forms, you can set the controller's ViewFactory to something else. You can implement the IViewFactory interface, and then you've got complete control over how the output is generated. This is useful for generating RSS, XML, or even graphics.

The routing system is quite flexible, as you've already seen. But there's nothing in the routing system that is specific to MVC. Every route has a RouteHandler property; so far, I've always set this to MvcRouteHandler. But it is possible to implement the IRouteHandler interface and hook the routing system up to other Web technologies. A future drop of the framework will be shipping with a WebFormsRouteHandler, and other technologies will be taking advantage of the generic routing system in the future.

Controllers don't have to derive from System.Web.Mvc.Controller. All a controller need do is implement the IController interface, which has only a single method called Execute. From there you can do whatever you like. If you only want to tweak a few behaviors of the base Controller class, on the other hand, Controller has many virtual functions you can override:

  • OnPreAction, OnPostAction, and OnError let you hook up generic pre- and post-processing on every action that gets executed. OnError gives you a controller-wide error handling mechanism.
  • HandleUnknownAction is called when a URL gets routed to the controller but that controller doesn't implement the action requested in the route. By default, this method throws an exception, but you can override it to do anything you'd like.
  • InvokeAction is the method that figures out what action method to call and calls it. If you'd like to customize the process (for example, to get rid of the requirement for [ControllerAction] attributes), this is the place to do it.

There are several more virtual methods on Controller, but they're primarily there as testing hooks rather than as extension points. For example, RedirectToAction is virtual so that you can create a derived class that doesn't actually redirect. This lets you test actions that redirect without needing a full Web server running.

Farewell Web Forms?

At this point you may be wondering, "What's happening to Web Forms? Is MVC replacing it?" The answer is no! Web Forms is a well-understood technology, and Microsoft will continue to support and enhance it. There are many applications where Web Forms work very well; the typical intranet database reporting app, for instance, can be created using Web Forms in a fraction of the time it would take to write it in MVC. Plus Web Forms supports a vast marketplace of controls, many of which are extremely sophisticated and save massive amounts of work.

So, when should you choose MVC over Web Forms? In many ways, it comes down to your requirements and your preferences. Are you struggling to get your URLs formed the way you want? Do you want to unit test your UI? Either of those scenarios would lean toward MVC. On the other hand, are you doing a lot of data display, with editable grids and fancy tree view controls? Then you're probably better off with Web Forms for the moment.

Over time, the MVC Framework will probably catch up in the UI control department, but most likely it'll never be as easy to get started with as with Web Forms, where massive amounts of functionality are just a drag and drop away. But in the meantime, the ASP.NET MVC Framework offers Web developers a new way to build Web applications in the Microsoft .NET Framework. The Framework is designed for testability, embraces HTTP instead of trying to abstract it away, and is extensible at just about every point. It is a compelling complement to Web Forms for those developers who want complete control over their Web applications.

Chris Tavares is a Developer on the Microsoft patterns & practices team, where he works to help the development community understand the best practices for building systems on Microsoft platforms. He is also a virtual member of the ASP.NET MVC team, helping to design the new framework. Chris can be contacted at cct@tavaresstudios.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Loosen Up: Tame Your Software Dependencies for More Flexible Apps

Loosen Up

Tame Your Software Dependencies for More Flexible Apps

James Kovacs
This article discusses:
  • What's wrong with tightly coupled architectures
  • Testing and dependency woes
  • Dependency inversion
  • Dependency injection
This article uses the following technologies:
.NET Framework
Code download available at: DependencyInjection2008_03.exe (5408KB)
Contents
The Inner Dependency Problem
Dependency Inversion
Dependency Injection
Containers
Full-Fledged IoC Containers
Lifetime Management
Auto-Wiring Dependencies
Loosen Up for a Change

F
ew would disagree that striving for a loosely coupled design is a bad thing. Unfortunately, the software we typically design is much more tightly coupled than we intend. How can you tell whether your design is tightly coupled? You can use a static analysis tool like NDepend to analyze your dependencies, but the easiest way to get a sense of the coupling in your application is to try to instantiate one of your classes in isolation.

Pick a class from your business layer, such as an InvoiceService, and just copy its code into a new console project. Try to compile it. You'll likely be missing some dependencies, such as an Invoice, InvoiceValidator, and so forth. Copy those classes to the console project and try again. You'll likely discover other missing classes. When you are finally able to compile, you may find that you have a significant fraction of your codebase in the new project. It's like pulling on a loose thread and watching your entire sweater fall apart. Every class is directly or indirectly coupled to every other class in your design. Making changes in this type of system is difficult at best because a change in any one class can have a ripple effect throughout the rest of the system.

Now the point is not to eschew coupling completely, as that would be impossible. For instance:

string name = "James Kovacs";

I've coupled my code to the System.String class in the Microsoft® .NET Framework. Is that a bad thing? I would argue not. The likelihood of the System.String class changing in undesirable ways is very low, as is the likelihood that requirements will change in such a way that would necessitate modifying how I interact with System.String. So I am comfortable with that coupling. The point of this thought exercise is not to eliminate coupling but to be aware of it and to make sure you choose your couplings wisely.

Let's take another example of code often found in the data layer of many applications:

SqlConnection conn = new SqlConnection(connectionString);

Or even this one:

XmlDocument settings = new XmlDocument();
settings.Load("Settings.xml");

How confident are you that your data layer will only talk to SQL Server® or that you'll always load your app settings from an XML document named Settings.xml? The intent here is not to build an infinitely extensible but hugely complex and unusable generic framework. It is about reversibility. How easily can you change your mind with respect to design decisions? Do you have an application architecture that responds well to change?

Why am I worried about change? Because practically the only constant in this industry is change. Requirements change, technologies change, developers change, and the business changes. Are you putting yourself in a position to respond to those changes? By creating loosely coupled designs, software can better respond to inevitable, and many times unforeseeable, changes.

The Inner Dependency Problem

Let's examine a typical highly coupled design that you'll find in your average layered application architecture (see Figure 1). A simple layering scheme has a UI layer that talks to a service (or business) layer that talks to a repository (or data) layer. The dependencies between these layers flow the same way down. The repository layer is unaware of the service layer, which is unaware of the UI layer.

Figure 1 Typical Layered Architecture
Figure 1 Typical Layered Architecture

Sometimes you have a few more layers, such as a presentation or workflow layer, but the pattern of layers being aware only of the layer beneath them is fairly typical. Layers as coherent clusters of responsibility are a good design technique. Direct coupling of upper layers to lower layers, however, increases coupling and makes the application difficult to test.

Why am I concerned with testability? Because testability is a good barometer of coupling. If you can't easily instantiate a class in a test, you have a coupling problem. For example, the service layer is intimately familiar with and dependent on the repository layer. We cannot test the service layer in isolation of the repository layer. On a practical level, this means that most tests access the underlying database, file system, or network. This leads to a variety of problems, including slow tests and high maintenance costs.

Slow Tests If tests can execute strictly in memory, time per test can be in the millisecond range. If tests access external resources, such as a database, file system, or network, time per test is often 100 milliseconds or more. Considering that a typical project with good test coverage has hundreds or thousands of tests, this can mean the difference between running your tests in a few seconds versus minutes or hours.

Poor Error Isolation A failure in a data layer component often causes tests of upper-layer components to fail, too. Rather than having a few tests fail, which helps you quickly isolate the problem, you have hundreds of failing tests, which makes finding the problem difficult and more time-consuming.

High Maintenance Costs Most tests require some initial data. If those tests touch the database, you must ensure that your database is in a known state before each test. In addition, you must ensure that each test's initial data is independent of other tests' initial data, or you can run into test ordering problems where certain tests fail if run out of order. Maintaining the database in a known good state is time-consuming and error prone.

Additionally, if you need to change the implementation of a lower layer, you're often forced to modify the upper layers as well due to implicit or explicit dependencies that those layers have on the lower layer. Although you have layered the application, you have not achieved loose coupling.

Let's take a look at a concrete example—a service that accepts invoices (see Figure 2). For InvoiceService.Submit to be able to accept an invoice submission, it depends on an AuthorizationService, InvoiceValidator, and InvoiceRepository, which are created in the class's constructor. You cannot unit test InvoiceService without its concrete dependencies. This means that before you run your unit tests, you must ensure that the database is in a state such that you will not cause any primary or unique key violations when InvoiceRepository inserts the new invoice, nor must InvoiceValidator report any validation failures. You must also ensure that the user running the unit tests has the correct permissions so that AuthorizationService will permit the "Submit" operation.

Figure 2 Invoice Service (Close Figure 2)
public class InvoiceService {
  private readonly AuthorizationService authoriazationService;
  private readonly InvoiceValidator invoiceValidator;
  private readonly InvoiceRepository invoiceRepository;

  public InvoiceService() {
    authoriazationService = new AuthorizationService();
    invoiceValidator = new InvoiceValidator();
    invoiceRepository = new InvoiceRepository();
  }

  public ValidationResults Submit(Invoice invoice) {
    ValidationResults results;
    CheckPermissions(invoice, InvoiceAction.Submit);
    results = ValidateInvoice(invoice);
    SaveInvoice(invoice);
    return results;
  }

  private void CheckPermissions(Invoice invoice, InvoiceAction action) {
    if(authoriazationService.IsActionAllowed(invoice, action) == false) {
      throw new SecurityException(
        "Insufficient permissions to submit this invoice");
    }
  }

  private ValidationResults ValidateInvoice(Invoice invoice) {
    return invoiceValidator.Validate(invoice);
  }

  private void SaveInvoice(Invoice invoice) {
    invoiceRepository.Save(invoice);
  }
}

That is a tall order. If there are problems in any of these dependent components, either code or data errors, the InvoiceService tests will fail unexpectedly. Even if the test passes, the total execution time will be a few hundred milliseconds between setting up the correct data in the database, executing the tests, and cleaning up any data created by the test. Even if you amortize the cost of the setup and cleanup by grouping the tests into a batch and then running the scripts before and after the batch, the execution time is still a lot longer than if you could have run the tests in-memory.

Additionally, there is an even subtler problem here. Supposing that you wanted to add auditing support for the InvoiceRepository, you would be forced to create an AuditingInvoiceRepository or modify InvoiceRepository itself. Because of the coupling between InvoiceService and its child components, you don't have many options with regard to introducing new functionality into the system.

Dependency Inversion

You can decouple your higher-level component, InvoiceService, from its lower-level dependencies by interacting with your dependencies via interfaces rather than their concrete classes:

public class InvoiceService : IInvoiceService {
    private readonly IAuthorizationService authService;
    private readonly IInvoiceValidator invoiceValidator;
    private readonly IInvoiceRepository invoiceRepository;
    ...
}

This simple change to using interfaces (or an abstract base class) means that you can substitute an alternate implementation for any of the dependencies. Rather than creating an InvoiceRepository, you can create an AuditingInvoiceRepository (assuming that AuditingInvoiceRepository implements IInvoiceRepository). This also means that you can substitute fakes or mocks during testing. This design technique is known as programming to contract.

The principle I'm applying in decoupling my high-level and lower-level components is called the dependency inversion principle. As Robert C. Martin says in his article on the topic (objectmentor.com/resources/articles/dip.pdf), "High-level modules should not depend on low-level modules. Both should depend on abstractions."

In this case, both InvoiceService and InvoiceRepository now depend on the abstraction provided by IInvoiceRepository. However, I haven't completely solved the problem; I merely shifted it. Although the concrete implementations only depend on an interface, the question remains how the concrete classes "find" each other.

InvoiceService still needs concrete implementations of its dependencies. You could simply instantiate those dependencies in InvoiceService's constructor, but you wouldn't be much better off than before. If you wanted to use an AuditingInvoiceRepository, you would still have to modify InvoiceService to instantiate an AuditingInvoiceRepository. Additionally, you would need to modify every class that's dependent on IInvoiceRepository to instantiate an AuditingInvoiceRepository instead. There is no easy way to globally swap AuditingInvoiceRepository for InvoiceRepository.

One solution is to use a factory for creating IInvoiceRepository instances. This provides a central place to switch to AuditingInvoiceRepository by simply changing the factory method. Another name for this technique is service location, and the factory class responsible for managing instances is called a service locator:

public InvoiceService() {
  this.authorizationService =
    ServiceLocator.Find<IAuthorizationService>();
  this.invoiceValidator = ServiceLocator.Find<IInvoiceValidator>();
  this.invoiceRepository = ServiceLocator.Find<IInvoiceRepository>();
}

The functionality within the ServiceLocator could be based on data read from a configuration file or database, or it could be directly wired up with code. In either case, you now have centralized object creation for your dependencies.

Unit testing of isolated components can be accomplished by configuring the service locator with fake or mock objects rather than the real implementations. So for example, during testing, ServiceLocator.Find<IInvoiceRepository> could return a FakeInvoiceRepository, which assigned a known primary key to the invoice when it was saved but did not actually save the invoice to the database. You can eliminate your complex setup and teardown of the database and return known data from your fake dependencies (see the sidebar, "Is Faking Dependencies Wise?").

Service location has a few disadvantages, however. First, dependencies are hidden in the higher-level class. You cannot tell that InvoiceService depends on AuthorizationService, InvoiceValidator, or InvoiceRepository from its public signature—only by examining its code.

If you need to supply different concrete types for the same interface, you must resort to overloaded Find methods. This requires you to make decisions about whether an alternate type is required when you're implementing the factory class. For example, you can't reconfigure the ServiceLocator to substitute an AuditingInvoiceRepository at deployment time for specific IInvoiceRepository requests. But even with these disadvantages, service location is easy to understand and better than hardcoding your dependencies.

Dependency Injection
Is Faking Dependencies Wise?

You might be wondering whether or not it's dangerous to "fake" your dependencies. Won't you get false successes? Well, the reality is that you should already have tests that verify the proper functioning of your dependencies, like the real InvoiceRepository. These tests should talk to the actual database and verify that InvoiceRepository is behaving correctly.

If you know that InvoiceRepository.Save works, why do you need to test it again with every single test that depends on InvoiceRepository? You'll just slow down your higher-level tests by connecting to the database, and if there is a problem with InvoiceRepository, not only will your InvoiceRepository tests fail but so will your InvoiceService tests and any other components that depend on InvoiceRepository.

If an InvoiceService test failed, but InvoiceRepository did not, it means that you missed a test on InvoiceRepository. This type of defect is better caught by integration tests, which test the component with its concrete dependencies. These run more slowly but are run less frequently than unit tests with faked/mocked dependencies.

Now assuming that InvoiceRepository works because its unit tests pass, you have two choices. You can create and maintain complex scripts to ensure that the data in the database is correct so that InvoiceRepository returns the expected data for each InvoiceService test. Or, you can create a fake or mock implementation of InvoiceRepository that returns the expected data. This second option is much easier and works well in practice.

When unit testing a higher-level component, you want to provide fake or mock implementations for its dependencies. But rather than configuring a service locator with the fakes or mocks and then having the higher-level component look them up, you could just pass the dependencies directly to the higher-level component via a parameterized constructor. This technique is known as dependency injection. An example is shown in Figure 3.

Figure 3 Dependency Injection (Close Figure 3)
[Test]
public void CanSubmitNewInvoice() {
  Invoice invoice = new Invoice();
  ValidationResults validationResults = new ValidationResults();
  IAuthorizationService authorizationService =
    mockery.CreateMock<IAuthorizationService>();
  IInvoiceValidator invoiceValidator =
    mockery.CreateMock<IInvoiceValidator>();
  IInvoiceRepository invoiceRepository =
    mockery.CreateMock<IInvoiceRepository>();

  using(mockery.Record()) {
    Expect.Call(authorizationService.IsActionAllowed(
      invoice, InvoiceAction.Submit)).Return(true);

    Expect.Call(invoiceValidator.Validate(invoice))
      .Return(validationResults);
    invoiceRepository.Save(invoice);
  }

  using(mockery.Playback()) {
    IInvoiceService service = new InvoiceService(authorizationService,
      invoiceValidator, invoiceRepository);
    service.Submit(invoice);
  }
}

In this example, I'm creating mock objects for InvoiceService's dependencies and then passing them to the InvoiceService constructor. (For more information on mock object frameworks, see Mark Seemann's "Unit Testing: Exploring the Continuum of Test Doubles" at msdn.microsoft.com/msdnmag/issues/07/09/MockTesting.) In sum, you specify the behavior of the InvoiceService by defining how it will interact with the mocks rather than verifying the state of the InvoiceService after the test is run.

By using dependency injection, you can easily supply your higher-level components with their dependencies in unit tests. However, you still have the problem of how to locate a class's dependencies outside of a unit test—either while running the application or in an integration test. It would be foolish to expect the UI layer to provide the service layer with its dependencies or the service layer to supply the repository layer with its dependencies. You would end up with an even worse problem than what you started with. But let's suppose that the UI layer was responsible for supplying the service layer with its dependencies:

// Somewhere in UI Layer
InvoiceSubmissionPresenter presenter =
  new InvoiceSubmissionPresenter(
    new InvoiceService(
      new AuthorizationService(),
      new InvoiceValidator(),
      new InvoiceRepository()));

As you can see, the UI would have to be aware of not just its own dependencies but also the dependencies of its dependencies, ad infinitum, down to the data layer. This is obviously not an ideal situation. The easiest way to solve this dilemma is with a technique that we'll call the poor man's dependency injection.

Poor man's dependency injection uses the default constructor of the higher-level component to supply the dependencies:

public InvoiceService() :
  this(new AuthorizationService(),
    new InvoiceValidator(),
    new InvoiceRepository()) { }

Notice how I'm delegating to the most overloaded constructor. This ensures that the class's initialization logic is identical regardless of which constructor is used to create an instance. The only place the class is coupled to concrete dependencies is through the default constructor. The class remains testable because you still have the overloaded constructor that allows you to supply the class's dependencies during unit testing.

Containers

Now it's time to introduce inversion of control (IoC) containers, which provide a central place to manage dependencies. In reality, a container is nothing more than a fancy dictionary of interfaces versus implementing types. In its simplest form, an IoC container is just a service locator by a different name. Later on, I'll examine how a container can do much more than just service location.

Getting back to the problem at hand, you would like to completely decouple InvoiceService from concrete implementations of its dependencies. Like all problems in software, you can solve it by adding another layer of indirection. You introduce the notion of a dependency resolver, which maps an interface to a concrete implementation. You then use a generic method that takes an interface T and returns a type implementing that interface:

public interface IDependencyResolver {
    T Resolve<T>();
}

Let's implement the SimpleDependencyResolver, which uses a dictionary to store mapping information between interfaces and objects implementing those interfaces. We need a way to initially populate the dictionary, which is done by the Register<T>(object obj) method (see Figure 4). Note that the Register method does not need to be on the IDependencyResolver interface because only the creator of the SimpleDependencyResolver will register dependencies. Typically this is done by a helper class called in the Main method during application startup.

Figure 4 SimpleDependencyResolver (Close Figure 4)
public class SimpleDependencyResolver : IDependencyResolver
{
  private readonly Dictionary<Type, object> m_Types =
    new Dictionary<Type, object>();

  public T Resolve<T>() {
    return (T)m_Types[typeof(T)];
  }

  public void Register<T>(object obj) {
    if(obj is T == false) {
      throw new InvalidOperationException(
        string.Format("The supplied instance does not implement {0}",
        typeof(T).FullName));
    }
    m_Types.Add(typeof(T), obj);
  }
}

How does CompanyService find the SimpleDependencyResolver so that it can locate its dependencies? We could pass an IDependencyResolver into every class that needs it, but this quickly becomes cumbersome. The easiest solution is to place the configured SimpleDependencyResolver instance into a globally accessible location, which you can do using the static gateway pattern. (You could have also used the singleton pattern, but singletons are notoriously difficult to test. They are one of the biggest causes of tightly coupled code that's difficult to test, as they're little more than global variables in disguise. Avoid them if possible.)

Let's take a look at the static gateway, which I'll call IoC. (Another possible name is DependencyResolver, but IoC is shorter.) The static methods on IoC match the methods on IDependencyResolver. (Note that IoC doesn't implement IDependencyResolver because static classes cannot implement interfaces.) There is also an Initialize method that accepts the real IDependencyResolver. The IoC static gateway simply forwards all Resolve<T> requests to the configured IDependencyResolver:

public class IoC {
  private static IDependencyResolver s_Inner;

  public static void Initialize(IDependencyResolver resolver) {
    s_Inner = resolver;
  }

  public static T Resolve<T>() {
    return s_Inner.Resolve<T>();
  }
}

During application startup, you initialize IoC with the configured SimpleDependencyResolver. You can now replace poor man's dependency injection with IoC.Resolve in the default constructor:

public InvoiceService() :
  this(IoC.Resolve<IAuthorizationService>(),
  IoC.Resolve<IInvoiceValidator>(),
  IoC.Resolve<IInvoiceRepository>()) { }

Note that you do not need to synchronize access to the inner IDependencyResolver, as it is only read but never updated, after application startup.

The IoC class provides another benefit—it acts as an anticorruption layer in your application. If you want to use a different IoC container, you simply need to implement an adapter that implements IDependencyResolver. Even though IoC will be used extensively throughout your application, you have not coupled yourself to any particular container.

Full-Fledged IoC Containers

A simple IoC container such as SimpleDependencyResolver enables you to stitch together loosely coupled components. However, it lacks many of the features present in full-fledged IoC containers, including:

  • Wider configuration options, such as XML, code, or script
  • Lifetime management, such as singleton, transient, per-thread, or pooled
  • Auto-wiring dependencies
  • The ability to wire in new functionality

Let's discuss each of these features in more depth. I will use Castle Windsor, a widely used open source IoC container, as a concrete example. Many containers can be configured through an external XML file. For example, Windsor can be configured as follows:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <components>
    <component id="Foo"
      service="JamesKovacs.IoCArticle.IFoo, JamesKovacs.IoCArticle"
      type="JamesKovacs.IoCArticle.Foo, JamesKovacs.IoCArticle"/>
  </components>
</configuration>

XML configuration is advantageous, as it can be modified without recompiling the application, though it does often require an application restart for the changes to be applied. It is not without its disadvantages though, as XML configuration tends to be very verbose; errors aren't detected until runtime; and generic types are declared using the CLR's backtick notation, rather than the more familiar C# generic notation. (Company.Application.IValidatorOf<Invoice> is written as Company.Application.IValidatorOf`1[[Company.Application.Invoice, Company.Application]], Company.Application.)

In addition to XML, you can configure Windsor using C# or any other Microsoft .NET Framework-compliant language. If you isolated your configuration code in a separate assembly, changing configuration would mean simply recompiling the configuration assembly and restarting the application.

You can script Windsor configuration using Binsor, which is a domain-specific language (DSL) built specifically for configuring Windsor. Binsor allows you to write your configuration files in Boo. (Boo is a statically typed CLR language that focuses on language and compiler extensibility, thus making it well suited for writing DSLs.) In Binsor, the previous XML configuration file could be rewritten as:

import JamesKovacs.IoCArticle
Component("Foo", IFoo, Foo)

Things get more interesting when you realize that Boo is a full-fledged programming language, which means that you can use Binsor to automatically register types in Windsor without having to manually add component registrations, as you would with XML-based configuration:

import System.Reflection
serviceAssembly = Assembly.Load("JamesKovacs.IoCArticle.IoCContainer")
for type in serviceAssembly.GetTypes():
  continue if type.IsInterface or type.IsAbstract or
    type.GetInterfaces().Length == 0
  Component(type.FullName, type.GetInterfaces()[0], type)

Even if you're not familiar with Boo, the intent of the code should be clear. Simply by adding a new service to the JamesKovacs.IoCArticle.Services namespace, that service is automatically registered as the default implementation for its service interface. Let's say I create the following class:

public class AuthorizationService : IAuthorizationService {
   ...
}

If any other class declares a dependency on IAuthorizationService by including it as a parameter to its constructor, Binsor will automatically wire it up without having to specify that dependency explicitly in a configuration file! You can find more information on Binsor at ayende.com/Blog/category/451.aspx and Boo at boo.codehaus.org.

Lifetime Management

The SimpleDependencyResolver always returns the same instance that was registered for an interface, which effectively makes that instance a singleton. You could modify the SimpleDependencyResolver to register a concrete type rather than an instance. Then you could use different factories to create instances of the concrete type. A singleton factory would always return the same instance. A transient factory would always return a new instance. A per-thread factory would maintain one instance per requesting thread.

Your instancing strategy is only limited by your imagination. This is exactly what Windsor provides. By applying attributes in the XML configuration file, you can change which type of factory is used to create instances of a particular concrete type. By default, Windsor uses singleton instances. If you wanted to return a new Foo every time an IFoo was requested from the container, you would simply change the configuration to:

<component id="Foo"
  service="JamesKovacs.IoCArticle.IFoo, JamesKovacs.IoCArticle"
  type="JamesKovacs.IoCArticle.Foo, JamesKovacs.IoCArticle"
  lifestyle="transient"/>

Auto-Wiring Dependencies

Auto-wiring of dependencies means that the container can examine the dependencies of the requested type and create those dependencies without the developer having to supply a default constructor:

public InvoiceService(IAuthorizationService authorizationService,
  IInvoiceValidator invoiceValidator,
  IInvoiceRepository invoiceRepository) {
  ...
}

When a client requests an IInvoiceService from the container, the container will notice that concrete type requires concrete implementations of IAuthorizationService, IInvoiceValidator, and IInvoiceRepository. It will look up the appropriate concrete types, resolve any dependencies that they might have, and construct them. It will then use these dependencies to create the InvoiceService. Auto-wiring obviates the need to maintain default constructors, thus simplifying code and removing many classes' dependency on the IoC static gateway.

By coding to contracts rather than concrete implementations and using a container, your architecture will be much more flexible and amenable to change. How could you implement configurable audit logging for the InvoiceRepository? In a tightly coupled architecture, you would have to modify InvoiceRepository. You would also need some application configuration setting to indicate whether audit logging was turned on.

In a loosely coupled architecture, is there a better way? You could implement an AuditingInvoiceRepositoryAuditor, which implements IInvoiceRepository. The auditor only implements auditing functionality and then delegates to the real InvoiceRepository, which is supplied in its constructor. This pattern is known as a decorator (see Figure 5).

Figure 5 Using the Decorator Pattern (Close Figure 5)
public class AuditingInvoiceRepository : IInvoiceRepository {
  private readonly IInvoiceRepository invoiceRepository;
  private readonly IAuditWriter auditWriter;

  public AuditingInvoiceRepository(IInvoiceRepository invoiceRepository,
    IAuditWriter auditWriter) {
    this.invoiceRepository = invoiceRepository;
    this.auditWriter = auditWriter;
  }

  public void Save(Invoice invoice) {
    auditWriter.WriteEntry("Invoice was written by a user.");
    invoiceRepository.Save(invoice);
  }
}

To turn on auditing, you configure the container to return an InvoiceRepository decorated with an AuditingInvoiceRepository when asked for an IInvoiceRepository. Clients will be none the wiser since they are still talking to an IInvoiceRepository. This approach has a number of benefits:

  1. Since InvoiceRepository has not been modified, there is no chance that you would have broken its code.
  2. AuditingInvoiceRepository can be implemented and tested independently of InvoiceRepository. Thus you can ensure that auditing works correctly regardless of whether you have an actual database.
  3. You can compose multiple decorators for auditing, security, caching, or other purposes without increasing the complexity of InvoiceRepository. In other words, the decorator approach in a loosely coupled system scales better when adding new functionality.
  4. Containers provide an interesting application extensibility mechanism. There is no reason that AuditingInvoiceRepository had to be implemented in the same assembly as InvoiceRepository or IInvoiceRepository. It could easily have been implemented in a third-party assembly that was referenced by the configuration file.

Loosen Up for a Change

Even though your software architecture is layered, you may still have tight coupling between your layers, which can impede testing and evolution of your app. But you can decouple the design. Through the use of dependency inversion and dependency injection, you can reap the benefits of coding to contract rather than concrete implementation. By introducing an inversion of control container, you can increase the flexibility of your architecture. In the end, your loosely coupled design will be more responsive to change.

James Kovacs is an independent architect, developer, trainer, and jack-of-all trades living in Calgary, Alberta, specializing in agile development using the .NET Framework. He is a Microsoft MVP for Solutions Architecture and received his master's degree from Harvard University. James can be reached at jkovacs@post.harvard.edu or www.jameskovacs.com.


CI Server: Redefine Your Build Process with Continuous Integration

CI Server

Redefine Your Build Process with Continuous Integration

Jay Flowers
This article discusses:
  • A basic CI server
  • Launching builds and tests
  • Source control and reports
  • Setting up CI Factory
This article uses the following technologies:
MSBuild
Code download available at: ContinuousIntegration2008_03.exe (168KB)
Contents
Lowering the Cost of Adoption
Getting Started
Build and Test
Source Control
Logging and Reports
Triggering a Build
Running the CI Server
Customizing CI Factory
The Rest of the Work

W
hen using a Continuous Integration (CI) system, team members integrate application components early and often, up to several integrations a day per developer. Lest that idea set your teeth on edge, note that each integration is immediately checked and verified by an automated build to find any errors instantly. While the concept of CI might take some getting used to, it can be a valuable system.

A basic principle of a CI system is that the word build needs to be redefined. Formerly, you might have thought of a build as simply a compilation. When it comes to CI, you must understand that the definition of build now includes the all-important steps of verification and testing. CI depends on the presence of tests. Without tests, all you have is continuous compilation.

Since developers are the main users of a CI system, they will determine whether the system is a success or a failure. Developers really only want two things: a build that takes less than five minutes and an easy way to find the cause of a build failure. Success will also depend on your team's build etiquette. I recommend these simple guidelines:

  • Never leave the building if you have broken the build.
  • Don't submit to a broken build.
  • Be nice. When you see that the build is broken, look at the build report. If you know how to fix the issue, contact the developer that broke the build and offer to help.

Lowering the Cost of Adoption

My recent experience suggests that a growing number of development shops are adopting CI as a best practice, and many more would get on the wagon if the cost of adoption was lower. It can take a veteran programmer a couple of days to get a basic CI server up and running; a neophyte might take up to a week.

The frustrating thing is that 80 percent of the work performed preparing a CI server and 80 percent of the work done by the server are the same—clean up after the previous build, get the latest source code, increment versions, compile, run tests, execute static analysis, and create installers. Very little, if anything, varies between CI server instances. This commonality is the key for lowering the cost of adopting continuous integration.

In this article I will explore the work required to set up and use a CI server and various solutions that help get the work done. I will first create a CI server manually using CruiseControl.Net, MSBuild 2.0, MbUnit, and WatiN. Following that, I will show you how to use an open source project called CI Factory to do the same thing with far less effort.

The sample application I will use is a simple guestbook from the open source project NunitAsp. I will compile the application, create an Microsoft Installer (MSI) setup app, install it, and test it with WatiN. The finished guestbook application allows you to save your name and a comment, as shown in Figure 1.

Figure 1 Finished Sample App Saves Names and Comments
Figure 1 Finished Sample App Saves Names and Comments(Click the image for a smaller view)
Figure 1 Finished Sample App Saves Names and Comments
Figure 1 Finished Sample App Saves Names and Comments (Click the image for a larger view)

As you might have guessed, the app is storing the name and comment in the session state. The WatiN test for this is simple, as you can see in Figure 2.

Figure 2 Testing the Sample App (Close Figure 2)
[Test]
public void Register()
{
  Ie = new IE("about:blank");
  Ie.GoTo("http://localhost/GuestBook/GuestBook.aspx");
  Ie.TextField(Find.ByName("name")).TypeText("Jay");
  Ie.TextField(Find.ByName("comments")).TypeText("Hello");
  Ie.Button(Find.ByName("save")).Click();

  Assert.AreEqual("Jay", Ie.TableCell(Find.By("innerText", "Jay"))
    .Text, @"innerText does not match");
  Assert.AreEqual("Hello", Ie.TableCell(Find.By("innerText",
    "Hello")).Text, @"innerText does not match");
}

WatiN is a Microsoft® .NET testing framework built on Internet Explorer®. It will open a new instance of Internet Explorer and drive the test. The test will enter the name Jay and the comment Hello, click the save button, and validate that the guests table contains the expected content.

The guestbook application does not come with an installer. I created an installer by adding a Visual Studio® Web Setup Project to the solution. I then added the primary output of the guestbook project to the bin folder and the files Global.asax, GuestBook.aspx, and web.config to the Web Application Folder. With that I have an installer!

This leaves a solution containing three projects: the production application guestbook, the test project GuestBook.Test, and the Deployment Project GuestBookSetup. If only we had a CI system to build it ...

Getting Started

From a developer's perspective, I find it helpful to think of CI as task-based development. After I complete a task, I submit to the build. Let's go a little deeper here and talk about task size. Tasks are things that you can complete in a day or less. Tasks are things that, when complete, leave the system in working order. In addition, tasks are generally smaller than requirements.

There are many tasks to be performed when setting up a CI server. Let's start with the physical server that will host the CI server. First, the development environment needs to be installed on the CI server. This can include external dependencies needed for integration testing like Microsoft SQL Server® as well as controls or other runtime dependencies needed for compilation. The remainder of the development environment will contain the development directory tree. Tree Surgeon is a free tool for creating a development tree (codeplex.com/treesurgeon). The tree should have clear locations for build scripts, production source code, test source code, and third-party dependencies. With the tree defined, I can begin a build script (see Figure 3).

Figure 3 CI Build Script (Close Figure 3)
<PropertyGroup>
  <ProjectName>GuestBook</ProjectName>
  <RootDirectory>c:\Projects</RootDirectory>
  <ProjectRootDirectory>$(RootDirectory)\$(ProjectName)
  </ProjectRootDirectory>
  <ProjectCodeLineDirectory>$(ProjectRootDirectory)\Trunk
  </ProjectCodeLineDirectory>
  <BuildDirectory>$(ProjectCodeLineDirectory)\Build</BuildDirectory>
  <SourceDirectory>$(ProjectCodeLineDirectory)\Source</SourceDirectory>
  <ThirdPartyDirectory>$(ProjectCodeLineDirectory)\Third Party
  </ThirdPartyDirectory>
  <UnitTestDirectory>$(SourceDirectory)\Unit Test</UnitTestDirectory>
  <InstallDirectory>$(SourceDirectory)\Install</InstallDirectory>
</PropertyGroup>

Without getting radical, you should attempt to have no external dependencies. This increases the simplicity of your development environment. If you have no external dependencies, all that a new developer need do is copy the dev tree from the source on your server. That is the ultimate in simplicity. However, most projects include some external dependencies, so all of your external dependencies should be codified in an installation script. Check to see whether the dependency is present, and if not, install it. The script could be executing an install from a shared drive on the network or even downloading the installer from the Web.

Configuration of external dependencies should be codified in your script as well (databases, IIS, registry entries, environment variables, and so on). The key is to spend the few minutes it takes to codify your development environment. Doing this will make setting up a new environment simple when you need to set up additional CI servers.

Build and Test

At this point I've laid the groundwork for a basic build script to manage external dependencies, compile, and execute unit tests, like the following:

<Target Name="Build">
  <CallTarget Targets="SetupVirtualDirectory"/>
  <CallTarget Targets="Compile"/>
  <CallTarget Targets="RunTests"/>
</Target>

The first target, SetupVirtualDirectory (see Figure 4), will map the Web Application Project directory in IIS as a virtual directory. This is the only external dependency for this project. Also, SetupVirtualDirectory allows a developer to avoid the time-consuming step of deploying the Web application. Later, when I am executing this script on the CI server, I will actually perform a full install.

Figure 4 Targeting SetupVirtualDirectory (Close Figure 4)
<Target Name="SetupVirtualDirectory">
  <Exec
    WorkingDirectory="$(SystemRoot)\System32\inetsrv\APPCMD.EXE"
    Command='Add APP /site.name:"Default Web Site"
    /path:/GuestBook/physicalPath:$(SourceDirectory)
    \Production\GuestBook' />
  <Exec
    WorkingDirectory="$(SystemRoot)\System32\inetsrv\APPCMD.EXE"
    Command='Set APP "Default Web Site/GuestBook"
    /virtualDirectoryDefaults.physicalPath:$(SourceDirectory)
    \Production\GuestBook' />
</Target>

The compile target is simple as well. It scans the Source directory for any C# projects and compiles them:

<ItemGroup>
  <Projects Include="$(SourceDirectory)\**\*.csproj" />
</ItemGroup>

<Target Name="Compile">
  <MSBuild Projects="@(Projects)" Targets="Build"
      StopOnFirstFailure="false" ContinueOnError="false">
  </MSBuild>
</Target>

There are two C# projects—the app project and the test project:

C:\Projects\GuestBook\Trunk\Source\
  Unit Test\GuestBook.Test\GuestBook.Test.csproj
C:\Projects\GuestBook\Trunk\Source\Production\
  GuestBook\GuestBook.csproj

Note that while this approach allows for new projects to be added without requiring a change to the build script, you do lose the ability to control the order in which your projects will be built. This creates a significant problem if you have dependencies between projects. In the case where you need to control the build order, explicitly list your projects in the ItemGroup.

To run the unit tests, I will use MbUnit via its associated MSBuild task (see Figure 5). It takes a list of assemblies as an argument and executes all the tests in each. The script is set to scan the Unit Test directory for each and every file in a bin directory matching the format *.Test.dll. As long as developers follow this convention, the build scripts don't need to change for additional test projects.

Figure 5 Running Unit Tests with MbUnit and MSBuild (Close Figure 5)
<UsingTask
  TaskName="MbUnit.MSBuild.Tasks.MbUnit"
  AssemblyFile="$(ThirdPartyDirectory)\MbUnit\MbUnit.MSBuild.Tasks.dll"
  />

<ItemGroup>
  <TestAssemblies Include="$(UnitTestDirectory)\**\bin\**\*.Test.dll" />
</ItemGroup>

<Target Name="RunTests">
  <MbUnit
    Assemblies="@(TestAssemblies)"
    ReportTypes="Xml"
    ReportFileNameFormat="unittest"
    ReportOutputDirectory="$(BuildDirectory)\UnitTestReport"
  />
</Target>

This script, named GuestBook.msbuild, is located in the Build directory, and I placed a batch file, Build.bat, in the Source directory. Developers will be living out of the Source directory, so having a batch file there will make it easy for them to execute the build script:

call "%ProgramFiles%\Microsoft Visual Studio 8\VC\vcvarsall.bat"
msbuild ..\Build\GuestBook.msbuild /t:Build
SET /P variable="Hit Enter to exit."

When running the build script on the CI server, I may want to test in a more true-to-life environment. This is easily accomplished by executing an installer. The guestbook solution includes a Web Deployment Project: GuestBookSetup.vdproj. To build this project I need to call devenv.exe instead of MSBuild, since deployment projects are not currently supported by MSBuild:

<Target Name="CreateInstaller">
  <Exec
    WorkingDirectory="$(SourceDirectory)"
    Command='"C:\Program Files\Microsoft Visual
    Studio 8\Common7\IDE\devenv.exe" $(ProjectName).sln
    /Build Debug /project GuestBookSetup' />
</Target>

I will need to execute the install silently, as this will be a fully automated build. This is something that is best known up front when writing an installer. It can be troublesome to try and add this capability after the installer has been written. In the MSBuild script I need a target to install and uninstall, like so:

<Target Name="InstallApplication">
  <Exec
    WorkingDirectory="$(InstallDirectory)\GuestBookSetup\Debug"
    Command='msiexec /i GuestBookSetup.msi /q'
  />
</Target>

<Target Name="UninstallApplication">
  <Exec
    WorkingDirectory="$(InstallDirectory)\GuestBookSetup\Debug"
    Command='msiexec /x GuestBookSetup.msi /q'
  />
</Target>

These targets use the Exec task to execute the MSI file with the application msiexec.exe, found in PATH. The switch /i is for install, /x is for uninstall, and /q is for quiet. This installer package requires no user input; the default settings are acceptable for testing. However, it is possible to pass parameters to an installer package with msiexec.

The entry target for the CI server looks like this:

<Target Name="Triggered">
  <CallTarget Targets="UninstallApplication" ContinueOnError="true"/>
  <CallTarget Targets="Compile"/>
  <CallTarget Targets="CreateInstaller"/>
  <CallTarget Targets="InstallApplication"/>
  <CallTarget Targets="RunTests"/>
</Target>

I now have a build script that can perform a personal build on a developer's machine as well as a public build on a CI server. I use CruiseControl.Net, better known as CCNet, to kick off this new build script. CCNet installs to the Program Files directory and can be shared by many different projects. There are two files that are used to configure CCNet: the server's ccnet.config and the dashboard's dashboard.config. The dashboard configuration is simply for display purposes and does not play a part in creation of the product, in this case guestbook. As such, its .config file will not be kept in source control.

The server .config file is a different story and is critical to the build process. In order to manage the .config for guestbook in source control, I will create a new XML file and define the configuration using a Document Type Definition (DTD) referenced from the CCNet server's .config file:

C:\Program Files\CruiseControl.NET\server\ccnet.config
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE cruisecontrol [
  <!ENTITY GuestBook SYSTEM
    "C:\Projects\GuestBook\Trunk\Build\CCNetProject.xml">
]>
<cruisecontrol>
  &GuestBook;
</cruisecontrol>

Source Control

Let's add a new dimension and introduce source control. A CI server should be able to perform a clean build, deleting not only the compiled artifacts, but also the source files and test execution environment. It is a bad practice to perform dirty builds. It is even worse to perform a "sourdough" build. This is where you require a previous build be present in order to successfully build.

Notice in the source control block that a fresh copy of the program source is pulled for every build. Cleancopy is set to true, like so:

<sourcecontrol type="vss" autoGetSource="true" applyLabel="true">
  <project>$/GuestBook/Trunk</project>
  <username>Build</username>
  <password>password</password>
  <workingDirectory>c:\projects\GuestBook\Trunk</workingDirectory>
  <executable>C:\Program Files\Microsoft Visual
    Studio\VSS\win32\SS.EXE</executable>
  <ssdir>c:\Source Safe Databases\GuestBook</ssdir>
  <cleanCopy>true</cleanCopy>
</sourcecontrol>

Most source control systems require the same type of configuration information: location of the repository, credentials, path to the root folder of the project, and the working folder on disk. And while the sample for this article uses Visual SourceSafe® (VSS), all the source control blocks in CCNet look similar for different source control providers (Visual Studio Team Foundation Server, Subversion, Perforce, Vault, and so on).

Besides performing a clean get at the beginning of the build, it will also create a label at the end of the build, provided that the build is successful. The label is applied to the project node—Trunk in this case. Remember that Trunk contains everything needed to build this version of the product. So if you needed to come back to this version six months from now, pulling from this label will provide everything you need to successfully build the application. This is something that I have seen many teams neglect.

Many teams fail to see the CI system as part of their product. It may not be something that you deliver to the customer, yet it still plays a critical role in your ability to deliver. Treat it with the respect it deserves and include it in source control, labeling it with your product's source.

Another benefit of this setup is that it lessens the need for anyone to make changes from the build server. Changes to the MSBuild script are easily made from your workstation, checked into source control, and then pulled to the build server by CCNet. This allows you to avoid the troublesome practice of making changes on the build server and then forgetting to check those changes in. The CCNet MSBuild task compiles the solution like so:

<msbuild>
  <executable>
    C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe
  </executable>
  <workingDirectory>
    C:\Projects\GuestBook\Trunk\Source
  </workingDirectory>
  <projectFile>
    C:\Projects\GuestBook\Trunk\Build\GuestBook.msbuild
  </projectFile>
  <buildArgs>
    /noconsolelogger /p:Configuration=Debug /v:diag
  </buildArgs>
  <logger>
    C:\...\ThoughtWorks.CruiseControl.MSBuild.dll
  </logger>
</msbuild>

Logging and Reports

MSBuild does not ship with an XML logger, nor does CCNet. You can download one from the CCNet wiki. It is referenced here from the Build directory. XML logs are important to CCNet, as the dashboard UI applies XSLT against the XML output from the build to produce HTML. The MSBuild task will collect the XML and include it in the CCNet XML log file for each build. To see a nicely formatted MSBuild report on the dashboard, you will need to add a configuration entry to the dashboard.config file, located at C:\Program Files\CruiseControl.NET\webdashboard\dashboard.config. Add this node at the location /dashboard/plugins/buildPlugins, as in the code here:

<xslReportBuildPlugin description="MSBuild Output"
  actionName="MSBuildOutputBuildPlugin"
  xslFileName="xsl\msbuild.xsl" />

Look back at the build script target RunTests and notice that the unit test execution emits an XML report to the directory \Trunk\Build\UnitTestReport. This XML needs to be included so that unit test reports can be generated. CCNet can then merge this into the final build log for reporting on the Web dashboard by adding a file merge publisher block into the ccnet.config file:

<merge>
  <files>
    <file>c:\projects\GuestBook\Trunk\Build\
      UnitTestReport\unittest.xml</file>
  </files>
</merge>

The unit test reports for MbUnit can be shown in two places: the summary build report and a detailed report. The detailed report metadata should be added to the dashboard .config file at the location /dashboard/plugins/buildPlugins, like this:

<xslReportBuildPlugin
  description="Unit Details"
  actionName="UnitDetailsBuildReport"
  xslFileName="xsl\MBUnitDetails.xsl" />

Likewise, the summary report should be added at the location /dashboard/plugins/buildReportBuildPlugin/xslFileNames:

<xslFile>xsl\MBUnitSummary.xsl</xslFile>

Triggering a Build

While all the major parts of the CCNet configuration are complete, we still need to add a trigger to cause the build to occur:

<triggers>
  <intervalTrigger
    seconds="60"
    buildCondition="IfModificationExists" />
</triggers>

This trigger block will potentially cause a build to occur every 60 seconds. CCNet will ask the source control repository if there have been modifications since the last build took place; if the answer is yes, it will trigger a build.

There are many more configuration elements that can be added to the CCNet .config file. It can send e-mail to notify you of the result of a build. You can control the label or version number for each build. There are many interesting things you can do.

Running the CI Server

Now that I have finished configuring CCNet, there are just a few items left before the CI server is fully operational: I need to add everything to source control and start the CCNet server.

Depending on how you installed CCNet, you turn on the server either by starting the Windows service or by executing the ccnet.exe console application. If you have chosen the Windows service, you need to make sure the desktop has access to the service because WatiN requires it.

To view the dashboard, open http://localhost/ccnet in a browser. The main dashboard project grid shows a little information on the progress of a build. There is a download link for the CCTray tool on the left. To use CCTray on your desktop, configure it after installation so that it knows about the build server and which projects you want to subscribe to. This tool then resides in the system tray and alerts you to changes in the status and activity of the build server. It even tries to predict when the build will be complete. Figure 6 shows a successful build report for the guestbook project.

Figure 6 CCNet Report for a Successful Build
Figure 6 CCNet Report for a Successful Build(Click the image for a smaller view)
Figure 6 CCNet Report for a Successful Build
Figure 6 CCNet Report for a Successful Build (Click the image for a larger view)

The build report includes a summary page and several detailed reports. Notice the unit test summary report and links to detail reports on the left. The MSBuild Output detail report is available on the left, too. The View Build Log link displays the raw XML log file for this build instance.

Using CI Factory

Now I will demonstrate how to accomplish the same thing and more with CI Factory, but with much less effort. After downloading and installing CI Factory, open the solution file CI Factory.sln. Open the Arguments.xml file from the Solution Explorer to configure a new project.

First I set the name of the Project to GuestBook:

<property name="ProjectName" value="GuestBook" />

Next I set the names of the folders for the dev tree. Rather than use the default values, I changed these to reflect my development tree, as shown in Figure 7.

Figure 7 Setting the Folder Names for the Dev Tree (Close Figure 7)
<property name="ProjectsDirectory"
  value="c:\Projects"/>
<property name="ProjectCodeLineName"
  value="Trunk"/>
<property name="ProductDirectoryName"
  value="Source"/>
<property name="ThirdPartyDirectoryName"
  value="Third Party"/>
<property name="ProductionDirectoryName"
  value="Production"/>
<property name="UnitTestDirectoryName"
  value="Unit Test"/>
<property name="InstallDirectoryName"
  value="Install"/>

Now set the build master and developer e-mail addresses. CI Factory will configure CCNet to e-mail a build report to developers when a build to which they have contributed modifications completes. It will additionally e-mail the build owner when a build fails, as you can see from this code:

<property name="BuildMaster.Email" value="alice@contoso.com"/>
<largeproperty name="Developer.List">
  <value expand="true" xml="true">
    <user name="MyBrotherDarell"
      group="developer"
      address="bob@contoso.com"/>
    <user name="MyOtherBrotherDarell"
      group="developer"
      address="charles@contoso.com"/>
  </value>
</largeproperty>

There are other options that can be set in this file, though most of the defaults are fine for my purposes here. For example, you can collect coverage statistics on the WatiN tests by setting the following property to true:

<property name="NCover.IIS" value="test" />

Finally, I select the tools stack for the build as shown in Figure 8.

Figure 8 Configuring Build Tools (Close Figure 8)
<strings id="Packages.InstallList">
  <string value="VisualSourceSafe"/>
  <string value="CSDiff" />
  <string value="SourceModificationReport"/>
  <string value="Versioning"/>
  <string value="MSBuild" />
  <string value="VS.NETDeploy" />
  <string value="NCover" />
  <string value="DotNetUnitTest"/>
  <string value="nDepend"/>
  <string value="FxCop"/>
  <string value="Simian" />
  <string value="Analytics" />
  <string value="Alerts" />

  <string value="Deployment"/>
  <string value="Workspace"/>
</strings>

CI Factory is organized with the concept of packages as a means for extension. The VisualSourceSafe package will provide integration with source control. CSDiff will provide diff reports on the dashboard. Versioning will maintain version information for both the assemblies and files. MSBuild will manage the compilation and reporting. NCover will ... well, you get the idea.

At this point, I am finished configuring the CI Factory installer, so I execute the run batch file C:\Tools\CI Factory\run.bat to create the guestbook project. The script will install required software that is missing and prompt for acceptance to licenses.

Next, the batch file starts the CCNet server as a console application and opens a Visual Studio solution of all the build files. I have found it convenient to have a project dedicated to managing the build scripts. First, only the files that I need to edit are included in the project. It also saves time to have the assistance of the Visual Studio editor when writing XML. It is so much more difficult to create malformed XML in Visual Studio; the editor does a great job of bringing errors to your attention. You can also take advantage of what an XSD (XML Schema Definition) enables the editor to do: provide IntelliSense®! In contrast, Notepad has never brought an error to my attention.

At this point I have a plain installation of CI Factory. The tree it created is very similar to what I created by hand earlier. You may notice the server and dashboard directories in the Build directory. These contain a copy of CCNet dedicated to serving the guestbook project. They are internal dependencies here as opposed to the external and quite possibly shared dependencies in the previous example. The CCNet project configuration file is in the same location as in the previous example: C:\Projects\GuestBook\Trunk\Build\ccnetproject.xml.

The main difference I would draw your attention to is that there are two projects—one to build the guestbook product and one to manage the build scripts. The build scripts project monitors the build directory in source control, simply updating the files on disk from the repository when triggered. The project that builds the product ignores changes made to the build directory.

In my previous example, I had to write an MSBuild script, but here most of that has been taken care of during installation and is maintained in large part by the Main.Build.xml script. Like the MSBuild script I wrote, Main.Build.xml contains a target named triggered, but it does not contain a target named Build. The Build target is in a script named Personal.Build.xml and is located in the Source directory.

There are more batch files than Build.bat. For example, Test.bat will execute just the test portion of the build script. But Build.bat includes more functionality, such as updating a developer workspace from source control:

..\Build\nant\bin\nant.exe -buildfile:Personal.Build.xml UpdateSource
IF NOT %ERRORLEVEL%==0 exit /B %ERRORLEVEL%
..\Build\nant\bin\nant.exe -buildfile:Personal.Build.xml %1 %2 %3 %4 %5 %6 %7 %8
SET /P variable="Hit Enter to continue."

Customizing CI Factory

This default install of CI Factory meets a significant portion of my needs, but you can do a lot to personalize the environment through postinstall configuration and other customizations. One thing to point out when customizing CI Factory is that it is configured using NAnt, so even though we use MSBuild to control our build through CCNet, we must use NAnt's XML syntax to customize CI Factory.

The CCNet dashboard is already configured, so there is no need to add configuration information for using different XSLT files. The default install even configured IIS, creating several virtual directories. I altered the dashboard.config slightly, as shown in Figure 9, simply changing the order of the sections shown in the main build report. I prefer to have the deployment files near the top of the report, as that is the section that lists all the hyper links to download files like MSI files, executables, and ZIP files created during the build.

Figure 9 Customizing the CCNet Build Report (Close Figure 9)
<buildReportBuildPlugin imageFileName="images/BuildReport.gif">
  <xslFileNames>
    <xslFile>xsl\header.xsl</xslFile>
    <xslFile>xsl\modifications.xsl</xslFile>
    <xslFile>Packages\Deployment\deployment.xsl</xslFile>
    <xslFile>xsl\compile.xsl</xslFile>
    <xslFile>Packages\VS.NETCompile\Compile.xsl</xslFile>
    <xslFile>Packages\MSBuild\compile-msbuild.xsl</xslFile>
    <xslFile>Packages\DotNetUnitTest\MbUnitDisplaySummary.xsl</xslFile>
    <xslFile>Packages\NCover\NCoverDisplaySummary.xsl</xslFile>
    <xslFile>Packages\Simian\SimianDisplaySummary.xsl</xslFile>
    <xslFile>Packages\nDepend\nDependSummaryDisplay.xsl</xslFile>
    <xslFile>Packages\FxCop\FxCopSummaryDisplay.xsl</xslFile>
  </xslFileNames>
</buildReportBuildPlugin>

I also made a configuration change to the MSBuild package. By default, it will create a ZIP file of the primary output of all projects in the production tree. In this example, there is only one such project, guestbook, so I don't need a ZIP file; an MSI will be fine. Creation of the ZIP file is disabled by editing these two properties in the package, setting them to false:

<property name="Compile.ToCopyToBin" value="false" overwrite="false"/>
<property name="Compile.ToDeployZip" value="false" overwrite="false"/>

Just like in the MSBuild script that I wrote earlier, I need to install and uninstall the MSI file. So I edit the Main.Build.xml file in the Build directory, adding exec tasks before and after the call to run the unit tests:

<exec program="msiexec"
  commandline="/i ${Install.MsiSourceDir}\${Install.MsiSourceFile} /q"
  verbose="true"/>
<call target="UnitTest.RunTests" />
<exec program="msiexec"
  commandline="/x ${Install.MsiSourceDir}\${Install.MsiSourceFile} /q"
  verbose="true"/>

I also want to maintain the good developer experience from the previous example and ensure that developers have the guestbook Web Application Project directory mapped in IIS as a virtual directory. I do this by adding a configuration script to the Workspace package. I create a new directory named IIS in the workspace's Configuration directory and add a new file, IIS.Script.xml. The script is simple, containing only one target to create the virtual directory:

<target name="Workspace.Configuration.IIS">
  <mkiisdir
    dirpath="${ProductionDirectory}\GuestBook"
    vdirname="GuestBook"
    defaultdoc="GuestBook.aspx"
    enabledefaultdoc="true"
  />
</target>

To have this target called, I need to register it with the Workspace package. This is done by adding an include task as well as adding IIS to the list of configuration scripts:

<strings id="Workspace.Configuration">
  <string value="MbUnit"/>
  <string value="IIS"/>
</strings>

<include buildfile="Configuration\IIS\IIS.Script.xml"/>

Now that I have a working build server, I'll add the guestbook product to source control and see the build triggered.

CI Factory's main dashboard project grid (see Figure 10) shows lots of information about current build status and activity, such as the build time elapsed and the changes that have been detected. The download for CCTray is on the left, same as before, though this installer will configure CCTray for you.

Figure 10 Build Status Reported in the Dashboard
Figure 10 Build Status Reported in the Dashboard(Click the image for a smaller view)
Figure 10 Build Status Reported in the Dashboard
Figure 10 Build Status Reported in the Dashboard (Click the image for a larger view)

When the build is complete I can view the report for version 1.0.0.1, as shown in Figure 11. You will notice the link to the MSI file is included in the build report.

Figure 11 Build Report
Figure 11 Build Report(Click the image for a smaller view)
Figure 11 Build Report
Figure 11 Build Report (Click the image for a larger view)

Scrolling down you can see summary reports from some of the other tools that I chose to include in the Package list during the install: NCover, Simian, and FxCop. You can also see links to their detail reports, including NDepend. These packages require no extra configuration on my part.

The Rest of the Work

What I demonstrated in this article is 80 percent of the work that is common to all CI systems. You might be wondering, what is in the other 20 percent? Managing the database for an application is a common example. I use a source control repository to help keep my team in sync. The CI system can play the leading role in synchronizing the team's databases both for personal use and public build server testing.

Another common example is system deployment, not just to a development or test environment but to a customer environment as well. Some of the most advanced and successful CI systems are handling deployment directly into customer environments. This is today's ultimate goal—CI all the way out to the user desktop.

Some interim goals to help give you headroom are things like automating the creation of your release notes, extending the CI system to the test team's desktops (similar to how the build.bat extends into developers' desktops), and the creation of a virtual machine or even a set of virtual machines in a release build in addition to installers. The common theme here is automating what can be automated and running that automation often.

Jay Flowers is the creator of the CI Factory open source project. He contributed the CI Factory section to the book Windows Developer Power Tools. Jay has also been involved in the MbUnit project and other smaller open source projects. His enthusiasm for software development is evident in the articles and blog posts on jayflowers.com.


Performance: Find Application Bottlenecks with Visual Studio Profiler

Performance

Find Application Bottlenecks with Visual Studio Profiler

Hari Pulapaka and Boris Vidolov
This article discusses:
  • Targeting performance bottlenecks
  • Application code profiling
  • Comparison of profiling data
  • Performance reports
This article uses the following technologies:
Visual Studio 2008
Contents
Application Profiling
Optimizing the Application
Targeted Profiling

T
he past decade has brought us many new software technologies and platforms. Each of these new technologies requires special knowledge to create great-performing applications. As Internet technologies like blogs have now enabled frustrated users to easily cast a negative light on your applications, you really need to make performance a primary priority. Early in planning, you should add requirements for responsiveness and create prototypes to identify possible technology restrictions. Throughout development, you should also measure different performance aspects of the application to find possible regression, and ensure that testers file and track bugs for slow scenarios.

Even with best planning, you may still have to investigate performance problems during product development. In this article, we will show you how you can use Visual Studio® Team System Development Edition or Visual Studio Team Suite to identify performance bottlenecks in your application. We'll introduce you to the Visual Studio Profiler by walking through a sample performance investigation. Please note that while we used C# to write the code samples in this article, most of the examples here are equally valid for native C/C++ and Visual Basic® code as well.

Application Profiling

For our purpose, we'll use the profiler that comes with the two previously mentioned editions of Visual Studio. We will start with a small sample project that draws a Mandelbrot fractal as shown in Figure 1. This application is not very efficient and it takes approximately 10 seconds to draw the fractal.

Figure 1 Target Program for Performance Testing
Figure 1 Target Program for Performance Testing(Click the image for a smaller view)
Figure 1 Target Program for Performance Testing
Figure 1 Target Program for Performance Testing (Click the image for a larger view)

To start our investigation, we launch the Performance Wizard from the new Analyze menu in Visual Studio 2008. In Visual Studio 2005 this functionality is available from the Tools | Performance Tools menu. This launches a three-step wizard, where the first step lets us specify the target project or Web site. The second step provides two different profiling methods: sampling and instrumentation. (See the "Performance Profiling Explained" sidebar for details on these profiling methods.) For now, we will pick the defaults.

After the wizard completes, a Performance Explorer dialog is displayed and a new performance session is created. The session contains the target application, Mandel in our case, and no reports. To start profiling we click the Launch with Profiling button in the tool window toolbar.

After the application draws the fractal, we immediately close the form to stop profiling. Visual Studio automatically adds a newly created report to our performance session and starts analyzing it. When the analysis completes, Visual Studio Profiler displays the Performance Report Summary, which lists the most expensive functions (see Figure 2). The report shows these functions in two ways. The first measures the work performed directly or indirectly by listed functions. For each function, the numbers represent the accumulated samples collected in both the body of the function and in all of its child calls. The second list does not count the samples collected in child calls. The summary page shows that Visual Studio Profiler collected 30.71 percent of the samples during execution of the DrawMandel method. The remaining 69 percent of the samples are scattered among different functions and not shown here. To learn more about the options for reporting, see the sidebar "Report Visualization Options."

Figure 2 Performance Test Shows Expensive Function Calls
Figure 2 Performance Test Shows Expensive Function Calls(Click the image for a smaller view)
Figure 2 Performance Test Shows Expensive Function Calls
Figure 2 Performance Test Shows Expensive Function Calls (Click the image for a larger view)

Looking at the Call Tree view of the report, shown in Figure 3, the Inclusive Samples % column represents the samples collected in the function and its children. The Exclusive Samples % column represents samples collected in the body of the function only. You can see that the DrawMandel method directly calls Bitmap.SetPixel. While DrawMandel itself caused 30.71 percent of the total samples, Visual Studio Profiler collected 64.54 percent from Bitmap.SetPixel and its children. The body of Bitmap.SetPixel caused only 0.68 percent and this is why it did not show up on the summary page. Yet, Bitmap.SetPixel caused most of the processing through its children. This is the real bottleneck of the application.

Figure 3 Call Tree Samples for the Tested Application
Figure 3 Call Tree Samples for the Tested Application(Click the image for a smaller view)
Figure 3 Call Tree Samples for the Tested Application
Figure 3 Call Tree Samples for the Tested Application (Click the image for a larger view)

Obviously, Bitmap.SetPixel is not optimal for the purposes of the Mandel project. Our application needs a faster way to access all the pixels on the form. Luckily for us, the Bitmap class provides yet another useful API: Bitmap.LockBits. This function allows the program to write directly into the bitmap memory, thus reducing the overhead of setting individual pixels. Furthermore, to optimize the drawing, we will create a flat integer array and fill it with the color values of each pixel. Following that, we will copy the values of that array into the bitmap in one single operation.

Optimizing the Application
Performance Profiling Explained

When profiling using the sampling method, the profiler attaches to the running process in a way similar to the debugger. Then the profiler periodically interrupts the process and inspects which function is on the top of the stack, along with the code path that led to the function. In other words, Visual Studio Profiler collects a sample of the current process state. Sampling is a nonintrusive, statistical approach to profiling. The more samples collected in a function, the more processing the function has likely performed.

Visual Studio Profiler also collects information about the call path that led to this execution. Thus, the tool is able to display the full call stack after analyzing the collected data. By default, Visual Studio Profiler collects one sample every 10 million CPU cycles. In addition to CPU cycles, sampling can occur on many other events like page faults, system calls, CPU cache misses, and so on. The properties of the profiling session control what the profiler samples on and how often.

As a low overhead solution, sampling is often the recommended option. It is important to note, however, that sampling collects information only when the program actively uses the CPU. Thus, while your process is waiting for disk, network, or any other resource, Visual Studio Profiler does not collect samples. This is why, if your application does not actively use the CPU, we recommend using instrumentation profiling.

In instrumentation mode, Visual Studio Profiler modifies (instruments) the binaries by injecting special instructions (called probes) at the beginning and at the end of each function. The probes allow the profiler to measure how long it takes to run each function. Additionally, the profiler also adds a pair of probes around each external function call, enabling it to determine how expensive those external calls are.

With instrumentation profiling you can measure data such as exactly how long the function took to run (Elapsed Time), how many times it was called, and even how long the function was actively using the CPU (Application Time) and not switched out by the OS. The drawback of instrumentation is that it collects large amounts of data that takes longer to analyze. In addition, this profiling mode also has higher runtime overhead. The higher overhead may inadvertently change the performance characteristics of your application being profiled.

With both sampling and instrumentation, you can also collect memory allocation data for applications based on the Microsoft® .NET Framework. Users could enable and tune the collection of .NET memory allocation data using the performance session property pages. This is often referred to as memory profiling, and there is an extensive MSDN® documentation on that topic. Note that this is the only feature in the Profiler that is only available for .NET Framework-compliant code. For the rest of the features, Visual Studio Profiler has complete parity between native C/C++ and .NET-based applications.

Let's modify the DrawMandel method to use LockBits instead of SetPixel and see what kind of performance we gain with that change. After creating the bitmap, add the following lines to lock the bitmap bits and obtain a pointer to bitmap memory:

BitmapData bmpData =
    bitmap.LockBits(
        new Rectangle(0, 0, Width, Height),
        ImageLockMode.ReadWrite,
        bitmap.PixelFormat);
IntPtr ptr = bmpData.Scan0;
int pixels = bitmap.Width * bitmap.Height;
Int32[] rgbValues = new Int32[pixels];

Then, in the inner loop where we set the pixels, comment out the call to Bitmap.SetPixel and replace it with a new statement like the following:

//bitmap.SetPixel(column, row, colors[color]);
rgbValues[row * Width + column] =
    colors[color].ToArgb();

In addition, add the following lines to copy the array into the bitmap memory:

Marshal.Copy(rgbValues, 0, ptr, pixels);
bitmap.UnlockBits(bmpData);

Now, if we rerun the application under the profiler, we can see that the fractal draws nearly three times faster (see Figure 4). Note that the summary page of the new performance report shows that body of DrawMandel directly causes 83.66 percent of the total samples. Since we optimized the drawing, the bottleneck is now in the calculation of the fractal.

Figure 4 Performance Profile for the Revised Code
Figure 4 Performance Profile for the Revised Code(Click the image for a smaller view)
Figure 4 Performance Profile for the Revised Code
Figure 4 Performance Profile for the Revised Code (Click the image for a larger view)

We will now go one step further and optimize that calculation, too. Unfortunately, this time we will need to find the bottleneck in a single function. DrawMandel is a complex method making it difficult to know which calculations to focus on. Luckily, the Visual Studio 2008 sampling profiler also collects line-level data by default, which will help identify which lines within a function are most expensive.

To view line-level data, let's inspect the performance report from another point of view. From the Current View menu, switch to the Modules view. Unlike the Call Tree view, the Modules view does not show information on how functions call each other and how much those calls cost in the context of the parent function. Instead, the Modules view contains the total accumulated samples per executable (assembly or DLL) and per function in that executable. Visual Studio Profiler accumulates that data from all call stacks.

The Modules view is good for observing the bigger picture. For instance, if we sort by the Exclusive Samples % column, we can see that Mandel.exe performs 87.57 percent of the processing itself. As result of our optimizations, GDI+ is responsible for less than 3 percent. Expanding these modules, we can see the same information for individual methods. Additionally, in Visual Studio 2008 we have the ability to expand the tree beyond function level and see the same data for the individual lines or even individual instructions in those lines (see Figure 5).

Figure 5 Jump to Profiled Line of Code
Figure 5 Jump to Profiled Line of Code(Click the image for a smaller view)
Figure 5 Jump to Profiled Line of Code
Figure 5 Jump to Profiled Line of Code (Click the image for a larger view)

Jumping to the source uncovers the code shown in Figure 6. The code calculates the square root inside the inner most loop condition. This operation is expensive and is responsible for 18 percent of total application processing. The highlighted lines in Figure 6 show the code we can optimize. The first line uses an unnecessary square root, and the second is invariant for the while loop.

Figure 6 Code-Level Optimizations (Close Figure 6)

Original Code

for (int column = 1; column < Width; column++)
{
 y = yStart;
 for (int row = 1; row < Height; row++)
 {
  double x1 = 0;
  double y1 = 0;
  int color = 0;
  int dept = 0;
  while (dept < 100 && Math.Sqrt((x1 * x1) + (y1 * y1)) < 2)
  {
   dept++;
   double temp = (x1 * x1) - (y1 * y1) + x;
   y1 = 2 * x1 * y1 + y;
   x1 = temp;
   double percentFactor = dept / (100.0);
   color = ((int)(percentFactor * 255));
  }
  //Comment this line to avoid calling Bitmap.SetPixel:
  //bitmap.SetPixel(column, row, colors[color]);
  //Uncomment the block below to avoid Bitmap.SetPixel:
  rgbValues[row * Width + column] = colors[color].ToArgb();

  y += deltaY;
 }
 x += deltaX;
}

Optimized Code

for (int column = 1; column < this.Width; ++column)
{
 y = yStart;
 int index = column;
 for (int row = 1; row < Height; row++)
 {
  double x1 = 0;
  double y1 = 0;
  int dept = 0;
  double x1Sqr, y1Sqr;
  while (dept < 100 && ((x1Sqr = x1 * x1) + (y1Sqr = y1 * y1)) < 4)
  {
   dept++;
   double temp = x1Sqr - y1Sqr + x;
   y1 = 2 * x1 * y1 + y;
   x1 = temp;
  }
  rgbValues[index] = colors[((int)(dept * 2.55))].ToArgb();
  index += Width;

  y += deltaY;
 }
 x += deltaX;
}

With that fix in, let's re-profile the application and check the performance of our optimized code. After building and running the app, the fractal now redraws in 1-2 seconds. We have reduced the startup time of the application noticeably.

Visual Studio 2008 includes a new feature that allows you to compare two performance reports. To see this feature in action, we rerun the application under the profiler and capture the latest performance report. To see the difference between the two versions of the application, select the original and the latest report in the Performance Explorer. Right-click on the report and from the context menu, click the Compare Performance Reports option. The command brings up a new report showing a flat view of all functions and the difference between the respective Exclusive Samples % value for that function in both reports. Because we cut the total execution time, DrawMandel's relative percent has raised from 31.76 to 70.46.

To better view the actual optimization, in the comparison options pane we will change the column to Inclusive Samples (see Figure 7). We also increase the threshold to 1500 samples, so that we do not see any minor fluctuations. In addition, you may note that, by default, the report shows the negative data, or least optimized functions first, as it is often used for finding regressions. For the purpose of our optimization, however, we will reverse sort the Delta column, so that we can see the most optimized functions on the top. Note that DrawMandel and its children changed from 2,064 samples to 175. This is more than tenfold optimization! To show off the performance improvements made, you can copy and paste any portion of that report.

Figure 7 Comparing Optimization Results for DrawMandel
Figure 7 Comparing Optimization Results for DrawMandel(Click the image for a smaller view)
Figure 7 Comparing Optimization Results for DrawMandel
Figure 7 Comparing Optimization Results for DrawMandel (Click the image for a larger view)

Targeted Profiling
Report Visualization Options

Visual Studio can view the performance data in different ways, using various performance report options: Call Tree, Modules, Functions, and several others. Summary is the default view when the report opens. For example, to locate the call path that caused most of the processing in Visual Studio 2008 select the Call Tree view from the Current View menu. (In Visual Studio 2005, select the Call Tree tab at the bottom of the report.) The Call Tree view contains the aggregated tree of all call stacks. The Inclusive Samples % column shows the aggregated expense of each branch in those code paths. By following the most expensive branches, you can find the performance bottleneck.

In Visual Studio 2008, the profiler team added two new features to simplify the usage of performance reports. The first is the addition of noise reduction options. By default, the reports now trim insignificant small functions, thus allowing users to see the ones with bigger impact easily. This option is often referred to as trimming. In addition, the team reduced the depth of the call trees by stacking together functions that do not have any processing on their own and just call other functions to do the processing. Visual Studio Profiler refers to this as folding.

The noise reduction options in the performance report control the threshold values for trimming and folding. You can turn off noise reduction if you experience problems finding a specific function in your performance report.

The second big improvement to call trees in Visual Studio 2008 is the Hot Path button and respective context menu. Hot Path highlights the most expensive code path in the program and follows this path until it sees significant processing performed (and not delegated) by a single function. Hot Path then highlights that function. If there are two or more separate important code paths, Hot Path will stop at the point in the tree where the branching occurs. If Hot Path provides more than one branch for your application, you could choose the more interesting one and reapply Hot Path for that particular branch.

So far, we have demonstrated how to use the Visual Studio Profiler for improving the performance of an application. But many real-world apps require that you go through multiple user actions to get to the performance problem. Typically, you would prefer to disregard any data collected prior to the start of your scenario. In addition, you may want to collect data from multiple scenarios in a single run.

To demonstrate how to use the profiler for such cases, we will switch gears and profile a sample e-commerce Web site (in fact, we used a modified version of TheBeerHouse sample available from asp.net/downloads/starter-kits/the-beer-house). This Web site takes a long time to load, but since that is a one-time cost, we're not as interested in startup time as we are in why the product catalog takes a long time to load and why adding an item to the shopping cart is so slow.

For the purpose of this article, we will show you the investigation of the first scenario only. We will, however, collect data for both scenarios and show you how to filter that data to focus on the performance problems of a specific scenario.

First, we create a new profiling session. As before, launch the Performance Wizard through the Analyze menu, then choose the defaults through the pages of the wizard. Note that for Web sites, Instrumentation Profiling is the default option. Web sites are often not CPU-bound; they usually rely on database server applications to do the heavy lifting. Thus, instrumentation is a better choice.

After creating the performance session, we launch the Web site under the profiler, but we will avoid the start-up time and focus on one scenario at a time. To do this in Visual Studio 2008, under the Performance Explorer, we can use the Launch with Profiling Paused option to start the application with Visual Studio Profiler attached, but the profiler will not collect any data until the user resumes profiling (see Figure 8).

Figure 8 Pausing the Profiler at Launch
Figure 8 Pausing the Profiler at Launch

As the Web site is loading, we switch back to Visual Studio. Note that Visual Studio Profiler shows a new tool window called Data Collection Control. This window allows you to pause and resume collection multiple times. An important part of that control is the list of predefined marks. Those are bookmarks or labels that you can insert into profiling data to denote interesting points of time. We use these marks to delimit the beginning and end of each user scenario.

First, we'll rename four of those marks using the Rename Mark command in the context menu. We will also remove unused marks (see Figure 9). So far, we have kept profiling paused to avoid start up time collection and prepare our scenarios. After the Web site loads, we will resume profiling.

Figure 9 Naming Profile Marks for Test Scenarios
Figure 9 Naming Profile Marks for Test Scenarios

We are ready to start our first scenario. We'll mark the beginning of that scenario by selecting the Product Catalog Request mark and clicking the Insert Mark button. We then switch back to Internet Explorer® and complete the first scenario by showing the product catalog. After the Web site displays it, we insert the Product Catalog Rendered mark, denoting the end of this scenario. To transition to the next scenario, we select the Beer cap product. Again, we insert the respective marks before and after the addition. As this completes all of our scenarios, we then exit the application.

After analysis of the data completes, Visual Studio Profiler presents the Performance Report Summary. This report is slightly different from the sampling report as it shows the most called functions, as well as the duration of the functions taking the longest. It is important to note that this data is aggregated over the lifetime of the application and includes both of our scenarios and any preceding activities.

Obviously, we would like to have the performance report show us only the data for a given scenario and filter out the rest. In Visual Studio 2008, the profiler has a new Marks view that lists all inserted marks. (Note that Visual Studio Profiler inserts additional automatic marks at the beginning and end of program.) To create a filter for our first scenario, we select the marks denoting the beginning and end of that scenario and select Add Filter on Marks in the context menu. This automatically creates the needed filter (see Figure 10). Besides marks, we could also filter by Thread, Process, or time interval. As we have our filter setup, we can go ahead and execute it.

Figure 10 Target Program for Performance Testing
Figure 10 Target Program for Performance Testing(Click the image for a smaller view)
Figure 10 Target Program for Performance Testing
Figure 10 Target Program for Performance Testing (Click the image for a larger view)

Note that this filtering applies to all views in the performance report. This is why Visual Studio Profiler automatically shows up the new summary page for the filtered data. This summary page is specific to the product catalog rendering scenario. For instance, we could see that System.IDisposable.Dispose is taking 3.4 seconds or 61 percent of the scenario execution time, while before it was responsible for 41 percent. Through filtering, we were able to see exactly how important this function is for our particular problem.

Now we'll fix this performance issue. We need to find the function in our code that caused those object disposals. The easiest way is to use the Call Tree with the Hot Path feature, as shown in Figure 11. This immediately shows that the SetInputControlsHighlight function is causing most of the calls to IDisposable.Dispose.

Figure 11 Finding the Problem Using Hot Path
Figure 11 Finding the Problem Using Hot Path(Click the image for a smaller view)
Figure 11 Finding the Problem Using Hot Path
Figure 11 Finding the Problem Using Hot Path (Click the image for a larger view)

It turns out that the function contains an inefficient logging mechanism:

foreach (Control ctl in container.Controls) {
  log += "Setting up control: " + ctl.ClientID;
  string tempDir =
    Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
  using (StreamWriter sw = new StreamWriter(
    Path.Combine(tempDir, "WebSite.log"), true)) {
    sw.WriteLine(log);
  }
 ...

This is extensive logging left by mistake during debugging and does not serve any specific diagnostic purpose anymore so it is safe to remove it. Again, the Hot Path feature in Visual Studio 2008 allowed us to quickly identify the bottleneck in our app.

Whether you write your applications in native C/C++, C#, or Visual Basic, Visual Studio Profiler significantly simplifies performance investigations and helps you write faster and more efficient applications. Visual Studio 2008 brings additional improvements to Visual Studio Profiler, making it easier than ever to find the performance bottlenecks in your programs.

Hari Pulapaka is a Software Design Engineer in Test who works on the Visual Studio Profiler team at Microsoft. You can reach Hari at haripul@microsoft.com.

Boris Vidolov is a Program Manager at Microsoft. In his more than 10 years in the software industry, he has worked on many performance-critical applications. You can learn more about Boris at his personal blog, perfcop.spaces.live.com.


Office Development: OBA Solution Patterns In The Real World

Office Development

OBA Solution Patterns In The Real World

Steve Fox
This article discusses:
  • OBA basics
  • OBA patterns
  • Microsoft Office and LOB integration
  • Real-world applications of OBA patterns
This article uses the following technologies:
Visual Studio 2008
Contents
OBA Patterns
Using the OBA Solution Patterns
The Problem
OBA Applications as a Reach Channel
Getting Down to Design
Other Uses

L
arge-scale systems such as SAP and PeopleSoft, and other comprehensive line of business (LOB) systems, are critical for the successful management of all types of business data and processes. However, not everyone in an organization has access to these systems, so the business data in them is often available to only a select few. This often results in data being extracted out of the system for processing, creating a disconnect between the business data source and the information workers who consume that data.

Office Business Applications (OBAs) address this problem by using Microsoft® Office to bridge the divide between business data in LOB systems and the information worker. Office features allow you to perform tasks such as integrating customer relationship management (CRM) data into Microsoft Outlook® through custom form regions and folders, integrating business intelligence into Microsoft Office SharePoint® Server (MOSS) to provide views into Sales performance data, and even integrating Microsoft Excel® with finance data to provide forecast templates that use data directly from LOB systems, to name a few.

To understand an OBA, you can think of it as a very simple model that includes three major parts: the LOB system, the customized Office client that integrates with the LOB system, and the server component (essentially MOSS) that also integrates with the LOB system. Note that when building OBAs you can also leverage other Microsoft server products such as Exchange Server 2007, PerformancePointTM Server 2007, and so on.

From an architectural perspective, you'll predominantly use a service-oriented architecture (SOA) to integrate with LOB systems, but with an OBA, you can customize the client in a number of ways as the interface to that services layer. More than likely, you'll use Visual Studio® Tools for Office (VSTO) to do this. For example, Figure 1 illustrates an interface with a custom ribbon and a custom task pane. Both interface with a services layer that populates the Excel 2007 spreadsheet with event information from SAP, and both are built using VSTO 3.0—the latest release of VSTO included in Visual Studio 2008 Professional and above. You can also design your interface to use Microsoft Word content controls and bind those controls to data, or extend Outlook with a custom toolbar, form regions, or custom ribbons. For more information on how you can customize the client, check out the "OBA Information Resources" sidebar.

Figure 1 Integrating SAP and Excel 2007
Figure 1 Integrating SAP and Excel 2007(Click the image for a smaller view)
Figure 1 Integrating SAP and Excel 2007
Figure 1 Integrating SAP and Excel 2007 (Click the image for a larger view)

With MOSS 2007, you have a number of options that are open to you. For example, you can build a site that hosts different types of Web Parts. Popular Web Parts for OBAs include Business Data Catalog (BDC) Web Parts, (which manage enterprise-wide entities and their related services or ADO.NET connections), Excel Services Web Parts (which link spreadsheet data to a Web Part), and Key Performance Indicator (KPI) Web Parts (which provide performance metrics). There are of course many others that you can use in the creation of your Office Business Applications. Figure 2 illustrates a BDC Web Part and an Excel Services Web Part in a SharePoint site.

Figure 2 BDC and Excel Services Web Parts
Figure 2 BDC and Excel Services Web Parts(Click the image for a smaller view)
Figure 2 BDC and Excel Services Web Parts
Figure 2 BDC and Excel Services Web Parts (Click the image for a larger view)

While the possibilities seem endless, there is help out there to get you started. This help is provided in a core set of seven OBA solution patterns. Next I'll introduce them to you and then actually apply one to the design and development of an OBA.

OBA Patterns

The goal of OBA patterns is to show how you can architect and develop an OBA, show what technologies you would use within a particular pattern, and illustrate how you might go about developing your OBA. Some of the technologies involved include OpenXML, MOSS 2007, Web services, and VSTO 3.0.

Of the seven core patterns, some have sub-patterns. And because OBAs are composite applications that leverage multiple parts of the Office platform, you'll find that there are relationships that span patterns. The seven patterns, along with the goals of each, are listed in Figure 3.

Figure 3 OBA Patterns (Close Figure 3)

Pattern | Description
OBA Applications as a Reach Channel | Shows how to extend LOB application functionality to a broader user base.
Document Integration | Provides guidance on how to generate Office documents from LOB applications and embed LOB data in Office documents.
Composite User Interface | Describes the composition of multiple application user interfaces in an Office document or SharePoint Web page.
Complementary Document Workflow | Shows how to control and monitor document-centric processes through the use of workflow.
Discovery Navigation | Provides a more natural way of interacting with LOB system business data to show how you can discover data by searching across multiple LOB applications.
Collaborative Site | Provides guidance on how to augment structured business processes with unstructured human collaboration.
Application-Generated Tasks and Notifications | Shows how to use Outlook as a primary user interface to receive and act on LOB application-generated tasks and alerts.

Each one of the patterns represents a single pattern or is composed of sub-patterns that provide more detailed guidance. For example, OBA Application as a Reach Channel is composed of two sub-patterns (Direct Integration and Mediated Integration), which provide specific guidance around an SOA versus a non-SOA solution. I'll discuss the OBA Application as a Reach Channel pattern in more detail later.

What's also important about the patterns is that they provide some guidance on the technologies within the Office platform that you would use to develop your OBA. So, for example, the Document Integration pattern and its sub-patterns suggest OpenXML, VSTO, the BDC, or, more generally, document markup. Figure 4 provides an overview of the sub-patterns, if any, that make up the seven core OBA solution patterns, along with the technologies that you would need when using that particular OBA pattern to design your OBA solution.

Figure 4 OBA Solution Patterns, Sub-Patterns, and Associated Technologies (Close Figure 4)

Patterns | Sub-Patterns | Technologies Used
OBA Applications as a Reach Channel | Direct Integration—custom components that deliver business data in a custom manner in the composite UI without using services. Mediated Integration—middle layer, abstracts the communication between both ends; more loosely coupled, declarative, and service-oriented. | VSTO MOSS 2007 BDC
Document Integration | Application-Generated Document—generated from LOB system (using server-side batch process). Can also use Direct or Mediated Integration. Intelligent Documents—LOB system data embedded directly in document (content controls, named ranges). | OpenXML VSTO BDC Document Markup
Composite User Interface | Context-Driven Composite User Interface—custom UI components tied to events and application context (specific data or user profiles). Mesh Composite View—connectable Web Parts to create master/detail composite user interfaces (BI views in SharePoint). RSS and Web Services Composition—mesh composite view for RSS and Web services. Analytics—mesh composite view for data analysis dashboard. | Web Parts VSTO (Ribbon, Custom Task Panes, and so on) BDC Excel Services and Dashboards
Complementary Document Workflow | LOB-Initiated Document Workflow—documents mapped to workflow are transformed, enhanced, or populated with business data. Additional workflow functionality is embedded in the documents. Cooperating Document Workflow—workflow is part of the document's container, so many people are involved and need to collaborate in a structured way with the document. | Windows Workflow Foundation SharePoint Storage BDC
Discovery Navigation | N/A | Enterprise Search BDC
Collaborative Site | N/A | Windows SharePoint Services InfoPath® Forms Services
Application-Generated Tasks and Notifications | Simple Task and Notification Delivery—one-way flow of a task and assignment to someone through e-mail notification. Task Synchronization—two-way synchronization of tasks in Outlook. Intelligent Tasks and Notifications—describes how to take actions based on the assigned task. Form-Based Tasks and Notifications—describes how to integrate InfoPath forms to provide richer validation and automation of business rules. | Outlook 2007 and MOSS Integration VSTO (Custom Task Panes, Outlook Form Regions, and so on) InfoPath Forms Services

You can use the patterns as a part of your consumption of other OBA content to get a feel for what constitutes an OBA and the technologies you would use to build one, or use the patterns in the design of your own OBA. There also are a number of Reference Architecture Packs available. These are OBA architecture (and application) guidance packs for specific industries—for example manufacturing or banking. For more information, see the "OBA Information Resources" sidebar.

Now that we've covered some of the OBA pattern essentials, let's apply the OBA solution patterns to real-world software design.

Using the OBA Solution Patterns

To begin, let's first look at where in a software development lifecycle (SDLC) process you would actually consult with the OBA solution patterns. I begin by identifying a problem. If you don't have a problem (or a need), I'd question why you're developing the software in the first place. Once you identify the problem, you're next going to identify the requirements that map back to that problem. These two steps are the most important part of the process. If you don't get them right, problems will cascade throughout the SDLC. Once you've nailed down the requirements, you can begin to design.

In the design phase, you're going to ask yourself a number of questions. For example: do the requirements specify a particular environment (thus implying design constraints)? Is certain software (either off the shelf or custom) required to make this work? What hardware is needed? Figure 5 illustrates how you might proceed in the design phase to incorporate the OBA patterns.

Figure 5 OBA Patterns in the Design Phase
Figure 5 OBA Patterns in the Design Phase(Click the image for a smaller view)
Figure 5 OBA Patterns in the Design Phase
Figure 5 OBA Patterns in the Design Phase (Click the image for a larger view)

With a set of requirements in hand and an understanding of an OBA, you'd next take a look at the OBA patterns to determine if one pattern fits with your requirements and, if so, which one. Further, you would want to dig into a specific pattern to see if a sub-pattern maps to your requirements. Assuming you do find a mapping between one of the patterns and your requirements, you could use the pattern as a template for design.

The Problem

As I've discussed, software development typically starts out with some problem. For our example, the problem is that a sales management team needs to view up-to-the-minute sales data. To tackle this problem, we're going to need to know a little bit more about the team, the environment, and some baseline requirements.

Here's what we know about this particular environment: the sales management team has 100 members, all of whom are all running Windows Vista®, the 2007 Office system, SharePoint, and a recent version of Internet Explorer®. The members are mostly remote, but they all can access a central SharePoint site or server over a corporate network.

The requirements, which are simplified for illustrative purposes, are as follows: because they are not consistently connected to the corporate network, the sales team records their sales data in an Excel spreadsheet throughout the day, and then manually reenters the data into an LOB system when connected to the corporate network. Additionally, there are specific steps in this process such as data validation which are also performed manually. The team wants a system that can support the intermittently connected sales team and can automate many of the manual processes, such as validation. (The sales data entry and validation process takes an average of 30 minutes each day. The team wants to reduce this time to 5 minutes.) Because the sales team is already comfortable using tools like Excel and Internet Explorer, the management team wants an Excel-based solution and a SharePoint view so that the members can build reports on their team management site.

Looking at the problem definition, you can quickly see that what the sales management needs is a way to extend their LOB applications used for managing sales data directly to their remote sales force. Comparing this definition to the list of OBA solution patterns, we find a good match in the OBA Applications as a Reach Channel pattern.

OBA Applications as a Reach Channel

The goal of this pattern is to show you how you can extend LOB system functionality to a broader user base using an Office application as the vehicle. The pattern implementation will give more people access to the LOB system's business data. Following this pattern, you can integrate with the LOB system using Outlook or Excel as a rich client, for example, or a SharePoint Web Part as a server-side thin client. In this case, the pattern is appropriate because you're rolling out an OBA to a small organization, so Excel and SharePoint will provide two good ways of interfacing with the sales data. In fact, ideally you'd like to have a solution that provides both client-side and thin-client access to the LOB system. As such, our Sales Data OBA looks like what's shown in Figure 6.

Figure 6 Client-Side Access and Thin-Client Access to the LOB System
Figure 6 Client-Side Access and Thin-Client Access to the LOB System(Click the image for a smaller view)
Figure 6 Client-Side Access and Thin-Client Access to the LOB System
Figure 6 Client-Side Access and Thin-Client Access to the LOB System (Click the image for a larger view)
Figure 6a Client-Side Access and Thin-Client Access to the LOB System
Figure 6a Client-Side Access and Thin-Client Access to the LOB System(Click the image for a smaller view)
Figure 6a Client-Side Access and Thin-Client Access to the LOB System
Figure 6a Client-Side Access and Thin-Client Access to the LOB System (Click the image for a larger view)
Figure 6b Client-Side Access and Thin-Client Access to the LOB System
Figure 6b Client-Side Access and Thin-Client Access to the LOB System(Click the image for a smaller view)
Figure 6b Client-Side Access and Thin-Client Access to the LOB System
Figure 6b Client-Side Access and Thin-Client Access to the LOB System (Click the image for a larger view)

Through a closer review you notice that there are two sub-patterns to the Applications as a Reach Channel: Direct Integration and Mediated Integration. So let's take a closer look at each of these.

With Direct Integration, access to the underlying LOB system is integrated directly into an Office client or SharePoint Web Part. Using this pattern, you would typically build connections directly into the back-end process or data source and should have minimal additional business logic.

While Direct Integration is common, it is difficult to reuse across multiple systems. Mediated Integration requires creating a services layer that mediates integration between the LOB system and the custom client components (for example, the custom task pane), thus facilitating reuse across multiple systems and loose coupling between the client and server. For the Sales Data OBA, this is good news. We can now create a Web service that wraps around the sales data module (where all of the sales data is entered) and integrate that Web service with all client interfaces. So if you want to integrate the LOB system with Excel, you can. You can then use this service through a reference you create within Visual Studio 2008 or through the BDC within SharePoint.

Now you're getting somewhere: you've nailed down the problem statement and requirements and are ready to apply an OBA pattern. The resulting architecture leverages the Mediated Integration pattern; that is, an OBA that leverages Web services to provide sales functionality in Excel and SharePoint. Notice that things are slightly different in Figure 7 (as compared to Figure 6). That's because the BDC only operates as a read-only vehicle for data visualization, whereas you can issue read/write calls from the client-side customization. This must be reflected in the architectural diagram.

Figure 7 Sales Data OBA Architecture
Figure 7 Sales Data OBA Architecture(Click the image for a smaller view)
Figure 7 Sales Data OBA Architecture
Figure 7 Sales Data OBA Architecture (Click the image for a larger view)

Now the architecture includes an Excel 2007 add-in, a MOSS 2007 site, and a Web service which acts as a mediated connection back to the LOB system. Further, you now have an understanding of the technologies involved, so you can write the more detailed specification and build the development environment (using the technologies involved in the OBA solution pattern).

You might assume that at this point you can put the OBA solution patterns aside. However, you might want to compare the pattern you chose against other possible patterns within the core seven solution patterns. This will validate your architecture and might spark some ideas for a version 2.0 of your solution that might include more of the servers, tools, and services of the Office development platform.

One final step is to consider what it would take to build each of the components in the architecture. I'll provide a high-level overview along with some additional links where you can drill down on each of the technologies I've discussed.

Getting Down to Design
OBA Information Resources

OBA development is a fairly broad area, so I've compiled a few recommended readings:
6 Microsoft Office Business Applications for Office SharePoint Server 2007
microsoft.com/mspress/books/9471.aspx
Programming Microsoft Office Business Applications
microsoft.com/mspress/books/12194.aspx
Getting Started with Office Business Applications
msdn2.microsoft.com/bb614541
OBA Central
obacentral.com
OBA Reference Architecture Packs
msdn2.microsoft.com/bb265266
VSTO Developer Portal
msdn2.microsoft.com/aa905533
Office Developer Portal
msdn2.microsoft.com/office
SharePoint Developer Portal
msdn2.microsoft.com/sharepoint
Creating a Database Connection by Using the Business Data Catalog Definition Editor
msdn2.microsoft.com/bb736296
Creating a Web Service Connection by Using the Business Data Catalog Definition Editor
msdn2.microsoft.com/bb737887
Integrating LOB Systems with the Microsoft Office System
msdn2.microsoft.com/bb896607

With your architecture in hand, you'll want to understand how you go about building this OBA. You know you'll need to have a MOSS site setup (or have admin privileges on the existing site so you can develop against it). If you choose to set up a new site, you'll need to install MOSS (and all of the prerequisites) on Windows Server® 2003 (or Windows Server 2008). You'll also need to make sure you have Visual Studio 2008 Professional Edition installed, along with the 2007 Office system Professional Edition and the Office Primary Interop Assemblies (PIAs)—which is a setup option in Visual Studio 2008. You'll also need access to your LOB system because you'll need to develop and test a Web service against an instance of it.

It will take at least a day to set up this environment. I would proceed by setting up the server operating system, setting up MOSS on the server, installing Office on the server, and then installing Visual Studio 2008 (and the PIAs) on the local machine. At this point, you'll be able to test access to any services that you create (or that exist already).

Once your environment is set up, you'll need to create the Web service. For the Sales Data OBA, you will likely want to create at least two Web services: one that gets sales data (which is displayed in the Excel client add-in and the MOSS site) and one that updates the data from the Excel client add-in. While the LOB system is generic, you could use the Web service tools to create a service that wraps more domain-specific functionality around, for example, an existing business module—using SAP speak.

Different LOB systems will obviously work in different ways, but the goal should be to end up with a service that you can consume within the BDC and within your VSTO add-in. To get help here from a related area (that is SAP and Office integration), you can review the OBA Starter Kit for SAP, which provides help on how to create Web services for SAP (msdn2.microsoft.com/bb498189).

Now that you've created the Web service, you can create the VSTO add-in. To do so, use Visual Studio 2008 to create either a VSTO application-level add-in or document-level solution. Each will consume a Web service in a similar way. You'll likely create a custom task pane in your add-in to manage the sales data (a filter to control which data is shown) and a custom ribbon to manage views on that data (to create a chart, exercise functions against the data, and so forth). There is a lot of great information on how to do this on the VSTO Developer Center at msdn2.microsoft.com/aa905533.

The next step would be to add the Web service to the MOSS site. This would mean creating an application definition file (ADF) for the Web service (which means either hand coding an XML file or using a tool like the BDC Definition Editor or BDC MetaMan to create it), importing the ADF into SharePoint, creating a BDC Web Part, and then tying the Web Part to the ADF to complete the integration between the BDC Web Part and the LOB system. For more information on how to do this, see msdn2.microsoft.com/bb737887.

To deploy this solution means having both a client-side deployment mechanism and the MOSS site deployed where the sales team has access to the BDC Web Parts and Web services. You can publish the VSTO client-side assembly to the SharePoint site, and the sales management team can then download the assemblies to their local clients, which will check for updates on the server via a deployment manifest each time the add-in is invoked.

Deploying the BDC Web Part requires that you have an existing MOSS site set up and configured for use by the team because the deployment process for the Web Part involves creating a Web Part on the existing site. For more information on deployment, see both the VSTO Developer Center (msdn2.microsoft.com/aa905533) and the SharePoint Developer Center (msdn2.microsoft.com/sharepoint).

Other Uses

While I walked you through how I might apply OBA patterns, you might want to apply them in a different way to suit your own purposes. For example, you might use them solely to understand existing and suggested patterns as guidance for your architectural design and never need them again, or you might use them in your design as an architectural template for your OBA solution, or you could use them repeatedly as a way to provide structure to your OBA solutions—and subsequently make them a prerequisite for your specification process. As we move forward, you'll likely see additional guidance on OBA development, which will probably include, among other things, enhancements to the existing patterns. Be on the lookout for it!

Steve Fox is a Program Manager with the Visual Studio Tools for Office (VSTO) team at Microsoft. He divides his time working externally with VSTO customers, providing VSTO training, delivering VSTO sessions at conferences, and working internally with the development team to help evolve the VSTO product.


Editor's Note: Too Many Swimming Pools

Editor's Note
Too Many Swimming Pools
Howard Dierking

I have always loved design. From my very first design project in the fourth grade when I designed my dream house, I have always derived a great deal of satisfaction from being able to put all of my crazy ideas out there to visualize and then to explore the alternatives and new possibilities they present. There's nothing quite like that feeling when all of the pieces just fit together in that perfect way.

I have spent the majority of my career seeking out that perfect design in software systems. I've studied the patterns, I've read the UML specification, I've embraced practices such as Test-Driven Development (TDD) and Domain-Driven Design (DDD), and I've built some systems that I remain very proud of to this day.

I've also built some systems that I've had to rewrite almost immediately because I was so focused on applying the latest and greatest design pattern that I completely missed the business problem that I had been asked to solve.

So what have I learned over the years in the area of design? Jumping back to my fourth grade dream house project for a moment, I should mention that because I wasn't required to actually build the house, my design became somewhat disconnected from reality. By that, I mean to say that I designed a 14-story house with four swimming pools, a couple of bowling alleys and a few gymnasiums (come to think of it, I don't remember adding any bedrooms) that was 400 feet underground with a rollercoaster connecting the house to the earth's surface.

When I look back at those systems I designed that were complete failures, two patterns (or anti-patterns) emerge. The first is that much of the initial design work was done in a vacuum using a modeling tool rather than a programming language. The second is that many of the designs were based around being able to use a pattern rather than letting the patterns emerge. As such, the resulting applications came out looking much like my dream house—they looked great on paper but were nearly impossible to implement. In other words, they had too many swimming pools and not enough living space.

So what does this mean for you as a reader of MSDN Magazine? After all, one of the key pieces of information that we learned from our latest reader survey is that you want to see more content on software design. I'm happy to let you know that we are going to be bringing you more of this type of content. As you can see, we've started in this issue by covering topics like dependency injection and continuous integration. Additionally, we are currently in the process of planning a new column specifically around software design best practices. That said, we will maintain a commitment to staying grounded in reality, so that you can apply these techniques in a way that will keep your projects from looking like my fourth grade dream house.

And, as always, we will continue to maintain a balance of content across all of the other programming topics, from service and Web 2.0 development to concurrency and security. This brings me to a few announcements. Keith Brown recently decided to stop writing the Security Briefs column in order to pursue other interests. While this is obviously a great loss to us, I am happy to let you know that Michael Howard is coming on board to continue the column. We are very excited about the wealth of experience that Michael brings to the column, and we anticipate great things going forward.

Finally, starting with next month's issue we are launching an entirely new column focused on all things mobile. With the seemingly constant barrage of new and more sophisticated devices, we anticipate an even greater need for systems that can run anywhere and on any device—and the goal of this column is to have a more regular conversation around how to meet that need as a community of developers.

Thanks to the following Microsoft technical experts for their help with this issue: Glenn Block, James Brundage, Ben Carter, John Cunningham, Elisa Flasko, Steve Fox, Scott Hanselman, Habib Heydarian, Lee Holmes, Ade Miller, Larry Olson, Polita Paulus, Michael Puleio, Karthik Ravindran, Stephen Toub, Eric White, and Jon White.


Toolbox: Easy Web Debugging, Extending Reflector, and More

Toolbox
Easy Web Debugging, Extending Reflector, and More
Scott Mitchell

Contents
Client-Side Web Development
Blogs of Note
Extend Lutz Roeder's Reflector
The Bookshelf

Client-Side Web Development

While the design-time experience of Visual Studio® truly simplifies the task of writing HTML, CSS, and JavaScript for a Web application, it's not perfect. The designer paints a fairly accurate picture of how a page will look in a browser, but it's not always exact and, of course, it can't support scenarios where HTML or CSS is dynamically added. Furthermore, the designer doesn't execute client-side script. Consequently, testing a Web page's appearance and client-side script involves quite a bit of switching back and forth between the designer and the browser.

Fortunately there are tools available to inspect and modify the markup, CSS, and script of a rendered Web page directly in the browser. One such tool is Joe Hewitt's Firebug 1.05. Firebug is a free, open-source add-on for the Mozilla Firefox browser that displays information about a page's HTML, CSS, JavaScript, images, and Flash content in a tabbed interface.

With Firebug, there's no more need for View/Source, as the page's HTML elements are displayed in a searchable tree view. Mousing over a node in the tree highlights the corresponding region in the browser, and selecting a node displays its CSS styles as well as its Document Object Model (DOM) attributes. Firebug also lists the page's CSS definitions and JavaScript.

In addition, the HTML and CSS can be edited in Firebug, with the effects immediately reflected in the browser. With the click of a mouse, you can add or remove HTML elements, modify attributes, and create, update, or disable CSS definitions. These are in-memory modifications within the browser—you still need to return to Visual Studio to more permanently apply the changes—but Firebug provides immediate feedback and makes testing and debugging layout and CSS changes much faster.

Firebug is also useful for finding and fixing script errors. A dropdown list enumerates the external JavaScript files used by the page, and selecting an item displays its script in a powerful JavaScript debugger. The debugger supports breakpoints, includes a watch window, and allows you to step through the executing JavaScript. In short, it's a must-have tool that belongs in every Web developer's toolbox.

Price: Free, open-source.

getfirebug.com

Firebug Displays HTML, CSS, JavaScript, Images, and Flash Information
Firebug Displays HTML, CSS, JavaScript, Images, and Flash Information (Click the image for a smaller view)
Firebug Displays HTML, CSS, JavaScript, Images, and Flash Information
Firebug Displays HTML, CSS, JavaScript, Images, and Flash Information (Click the image for a larger view)

Blogs of Note

One of my favorite bloggers is Scott Guthrie, General Manager in the Microsoft Developer Division. What I like best about his blog is his in-depth, multi-part tutorial series. Scott has written detailed tutorials on data-driven Web sites, the ASP.NET Modal View Controller (MVC) Framework, and LINQ to SQL. Each tutorial includes step-by-step instructions and an abundance of screenshots, making it easy for developers of all skill levels to follow along. And there's no shortage of critical details; all the information you need is right there.

Scott also blogs on other forward-looking topics such as product roadmaps. He offers tips, tricks, and gotchas for Visual Studio, the Microsoft® ASP.NET AJAX Framework, working with data, security, performance, LINQ, Silverlight®, and other technologies. And once a month he shares his favorite blogs, articles, and books.

Another great quality of Scott's blog is his responsiveness to reader comments. If you are a developer using any of the products Scott helps create, this is one blog you need to subscribe to.

Price: Free.

Weblogs.asp.net/scottgu

Scott Guthrie Announces .NET Code Release
Scott Guthrie Announces .NET Code Release (Click the image for a smaller view)
Scott Guthrie Announces .NET Code Release
Scott Guthrie Announces .NET Code Release (Click the image for a larger view)

Extend Lutz Roeder's Reflector

Reflector for .NET version 5.0 (aisto.com/roeder/dotnet) is a free class browser, analyzer, and decompiler for the Microsoft .NET Framework that has been around since the early days of the .NET Framework 1.0. Reflector displays the types, methods, properties, and events for any managed assembly. In addition to a simple class browser interface, Reflector can analyze how a particular type, method, or property is called by other methods and properties, and how it depends on other methods and properties. Moreover, Reflector can decompile a method or property into corresponding C# or Visual Basic® code.

Reflector is a powerful tool for inspecting and analyzing managed assemblies. As a decompiler, it can be used to examine and retrieve the source code of a .NET assembly whose code has been lost. Its analyzer can quickly highlight dependencies among methods and properties. Best of all, Reflector was designed to support add-ins, and a number of developers in the .NET community have taken the time to create and share 30 powerful Reflector add-ins at codeplex.com/reflectoraddins.

The Code Metrics add-in computes a number of code quality metrics on the selected assemblies, providing an overview of the largest, most used, and most complex types, methods, and properties, among other information. The Graph Add-In visually represents assembly dependencies using a dependency graph and can illustrate type hierarchies using class diagrams. And the Diff Add-In is useful for comparing differences between two versions of an assembly.

Other useful Add-Ins include FileDisassembler, which saves an entire assembly's decompiled source code to .vb or .cs files; SQL2005Browser, for browsing the .NET assemblies stored in a SQL Server® 2005 database; and CodeSearch, which enhances Reflector's search capabilities to search for strings and regular expressions in the decompiled code.

Price: Free.

Reflector: aisto.com/roeder/dotnet

Reflector Add-Ins: codeplex.com/reflectoraddins

Code Metrics Add-Ins in Reflector
Code Metrics Add-Ins in Reflector (Click the image for a smaller view)
Code Metrics Add-Ins in Reflector
Code Metrics Add-Ins in Reflector (Click the image for a larger view)

The Bookshelf

Most books that teach SQL do so with dry prose and focus on business and accounting scenarios. O'Reilly's Head First SQL by Lynn Beighley turns this approach on its head, and to great effect. The topics covered in Head First SQL are familiar: the fundamentals of the SELECT, INSERT, UPDATE, and DELETE queries; JOINs; subqueries; data normalization; data and relational integrity; and so on. But the unique presentation in Head First SQL makes it fun to learn and easy to remember.

I'm a big fan of O'Reilly's entire Head First series, whose titles present various technologies in a visually captivating manner. Unlike the typical computer book, Head First titles dispense with the traditional layout of screen shots interspersed amongst paragraphs of text and replace it with a bevy of figures, diagrams, and code snippets. And the overdone corporate examples found in most books are replaced with more interesting ones that tell a story. For example, instead of the archetypical employee database example, Head First SQL follows Greg as he converts his paper-based matchmaking system into a database-driven one.

Head First SQL shines in its ability to explain concepts in a way that makes even the most complicated scenarios seem like common sense. The chapter on JOINs is the most lucid and digestible description I've read yet. The same is true for the chapter on subqueries.

Head First books are a great way to learn a new technology, but they don't typically contain the depth of information found in those stodgy computer books. While paragraph after paragraph of text may be less engaging than the fun layout found in the Head First books, mastering a technology or explaining an especially complex or profound topic often requires the more traditional approach. In short, if you or a colleague needs to get up to speed rapidly on your SQL skills, consider picking up a copy of Head First SQL, but you shouldn't expect the depth of material needed to truly master the topic.

Price: $44.99.

oreilly.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out: Improving Application Startup Performance

CLR Inside Out
Improving Application Startup Performance
Claudio Caldato

Contents
Cold Startup
Identify Code Loaded from Disk
System Assemblies and Other Processes
NGen Performance
Authenticode Verification
Wrapping Up

a
s waiting for an application to start is frustrating to many users, focusing on your client application's startup performance can greatly enhance your customers' first and lasting impressions of your handiwork. And because startup performance matters to users, it's worth exploring the factors that impact it so you can avoid the most common mistakes.

Application startups are typically classified as either cold or warm. In the context of a managed application, cold startup is when neither the Microsoft® .NET Framework system assemblies nor the application's code are in memory, so they all need to be fetched from disk. Warm startup is either a subsequent startup of an application or an application startup when most of the system code is already in memory because it was used previously by another managed application.

Cold Startup

In most cases, cold startup is I/O bound. In other words, more time is spent waiting for data than is spent processing instructions. The time it takes to launch the application is equal to the time it takes the OS to fetch code from disk plus the time it takes to perform additional processing such as JITing the IL code and any other initialization that is performed in the startup path of the application. Since processing is usually not the bottleneck in a cold startup, the initial goal of any application startup performance investigation is to reduce disk access by reducing the amount of code that is loaded.

The way in which the application's code is written also has a significant impact on cold startup so it is important to find out if, for example, the application opens additional files or launches other processes that could compete for I/O resources at startup.

Because cold startup is an I/O bound scenario, the use of a traditional CPU profiler (whether instrumentation-based or sampling-based) will not help investigations significantly. Instrumentation-based profilers will report the time spent waiting on I/O as blocked time. The problem is that, even if you are able to attribute the blocked time to a specific call stack, blocked time is counted only once. Any following disk I/O is not taken into account, resulting in a partial picture of the actual contribution of disk I/O to the total execution time.

With sampling-based profilers, the information collected can even be misleading. Here CPU usage, not I/O, is tracked, so all the time spend in I/O is not accounted for in the profiler's reports.

You can see for yourself in Figure 1 that cold startup is I/O-bound by launching your application twice in a row. The first launch will likely be much slower than the second (where most of the code needed for the execution is already in memory due to the first launch, saving time by avoiding disk access). Of course, to make sure the first launch is indeed a cold startup, restart your machine before and make sure there are no managed applications in the startup folder and that no Windows® service using managed code runs when the user logs on.

Figure 1 Time Reading Disk and CPU Time in Cold Startup
Figure 1 Time Reading Disk and CPU Time in Cold Startup

Note that to perform an ideal test of cold startup, you should disable the SuperFetch service, which may otherwise preload some of the code needed by your app, creating a warmer startup scenario. The benefit of measuring with SuperFetch turned off is that you can count on the fact that all the code needed by the application was loaded into memory on application startup, and hence you can more accurately measure the cost of I/O. You should keep in mind, however, that what you are measuring is not necessarily the actual user experience, so don't draw any concrete conclusions about the actual performance of your application with the data you collect with SuperFetch turned off.

Two performance counters that you can use to see the impact of cold startup on I/O are % processor time and % disk read time. If I/O dominates cold startup, which it should, you should see a big difference between % processor time and disk read time. Performance counters can be collected using PerfMon (see the "Startup Performance Resources" sidebar for more information).

In Figure 1, the red line represents % disk read time and the green line represents % processor time. In the case of cold startup, you can see that CPU usage is relatively low compared with the time spent reading from disk.

The second time the application is launched, you are in the warm startup scenario so the performance counter should show a different picture. In Figure 2, the scenario is CPU bound, and as you can see, % disk read is very low compared with % processor time.

Figure 2 Shorter Times in Warm Startup
Figure 2 Shorter Times in Warm Startup

Warm startup is CPU-bound because the code is already in memory (so there is no need for additional I/O) but code needs to be JITed before the application can run. With the .NET Framework today, the native code generated by the JIT is not saved from one execution of an application to the next.

If you see that warm startup is not significantly smaller than cold startup, you need to discover what is consuming CPU cycles (because most of the code is preloaded in a warm startup and is unlikely to be I/O bound). The likely culprits, then, include large volumes of code that must be JITed or complex computations that the application must perform.

To determine if JITing is the issue, you can check the performance counter .NET CLR JIT\% time in JIT. If the value is not high (for instance more than 30%-40% for most of the startup time), it means that JIT is not likely to be a major contributor and you should use a profiler to determine which functions in your app consume the most CPU time. Keep in mind that the counter is updated only if methods are actually JITed. This means that after the last method has been JITed the counter will still report the last value; it will not drop to zero. Therefore, make sure you look at the counter only for the first few seconds of the app's startup; at that time you are likely to see the counter increase very fast, indicating that the spike in CPU utilization is caused by the JIT compiler.

You should also be aware that any application that's loaded when the user logs on will have to compete for I/O with other services and applications, making startup time even worse. Therefore, avoid adding applications to the startup group whenever possible (a good tool for determining what applications are set to run at machine setup is AutoRuns, available at microsoft.com/technet/sysinternals/Security/Autoruns.mspx).

Identify Code Loaded from Disk
Startup Performance Resources

The following previously published CLR Inside Out columns all contain valuable information for further investigation:
Improving Application Startup Time by Claudio Caldato
msdn.microsoft.com/msdnmag/issues/06/02/CLRInsideOut
Includes some useful tips on how to write code in such a way that it has less impact on the application startup.
The Performance Benefits of NGen by Surupa Biswas
msdn.microsoft.com/msdnmag/issues/06/05/CLRInsideOut
Provides a good overview of the NGen technology with a focus on performance. This article is useful for understanding the costs and benefits of using NGen when considering different scenarios such as warm startup, code sharing, cold startup, and so on.
Investigating Memory Issues by Claudio Caldato and Maoni Stephens
msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut
This piece is not related to startup per se, although it discusses working set, which is another important performance metric. The article covers some more advanced techniques that can be used to determine why a managed application is using too much memory.

Also, you might want to check out the following articles that provide further information to help you reduce startup time:
Bug Bash: Let the CLR Find Bugs for You with Managed Debugging Assistants by Stephen Toub
msdn.microsoft.com/msdnmag/issues/06/05/BugBash
Talks about MDAs, which are debugging aids that work with the CLR to provide information on the state of the runtime. They provide information about runtime events that you would ordinarily not be able to generate.
Inside the Windows Vista Kernel by Mark Russinovich microsoft.com/technet/technetmag/issues/2007/03/VistaKernel
Explains how SuperFetch operates and how ReadyBoost adds to RAM via USB without adding RAM.
Signing and Checking Code with Authenticode msdn2.microsoft.com/ms537364
Gives you all the information you need to understand the Authenticode signing process.
Runtime Profiling
msdn2.microsoft.com/w4bz2147
Explains how to use PerMon and what the different counters mean.

The next step is to determine what is loaded from disk and find out if there is code that is being loaded unintentionally. The quickest way to determine what is loaded into memory is to use the VADump tool (you can find it in the Windows Platform SDK). Figure 3 shows an excerpt of the report that's generated by running the following command:

Figure 3 VADump Output (Close Figure 3)
Category                        Total        Private Shareable    Shared
                           Pages    KBytes    KBytes    KBytes    KBytes
      Page Table Pages       177       708       708         0         0
      Other System            39       156       156         0         0
      Code/StaticData       8169     32676      2160      8336     22180
      Heap                 14042     56168     56168         0         0
      Stack                    0         0         0         0         0
      Teb                      0         0         0         0         0
      Mapped Data              8        32         0         4        28
      Other Data               1         4         4         0         0

      Total Modules         8169     32676      2160      8336     22180
      Total Dynamic Data   14051     56204     56172         4        28
      Total System           216       864       864         0         0
Grand Total Working Set    22436     89744     59196      8340     22208

Module Working Set Contributions in pages
    Total   Private Shareable    Shared Module
       72         2        70         0 HeadTrax - HeadTrax.exe
      107         7         0       100 ntdll.dll
       37         4         6        27 mscoree.dll
       77         3         0        74 KERNEL32.dll
        6         2         0         4 LPK.DLL
       27         4         0        23 USP10.dll
      116         4         0       112 comctl32.dll
      878        23        79       776 mscorwks.dll

Heap Working Set Contributions
   0 pages from Process Heap (class 0x00000000)
   0 pages from Process Heap (class 0x00000000)
9332 pages from Process Heap (class 0x00000000)
  0x0255850F - 0xC255350F 9332 pages
   0 pages from Process Heap (class 0x00000000)
   0 pages from Process Heap (class 0x00000000)
4710 pages from Process Heap (class 0x00000000)
  0x00040000 - 0x10040000 4710 pages
   0 pages from Process Heap (class 0x00000000)

Stack Working Set Contributions
   0 pages from stack for thread 00001018
   0 pages from stack for thread 000017EC
   0 pages from stack for thread 0000187C

VADump –sop <proc ID>

One important thing to keep in mind is that VADump shows only what is loaded in memory at the time the tool is run, so it might miss modules that are loaded in memory for only a short period of time. It also doesn't show the part of the app (either code or data) that has been paged out to disk. The goal is to review the VADump report to determine if it makes sense to load all the modules in the list. For instance, if your application does not use XML and you see that System.Xml was loaded, you need to investigate.

You can find out what loaded an assembly by using the sxe command in the Windows debugger (windbg). The "sxe ld:<dll name>" command causes the debugger to break when the specified DLL is loaded. You can then check the call stack to find out which function caused the DLL to be loaded in memory. This aspect of the investigation should not be underestimated. It is very easy to lose sight of what the application actually loads in memory.

System Assemblies and Other Processes

Once you have eliminated the loading of all unnecessary assemblies at startup (for further improvement, you can also modify the application code to delay some of the initialization work done at startup), the next thing to do is to reduce the amount of code that is loaded from system assemblies. Unfortunately, I know of no tools that can tell you how much code is pulled in if a system API is used. That would be very useful because the developer could use APIs in startup code that need less code to be loaded from the system assemblies. Until such tools become available, you can get an approximate page cost of an API by using an instrumentation-based profiler (for instance, the Visual Studio® Performance Tools).

By looking at the profile data, you can try to avoid APIs that involve system calls with big call trees (big tree, deep-nested calls mean that the code for each method call was pulled out from disk—therefore, it is a way to approximate how expensive the call is). If you can implement the same functionality by calling an API that doesn't have a deep system call tree, you will save time. This is not a scientific approach because it is not easy to determine how much code can be saved by cutting a call tree, but it generally makes sense that the bigger the call tree is, the greater the amount of code loaded from disk will be.

In some cases, your application explicitly or implicitly might launch other processes at startup. You can easily find out what they are by using the –o option in the Windows debugger (windbg). The –o option causes the debugger to attach to any child process. A typical example of a process being launched implicitly by the application occurs when the app uses XML serialization and doesn't precompile the serialization classes (using the Sgen utility). When this happens, the C# compiler is launched to compile them. Launching other processes is usually a very expensive operation that can have a significant impact on startup.

NGen Performance

Native image generation (NGen) always helps improve warm startup because it avoids the cost of JITing code. NGen can also help in cold startup scenarios if mscorjit.dll does not need to be loaded because all code used by the application is already precompiled using NGen. If only one of the modules does not have a corresponding native image, however, the mscorjit.dll will still be loaded. Then not only will the code will be JITed, consuming CPU cycles, but also many pages in the NGen images will be touched because the JIT compiler needs to read metadata. This will result in an even worse startup. For this reason, it is recommended that you remove any code that might cause JITing during startup. Of course, whether or not this approach should be taken can only be determined after measuring performance of cold startup with and without generated native images because the actual benefit of NGen on cold startup depends on the application code and size, so it is not guaranteed to give a significant startup improvement even if there is no JIT involved at startup time.

One way to determine if and when JITing is happening is to use the Managed Debugging Assistant (MDA). JIT MDA allows you to either break into the debugger or print debug information when a method is being JITed. The MDA can be enabled by setting an environment variable, as follows:

COMPLUS_MDA=JitCompilationStart

The application will break into the debugger when code is being JITed. The MDA can also be set using the registry or the application's .config file. See the "Startup Performance Resources" sidebar for more details on how to use MDA.

In general, in order to ensure that NGen is going to benefit cold startup performance, make sure that:

  • The entire application is NGen-ed.
  • There is no rebasing. Rebasing is a very expensive operation, and rebased code cannot be shared. You can find more details on how to set the base address at msdn.microsoft.com/msdnmag/issues/06/05/CLRInsideOut.
  • Assemblies are installed in the Global Assembly Cache (GAC). Strong name verification requires touching the entire file, but strong name verification is skipped for all assemblies installed in the GAC.

Authenticode Verification

Assemblies can be authenticode-signed using the signcode tool. Authenticode verification always has a negative impact on startup because authenticode-signed assemblies need to be verified by a Certificate Authority (CA). The verification requires validating the CA used to sign the assemblies, which is a very expensive operation that requires network access (if the CA is not installed locally on the same machine).

Ideally, you want to avoid authenticode-signing of your assemblies and use strong name signatures instead. If authenticode-signing cannot be avoided, the verification can be skipped in the .NET Framework 3.5 by using the following configuration option:

<configuration>
    <runtime>
        <generatePublisherEvidence enabled="false"/>
    </runtime>
</configuration>

Note, however, that even when authenticode-signing is necessary, most of the time required for the authentication can still be saved simply by installing the CA certificate on the client machine.

Wrapping Up

In order to achieve good cold startup performance, the best thing to do is to keep the code that is executed at startup time very lean. This means postponing initialization that is not strictly needed, checking all the references to make sure they are not loaded too soon and trying to use classes and methods that don't require a lot of code to be loaded. Remember, the goal is to reduce disk access. This is not an easy task, but Xperf, a new, useful tool included in the upcoming Windows Server® 2008 SDK uses Event Tracing for Windows (ETW) to track loaded modules, context switches, and other events to help determine what happens during application startup. With XPerf it will be possible to collect very precise metrics on application startup time. The "Startup Performance Resources" sidebar contains many more references that you'll find helpful.

Send your questions and comments to clrinout@microsoft.com.

Claudio Caldato is a Program Manager for Performance and Garbage Collector on the CLR team at Microsoft.


Data Points: Standard Query Operators with LINQ

Data Points
Standard Query Operators with LINQ
John Papa

Code download available at: DataPoints2008_03.exe (958KB)
Contents
Operators and LINQ
Types of Operators
Lambda Expressions
First and Single
Aggregates, Hierarchies, and Projections
Projections and Ordering
Quantifiers and Conversion
Wrapping Up

L
anguage Integrated Query (LINQ) allows developers to write SQL-like queries with Microsoft® .NET Framework 3.5 code using a strongly typed syntax. Different LINQ providers, such as LINQ to Objects (which allows you to write queries against object hierarchies) and LINQ to Entities (which allows you to write queries against the Entity Framework's conceptual model), are then able to effectively process these queries according to the nuances of their representative data stores.

Besides the strongly typed syntax, LINQ queries also have an arsenal of standard query operators to enhance their functionality. These standard query operators operate on sequences and allow you to perform operations such as determining if a value exists in the sequence and performing an aggregated function, such as a summation, over a sequence.

In this month's column I will perform practical queries and operations with LINQ, using both LINQ to Objects and LINQ to Entities. I'll query a single collection of entities and dive deeper into a hierarchical set of entities using their navigation properties. I'll also show how you can apply many of the standard query operators to arrays and collections. I will demonstrate how LINQ's standard query operators are enhanced using lambda expressions and how they can let you parse specific information from a sequence and perform complex logic over a sequence. All code examples are available in the download provided for this column (see msdn.microsoft.com/msdnmag/code08.aspx).

Operators and LINQ

LINQ is very powerful all on its own whether you are using LINQ to XML, LINQ to DataSets, LINQ to Entities, LINQ to Objects, or any other LINQ provider that comes along. The core power of LINQ is in its strongly typed query syntax, which can be used to operate over any of these providers. When LINQ is combined with one or more standard query operators, the result is an even more powerful toolset that can give you granular control over a set of data.

Standard query operators exist in the System.Core.dll assembly within the System.Linq namespace as extension methods on the static classes Enumerable and Queryable, and they can be used on objects that implement IEnumerable<T> or IQueryable<T>. This allows them to operate on a variety of types, from in-memory collections and arrays (sequences) to remote databases using providers like LINQ to Entities and LINQ to SQL.

It is easy to determine which set of operators you have at your disposal for a specific task. When you want to use an operator in a LINQ query, you can use an operator from the available list of extension methods in the Queryable static class. When you want to use an operator on a sequence that implements IEnumerable<T>, you can use one of the extension methods found in the Enumerable static class. Keep in mind, however, that not all of the operators found in the Queryable class are applicable to the underlying data store, and hence some may not be supported at run time.

Types of Operators

There are many different types of operators (all of which can be found using the object browser to look at the Enumerable and Queryable classes). Figure A shows a categorization of the different types of operators in alphabetical order. It offers you a glimpse at the functionality the operators provide. I will be demonstrating a subset of these operators using both LINQ to Objects and LINQ to Entities to show how they can be beneficial in a real-world application.

Figure A Categories of Operators (Close Figure A)

Operator | Description
Aggregation |
Aggregate | Performs a custom method over a sequence
Average | Computes the average of a sequence of numeric values
Count | Returns the number of the items in a sequence as an int
LongCount | Returns the number of the items in a sequence as a long
Min | Finds the minimum number of a sequence of numbers
Max | Finds the maximum number of a sequence of numbers
Sum | Sums the numbers in a sequence
Concatenation |
Concat | Concatenates two sequences into one sequence
Conversion |
Cast | Casts elements in a sequence to a given type
OfType | Filters elements in a sequence of a given type
ToArray | Returns an Array from a sequence
ToDictionary | Returns a Dictionary from a sequence
ToList | Returns a List from a sequence
ToLookup | Returns a Lookup from a sequence
ToSequence | Returns an IEnumerable sequence
Element |
DefaultIfEmpty | Creates a default element for an empty sequence
ElementAt | Returns the element at a given index in a sequence
ElementAtOrDefault | Returns the element at a given index in a sequence or a default value if the index is out of range
First | Returns the first element of a sequence
FirstOrDefault | Returns the first element of a sequence or a default value if no element is found
Last | Returns the last element of a sequence
LastOrDefault | Returns the last element of a sequence or a default value if no element is found
Single | Returns the single element of a sequence
SingleOrDefault | Returns the single element of a sequence or a default value if no element is found
Equality |
SequenceEqual | Compares two sequences to see if they are equivalent
Generation |
Empty | Generates an empty sequence
Range | Generates a sequence given a range
Repeat | Generates a sequence by repeating an item a given number of times
Grouping |
GroupBy | Groups items in a sequence by a given grouping
Joining |
GroupJoin | Performs a grouped join on two sequences
Join | Performs an inner join on two sequences
Ordering |
OrderBy | Orders a sequence by value(s) in ascending order
OrderByDescending | Orders a sequence by value(s) in descending order
ThenBy | Orders an already-ordered sequence in ascending order
ThenByDescending | Orders an already-ordered sequence in descending order
Reverse | Reverses the order of the items in a sequence
Partitioning |
Skip | Returns a sequence that skips a given number of items
SkipWhile | Returns a sequence that skips items that do not meet an expression
Take | Returns a sequence that takes a given number of items
TakeWhile | Returns a sequence that takes items that meet an expression
Projection |
Select | Creates a projection of parts of a sequence
SelectMany | Creates a one-to-many projection of parts of a sequence
Quantifiers |
All | Determines if all items in a sequence meet a condition
Any | Determines if any items in a sequence meet a condition
Contains | Determines if a sequence contains a given item
Restriction |
Where | Filters the items in a sequence
Set |
Distinct | Returns a sequence without duplicate items
Except | Returns a sequence representing the difference between two sequences
Intersect | Returns a sequence representing the intersection of two sequences
Union | Returns a sequence representing the union of two sequences

Lambda Expressions

Many standard query operators use Func delegates to process individual elements as they operate over a sequence. Lambda expressions can work in concert here with the standard query operators to represent the delegates. A lambda expression is a shorthand form for creating an implementation of a delegate and can be used anywhere an anonymous delegate can be used. Lambda expressions are supported by both C# and Visual Basic® .NET. It is important to note, though, that because Visual Basic .NET does not yet support anonymous methods, lambda expressions may consist of only a single statement.

Let's see how the Single operator can be used on an array of integers. An array of integers, each element representing a power of 2 from 1 through 10, is created and then the Single operator is used to retrieve the single integer element that meets the condition specified in the lambda expression:

int[] nums = { 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 };
int singleNum = nums.Single(x => x > 16 && x < 64);
Console.WriteLine(singleNum.ToString());

Lambda expressions have a few key parts. The lambda expression starts by defining the variables passed into the delegate. In the previous code example, x (declared on the left-hand side of the => operator) is the argument that represents each element of the nums array that is passed to it. The remaining portion of the lambda expression represents the logic that is evaluated for each element in the array. The previous expression could have just as easily been rewritten using an anonymous delegate, as follows:

int singleNum = nums.Single<int>(
  delegate(int x) {return (x > 16 && x < 64); }
) ;

However, this code is less readable than using the lambda expression. C# 2.0 introduced anonymous delegates, which made passing delegates a little easier; however, lambda expressions took that to new heights of simplicity with its terse syntax.

First and Single

When it is necessary to get a single value from a sequence, the First, FirstOrDefault, Single, and SingleOrDefault operators are very useful. The First method returns the first element in a sequence. There is an overloaded method for First that allows you to pass in a lambda expression to represent a condition. For example, if you want to return the first element in a sequence of integers where the integer element is greater than 50, you could use the following code sample:

int[] nums = { 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 };
int num1 = nums.First<int>();
int num2 = nums.First<int>(x => x > 50);
int num3 = nums.FirstOrDefault<int>(x => x > 5000);

Console.WriteLine(
  num1.ToString() + "-" +
  num2.ToString() + "-" +
  num3.ToString());

This code will find the first element (1), the first element that is greater than 50 (64), and the first element greater than 5,000. Because there is no element in the array that satisfies the third lambda expression (no integer in the array is greater than 5,000), an exception would be thrown if the code used the First operator instead of FirstOrDefault. By using the FirstOrDefault operator, 0 is returned if no element satisfies the lambda expression. The First operator can also be used in a LINQ to Entities query, as you see here:

using (Entities entities = new Entities())
{
  var query = (from c in entities.Customers
               select c).First(c => c.City.Equals("London"));
  Console.WriteLine(query.CompanyName);
}

In this example, the first customer in the city of London will be returned. As you can see, the syntax for using the First method does not change when using it with different LINQ providers—in this case LINQ to Objects and LINQ to Entities.

The First operator is very useful in the context of LINQ to Entities, especially when you know a single record will be returned from a query. For example, you might have a query that always gets a customer record when given the CustomerID. In this situation you would always have 0 or 1 records returned, so a sequence does not help you nearly as much as the entity itself. In other words, you would rather just fetch the Customer entity instead of a sequence of 1 Customer entity. This is a case where the First method can help, as shown in the code snippet that follows. (Because the Entity Framework does not try to distribute the execution of a single query across client and server, the single method is not supported in LINQ to Entities, so using the First method is an easy alternative.)

using (Entities entities = new Entities())
{
  var query = (from c in entities.Customers
               where c.CustomerID.Equals("BOLID")
               select c).First();
  Console.WriteLine(query.CompanyName);
}

Aggregates, Hierarchies, and Projections

Using an aggregate operator such as Sum on a LINQ to Entities query can help simplify a query. For example, the following code retrieves a sequence of orders where the order total is more than $10,000:

using (Entities entities = new Entities())
{
  var query = from o in entities.Orders
              where o.OrderDetails.Sum(
                od => od.UnitPrice * od.Quantity) >= 10000
              select o;
  foreach (Orders order in query)
    Console.WriteLine(order.OrderID);
}

Because LINQ can query hierarchical entity collections, standard query operators can also be used to perform operations over the nested sequences of entities. This can be very useful when derived data must be calculated or interrogated. The derived data may only exist in its base form, such as in the case where the details of the customer orders have only values for unit price and quantity. In this case, aggregated data representing the total amount for an order is not provided at any point in the model. By applying the Sum operator to your LINQ query, however, you can still retrieve all customers who have spent more than $20,000, as shown here:

using (Entities entities = new Entities())
{
  var query = from c in entities.Customers
              where c.Orders.Sum(
                o => o.OrderDetails.Sum(
                  od => od.UnitPrice * od.Quantity)) >= 25000
              select c;
  foreach (Customers customer in query)
    Console.WriteLine(customer.CompanyName);
}

This example demonstrates how you can apply the standard query operators at multiple levels of a LINQ query. The query eventually returns a sequence of Customers entities, but to get there it must first dive into each customer's orders and each order's order details to get the data it needs so that it can calculate each line item's price, sum the total of the line items for each order, and sum the order totals for each customer.

The Count operator is another aggregate standard query operator. You can determine how many customers spent more than $25,000 by using the following code:

using (Entities entities = new Entities())
{
  var query = (from c in entities.Customers
               where c.Orders.Sum(
                 o => o.OrderDetails.Sum(
                   od => od.UnitPrice * od.Quantity)) >= 25000
               select c).Count();
  Console.WriteLine(query);
}

The Max operator can be used to determine the best customer. The following code sample will return the amount that the highest-spending Customer spent. It uses a combination of the Sum and the Max aggregators over multiple levels of a hierarchy:

using (Entities entities = new Entities())
{
  var query = (from c in entities.Customers
               select new
               {
                 c.CustomerID,
                 Total = c.Orders.Sum(
                   o => o.OrderDetails.Sum(od => od.UnitPrice))
               }).Max(c2 => c2.Total);
  Console.WriteLine(query);
}

Projections and Ordering

You also may have noticed that I snuck in a projection in the previous example. Before using the Max operator, the LINQ query does not return a list of customers. Instead, it returns a projection, which creates a new entity that contains a property for the CustomerID and a property for the Total (the customer's entire spending amount). Projections are integral to LINQ, and when they are projected into sequences, as in the previous example just shown, they can be further processed using standard query operators.

Figure 1 shows how to create a projection of a new entity that contains a CustomerID and the sum of that customer's order totals (using the Sum operator discussed previously). Figure 1 also uses the OrderByDescending operator to order the sequence of projected entities by the calculated total. If two customers had the same total, an additional ordering operator could be used to further define the ordering. For example, the foreach statement in Figure 1 could be amended to use the following code to further qualify the ordering rules:

Figure 1 Aggregates, Projections, and Ordering (Close Figure 1)
using (Entities entities = new Entities())
{
  var query = from c in entities.Customers
              where c.Orders.Sum(
                o => o.OrderDetails.Sum(od => od.UnitPrice)) > 0
              select new
              {
                c.CustomerID,
                Total = c.Orders.Sum(
                  o => o.OrderDetails.Sum(od => od.UnitPrice))
              };
  foreach (var item in query.OrderByDescending(x => x.Total))
    Console.WriteLine(item.CustomerID + " == " + item.Total);
}

foreach (var item in
  query.OrderByDescending(x => x.Total)
  .ThenBy(x => x.CustomerID))
{
  Console.WriteLine(item.CustomerID + " == " + item.Total);
}

In this code snippet, I added the ThenBy operator and a lambda expression to indicate that the sequence should be ordered first in descending order by the Total property and then in ascending order by the CustomerID property of the projection.

Quantifiers and Conversion

When you need to determine if a value exists in a sequence, the standard query operator Any can be used. Quantifiers such as Any, All, and Contains search a sequence of elements and evaluate whether or not the sequence meets a lambda expression's condition. This can be very helpful when you need to examine a sequence to determine things like whether a customer from a specific address exists, if all customers are from the same country, or any other analytical deterministic question.

For example, the following LINQ query checks to see if all customers from the United Kingdom are located in London. This uses the quantifier All and passes it the lambda expression that simply evaluates if the city is London. If every element in the sequence meets this criterion and returns true from the lambda expression, then the All operator will return true:

using (Entities entities = new Entities())
{
  bool allUKCustomerAreFromLondon = (from c in entities.Customers
                                     where c.Country == "UK"
                                     select c).All(
                                       c => c.City.Equals("London"));
  Console.WriteLine(allUKCustomerAreFromLondon ? "Yes" : "No");
}

Another question that might need to be asked in this query is if there are any entities in the sequence from the city Cowes in the United Kingdom. For this question, you can use the Any quantifier to operate on the sequence, as shown here:

using (Entities entities = new Entities())
{
  bool isOneUKCustomerFromCowes = (from c in entities.Customers
                                   where c.Country == "UK"
                                   select c).Any(
                                     c => c.City.Equals("Cowes"));
  Console.WriteLine(isOneUKCustomerFromCowes? "Yes" : "No");
}

The Contains operator is similar to the Any operator in that it evaluates whether the sequence contains what you are looking for. The Any operator can determine if a value exists within an item in the sequence, but the Contains operator determines if a specific item instance exists in the sequence. For example, before you add an object to a sequence, you might want to make sure the sequence does not already contain that object. Figure 2 demonstrates how to check.

Figure 2 Using Contains and Conversion (Close Figure 2)
using (Entities entities = new Entities())
{
    Customers customerBSBEV = (from c in entities.Customers
                               where c.CustomerID == "BSBEV"
                               select c).First();

    var customersUK = from c in entities.Customers
                      where c.Country == "UK"
                      select c;

    bool isCustomerInSequence = customersUK.Contains(customerBSBEV);

    Console.WriteLine(isCustomerInSequence? "Yes" : "No");
}

Notice in Figure 2 that a Customers entity is retrieved for the BSBEV customer. Then the sequence of Customers entities where the customers are from the United Kingdom is retrieved. Finally, the Contains operator is used to check whether the sequence of customers contains the instance held by the customerBSBEV variable.

The implementation of the Contains operator shown in Figure 2 works in situations where you can confidently compare objects based on their actual instance. However, what if you need the Contains operator to test based on a logical identity? Fortunately, the Contains operator has an overload allowing you to pass in an object implementing the IEqualityComparer<T> interface. To use Contains based on CustomerID, the code in Figure 2 can be rewritten as follows

using (Entities entities = new Entities())
{
  ...

  bool isCustomerInSequence = customersUK.Contains(customerBSBEV,
    new CustomerComparer());

  Console.WriteLine(isCustomerInSequence? "Yes" : "No");
 }

where CustomerComparer is defined as

private class CustomerComparer : IEqualityComparer<Customers>
{
  public bool Equals(Customers x, Customers y) {
    if (x == null || y == null)
      return false;
    return x.CustomerID.Equals(y.CustomerID);
  }

  ...
}

Wrapping Up

There are many standard query operators that are defined as extension methods for the Enumerable and Queryable sequence classes. These operators can help extend the functionality of LINQ, as I've show you here. I've also demonstrated how many of the new enhancements to the .NET Framework 3.5 (including lambda expressions, LINQ, the Entity Framework, and implicitly typed variables) come together to make writing robust code and logic easier.

Send your questions and comments for John to mmdatat@microsoft.com.

John Papa is a senior .NET consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP and INETA speaker, has authored several books on data access technologies. He can often be found speaking at industry conferences or blogging at www.johnpapa.net.


Advanced Basics: Office 2007 Files and LINQ

Advanced Basics
Office 2007 Files and LINQ
Ken Getz

Code download available at: AdvancedBasics2008_03.exe (162KB)
Contents
Getting Started
Investigating the Code
Wrapping It Up

I
n the June 2006 installment of this column, I described how you can programmatically interact with the Microsoft® 2007 Office system Open XML File Formats using classes in the System.IO.Packaging and System.Xml namespaces (see msdn.microsoft.com/msdnmag/issues/06/06/AdvancedBasics). As I was writing that column, I was aware of some new technologies that would make my code simpler, but they weren't quite ready yet. Now that Visual Studio® 2008 has been released, I'll revisit the code using LINQ to XML and the Community Technology Preview (CTP) edition of the Microsoft SDK for Open XML Formats, which you'll need to download from go.microsoft.com/fwlink/?LinkId=106944 in order to compile and test the code presented here. These two technologies can change the way in which you access the parts of the 2007 Office system Open XML document (created by Microsoft Word, Excel®, and PowerPoint®) when retrieving or modifying data.

As I described in the June 2006 column, in order to programmatically interact with the Office Open XML File Formats you must be able to navigate to the exact document part you require and then interact with the XML in that part once you retrieve a reference to it. Usually you use the System.IO.Packaging namespace to navigate through the file format's hierarchy of parts. Once you get to the part that you want to program against, you can use any existing XML technology to interact with it; generally, you'll either use the XmlDocument class or the XmlReader and XmlWriter classes.

The classes in the System.IO.Packaging namespace certainly make it possible to navigate the file format's hierarchy of parts, but because it's so general (it works with any Open XML File Format, not just 2007 Office system documents), the code you write using this API is generally more repetitive than you might like. To make this navigation easier, members of the Office team at Microsoft have been working on an SDK that encapsulates the generalized System.IO.Packaging namespace, providing an API that's focused directly on the 2007 Office system file formats. I decided to use this SDK, even though it's currently only available as a CTP, because it so greatly simplifies code that works with Office documents.

The 2007 Office system Open XML SDK only handles the navigation to the individual part of interest; it is not used for manipulating the part's XML content to retrieve or modify the document information. In the earlier column, I used the XmlDocument class to work with document properties. Now, you can use LINQ to XML, a new feature in the .NET Framework 3.5 (exposed in Visual Studio 2008) to make working with the XML content far easier.

For this column, I'll modify the WDSetCustomProperty code, which was one of the forty snippets I wrote when the 2007 Office system first launched (go.microsoft.com/fwlink/?LinkId=106943). Before investigating the code in this article, you might want to examine the original code snippet to see how it works. (You'll find a walkthrough of the code, plus a video demonstration, at msdn2.microsoft.com/bb308936.)

You should also be relatively comfortable with the 2007 Office system Open XML File Formats, so begin by reviewing my earlier column. The code related to the 2007 Office system Open XML SDK is fairly self-evident, as you'll see when you review the sample code for this article. (In order to use the SDK, you'll need to download and install it, and within your project, set a reference to the Microsoft.Office.DocumentFormat.OpenXml.dll assembly included in the SDK.) The final technology I'll cover here, LINQ to XML, is a broad topic on its own, so all I'll do is discuss how I've used it in the context of this column. Although much has already been written on this important subject, you may want to start with the video presentation "How Do I Get Started with LINQ to XML?" at msdn2.microsoft.com/bb466226.

In creating the code sample for this column, 90 percent of my time was devoted to XML namespace manipulation. Visual Basic® provides some amazing support for LINQ to XML in its latest version, but it's easy to get tangled up in namespace manipulation. The description of the feature at msdn2.microsoft.com/bb687701 will be useful to you—it certainly helped me work things out!

Getting Started

In order to set or retrieve Word 2007 custom document properties, you'll need to work with the docProps/custom.xml part. As you attempt to set a property, one of four situations can occur:

  • The custom property part may not yet exist, in which case, you must create it.
  • The custom property may not exist, in which case, you must create it.
  • The custom property may exist, and its type matches the type of the new value. In this case, you simply set the property's value.
  • The custom property may exist, but its type differs from the type of the new value. In this case, delete the property, and recreate it using the new type. (You could devise other means of resetting the type, but the code you'll see here follows the same path if the property doesn't yet exist, or it does exist and has the wrong type. Either way, the code creates a new element for the property).

A sample set of custom properties might look like Figure 1.This particular document contains four custom properties: Author (Ken), Birthdate (1/1/1956), Chapter (2), revisions (5). Note that the XML content is scoped to the default namespace:

Figure 1 Some Custom Properties (Close Figure 1)
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<Properties
  xmlns:vt="http://schemas.openxmlformats.org/officeDocument/2006
    /docPropsVTypes"
  xmlns="http://schemas.openxmlformats.org/officeDocument/2006
    /custom-properties">
  <property fmtid="{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"
    pid="2" name="Author">
    <vt:lpwstr>Ken</vt:lpwstr>
  </property>
  <property fmtid="{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"
    pid="3" name="Birthdate">
    <vt:filetime>1956-01-01T06:00:00Z</vt:filetime>
  </property>
  <property fmtid="{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"
    pid="4" name="Chapter">
    <vt:i4>2</vt:i4>
  </property>
  <property fmtid="{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"
    pid="5" name="Revisions">
    <vt:i4>5</vt:i4>
  </property>
</Properties>

http://schemas.openxmlformats.org/officeDocument/2006/custom-properties

The content also includes a non-default namespace that it uses to describe the property types:

vt="http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes"

Any code you write that works with this part needs to respect and maintain those namespaces or Word 2007 simply won't load your document (a fact I learned the hard way, after many failed attempts.)

Each property element contains an fmtid attribute with the same GUID value ({D5CDD505-2E9C-101B-9397-08002B2CF9AE}). Each property also includes a pid attribute, which must be a numeric value, starting with 2, and increasing in value. Although Word 2007 allows you to skip values (you can have properties with pid values of 2, 3, and 5, for example), these values cannot be the same, and they must appear in numeric order. Finally, each property includes a name attribute containing the name of the property.

Within each property element, you must add an element that contains the value of the property. The element name, in each case, indicates the type of value the property contains. The sample code provides an enumeration with a value representing each of these types:

Public Enum PropertyTypes
  YesNo
  Text
  DateTime
  NumberInteger
  NumberDouble
End Enum

In the XML content, these types correspond to bool, lpwstr, filetime, i4, and r8, respectively. Each of these type names belongs to the XML namespace which has been assigned to the alias vt.

In order to make it easy to interact with XML namespaces, Visual Basic overrides the behavior of the file-level Imports statement. You can specify abbreviations for namespaces at the top of a file and use those abbreviations within the file. The sample code uses the following Imports statements:

Imports System.IO
Imports Packaging = Microsoft.Office.DocumentFormat.OpenXml.Packaging
Imports <xmlns="http://schemas.openxmlformats.org/officeDocument/2006
/custom-properties">
Imports <xmlns:vt="http://schemas.openxmlformats.org/officeDocument
/2006/docPropsVTypes">

I've used the Packaging alias so that it's clear within the code when I'm using code from the 2007 Office system Open XML SDK. The code also includes Imports statements for the default namespace and the namespace with the alias vt. (The fact that you can import namespaces in this manner makes coding easier, but it does force one limitation: you can define only one default namespace per file. If you want to include XML code that uses a different default namespace, you'll need to place that code in a separate file. Of course, because you can split a single class into multiple files by creating partial classes, this isn't a restriction except with regard to file management in your project.)

In order to set the document property, you'll need to create a method that can be called like this:

WDSetCustomProperty("C:\YourDocument.docx", "PropertyName", _
  "PropertyValue", PropertyTypes.Text)

If the property value you supply doesn't match the PropertyTypes enumerated value you specified, the code should raise an exception and return without attempting to perform any operation.

Throughout the sample code, you'll see examples of features that are new in Visual Studio 2008. Among other features that I'll call out specifically, you'll see code that uses implicit type declaration. Rather than specifying a type for each variable, Visual Basic uses type inference to infer the type at compile time. This means that you can write code like this:

Using wdPackage = _
  Packaging.WordprocessingDocument.Open(docName, True)
  ...
End Using

The explicit declaration would look like this:

Using wdPackage As Packaging.WordprocessingDocument = _
  Packaging.WordprocessingDocument.Open(docName, True)
  ' Code removed here.
End Using

Not specifying the exact type doesn't detract from the readability of the code, and Visual Basic compiles both versions in the same way. (This feature is controlled by the Option Infer file-level option, which is set to On by default.)

Investigating the Code

Now I'll investigate the sample code, explaining each block. The entire WDSetCustomProperty procedure is available in the code download on the MSDN® Magazine Web site. You might want to refer to it as the text here walks through the fragments of the procedure. It begins by investigating the parameters, converting the input value as necessary, as shown in Figure 2.

Figure 2 Checking the Types and Performing Conversions (Close Figure 2)
Dim propertyTypeName As String = Nothing
Dim propertyValueString As String = Nothing

' Calculate the correct type:
Select Case propertyType
  Case PropertyTypes.DateTime
    propertyTypeName = "filetime"
    ' Make sure you were passed a real date,
    ' and if so, format in the correct way.
    ' The date/time value passed in should
    ' represent a UTC date/time.
    If TypeOf (propertyValue) Is DateTime Then
      propertyValueString = String.Format("{0:s}Z", _
        Convert.ToDateTime(propertyValue))
    End If
  Case PropertyTypes.NumberInteger
    propertyTypeName = "i4"
    If TypeOf (propertyValue) Is Integer Then
      propertyValueString = propertyValue.ToString()
    End If

  Case PropertyTypes.NumberDouble
    propertyTypeName = "r8"
    If TypeOf propertyValue Is Double Then
      propertyValueString = propertyValue.ToString()
    End If

  Case PropertyTypes.Text
    propertyTypeName = "lpwstr"
    propertyValueString = propertyValue.ToString()

  Case PropertyTypes.YesNo
    propertyTypeName = "bool"
    If TypeOf propertyValue Is Boolean Then
      ' Must be lower case!
      propertyValueString = _
        Convert.ToBoolean(propertyValue).ToString().ToLower()
    End If
End Select

If propertyValueString Is Nothing Then
  ' If the code wasn't able to convert the
  ' property to a valid value, throw an exception:
  Throw New InvalidDataException("Invalid parameter value.")
End If

Based on the enumerated value you supply, indicating the type of the value, the code attempts to convert the Object value you've provided to the correct type. Note that in order for Word 2007 to accept the values, date/time values must be in Universal Time Coordinate (UTC) time (that is, based on Greenwich mean time) and Boolean values must be in lower case. If, after investigating the values, the propertyValueString variable is still Nothing, the code raises an exception back to the caller, ending the procedure.

Next, the code uses the Microsoft SDK for Open XML Formats to find the document part and the custom properties part within the package. Compare this code to the code from the original snippet, and you'll have to agree that the SDK makes it far easier to navigate within the document package:

Using wdPackage = _
  Packaging.WordprocessingDocument.Open(docName, True)

  Dim documentPart = wdPackage.MainDocumentPart
  Dim customPropsPart = wdPackage.CustomFilePropertiesPart
  ...
End Using

If the code is able to retrieve a reference to the custom properties part (that is, if the customPropsPart variable isn't Nothing), the code loads the contents of the part using the XDocument.Load method (see Figure 3), allowing the GetStream method of the part to retrieve the XML contents. If the part doesn't exist, the code uses the SDK to create the custom properties part. If the part does exist, the code also attempts to retrieve the first property node for which the name attribute matches the property name you've specified. Again, compare this to the original code snippet—you'll be amazed how much easier this technique is.

Figure 3 Find the Custom Properties Part and Set Properties (Close Figure 3)
Dim customDoc As XDocument = Nothing
Dim propertyNode As XElement = Nothing
If customPropsPart IsNot Nothing Then
  ' Load the existing custom properties part:
  customDoc = XDocument.Load( _
    New StreamReader(customPropsPart.GetStream()))

  propertyNode = _
    (From node In customDoc.<Properties>.<property> _
     Where node.@name = propertyName).FirstOrDefault()
Else
  customPropsPart = wdPackage.AddCustomFilePropertiesPart
End If

This code statement makes use of LINQ to XML and the Visual Basic axis properties to simplify the XML manipulation:

propertyNode = _
  (From node In customDoc.<Properties>.<property> _
   Where node.@name = propertyName).FirstOrDefault()

The "." (period) separator indicates a node one level beneath the parent (customDoc). The <> syntax indicates an element name. The @ syntax indicates an attribute name. These are specific Visual Basic features (in C#, you would need to use the Element or Attribute property of the XDocument object to get the same information). The code also uses the FirstOrDefault LINQ extension method. This method returns the first matching node, or the default value (Nothing) if it finds no match. You could also use the First or Single extension methods, but each of those raises an exception if no match is found. Here, the code handles the case in which no match was found without raising an exception.

After this code executes, customPropsPart contains either a reference to the custom properties part, or Nothing. Likewise, propertyNode contains either a reference to the XElement object corresponding to the property you want to modify, or Nothing if the property doesn't yet exist.

If the propertyNode variable isn't equal to Nothing, the code retrieves the node's first child element, using the First method (in this case, if the code can't find the first child element, it raises an exception—the property node must contain an element, and if it doesn't, the code can't continue because the document is damaged). If the name of the child element matches the type of the new property value, the code changes the value of the element. If not, the code removes the node and sets the propertyNode variable to Nothing. After this code runs, the propertyNode variable either contains the new value or is set to Nothing, as you see in Figure 4.

Figure 4 Setting the PropertyNode Variable Value (Close Figure 4)
If propertyNode IsNot Nothing Then
  Dim propertyContentNode = _
    propertyNode.Elements.First

  If propertyContentNode.Name.LocalName = _
    propertyTypeName Then

    ' Type is the same, so just change the value.
    propertyContentNode.Value = propertyValue.ToString()
  Else
    ' Type is different, so delete the node.
    propertyNode.Remove()
    propertyNode = Nothing
  End If
End If

If the propertyNode variable is Nothing, it's time to create the node and set its content. The code in Figure 5 starts out by creating the fmtid constant and calculates the pid value to use. To calculate the pid to use for the new element, the code retrieves a reference to the Properties element and then finds the last child of the element. If the last child exists, the code retrieves the child's pid value, increments it, and uses the new value. Note the various LINQ to XML features used in Figure 5: the First and LastOrDefault methods provide simple ways to access child elements, given specific criteria.

Figure 5 Creating the Node (Close Figure 5)
If propertyNode Is Nothing Then
  Const fmtid As String = _
    "{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"

  Dim pid As String = "2"

  Dim propertiesNode As XElement = Nothing
  If customDoc IsNot Nothing Then
    propertiesNode = customDoc.<Properties>.First

    Dim lastChild = propertiesNode.Elements.LastOrDefault
    If lastChild IsNot Nothing Then
      pid = (Integer.Parse(lastChild.@pid) + 1).ToString()
    End If
  End If
  ...
End If

Next, the code uses some features specific to Visual Basic to create the new property node:

Dim propertyTypeXName As XName = _
  GetXmlNamespace(vt) + propertyTypeName

propertyNode = _
  <property fmtid=<%= fmtid %>
    pid=<%= pid %> name=<%= propertyName %>>
    <<%= propertyTypeXName %>><%= propertyValueString %></>
  </property>

In order to create the property node's value element, the code requires an XName object containing the full type name (vt:i4, for example). To create that XName object, you use the new GetXmlNamespace keyword. Supplying the namespace abbreviation that you created in the Imports statement enables GetXmlNamespace to retrieve the corresponding XML namespace. Visual Basic overloads the + operator when used with XML namespaces, so you can write a statement like this to create the full XName instance:

Dim propertyTypeXName As XName = _
  GetXmlNamespace(vt) + propertyTypeName

Next, the code uses one of the best new features in Visual Basic—replacement tokens within XML literals. Visual Basic allows you to embed XML content directly in the code, so you can refer to and create XML without needing to directly program against the Document Object Model (DOM) objects. In this case, the code creates a new XML element:

propertyNode = _
  <property fmtid=<%= fmtid %>
    pid=<%= pid %> name=<%= propertyName %>>
    <<%= propertyTypeXName %>><%= propertyValueString %></>
  </property>

After this statement executes, the propertyNode variable contains an XElement object containing XML that looks like this:

<property fmtid="{D5CDD505-2E9C-101B-9397-08002B2CF9AE}"
  pid="4" name="Chapter">
  <vt:i4>2</vt:i4>
</property>

Comparing the two code fragments, it's easy to see how Visual Basic makes the replacements: each time it encounters a replacement token (delimited with <%= %>) it replaces the named token with its value. The tokens can even contain expressions, as you'll see later. Note that the replacement token for an element name must contain an XName reference, not a string or some other value. (I have to thank Avner Aharoni, of the Visual Basic team, for helping me work out this little detail.)

The final goal, before saving the custom XML part back to the document, is to recreate the entire contents of the part. The code must create the container XML document, insert all the existing custom properties, and then append the new custom property.

It's possible to take the existing custom properties part and simply call the Add method of the Properties element to add a new property child element (and that's how I originally wrote the code). Doing so bypasses an important feature of Visual Basic namespace handling. When you use the Add method, Visual Basic can't determine how it should handle namespace factoring, resulting in the insertion of redundant namespace information. In this case, Visual Basic inserts an explicit default namespace reference. Although the XML it creates is perfectly valid, the XML parser in Word fails when it tries to load the custom properties part. If you use the Add method, you'll often find that you must explicitly remove redundant namespace references. (Thanks again to Avner, who explained this behavior to me.)

To avoid this problem, the sample code completely recreates the custom properties part. Using this technique allows Visual Basic to factor out redundant namespace references and creates exactly the XML that Word expects:

customDoc = _
  <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  <Properties
    xmlns:vt=
    "http://schemas.openxmlformats.org/officeDocument
      /2006/docPropsVTypes">
    <%= If(propertiesNode IsNot Nothing, _
      propertiesNode.Elements, Nothing) %>
    <%= propertyNode %>
  </Properties>

This code doesn't need to reference the default namespace; because Visual Basic already has the information about the default namespace from the Imports statements, it can infer that information. For any specific namespaces that the code requires, you'll need to include the information in the XML literal, as you just saw for the vt namespace.

This XML content includes a replacement token for all the existing property elements and uses the new If function in Visual Basic. The If function, unlike the older IIf function, doesn't have the side effect of the original; instead of evaluating both expressions and determining which to return, the If function evaluates its parameters from left to right. This bypasses an error that would have occurred in this case using the IIf function (if propertiesNode was Nothing, the IIf function would still have attempted to retrieve its Elements property, causing an exception). After outputting the existing property elements (if they exist), the code outputs the new property node. This results in customDoc containing the full XML content of the custom properties part.

There is one final challenge left—saving the part back to file storage. The XDocument class and the SDK provide the solution here. You can call the Save method of the XDocument object, customDoc, which allows you to specify a stream into which to write the XML content. Conveniently, the CustomFilePropertiesPart class provides a GetStream method, returning the stream corresponding to the part's storage within the package:

customDoc.Save(New StreamWriter( _
  customPropsPart.GetStream( _
  FileMode.Create, FileAccess.Write)))

Wrapping It Up

Although this seems like a lot of code to satisfy a rather simple goal, once you remove the comments and the code that formats the parameters, the code that actually performs the work with the document is relatively short. Compare it to the code in the original snippet, and you'll have to agree that using the Microsoft SDK for 2007 Office system Open XML Formats and LINQ to XML in Visual Basic dramatically simplifies the process of working with 2007 Office system Open XML documents.

If you search the Web for LINQ to XML, you'll find many documents to help get over the hurdle of working with XML using LINQ. Although not specific to Visual Basic, the document ".NET Language-Integrated Query for XML Data" (msdn2.microsoft.com/bb308960) covers much of what you'll need to know. You'll also find the Visual Basic team's blog (blogs.msdn.com/vbteam) to be a good source. I'd love to have the bandwidth to go back and revise all 40 of the original code snippets to use these new technologies, but it's not likely I'll have time to do that. You can, however, take what you've learned here and use it to modify any code you have written that interacts with the 2007 Office system Open XML File Formats. You'll certainly reduce the complexity and the volume of code in doing so.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a Senior Consultant with MCW Technologies, and a courseware author for AppDev (www.appdev.com). He is coauthor of ASP .NET Developers Jumpstart, Access Developer's Handbook, and VBA Developer's Handbook, 2nd Edition. Reach him at keng@mcwtech.com.


Office Space: Simplify SharePoint Development with STSDEV

Office Space
Simplify SharePoint Development with STSDEV
Ted Pattison

Code download available at: OfficeSpace2008_03.exe (762KB)
Contents
The RootFiles Directory
Automating the Building of Solution Packages
Getting Started with STSDEV
Creating Custom Build Targets with MSBuild
Extending the STSDEV Utility

I
started developing for Windows® SharePoint® Services (WSS) 3.0 and Microsoft® Office SharePoint Server (MOSS) 2007 back in the summer of 2005. Since that time I have created hundreds of new class library projects and configured them by hand to develop various types of SharePoint components such as Features, Application Pages, Page Templates, and Web Parts.

After a while I started to standardize on a common project structure of files and folders within the solutions I created with Visual Studio® 2005 for SharePoint development. However, this had a painful aspect: for each project, it would always take me about five or ten minutes just to create the basic structure that would be my starting point. Very little was different in the source files from project to project other than the project name.

Another painstaking issue many developers face with SharePoint development has to do with creating the necessary files to generate the solution package file (.wsp) required for testing and deployment. Like many other SharePoint developers, I grew accustomed to creating and updating a manifest.xml file and a .ddf file by hand in order to generate my solution packages.

Over the past year I have been collaborating with some other SharePoint MVPs including John Holliday, Andrew Connell, and Scot Hillier, who had all come up with very similar yet slightly different schemes to accomplish the same goal. We discussed the idea of designing and implementing a code generator that could automatically produce all the source files needed for a new Visual Studio project and solution. The goal, of course, would be to provide a much faster technique for getting up and running with a new SharePoint development project.

Once we created a code generator that could build every new Visual Studio solution with a common folder structure, it would then also be possible to write the code required to regenerate the manifest.xml file and the .ddf file required to produce the solution package .wsp file. This would eliminate the need to modify manifest.xml or the .ddf file by hand.

This month's column is accompanied by a proof-of-concept application named STSDEV.EXE that demonstrates how to create a simple code generator for SharePoint development. STSDEV is a console application that makes it easy to develop components for MOSS 2007. Furthermore, the fact that it generates code in a repeatable fashion also makes it possible to build Visual Studio solutions for SharePoint development that adhere to best practices.

My goal in this month's column is to discuss how to use the STSDEV utility to create and extend a Visual Studio project that produces an effective and flexible environment for SharePoint development. The approach outlined here provides flexibility because it can be used with either Visual Studio 2005 or Visual Studio 2008. Furthermore, this approach will ease migration when it's time to move from one version of Visual Studio to the next.

The RootFiles Directory

An important aspect of understanding how deployment works in SharePoint development is knowing where your templates files need to be copied on the front-end Web server. A significant percentage of the files must be deployed on each and every front-end Web server inside a location known as the SharePoint RootFiles directory, which in WSS 3.0 is located here:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12

Take a look at Figure 1 to review the child directories nested inside the RootFiles directory. The directory into which you most commonly need to deploy custom templates files is the TEMPLATE directory. But it is also sometimes necessary to deploy files inside some of these other child directories, such as the ISAPI directory when you are deploying a custom Web service and the Resources directory when you need to deploy .resx files with global resources.

Figure 1 Folders Within RootFiles
Figure 1 Folders Within RootFiles

The TEMPLATE directory also contains several important child directories. For example, each custom feature requires its own named directory within the FEATURES directory that contains a feature manifest named feature.xml. Application pages should be copied to a child directory within the LAYOUTS directory.

Automating the Building of Solution Packages

It is a best practice in MOSS 2007 development to deploy all custom templates and components using solution packages. In fact, if you ever discover a component type that cannot be deployed via a solution package, you should consider avoiding it when designing and developing SharePoint-based solutions because it will compromise your ability to install, maintain, and upgrade your development efforts in a production or staging environment.

When you are developing for WSS 3.0 or MOSS 2007, you should configure Visual Studio projects to generate a solution package for you, and you should do it early in the development cycle of any project. I have seen many developers defer creating the solution package for testing purposes until late in the development lifecycle, which can cause problems and unexpected delays. It is much better to do the extra work of configuring your Visual Studio project to build the solution package right away and to conduct all testing using solution package deployment.

So how do you configure a Visual Studio project to automate the build of the .wsp file that represents the solution package? You have a few choices. For example, you can write a batch file that calls the MAKECAB.EXE utility to generate the .wsp file and then call it from a build event in Visual Studio. But while this method provides a quick and easy way to automate running commands from the Visual Studio build process, it is more flexible and more maintainable to use custom build targets instead.

The STSDEV utility was designed to create Visual Studio solutions that allow you to run many SharePoint-specific command sequences right from within the Visual Studio development environment. Whenever the STSDEV utility creates a new project, it does so with several different configurations and adds an MSBuild custom build target to each configuration. That makes it possible to simply switch between configurations and then run the standard Visual Studio build command to execute a specific build target. This target contains a common sequence of commands used in SharePoint development, such as using the MAKECAB.EXE utility to build a solution package and then using the STSADM.EXE utility to install and deploy the solution package in the local farm.

Getting Started with STSDEV

Let's start off with a quick walkthrough of how the STSDEV utility works. You can begin by running STSDEV without passing any parameters. The utility brings up a dialog that prompts the user to select the type of SharePoint development solution to be generated (see Figure 2).

Figure 2 Creating a Visual Studio Solution with STSDEV
Figure 2 Creating a Visual Studio Solution with STSDEV(Click the image for a smaller view)
Figure 2 Creating a Visual Studio Solution with STSDEV
Figure 2 Creating a Visual Studio Solution with STSDEV (Click the image for a larger view)

The STSDEV utility was designed with extensibility in mind. Each of the available solution types is basically a solution provider class that implements a very simple STSDEV-specific interface named ISolutionProvider:

   public interface ISolutionProvider {
     string Title { get; }
     void InitializeSolution();
     void AddSolutionItems();
   }

The goal behind this design is that advanced SharePoint developers can easily extend STSDEV by creating their own solution providers, which can then be used to generate Visual Studio projects for a particular scenario or to meet the standards and guidelines of a particular company.

However, every solution and project created using STSDEV will always share a common structure and a standard set of deployment files that are used to generate the output solution package. The STSDEV utility also creates each Visual Studio solution with a standard set of configurations and custom build targets that allow developers to automate many of the operations needed during development such as installing, deploying, or retracting the current project's solution package.

To get a better idea, let's examine the structure of a HelloWorld solution generated using the Empty Solution provider. As you can see from Figure 3, STSDEV generates a solution file, named HelloWorld.sln, and a project file, HelloWorld.csproj, that can be created using either the Visual Studio 2005 solution format or the Visual Studio 2008 solution format. Also keep in mind that the STSDEV utility creates Visual Studio projects using the standard Class Library Project type, which makes it easy to upgrade a Visual Studio 2005 project to Visual Studio 2008.

Figure 3 HelloWorld Project Structure
Figure 3 HelloWorld Project Structure

Using its standard project structure, the STSDEV utility creates a folder named DeploymentFiles and another folder named RootFiles. The DeploymentFiles folder contains four files that are used to build, deploy, and debug the output solution package:

Manifest.xml The solution package manifest required by SharePoint that must be built into the output solution package file (for example, HelloWorld.wsp).

SolutionPackage.ddf A diamond definition file used as input to the MAKECAB.EXE utility to build the solution package file.

SolutionConfig.xml A file used by STSDEV to track additional solution metadata required to regenerate manifest.xml and SolutionPackage.ddf. This metadata includes a GUID for Solution ID, the output solution package name, and optionally additional assembly deployment information, such as SafeControl entries and custom Code Access Security (CAS) settings.

Microsoft.SharePoint.targets A file containing custom build targets in MSBuild format that are used in SharePoint development.

While the DeploymentFiles folder always contains these four files, the contents of the RootFiles folder will vary depending on what type of solution you selected. The RootFiles folder initially contains no files or folders when you create an empty solution. However, the RootFiles folder is where you add each template file that needs to be deployed within the RootFile directory of each front-end Web server.

Each time you add a new file into the RootFiles folder, it will require updates to the manifest.xml file as well as SolutionPackage.ddf. However, the STSDEV utility provides the convenience of updating these files for you each time you rebuild the project. It does so with the RefreshDeploymentFiles command, which regenerates the manifest.xml and SolutionPackage.ddf files by enumerating through the folders and files it finds within the RootFiles directory.

Let's look at an example. When you create an empty solution, the manifest.xml file looks like this:

<Solution
  SolutionId="24F91DED-8BA7-4633-8BA0-4C9B2A4387D7"
  ResetWebServer="True"
  xmlns="http://schemas.microsoft.com/sharepoint/" >

</Solution>

Likewise, the SolutionPackage.ddf file looks like this:

; Generated by STSDEV at 1/7/2008 8:42:21 AM

.OPTION EXPLICIT
.Set CabinetNameTemplate=HelloWorld.wsp
.set DiskDirectoryTemplate=CDROM
.Set CompressionType=MSZIP
.Set UniqueFiles=off
.Set Cabinet=on
.Set DiskDirectory1=DeploymentFiles

DeploymentFiles\manifest.xml

However, what happens if you add some new folders within the RootFiles folder and copy an image file, such as AfricanPith32.gif, into it? The next time you run the Visual Studio build command, there is a custom build target to call into the STSDEV utility and execute the RefreshDeploymentFiles command. The manifest.xml is then updated to look like this:

<Solution
  SolutionId="24F91DED-8BA7-4633-8BA0-4C9B2A4387D7"
  ResetWebServer="True"
  xmlns="http://schemas.microsoft.com/sharepoint/" >
  <!--TEMPLATE files-->
  <TemplateFiles>
    <TemplateFile
      Location="IMAGES\HelloWorld\AfricanPith32.gif" />
  </TemplateFiles>
</Solution>

And the SolutionPackage.ddf file is updated to include this:

...
;*** Solution manifest
DeploymentFiles\manifest.xml

.Set DestinationDir=IMAGES\HelloWorld
RootFiles\TEMPLATE\IMAGES\HelloWorld\AfricanPith32.gif

While the RefreshDeploymentFiles command is able to determine what files to add to manifest.xml and SolutionPackage.ddf, there is still a need to track additional metadata about the solution itself and any assemblies that should be deployed inside the solution package. Therefore, STSDEV has been designed to generate an XML metadata file named SolutionConfig.xml and to read the contents of this file any time it regenerates manifest.xml and SolutionPackage.ddf.

The most basic version of a SolutionConfig.xml file looks like the following:

<Solution>  <SolutionId>24F91DED-8BA7-4633-8BA0-4C9B2A4387D7</SolutionId>
  <SolutionName>HelloWorld</SolutionName>
  <ResetWebServer>True</ResetWebServer>
  <AssemblyDeployment>False</AssemblyDeployment>
</Solution>

As you can see, the SolutionConfig.xml file tracks the GUID with the solution ID, the solution name, and a Boolean value named ResetWebServer that is used add the instruction in manifest.xml that controls whether WSS will run IISRESET on each front-end Web server after the output solution package has been deployed, upgraded, or retracted.

SolutionConfig.xml also contains an element named AssemblyDeployment that is read by the RefreshDeploymentFiles command. This is used to add configuration about one or more of the assemblies that need to be deployed in the output solution package along with the required assembly configuration information.

For example, if you created the HelloWorld project by selecting Empty Solution (C# assembly), then the initial state of the SolutionConfig.xml file would look like the one shown in Figure 4. The binary file and metadata information would then be added into manifest.xml and SolutionPackage.ddf for assembly deployment.

Figure 4 Deploying an Assembly with SolutionConfig.xml (Close Figure 4)
<!-- SolutionConfig.xml file for an Empty Solution with C# assembly -->
<Solution>
  <SolutionId>24F91DED-8BA7-4633-8BA0-4C9B2A4387D7</SolutionId>
  <SolutionName>HelloWorld</SolutionName>
  <ResetWebServer>True</ResetWebServer>
  <AssemblyDeployment>True</AssemblyDeployment>
  <!--Assembly files-->
  <Assemblies>
    <Assembly Location="HelloWorld.dll"
              DeploymentTarget="GlobalAssemblyCache">
      <SafeControls>
        <SafeControl Assembly="HelloWorld, [full 4-part assembly name]"
                     Namespace="HelloWorld"
                     TypeName="*"
                     Safe="True" />
      </SafeControls>
    </Assembly>
  </Assemblies>
</Solution>

Creating Custom Build Targets with MSBuild

Instead of using batch files, STSDEV leverages the capabilities of MSBuild to create custom build targets that can be integrated into Visual Studio solutions. In particular, STSDEV creates a file named Microsoft.SharePoint.targets in the DeploymentFiles folder that contains custom build targets. This makes it possible to create each new solution with a complete set of commands commonly used in SharePoint development. Figure 5 provides a list of the custom build targets generated for each new project.

Figure 5 Custom Build Targets (Close Figure 5)

Target | Description
DebugBuild | After building the debug version of the project's output assembly DLL, this command refreshes the solution's deployment files and then builds the output solution package .wsp file using MAKECAB.EXE.
DebugInstall | This command installs the solution package .wsp file into the solution package store of the local farm using the STSADM –addsolution operation.
DebugDeploy | This command deploys the solution package in the farm using the STSADM –deploysolution operation.
DebugUpgrade | This command upgrades the solution package using the STSADM –upgradesolution operation.
DebugQuickCopy | This command performs a simply copy command to copy the contents of the solution's RootFiles folder into the WSS RootFiles directory of the local Web server.
DebugRetract | This command retracts the solution package from the farm using the STSADM –retractsolution operation.
DebugDelete | This command deletes the solution package .wsp file from the solution package store of the local farm using the STSADM –deletesolution operation.
ReleaseBuild | After building the release version of the project's output assembly DLL, this command refreshes the solution's deployment files and then builds the output solution package .wsp file using MAKECAB.EXE.

Note that the use of custom build targets brings up security issues. Visual Studio warns you whenever you open a project or solution with custom build targets to protect developers against unexpected attacks. Therefore, when you open a project created with the STSDEV utility for the first time, you should expect to see the security warning.

This is a good time for me to drill into the contents of the Microsoft.SharePoint.targets file. If you open this file, you should see XML formatted in a structure similar to that shown in Figure 6. As you can see, MSBuild requires a top-level Project element. Within the Project element there are child elements such as PropertyGroup and Target.

Figure 6 Microsoft.SharePoint.targets (Close Figure 6)
<?xml version="1.0" encoding="utf-8" ?>

<Project DefaultTargets="DebugBuild"
  xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>
    <PackageName>HelloWorld.wsp</PackageName>
    <PackageFile>HelloWorld.wsp</PackageFile>
    <WssRootFilesFolder> <!--RootFiles path--> </WssRootFilesFolder>
    <MAKECAB> <!-- MAKECAB.EXE path--> </MAKECAB>
    <STSADM> <!-- STSADM.EXE path--> </STSADM>
    <STSDEV> <!—STSDEV.EXE path--> </STSDEV>
  </PropertyGroup>

<Target Name="DebugBuild">
    <!—custom build target 1 -->
  </Target>

  <Target Name="DebugInstall" DependsOnTargets="DebugBuild">
    <!—custom build target 2 -->
  </Target>
</Project>

Note that the PropertyGroup section at the top of Microsoft.SharePoint.targets is used to track metadata and values like the path and name of files that are needed within individual target commands. You can see that Microsoft.SharePoint.target tracks the file name of the output solution package as well as the fully qualified file path to utilities such as MAKECAB and STSADM. These is also a property that tracks the local file path to STSDEV itself so that command targets are able to call into the STSDEV utility and execute the RefreshDeploymentFiles command.

Now let's look at a few of the build targets. I will start by examining the commands defined inside the build target named DebugBuild, shown in Figure 7.

Figure 7 DebugBuild Build Target (Close Figure 7)
<Target Name="DebugBuild">
  <Message Text="Refreshing Deployment Files..." Importance="high" />
  <Exec Command="$(STSDEV) /refresh $(TargetName) $(SolutionDir)"
    ContinueOnError="false" />
  <Message Text="Deleting Solution Package File..." Importance="high" />
  <Delete Files="$(ProjectDeploymentFilesFolder)\$(PackageFile)"
    ContinueOnError="true" />
  <Message Text="Building Solution Package (Debug Version)"
    Importance="high" />
  <Exec Command="$(MAKECAB) /F
    $(ProjectDeploymentFilesFolder)\SolutionPackage.ddf /D
    CabinetNameTemplate=$(PackageFile)" ContinueOnError="false" />
</Target>

As you can see, the first command inside the DebugBuild target is the one that calls into STSDEV and passes the /refresh parameter to run the RefreshDeploymentFiles command. This causes the STSDEV utility to regenerate manifest.xml and SoutionPackage.ddf as the first part of the solution package build process. After that, there is an MSBuild command executed to delete any existing .wsp file and to rebuild the solution package using the MAKECAB.EXE utility. You should also note the use of Message commands, which are used to display informational messages to the developer inside of the Visual Studio output window while the commands are running.

The DebugInstall command has been written to use the addsolution operation supplied by STSADM to install the newly generated solution package .wsp file into the solution package store of the local farm. Also note the use of the DependsOnTargets attribute, which will first run the DebugBuild target commands anytime the DebugInstall target is executed:

<Target Name="DebugInstall" DependsOnTargets="DebugBuild">
<Message Text="Installing Solution..." Importance="high" />
  <Exec Command="$(STSADM) -o addsolution -filename
    $(ProjectDeploymentFilesFolder)\$(PackageFile)"
    ContinueOnError="true" />
  <Exec Command="$(STSADM) -o execadmsvcjobs" />
  <Message Text="" Importance="high" />
</Target>

Many of the operations exposed by the STSADM utility are run asynchronously using timer jobs. Therefore, there is another call to STSADM to run the execadmsvcjobs operation. This operation causes the command to pause and wait until the local SharePoint Web server has been able to run all pending timer jobs to completion. The execadmsvcjobs operation is also called in the DebugDeploy target directly after calling the deploysolution operation:

<Target Name="DebugDeploy" DependsOnTargets="DebugInstall">
  <Message Text="Deploying Solution..." Importance="high" />
  <Exec Command="$(STSADM) -o deploysolution -name
    $(PackageName) -local -allowGacDeployment -force" />
  <Exec Command="$(STSADM) -o execadmsvcjobs" />
  <Message Text="" Importance="high" />
</Target>

Now that you have seen the different custom build targets that exist within the Microsoft.SharePoint.targets file generated by the STSDEV utility, you need to know how to execute them from within the Visual Studio environment. It's very simple. You simply select the Visual Studio configuration associated with the desired build target and run the standard Visual Studio build command, as shown in Figure 8.

Figure 8 Executing an STSDEV-Generated Build Target
Figure 8 Executing an STSDEV-Generated Build Target(Click the image for a smaller view)
Figure 8 Executing an STSDEV-Generated Build Target
Figure 8 Executing an STSDEV-Generated Build Target (Click the image for a larger view)

The MSBuild technique that is being used to wire up these custom build targets requires adding a Target element into the Visual Studio project file (for example, HelloWorld.csproj). This is something that STSDEV does automatically anytime you create a new solution. If you open up one of the .csproj files and look at the bottom, you will see a Target section that looks like Figure 9.

Figure 9 Targets in a Visual Studio Project (Close Figure 9)
<Target Name="AfterBuild">
  <CallTarget Targets="DebugBuild"
   Condition="'$(Configuration)|$(Platform)' == 'DebugBuild|AnyCPU'"/>
  <CallTarget Targets="DebugInstall"
   Condition="'$(Configuration)|$(Platform)' == 'DebugInstall|AnyCPU'"/>
  <CallTarget Targets="DebugDeploy"
   Condition="'$(Configuration)|$(Platform)' == 'DebugDeploy|AnyCPU'"/>
  <CallTarget Targets="DebugRedeploy"
   Condition="'$(Configuration)|$(Platform)' == 'DebugRedeploy|AnyCPU'"/>
  <CallTarget Targets="DebugUpgrade"
   Condition="'$(Configuration)|$(Platform)' == 'DebugUpgrade|AnyCPU'"/>
  <CallTarget Targets="DebugQuickCopy"
   Condition="'$(Configuration)|$(Platform)' == 'DebugQuickCopy|AnyCPU'"/>
  <CallTarget Targets="DebugRetract"
   Condition="'$(Configuration)|$(Platform)' == 'DebugRetract|AnyCPU'"/>
  <CallTarget Targets="DebugDelete"
   Condition="'$(Configuration)|$(Platform)' == 'DebugDelete|AnyCPU'"/>
  <CallTarget Targets="ReleaseBuild"
   Condition="'$(Configuration)|$(Platform)' == 'ReleaseBuild|AnyCPU'"/>
</Target>

You should notice that the Target section has a name of AfterBuild. This is a well-known target name that Visual Studio automatically runs at the completion of every build. This makes it easy for you to add one CallTarget command per build target with a condition to execute the appropriate custom build target depending on the configuration that has been selected.

Extending the STSDEV Utility

The STSDEV utility ships with a handful of sample solution providers. However, the real potential is in extending STSDEV by adding your own solution provider. If you would like to try this, open the source code for the STSDEV project that accompanies this month's column and examine the class implementation for some of the existing solution providers.

If you open up source files such as SimpleFeatureSolutionProvider.cs or WebPartSolutionProvider.cs and walk through the code in the InitializeSolution and AddSolutionItems methods, you can see that it's pretty easy to build a solution provider. There is a class named SolutionBuilder that contains a public static method used to create many of the common files needed in a SharePoint development project. You can also see that some solution providers (such as the one in SimpleFeatureSolutionProvider.cs) interact with the user to prompt for additional configuration items using a modal dialog box created using Windows Forms.

I hope you find the STSDEV utility both educational and useful in your everyday life as a SharePoint developer. This utility is a product of the SharePoint developer community and the latest version of its source code will always be available on the CodePlex Web site at codeplex.com/stsdev. At the STSDEV site on CodePlex you will find other resources such as an introductory set of screencasts to get you up and running quickly with this utility. Please feel free to provide feedback and to post examples of any solution providers that you create in your own SharePoint development efforts.

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. His new book is Inside Windows SharePoint Services 3.0. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group (www.TedPattison.net).


Test Run: Web UI Automation with Windows PowerShell

Test Run
Web UI Automation with Windows PowerShell
Dr. James McCaffrey

Code download available at: TestRun2008_03.exe (155KB)
Contents
The Web App
Test Automation Scripting with Windows PowerShell
Taking It a Step Further

W
indows PowerShellTM, the new Microsoft command shell and scripting language, is a great platform for several kinds of lightweight test automation. In this month's Test Run column, I show you how to use Windows PowerShell to create quick and easy UI test automation for any kind of Web application by automating Internet Explorer®. This column is primarily intended for beginners, but experienced engineers will find some interesting information here, too.

Before you start, make sure you have added whatever sites you are going to test to your "Trusted Sites" list in Internet Explorer—otherwise the scripts may not work. I begin my automation demonstration by issuing the Windows PowerShell command:

PS C:\> $ie = new-object -com "InternetExplorer.Application"

This creates an instance of the classic InternetExplorer COM automation object from the SHDocVw.dll library. The new-object keyword is a Windows PowerShell cmdlet (pronounced command-let). There are approximately 130 cmdlets, and they form the heart of Windows PowerShell functionality. You can explore what the cmdlets are by running get-command and you can always get help on a cmdlet by running get-help Command. I supply a -com switch (which is actually a shortcut for -comObject) to new-object. This specifies that I am instantiating a classic COM object using the object's ProgID rather than instantiating a managed object.

I store the resulting object into the $ie variable (all Windows PowerShell variables are preceded by a $ character, making them easy to distinguish from other token types). Next, I use the Navigate method to load my dummy MiniCalc Web application under test into my browser automation object:

PS C:\> $ie.navigate("http://localhost/MiniCalc/Default.aspx")
PS C:\> $ie.visible = $true

One of the great features of Windows PowerShell is that it can help you explore an object's capabilities. With older scripting technologies, if I didn't know that the InternetExplorer object has a Navigate method, I'd be forced to look that information up using some sort of external reference. But with Windows PowerShell, I have several quick ways to see the available methods and properties for an object. For example, I can use tab completion by typing "$ie." and then pressing the Tab key. After each key press, an available property or method will be displayed. I can also use the get-member cmdlet to get a list of all available properties and methods, and their signatures:

PS C:\> $ie | get-member | more

Another Windows PowerShell discovery capability is command completion. For instance, I can type "$ie.vi" and then press the Tab key, and Windows PowerShell will finish typing my ie.visible statement for me. These discovery features in Windows PowerShell are huge time savers.

Next, I obtain references to all input controls on my application under test:

PS C:\> $doc = $ie.document
PS C:\> $tb1 = $doc.getElementByID("TextBox1")
PS C:\> $tb2 = $doc.getElementByID("TextBox2")
PS C:\> $add = $doc.getElementByID("RadioButton1")
PS C:\> $btn = $doc.getElementByID("Button1")

I use the Document property to fetch the active document and then use the getElementById method to obtain references to each control. Notice that for this technique to work, all of my HTML elements must have an ID value. In most situations this is not a problem. If you create your Web application with Visual Studio®, all controls automatically receive IDs. If you are writing test automation for a Web application where the elements do not have IDs, you can also use the getElementsByTagName method to return a collection of elements and then access a specific element by index.

Next, my automation simulates some user input:

PS C:\> $tb1.value = 5
PS C:\> $tb2.value = 7
PS C:\> $add.checked = $true
PS C:\> $btn.click()

Notice that although the TextBox1 and TextBox2 values referenced by $tb1 and $tb2 are string types, I can omit quotes because Windows PowerShell will correctly infer the correct data type for me even though I could have typed the command as:

PS C:\> $tb1.value = "5"

I finish my interactive automation by examining the resulting state of the MiniCalc Web application:

PS C:\> $tb3 = $doc.getElementById("TextBox3")
PS C:\> $ans = $tb3.value
PS C:\> if ($ans -eq "12.0000") { 'Pass' } else { '*FAIL*' }

I first get a reference to the TextBox3 control. I did not do this earlier because, after calling the $btn click method, an HTTP request is sent to the Web server and a new page with a new TextBox3 value is generated and returned to the client browser. For brevity, I can simply type 'Pass' instead of write-host 'Pass' because the default Windows PowerShell action for a string value is to output the value to the host.

In the following sections of this month's column, I briefly describe the dummy MiniCalc Web application under test so you'll know exactly what is being tested. Then I refactor the interactive commands shown in Figure 1 to a more practical Windows PowerShell script and demonstrate how to deal with tricky timing issues. I conclude by examining some pros and cons of using Windows PowerShell for Web UI automation compared to alternative approaches such as using a commercial test framework, writing a custom C# harness with Visual Studio, and writing custom automation that uses a JavaScript approach. I think you'll find the techniques I present here to be an extremely useful and valuable addition to your software testing toolkit.

Figure 1 Web App UI Automation with Windows PowerShell
Figure 1 Web App UI Automation with Windows PowerShell(Click the image for a smaller view)
Figure 1 Web App UI Automation with Windows PowerShell
Figure 1 Web App UI Automation with Windows PowerShell (Click the image for a larger view)

The Web App

Let's begin by examining the MiniCalc ASP.NET Web app shown in the background of Figure 1. This is the target of my UI test automation. MiniCalc is a simple ASP.NET Web app. The techniques I present in this month's column can automate ASP.NET Web applications, classic ASP Web applications, and applications created with technologies such as PHP and Ruby.

To keep my Web application code as short and as simple as possible, I coded the app in Notepad and placed both logic code and display code in a single file. The entire code is listed in Figure 2. (All the code discussed in this column is available on the MSDN® Magazine Web site.) I name my Web application Default.aspx and placed the app at Web site http://localhost/MiniCalc.

Figure 2 Web Application under Test (Close Figure 2)
<%@ Page Language="C#" %>
<script runat="server">
  private void Button1_Click(object sender, System.EventArgs e)
  {
    int alpha = int.Parse(TextBox1.Text.Trim());
    int  beta = int.Parse(TextBox2.Text.Trim());

    System.Threading.Thread.Sleep(3000);

    if (RadioButton1.Checked) {
      TextBox3.Text = Sum(alpha, beta).ToString("F4");
    }
    else if (RadioButton2.Checked) {
      TextBox3.Text = Product(alpha, beta).ToString("F4");
    }
    else
     TextBox3.Text = "Select method";
  }

  private static double Sum(int a, int b) {
    double ans = a + b;
    return ans;
  }

  private static double Product(int a, int b) {
    double ans = a * b;
    return ans;
  }
</script>

<html>
  <head>
    <style type="text/css">
     fieldset { width: 16em }
     body { font-size: 10pt; font-family: Arial }
    </style>
    <title>Default.aspx</title>
  </head>
  <body bgColor="#ccffff">
    <h3>MiniCalc by ASP.NET</h3>
    <form method="post" name="theForm" id="theForm"
      runat="server" action="Default.aspx">
      <p>
        <asp:Label id="Label1" runat="server">
          Enter integer:&nbsp&nbsp
        </asp:Label>
        <asp:TextBox id="TextBox1" width="100" runat="server" />
      </p>
      <p>
        <asp:Label id="Label2" runat="server">
          Enter another:&nbsp
        </asp:Label>
        <asp:TextBox id="TextBox2" width="100"  runat="server" />
      </p>
      <p></p>
      <fieldset>
        <legend>Arithmentic Operation</legend>
        <p>
          <asp:RadioButton id="RadioButton1"
            GroupName="Operation" runat="server"/>
            Addition
        </p>
        <p>
          <asp:RadioButton id="RadioButton2"
            GroupName="Operation" runat="server"/>
            Multiplication
        </p>
        <p></p>
      </fieldset>
      <p>
        <asp:Button id="Button1" runat="server"
          text=" Calculate " onclick="Button1_Click" />
      </p>
      <p>
        <asp:TextBox id="TextBox3" width="120"  runat="server" />
      </p>
    </form>
  </body>
</html>

To perform Web application UI automation, you must know the IDs of the user controls you wish to manipulate and examine. If you look over the code in Figure 2, you'll see that I use Visual Studio-style default IDs for my controls. I use the IDs TextBox1 and TextBox2 for the two textbox controls that will hold the integers entered by the user:

<asp:TextBox id="TextBox1" width="100" runat="server" />

I use RadioButton1 as the ID for the RadioButton control, which allows the user to select either Addition or Multiplication; Button1 as the ID for the Button control, which causes the Web app to add or multiply the values in TextBox1 and TextBox2; and I use TextBox3 as the ID for the textbox control, which holds the result. All my logic is contained in the Web app's Button1_Click method, which handles the Button1 click event.

I begin by fetching the user input and converting from string to type int, as shown here:

int alpha = int.Parse(TextBox1.Text.Trim());
int  beta = int.Parse(TextBox2.Text.Trim());
System.Threading.Thread.Sleep(3000);

After capturing the user input, I insert a Thread.Sleep statement in order to simulate some processing time, such as that which would occur in a real Web application that accesses a back-end database. As you'll see shortly, dealing with unpredictable HTTP response times is the most difficult part of writing lightweight UI test automation for Web applications.

After I have the two user-supplied integers I need, I check which operation (addition or subtraction) the user wants, compute the result and place that result into the TextBox3 control:

if (RadioButton1.Checked) {
  TextBox3.Text = Sum(alpha, beta).ToString("F4");
}
else if (RadioButton2.Checked) {
  TextBox3.Text = Product(alpha, beta).ToString("F4");
}
else
 TextBox3.Text = "Select method";
}

I implicitly cast my answer to type double, and when I place the result in the TextBox3 control, I format to four decimal places by using an "F4" argument to the ToString method.

Test Automation Scripting with Windows PowerShell

Although the interactive test automation with Windows PowerShell described in the first section of this column can be useful in many testing situations, you will often want to create test automation scripts. Figure 3 shows one way to test the MiniCalc Web application. Figure 4 shows the script in action.

Figure 3 UI Test Automation with Windows PowerShell (Close Figure 3)
# file: testScript.ps1

function main()
{
  write-host "`nBegin test automation using Windows PowerShell`n"

  write-host 'Launching IE'
  $ie = new-object -com "InternetExplorer.Application"
  $ie.navigate("about:blank")
  $ie.visible = $true
  [System.Threading.Thread]::Sleep(2000)

  write-host "`nResizing IE to 425 x 535"
  $ie.height = 535
  $ie.width = 425

  write-host "`nNavigating to MiniCalc Web application"
    navigateToApp $ie "http://localhost/MiniCalc/Default.aspx"`
      "TextBox1" 100 2

  write-host "`nGetting input controls"
  $doc = $ie.document
  $tb1 = $doc.getElementByID("TextBox1")
  $tb2 = $doc.getElementByID("TextBox2")
  $add = $doc.getElementByID("RadioButton1")
  $btn = $doc.getElementByID("Button1")

    if ($tb1 -eq $null -or $tb2 -eq $null -or $add -eq $null -or $btn`
      -eq $null) {
    write-host "One or more controls are null" -backgroundcolor "red"`
      -foregroundcolor "yellow"
  }
  else {
    write-host "All controls found"
  }

  write-host "`nSetting TextBox1 to 5"
  $tb1.value = 5
  write-host "Setting TextBox2 to 7"
  $tb2.value = 7

  write-host "Selecting 'Addition' operation"
  $add.checked = $true

  write-host "`nClicking 'Calculate' button"
  $tb3 = $doc.getElementByID("TextBox3")
  $old = $tb3.value
  $btn.click()
  $wait = $true
  $numWaits = 0
  while ($wait -and $numWaits -lt 100) {
    $numWaits++
    [System.Threading.Thread]::Sleep(50)
    $tb3 = $doc.getElementByID("TextBox3")
    if ($tb3.value -ne $old) {
      $wait = $false
    }
    else {
      write-host "Waiting for app to respond $numWaits . . ."
    }
  }
  if ($numWaits -eq 100) {
    throw "Application did not respond after 100 delays"
  }
  else {
    write-host "Application has responded"
  }

  write-host "`nChecking for 12.0000"
  $tb3 = $doc.getElementByID("TextBox3")
  $ans = $tb3.value

  if ($ans -eq '12.0000') {
    write-host "Target value found"
    write-host "`nTest scenario: Pass" -foregroundcolor 'green'
  }
  else {
    write-host "Target value not found"
    write-host "`nTest scenario: *FAIL*" -foregroundcolor 'red'
  }

  trap {
    write-host "Fatal error: " $_.exception.message -backgroundcolor red`
      -foregroundcolor yellow
  }

  write-host "`nEnd test automation`n"
} # main

function navigateToApp($browser, [string] $url, [string] $controlID,`
  [int] $maxDelays, [int] $delayTime)
{
  $numDelays = 0
  $loaded = $false
  $browser.navigate($url)
  while ($loaded -eq $false -and $numDelays -lt $maxDelays) {
    $numDelays++
    [System.Threading.Thread]::Sleep($delayTime)
    $doc = $browser.document
    if ($doc -eq $null) {
      continue
    }
    $controlRef = $doc.getElementByID($controlID)
    if ($controlRef -eq $null) {
      write-host "Waiting for Web app to load $numDelays . . ."
    }
    else {
      write-host "Web app loaded after $numDelays pauses"
      $loaded = $true
    }
  }

  if ($numDelays -eq $maxDelays) {
    throw "Browser not loaded after $maxDelays delays"
  }
}

main
# end script
Figure 4 Executing Test Automation with a Script
Figure 4 Executing Test Automation with a Script(Click the image for a smaller view)
Figure 4 Executing Test Automation with a Script
Figure 4 Executing Test Automation with a Script (Click the image for a larger view)

If you examine Figure 4, you'll see that after I launch a Windows PowerShell shell, I verify that my Windows PowerShell session's execution policy allows script execution and then invoke my script. Notice that under Windows PowerShell I must specify the path to the script (.\ if the script is in the current directory) even when the script is in the current directory.

The overall structure of the test script is:

# file: testScript.ps1
function main
{
  # code
}

function navigateToApp($browser, [string] $url,
 [string] $controlID, [int] $maxDelays,
 [int] $delayTime)
{
  # code
}

main
# end script

I call my main function main, but there is no default Windows PowerShell script entry point, so I could have named this function anything. After defining an auxiliary navigateToApp function, I issue the single statement main to launch my script's execution. The first few lines of the main function are:

write-host "`nBegin test automation using Windows PowerShell`n"
write-host 'Launching IE'
$ie = new-object -com "InternetExplorer.Application"
$ie.navigate("about:blank")
$ie.visible = $true
[System.Threading.Thread]::Sleep(2000)

My first two write-host statements show how, in Windows PowerShell, double quotes are intelligent in the sense that certain escape sequences, such as the 'n newline character and object references beginning with the $ character, are evaluated by the script execution engine. Single-quote-delimited strings are interpreted literally.

As explained earlier, I use the new-object cmdlet to instantiate an instance of a classic InternetExplorer COM automation object. I then navigate to the about:blank page and make my browser visible. I call directly into the Microsoft® .NET Framework to access the Thread.Sleep method and pause my test automation for two seconds. There are two points here. First, the ability to directly call into the .NET Framework is a key advantage of Windows PowerShell over most other scripting technologies. Second, although sleeping the thread in order to give the Web application under test a chance to respond works, there is a much better approach, which I will describe shortly.

The next few lines of my automation script set the Internet Explorer browser to a known state:

write-host "`nResizing IE to 425 x 535"
$ie.height = 535
$ie.width = 425

In general, when performing most types of Web application UI test automation, it's a good idea to set characteristics of the browser to a known state so that any bugs that are revealed by the automation can be observed more easily. For example, consider a Web app that is a front end to a SQL Server® data store and has a search functionality that displays results in a listbox control. Suppose the app's logic forgets to clear the listbox control after a particular search. Now starting from the standard default state, which will have an empty results listbox, test automation might determine that the Web app works correctly and not reveal the bug. But if you start the automation from a state that already contains some results in the listbox, the new results would be concatenated with the old results and the bug would be revealed. You should also write scripts that allow variations in initial state to aid in uncovering bugs that are sensitive to those variations.

Next, I point my browser to the MiniCalc Web application under test:

write-host "`nNavigating to MiniCalc Web application"
navigateToApp $ie "http://localhost/MiniCalc/Default.aspx" "TextBox1" 100 80

I invoke the navigateToApp helper function. You can interpret this call to mean "navigate Internet Explorer to URL http://localhost/MiniCalc/Default.aspx and then wait until a reference to an HTML element with ID equal to TextBox1 is accessible, pausing 80 milliseconds between attempts to access TextBox1, up to a maximum of 100 attempts." Determining when your application is loaded is not trivial. The crude alternate to load my Web app is simple but generally not as effective:

$ie.navigate("http://localhost/MiniCalc/Default.aspx")
 [System.Threading.Thread]::Sleep(8000)

The two problems with this approach are that there is no good way to predict how long to pause your automation, and there's no clear way to deal with a situation where the application under test does not load within the allotted time. The navigateToApp function solves both these problems and is listed in Figure 5.

Figure 5 NavigateToApp Helper Function (Close Figure 5)
function navigateToApp($browser, [string] $url, [string] $controlID,
  [int] $maxDelays, [int] $delayTime)
{
  $numDelays = 0
  $loaded = $false
  $browser.navigate($url)
  while ($loaded -eq $false -and $numDelays -lt $maxDelays) {
    $numDelays++
    [System.Threading.Thread]::Sleep($delayTime)
    $doc = $browser.document
    if ($doc -eq $null) {
      continue
    }
    $controlRef = $doc.getElementByID($controlID)
    if ($controlRef -eq $null) {
      write-host "Waiting for Web app to load $numDelays . . ."
    }
    else {
      write-host "Web app loaded after $numDelays pauses"
      $loaded = $true
    }
  }

  if ($numDelays -eq $maxDelays) {
    throw "Browser not loaded after $maxDelays delays"
  }
}

The navigateToApp function essentially goes into a delay loop, checking in each iteration to see whether a specified user control reference is available. The loop also will exit if some maximum number of iterations through the loop is exceeded to prevent an infinite loop condition.

Although Windows PowerShell is object-based, it is considered acceptable to refer to simple objects as variables. You can see that the navigateToApp function uses local variables $numDelays and $loaded, but they don't have to be explicitly declared to be local variables. I could have preceded these variables with the $private: qualifier to make their local scope explicit or used the $global: qualifier to make the values of these variables available outside the scope of the function. The control logic in navigateToApp checks if the document object is available; if not, I use the continue statement to short-circuit out of the current loop iteration and then try again after a delay. If I do have a valid reference to the document object, then I attempt to get a reference to a target element.

I also could have used the Windows PowerShell elseif control structure. Additionally, instead of using an explicit $loaded variable, I could have used the Windows PowerShell break statement to exit the delay loop. Windows PowerShell has a rich set of control structures that allow you to program in many different styles, including whatever programming style you are accustomed to, and this speeds up your learning curve.

After exiting the delay loop, I check to see whether the loop exited because of exceeding the maximum number of delays (which means I never obtained a reference to a target user control, signaling that the Web application never loaded successfully). In this case, I throw an exception that I will catch in the main function by using the Windows PowerShell trap mechanism.

After the Web application under test is fully loaded, I obtain references to all user input controls:

write-host "`nGetting input controls"
$doc = $ie.document
$tb1 = $doc.getElementByID("TextBox1")
$tb2 = $doc.getElementByID("TextBox2")
$add = $doc.getElementByID("RadioButton1")
$btn = $doc.getElementByID("Button1")

As I mentioned earlier, the getElementById method requires that all controls have an ID attribute, but in situations where you need to access controls without IDs, you can also use the getElementsByTagName method.

Next, I perform a quick check to make sure my HTML element references are valid:

if ($tb1 -eq $null -or $tb2 -eq $null –or
    $add -eq $null -or $btn -eq $null) {
  write-host "One or more controls are null"
    -backgroundcolor "red" -foregroundcolor "yellow"
}
else {
    write-host "All controls found"
}

When writing Windows PowerShell-based UI test automation, it is generally a matter of personal coding style whether to throw an exception or to simply display a message using the write-host cmdlet when you error check. Conceptually, throwing an exception is the more logical approach, but using write-host allows you to specify easily visible text using the –backgroundcolor and –foregroundcolor arguments.

After I have verified that all user input controls are available, I can easily manipulate them, like so:

write-host "`nSetting TextBox1 to 5"
$tb1.value = 5
write-host "Setting TextBox2 to 7"
$tb2.value = 7
write-host "Selecting 'Addition' operation"
$add.checked = $true

Now I am ready to simulate the user action that will trigger a post to the Web server—a button-click in this case—and then wait for the response from the server. This is not so easy. Similar to loading the application under test, a simplistic approach such as the following just isn't effective because there is no reliable way to know in advance how long to pause your test automation:

write-host "`nClicking 'Calculate' button"
$btn.click()
[System.Threading.Thread]::Sleep(5000)

One of several possible solutions is to first get some prerequest control value on the Web application, then trigger the HTTP request, and then use a delay loop until the prerequest control value has changed. For example, first I can fetch the value in the TextBox3 control and save it:

$tb3 = $doc.getElementByID("TextBox3")
$old = $tb3.value

Now I can simulate a user clicking on the Calculate button:

$btn.click()

And then I can go into a delay loop until either the value in TextBox3 has changed or I exceed some maximum number of delays:

$wait = $true
$numWaits = 0
while ($wait -and $numWaits -lt 100) {
  $numWaits++
  [System.Threading.Thread]::Sleep(50)
  $tb3 = $doc.getElementByID("TextBox3")
  if ($tb3.value -ne $old) {
    $wait = $false
  }
  else {
    write-host "Waiting for app to respond $numWaits . . ."
  }
}

After my delay loop terminates, I check to see if the exit occurred because of exceeding the maximum number of attempts to find a change in the target control's value:

if ($numWaits -eq 100) {
  throw "Application did not respond after 100 delays"
}
else {
  write-host "Application has responded"
}

At this point in my automation, I have successfully loaded the Web app under test, manipulated elements, triggered an HTTP request, and determined that there has been a response from the server. Now I can check the resulting state of the Web application to determine a pass or fail result. First, I get a reference to, and the value of, TextBox3:

write-host "`nChecking for 12.0000"
$tb3 = $doc.getElementByID("TextBox3")
$ans = $tb3.value

I cannot get a reference to TextBox3 until this point because a reference obtained before the HTTP request would be lost after the HTTP response. Now I can check the resulting value and display the test scenario result:

if ($ans -eq '12.0000') {
  write-host "Target value found"
  write-host "`nTest scenario: Pass" -foregroundcolor 'green'
}
else {
  write-host "Target value not found"
  write-host "`nTest scenario: *FAIL*" -foregroundcolor 'red'
}

At the end of my main function, I use the trap statement to deal with any exceptions that may have been thrown during the test run:

trap {
  write-host "Fatal error: " $_.exception.message
  -backgroundcolor red -foregroundcolor yellow
}

Here I simply display the exception message. In some cases, you may want to use the continue statement to force your test automation to continue running even on a fatal error. Or you may want to exit your automation altogether.

Taking It a Step Further

I hardcoded many values for clarity, but you will likely want to parameterize your automation scripts in a production environment. For example, I hardcoded the path to the Web application under test. Windows PowerShell has good mechanisms for passing command-line arguments to scripts—you can add parameters to a script by adding param($param1, $param2), and so on, to the top of your script.

Additionally, you may want to extend your automation scripts by parameterizing test case input values and corresponding expected values. Again, Windows PowerShell has elegant ways to read test case data from an external flat text file, an external XML file, a SQL database, or other test case data store.

Similarly, my example test automation script writes its pass/fail result to the shell. Windows PowerShell also allows you to easily save results to any type of data store that you wish. One interesting example of this is to write your test results to a Microsoft Team Foundation Server, which gives you great test management capabilities. For details on integrating with Team Foundation Server, please see my Test Run column in the MSDN Magazine Launch 2008 issue (msdn.microsoft.com/msdnmag/issues/08/LA/TestRun).

In much the same way that test automation complements rather than replaces manual testing, ultralightweight software test automation with Windows PowerShell complements rather than replaces other types of test automation and test frameworks. For example, because the UI testing technique I've presented here uses the Internet Explorer object model, you cannot use it to test Web applications running on other Web browsers or devices. In situations like those, you can employ a JavaScript approach (see the February 2007 Test Run column, "AJAX Test Automation", at msdn.microsoft.com/msdnmag/issues/07/02/TestRun), or use a commercial test framework.

The techniques I've presented here are available to some extent in other scripting languages. But based on my experience, using Windows PowerShell for script-based test automation has five small, but significant, characteristics that give it an advantage over other scripting environments.

First, Windows PowerShell can directly access both COM objects and the .NET Framework (rather than go through a wrapper mechanism).

Second, the interactive mode of Windows PowerShell allows you to quickly experiment while developing your automation scripts, which greatly speeds up the script creation process.

Third, the built-in discovery mechanisms of Windows PowerShell, such as dot-tab completion and the get-member cmdlet, provide you with what is essentially a virtual documentation help feature.

Fourth, the Windows PowerShell built-in collection of cmdlets simplifies many mundane test automation tasks.

Fifth, in my opinion, Windows PowerShell is simply easier and more intuitive to use.

While none of these advantages by themselves are huge, when taken together these characteristics provide you with a high rate of return on your test automation with Windows PowerShell relative to the cost you pay to write your automation.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at the Microsoft Redmond, Wash., campus. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes and can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Security Briefs: Protecting Your Code with Visual C++ Defenses

Security Briefs
Protecting Your Code with Visual C++ Defenses
Michael Howard

Contents
Stack-Based Buffer Overrun Detection (/GS)
Safe Exception Handling (/SafeSEH)
DEP Compatibility (/NXCompat)
Image Randomization (/DynamicBase)
Safer Function Calls
C++ Operator::new
What if It Fails?

A
lot of code is written in C and C++ and unfortunately a lot of this code has security vulnerabilities that many developers do not know about. Programs written in any language can have vulnerabilities that leave their users open to attack, but it's the C and C++ languages that have a special place in Internet history because so many security vulnerabilities are due to the very thing that makes these two programming languages so popular: the unbridled access to computer hardware and the performance that comes with it. When you read about security and C or C++ crops up, the words "buffer" and "overrun" are usually pretty close by because buffers are typically an example of direct access to memory. This kind of direct access is very powerful—and very, very dangerous.

There are a number of reasons for the many buffer overruns in production C and C++ code. The first I have already mentioned: the languages provide direct access to vulnerable memory. Second, developers make mistakes. And third, there are normally no defenses offered by compilers. It is feasible to provide remedies for the first issue, but then C and C++ start to become different languages.

The reason for developers making mistakes could be partially addressed through education, but I really don't see the educational institutions stepping up. Sure there is a place in industry for security education, too, but we are all part of the solution or part of the problem, and I would love to see colleges doing more to educate students about software security. You're probably asking, "Why are educational institutions not attempting to teach this critically important subject?" To be honest, I have absolutely no idea. It's kind of depressing, actually.

Finally, even with an excellent education, some security issues are complex enough that even well-educated engineers won't catch everything. We humans are not perfect.

The issue about needing to build more defenses into compilers is something the Microsoft Visual C++ team has been working on and slowly improving over the years, with help from our security team. This column outlines some of the buffer overrun defenses available in Visual C++® 2005 and beyond. Note that some other compilers do offer defenses, but Visual C++ has two major advantages over the likes of gcc. First, all these defenses are in the toolset by default; there is no need to download some funky add-in. Second, the options are easy to use.

In no particular order, the defenses offered by the Visual C++ toolset are:

  • Stack-based Buffer Overrun Detection (/GS)
  • Safe Exception Handling (/SafeSEH)
  • Data Execution Prevention (DEP) Compatibility (/NXCompat)
  • Image Randomization (/DynamicBase)
  • Automatic use of safer function calls
  • C++ operator::new

Before I discuss each of these in detail I want to point out that these defenses do not compensate for insecure code. You should always strive to create the most secure code possible, and if you don't know how to do that, then run right out and read some of the very good books available on the subject.

Typical Stack Compared to One Compiled with /GS (Click the image for a smaller view)

Typical Stack Compared to One Compiled with /GS (Click the image for a larger view)

I also want to point out that these are all Security Development Lifecycle (SDL) requirements at Microsoft, which means that C and C++ code must use these options in order to ship. There are occasional exceptions, but they are relatively rare, so I won't be going into detail about them here.

Finally, you must keep in mind this important point: it is possible, depending on the code in question, to circumvent these elaborate defenses. The more defenses used by the code, the harder they are to work around, but no defense is perfect. They are all speed bumps to reduce the chance an exploit will succeed. You have been warned! The only variant is the use of safer function calls, which are real defenses and can remove vulnerabilities. Let's look at each defense in detail.

Stack-Based Buffer Overrun Detection (/GS)

Stack-based buffer overrun detection is the oldest and most well-known defense available in Visual C++. The goal of the /GS compiler flag is simple: reduce the chance that malicious code will execute correctly. The /GS option is on by default in Visual C++ 2003 and later, and it detects certain kinds of stack smash at run time. It goes about doing this by including a random number in a function's stack just before the return address on the stack, and when the function returns, function epilogue code checks this value to make sure it has not changed. If the cookie, as it's called, has changed, execution is halted.

The function prologue code that sets the cookie looks like this:

sub    esp, 8
mov    eax, DWORD PTR ___security_cookie
xor    eax, esp
mov    DWORD PTR __$ArrayPad$[esp+8], eax
mov    eax, DWORD PTR _input$[esp+4]

And shown here is the function epilogue code that checks the cookie:

mov    ecx, DWORD PTR __$ArrayPad$[esp+12]
add    esp, 4
xor    ecx, esp
call    @__security_check_cookie@4
add    esp, 8

Visual C++ 2005 also moves data around on the stack to make it harder to corrupt that data. Examples include moving buffers to higher memory than non-buffers. This step can help protect function pointers that reside on the stack, for example. Alternately, moving pointer and buffer arguments to lower memory at run time mitigates various buffer overrun attacks. See the diagram to compare a typical stack with a /GS stack.

The /GS compiler option is not applied in any of the following situations:

  • Functions do not contain a buffer.
  • Optimizations are not enabled.
  • Functions are defined to have a variable argument list.
  • Functions are marked with the naked keyword (C++).
  • Functions contain inline assembly code in the first statement.
  • The buffer isn't an 8-byte type and is less than 4 bytes in size.

An option was added to Visual C++ 2005 SP1 to make the /GS heuristics much more aggressive so that more functions would be protected. Microsoft added this because a small number of security bulletins were issued in code that had stack-based buffer overruns and the code, even though it was compiled with /GS, was not protected by a cookie. This new option increases the number of protected by functions substantially.

Use this option by placing the following line in modules where you want added protection, such as code that handles data from the Internet:

#pragma strict_gs_check(on)

This pragma is just one example of how Microsoft is constantly evolving the /GS capability. The original version in Visual C++ 2003 was pretty simple, and then it was updated in Visual C++ 2005 SP1 and again in Visual C++ 2008 as we learned about new attacks as well as new ways to circumvent existing attacks. In our analysis, we have found that /GS causes no real compatibility or performance problems.

Safe Exception Handling (/SafeSEH)

The CodeRed worm that affected Internet Information Server (IIS) 4.0 was caused by a stack-based buffer overrun. Interestingly, /GS would not have caught the problem exploited by the worm because the code did not overrun the affected function's return address; rather, it corrupted an exception handler on the stack. This is a great example of why you must constantly focus on writing secure code and not rely solely on these types of compiler-based defenses.

An exception handler is code executed when an exceptional condition occurs, such as division by zero. The address of the handler is held on the stack frame of the function and is therefore subject to corruption. The linker included with Visual Studio® 2003 and later includes an option to store the list of valid exception handlers in the image's PE header at compile time. When an exception is raised at run time, the operating system checks the image header to determine whether the exception handler address is correct; if it is not, the application is terminated. This would have prevented CodeRed if the technology existed at the time the code was linked. There is no performance hit whatsoever from the /SafeSEH option, other than when an exception is raised, so you should always link with this option.

DEP Compatibility (/NXCompat)

Just about every CPU produced today supports a no execute (NX) capability, which means the CPU will not execute non-code pages. Think about the implications of this for a moment: just about every buffer overrun vulnerability is a data bug; the attacker then injects data into the process by way of the buffer overrun and then continues execution within the malicious data buffer. Why on earth is the CPU running data?

Linking with the /NXCompat option means your executable will be protected by the CPU's no execute capability. In our experience, the security team at Microsoft has seen very few compatibility issues because of this option, and there is no performance degradation whatsoever.

Windows Vista® SP1 also adds a new API that will enabled DEP for your running process and once it is set, it cannot be unset:

SetProcessDEPPolicy(PROCESS_DEP_ENABLE);

Image Randomization (/DynamicBase)

Windows Vista and Windows Server® 2008 support image randomization, which means, when the system boots, it shuffles operating system images around in memory. The purpose of this feature is to simply remove some of the predictability from attackers. This is also known as Address Space Layout Randomization (ASLR). Note that for ASLR to be of any use whatsoever, you must also have DEP enabled.

By default, Windows® will only juggle system components around. If you want your image to be moved around by the operating system (highly recommended), then you should link with the /DynamicBase option. (This option is available in the Visual Studio 2005 SP1 and later toolset.) There is an interesting side effect of linking with /DynamicBase—the operating system will also randomize your stack, which helps reduce predictability and makes it much harder for attackers to successfully compromise a system. Note also that the heap is also randomized in Windows Vista and Windows Server 2008, but this is there by default, you do not need to compile or link with any special options.

Safer Function Calls

Take a look at the following lines of code:

void func(char *p) {
    char d[20];
    strcpy(d,p);
    // etc
}

Assuming *p contains untrusted data, this code represents a security vulnerability. The perverse thing about this code is that the compiler could potentially have promoted the call to strcpy to a safer function call that bounded the copy operation to the size of the destination buffer. Why? Because the buffer size is static and known at compile time!

With Visual C++ you can add the following line to your stdafx.h header file:

#define _CRT_SECURE_COPP_OVERLOAD_STANDARD_NAMES 1

The compiler will then go ahead and emit the following code from the initial, unsafe function:

void func(char *p) {
    char d[20];
    strcpy_s(d,__countof(d), p);
    // etc
}

As you can see, the code is now secure, and yet the developer did nothing other than add a #define. This is one of my favorite additions to Visual C++ because about 50 percent of unsafe function calls can be promoted to safer calls automatically.

C++ Operator::new

Finally, Visual C++ 2005 and later adds a defense that detects integer overflow when calling operator::new. You can start with code like this:

CFoo *p = new CFoo[count];

Visual C++ compiles it to this:

00004    33 c9           xor    ecx, ecx
00006    8b c6           mov    eax, esi
00008    ba 04 00 00 00  mov    edx, 4
0000d    f7 e2           mul    edx
0000f    0f 90 c1       seto    cl
00012    f7 d9           neg    ecx
00014    0b c8            or    ecx, eax
00016    51             push    ecx
00017    e8 00 00 00 00 call    ??2@YAPAXI@Z    ; operator new

After the amount of memory to allocate is calculated (mul edx), the CL register is set or not set depending on the value of the overflow flag after the multiplication, so, ECX will be 0x00000000 or 0xFFFFFFFF. Because of the next operation (or ecx) the ECX register will either be 0xFFFFFFFF or the value held in EAX, which is result of the initial multiply. This is then passed to operator::new, which will fail in the face of the 2^N-1 allocation.

This defense is free. There is no compiler switch; this is simply what the compiler does.

What if It Fails?

Ah! This thorny question! If any of the defenses listed above are triggered, there is a pretty nasty result: the application quits. That's hardly optimal, but it is a heck of a lot better than running the attacker's malicious code.

The SDL mandates that new code use all these defensive options simply because there are so many attacks out there and you can never verify that your code is 100 percent free of vulnerabilities. One of the catch phrases of the SDL is "assume your code will fail; now what?" In the real world "now what" means put up a fight! Don't let the attacker's code have an easy time; make it hard for the attacker to get exploits to work. Don't give up!

So compile with the latest version of the C++ compiler to get a better /GS, and link with the latest linker to get the benefit of CPU and operating systems defenses.

Send your questions and comments to briefs@microsoft.com.

Michael Howard is a Principal Security Program Manager at Microsoft focusing on secure process improvement and best practices. He is the coauthor of five security books including Writing Secure Code for Windows Vista, The Security Development Lifecycle, Writing Secure Code, and 19 Deadly Sins of Software Security. View his blog at blogs.msdn.com/michael_howard.


Extreme ASP.NET: The Only Data-binding Control You'll Ever Need

Extreme ASP.NET
The Only Data-binding Control You'll Ever Need
Fritz Onion

Code download available at: ExtremeASPNET2008_03.exe (192KB)
Contents
ListView Basics
ListView and CSS
Pagination
Sorting, Editing, Inserting, and Deleting
Grouping
Putting ListView to Work

T
he ASP.NET 3.5 release that accompanies Visual Studio® 2008 introduces a new data-binding control—the ListView. I know what you're thinking: why do we need yet another data-bound control in ASP.NET? After all, we already have more than 10 controls to choose from when displaying collections of data, including the semi-retired DataGrid, the new and improved GridView, the ever-solid and simple Repeater, the unique and flexible DataList, the handy FormView and its somewhat redundant sibling the DetailsView. And of course there are the single-dimensional list controls BulletedList, ListBox, DropDownList, RadioButtonList and CheckBoxList.

Well, the ListView can literally replace all other data-binding controls in ASP.NET. That's right, every last one. You could literally eschew every other control in the above list in lieu of the ListView control. The ListView also makes several data-binding tasks much easier than they were with previous controls, including styling with CSS, flexible pagination, and the full complement of sorting, inserting, deleting, and updating features.

Let me begin by describing the typical usage model of the ListView, and then I will progress through the features of the control, illustrating its flexibility and power. By the end of this column you should have enough information to decide how many databound controls to keep in your ASP.NET toolbox.

ListView Basics

The ListView is a template-driven control, meaning that it will not render anything by default—you must completely specify the HTML you want it to render in the form of templates. Like most template controls, the ItemTemplate is going to be the focus of most of your efforts, as that is where you place the HTML content that will be repeated for each row of the bound data set.

What is new in the ListView—and what really sets it apart from other controls—is the introduction of a LayoutTemplate. The LayoutTemplate is where you define the top-level HTML to be output as part of the control rendering. If you want the ListView to render as a table, for example, you would include the top-level <table> and perhaps <thead> elements in the LayoutTemplate, leaving the row and cell rendering to the ItemTemplate as shown in Figure 1 (in this case, bound to a data source exposing a simple table containing movie titles and release dates). Figure 2 shows the browser rendering.

Figure 1 Using LayoutTemplate and ItemTemplate (Close Figure 1)
<asp:ListView runat="server" ID="_simpleTableListView"
  DataSourceID="_moviesDataSource">
  <LayoutTemplate>
    <table>
      <thead>
        <tr>
          <th>ID</th>
          <th>Title</th>
          <th>Release Date</th>
        </tr>
      </thead>
      <tbody>
        <asp:PlaceHolder runat="server" ID="itemPlaceholder" />
      </tbody>
    </table>
  </LayoutTemplate>
  <ItemTemplate>
    <tr>
      <td><%# Eval("movie_id") %></td>
      <td><%# Eval("title") %></td>
      <td><%# Eval("release_date", "{0:d}") %></td>
    </tr>
  </ItemTemplate>
</asp:ListView>
Figure 2 The List Displayed in a Table
Figure 2 The List Displayed in a Table(Click the image for a smaller view)
Figure 2 The List Displayed in a Table
Figure 2 The List Displayed in a Table (Click the image for a larger view)

The association between the LayoutTemplate and the ItemTemplate is made by the single server-side control in the LayoutTemplate with an ID set to itemPlaceholder. (You can change the ID string from its default with the ItemPlaceholderID property of the ListView.) In my first example, I used the technique of placing an instance of a PlaceHolder control at the location in the template where I wanted the contents of the ItemTemplate to be injected. Note that although it must support child controls, there is no restriction on what type of control you use as a placeholder—the ID is what is important. For example, I could have written the LayoutTemplate using a server-side table row instead of the PlaceHolder control to achieve the same effect:

<LayoutTemplate>
  <table>
    <thead>
      <tr>
        <th>ID</th>
        <th>Title</th>
        <th>Release Date</th>
      </tr>
    </thead>
    <tbody>
      <tr runat="server" ID="itemPlaceholder" />
    </tbody>
  </table>
</LayoutTemplate>

In general, I prefer to use the generic PlaceHolder control for two reasons. The first is because the names match up nicely. And furthermore, this control renders no HTML of its own and is instead replaced with the contents of the ItemTemplate, so using a control that has no purpose besides reserving a place in the control hierarchy seems a more logical choice.

Of course, what makes the ListView so flexible is the fact that you completely control what goes into the LayoutTemplate. You're not restricted to using a table—you can place whatever arbitrary HTML you want to render in the LayoutTemplate, as long as the contents of your ItemTemplate make sense when they are injected at the itemPlaceholder control's location. Here's an example of a ListView bound to the same movie data source, but this time instead of a table, the movies are displayed in a bulleted list with the title and release date (the resulting list is shown in Figure 3):

Figure 3 Same List, Different Format
Figure 3 Same List, Different Format(Click the image for a smaller view)
Figure 3 Same List, Different Format
Figure 3 Same List, Different Format (Click the image for a larger view)

   <asp:ListView runat="server"
     ID="_simpleTableListView"
     DataSourceID="_moviesDataSource">
     <LayoutTemplate>
       <ul>
         <asp:PlaceHolder runat="server"
              ID="itemPlaceholder" />
       </ul>
     </LayoutTemplate>
     <ItemTemplate>
       <li><%# Eval("title") %>,
           <%# Eval("release_date", "{0:d}") %> </li>
     </ItemTemplate>
                         </asp:ListView>

ListView and CSS

ASP.NET developers have long been at the mercy of individual controls when it comes to creating CSS-driven sites. Many of the default controls render inline styles or make it difficult to associate CSS classes with portions of their HTML output. Microsoft actually released a toolkit in April 2006 called the CSS Control Adapter Toolkit that provided an alternate rendering mechanism for several controls (including the GridView) that were completely CSS-driven to help remedy this problem (see the October 2006 Extreme ASP.NET column for more details at msdn.microsoft.com/msdnmag/issues/06/10/ExtremeASPNET). These alternate renderings were never incorporated into the full release, though, so they remain a separate installation and lack designer support.

The ListView offers a refreshing simplicity to leveraging CSS in your site by putting you in complete control of where and when to apply stylesheets. One very common scenario is for a developer to be handed a preset design for a particular page, typically consisting of HTML and CSS. Using a traditional GridView in order to render a specific design for a table has always been tricky to get right because the GridView class only provides a limited set of hooks modifying the resulting HTML.

I have seen many developers resort to trial and error, applying style attributes to a grid, viewing the source of the page to understand exactly where the styles were placed, and iterating until the grid rendered as desired. With the ListView, the guesswork is gone as you are now responsible for the layout as well as the content.

As an example, suppose you were given a table that needed to look like the one shown in Figure 4, with the design consisting of the .htm and .css files shown in Figure 5.

Figure 5 HTML and CSS for the Table (Close Figure 5)

HTML
<div class="PrettyGrid">
  <table cellpadding="0" cellspacing="0" summary="">
    <thead>
      <tr>
        <th scope="col"><a href="http://.">ID</a></th>
        <th scope="col"><a href="http://.">Title</a></th>
        <th scope="col"><a href="http://.">Release date</a></th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <td>1</td>
        <td>Where the Wild Things Are</td>
        <td>12/15/2008</td>
      </tr>
  <!-- ... -->
    </tbody>
  </table>
  <div class="Pagination">
    <span>1</span>
    <a href="http://.">2</a>
    <a href="http://.">3</a>
  </div>
</div>
CSS
.PrettyGrid
{
  width: 100%;
}

.PrettyGrid div.Pagination,
.PrettyGrid div.Pagination a,
.PrettyGrid div.Pagination span
{
  color: #00FFFF;
  background: #284775;
  font-weight: normal;
  padding: 2px;
}
.PrettyGrid table
{
  border: solid 1px #CCCCCC;
  width: 100%;
}
/*...*/
Figure 4 Target Design for a Table
Figure 4 Target Design for a Table(Click the image for a smaller view)
Figure 4 Target Design for a Table
Figure 4 Target Design for a Table (Click the image for a larger view)

You could quickly construct a ListView that renders exactly as the HTML/CSS combination requires by plucking the appropriate sections from the HTML and placing them within the corresponding templates as shown in Figure 6. The end result will look exactly like the original HTML styled completely with CSS. It's also easy to make modifications to the design by simply updating the HTML or corresponding CSS.

Figure 6 ListView to Construct the Table (Close Figure 6)
<asp:ListView ID="_moviesGrid" runat="server" DataKeyNames="movie_id"
  DataSourceID="_moviesDataSource">
  <LayoutTemplate>
    <div class="PrettyGrid">
      <table cellpadding="0" cellspacing="0" summary="">
        <thead>
          <tr>
            <th scope="col"><a href="http://.">ID</a ></th>
            <th scope="col"><a href="http://.">Title</a></th>
            <th scope="col"><a href="http://.">Release date</a></th>
          </tr>
        </thead>
        <tbody>
          <asp:PlaceHolder ID="itemPlaceholder" runat="server" />
        </tbody>
      </table>
      <div class="Pagination">
        <span>1</span>
        <a href="http://.">2</a>
        <a href="http://.">3</a>
      </div>
    </div>
  </LayoutTemplate>

  <AlternatingItemTemplate>
    <tr class="Alternate">
      <td><asp:Label ID="movie_idLabel" runat="server"
        Text='<%# Eval("movie_id") %>' /></td>
      <td><asp:Label ID="titleLabel" runat="server"
        Text='<%# Eval("title") %>' /></td>
      <td><asp:Label ID="release_dateLabel" runat="server"
        Text='<%# Eval("release_date", "{0:d}") %>' />  </td>
    </tr>
  </AlternatingItemTemplate>

  <ItemTemplate>
    <tr>
      <td><asp:Label ID="movie_idLabel" runat="server"
        Text='<%# Eval("movie_id") %>' /></td>
      <td><asp:Label ID="titleLabel" runat="server"
        Text='<%# Eval("title") %>' /></td>
      <td><asp:Label ID="release_dateLabel" runat="server"
        Text='<%# Eval("release_date", "{0:d}") %>' /> </td>
    </tr>
  </ItemTemplate>
</asp:ListView>

Pagination

The original HTML design that I started with in the previous section had both pagination and sorting implied by the design, so my task to fully implement this grid based on the specification is not yet complete. Let's start with pagination, and I'll get to sorting next.

Pagination is accomplished in the ListView control through the introduction of another new control, the DataPager. By separating paging into a separate control, the DataPager decouples the paging UI from whatever the ListView is using to present the data. This means that you can place the paging UI pretty much wherever you like on the page, and you can also create as many DataPager controls as you like. One common application of multiple pager controls is to provide a pagination interface both at the top and the bottom of a grid of data so the user doesn't have to scroll the grid to navigate to the next page—something that is easily accomplished with the DataPager.

Let's start by implementing pagination in the ListView example from the previous section. The simplest way to create a DataPager control associated with a ListView is to actually embed the DataPager control within the LayoutTemplate of the ListView:

<asp:ListView ID="_moviesGrid"
  runat="server" DataKeyNames="movie_id"
  DataSourceID="_moviesDataSource">
  <LayoutTemplate>
    <!-- ... -->
    <div class="Pagination">
      <asp:DataPager ID="_moviesGridDataPager" runat="server">
        <Fields>
          <asp:NumericPagerField />
        </Fields>
      </asp:DataPager>
    </div>
  </LayoutTemplate>
</asp:ListView>

By embedding the DataPager within the LayoutTemplate of the ListView, they are implicitly associated. The other option would be to place the DataPager elsewhere on the page and associate it with the ListView by setting its PagedControlID to the ID of the associated ListView.

In this particular case, the NumericPagerField displays the exact interface I wanted—just a series of numbers displayed as hyperlinks to navigate pages. The DataPager supports three types of fields:

  • NumericPagerField displays the 1 2 3... pagination interface.
  • NextPreviousPagerField displays a Next, Previous, First, and Last buttons to iterate across rows.
  • TemplatePagerField lets you define the exact design and function of the pagination interface using a PagerTemplate.

The DataPager control is implemented generically to provide paging support for any control that implements the IPageableItemContainer interface (currently the ListView is the only control that implements this interface), like so:

public interface IPageableItemContainer
{
    event EventHandler<PageEventArgs> TotalRowCountAvailable;
    void SetPageProperties(int startRowIndex, int maximumRows,
                           bool databind);
    int MaximumRows { get; }
    int StartRowIndex { get; }
}

Figure 7 shows the relationship among the ListView, a DataPager, and an associated DataSource control. The DataPager never interacts directly with the DataSource used to populate the ListView, but rather queries the data it needs through this interface.

Figure 7 Relationship among a ListView, a DataPager, and a DataSource
Figure 7 Relationship among a ListView, a DataPager, and a DataSource(Click the image for a smaller view)
Figure 7 Relationship among a ListView, a DataPager, and a DataSource
Figure 7 Relationship among a ListView, a DataPager, and a DataSource (Click the image for a larger view)

The first thing that happens in preparing the pagination is the ListView queries the DataSource to see whether it supports paging, and if so, whether it can return the total row count. If it can, the ListView retrieves the total number of rows in the data source and then raises the TotalRowCountAvailable event implemented as part of its IPageableItemContainer interface. Any associated DataPager controls will have subscribed to this event and will use the total rowcount number to initialize the fields needed to present the pagination interface. The DataPager will then invoke the SetPageProperties method of the ListView to set the initial row index as well as the maximum number of rows to return.

When the ListView retrieves data from the associated data source, it will request only a subset of rows based on the values set by the DataPager. Whenever the DataPager changes its current page index (generally because of user interaction), it will call the ListView's SetPageProperties again to reflect the current subset of rows to retrieve. You can change the number of records displayed on a page by setting the PageSize property of the DataPager control, which affects the maximum row count information that it sets in the corresponding ListView.

The DataPager also supports the QueryStringField property, which rather drastically changes how the DataPager works. By setting the QueryStringField property to some string (pageNum, for example), you are requesting that the DataPager issue an HTTP GET request in response to the user clicking page numbers, with the requested page number sent through a query string parameter using the string you assign instead of the traditional POST-back model.

One useful side-effect of this change is that it makes it possible for clients to create bookmarks to specific pages in your databound ListView control since the page number will be visible in the URL. Beware that if you switch to this GET model of communication, the POST-back hooking mechanism used by the ASP.NET AJAX UpdatePanel control will not be able to intercept the pagination requests and turn them into asynchronous postbacks:

<asp:DataPager ID="_moviesGridDataPager" runat="server"
  QueryStringField="pageNum" >
  <Fields>
    <asp:NumericPagerField />
  </Fields>
</asp:DataPager>

Note that because the DataPager is completely reliant on the ListView for performing the actual pagination of data, which in turn relies on the associated DataSource control, you have the same constraints on paging that you do with other databound controls. For example, paging only works for SqlDataSource controls when they are set to DataSet mode, which means the entire resultset is loaded into memory to perform pagination. You can, of course, implement your own custom paging either with a custom DataSource control or by using the ObjectDataSource control.

Sorting, Editing, Inserting, and Deleting

The ListView would not be complete without the ability to support sorting and the full complement of create, read, update, and delete (CRUD) operations. Its implementation for each of these commands is similar to the way the FormView control implements commands.

Because the ListView is entirely template driven, it recognizes certain buttons within its templates that have the CommandName property set to one of seven specific command strings: Cancel, Delete, Select, Edit, Insert, Update, and Sort. Each command initiates the corresponding action on the ListView—so if you want to add sorting to your ListView, you need to place a button (the example in Figure 8 uses a LinkButton) within the LayoutTemplate with a CommandName property set to Sort and a CommandArgument set to the column name on which you want the data source to sort. In Figure 8 I have modified the previously static header links for each column in my grid to be clickable links that request the ListView to sort the data based on the column.

Figure 8 Sorting in ListView (Close Figure 8)
<asp:ListView ID="_moviesGrid" runat="server" DataKeyNames="movie_id"
  DataSourceID="_moviesDataSource">
  <LayoutTemplate>
    <div class="PrettyGrid">
      <table cellpadding="0" cellspacing="0" summary="">
        <thead>
          <tr>
            <th scope="col">
              <asp:LinkButton ID="_movieIdSortLink"
                CommandName="Sort" CommandArgument="movie_id"
                runat="server">ID</asp:LinkButton>
            </th>
            <th scope="col">
              <asp:LinkButton ID="_titleSortLink"
                CommandName="Sort" CommandArgument="title"
                runat="server">Title</asp:LinkButton>
            </th>
            <th scope="col">
              <asp:LinkButton ID="_releaseDateSortLink"
                CommandName="Sort" CommandArgument="release_date"
                runat="server">Release date</asp:LinkButton>
            </th>
          </tr>
        </thead>
    <!-- ... -->
  </LayoutTemplate>
</asp:ListView>

You can similarly add command buttons for initiating editing mode, deleting a row, or inserting a new row into the data set, the details of which are essentially identical to other template-based data-binding controls (like the FormView and GridView) and thus will not be shown here.

Grouping

The last major feature of the ListView is the ability to group data into subsets, much like the DataList control provides. The DataList is a tabular control that renders a single row of data in each cell of the rendered table. You control how many rows of the underlying dataset are grouped into a single table row by setting the RepeatColumns property.

Since the ListView is not constrained to render as a table, it needs a more generic way of specifying groups of items to be rendered together, which is what GroupTemplate does. Figure 9 shows the relationship among the LayoutTemplate, GroupTemplate, and ItemTemplate elements within a ListView. The GroupTemplate lets you specify the surrounding HTML for every n elements in the underlying dataset, where n is specified by the GroupItemCount property of the ListView.

Figure 9 Templates in a ListView
Figure 9 Templates in a ListView(Click the image for a smaller view)
Figure 9 Templates in a ListView
Figure 9 Templates in a ListView (Click the image for a larger view)

When you use a GroupTemplate within a ListView, you won't specify a control with an ID of itemPlaceholder within your LayoutTemplate—that control now needs to be in your GroupTemplate. Instead, you will specify a control with an ID of groupPlaceholder in the LayoutTemplate (you can change the control ID by setting the GroupPlaceholderID property of the ListView) to describe where the contents of the GroupTemplate should be injected for each n items encountered in the underlying dataset.

For example, the ListView in Figure 10 shows how I could display four movies from my database in each row of a table by defining a GroupTemplate to delineate rows and leaving the ItemTemplate to lay out only the cell. The result is shown in Figure 11.

Figure 10 Defining Rows with GroupTemplate (Close Figure 10)
<asp:ListView ID="_groupListView" runat="server"
  DataKeyNames="movie_id" DataSourceID="_moviesDataSource"
  GroupItemCount="4" >
  <GroupTemplate>
    <tr>
      <asp:PlaceHolder runat="server" ID="itemPlaceholder" />
    </tr>
  </GroupTemplate>
  <LayoutTemplate>
    <table>
      <asp:PlaceHolder ID="groupPlaceholder" runat="server" />
    </table>
  </LayoutTemplate>
  <ItemTemplate>
    <td>
      movie_id:
      <asp:Label ID="_movie_idLabel" runat="server"
        Text='<%# Eval("movie_id") %>' /> <br />
      title:
      <asp:Label ID="_titleLabel" runat="server"
        Text='<%# Eval("title") %>' /> <br />
      release_date:
      <asp:Label ID="_release_dateLabel" runat="server"
        Text='<%# Eval("release_date", "{0:d}") %>' /> <br />
      <br />
    </td>
  </ItemTemplate>
</asp:ListView>
Figure 11 GroupTemplate Rows in the Resulting Web Page
Figure 11 GroupTemplate Rows in the Resulting Web Page(Click the image for a smaller view)
Figure 11 GroupTemplate Rows in the Resulting Web Page
Figure 11 GroupTemplate Rows in the Resulting Web Page (Click the image for a larger view)

This is very similar to what you could do with a DataList, but because you are working with a ListView, you can just as easily add both pagination and sorting as you did earlier with your grid rendering, a task that would be rather daunting with the DataList. The code download for this article contains an example that implements both pagination and sorting for your reference.

Putting ListView to Work

You may want to begin your experiments with the ListView control by using the designer in Visual Studio 2008, which lets you select from among five different layouts: grid, tiled, bulleted list, flow, and single row. You can quickly see the different layout options that are possible—but the real power of the ListView lies in the control you have over the HTML that it renders, so you're more likely to spend your time building the LayoutTemplate yourself in real projects. Will you end up using the ListView for every data-binding opportunity in the future? That may be a bit extreme—but it's nice to know you could. I know I'll be reaching for this flexible data-binding control more often than not.

Send your questions and comments for Fritz to xtrmaspt@microsoft.com.

Fritz Onion is a co-founder of Pluralsight, a Microsoft .NET training provider, where he heads the Web development curriculum. Fritz is the author of Essential ASP.NET and Essential ASP.NET 2.0. Reach him at pluralsight.com/fritz.


Foundations: Vector Graphics and the WPF Shape Class

Foundations
Vector Graphics and the WPF Shape Class
Charles Petzold

Code download available at: Foundations2008_03.exe (192KB)
Contents
The Role of the Shapes Namespace
Exposing the Pen
Two Drawing Paradigms
Three Get-Only Properties
Parallel Lines and Curves
Minimizing the Heap Allocations
A Better Widened Path
The Other Paradigm

E
ven in the relatively prosaic world of two-dimensional vector graphics, Windows® Presentation Foundation (WPF) has forced programmers to learn many new concepts. In WPF, graphical objects have been elevated to a status nearly equal to controls, often participating in layout and receiving mouse, keyboard, and stylus input. In addition, these graphical objects are retained by the graphical system so they needn't be redrawn as frequently as graphics in the past, but they can also be animated and serve as the targets of data bindings.

When I first began examining WPF, I immediately thought of the System.Windows.Shapes namespace as containing the "baby" graphics classes. These classes seemed suitable for simple displays of lines and rectangles, but I thought grownup WPF programs would probably want to get down-and-dirty by overriding the OnRender method and calling methods in the DrawingContext class.

The DrawGeometry method seemed particularly enticing: in WPF a Geometry object is a combination of connected and unconnected straight lines, arcs, and Bézier curves that in traditional graphics programming is known as a "path." The three arguments to DrawGeometry include a Geometry object, a Pen to stroke the lines and curves of the Geometry, and a Brush to fill enclosed interiors.

The Role of the Shapes Namespace

In time, I realized that my first impression of WPF vector graphics was wrong. Most WPF programs have no need to override the OnRender method and make calls to methods in the DrawingContext class. Overriding OnRender is a good training exercise, but usually not required in most mainstream applications.

With this realization, the System.Windows.Shapes namespace became, at least in my mind, the namespace of choice for rendering two-dimensional vector graphics in WPF. The System.Windows.Shapes namespace consists of these classes: Shape (which is abstract) plus Line, Polyline, Polygon, Path, Rectangle, and Ellipse (all of which are sealed).

The Shape class itself derives from FrameworkElement. The most important Shape derivative is undoubtedly Path; this class has the same degree of power as the DrawGeometry method of DrawingContext but causes much less hassle. When you use the Path class in XAML, you can even define the Geometry object with a string of coded drawing commands.

This is not to say that the Shapes classes constitute an all-purpose vector graphics solution for every application. Each instance of one of these classes is a full-fledged WPF element, and that might be more overhead than you need. Moreover, each of these classes has only one stroke pen and one fill brush, and that might be less color than you require.

For rendering complex vector images containing multiple colors, you have a couple of options. You could create multiple Path objects, of course, but that might be overkill if you want to treat the complex image as its own entity. A better solution in that case involves the DrawingGroup class, which can have multiple GeometryDrawing objects, each of which consists of a Geometry, a stroke pen, and a fill brush. This DrawingGroup object is perhaps the closest thing in WPF to a traditional graphics metafile. The DrawingGroup object can be the basis for a brush (via DrawingBrush) or you can turn it into a DrawingImage for display by the Image class.

When you need only a moderate number of graphics primitives—and particularly when these objects need to receive mouse, keyboard, or stylus input, or be subjected to their own transforms—the classes in the Shapes namespace are ideal.

Now I'm going to discuss deriving from Shape, which is the only unsealed class in the Shapes namespace. You can derive from Shape to implement your own custom vector graphics primitives. Deriving from Shape is the easiest way to ensure that these custom primitives exploit the protocols of the WPF layout system.

Exposing the Pen

I haven't seen the source code for Shape, but I know a few things about the class from general principles. Because Shape derives from FrameworkElement, it overrides the MeasureOverride, ArrangeOverride, and OnRender methods. I suspect that the OnRender override is quite simple and consists solely of a call to the DrawGeometry method of DrawingContext, passing to it the Brush, Pen, and Geometry objects.

While the Brush argument to the DrawGeometry call undoubtedly comes from the Shape's Fill property, Shape does not define a property of type Pen. Instead, Shape defines nine separate properties that it constructs internally into a Pen object. These nine properties all begin with the word Stroke, and actually help make the use of Shape derivatives fairly easy, both in XAML and code.

For example, if you were defining a GeometryDrawing object containing an EllipseGeometry in XAML, it might look like this:

<GeometryDrawing Brush="Red">
    <GeometryDrawing.Geometry>
        <EllipseGeometry ... />
    </GeometryDrawing.Geometry>
    <GeometryDrawing.Pen>
        <Pen Brush="Blue" Thickness="3" />
    </GeometryDrawing.Pen>
</GeometryDrawing>

Notice that the Pen object needs to be defined within a property element, and even with this markup you still haven't specified how the GeometryDrawing will actually be rendered. Equivalent XAML for the Ellipse class, however, looks like this:

<Ellipse Fill="Red" Stroke="Blue" StrokeThickness="3" ... />

When the Shape class calls the DrawGeometry method of DrawingContext, it also requires a Geometry object. This Geometry object is the focus of much of the rest of this column.

Two Drawing Paradigms

One of the most confusing aspects of the Shape class involves its ability to encapsulate two different drawing paradigms.

The first paradigm is more traditional. Let's call it the "coordinate" paradigm. When using the Line, Polyline, Polygon, and Path classes you specify actual coordinate points that define the straight lines and curves that make up the figure, perhaps like this:

Line line = new Line();
line.X1 = 100;
line.Y1 = 50;
line.X2 = 200;
line.Y2 = 150;
line.Stroke = Brushes.Blue;
line.StrokeThickness = 12;

The result is shown in Figure 1. While a Canvas panel is probably the most common parent of a Line element, you can actually put a Line anywhere you'd put another control or element. The program in Figure 1 sets the Line to the Content property of a Window.

Figure 1 A Line Element in a Window
Figure 1 A Line Element in a Window

The Line, Polyline, Polygon, and Path classes inherit a Stretch property from Shape, which then defines the default value as Stretch.None. If you set the property to Stretch.Fill, the Line fills its parent, as shown in Figure 2.

Figure 2 A Line Element with Stretch Set to Fill
Figure 2 A Line Element with Stretch Set to Fill

Shape also implements a different drawing paradigm that's perhaps more of a WPF style of rendering. We can call this the "autosize" paradigm. This paradigm is realized in the Rectangle and Ellipse classes. Figure 3 shows an Ellipse element set to the Content property of a Window. The Ellipse element is created like this:

Figure 3 An Ellipse Element in a Window
Figure 3 An Ellipse Element in a Window

   Ellipse elips = new Ellipse();
   elips.Fill = Brushes.Red;
   elips.Stroke = Brushes.Blue;
   elips.StrokeThickness = 12;

The Ellipse element is not given any coordinate or size information; the figure fills the interior of its parent by default. Rectangle and Ellipse set the default value of the Stretch property to Stretch.Fill. If you set the property to Stretch.None, the ellipse collapses into a tiny ball with only its border visible, as shown in Figure 4.

Figure 4 An Ellipse Element with Stretch Set to None
Figure 4 An Ellipse Element with Stretch Set to None

You can also create this effect by setting the HorizontalAlignment and VerticalAlignment properties of Rectangle or Ellipse to something other than Stretch. (Included with the downloadable code for this column is a program named StretchExplore that lets you see different stretch options for Line and Ellipse as well as the two Shape derivatives that I'll develop here.)

If you need a Rectangle or Ellipse element to be a particular size, you can use the Width and Height properties defined by FrameworkElement, or you can set MinWidth, MaxWidth, MinHeight, and MaxHeight to specify a range of values. You must set these properties when rendering Rectangle or Ellipse on a Canvas panel.

When deriving a custom graphics primitive from Shape, it is easier to implement the "coordinate" paradigm rather than the "autosize" paradigm.

Three Get-Only Properties

Shape defines 14 properties inherited by its derivatives, and I've already mentioned 11 of them: the 9 pen-related properties that begin with the word Stroke, the Fill property, and Stretch. The other 3 properties are defined with get accessors only: DefiningGeometry (protected and abstract); GeometryTransform (public and virtual); and RenderedGeometry (public and virtual).

When deriving from Shape to implement a graphics primitive with the "coordinate" paradigm, the only property you need to override is DefiningGeometry. As the name implies, you implement DefiningGeometry by returning an object of type Geometry that defines your graphics primitive. It's very likely that your Shape derivative class will include additional properties that define the graphic. In most cases, you should back these properties with dependency properties so they can be the target of data bindings and animations.

Because DefiningGeometry can be called at any time, and particularly whenever any property that affects the primitive changes, it's important to implement DefiningGeometry without routinely allocating memory from the heap. If every call to DefiningGeometry results in a heap allocation, eventually the Microsoft® .NET Framework garbage collector will need to take action. You should try to purge your DefiningGeometry code of any class instantiations, and you should also be aware of implicit heap allocations associated with some methods. Next I'll show you several techniques for avoiding heap allocations in your Shape derivatives.

As a first simple example, suppose you'd prefer an alternative to the Line primitive that lets you set the starting and ending coordinate points with Point objects rather than pairs of double objects. The PointLine class that derives from Shape is shown in Figure 5.

Figure 5 PointLine.cs—the PointLine Class (Close Figure 5)
using System;
using System.Windows;
using System.Windows.Media;
using System.Windows.Shapes;

namespace Petzold.Shapes
{
    public class PointLine : Shape
    {
        LineGeometry linegeo = new LineGeometry();

        // Dependency properties
        public static readonly DependencyProperty StartPointProperty =
            LineGeometry.StartPointProperty.AddOwner(
                typeof(PointLine),
                new FrameworkPropertyMetadata(new Point(0, 0),
                    FrameworkPropertyMetadataOptions.AffectsMeasure));

        public static readonly DependencyProperty EndPointProperty =
            LineGeometry.EndPointProperty.AddOwner(
                typeof(PointLine),
                new FrameworkPropertyMetadata(new Point(0, 0),
                    FrameworkPropertyMetadataOptions.AffectsMeasure));

        public Point StartPoint
        {
            set { SetValue(StartPointProperty, value); }
            get { return (Point)GetValue(StartPointProperty); }
        }

        public Point EndPoint
        {
            set { SetValue(EndPointProperty, value); }
            get { return (Point)GetValue(EndPointProperty); }
        }

        // Required DefiningGeometry override
        protected override Geometry DefiningGeometry
        {
            get
            {
                linegeo.StartPoint = StartPoint;
                linegeo.EndPoint = EndPoint;
                return linegeo;
            }
        }
    }
}

Notice that StartPoint and EndPoint are both defined with dependency properties. I used the same names as the corresponding properties in the LineGeometry class so I could simply add the PointLine class as a new owner to these properties. Both properties have the AffectsMeasure flag set because both properties affect the size of the element. When either property changes, a new layout pass occurs that culminates in a call to OnRender implemented in Shape. If you define a property that affects only the appearance of the shape and not its size, you can use the AffectsRender flag instead to avoid initiating a layout pass.

The get accessor of the DefiningGeometry property returns a LineGeometry object based on the StartPoint and EndPoint properties. You should notice that the class reuses a single LineGeometry object that is defined as a field rather than creating a new LineGeometry object during each DefiningGeometry call. This is a crucial technique when deriving from Shape in order to avoid cluttering the heap with new object instances.

Figure 6 shows another relatively straightforward Shape derivative. The CenteredEllipse class lets you draw an ellipse by specifying its center, its horizontal and vertical radii, and a rotation angle. CenteredEllipse adds itself as an owner to the Center, RadiusX, and RadiusY properties defined by EllipseGeometry, and also appropriates the RotationAngle property from ArcSegment. Notice that CenteredEllipse creates a RotationTransform stored as a field and reuses it to associate a transform with the EllipseGeometry object.

Figure 6 CenteredEllipse.cs—the CenteredEllipse Class (Close Figure 6)
using System;
using System.Windows;
using System.Windows.Media;
using System.Windows.Shapes;

namespace Petzold.Shapes
{
  public class CenteredEllipse : Shape
  {
    EllipseGeometry elipGeo = new EllipseGeometry();
    RotateTransform xform = new RotateTransform();

    // Dependency properties
    public static readonly DependencyProperty CenterProperty =
      EllipseGeometry.CenterProperty.AddOwner(
        typeof(CenteredEllipse),
        new FrameworkPropertyMetadata(new Point(0, 0),
          EllipsePropertyChanged));

    public static readonly DependencyProperty RadiusXProperty =
      EllipseGeometry.RadiusXProperty.AddOwner(
        typeof(CenteredEllipse),
        new FrameworkPropertyMetadata(0.0,
          EllipsePropertyChanged));

    public static readonly DependencyProperty RadiusYProperty =
      EllipseGeometry.RadiusYProperty.AddOwner(
        typeof(CenteredEllipse),
        new FrameworkPropertyMetadata(0.0,
          EllipsePropertyChanged));

    public static readonly DependencyProperty RotationAngleProperty =
      ArcSegment.RotationAngleProperty.AddOwner(
        typeof(CenteredEllipse),
        new FrameworkPropertyMetadata(0.0,
          TransformPropertyChanged));

    static void EllipsePropertyChanged(DependencyObject obj,
      DependencyPropertyChangedEventArgs args)
    {
      (obj as CenteredEllipse).EllipsePropertyChanged(args);
    }

    static void TransformPropertyChanged(DependencyObject obj,
      DependencyPropertyChangedEventArgs args)
    {
      (obj as CenteredEllipse).TransformPropertyChanged(args);
    }

    void EllipsePropertyChanged(DependencyPropertyChangedEventArgs args)
    {
      elipGeo.Center = Center;
      elipGeo.RadiusX = RadiusX;
      elipGeo.RadiusY = RadiusY;
      InvalidateMeasure();
    }

    void TransformPropertyChanged(DependencyPropertyChangedEventArgs args)
    {
      xform.Angle = RotationAngle;
      xform.CenterX = Center.X;
      xform.CenterY = Center.Y;
      InvalidateMeasure();
    }

    public CenteredEllipse()
    {
      elipGeo.Transform = xform;
    }

    // Public CLR properties
    public Point Center
    {
      set { SetValue(CenterProperty, value); }
      get { return (Point)GetValue(CenterProperty); }
    }

    public double RadiusX
    {
      set { SetValue(RadiusXProperty, value); }
      get { return (double)GetValue(RadiusXProperty); }
    }

    public double RadiusY
    {
      set { SetValue(RadiusYProperty, value); }
      get { return (double)GetValue(RadiusYProperty); }
    }

    public double RotationAngle
    {
      set { SetValue(RotationAngleProperty, value); }
      get { return (double)GetValue(RotationAngleProperty); }
    }

    // Required DefiningGeometry override
    protected override Geometry DefiningGeometry
    {
      get
      {
        return elipGeo;
      }
    }
  }
}

CenteredEllipse illustrates an alternative structure that is appropriate if defining the Geometry requires significant processing time. Rather than setting the AffectsMeasure flag in the dependency properties, the class defines property-changed handlers that set the properties of the two private fields and then call InvalidateMeasure to initiate the new layout pass.

You'll discover that these PointLine and CenteredEllipse classes behave the same as the regular Line class. If you set the Stretch property to something other than Stretch.None, the Shape class calculates a transform necessary to resize the geometry and shift it so it occupies the interior of its parent. Obviously this transform is based on the size of the Geometry itself (which is available through the Bounds property), but it also takes into account the StrokeThickness property so that at least most of the object stays within the parent's rectangle.

These calculations occur behind the scenes within the Shape class itself. The transform that Shape calculates is available through the public GeometryTransform property. (The default value is the static property Transform.Identity.) Shape also calculates the RenderedGeometry property as the DefiningGeometry property transformed by the GeometryTransform property. It is this RenderedGeometry property that the OnRender method in Shape uses to draw the graphic.

It's important for Shape to apply this transform to the geometry rather than to the figure itself. If the transform were applied to the figure, it would also affect the width of the pen that is used to stroke the lines.

The good news, of course, is that you don't have to worry about this transform calculation. All you need to supply is the DefiningGeometry.

The source code for this column consists of a single Visual Studio solution named DerivingFromShape. The Shape derivatives themselves are in a DLL project named Petzold.Shapes. This DLL also includes Shape derivatives that implement line and polyline primitives tipped with arrows that I wrote for an April 2007 blog entry (charlespetzold.com/blog/2007/04/191200.html).

Parallel Lines and Curves

Let's leap into something a bit more challenging. Suppose you want a class that is essentially a souped-up version of Path. The Path class defines a single property of its own named Data of type Geometry. It's very likely that Path returns this same Geometry object from its DefiningGeometry property, qualifying Path as surely the simplest of the Shape derivatives.

Our new class will itself define a Data property by adding an owner to Path's Data property. But rather than just rendering the Geometry object, our new class will render the Geometry in parallel, as shown in Figure 7. The new ParallelPath class, as it is called, has a property named Number for the number of lines (5 in Figure 7) and Gap for the gap between the lines. In Figure 7, StrokeThickness is set to 3 and Gap to 4.

Figure 7 The ParallelPathDemo Program
Figure 7 The ParallelPathDemo Program(Click the image for a smaller view)
Figure 7 The ParallelPathDemo Program
Figure 7 The ParallelPathDemo Program (Click the image for a larger view)

I'll warn you right now that the algorithm used by ParallelPath breaks down under some conditions and causes extraneous lines to shoot off sometimes, but it works fairly well if you keep the number of parallel paths within reason and maintain smoothness between the various path segments. ParallelPath might also give unexpected (although completely deterministic) results when you fill it with a brush.

The concept of generating parallel lines is fairly simple when only straight lines are involved, but the ParallelPath class also works for Bézier curves and arcs. Just as with Path, you can set the Data property of ParallelPath to any object of type Geometry, which is an abstract class from which seven sealed classes derive: CombinedGeometry, EllipseGeometry, GeometryGroup, LineGeometry, PathGeometry, RectangleGeometry, and StreamGeometry.

The image in Figure 7 is generated by the following code:

<ps:ParallelPath Stroke="Black" StrokeThickness="3"
  Number="5" Gap="4" Tolerance="10">
  <ps:ParallelPath.Data>
    <PathGeometry>
      <PathFigure StartPoint="200 100">
        <PolyLineSegment Points="350 200, 450 50" />
        <PolyBezierSegment
          Points="500 0, 600 300, 350 300,
                  100 300, 50 0, 200 100" />
      </PathFigure>
    </PathGeometry>
  </ps:ParallelPath.Data>
</ps:ParallelPath>

The "ps" XML namespace is associated with the Petzold.Shapes CLR namespace. You can also set the Data property to a string in the path mini-language; in that case, however, the individual coordinates cannot be the targets of data bindings or animations.

The most versatile of the Geometry derivatives is PathGeometry. A PathGeometry object is a collection of PathFigure objects, each of which is a collection of connected straight lines and curves stored as a collection of PathSegment objects. PathSegment is an abstract class from which seven classes derive for rendering straight lines, arcs, and Bézier curves. Any coordinate point in any of these segments can be the target of a data binding or animation.

Regardless what type of Geometry object you set to the Data property, ParallelPath generates a PathGeometry object describing the parallel straight lines and curves. The algorithm does not, however, attempt to find a Bézier curve that is parallel to another Bézier curve. The algorithm is instead based entirely on polylines: The input is one or more polylines and the output consists of multiple polylines for each input polyline. For this reason, ParallelPath needs to "flatten" the input geometry—which means converting the entire geometry (including arcs and Bézier curves) into a polyline approximation.

If ParallelPath is not giving you the results you anticipate—if, for example, there are some extraneous lines that shoot off at the endpoints of some segments—you can set the Tolerance property to a value greater than 1; perhaps to 10 as the ParallelPathDemo program does. This Tolerance value is (roughly) the number of device-independent units in each polyline of the flattened geometry.

Fortunately, the Geometry class includes a method for flattening geometries. The method is named GetFlattenedPathGeometry and returns a PathGeometry object consisting entirely of PathFigure objects with PolyLineSegment objects. (The PathFigure class itself defines a GetFlattenedPathFigure method that returns another PathFigure.)

Unfortunately, the GetFlattenedPathGeometry method needs to allocate memory from the managed heap to create the PathGeometry object, as well as to create the PathFigure and PolyLineSegment children that make up the geometry. These memory allocations are potentially a problem. Suppose the Data property of ParallelPath is set to a PathGeometry object, and one of the coordinate points of the PathGeometry is animated. That means that ParallelPath needs to regenerate the DefiningGeometry object for every change in that animated coordinate point. By calling GetFlattenedPathGeometry, ParallelPath is implicitly performing repeated heap allocations, which will eventually require that the .NET garbage collector runs.

Minimizing the Heap Allocations

Because GetFlattenedPathGeometry likely performs many memory allocations, I decided to write my own path-flattening routine. Normally that would require creating new instances of PathGeometry, PathFigure, and PolyLineSegment objects, but I also wrote some simple methods that cache and reuse these objects. The path-flattening and caching methods are in a class named PathGeometryHelper. The class obviously needs to perform some memory allocations, at least initially, but not routine memory allocations, and that (I hope) makes a difference overall.

There are two cases where my PathGeometryHelper class cannot implement its own flattening algorithms. One case is when the Geometry is of type StreamGeometry. This is because StreamGeometry is built from calls to the StreamGeometryContext object, but the geometry itself is inaccessible except through the static PathGeometry.CreateFromGeometry method. The second case is an object of type CombinedGeometry, which consists of two Geometry objects in Boolean combinations.

For these two cases, my PathGeometryHelper class gives up and just calls GetFlattenedPathGeometry. (Actually, because a GeometryGroup object can have children or grandchildren of type StreamGeometry or CombinedGeometry object, I call GetFlattenedPathGeometry if any part of the path is an object of these types.) These exceptions represent compromises, but not very severe ones. An object of type StreamGeometry cannot be animated anyway, and the use of CombinedGeometry is probably quite rare.

To help the class further avoid memory allocations, the ParallelPath class defines two fields of type PathGeometry. One, named pathGeoSrc ("PathGeometry source"), stores the flattened path; the other, named pathGeoDst ("PathGeometry destination"), stores the path expanded into parallel lines. Another field stores an instance of PathGeometryHelper named pathHelper

The four properties that ParallelPath defines are associated with two property-changed callbacks named SourcePropertyChanged and DestinationPropertyChanged. Whenever the Data or Tolerance property changes, the SourcePropertyChanged handler first caches the previous PathGeometry source, like so:

pathHelper.CacheAll(pathGeoSrc);

The CacheAll method caches the PathGeometry object itself as well as all PathFigure and PolyLineSegment objects contained in the PathGeometry. However, if the previous PathGeometry were obtained from a call to the GetFlattenedPathGeometry method of Geometry, then objects will be frozen and can no longer be modified. Such objects are not cached.

The SourcePropertyChanged handler then flattens the incoming geometry and stores it as pathGeoSrc, using this code:

pathGeoSrc = pathHelper.FlattenGeometry(Data, Tolerance);

Whenever possible, the FlattenGeometry method uses PathGeometry, PathFigure, and PolyLineSegment objects from the cache. SourcePropertyChanged concludes by calling the property-changed callback associated with the Number and Gap properties, like so:

DestinationPropertyChanged(args);

The DestinationPropertyChanged callback begins by returning the components of the PathGeometry containing the parallel lines to the cache, as you see here:

pathHelper.CacheAll(pathGeoDst);

The callback then calls the GenerateGeometry method in ParallelPath to generate multiple parallel paths, like so:

pathGeoDst = GenerateGeometry(pathGeoSrc);

The GenerateGeometry method also uses PathGeometry, PathFigure, and PolyLineSegment objects from the cache, as well as an object of type List<Point> stored as a field and reused on each call. If sufficient objects exist in the cache, the entire process occurs without any heap allocations. The DestinationPropertyChanged callback concludes by initiating the layout pass, as shown here:

InvalidateMeasure();

The DefiningGeometry property simply returns pathGeoDst.

A Better Widened Path

After I had written the ParallelPath class, another class occurred to me. This class afforded me an opportunity to do something about a problem that's been bothering me for more than 15 years.

Sometimes when programmers start working with wide lines, they wonder if there's a way to outline the line. The whole concept sounds weird until you actually see it. But it is quite possible and supported from the very beginnings of Win32®. In WPF, the facility is built into the Geometry class in the form of the GetWidenedPathGeometry method.

Let's look at an example. Suppose you had a Geometry named geo, and you used it with a Path object, as in the following code:

path = new Path();
path.Data = geo;
path.Stroke = Brushes.Blue;
path.StrokeThickness = 40;

Figure 8 shows this Path object.

Figure 8 A Wide Line Drawn with Path
Figure 8 A Wide Line Drawn with Path

The GetWidenedPathGeometry of the Geometry class requires a Pen object, but it ignores the Brush property and uses only the physical dimensions and characteristics of the Pen, like so:

Pen pen = new Pen(null, 40);
PathGeometry pathGeo = geo.GetWidenedPathGeometry(pen);

Notice that I've specified the same pen thickness of 40 units as in the Path code. This new PathGeometry describes the outline of the Geometry as if it had been stroked with the Pen. Now let's draw a new Path figure with this new Geometry, but instead specifying a thinner pen and a fill brush:

path = new Path();
path.Data = pathGeo;
path.Stroke = Brushes.Red;
path.StrokeThickness = 6;
path.Fill = Brushes.Blue;

The result is shown in Figure 9. The original line has essentially been outlined.

Figure 9 A Widened Path Drawn with Path
Figure 9 A Widened Path Drawn with Path

Obviously this path-widening method works, but there are two problems with it. The first problem is that GetWidenedPathGeometry is a method. If you created a PathGeometry with some animatable points, you'd have to make repeated calls to GetWidenedPathGeometry to animate the widened path.

The second problem with path widening is the obvious artifacts. It's easy to see where they come from—they are a familiar sight to anybody who has ever worked with widened paths in Windows.

For these reasons, I created a WidenedPath class that has a Data property like Path and a WideningPen property for the widening parameters. The image in Figure 10 is created by the following code:

Figure 10 A Line Element in a Window
Figure 10 A Line Element in a Window

   WidenedPath widePath = new WidenedPath();
   widePath.Data = geo;
   widePath.WideningPen = pen;
   widePath.Stroke = Brushes.Red;
   widePath.StrokeThickness = 6;
   widePath.Fill = Brushes.Blue;

You might think that WidenedPath would be a fairly easy adaptation of ParallelPath, but there are other complications involved. If it is done properly, the widening of the path must take account of line caps and joins—the Pen properties StartLineCap, EndLineCap, and LineJoin. This requires the path-widening algorithm to potentially add curves to the path, as shown in Figure 11.

Figure 11 WidenedPath Class with Ends and Joins
Figure 11 WidenedPath Class with Ends and Joins

However, WidenedPath does not take account of the Pen properties DashStyle, DashCap, or MiterLimit. Although WidenedPath creates fewer artifacts than GetWidenedPathGeometry, it is not entirely immune from them. As with ParallelPath, you can minimize these artifacts by increasing the Tolerance property.

The Other Paradigm

Although I have focused on classes that use the "coordinate" paradigm implemented by the Shape class, I'm sure you haven't forgotten about the "autosize" paradigm.

Judging from some experimentation with Rectangle and Ellipse, these classes always return Transform.Identity from their GeometryTransform methods and instead base the size of the DefiningGeometry on information obtained by overriding the MeasureOverride and ArrangeOverride methods.

After experimenting with that approach in creating a RegularPolygon class, I tried coding the class as if it instead used the "coordinate" paradigm. I didn't attempt to take account of any coordinates or sizes, but instead based the polygon on a circle with the center at (0, 0) and the radius as 1, and then let WPF handle the rest.

The result was definitely not the same as Rectangle or Ellipse, but it varied the most when the Width and Height properties are set to NaN, and Stretch is set to Uniform or Fill. In many of these cases, RegularPolygon is visible while Ellipse is a tiny ball.

Included with the source code is a program called StretchExplore that lets you view Ellipse, RegularPolygon, Line, and PointLine figures under all combinations of Stretch, HorizontalAlignment, VerticalAlignment, Width, and Height settings. You can then decide for yourself whether my simple approach in RegularPolygon works in your own applications.

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a Contributing Editor to MSDN Magazine. His most recent book is 3D Programming for Windows.


.NET Matters: Asynchronous Stream Processing

.NET Matters
Asynchronous Stream Processing
Stephen Toub

Q I've read about some neat tricks you can play to ease the development of asynchronous implementations using C# iterators. (Jeffrey Richter was extolling the virtues of such techniques in a previous issue of MSDN® Magazine.) However, I was trying to figure out how to do similar things without such code transformations, and I was having a very hard time. For example, let's say I want to copy from one stream to another, using only the asynchronous methods on Stream. I understand how to issue the asynchronous requests once (to do one read and one write), but how can I loop such requests continually in order to process the whole stream? Is that even possible?

A Yes, it's possible. I'll show how, using your example, with both C# and Visual Basic® (noting that Visual Basic doesn't provide compiler support for iterators). To set the stage, Figure 1 shows a synchronous implementation of what we want to implement asynchronously, which could be called with code such as this:

Figure 1 Synchronous CopyStreamToStream (Close Figure 1)
public static void CopyStreamToStream(
        Stream source, Stream destination,
        Action<Stream, Stream, Exception> completed)
{
    byte[] buffer = new byte[0x1000];
    int read;
    try
    {
        while ((read = source.Read(buffer, 0, buffer.Length)) > 0)
        {
            destination.Write(buffer, 0, read);
        }
        if (completed != null) completed(source, destination, null);
    }
    catch (Exception exc)
    {
        if (completed != null) completed(source, destination, exc);
    }
}

FileStream input = ...;
FileStream output = ...;
CopyStreamToStream(input, output, (src,dst,exc) => {
    src.Close();
    dst.Close();
});

With the synchronous implementation, CopyStreamToStream will block until the stream has been copied and the completed callback has been executed. You can easily implement this as an asynchronous invocation in a few ways. One would be to explicitly queue to the ThreadPool a work item that calls this method:

ThreadPool.QueueUserWorkItem(delegate
{
    CopyStreamToStream(input, output, (src,dst,exc) =>
    {
        src.Close();
        dst.Close();
    });
});

Another would be to use asynchronous delegate invocation, which would have largely the same effect:

delegate void CopyStreamToStreamHandler(
    Stream source, Stream destination,
    Action<Stream, Stream, Exception> completed);
...
CopyStreamToStreamHandler handler = CopyStreamToStream;
handler.BeginInvoke(source, destination, (src,dst,exc) =>
{
    src.Close();
    dst.Close();
}, iar => handler.EndInvoke(iar), null);

But neither of those approaches is asynchronous in the way you're looking for. Yes, they're both asynchronous with respect to the thread starting their execution (meaning that the calling thread won't be blocked waiting for their completion); however, in both cases the CopyStreamToStream method is just being run on a thread from the pool, and all of the reads and writes it's issuing are synchronous with regard to the rest of the method call.

If I were to launch CopyStreamToStream asynchronously in this fashion 100 times, 100 threads (possibly reused) would be needed to execute those copies. Not only is that significant overhead, but given the number of threads available in the thread pool and the rate at which the thread count is increased, a significant number of the copies probably wouldn't begin execution until others had finished, increasing the overall time to completion.

What you're looking for is a true asynchronous implementation of CopyStreamToStream, where instead of issuing Read and Write calls on the Streams, you issue BeginRead and BeginWrite calls, with appropriately matched calls to EndRead and EndWrite.

As you can see in the synchronous implementation, you need to issue a read request against the source Stream, take the retrieved data and write it to the destination Stream, and then rinse and repeat. With the asynchronous implementation, rather than using a loop, as I did in the synchronous implementation, I'll use continuation-passing style to package everything that comes after the asynchronous request, and pass that as a closure to be executed when the asynchronous request completes. To begin, I issue an asynchronous read:

byte[] buffer = new byte[0x1000];
source.BeginRead(buffer, 0, buffer.Length, readResult =>
{
    int read = source.EndRead(readResult);
    ... // process read results in buffer here
}, null);

The BeginRead method accepts the same three parameters that the Read method accepts: the target buffer in which to place the data, the index into the buffer specifying where the data will be placed, and the amount of data to be read. The BeginRead method also takes two additional parameters: the first represents the delegate to be called when the read request completes, and the second is any state that I want to be available in the callback. Within the callback, I issue an EndRead call against the source, providing it with the IAsyncResult from the BeginRead call and allowing me to retrieve the result of the read. (EndRead will also throw any exceptions that may have occurred as a result of the read.) After that, I want to write that data out to the destination Stream:

int read = source.EndRead(readResult);
if (read > 0)
{
    destination.BeginWrite(buffer, 0, read, writeResult =>
    {
        destination.EndWrite(writeResult);
        ...
    }, null);
}

Again, rather than calling Write on the destination Stream, I call BeginWrite, passing it the same parameters I would have passed to Write, along with a delegate to be executed when the write request completes (as well as a state argument, which I'm ignoring). So far, so good. I've asynchronously issued a read request, and the thread on which I issued the request will not block waiting for the request to complete. When it does complete, I asynchronously issue a write request and the callback from the read request doesn't block waiting for the write request to complete.

But now I'm in a bind. In the BeginWrite callback, do I code another call to BeginRead in order to read more from the source stream? If so, what do I pass as the callback for that read? I can't explicitly code every read/write pair; there could be an infinite number of them, and that would make for a very long code file.

Instead, a good approach is to house the call to EndRead in a method that can be called through an AsyncCallback delegate. That method will serve as the callback for BeginRead. Then, in the BeginWrite callback, after I end the previous write, I can invoke the method that issues the next BeginRead request. To start, I issue a single call to BeginRead, passing the callback delegate I just described. That gives me an implementation like the following:

AsyncCallback rc = readResult =>
{
    int read = source.EndRead(readResult);
    if (read > 0)
    {
        destination.BeginWrite(buffer, 0, read, writeResult =>
        {
            destination.EndWrite(writeResult);
            source.BeginRead(buffer, 0, buffer.Length, rc, null);
        }
    }
}
source.BeginRead(buffer, 0, buffer.Length, rc, null);

All would appear to be well, until you try to compile this, at which point the friendly C# compiler will tell you:

error CS0165: Use of unassigned local variable 'rc'

In other words, I'm trying to use the 'rc' variable within my callback, and I'm initializing the 'rc' variable to my callback, but so as far as the C# compiler is concerned, 'rc' is undefined at the time. (For more details, see Eric Lippert's blog post at go.microsoft.com/fwlink/?LinkId=109263). The simple workaround is to first declare the 'rc' variable and initialize it to null. Then as a separate statement, assign the callback to 'rc':

AsyncCallback rc = null;
rc = readResult =>
{
    ...
};

The full implementation is shown in Figure 2. You'll notice some additional functionality added to this, over and above the transformation from Read/Write to BeginRead/BeginWrite. First, I've added exception handling to the implementation. As with the synchronous implementation, when an exception is caught, the completed delegate is called with the exception to notify the application that something went wrong and to allow the exception to be dealt with appropriately. Second, we're taking advantage of the AsyncOperation class, which was introduced in the Microsoft® .NET Framework 2.0. This class creates an instance of AsyncOperation using the AsyncOperationManager and captures the current SynchronizationContext, allowing me to invoke operations in the context of that SynchronizationContext. (For more information, see msdn.microsoft.com/msdnmag/issues/06/06/NETMatters/#qa7.)

Figure 2 Async CopyStreamToStream in C# (Close Figure 2)
public static void CopyStreamToStream(
    Stream source, Stream destination,
    Action<Stream,Stream,Exception> completed) {
    byte[] buffer = new byte[0x1000];
    AsyncOperation asyncOp = AsyncOperationManager.CreateOperation(null);

    Action<Exception> done = e => {
        if (completed != null) asyncOp.Post(delegate {
            completed(source, destination, e); }, null);
    };

    AsyncCallback rc = null;
    rc = readResult => {
        try {
            int read = source.EndRead(readResult);
            if (read > 0) {
                destination.BeginWrite(buffer, 0, read, writeResult => {
                    try {
                        destination.EndWrite(writeResult);
                        source.BeginRead(
                            buffer, 0, buffer.Length, rc, null);
                    }
                    catch (Exception exc) { done(exc); }
                }, null);
            }
            else done(null);
        }
        catch (Exception exc) { done(exc); }
    };

    source.BeginRead(buffer, 0, buffer.Length, rc, null);
}

For example, if CopyStreamToStream were invoked from the GUI thread of a Windows® Forms application, the captured SynchronizationContext would actually be a WindowsFormsSynchronizationContext, and calling Post on the resulting AsyncOperation would cause the delegate provided to execute on the GUI thread. This allows CopyStreamToStream to play nicely within whatever environment it may be running.

Now, at the beginning of my response I promised an asynchronous implementation in both C# and Visual Basic. As you've seen, while I didn't take advantage of iterators in C#, I did take advantage of other C# features, such as anonymous methods. And anonymous methods aren't yet supported in Visual Basic. (Visual Basic 2008 supports lambda expressions, but those are limited to single expressions.) Doing the same core task in Visual Basic is certainly feasible, but it requires more work, as shown in Figure 3. Most of the extra code here is associated with manually creating and passing around the closures that I was getting for free when using C#.

Figure 3 Async CopyStreamToStream in Visual Basic (Close Figure 3)
Public Shared Sub CopyStreamToStream( _
    ByVal source As Stream, ByVal destination As Stream, _
    ByVal completed As Action(Of Stream, Stream, Exception))

    Dim buffer(&H1000) As Byte
    Dim data = New CopyData With { _
        .Source = source, .Destination = destination, _
        .Buffer = buffer, .Completed = completed, _
        .AsyncOperation = AsyncOperationManager.CreateOperation(Nothing)}
    source.BeginRead(buffer, 0, buffer.Length, _
        AddressOf ReadCallbackSub, data)
End Sub

Private Shared Sub ReadCallbackSub(ByVal readResult As IAsyncResult)
    Dim copyData = CType(readResult.AsyncState, CopyData)
    Try
        Dim read = copyData.Source.EndRead(readResult)
        If read > 0 Then
            copyData.Destination.BeginWrite( _
                copyData.Buffer, 0, read, _
                AddressOf WriteCallbackSub, copyData)
        Else
            copyData.AsyncOperation.Post( _
                Function() Done(copyData, Nothing), Nothing)
        End If
    Catch ex As Exception
        copyData.AsyncOperation.Post( _
            Function() Done(copyData, ex), Nothing)
    End Try
End Sub

Private Shared Sub WriteCallbackSub(ByVal writeResult As IAsyncResult)
    Dim copyData = CType(writeResult.AsyncState, CopyData)
    Try
        copyData.Destination.EndWrite(writeResult)
        copyData.Source.BeginRead( _
            copyData.Buffer, 0, copyData.Buffer.Length, _
            AddressOf ReadCallbackSub, copyData)
    Catch ex As Exception
        copyData.AsyncOperation.Post( _
            Function() Done(copyData, ex), Nothing)
    End Try
End Sub

Private Shared Function Done( _
        ByVal copyData As CopyData, ByVal ex As Exception) As Boolean
    If copyData.Completed <> Nothing Then
        copyData.Completed(copyData.Source, copyData.Destination, ex)
    End If
    Return True
End Function

Private Class CopyData
    Public Source, Destination As Stream
    Public Buffer() As Byte
    Public AsyncOperation As AsyncOperation
    Public Completed As Action(Of Stream, Stream, Exception)
End Class

Regardless of whether I'm using the C# or Visual Basic implementation, I now have a truly asynchronous implementation that copies all of the data from one stream to another. The benefit of this, however, depends on how the streams themselves implement the asynchronous BeginRead and BeginWrite methods. The implementation of these methods on the base Stream class simply executes Read and Write from the ThreadPool using an asynchronous delegate invocation. So while each read and write will be asynchronous, from a performance perspective this will likely be slower overall than just doing the reads and writes synchronously.

Derived streams can choose to provide their own implementation of BeginRead and BeginWrite, and when they do, it's typically done because the domain for which they're intended has true asynchronous read and write support. For example, FileStream's implementation of BeginRead and BeginWrite provides access to the asynchronous I/O support available in the Windows file system. If you construct a FileStream by passing true for the useAsync constructor parameter, and if the version of Windows you're running supports asynchronous I/O, the FileStream's BeginRead and BeginWrite methods will be truly asynchronous. While data is being read from or written to the disk, no threads in your application will be blocked or utilized for these actions; only when read data is available or when a write has completed will the relevant callbacks be issued and a thread from the ThreadPool be used. This makes it possible to write scalable systems that minimize resource usage.

As you can see, while writing this kind of code by hand can create high-performance applications, doing so is also very tedious and error-prone. Hopefully in the future such constructs will be easier to create in mainstream languages like C# and Visual Basic.

F# has already made great progress down that road with its asynchronous workflows (see blogs.msdn.com/dsyme/archive/2007/10/11/introducing-f-asynchronous-workflows.aspx). There are also several libraries floating around the Web that take advantage of C# iterators to ease the pain of writing asynchronous code. An example of this is the Concurrency and Coordination Runtime (CCR), which you can read about at msdn.microsoft.com/msdnmag/issues/06/09/ConcurrentAffairs.

And as you mentioned in your question, Jeffrey Richter has created a more generic AsyncEnumerator implementation, which he started discussing in his column at msdn.microsoft.com/msdnmag/issues/07/11/ConcurrentAffairs and which he dives into further in his Concurrent Affairs column in next month's issue of MSDN Magazine.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager Lead on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.


{ End Bracket }: What is ALT .NET?

{ End Bracket }
What is ALT .NET?
Jeremy D. Miller

Last year David Laribee coined the term "ALT .NET" to describe a coalescing community of like-minded individuals within the larger world of the Microsoft® .NET Framework who felt a growing frustration that Microsoft tooling, guidance, and .NET culture at large did not reflect or support an important set of core values. In October 2007, some hundred-odd souls descended upon Austin, Texas, for the first ALT .NET Open Spaces event, turning the blogosphere buzzword into something more tangible and kicking off the formation of a real community.

So what is ALT .NET? And how does it differ from the .NET that we already know and love? What are these values that many of us think are missing? What are these alternative tools, techniques, and practices that ALT .NET'ers are espousing? Let's first examine the original tenets of being an ALT .NET developer.

  1. You're the type of developer who uses whatever works while keeping an eye out for a better way. One of the common topics at the ALT .NET event was closing gaps between requirements, testing, and code. There's still fat in the way we develop software that can be eliminated.
  2. You reach outside the mainstream to adopt the best of any community: Open Source, Agile, Java, Ruby. In no way does Microsoft or the .NET community have a monopoly on good software development. For instance, Agile processes and Design Patterns started with Smalltalk. Likewise, Inversion of Control tools and techniques originated in Java. And two fundamental Ruby on Rails principles—Don't Repeat Yourself and Convention over Configuration—are ones that we can adopt in .NET.
  3. You're not content with the status quo. Things can always be more elegant, more mutable, and of higher quality. We're all experimenting with techniques to more closely connect the coding and testing to the business domain. For example, Behavior Driven Development (BDD) refines and extends Test Driven Development (TDD) by specifying the intended behavior of the code in a more readable way than classic xUnit testing. And language-oriented programming has the potential to raise the abstraction layer up to the level of the domain logic.
  4. You realize that tools are great, but they only take you so far. It's the principles and knowledge that really matter. The best tools are those that embed the knowledge and encourage the principles (for example, ReSharper). Furthermore, you feel that the most important qualities of a solution are maintainability and sustainability. Maintainable code means good design. Good design arises from the skillful application of design knowledge. The .NET community has been placing too much focus on learning API and framework details and not enough emphasis on design and coding fundamentals.

Part of the catalyst for the raising of the ALT .NET banner was a demonstration of a beta of the new Entity Framework in March 2007. I, along with several others, was disappointed at certain elements of the Entity Framework, specifically the lack of "Persistence Ignorance" and the tight coupling to the database. At that time, the Entity Framework did not appear suitable for TDD, Domain Driven Design, or Continuous Design—all things that I consider valuable for maintainable development.

On the other hand, the new Model View Controller (MVC) Framework for ASP.NET development is exciting because it explicitly supports and enables the values I feel are important for successful software development. The MVC Framework clearly reflects the values of maintainability and testability with an emphasis on Separation of Concerns and testability throughout the design of the Framework.

It's easy to be cynical or apathetic about the state of development. However, instead of lamenting that Johnny can't code and complaining that it's hard to find good developers to hire, perhaps we should admit our main problem: we're terrible at growing strong developers.

Many developers would be happy to use Object/Relational Mapping and TDD if they just knew where to start. Likewise, folks already practicing these techniques want to get better by interacting with other practitioners. My chief hope for the ALT .NET movement is that it creates a community that assists both groups, helping them all become strong developers. To that end, the Altnetpedia Web site (altnetpedia.com) has just opened for business with the beginning of a list of resources for learning how to apply these techniques in .NET. The altnetconf message board (tech.groups.yahoo.com/group/altdotnet) is already up and going strong as a place for .NET practitioners to discuss development topics. Stop by!

Jeremy D. Miller is the Development Manager for BayerSoftware in Austin, TX. See his blog at codebetter.com/blogs/jeremy.miller.