 MSDN Magazine March 2006

MSDN Magazine March 2006

Winning Forms: Practical Tips For Boosting The Performance Of Windows Forms Apps

This article discusses techniques you can use to ensure that Windows Forms-based apps provide optimal performance to match the rich UI responsiveness they're known to provide.

Milena Salman Text Rendering: Build World-Ready Apps Using Complex Scripts In Windows Forms Controls

The System.Windows.Forms.TextRenderer class provides support for complex scripts in Windows Forms controls so you can render text the way you want and support international locales.

Miguel A. Lacouture Outlook Add-Ins: Improve Your Outlook With Visual Studio Tools For Office

Microsoft Visual Studio Tools for Office 2003 supported only Microsoft Word and Excel. The new version, however, has the tools you need to create managed code add-ins for Outlook 2003.

John R. Durant Office Unbound: Bring Your Documents To Life With Data Binding In Visual Studio Tools For Office

Thanks to Visual Studio 2005 Tools for the Microsoft Office System, Windows Forms controls can be bound to databases, Web services, or objects and added to a workbook or document. Find out how.

Eric Carter and Eric Lippert Agile Development: Extend Team Foundation Server To Enable Continuous Integration

How does Visual Studio 2005 Team System and Team Foundation Server facilitate the process of agile development and continuous integration? Here Ben Waldron explains it all.

Ben Waldron Reporting Services: Deliver SQL Server Reports To SharePoint To Enhance Team Collaboration

The solution covered here includes a custom SharePoint Web service that accepts SQL Server-generated report files and, through the WSS object model, stores the file in the appropriate library for enhanced collaboration possibilities.

Ed Hild The Perfect Host: Create And Host Custom Designers With The .NET Framework 2.0

The .NET Framework 2.0 introduces a set of classes that can be used to host designers right out of the box. With the understanding of designers you'll glean from this article, you'll be ready to host them in your own apps.

Dinesh Chandnani Editor's Note: When Did Clients Get So Smart?

This issue marks the sixth anniversary of our merger, when Microsoft Systems Journal and Microsoft Internet Developer combined to become MSDN Magazine. We've been around to witness the birth and growth of the Microsoft .

Toolbox: Automated Build Process, Real-World E-Mail Tasks, and More

While the Visual Studio® Build menu's Build Solution option is the usual way developers compile their current work on a project, the process of building the complete solution for testing, deployment, or production often requires many more steps.

Scott Mitchell Advanced Basics: Set Word Document Properties Programmatically

At the beginning of another lovely day of writing courseware in mad pursuit of unrealistic deadlines, I received a frantic call from a business partner. He was at the end of a long consulting project and had several hundred Microsoft® Word documents, all of which required their document properties to be set identically, except the Title property of the document, which was to be based on the document file name, minus the .

Ken Getz CLR Inside Out: Ensuring .NET Framework 2.0 Compatibility

If we learned only one thing about compatibility in the past few years, it is that compatibility is much more than avoiding breaking changes. On the Microsoft®. NET Framework and Visual Studio® teams, we do our part to ensure that the products we build are stable platforms that developers can truly rely on.

Jesse Kaplan Data Points: Using XQuery, New Large DataTypes, and More

SQL Server 2005 introduces a lot of new features, but it also enhances the popular and oft-used Transact-SQL (T-SQL) language. Changes include the introduction of new datatypes to store large values using the MAX indicator, the integration of enhanced XML querying and data modification with XQuery, and the new XML datatype.

John Papa Cutting Edge: Windows Workflow Foundation

In the January 2006 issue, Don Box and Dharma Shukla introduced Windows® Workflow Foundation and discussed the overall architecture of the framework and its constituent components (see WinFX Workflow: Simplify Development With The Declarative Model Of Windows Workflow Foundation).

Dino Esposito Test Run: Determining .NET Assembly and Method References

Before you can test any software system effectively, you must understand the system under test. If the system includes the Microsoft® . NET Framework, understanding the system under test includes understanding its assembly and method dependencies.

James McCaffrey Concurrent Affairs: Build a Richer Thread Synchronization Lock

In my last column, I showed the various thread synchronization mechanisms employed by the Microsoft® . NET Framework (see Concurrent Affairs: Performance-Conscious Thread Synchronization). I then examined the performance characteristics of all these mechanisms and determined that the Interlocked methods performed the best because the calling thread never has to transition to kernel mode.

Jeffrey Richter Bugslayer: Strengthening Visual Studio Unit Tests

Visual Studio 2005 brought so many new features to the table that it can seem almost overwhelming. One of the most exciting additions is the new unit testing features found in the Test menu on the main menu bar.

John Robbins .NET Matters: Abortable Thread Pool

Stephen Toub C++ at Work: Event Programming, Part 2

Last month I answered a question about implementing native events in C++ (see C++ At Work: Event Programming). I discussed events in general and showed how to use an interface to define event handlers for your class that clients must implement to handle events.

Paul DiLascia { End Bracket }: Legal Doesn’t Think the Way You Do

If you develop software for a large company, you've undoubtedly experienced the joys of dealing with "Legal. " The meetings, the endless explanations, the long waits. I'm here to report from the other side.

Don McGowan


Winning Forms: Practical Tips For Boosting The Performance Of Windows Forms Apps

Winning Forms

Practical Tips For Boosting The Performance Of Windows Forms Apps

Milena Salman
This article discusses:
  • Improving application startup time
  • Tuning up control creation and population
  • Improving painting performance
  • Rendering text and images
  • Efficient resource management
This article uses the following technologies:
Windows Forms, .NET Framework 2.0
Code download available at: WindowsFormsPerformance.exe (605KB)
Contents
Snappy Startup
Faster UI Loading
Control Population
Data Binding Optimization
Lightweight Layout
Painting Performance
Text and Images
Applying Painting Techniques
Resource Management
Conclusion

W
indows Forms allow you to build a rich and responsive user interface for your applications. In this article I'll discuss a number of techniques you can use to ensure that Windows® Forms-based apps provide optimal performance as well. I'll discuss common performance-critical scenarios such as startup, control population, and control painting. Plus, I'll discuss how to design and code for performance in your application. Together these techniques should give you a good, basic foundation for getting the most out of your UIs.

Snappy Startup

Startup time is an important performance metric for most applications. Quick startup leaves the user with a favorable perception of your application's performance and usability. There are two terms that describe how an application starts up: warm startup and cold startup. If you start any managed application immediately after restarting your computer, then close the app and start it again. You will usually notice a pretty significant difference in startup time. The first (cold) startup will have to perform disk I/O operations to bring all of the required pages into memory. The second (warm) startup is significantly faster because it reuses the pages that are already present in memory.

To get the warm startup effect, you don't necessarily have to have previously run the same application. Just starting any managed application will bring into memory many pages from the Microsoft® .NET Framework binaries, and this will improve startup time for any managed applications that start later.

When designing for performance, it is important to distinguish between these two scenarios and to set separate goals for them. While warm startup can be improved by reducing CPU utilization, cold startup is affected mostly by disk I/O. Therefore, some optimizations improve some scenarios and not others.

I will start with some general performance advice applicable to a starting application written in any managed code. Then I'll give you some tips specific to improving the performance of Windows Forms-based apps. For detailed guidelines on improving the startup performance of managed applications, see the CLR Inside Out column by Claudio Caldato in the February 2006 issue of MSDN®Magazine at CLR Inside Out: Improving Application Startup Time.

One way to improve cold startup is to reduce the number of DLLs that you load into your application. This directly affects the number of pages that have to be touched during application startup. Additionally, reducing the number of modules loaded reduces the CPU overhead associated with loading a module, also improving warm startup times.

You can diagnose how modules are being loaded into your application using the Load Module event filter within the WinDbg debugger in the Platform SDK. Alternatively you can use the following WinDbg command to see why a specific module was loaded:

sxe ld:foo.dll
Review the call stacks and see if some of the module loads could be avoided. You can also use the "ca ml" option of the MDbg managed debugger, available as part of the .NET Framework SDK. In some cases you can avoid loading additional modules by simply refactoring your code. If your application loads lots of DLLs that you own, consider merging them into fewer bigger DLLs where it makes sense to do so.

Just-in-time (JIT) compilation of methods can consume many CPU cycles and easily become a bottleneck during application startup. To avoid this overhead you can precompile assemblies using the Native Image Generator, NGen.exe (see Speed: NGen Revs Up Your Performance with Powerful New Features). NGen performs all of the work of the JIT compiler, but accomplishes this work up front, then persists those changes to disk, saving CPU cycles at run time. Using NGen to precompile your binaries has an additional performance benefit: native code pages can be shared between processes, while JIT-compiled code pages are private to a process and cannot be reused.

Despite the advantages, precompilation is not always a a panacea for startup time. While warm startup will likely benefit from NGen precompilation, cold startup may not improve or might even be a little slower because the pages containing compiled code now need to be loaded from disk. However, if JIT compilation is eliminated completely, you might see a benefit even for cold startup since the pages required for the JIT compiler itself would not need to be loaded. It is good to measure both the cold and warm startup times to assess the impact of NGen.

Another technique is to install strong-named assemblies in the Global Assembly Cache (GAC) to avoid strong-name signature verification, a costly process that touches every page of the assembly before it is loaded. If you decide to use NGen, it's even more important to have strong-named assemblies in the GAC and avoid DLL rebasing.

Every binary (DLL or EXE) has a preferred base address—a location in virtual memory where it is to be loaded. This address is specified at build time. If you use the Visual Basic® or C# compiler, then all of your binaries will get the same base address (0x400000) if no explicit build directions to the contrary are given. Thus, at load time, your EXE will be placed at this address and all of your DLLs will be have to be placed elsewhere in virtual memory (rebased), since their preferred base address is occupied.

When a DLL is rebased, the loader updates all of the absolute addresses in the DLL to reflect the new load address. This means every page that contains an address that needs to be adjusted will be touched when the DLL gets rebased. Moreover, in order to become writable, a page needs to be copied and backed by a pagefile. At that point the page is private to the process and cannot be shared with other processes.

To avoid the rebasing performance hit, you can explicitly specify a preferred base address by using the /baseaddress compiler switch for each DLL in your application. Assign base addresses to the DLLs beginning from some predetermined starting point (for example, 0x10000000), and leave gaps between the addresses that are large enough to give some room for growth to the DLL that will lie in that range.

NGen images tend to be significantly bigger than Intermediate Language (IL) images and this increases the impact of rebasing. NGen sets the same base address for the image it creates as the address specified in the IL image. Therefore, if you decide to NGen your assemblies, then when you assign base addresses for your IL images you should leave enough space to fit NGen images. Usually you will need two or three times more space for an NGen image than for the equivalent IL image.

You can check that your base address assignment worked properly by verifying actual load addresses and comparing them with preferred load addresses. To accomplish this, you can use the task list sample application, TList.exe, and the Microsoft COFF Binary File Dumper, Dumpbin.exe. This command shows the actual load address for each DLL loaded in the context of the application with process ID 1240:

tlist 1240
Dumpbin allows you to check the preferred base address for your DLL, as shown here:
dumpbin /headers test.dll
Looking at the output of this command, you'll see OPTIONAL HEADERS with an image base listed, something like this:
75F70000 image base (75F70000 to 75F78FFF).
This means the preferred base address of test.dll is 75F70000 and that the address range of 75F70000 to 75F78FFF should be available in order to load this DLL at its preferred base address.

Faster UI Loading

The perception of startup performance depends largely on the delay until the first UI is shown. You should minimize the logic required to display the UI. This includes any work performed in the Load event of the form because this is evaluated as the form is in the process of showing itself.

If you use expensive network or database calls, make them asynchronous on a different thread to avoid blocking the UI. In Windows Forms 2.0, the BackgroundWorker component can be used to push work onto a background thread. In Figure 1, BackgroundWorker is used to recursively find all files matching a given pattern (for example, "*.txt") in a given folder and its subfolders. The background thread updates the UI thread on its progress, and the UI thread adds the files or subfolders to a TreeView representation in response to progress updates. The resulting TreeView is filtered so that subfolders that don't contain matching files are not shown.

Figure 1 Using a Background Thread (Close Figure 1)
Imports System.IO

Public Class Form1
    Private DirInfo As DirectoryInfo
    Private Pattern As String

    Private Sub Form1_Load(ByVal sender As System.Object, _
            ByVal e As System.EventArgs) Handles MyBase.Load
        DirInfo = My.Computer.FileSystem.GetDirectoryInfo("C:\\")
        Pattern = "*.txt"
        Me.Text = "Searching for " & Pattern & " files"
        Me.TreeView1.Sorted = True
        Me.BackgroundWorker1.WorkerReportsProgress = True
        Me.BackgroundWorker1.RunWorkerAsync()
    End Sub

    Private Sub BackgroundWorker1_DoWork( _
        ByVal sender As System.Object, _
        ByVal e As System.ComponentModel.DoWorkEventArgs) _
        Handles BackgroundWorker1.DoWork

        For Each f As FileInfo In DirInfo.GetFiles(Pattern)
            Dim treeNode As TreeNode = New TreeNode(f.ToString)
            Me.BackgroundWorker1.ReportProgress(0, treeNode)
        Next

        For Each SubDir As DirectoryInfo In DirInfo.GetDirectories()
            Dim treeNode As TreeNode = GetMatchingFiles(SubDir)
            If Not treeNode Is Nothing Then
                Me.BackgroundWorker1.ReportProgress(0, treeNode)
            End If
        Next
    End Sub

    Private Sub BackgroundWorker1_ProgressChanged(_
        ByVal sender As Object, _
        ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
        Handles BackgroundWorker1.ProgressChanged

        Me.TreeView1.Nodes.Add((CType(e.UserState, TreeNode)))
    End Sub

    Private Sub BackgroundWorker1_RunWorkerCompleted( _
        ByVal sender As Object, _
        ByVal e As System.ComponentModel.RunWorkerCompletedEventArgs) _
        Handles BackgroundWorker1.RunWorkerCompleted

        Me.Text = "Ready"
    End Sub

    Private Function GetMatchingFiles(ByVal dir As DirectoryInfo) _
        As TreeNode

        Dim treeNode As TreeNode = Nothing
        For Each f As FileInfo In dir.GetFiles(Pattern)
            If treeNode Is Nothing Then
                treeNode = New TreeNode(dir.Name)
            End If
            treeNode.Nodes.Add(New TreeNode(f.ToString))
        Next
        For Each SubDir As DirectoryInfo In dir.GetDirectories()
            Dim subNode As TreeNode = GetMatchingFiles(SubDir)
            If Not subNode Is Nothing Then
                If treeNode Is Nothing Then
                    treeNode = New TreeNode(dir.Name)
                End If
                treeNode.Nodes.Add(subNode)
            End If
        Next
        Return treeNode

    End Function
End Class

The advantage of using a background thread here is that the UI is fully accessible while the search proceeds. Your user can scroll though the list of files found so far and expand subfolder nodes that have been added to the TreeView.

Operations that must be performed, but are not needed for displaying the first UI, can be done while the system is idle or on demand after the first UI is shown. For example, if you have a TabControl, populate only the topmost page on startup and retrieve the information for other pages when required. The code in Figure 2 includes a form with a TabControl containing six TabPages. I put all of the controls required for each TabPage on a UserControl and added this to a control collection of the TabPage when the TabPage is going to be shown. For simplicity I used the same UserControl for each of the TabPages and only slightly changed the context of some child controls, but you can easily have a separate UserControl corresponding to each TabPage.

Figure 2 Initializing Controls on Idle (Close Figure 2)
public partial class Form2 : Form
{
    int currentIndex = 1;
    public Form2(bool onDemand)
    {
        InitializeComponent();
        if (!onDemand) InitTab(-1);
        else
        {
            InitTab(0);
            Application.Idle += new EventHandler(Application_Idle);
        }
    }

    private void InitTab(int index)
    {
        if (index == -1)
        {
            for (int i = 0; i < tabControl1.TabCount; i++)
            {
                InitTab(i);
            }
            return;
        }

        // if the TabPage has already been populated
        if (tabControl1.TabPages[index].Controls.Count > 0) return;

        UserControl control = new myTabPage(index);
        if (control != null)
        {
            control.Dock = DockStyle.Fill;
            tabControl1.TabPages[index].Controls.Add(control);
        }
    }

    private void tabControl1_SelectedIndexChanged(
        object sender, System.EventArgs e)
    {
        InitTab(tabControl1.SelectedIndex);
    }

    private void Application_Idle(object sender, EventArgs e)
    {
        if (currentIndex >= tabControl1.TabCount)
        {
            Application.Idle -= new EventHandler(Application_Idle);
        }
        else InitTab(currentIndex++);
    }
}

For the topmost page I added the UserControl in Form's constructor. For other pages I did it in response to firing of the TabControl.SelectedIndexChanged event when the corresponding page is going to be shown. The user will experience some delay when a TabPage is selected for the first time. Then the next time it is selected, there's no delay since the TabPage has already been initialized.

In this example, I've intentionally slowed down the creation of the TabPages by showing a ListView with a large number of items. If I initialize all the TabPages up front, the time required to show a form is nine seconds on my system (Pentium III, 800 MHz, 512 MB RAM). By filling the TabPages on demand, showing each TabPage the first time takes about 1.5 seconds. If only the topmost page is populated, the entire form is shown in 1.5 seconds.

I can push this a little further by listening for the Application.Idle event and populating one TabPage at a time when the Idle event occurs. When the form is shown or a new TabPage is shown, it will take a user some time to look at the form and move the mouse to select another page. During this organic delay in the application operation, the Idle event will be processed and additional TabPages will be populated.

If other techniques fail to improve startup performance sufficiently, you might consider using splash screens and progress bars to keep users informed, like this example at Creating a Splash Screen Form on a Separate Thread. If you write your app in Visual Basic, you can use the My.Application.SplashScreen property of the .NET Framework 2.0 (see My.Application.SplashScreen Property).

Control Population

Many Windows Forms Controls such as ListView, TreeView, and Combobox display collections of items. Adding too many items to these collections can cause a bottleneck if no optimizations are applied. Fortunately, several control population improvements were incorporated into Windows Forms in the .NET Framework 2.0. These include the use of the BeginUpdate and EndUpdate methods internally within AddRange methods, optimizing TreeView population with AddRange, and optimizing sorted population of all controls.

The most common reason for slow control population is the repainting of the control after each change. A number of Windows Forms controls implement BeginUpdate and EndUpdate methods, which suppress repainting while the underlying data or control properties are manipulated. Using these methods allows you to make significant changes to your control (like adding items to its collection) while avoiding constant repainting. Here's an example of using these methods:

listView1.BeginUpdate();
for(int i = 0; i < 10000; i++)
{
    ListViewItem listItem = new ListViewItem("Item"+i.ToString() );
    listView1.Items.Add(listItem);
}
listView1.EndUpdate();

Bulk operations are always more efficient than individual changes. The preferred way to add items to the collections of controls such as ComboBox, ListView, or TreeView is to use the AddRange method, which lets you add an array of pre-created items at one time. Windows Forms does all of the possible optimizations to make this operation efficient. Often this just means that BeginUpdate and EndUpdate will be called for you. In some cases, however, additional optimizations will be done. For the TreeView control, for example, these optimizations are significant.

ListView population works considerably faster if it is performed after the ListView's handle is created. If you need to show a form having a ListView with many items, add items to ListView in the Form.Load or Form.Show event handler. At this point the ListView handle is created already. If you add ListView items in the Form's constructor (that is, before the ListView handle is created), the items will be added quickly, but showing the form will take significant time due to slow ListView rendering.

Unlike ListView, TreeView is populated significantly faster if nodes are added before the handle is created. If you use TreeView.AddRange you will not notice the difference; your TreeView will be populated quickly regardless of when you do it. However if you use the Add method and add nodes in the form's constructor before the TreeView's handle is created, your TreeView will be populated and shown as quickly as if you had used the AddRange method.

Data Binding Optimization

When populating databound controls such as ComboBox or ListBox, it is more efficient to set the DataSource property last, after ValueMember and DisplayMember are set. Otherwise your control will be repopulated as a result of the ValueMember change:

comboBox1.ValueMember = "Name";
comboBox1.DisplayMember = "Name";
comboBox1.DataSource = test;
For this reason, the code just shown is more efficient than this:
comboBox1.DataSource = test;
comboBox1.ValueMember = "Name";
comboBox1.DisplayMember = "Name"
BindingSource.SuspendBinding and BindingSource.ResumeBinding are two methods that allow the temporary suspension and resumption of data binding. SuspendBinding prevents changes from being pushed into the data source until ResumeBinding is called.

These methods are designed to be used with simple bound scenarios such as TextBox or ComboBox data binding. The code in Figure 3 demonstrates that the iteration that suspends and resumes binding works approximately five times faster than the iteration that does not.

Figure 3 Suspending and Resuming Data Binding (Close Figure 3)
private void button1_Click(object sender, EventArgs e)
{
    DataTable tbl = this.bindingSource1.DataSource as DataTable;
    Stopwatch sw1 = new Stopwatch();
    sw1.Start();
    for (int i = 0; i < 1000; i++)
    {
        tbl.Rows[0][0] = "row " + i.ToString();
    }
    sw1.Stop();

    Stopwatch sw2 = new Stopwatch();
    sw2.Start();
    this.bindingSource1.SuspendBinding();
    for (int i = 0; i < 1000; i++)
    {
        tbl.Rows[0][0] = "suspend row " + i.ToString();
    }
    this.bindingSource1.ResumeBinding();
    sw2.Stop();
    MessageBox.Show(String.Format("Trial 1 {0}\r\nTrial 2 {1}",
        sw1.ElapsedMilliseconds, sw2.ElapsedMilliseconds));
}

private void Form1_Load(object sender, EventArgs e)
{
    DataTable tbl = new DataTable();
    tbl.Columns.Add("col1");
    tbl.Rows.Add("one");
    tbl.Rows.Add("two");
    this.bindingSource1.DataSource = tbl;
    this.textBox1.DataBindings.Add("Text", this.bindingSource1, "col1");
}

Controls that implement complex data binding, such as the DataGridView control, update their values based on change events such as ListChanged, so calling SuspendBinding will not prevent them from receiving changes to the data source. You can use these methods in complex binding scenarios if you suppress ListChanged events by setting the RaiseListChangedEvents property to false.

Lightweight Layout

Changes such as resizing or realigning a control can also cause problems. For example, when child controls are added to a form or ToolStrip, a Control.Layout event is fired. A change to the size or location of a child control will cause a Layout event on a parent control. Font size changes will also cause a Layout event. In response to the Layout event, the form scales and arranges controls. Having too many Layout events processed while you create or resize your control may have a significant performance impact.

Component code generated by the Windows Forms designer starts with SuspendLayout and ends with ResumeLayout. This is done to avoid performing layout on the form while it is being created and populated with controls. The SuspendLayout method allows multiple actions to be performed on a control without generating a Layout event for each change. Use this technique whenever possible to minimize the number of Layout events.

Remember that SuspendLayout only prevents Layout events from being performed for that particular control. If controls are added to a panel, for example, SuspendLayout and ResumeLayout must be called for the panel and not for the parent form.

Changing properties such as Bounds, Size, Location, Visible, and Text for AutoSize controls will fire a Layout event. It will be even more expensive if these properties are changed in Form.Load, since all handles are created by then and thus many messages will be processed. This is another place where you should add SuspendLayout and ResumeLayout to prevent extra Layout events from occurring. If possible, make all of these changes within InitializeComponent—this way only one Layout event is ever needed.

Several properties dictate the size or location of a control: Width, Height, Top, Bottom, Left, Right, Size, Location, and Bounds. Setting panel1.Width and then panel1.Height causes twice the work of setting them both together via panel1.Size. Running the code in Figure 4 demonstrates the difference.

Figure 4 Testing Layout Performance (Close Figure 4)
private void button1_Click(object sender, EventArgs e)
{
    Stopwatch sw = new Stopwatch();

    sw.Start();
    for (int i = 1; i < 1000; i++) {
        panel1.Width = i;
        panel1.Height = i;
    }
    sw.Stop();

    Stopwatch sw2 = new Stopwatch();
    sw2.Start();
    for (int i = 1; i < 1000; i++) {
        panel1.Size = new Size(i, i);
    }
    sw2.Stop();
    MessageBox.Show(String.Format("Trial 1 {0}\r\nTrial 2 {1}",
        sw.ElapsedMilliseconds, sw2.ElapsedMilliseconds));
}

If the handles are created you will notice the difference, even if you use SuspendLayout and ResumeLayout calls. SuspendLayout only prevents Windows Forms OnLayout from being called. It will not prevent messages about size changes from being sent and processed. You should try to set the property that reflects the most information you have. If you're just changing the size, set Size; if you're changing the size and location, change Bounds.

Painting Performance

If your Windows Forms application is drawing-intensive, it could benefit from enhanced painting performance. First, minimize the work that is done on every Paint event. Make your Paint event handler as lightweight as possible by moving any expensive operations out of it. For example, if you need to render text based on your current ClientSize, you can create the appropriately sized Font object in a Resize event handler instead of on every Paint event. If you need to create a Brush for drawing, you could cache it instead of recreating it repeatedly.

When you want your control to be redrawn, you can call its Invalidate method. The entire control will be redrawn if you do not pass any arguments to this call. In many cases you can optimize drawing performance by carefully calculating the area that needs to be repainted and passing that area as an argument to Invalidate.

If this does not work well enough, you might want to use the ClipRectangle structure that is included in the PaintEventArgs parameter of the OnPaint event. For example, you can draw a part of an image that was invalidated by doing the following:

Protected Overrides Sub OnPaint(_
    ByVal e As System.Windows.Forms.PaintEventArgs)

    e.Graphics.DrawImage(Me.RenderBmp, e.ClipRectangle, _
        e.ClipRectangle, GraphicsUnit.Pixel)

End Sub
Using this technique, however, frequently requires careful calculations (such as if scrolling or scaling is involved).

Windows Forms and the underlying Win32® architecture expose two layers of painting for any control: background and foreground. The Control.OnPaintBackground function is responsible for drawing background effects (typically background image and back color) and Control.OnPaint function is responsible for drawing foreground effects (images and text). If your control is not using the background image or background color, but instead custom paints everything in OnPaint, using the Opaque control style may help performance by skipping over painting logic that is not being used. In particular, if ControlStyles.Opaque is set to true on the control, the OnPaintBackground function will be skipped, as it is assumed that the OnPaint function will perform all the painting work (including painting the background). Additionally, this style might be advantageous when your control is completely covered by other controls and its background need not be painted.

Text and Images

You can measure and draw text on a Windows Forms Control in the .NET Framework 2.0 using the TextRenderer class. TextRenderer has two methods, MeasureText and DrawText, each with several overloads. The efficiency of your rendering operations depends on the overloads you choose and the options that you set in the TextFormatFlags argument.

For measuring single-line strings, you're better off not using the TextFormatFlag.WordBreak flag. When this flag is set, GDI executes a reliable but costly algorithm to determine the places the text needs to be broken into. Similarly, do not use the TextFormatFlags.PreserveGraphicsClipping and TextFormatFlags.PreserveGraphicsTranslateTransform options if no clipping or transforms have been applied to the Graphics object because these options cause some expensive calculations to occur.

It's better to use the TextRenderer method overloads that do not get IDeviceContext as an argument. These methods are more efficient because they use a cached screen-compatible memory device context rather than retrieving the native handle for device context from the internal DeviceContext and creating an internal object to wrap it.

If your application displays image files and contains an alpha component, blending, you can improve performance significantly by prerendering the images into a special premultiplied bitmap format. If an image has an alpha component, its color components should be multiplied by alpha when the image is displayed. Prerendering an image with premultiplied alpha eliminates several (three for RGB images) multiplication operations for each pixel in the image.

To use this technique, first load the image from a file, then render it to a bitmap with PixelFormat.Format32bppPArgb. Figure 5 uses this technique to set the Background image on a control.

Figure 5 Rendering a Background Image (Close Figure 5)
Public Overrides Property BackgroundImage() As System.Drawing.Image
    Get
        Return Me.RenderBmp
    End Get
    Set(ByVal value As System.Drawing.Image)
        Me.RenderBmp = New Bitmap(Me.Width, Me.Height, _
            Imaging.PixelFormat.Format32bppPArgb)
        Dim g As Graphics = Graphics.FromImage(Me.RenderBmp)
        g.InterpolationMode = Drawing.Drawing2D.InterpolationMode.High
        g.DrawImage(value, New Rectangle(0, 0, Me.RenderBmp.Width, _
            Me.RenderBmp.Height))
        g.Dispose()
    End Set

End Property

In Windows Forms 2.0, the BackgroundImage property has a companion property called BackgroundImageLayout. In prior versions of the .NET Framework, the background image was automatically tiled by Windows Forms, letting the image repeat throughout the client area. The new BackgroundImageLayout property allows you to set your background image layout to Tile, Center, Stretch, or Zoom. To preserve application compatibility with previous versions of the .NET Framework, Tile layout is still the default.

In addition to adding a rich set of background image features to Windows Forms, the nondefault values of BackgroundImageLayout improve performance of your background image painting. The major drawback to the Tile setting is that a TextureBrush is required to repeat the image pattern. The creation of the TextureBrush is expensive as it involves scanning the entire image. Other layouts simply make use of the Graphics.DrawImage method, resulting in much better performance.

For layouts other than Tile, setting the BackgroundImage and BackgroundImageLayout properties may automatically turn on the DoubleBuffered property. In particular, if the image has some sort of transparency (as detected through ImageFlagsHasAlpha), the control will start using double buffering to increase performance. For Tile layout, the DoubleBuffered property is not turned on automatically, but you can always turn it on manually.

Double buffering is a technique used to make drawing faster and appear smoother by reducing flicker. The basic idea is to take the drawing operations used to paint your control and apply them to an off-screen buffer. Once all of the drawing operations have finished, this buffer is drawn as a single image onto the control. This usually reduces flicker and makes the application seem faster. This behavior can be achieved in the .NET Framework 2.0 by setting a control style to OptimizedDoubleBuffer, which is equivalent to setting the DoubleBuffer and UserPaint styles in previous versions of the Framework.

To fully enable double buffering, you must also set AllPaintingInWmPaint to true. When it is set, the window message WM_ERASEBKGRND is ignored, and both OnPaintBackground and OnPaint methods are called directly from the WM_PAINT message. If you set DoubleBuffering and AllPaintingInWmPaint to true, then OnPaintBackground and OnPaint will be called with the same buffered graphics object and everything will paint off-screen together and update all at once.

To benefit from double buffering you have to set both these styles to true or set the DoubleBuffered property on your control, like this:

SetStyle(ControlStyles.OptimizedDoubleBuffer |
    ControlStyles.AllPaintingInWmPaint, true);
// OR
DoubleBufferred = true;  // sets both flags
It is important to mention here that setting the DoubleBuffered property is not always the optimal solution to painting problems. This property should be used with care, depending on your application and targeted scenarios. The main drawback of setting DoubleBuffered is that it results in a significant amount of memory being allocated for painting. If you have to use double buffering, consider also invalidating small parts of the control so that only the most important parts need to be repainted.

Another side effect of setting AllPaintingInWmPaint to true can be a visible white trace over the child controls of your control when a window belonging to another application is moved across your window. This can happen because child control windows get many WM_ERASEBKGND messages and only one WM_PAINT message. Since WM_ERASEBKGND messages are ignored, repainting happens too rarely for them.

Applying Painting Techniques

The screen in Figure 6 demonstrates animation of text over a user control for which a background image is set. You can set different painting options to see how they affect painting smoothness and speed (this sample application is available for download from the MSDN Magazine Web site).

Figure 6 Sample Performance-Testing Application
Figure 6 Sample Performance-Testing Application

The main form has a user control called DrawingSurface, which has a background image, some text ("Hello World!" by default), and two timers. The first timer controls animation speed and is set to 10ms by default. This can be changed through the Interval setting. When the interval elapses, the text is redrawn in a different position on the drawing surface. Depending on the animation effect, the text is either bouncing or spinning.

The second timer is used to update actual animation speed displayed in the lower-right corner of the drawing surface. By actual animation speed I mean the number of paint events handled during the previous second. Actual animation speed might be lower than the desired animation speed. For example, a 10ms interval means that you would like the text to change its position 100 times per second.

The Background Image Layout options let you compare drawing performance between Tile and other image layout flags. I chose Center layout just as an example (Zoom or Stretch would have the same effect as Center). In this application I realigned the image to fit the drawing surface size independently of the image layout chosen. However, when the background is painted, Tile layout affects performance. Also when Center layout is chosen, the DoubleBuffered option is set automatically for you. That isn't the case when Tile layout is chosen.

You can see how different options affect performance by observing flickering effects and by checking the actual animation speed in the lower-right corner of the drawing surface. Animation speed depends on the drawing options, on the update interval you set, and on physical limitations of your machine.

In my tests, the best results were achieved when Double Buffering and Smart Invalidation techniques were applied, Background Image Layout was set to Center (or, more specifically, any layout other than Tile), and the BackgroundImage was prerendered using PixelFormat.Format32bppPArgb. Painting is smooth and flicker-free when all of these techniques are used. Plus, animation is faster.

Resource Management

The physical memory used by your application is an important performance metric. You should use the least possible memory and resources so there is as much as possible left over for other processes. This is not just about being a good citizen; your application will benefit from a lower memory footprint, and this benefit can be dramatic if your memory usage is big enough to consume available physical memory and push the machine into paging. But even if you are targeting high-end machines and paging is not your main threat, you should use memory wisely. The cost of memory management in the common language runtime (CLR) can be significant for applications that allocate memory carelessly.

Memory footprint, garbage collection, and managing native resources, such as window handles or GDI handles, are interdependent for Windows Forms applications. Failure to release unmanaged resources increases the pressure on the garbage collector (GC). Windows Forms attempts to track handle usage and may force additional garbage collection cycles by calling GC.Collect if it is in danger of running out of resources. Besides, the managed objects holding these resources will have to be finalized by the GC, which is more expensive than releasing them proactively in your application through the IDisposable interface.

When the lifetime of the object is explicitly known, the unmanaged resources associated with it should be released. If a class that you are using implements the Dispose pattern and you explicitly know when you are done with the object, definitely call Dispose. In the Dispose method you will call the same cleanup code that is in the Finalizer and inform the GC that it no longer needs to finalize the object by calling the GC.SuppressFinalization method. This will improve performance since all the unreferenced objects in generation zero will be collected during one GC cycle. If a collectable object needs to be finalized, at least two GC cycles will be required for collecting it.

Objects that implement IDisposable usually do so because they are holding onto resources that should be freed deterministically. Windows Forms controls, for example, hold Windows handles that need to be released. If you add and remove controls from your form dynamically, you should call Dispose on them—otherwise you'll accumulate extra unwanted handles in your process. You only need to dispose the topmost control since its children will be disposed automatically.

If you show a form modally (using the ShowDialog method) you need to Dispose it after it is closed. It is not disposed automatically in this case, to enable you to inquire on its state after it is closed. Note that other forms are automatically disposed when they are closed.

Graphics objects such as brushes, pens, and fonts implement IDisposable as well, since they hold onto GDI handles. Usually you create these objects in the OnPaint method, which is called each time the control needs to be redrawn. If you don't dispose them immediately, you may accumulate a large number of these objects and hold a lot of GDI handles before these objects are finalized by the GC. If you are writing your code in C# or Visual Basic, the using statement is a handy way to do this:

using (Pen p = new Pen(Color.Red, 1))
{
    e.Graphics.DrawLine(p, 0,0, 10,10);
}

Using p As Pen = New Pen(Color.Red, 1)
    e.Graphics.DrawLine(p, 0,0, 10,10)
End Using
If you're using C++, its stack-allocation semantics are also very useful in this regard:
Pen p(Color::Red, 1);
e->Graphics->DrawLine(%p, 0,0, 10,10);

You should try to avoid invoking GC.Collect if you possibly can. The GC is self-tuning and will adjust itself according to application memory requirements. In most cases, programmatically invoking the GC will hinder that automatic tuning. The only time GC.Collect might be useful is when you know that you have just released a lot of memory and you would like it to be collected immediately. Generally, however, it is more efficient to allow the GC to simply manage itself.

Twelve Performance Tips

  • Load fewer modules at startup
  • Precompile assemblies using NGen
  • Place strong-named assemblies in the GAC
  • Avoid base address collisions
  • Avoid blocking on the UI thread
  • Perform lazy processing
  • Populate controls more quickly
  • Exercise more control over data binding
  • Reduce repainting
  • Use double buffering
  • Manage memory usage
  • Use reflection wisely

If your form has more than 200 child controls, you might think of ways to design it more efficiently. Each control consumes native resources, and managing a large number of controls is expensive. Instead, consider using ListView, TreeView, DataGridView, ListBox, ToolStrip, and MenuStrip. The individual items for these controls generally do not require a native handle to back each item.

If physical memory is in short demand and paging occurs, physical disk usage will be high (since pages are written to disk). Your application will perform unbearably slowly in this case and you will desperately need to reduce memory usage. The profiling tool in Visual Studio Team System will be helpful here. You can also use this tool to look at managed memory allocations, or you can use the CLR's profiling support (CLR Profiler: No Code Can Hide from the Profiling API in the .NET Framework 2.0). Also you might want to use the Virtual Address Dump (VaDump) tool to look at the process working set:

VaDump –sop <pid>

Conclusion

The sidebar "Twelve Performance Tips " provides you with a grab bag of ways to optimize application performance. All of them provide you with definite performance advantages, used both individually or together. Now you should be well-equipped with skills that will allow you to write fast and efficient Windows Forms applications that will impress your users and keep them happy while using your applications.

Milena Salman is a Software Design Engineer on the .NET Client (Windows Forms) team at Microsoft. She focuses on performance tuning of Windows Forms. You can reach Milena at msalman@microsoft.com. Thanks to Shawn Burke and Jessica Fosler from the Windows Forms team for their valuable contribution to this article.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Text Rendering: Build World-Ready Apps Using Complex Scripts In Windows Forms Controls

Text Rendering

Build World-Ready Apps Using Complex Scripts In Windows Forms Controls

Miguel A. Lacouture
This article discusses:
  • The TextRenderer and Graphics classes
  • Using IDeviceContext
  • Printing with TextRenderer
  • Migrating from Graphics to TextRenderer
This article uses the following technologies:
Windows Forms, .NET Framework 2.0, GDI+
Code download available at: TextRendering.exe (154KB)
Contents
The TextRenderer Class
The IDeviceContext Interface
Limitations of TextRenderer
Using TextRenderer
UseCompatibleTextRendering
Printing with TextRenderer
The TextFormatFlags Utility
Conclusion

D
eveloping world-ready software is one of the requirements software companies must meet if they want to reach customers around the globe. Text-rendering libraries must excel at rendering text, as well as at supporting international locales (language packages). This is where the Windows® Forms System.Drawing.Graphics class presents some limitations—it is based on GDI+ and currently has limited support for complex scripts.

The System.Windows.Forms.TextRenderer class was developed to provide the same level of support for complex scripts in Windows Forms controls that we expect from the Windows operating system. This TextRenderer is based on the GDI text-rendering API, which uses the Windows Unicode Script Processor (Uniscribe).

What are Complex Scripts?

Among the scripts that the Windows operating system supports, there are some that require special processing when editing and displaying text. This is because characters can have different shapes depending on the context in which they are written. Furthermore, characters from these scripts are not laid out in a simple linear progression—like the left to right progression used in English.

Arabic scripts, for instance, are right-justified and are read from right to left (RTL). In addition, the Arabic languages have character contextual shaping: a character may be represented differently depending on its position within a word or what characters precede or follow it. These scripts also require complex semantics for word and line breaking and text justification. Due to the complexity of text processing required by these scripts, they are called complex scripts.

Figure 1 shows a piece of Sinhalese text written with and without full support for complex scripts. Some of the characters in the first row, where the Graphics class is used, are not rendered properly. Compare these to the characters in the second row, which shows text rendered with full script support using TextRenderer. (For more information about complex scripts, see the "What are Complex Scripts?" sidebar.)

Figure 1 Text Rendering Differences (Close Figure 1)

System.Drawing.Graphics | System.Windows.Forms.TextRenderer
Character shapes are not rendered accurately. | Characters are shaped properly with full script support.

The TextRenderer Class

The System.Windows.Forms.TextRenderer class, new to the Microsoft® .NET Framework 2.0, contains only two overloaded static methods: MeasureText and DrawText. As their names imply, one method is used to measure the size of text, and the other to draw text, in a device context. Two new types were introduced with the TextRenderer class: the System.Windows.Forms.TextFormatFlags enumeration and the System.Drawing.IDeviceContext interface.

If you're familiar with the Win32® DrawTextEx function, you'll see a lot of commonalities between the values of the TextFormatFlags enumeration and the values defined for the dwDTFormat parameter of the native DrawTextEx function. This is not a coincidence—TextRenderer is actually a wrapper around DrawTextEx and most of the formatting options supported by DrawTextEx are also supported by TextRenderer. The TextFormatFlags enumeration defines the formatting options that govern how the text is to be formatted, measured, and drawn in a device context.

There are a few differences between the native DrawTextEx function and the TextRenderer methods. For instance, the native formatting options DT_CALCRECT and DT_TABSTOP are not present in TextFormatFlags. The DT_CALCRECT option, which is used for measuring text, is not needed because TextRenderer has MeasureText. The DT_TABSTOP option, which sets tab stops, is not currently supported by TextRenderer. To set the tab stop length in DrawTextEx, you use the DRAWTEXTPARAMS parameter; this also lets the user specify text margins. While TextRenderer has no counterpart for this particular parameter, it can expand tabs in the text if the TextFormatFlags.ExpandTabs flag is specified.

Even though TextRenderer does not expose a way to specify custom text margins as does the native DrawTextEx function, TextRenderer can automatically add text margins depending on some flags defined by TextFormatFlags for this purpose: TextFormatFlags.NoPadding, TextFormatFlags.GlyphOverhangPadding, and TextFormatFlags.LeftAndRightPadding.

Figure 2 illustrates these flags in use. The text is drawn using the flags and using a font created from the Times New Roman family. The text at the right is drawn with the italic style. The rectangle around the text is the text bounding rectangle, the size of which is obtained by measuring the text using these flags.

Figure 2 Text Drawn with Padding Flags (Close Figure 2)

TextFormatFlags.NoPadding
TextFormatFlags.GlyphOverhangPadding
TextFormatFlags.LeftAndRightPadding

However, Figure 2 shows a problem. Take a look at the italicized text that is rendered with the NoPadding flag. Notice where the glyphs fall outside the bounding box in the lower-left and upper-right areas. If the bounding box is clipped, those parts will not be drawn. Some characters have this problem when drawn at the beginning or end of a text block with a particular font and style. The GlyphOverhangPadding flag, which is the default padding value in TextFormatFlags, was provided to address this problem. When GlyphOverhangPadding is specified, TextRenderer computes some left and right margins based on the font size and style.

When drawing controls—buttons, for example—some extra text margins can sometimes make the text in the control look better. This is particularly true when the button's size is calculated on the fly. The LeftAndRightPadding flag was provided for this purpose.

The TextFormatFlags enumeration also defines PreserveGraphicsClipping and PreserveGraphicsTranslateTransform flags. These flags help to improve interoperability with the Graphics class and are needed only when some coordinate translation and custom clipping information have been applied to the Graphics object. When transitioning from GDI+ to GDI, this information is lost. If these flags are set, TextRenderer reapplies the information to the GDI device context. One more property that is automatically preserved from the Graphics object is Graphics.TextRenderingHint, which helps choose the quality of the font to be used.

The TextFormatFlags enumeration is to TextRenderer what the System.Drawing.StringFormat class is to Graphics—both define the way text should be laid out. There isn't an exact mapping between the flags in the TextFormatFlags enumeration and the ones in the StringFormat class, but a pretty good approximation can be achieved. Figure 3 outlines the relationships between the values of these formatting types.

Figure 3 Mapping Values from StringFormat and TextFormatFlags (Close Figure 3)

StringFormat Property Value | TextFormatFlags Flag Value
Horizontal Alignment
Alignment.Near | Left
Alignment.Center | HorizontalCenter
Alignment.Far | Right
Vertical Alignment
LineAlignment.Near | Top
LineAlignment.Center | VerticalCenter
LineAlignment.Far | Bottom
Ellipsis
Trimming.EllipsisCharacter | EndEllipsis
Trimming.EllipsisWord | WordEllipsis
Trimming.EllipsisPath | PathEllipsis
Trimming.Character | N/A
Trimming.Word | N/A
Trimming.None | <default behavior>
Hotkey Prefix
HotkeyPrefix.None | NoPrefix
HotkeyPrefix.Show | <default behavior>
HotkeyPrefix.Hide | HidePrefix
N/A | PrefixOnly
Text Padding
FormatFlags.FitBlackBox * | NoPadding
N/A | LeftAndRightPadding
Text Wrapping
FormatFlags.NoWrap | SingleLine
<default behavior> | WordBreak
FormatFlags.LineLimit | WordBreak | TextBoxControl
N/A | NoFullWidthCharacterBreak
Other Flags
FormatFlags.DirectionRightToLeft | RightToLeft
FormatFlags.NoClip | NoClipping
FormatFlags.DisplayFormatControl | N/A
FormatFlags.NoFontFallBack | N/A
FormatFlags.MeasureTrailingSpaces | N/A
FormatFlags.DirectionVertical | N/A
N/A | PreserveGraphicsClipping
N/A | PreserveGraphicsTranslateTransform
N/A | ExternalLeading
N/A | Internal
N/A | ModifyString
SetTabStops method | ExpandTabs
* Actually means NoFitBlackBox

The TextFormatFlags utility I discuss later in this article can create a StringFormat object based on the values of TextFormatFlags, and vice versa and then copy it to the clipboard as C# code.

The IDeviceContext Interface

IDeviceContext is used to define a contract for getting and releasing a native handle to a device context (HDC).

The IDeviceContext interface was provided for two reasons. The first is for a way to interact seamlessly with the Graphics class. The type of most of the parameters passed to the TextRenderer methods come from the System.Drawing namespace. Thus, the Graphics stateless model was preserved in TextRenderer for consistency. The Graphics class implements this interface and can be used directly with TextRenderer methods.

The second reason for the IDeviceContext interface is to allow for custom initialization of a device context before calling into TextRenderer. There are several features in Graphics that are not supported by TextRenderer, like drawing inclined text. IDeviceContext actually lets you implement support for the missing features, assuming feature parity exists in both of the underlying technologies (GDI and GDI+), which is not always the case. This interface is also used by the System.Windows.Forms.VisualStyles API. This is a new API, introduced with the .NET Framework 2.0, used for rendering controls and other UI elements with visual styles.

Figure 4 shows the code that was used to render the text in Figure 2. This code shows the steps needed to properly render text in a device context that a Graphics object represents. Note that the value Size.Empty is used as the proposed size when measuring the text. This is one way to tell TextRenderer that there isn't a bounding restriction and that the box size should be that which best fits the text. Another way to express this is by passing an unbounded size with width and height set to the maximum possible.

Figure 4 Rendering Text with TextRenderer (Close Figure 4)
Size PaintText(Graphics g, string txt, Font font,
    Point pt, TextFormatFlags flags) {
    Size size = TextRenderer.MeasureText(g, txt,
        font, Size.Empty, flags);
    Rectangle box = new Rectangle(pt, size);
    g.DrawRectangle(Pens.Black, box);
    TextRenderer.DrawText(g, txt, font, box,
        SystemColors.ControlText, flags);
    return size;
}

Limitations of TextRenderer

The TextRenderer class was designed for rendering text in Windows Forms controls (display device context type). As such, the TextRenderer class, unlike the Graphics class, is not intended for device-independent drawing. This does not mean it cannot be used in other device context types, but some work will need to be done beforehand. For instance, when using TextRenderer for printing, all parameter values must be properly scaled based on the printer resolution. (I discuss this in more detail later in this article.) This is probably the main reason why TextRenderer appears under the System.Windows.Forms namespace rather than under the System.Drawing namespace.

TextRenderer does not support rendering text at an angle. A simple workaround is to draw the text into a bitmap and then rotate the bitmap as needed. Note that when text is rendered into a bitmap, transparency is lost. Though this solution won't work in all cases, I recommend that you fill the bitmap background color where you're placing the text with the same color used in the background of the bitmap image where the text will be rendered.

As mentioned previously, TextRenderer does not allow you to specify custom tab lengths. A solution for this is to format the string with the appropriate tab lengths and location using the rich string formatting options in the .NET Framework before passing it to TextRenderer.

Using TextRenderer
Should You Make the Switch?

Not sure whether you should continue using Graphics or switch to using TextRenderer? The right decision depends on your situation. Here are a few things you need to consider:

Can your application take advantage of TextRenderer? If after reading the previous sections you determine that your application will not benefit from TextRenderer, then stick with the Graphics class you've already implemented and tested.

Can you can afford to make the changes? There are a couple of things you would need to do. First, you would call Application.SetUseCompatibleTextRenderingDefault before creating any form. You would then replace calls to Graphics.MeasureString and Graphics.DrawString with TextRenderer.MeasureText and TextRenderer.DrawText, respectively. The biggest challenge when replacing the Graphics calls is determining what set of TextFormatFlags to use based on the StringFormat options. To make this process easier, the TextFormatFlags utility can give you a hand, as explained later.

Can you afford to test for changes that break? The way GDI and GDI+ calculate text metrics, like character spacing and word breaking, is not the same and in some cases the differences are noticeable. There are several factors that affect the way text is rendered, including the font and all of its variations (such as style, size, family, and so on), the device resolution, and system text options (such as text appearance effects). Also, there isn't a full mapping between the StringFormat options and the TextFormatFlags options. Clipped text or layout changes are among the most common cases of breaking changes to watch for when switching text-rendering engines.

TextRenderer has multiple benefits for drawing text in controls—it lets you obtain complex script support for free, and it allows your controls to display text using the same rendering engine, providing a consistent text layout across the UI. In the .NET Framework 2.0, all controls render text using GDI by default, and there are actually some controls that cannot render text using GDI+, like TextBox and ListView.

As with complex scripts, there are other cases where Graphics presents some limitations that TextRenderer can fix. For instance, you cannot use GDI+ when creating end-user-defined characters (EUDC) at run time because it locks the font, preventing the addition of new characters. An important point to keep in mind is to avoid measuring text using one class and then drawing it using the another. This could lead to issues like text clipping since the text layout algorithms are different in the two classes. Always use Graphics.MeasureString with Graphics.DrawString and TextRenderer.MeasureText with TextRender-er.DrawText. (If you're wondering whether moving your apps to TextRenderer will be worth the effort, see the sidebar "Should You Make the Switch?" for some advice.)

UseCompatibleTextRendering

All Windows Forms controls in the .NET Framework 2.0 provide the same level of support for complex scripts as does Windows. In previous releases, a few Windows Forms controls could only render text using the Graphics class with the GDI+ inherited complex scripts limitation. These controls now render text using the TextRenderer class by default. For compatibility with previous releases, these controls have a new property: UseCompatibleTextRendering. When this property is set to true, text is rendered as in previous releases using the Graphics class. The controls that have this new property are Button, RadioButton, CheckBox, CheckedListBox, GroupBox, Label, LinkLabel, and PropertyGrid.

For compatibility, the UseCompatibleTextRendering property is set to false by default so applications built with previous releases of the .NET Framework will continue to look the same even when run on the .NET Framework 2.0.

New applications can change the default value of this property by calling the static Application.SetUseCompatibleTextRenderingDefault method before running the first form in a process. The Visual Studio®designer for Windows Forms generates a call to this method in the Main method.

As a side effect of this feature, controls in Windows Forms apps now look more consistent with the rest of the Windows UI because they all share the same text-rendering engine (GDI).

Printing with TextRenderer

When working with System.Drawing.Printing, you are usually interested in handling the PrintPage event of the PrintDocument object. You can use the Graphics object of the PrintPageEventArgs in the same way you use the Graphics object obtained from the PaintEventArgs when handling a control's paint event. In many cases, you want to use a common method to draw—for example, drawing into a Form, the printer, and a print-preview dialog box. This ensures consistency between what the user sees on screen and on the printed page.

This technique, however, does not work with TextRenderer. The reason is that the Graphics object in the PrintPageEventArgs object is set up with the resolution of the printer and also works with a different coordinate unit—it is usually set for the coordinate unit to be hundredths of an inch. How does it work for the Graphics object? That is where GDI+ displays its beauty. GDI+ is a device-independent graphics library and knows how to scale units properly in this case. TextRenderer assumes all units are at 100 percent. In other words, no scaling is needed.

To make TextRenderer work with the printing API, unit scaling needs to be performed beforehand. Figure 5 shows how to scale a Point and a Font to be used with TextRenderer when the Graphics object comes from the printing library. Units are converted from hundredths of an inch to pixels using the resolution of the device, which is obtained from the Graphics object DpiX and DpiY properties.

Figure 5 Scaling Font and Point (Close Figure 5)
private Point ScalePoint( Graphics g, ref Point p ) {
   return new Point(
      (int) Math.Ceiling( p.X / 100f * g.DpiX ),
      (int) Math.Ceiling( p.Y / 100f * g.DpiY ) );
}

private Font ScaleFont( Graphics g, Font font ) {
   return new Font(
      font.FontFamily, font.SizeInPoints / 72f *
      g.DpiY, font.Style, GraphicsUnit.Pixel,
      font.GdiCharSet, font.GdiVerticalFont );
}

In Figure 6, the printing flag is set to true in the printDoc_Print-Page event handler method for the PrintPage event, and false in the control_Paint event handler method for the Paint event. The PaintWorker method checks this flag and, if printing, calls the scaling methods in Figure 5.

Figure 6 Paint Method for Display or Printer (Close Figure 6)
private void PaintWorker( Graphics g ) {
    Point pt = new Point(10, 10);
    Font f = this.Font;

    if( this.printing ) {
        pt = ScalePoint(g, ref pt);
        f = ScaleFont(g, f);
    }

    TextRenderer.DrawText(g, Text, f, pt, SystemColors.ControlText);

    if( f != this.Font) f.Dispose();
}

private void control_Paint( object sender, PaintEventArgs e ) {
    this.printing = false;
    PaintWorker( e.Graphics );
}

void printDoc_PrintPage( object sender, PrintPageEventArgs e ) {
    this.printing = true;
    PaintWorker( e.Graphics );
}

Even though this approach solves the problem encountered when using TextRenderer for printing and print-previewing, it does not guarantee that you will obtain the same WYSIWYG results you get with the Graphics class. The main reason is that the GDI character-spacing and word-breaking algorithm is not linear—the text rendered using the scaled values may end up with a different layout. Other reasons include the loss of precision when scaling the values and the possibility of obtaining a scaled font that is different from the intended scaling. This can occur if the intended scaled font does not exist and the font-mapping algorithm selects what it thinks is the closest match.

The TextFormatFlags Utility

Figure 7 shows the TextFormatFlags utility UI. Note that in this image, there is a small window on top of the main form. This window, called the drawing surface, is used to compare text rendered with both the TextRenderer and the Graphics classes. The text format can be chosen from the options form—TextFormatFlags options are chosen from the left panel and StringFormat options from the right.

Figure 7 TextFormatFlags Utility
Figure 7 TextFormatFlags Utility

One of the most interesting features of this tool is that it can obtain the text format options from one API based on the options from the other API and copy this information to the clipboard as C# code, like this:

TextFormatFlags textFlags = TextFormatFlags.NoPrefix |
    TextFormatFlags.WordBreak | TextFormatFlags.ExpandTabs;

StringFormat strFormat = new StringFormat();
strFormat.FormatFlags =
    StringFormatFlags.MeasureTrailingSpaces;

Although text format options do not map 100 percent from one API to the other API, this tool does a very good job of getting you started with your transition between APIs. This tool was very useful during the development of the TextRenderer class, as it was used to better understand the behavior of the underlying technologies. You can download it from the MSDN®Magazine Web site.

Conclusion

Applications that need to be ready for globalization must provide at least the same level of support for rendering text using different scripts as the underlying operating system. With the introduction of the System.Windows.Forms.TextRenderer class in the .NET Framework 2.0, it has become easier to give your Windows Forms applications the same level of support for complex scripts as the platform the application is targeted to. Now all Windows Forms controls can render text using the same underlying text-rendering engine when the UseCompatibleTextRendering property is set to true, making the user interface more consistent in appearance and readability.

Miguel A. Lacouture is a Software Design Engineer on the Windows Forms team and has been at Microsoft for seven years. You can reach him at mlacouture@msn.com.


Outlook Add-Ins: Improve Your Outlook With Visual Studio Tools For Office

Outlook Add-Ins

Improve Your Outlook With Visual Studio Tools For Office

John R. Durant
This article discusses:
  • Background scenario for an Outlook add-in
  • A high-level look at the add-ins architecture
  • Processing e-mail attachments
  • Generating customer correspondence
This article uses the following technologies:
Visual Studio 2005 Tools for the Microsoft Office System
Code download available at: OutlookAddIns.exe (158KB)
Contents
The Scenario
Add-In Architecture
Attachment Processing
Creating the Letter
Conclusion

M
icrosoft Visual Studio Tools for Office 2003, while useful to many developers who wrote tools for Office, supported only Microsoft® Word and Excel®. The new Visual Studio® 2005 Tools for the Microsoft Office System goes farther, adding tools that help you create managed code add-ins for Outlook® 2003.

You can create managed add-ins for Outlook using the Shared Add-in template in Visual Studio 2005 Standard Edition, but you'll miss out on the improvements that have been made in Visual Studio 2005 Tools for the Microsoft Office System that make the process easier and faster.

This article walks you through a sample add-in application for Outlook 2003. Along the way I'll show off some of the things that are possible when creating an Outlook add-in solution using this new version (Visual Studio Tools for Office). The add-in helps a company bring its inventory management data into a more meaningful and immediate sales promotion context.

The Scenario

The business scenario for the sample deserves a little bit of background to help you understand the overall application as well as the technical details. World Wide Importers sells products that other companies and institutions purchase as part of their own marketing campaigns. Items sold include branded key chains, pens, paper, letter openers, and other such office supplies.

In the past, World Wide Importers sales agents have received reports containing data about previous customer orders, but they had to sift through this data manually to find out what was meaningful, slowing the sales process, and causing some missed opportunities. Further, the sales agents felt they spent too much time creating letters and other material that suggested products to their customers. They wanted these materials to be both customer-specific and easy to compile.

My sample application, the Outlook Sales Order Prompt add-in, solved the problem. As seasonal item inventory comes in, the inventory manager runs a batch job from the database. The batch job in turn produces an Excel spreadsheet for each customer and sends the spreadsheet to the customer's sales agent as an attachment. These spreadsheets are not produced in the Excel binary format but rather in the XML spreadsheet format, SpreadsheetML.

The sales agent can review the data in the spreadsheet and, by clicking a button, create a promotional letter containing customer-specific recommended items from the spreadsheet and contact data from Outlook. A resulting promotional letter is saved as a Word document using the Word XML file format, WordProcessingML. An assistant can print these letters and send them to the customers. The entire process, up to the point of sending the physical letter, takes a fraction of a second.

Add-In Architecture

As you can see from the application's data flow, shown in Figure 1, the add-in handles transforming the Excel spreadsheet into two different XML documents. The first is a raw data document, and the second is a fully formatted Word document. While the transformation from spreadsheet to Word document can be done in one operation, there are benefits to having an interim XML document with no Word formatting. For example, what if another system or process needed access to the raw XML data? What if additional data elements were needed in the raw XML data file? It is easier to manage the transformations and various markup demands by isolating the operations.

Figure 1 Data Flow of Outlook Sales Order Prompt Add-In
Figure 1 Data Flow of Outlook Sales Order Prompt Add-In

Fortunately, users need not worry about any of these details as they interact with the add-in. Its simple menu contains three items, as shown in Figure 2. These menu items simply execute various data transformations supported by the add-in.

Figure 2 Add-In Menu Items
Figure 2 Add-In Menu Items

One key advantage of using Visual Studio Tools for Office to create an Outlook add-in is that the development experience has been greatly simplified. Creating add-ins using the old IDTExtensibility2 model meant that you had five different events to choose from, causing confusion about which event to use for different programming tasks. Now there are just two events, ThisApplication_Startup and ThisApplication_Shutdown. These events are contained in a ThisApplication file that the tools create automatically. In the case of the sample add-in, which was written in Visual Basic, the file is called ThisApplication.vb. The principles discussed here apply to add-ins created in C# as well.

The SalesOrderPrompt add-in runs code at startup to add its menu to the menu bar in Outlook (see Figure 3). The add-in also acquires a reference to a specific Outlook folder because it will need to work with this reference later on. Finally, the startup code gains a pointer to the active Explorer object so that the add-in can disable the menu if the user doesn't have the target folder selected. The only other work that precedes the ThisApplication_Startup method is the class-level variable declarations for the various folders, Explorers, buttons, strings, and other items used during the add-in's lifetime.

Figure 3 Class-Level Declarations and Startup Code (Close Figure 3)
Private WithEvents _Explorers As Outlook.Explorers
Private WithEvents _Explorer As Outlook.Explorer
Private _taskFolder As Outlook.Explorer
Private _SOPFolder As Outlook.MAPIFolder
Private _topMenu As Office.CommandBarPopup
Private _results As Outlook.Results
Private _searchTerm As String
Private _contactEntryID As String
Private _attachmentPath As String
Private WithEvents _createLetter1 As Office.CommandBarButton
Private WithEvents _createLetter2 As Office.CommandBarButton
Private WithEvents _createLetter3 As Office.CommandBarButton
Private _menuBar As Office.CommandBar
Dim WithEvents _mailTarget As Outlook.MailItem

Private search As Outlook.Search = Nothing

Private Sub ThisApplication_Startup( _
        ByVal sender As Object, ByVal e As System.EventArgs) _
        Handles Me.Startup
    BuildTopMenu()
    _SOPFolder = OpenMAPIFolder()
    _Explorer = Me.ActiveExplorer()
End Sub

Function OpenMAPIFolder() _
        As Microsoft.Office.Interop.Outlook.MAPIFolder
    Dim SOPFolder As Microsoft.Office.Interop.Outlook.MAPIFolder
    Dim f As Microsoft.Office.Interop.Outlook.MAPIFolder = Nothing
    SOPFolder = Me.ActiveExplorer.Session.GetDefaultFolder( _
        Outlook.OlDefaultFolders.olFolderInbox)
    For Each f In SOPFolder.Folders
        If f.Name = "SalesOrderPrompt" Then Exit For
    Next
    Return f
End Function

Building a menu in Outlook is easy. The add-in has a separate procedure to build the menu, and it does so by referencing the ActiveMenuBar property of the Outlook ActiveExplorer (see Figure 4). Each Explorer object in Outlook has a CommandBars collection that you can access to add, remove, or modify its collection of menu bars and their controls. The Sales Order Prompt add-in appends a new menu bar to the Outlook standard menu bar and then adds three new buttons to the menu.

Figure 4 Creating the Outlook Menu (Close Figure 4)
Private Sub BuildTopMenu()
    _menuBar = Me.ActiveExplorer().CommandBars.ActiveMenuBar

    _topMenu = CType(_menuBar.Controls.Add( _
        Office.MsoControlType.msoControlPopup, , True), _
        Office.CommandBarPopup)
    _topMenu.Caption = "SOP Add-in"
    _topMenu.Visible = True

    _createLetter1 = CType(_topMenu.Controls.Add( _
        Office.MsoControlType.msoControlButton, , , True), _
        Office.CommandBarButton)
    _createLetter1.Caption = "Create Basic Letter"
    _createLetter1.Visible = True
    _createLetter1.Enabled = False

    _createLetter2 = CType(_topMenu.Controls.Add( _
        Office.MsoControlType.msoControlButton, , , True), _
        Office.CommandBarButton)
    _createLetter2.Caption = "Create Personal Letter"
    _createLetter2.Visible = True
    _createLetter2.Enabled = False

    _createLetter3 = CType(_topMenu.Controls.Add( _
        Office.MsoControlType.msoControlButton, , , True), _
        Office.CommandBarButton)
    _createLetter3.Caption = "Create Premiere Letter"
    _createLetter3.Visible = True
    _createLetter3.Enabled = False
    _topMenu.Enabled = False
End Sub

Looking at Figure 4 closely you can see more of the advantages that Visual Studio Tools for Office offers. For example, the add-in provides easy access to the Outlook object model objects by setting up the reference to the primary Outlook application instance. This reference is an instance of the Application object in the Outlook object model. You access this object instance, ThisApplication, using the Me keyword in Visual Basic or the this keyword in C#.

Note that the Application object instance accessed through ThisApplication does not derive from Microsoft.Office.Interop.Outlook. Rather, it derives from Microsoft.Office.Tools.Outlook.Application. This is a custom namespace that accompanies Visual Studio Tools for Office. The key advantage in deriving from this class rather than the normal Primary Interop Assembly (PIA) class is that it allows your add-in code to be forward-compatible by decoupling it from the regular PIA namespace. If future versions of Office change the PIA namespace, your code can remain stable as far as the primary object reference is concerned.

Attachment Processing

The add-in's primary operation runs when the user clicks one of the buttons. The custom add-in menu is only enabled if the user has selected a custom folder associated with the Sales Order Prompt add-in (see Figure 5) and only one e-mail item is selected in the special folder.

Figure 5 Target Outlook Folder
Figure 5 Target Outlook Folder

Recall that the overall scenario specified that the company's database sends an e-mail containing a report for each customer. The customer's sales agent receives the e-mail, reviews the report, and fires off the main code of the add-in by clicking the custom menu button. Restricting the selection of only one e-mail item in the custom folder reduces the code complexity in the add-in. The code to enforce both of these key constraints is in the FolderSwitch and SelectionChange events (see Figure 6) of the Explorer object that the add-in references at startup.

Figure 6 Controlling UI Availability (Close Figure 6)
Private Sub _Explorer_FolderSwitch() Handles _Explorer.FolderSwitch
    _topMenu.Enabled = _
        (_Explorer.CurrentFolder.EntryID = _SOPFolder.EntryID)
End Sub

Private Sub _Explorer_SelectionChange() _
    Handles _Explorer.SelectionChange

    If _Explorer.CurrentFolder.EntryID = _SOPFolder.EntryID _
            And _Explorer.Selection.Count = 1 Then
        _mailTarget = _Explorer.Selection(1)
        _createLetter1.Enabled = True
        _createLetter2.Enabled = True
        _createLetter3.Enabled = True
    Else
        _createLetter1.Enabled = False
        _createLetter2.Enabled = False
        _createLetter3.Enabled = False
    End If
End Sub

With these conditions met, the user clicks the button to process the e-mail attachment in order to create a promotional letter. The attachment processing starts by gaining a reference to the selected e-mail's Excel spreadsheet attachment. It finds the name of the customer in the spreadsheet, searches among the user's Outlook contacts to find the contact information for the customer, and merges the contact information with selected data from the Excel spreadsheet report in a raw data XML file. The code converts the raw data XML file into a fully formatted Word document saved in a centrally accessible location. These steps begin in the click event for a button on the add-ins custom menu (see Figure 7).

Figure 7 Processing an E-Mail Attachment (Close Figure 7)
Private Sub _createLetter1_Click( _
    ByVal Ctrl As Microsoft.Office.Core.CommandBarButton, _
    ByRef CancelDefault As Boolean) Handles _createLetter1.Click

    Try
        If _mailTarget.Attachments.Count = 0 Then
           MessageBox.Show("E-mail message has no attachment to process")
           Exit Sub
        End If

        Dim attachment As Outlook.Attachment
        attachment = _mailTarget.Attachments.Item(1)

        Dim excelExport As New XmlDocument()
        _attachmentPath = Path.GetTempPath() & attachment.FileName
        attachment.SaveAsFile(_attachmentPath)
        excelExport.Load(_attachmentPath)

        Dim namespaceManager As System.Xml.XmlNamespaceManager
        namespaceManager = New _
           System.Xml.XmlNamespaceManager(excelExport.NameTable)
        namespaceManager.AddNamespace("ss", _
           "urn:schemas-microsoft-com:office:spreadsheet")

        _searchTerm = excelExport. _
           SelectSingleNode("/ss:Workbook/ss:Worksheet/" _
           & "ss:Table/ss:Row/ss:Cell/ss:NamedCell[@ss:Name=" _
           & "'Customer']/../ss:Data/node()", namespaceManager).Value
        UseAdvancedSearch()

    Catch exc As System.Exception
        MessageBox.Show(exc.Message)
    End Try
End Sub

No system is foolproof, so the add-in code checks to see whether the selected e-mail has no attachment and lets the user know if this is the case. With one attachment found, the add-in saves the attachment to a local temporary directory so that its contents can be loaded into an XmlDocument instance. A quick search using an XPath statement lets the add-in find the name of the customer so that it can find contact information.

Saving the file to a local temporary directory is necessary because Outlook exposes the attachment as a binary object with no clearly discernible type information. A potential improvement of the add-in could include the conversion of the attachment's binary stream to its simple text-based XML format.

In order to find the customer's contact information, the add-in leverages the AdvancedSearch object in Outlook, as shown in Figure 8. Using this object, the add-in can look for all contacts whose CompanyName property matches the search term extracted from the spreadsheet attachment.

Figure 8 Using the AdvancedSearch Object (Close Figure 8)
Private Sub UseAdvancedSearch()
    Dim contacts As Outlook.MAPIFolder = _
        Me.ActiveExplorer().Session.GetDefaultFolder( _
        Outlook.OlDefaultFolders.olFolderContacts)

    Try
        Dim filter As String = String.Format("{0} LIKE '%{1}%'", _
            "urn:schemas:contacts:o", _searchTerm)
        Dim scope As String = String.Format("SCOPE (" _
            & "'shallow traversal of ""{0}""')", _
            contacts.FullFolderPath)
        search = Me.AdvancedSearch(scope, filter, True, _searchTerm)

    Catch ex As System.Exception
        MessageBox.Show("There was an error looking for a customer:" _
            & _searchTerm, "Search")

    End Try
End Sub

The AdvancedSearch object in Outlook works asynchronously so that while the search is underway, users can return to other tasks and not have to wait for the search to complete. Because AdvancedSearch works asynchronously, there is no consistent linear progression from when a user clicks a button on the menu and when the search completes. It is possible for a user to launch the search, and then launch a second search that completes before the first search completes. Thus, the add-in needs a way to distinguish the results of one search from another.

One way to do this is shown in Figure 8 where the search term variable, _searchTerm, is passed as an argument to the AdvancedSearch constructor. This sets the Tag property of the AdvancedSearch object equal to the value of the search term. Then, as the search completes, the add-in can look at this property to determine which search the results belong to. When the search finishes, Outlook fires the AdvancedSearchComplete event and passes as an argument a Search object containing the results of the search (see Figure 9).

Figure 9 Handling AdvancedSearchComplete (Close Figure 9)
Private Sub ThisApplication_AdvancedSearchComplete( _
    ByVal SearchObject As Microsoft.Office.Interop.Outlook.Search) _
    Handles Me.AdvancedSearchComplete

    Dim contactResultsForm As ContactResultsForm
    Dim contactResultArray() As ContactResult = Nothing

    If SearchObject.Tag = _searchTerm Then
        _results = SearchObject.Results

        If _results.Count > 0 Then
            Dim i As Integer
            Dim _item As Outlook.ContactItem

            For i = 1 To _results.Count
                If _results.Item(i).Class = _
                    Outlook.OlObjectClass.olContact Then
                    _item = _results.Item(i)
                    ReDim Preserve contactResultArray(i - 1)
                    contactResultArray(i - 1) = New _
                        ContactResult(_item.FullName, _
                        _item.CompanyName, _item.EntryID)
                End If
            Next

            If contactResultArray.Length > 1 Then
                contactResultsForm = New _
                    ContactResultsForm(contactResultArray)
                contactResultsForm.ShowDialog()
                _contactEntryID = contactResultsForm.EntryID
                ProcessXmlAttachment()
            Else
                _contactEntryID = contactResultArray(0).EntryID
                ProcessXmlAttachment()
            End If
        End If
    End If
End Sub

Creating the Letter

In creating the actual letter, the add-in does its primary job of merging data, transforming XML, and producing Word documents to be sent to the customer. The add-in checks to see whether the search results are for the Sales Order Prompt search by looking at the Tag property and seeing if it matches the value of the _searchTerm variable that was set when the search began. As the add-in loops through the results, it verifies that each result's olObjectClass property is set to the contact olContact. The add-in does not process non-contact items returned from the search.

For each contact item found, the add-in creates an instance of a custom ContactResult class and places it into an array. If there is only one contact for a customer, then the add-in can proceed directly with merging the contact data with the spreadsheet data and doing the transformations. If more than one contact exists for a customer, the add-in displays a form that lets the user select one of the contacts.

Once the add-in knows which contact's data to include, it sets a class-level variable, _entryID, to the entryID property of the target contact. This allows the add-in to load the contact from another procedure, ProcessXmlAttachment (see the code download for details).

At an earlier point, the add-in saved the spreadsheet in a temporary directory. Now the add-in loads that XML file into a new XmlDocument instance. It then loads an XSL stylesheet from disk, the location of which is specified in an application-level setting named ExcelExport. With the source document and transformation document specified, the add-in transforms the source document and loads the resulting XML document into memory using the StringWriter object instance.

Next, the add-in merges the data about the sales agent and the company's contact into the new in-memory XML document. This means adding new nodes and appending them to the XML document. To reduce the clutter in this part of the code, the add-in repeatedly calls the AddXmlElement function to add the new node elements:

Private Function AddXmlElement(ByRef ParentDoc As XmlDocument, _
      ByVal ElementName As String, ByVal InnerText As String) As XmlNode
    Dim newNode As XmlNode
    newNode = ParentDoc.CreateElement(ElementName)
    newNode.InnerText = InnerText
    Return newNode
End Function

The add-in loads a special XML file in a directory located in the user's special settings: C:\Documents and Settings\[USERNAME]\Local Settings\Application Data\SOP. This directory contains a file called CurrentUser.XML, which contains information about the logged-on sales agent. Data from this file and from the contact returned by the Outlook search is added to the in-memory XML document.

Finally, the add-in loads another XSL stylesheet using a different application-level variable, ProposalLetter. This transformation converts the data-only XML into a fully formatted Word document.

The add-in produces the XML document using an instance of the XmlTextWriter object. It saves the final document in a special directory whose location it keeps in another application-level variable, DocStore. When it saves the file, it includes the name of the company and the current date in the document's name.

Finally, I don't think anyone likes to have menus cluttering up the user interface when they are not really needed. So the add-in responsibly removes the custom menu from the Outlook standard menu bar when the add-in is unloaded. Unloading the add-in can occur when the user quits Outlook or when the user disables it in the list of COM add-ins. In either case, the add-in's ThisApplication_Shutdown event fires:

Private Sub ThisApplication_Shutdown( _
        ByVal sender As Object, ByVal e As System.EventArgs) _
        Handles Me.Shutdown
    _topMenu.Delete()
End Sub

Another advantage of using Visual Studio Tools for Office to create your Outlook add-ins is that the tools create a setup project for you at project creation. Thus, your solution will contain two projects, the first being your add-in itself, and the second being the setup project for the add-in.

Now you can add special launch conditions and craft your setup package to suit your needs. But the tools have already configured the basic properties for you. For example, the output of your add-in has already been referenced along with dependencies on the Office PIA and the Microsoft .NET Framework 2.0, among others.

Conclusion

Many companies want to make Outlook and the data that it handles a more integral part of their core business processes. Writing solutions in managed code is one obvious way to achieve this. Until now, creating, debugging, and maintaining managed code add-ins for Outlook has been a sizable challenge. Debugging could be a frustrating experience, and there was too wide a gap between the COM world of Office and the managed world of the Microsoft .NET Framework.

Visual Studio Tools for Office narrows that gap considerably and delivers a dependable design time and run-time environment for managed code Outlook add-ins. Rather than spending your time troubleshooting development tools, you can now get right into the core programming tasks and create powerful Outlook add-ins more easily. For more information on developing add-ins for Outlook, see Visual Studio Tools for Office Technical Articles.

John R. Durant is a Program Manager for Visual Studio Tools for Office and is a frequent author and speaker on a wide variety of development topics (blogs.msdn.com/johnrdurant). He coauthored the latest edition of the XML Programming Bible (Wiley & Sons, 2003) and is authoring a new book on Office 12 programmability.


Office Unbound: Bring Your Documents To Life With Data Binding In Visual Studio Tools For Office

Office Unbound

Bring Your Documents To Life With Data Binding In Visual Studio Tools For Office

Eric Carter and Eric Lippert
This article discusses:
  • Defining data sources for Office documents
  • Creating data-bound documents
  • Understanding datasets, adapters, and sources
  • Binding to proxy objects
This article uses the following technologies:
Visual Studio 2005 Tools for the Microsoft Office System
Contents
Defining a Data Source
Data-Bound Documents
Datasets, Adapters, and Sources
Security Best Practices
DataSets
A Disconnected Strategy
Typed and Untyped Datasets
Adapters
Using Binding Sources as Proxies
Data-Bindable Controls
Special Considerations
Conclusion

M
icrosoft Visual Studio 2005 Tools for the Microsoft Office System provides a powerful new toolset for Office development within the Visual Studio® environment. One of the most important new features of Visual Studio Tools for Office is the ability to create documents that are data bound to databases, Web services, or objects. Elements of an Excel® workbook (called host controls in Visual Studio Tools for Office), such as a range or a list object or a bookmark in a Microsoft® Word document, can be data bound using the same tools and syntax used when creating Windows® Forms-based applications. Data-bound Windows Forms controls can also be added to a workbook or document using Visual Studio Tools for Office.

Creating a no-frills, data-bound customized document using the Visual Studio Tools for Office designer requires no coding, but a whole lot of mouse clicking instead. First we'll tell Visual Studio about a data source—in this case, the Northwind sample database that comes with Office—and then drag and drop some data-bound controls onto the spreadsheet.

Defining a Data Source

Let's start up Visual Studio and create a new Excel project. Select File, then New Project, and go to the Office folder under Visual C#® or Visual Basic® to select the Excel Workbook project.

From the Visual Studio Data menu, choose Show Data Sources to display the Data Sources pane. Click Add New Data Source to start the Data Source Configuration Wizard (see Figure 1).

Choose Database and click Next. Then click New Connection. A second wizard will appear listing a variety of data source options, such as Microsoft Access Database File, Microsoft ODBC Data Source, and Microsoft SQL Server™.

Figure 1 Starting the Data Source Configuration Wizard
Figure 1 Starting the Data Source Configuration Wizard

Choose Microsoft Access Database File and click Continue to go on to the Connection dialog. The Northwind database file is typically in the \Program Files\Microsoft Office\Office11\Samples directory. Click Browse and find the Northwind database. No security is enforced on this database file, so the default Admin username and a blank password will work (in a real-world application with a secured database, it would be a very bad idea to have a blank administrator password). Click OK to close the Connection Wizard and continue with the Data Source Wizard.

When you click Next, Visual Studio notes that you are creating a connection to a local database file that is not part of the current project. If you want this project to have its own copy of the database rather than modifying the original, you can choose to do this and Visual Studio will automatically update the connection to point to the new location. In this first example, there is no reason to make a copy of the database, so click No.

As you can see, all the information about the database connection that you have just created is saved in a connection string. For both convenience and security, it is a good idea to save that connection string in a configuration file rather than hard-coding it into your program.

Figure 2 Choose Your Tables
Figure 2 Choose Your Tables

The database to which we are connecting might have an enormous number of queries, tables, columns within those tables, and so on. To manage some of this complexity, Visual Studio lets you choose which portions of the database will display in Visual Studio. Let's select the entire Suppliers table and the ProductName, SupplierID, QuantityPerUnit, and UnitPrice columns from the Products table (see Figure 2). Finally, click Finish to exit the wizard.

Data-Bound Documents

The Data Sources window now contains an entry for the NorthwindDataSet. Expand the nodes in the tree view. There are a few interesting things here. First, Visual Studio has discovered from the database that the Products table has a relationship with the Suppliers table; the Products table appears both as a table in its own right and as a child node of the Suppliers table. This will allow us to more easily create master-detail views.

Second, notice that the icons for the columns have "named range" icons, indicating that if you drag the icon and drop it onto the worksheet, you will get a data-bound named range to this column. The default for a column is a named range, and the default for an entire table is a list object, but you can choose other controls by clicking the item and selecting from a dropdown. Another powerful feature of Visual Studio Tools for Office is that you can actually put Windows Forms controls, such as a combobox, into a Word or Excel document and data bind to Windows Forms controls in the document. Suppose you want to have a combobox bound to the CompanyName, for instance. You can choose ComboBox from the dropdown as the control to use for CompanyName, as shown in Figure 3.

Figure 3 Choose the Control Type
Figure 3 Choose the Control Type

Drag the CompanyName as a combobox, the ContactName as a named range, and the entire Products table onto the worksheet. Use the Products table that is the child of the Suppliers table in the tree view and you'll get a nice master-detail view.

A whole lot of stuff has magically appeared in the component tray below the Excel designer: a DataSet, two binding sources, and two table adapters. We'll get into the details of what these components are for later. For now, just compile and run the application. Without writing a single line of code, you have gotten a data-bound master-detail view on an Excel spreadsheet. As you select different items from the combobox, the named range and list objects automatically update themselves (see Figure 4).

Figure 4 Data-Bound Master-Detail Spreadsheet
Figure 4 Data-Bound Master-Detail Spreadsheet

We can create a similar data-bound document in Word using bookmarks rather than named ranges, and a DataGrid rather than an Excel list object. Create a new Word document project and again add the Northwind database as a data source to the Data Sources pane. Visual Studio should remember the connection string from last time, so you will not need to configure it again. Unfortunately, in this version of Visual Studio Tools for Office, there is no way to bind a data table to a Word table as you can with an Excel list object.

Figure 5 Master-Detail View at Run Time
Figure 5 Master-Detail View at Run Time

When we build and run the customized Word document, we have a master-detail view of a data table running in Word, once again without writing a single line of code (see Figure 5).

Datasets, Adapters, and Sources

Now that you have seen a couple of no-coding-required examples, let's take a peek under the hood and see how data binding actually works. The data-binding model used by Visual Studio Tools for Office is almost identical to the model used by Windows Forms. Several players make data binding work, many of which can be seen on the component tray or design surface:

  • A back-end data source, such as a Microsoft Access database, a remote SQL Server database, a Web service, or some other data storage and retrieval system, is where the data ultimately lives.
  • A DataSet provides a disconnected local cache of some portion of the back-end data source.
  • An adapter connects the DataSet to the back-end data source, both to fill the DataSet from the back-end source and to update the back end with any changes. There is usually one adapter per table, which is why there are two adapters in the preceding example.
  • A binding source acts as the intermediary between the user interface control and the DataSet. Although it is possible to bind a control directly to a DataSet, as we'll discuss later, it is usually more convenient to go through a dedicated binding source object.
  • A data-bindable control provides a user interface element that enables the user to read or write the data. Using Visual Studio Tools for Office some built-in elements in Word and Excel documents are data-bindable, such as a Word bookmark and an Excel range. Windows Forms controls can also be added to a Word document or Excel spreadsheet and are data-bindable as well.
The back-end data source is represented in a Visual Studio Tools for Office project by the connection string passed to the adapter; everything else is represented by a member of the customized host item (the worksheet or document) class.

Security Best Practices

As you probably noticed in the Connection Wizard, all the information required to connect to the back-end data source is stored in a connection string generated by the wizard. It typically looks something like this:

Server=MyDataServer;Database=Customers;Integrated Security=true;
That is, it says where the database is located, what it is called, and how the user should be authenticated. All of this is potentially sensitive information. Use caution when embedding connection strings into your programs. Remember, even without the source code, it is very easy to figure out which strings are embedded in a managed application. This particularly applies to connection strings where, instead of using security integrated into Windows NT®, you simply embed something like this directly:
UserID=eric;Password=BigSecret123

Furthermore, hardcoding embedded strings in your source code makes it hard for developers, testers, end users, and database administrators to update your application should the database connection information change over time. As discussed previously, Visual Studio gives you the option of embedding the connection string in the configuration file. The automatically generated configuration file for the example just shown looks something like this:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections/>
  <connectionStrings>
    <add name=
      "ExcelWorkbook11.Properties.Settings.NorthwindConnectionString"
      connectionString="Provider=Microsoft.Jet.OLEDB.4.0;Data
      Source=&quot;C:\Program Files\Microsoft
      Office\OFFICE11\SAMPLES\Northwind.mdb&quot;"
      providerName="System.Data.OleDb" />
  </connectionStrings>
</configuration>

It is also a good idea to use the principle of least privilege. This is one of the fundamental principles of secure design: grant exactly as much privilege as the user needs to get the job done—no more, and no less. For example, if your user needs to be able to read from the database but not write to it, do not specify a connection string that gives the user administrator rights to the database. Instead, choose a connection string that specifies a username and password with read-only access. That way, if the username and password are ever compromised, at least the attacker does not get administrator access out of it.

Better still, do not use stored user IDs and passwords at all; some databases use integrated Windows authentication so the logged-on user can seamlessly use his already authenticated credentials. Or, if your database system requires a username and password, make the user type them in rather than storing them. As you'll see later when we discuss adapters, you can manually change the connection string used by the adapter before it fills the DataSet. That way you can ask the user to type in a user ID and password and then generate a new connection string from that information.

DataSets

The cornerstone of the Visual Studio Tools for Office data model, and of ADO.NET in general, is the DataSet. We should explain the existence of DataSets by describing the old way of doing data access. Back in the twentieth century, you typically communicated with a database via "ADO Classic" which went something like the following:

  1. Create and open a connection to a database.
  2. Create and execute a database command such as:
    SELECT partnumber FROM invoices WHERE price > 100
  3. Enumerate the resulting record set.
  4. Close the connection.
This approach worked fairly well, but it has several drawbacks. The principal drawbacks are consequences of the fact that this model requires a live connection to a database. If there are going to be many live connections, the server needs to be scalable and robust, which can be expensive. Therefore, to minimize load upon the server, we want connections to be short-lived. But because the connection is open while the user is enumerating the record set, the connection is typically open for quite some time—as long as the user is working with the data. Furthermore, even if the server-side expense of keeping connections open is unimportant, this model does not work well in a world where you want to be able to work with your data even if you temporarily lack network connectivity.

A Disconnected Strategy

Database connections are both expensive and necessary and therefore must be managed carefully. In a typical ADO application, much developer effort is expended writing code to ensure that the connection is open for as little time as possible while still meeting the needs of the application's users. ADO.NET addresses the problems of ADO by going straight to the root: if we cannot make connections inexpensive, we can at least make them less necessary. ADO.NET is therefore fundamentally a disconnected strategy. A typical ADO.NET scenario goes something like this:

  1. Create a DataAdapter to manage the connection to a specific database or other data source.
  2. Set properties on the adapter that tell it what query to execute against the database.
  3. Create a DataSet to be filled.
  4. Invoke a method on the adapter to take care of the details of opening a connection, executing the query, saving the results in the DataSet, and closing the connection as soon as possible.
  5. Work with the data in the now-disconnected DataSet.
  6. When finished working with the data, invoke a method on the adapter to re-open the connection to the database and update it with any changes.
And indeed, as you will see later when we discuss adapters, Visual Studio Tools for Office does exactly this on your behalf.

Because the DataSet acts much like the original database, the connection only needs to be open as long as it takes to fill the DataSet. After the data has been copied to the DataSet, you can query and manipulate the DataSet for as long as you want without having to worry that you are consuming a valuable database connection.

Furthermore, there is no reason why the data used to fill the DataSet has to come from a connected database; you could also fill the DataSet from an XML file or write a program to add tables and rows to build one from scratch. DataSets have no knowledge of where the data they contain came from; if you need it, all that knowledge is encapsulated in the adapter.

Typed and Untyped Datasets

In the Solution Explorer of the Word and Excel projects we created earlier, you will find a NorthwindDataSet.xsd file containing the database schema. This is an XML document that describes the tables, columns, and relationships that make up the DataSet. One of the child nodes in the Solution Explorer tree view is NorthwindDataSet.Designer.cs. This file contains the automatically generated code for the DataSet and table adapters.

The first line of the declaration is interesting:

public partial class NorthwindDataSet : System.Data.DataSet {
The generated class is partial so that, if you need to add your own extensions to it, you can do so in a separate file. It is a bad idea to edit automatically generated files. More importantly, this DataSet extends the System.Data.DataSet class. A System.Data.DataSet consists of a collection of data tables. As you would expect, data tables consist of a collection of data columns and data rows. Each class exposes various collections as properties that allow you to navigate through the DataSet.

System.Data.DataSet is not an abstract class—you can create instances and fill them from any back-end data source. But that would be an untyped DataSet; the NorthwindDataSet is a typed DataSet. Untyped DataSets give you great flexibility but are so general that they are somewhat harder to use. For example, if you were to fill an untyped DataSet with data from the Northwind database file, you could access a particular datum with an expression such as this:

name = myDataSet.Tables["Products"].Rows[1]["ProductName"];
But that flexibility comes at a cost: you can accidentally pass in a bad table name or column name, or make a bad assumption about the type of the data stored in a column. Because the structure of the tables and the types of the columns is not known at compile time, the compiler is unable to verify that the code will run without throwing exceptions. Also, the IntelliSense® engine is unable to provide any hints about the DataSet's structure while you are developing the code.

Typed DataSets mitigate these problems. A typed DataSet is a class that extends the DataSet base class; it has all the flexible, untyped features of a regular untyped DataSet, but also has compile-time strongly typed properties that expose the tables by name. A typed DataSet also defines typed data table and data row subclasses.

As you can see from the NorthwindDataSet.Designer.cs file, the typed DataSet has public properties that enable you to write much more straightforward code, such as the following :

name = myDataSet.Products[1].ProductName;
Typed DataSets extend untyped DataSets in many ways; some of the most important are as follows:
  • Tables are exposed as read-only properties typed as instances of typed data tables.
  • Tables have read-only properties for each column.
  • Tables have an indexer that returns a typed data row.
  • Event delegates for row change events pass typed change event arguments. Each row type has a row-changing, row-changed, row-deleting, and row-deleted event. (You might be wondering where the row-adding and row-added events are. The changing/changed events pass a DataRowAction enumerated type to indicate whether the row in question was newly created.)
  • Tables provide methods for adding and removing typed data rows.
  • Rows provide getters, setters, and nullity testers for each column.
In short, it is almost always a good idea to use a typed DataSet. Weakly typed code is harder to read, harder to understand, and harder to maintain.

Adapters

Take a look at the Startup event handler in the Word or Excel examples discussed previously. Visual Studio has automatically generated the code (shown in Figure 6) on your behalf. The one new thing here is the NeedsFill method. NeedsFill is a special Visual Studio Tools for Office method that works with Visual Studio Tools for Office data caching feature. This feature allows you to mark a DataSet as cached so that the DataSet is saved in the document or spreadsheet being customized. This article doesn't cover this feature in any additional detail—for now, the rest of the code should look fairly straightforward. If the two tables need to be filled from the back-end data source, the adapters fill the appropriate tables.

Figure 6 Auto-Generated Table-Filling Code (Close Figure 6)
public partial class ThisDocument
{
    private void ThisDocument_Startup(
        object sender, System.EventArgs e)
    {
        // TODO: Delete this line of code to remove the default
        //       AutoFill for 'northwindDataSet.Products'.
        if (this.NeedsFill("northwindDataSet"))
        {
            this.productsTableAdapter.Fill(
                this.northwindDataSet.Products);
        }

        // TODO: Delete this line of code to remove the default
        //       AutoFill for 'northwindDataSet.Suppliers'.
        if (this.NeedsFill("northwindDataSet"))
        {
            this.suppliersTableAdapter.Fill(
                this.northwindDataSet.Suppliers);
        }
    }
}

There are a number of reasons why you might not want to automatically fill the data tables in the Startup event, which is why the comment points out that you can remove the auto-generated code. For example, as mentioned earlier, you might want to require that the user enter a database password before attempting to fill the DataSet. You can generate a new connection string and then set the adapter's Connection.ConnectionString property.

Or perhaps you want to give the user the option of whether to connect to the back end. If the user is on an expensive or slow connection, he might want to skip downloading a large chunk of data. For any number of reasons, you might not want to connect right away or use the default connection string, so Visual Studio allows you to modify this startup code.

Visual Studio generates strongly typed custom adapters at the same time as it generates the typed DataSet. If you read through the generated adapter code in NorthwindDataSet.Designers.cs, you will see that the generated adapter has been hardcoded to connect to the database specified by the connection string in the configuration file. The bulk of the generated adapter code consists of the query code used to read from the back-end data store into the typed DataSet, and to then take any changes in the DataSet and update or delete the appropriate rows in the store.

The adapter takes care of all the details of opening the connection, executing the query, copying the data into the DataSet, and closing the connection. At this point, you have a local copy of the data, which you can use to your heart's content without worrying about taxing the server further.

When you are finished editing the local copy of the data in the DataSet, you can use the adapter in order to update the database with the changes by calling the Update method of the adapter. The adapter will then make the various additions, changes, and deletions on the back-end database.

Using Binding Sources as Proxies

Why does Visual Studio bind the controls to a BindingSource proxy object, rather than binding controls directly to the data table? The reason is because the control can bind to the proxy even if the data in the table is not currently available. For instance, perhaps the data is going to be derived from a call to a Web service, which will not happen until long after the initialization is complete, or until the user types in his password or presses a button to start the database connection.

The proxy object is created when the customization starts up and controls can be bound to it even if there is no real data available. When the real data is available, the binding source updates the controls. It is essentially just a thin shim that makes it easier to set up bindings before all the data is available.

As you saw in the examples, multiple controls can share the same binding source and therefore have the same currency. That is, when one control is updated, every other control linked to the same binding source is also updated automatically. Controls on different worksheets can share binding sources and thereby share currency. It is easy to change the currency and move to another record in the database by using the MoveNext and MovePrevious methods on the BindingSource object.

Data-Bindable Controls

The last piece of the data binding puzzle is the host control (such as a Word bookmark or Excel range) or Windows Forms control on the spreadsheet or document that actually displays the bound data. There are two flavors of data-bindable controls: simple and complex. Controls that can bind a single datum to a particular property are simple-data-bindable. Controls that can bind multiple rows and/or columns are complex-data-bindable.

In the preceding examples, the list object in Excel and the combobox and DataGrid Windows Forms controls are complex-data-bindable; the list object and DataGrid display multiple rows and columns from a table, and the combobox displays multiple rows from a single column. The bookmark and named range controls by contrast are simple-data-bindable; only a single datum is bound to the Value property of the named range.

All the Windows Forms controls are simple-data-bindable, as are almost all of the Word and Excel host items and host controls. (There is one exception: the Word XMLNodes host control is neither simple- nor complex-data-bindable.) Of the host items and host controls, only the Excel list object is complex-data-bindable.

Special Considerations

It is easy to use data binding to move the data one way into the document or workbook. But to get the data back from Excel or Word to the database (presuming you want the user to be able to edit the data and push it back to the database) requires some extra steps.

First of all, you need to use the WriteValue method available on any simple data-bound host control or Windows Forms control. You do this by writing code like this:

myNamedRange1.DataBindings.Item(0).WriteValue()
You typically want to write this code within a change event for the named range or bookmark or at some other point before you are going to write back to the database.

This line of code gets the changed data from the named range in this case back to the DataSet. Next, you need to tell the bound DataSet to accept changes made to it (because of WriteValue) by calling AcceptChanges on the DataSet:

MyDataSet.AcceptChanges()
Alternatively, if you are using a BindingSource object, you can call EndEdit on the BindingSource object.

Finally, now that the change has been WriteValue'd back to the DataSet from the data-bound named range, and the change has been Accepted by the DataSet, you need to write the modified DataSet back to the actual database by using the table adapter that was used to fill the DataSet and calling Update on the table adapter:

Me.MyTableAdapter.Update(Me.MyDataSet.MyTable)
Note that you don't have to do the WriteValue and AcceptChanges steps if you are using a data-bound Excel list object—it takes care of these two steps automatically because it is doing complex data binding rather than simple data binding.

Conclusion

Visual Studio 2005 Tools for Office provides powerful tools for data binding that, for the most part, use the same model used by Windows Forms. It allows you to use your existing knowledge of data sources and data binding in Visual Studio 2005 to create data-bound Office solutions without having to understand the complex object models of Excel and Word.

Eric Carter and Eric Lippert are developers at Microsoft. They discuss Visual Studio Tools for Office in depth in the books Visual Studio Tools for Office: Using C# with Excel, Word, Outlook, and InfoPath (Addison-Wesley Professional, 2005) and Visual Studio Tools for Office: Using Visual Basic 2005 with Excel, Word, Outlook, and InfoPath (Addison-Wesley Professional, 2006). Portions of this article have been adapted from the book Visual Studio Tools for Office: Using C# with Excel, Word, Outlook, and InfoPath with permission from Addison-Wesley.


Agile Development: Extend Team Foundation Server To Enable Continuous Integration

Agile Development

Extend Team Foundation Server To Enable Continuous Integration

Ben Waldron
This article discusses:
  • Agile development methodologies
  • Visual Studio Team System and Team Foundation Server
  • Setting up a team project
  • Extending Team Foundation Server
This article uses the following technologies:
Visual Studio 2005 Team System, Team Foundation Server
Code download available at: TeamSystem.exe (173KB)
Contents
Agile Methodologies
Continuous Integration
Visual Studio 2005 Team System
Project Unit Testing
Creating a Team Project
Creating the Build Unit Tests
Defining the Build
Extending Team Foundation Server
Assigning a Work Item
Conclusion

M
any development teams have adopted "agile" methodologies to manage change and to improve software quality. These methodologies promote continuous integration as a practice to build and test software products incrementally as new features are included, bugs are fixed, and code is refactored. So how does Visual Studio® 2005 Team System and Team Foundation Server facilitate the process of agile development and continuous integration?

This article answers that question by creating an example project using agile concepts such as test-driven development (TDD) using the new unit testing features in Visual Studio 2005 Team System. After the project is completed, I'll show how to create a team project using Team Foundation Server and use this technology's extensibility features to build a custom Web service that enables continuous integration to build the application as code is checked into source control.

Agile Methodologies

Before drilling into specific agile development concepts, it is useful to revisit the history of agile development and define what makes a specific methodology agile.

Several methodologies were born in the 1990s as alternative approaches to process intensive development models like the Rational Unified Process (RUP). Some developers felt that these models did not represent the dynamic nature of software development where priorities often change and user feedback is critical. Through experience, many development teams recognized that software projects are built in many shapes and sizes, and that heavyweight software methodologies do not fit in many cases, especially for smaller, more dynamic development efforts. The predictive waterfall methodology, for example, does not scale well for rapidly changing development environments.

The extreme programming, adaptive software development, and Dynamic Systems Development Method (DSDM) methodologies were all created to allow project management and developers to adapt to change quickly without sacrificing quality. In retrospect, it was clear that these methodologies had many common themes, such as the ability to respond quickly to customer feedback and business demands, an understanding that customers have a strong voice in how the product evolves, short development iterations, and full testing coverage. Development teams began adopting the methodology that was the best fit for their organization and fed improvements back to evolve the methods over time.

Resources for Agile Development

Agile Development
Manifesto for Agile Software Development
Agile Alliance

Extreme Programming
Extreme Programming: A gentle introduction
Extreme Programming Wiki

Adaptive Software Development
Messy, Exciting, and Anxiety-Ridden Adaptive Software Development

Dynamic Systems Development Method
DSDM Consortium

Scrum
Jeff Sutherland's Blog
Control Chaos

Microsoft Solutions Framework
MSF for Agile Software Development

In 2001, a group of wise software professionals convened to discuss commonality among these methodologies—a meeting of the "five families" of adaptive software development. These software professionals coined the term "agile" to encompass all that was in common among the methodologies.

Since that time, agile development has grown and tools have been created to help. Meanwhile, development teams are reaping the benefits. Unit testing tools like NUnit have become integral parts of the development process. Likewise, continuous integration has become a standard practice that many development teams have adopted even without fully adopting an agile methodology. For more information on this topic, take a look at the "Resources for Agile Development" sidebar.

Continuous Integration

Continuous integration is a simple, powerful technique that Martin Fowler promoted as a best practice in software development, and a technique that fits easily into an agile development process (see Continuous Integration). Continuous integration takes the notion of a daily build a step further by building a software product each time any unit of the software changes. In most cases, a build is created each time a developer checks any code or configuration change into a source control. The build takes all files from the central source control repository and compiles the build. The status of the build is communicated to developers and testers on an ongoing basis so that failed builds can be caught quickly and corrected. Variations on this take it one step further, and perform the build before the check-in is finalized, allowing the check-in to be aborted if it would cause a break.

Extreme programming in particular emphasizes TDD, a natural fit into continuous integration. TDD proposes that the developer write a test case (typically a unit test), write the code exercised by that test, run the automated test, and refactor the code as necessary. Writing these tests effectively yields unit tests that give adequate test coverage across the product. These automated tests are placed in a common area where all developers and testers have access to run them before checking any code into the build.

This fits in well with continuous integration because, after a build is compiled, all unit tests are run on the latest build. If any unit test fails, then the build fails because something is not working properly. It is very important that the test cases cover as much of the product as possible to ensure a quality product. Another good practice is to produce unit tests for bugs that are fixed so that bugs can be verified and checked on an ongoing basis in an automated way.

One obvious benefit of continuous integration is transparency. Failed builds and tests are found quickly rather than waiting for the next build. The developer who checked in the offending code is probably still nearby and can quickly fix or roll back the change. If test cases are plentiful and thoughtfully composed, there are fewer burdens on quality assurance (QA) resources to find latent bugs, which gives you peace of mind regarding software quality. The result is more time for developing features and refactoring and less time spent trying to find integration errors.

Although the remainder of this article focuses on unit testing and continuous integration using Visual Studio 2005 Team System and Team Foundation Server, it is certainly possible to implement continuous integration without these tools. CruiseControl.NET is another example of a tool you can employ to implement continuous integration for your projects. However, I will show you how Team Foundation Server can automate builds to integrate seamlessly with the new unit-testing features in Visual Studio 2005.

Visual Studio 2005 Team System

Much has been written about Visual Studio 2005 Team System and its new capabilities for development teams. A good resource for learning about these features is the article, "Get All Your Devs in a Row with Visual Studio 2005 Team System" by Chris Menegay.

Team Foundation Server is the back-end server component that drives much of the Visual Studio Team System functionality. The feature that jumps out is the new source control system. This can replace Visual SourceSafe® and it even has a nice conversion utility to migrate your existing source code. A new version of Visual SourceSafe has also been released if you choose not to migrate.

Team Foundation Server has much more to offer than source control. You can track work items such as bugs, tasks, and project documentation. These items are stored and versioned, and are considered as valuable and as necessary as source code. Items that all team members, not just developers, need on a daily basis to complete tasks are stored within Team Foundation Server and presented through the Visual Studio user interface, allowing you to have this information in the tool you use most.

Team Foundation Server is driven by Web services that handle the interaction with Visual Studio clients. SQL Server™ 2005 is used to store work items and maintains sources as compressed reverse deltas from version to version. SQL Server Analysis Services and Reporting Services provide project-related reports for test results, metrics, and code churn. Windows® SharePoint® Services provides a portal for project collaboration and document repositories. The project team site lets team members who do not work in Visual Studio have access to project information and resources.

Source code is compiled centrally using the Team Foundation Build Server which can be installed on a standalone server. Code is checked out from the source control system, built, and tested, and the resulting build output is accessible through a network share. Multiple build types are supported, enabling developers and configuration managers to control how the project is built.

Since much of Team Foundation Server is written using the Microsoft® .NET Framework and Web services, it is easy to customize by subscribing to events and using the Team Foundation object model to automate tasks. I'll show an example of this that enables continuous integration by responding to source control check-in events.

Project Unit Testing

As an example, I'm creating, testing, and building a simple .NET library following the principle of extreme programming. I'm developing a math component that provides basic arithmetic functions using generics in the .NET Framework 2.0. A generic interface defines methods for the arithmetic functions such as add and subtract. This allows the Calculator library to invoke the correct methods based on type. The example library is not that important to the concepts conveyed, but can be explored to show the use of .NET generics. Note that all of the code for this article is available in both C# and Visual Basic®.

As all good extreme programming developers do, I first create unit tests for each of the public methods that I intend to implement. Typically, I stub out the public properties and methods before constructing the unit test. The new Class Diagram feature in Visual Studio makes it much faster to do this (see Figure 1).

Figure 1 CalculatorLibrary Class Diagram
Figure 1 CalculatorLibrary Class Diagram

Unit tests are created for the public methods of the CalculatorLibrary class. With Visual Studio Team System, unit tests are added to the solution by creating a new unit test project, which allows you to create Web page tests, load tests, code unit tests, and even document tests that can be run manually. In this case a code unit test class helps test accuracy of the mathematic methods. Figure 2 shows an example of a unit test that makes sure the sum of all items added to the list equals 86. If an exception is encountered or the value is anything other than 86, the test will fail.

Figure 2 Unit Test for the Sum Method (Close Figure 2)
'''<summary>Initialize the test</summary>
<TestInitialize()> _
Public Sub Initialize()
    calculatorLibrary = New CalculatorLibrary(Of Integer)( _
        New Math.Integer.Calculator())
End Sub

''' <summary>
''' Test Sum Method for Int type.  (11+2+73=86)
''' </summary>
<TestMethod()> _
Public Sub TestIntSumMethod()
    Dim itemsToAdd As List(Of Integer) = New List(Of Integer)
    itemsToAdd.Add(11)
    itemsToAdd.Add(2)
    itemsToAdd.Add(73)
    Dim result As Integer = calculatorLibrary.Sum(itemsToAdd)
    Assert.AreEqual(86, result)
End Sub

The Initialize method defines any work necessary to set up the tests. In this case, the Calculator object is created. Test initialization methods must be decorated with the TestInitialize attribute. Likewise, test methods are decorated with the TestMethod attribute. These attributes are necessary for the methods to be recognized by the testing framework in Visual Studio, and later on when I build the product.

Figure 3 Reviewing Local Test Results
Figure 3 Reviewing Local Test Results

Once unit tests are completed, the Calculator library can be implemented. Then I'm ready to run the unit tests locally. The Test Manager allows selection and execution of all tests located in the current solution. The Test Results window in Figure 3 shows that all my tests have passed and I'm ready to continue with the next step; specifically, creating a new team project in Team Foundation Server and adding the solution to it.

Creating a Team Project

After Team Foundation Server has been set up on the network, connecting to the server in Visual Studio is straightforward. In the Tools menu, just select Connect to Team Foundation Server. There is an option to add a new server, typically through a URL, to the server running on the default port of 8080. In my case, the URL, is TeamFoundation:8080.

When you are connected to the server, you can create a new project. Usually the name of the project is more encompassing than the name of the solution. I chose to call my project MSDN Calculator, as shown in Figure 4. Typically, a team project will be created before any code is written, as the team is going through the definition phase for the application being built. The team project might contain many solutions and Visual Studio projects, but the more interesting thing is that it's about much more than source control. Team projects have members with specific roles, they follow a specific software development methodology, they have a set of work item types corresponding to their methodology, they have a portal for collaborating and communicating projectrelated information, and so on. As such, a more appropriate team project name might be something like "Enterprise Infrastructure," as companies frequently have a set of common classes shared by many applications and need a team project to manage their development. However, I've gone with MSDN Calculator for simplicity of example.

Figure 4 Creating a New Team Project
Figure 4 Creating a New Team Project

The next series of steps allow me to define the project in more detail. First I can define the process template that corresponds to the development methodology I'm using. As of Beta 3 the choices include Microsoft Solutions Framework (MSF) for Agile Software Development and MSF for CMMI Process Improvement. Since I'm using an extreme programming methodology, I'll choose the agile development template. Note, however, that these templates can be customized and you can create new templates as well. These templates are important because they define the work items, reports, and security settings for the project.

The next step is setting up source control for the project. The interface for doing this is much like Visual SourceSafe and will be intuitive to users who are migrating from that environment. For this project I accept the default to create an empty source control folder named after the project.

After completing this step, a dialog box confirms the settings I made and a new project is created. This takes some time because much work goes into creating a new project. On the back end, a SharePoint site is created, work items are generated, reports are defined, and all other setting are configured according to the process template. At this time, I can configure security on the project by defining roles for developers, testers, and other contributors. Now I can check in some items and define my build settings. In this case, I can also extend Team Foundation Server to support continuous integration.

Thus far I have created a .NET library, complete with a suite of unit tests, and I have created a new team project. The next step is to prepare the units tests so that they can run remotely on the Team Foundation Build Server.

Creating the Build Unit Tests

Within Visual Studio Team System, the Test Manager will automatically create a test metadata file with a .vsmdi extension. This file contains the unit test configuration for the project, which enables a unit test to be run with a remote build.

Within Test Manager I create two lists of tests by right-clicking in the left pane and selecting New Test List. I create one list that houses the unit tests for passing Ints to CalculatorLibrary and another for the tests that cover passing the Doubles (see Figure 5). Note that I did not include any of the manual tests since the build will be happening in the background with no human interaction. This data is saved in the .vsmdi file, which is checked into source control.

Figure 5 Project Test Configuration in Test Manager
Figure 5 Project Test Configuration in Test Manager

Within Team Foundation Server is the ability to define source control check-in policies, but I have chosen not to configure that for this project. A check-in policy is a great feature for imposing uniformity across a team of developers.

Figure 6 Project Contents
Figure 6 Project Contents

At this point, the solution and project settings are configured as shown in Figure 6. This shows the main project that contains the Calculator library and the unit testing project. The test metadata file is also shown as a solution item that is bound to source control.

Defining the Build

I'm now ready to define build types for this project. In this case I am creating a build that will compile both the math library and the unit test project as debug builds and then run both of the test lists that I have defined. I name the build Continuous Integration. This name will be used later when I customize the build. If the build or any of the unit tests fail, I can notify the developers in various ways to correct the problem.

Within the Team Explorer window, the Team Builds option allows you to create new build types. The wizard takes you through the process to name the build type, select the projects to compile, and specify the build machine. In my configuration, I have the build service installed and running on the Team Foundation Server, but that will probably not be the typical scenario since the build is likely to be done on a standalone server. The last page of the wizard enables me to select the unit tests that the build will run after the project has compiled. In this case, I select both sets of unit tests that are defined (see Figure 7).

Figure 7 Selecting Tests to Run After a Build
Figure 7 Selecting Tests to Run After a Build

Once the Continuous Integration build is defined, the best way to test it is to force a build through the Visual Studio Team Explorer user interface. The Continuous Integration build can be seen under the Team Build folder within Team Explorer. A simple right-click will show the option to build the team project. Build status is reported within Visual Studio.

Extending Team Foundation Server

Team Foundation Server was built with extensibility in mind. One of the first indications of this feature that you're likely to notice is the ability to subscribe to events, particularly source control events. A command-line utility called BisSubscribe.exe is included with the Team Foundation Sever installation and is used to subscribe to these types of events. But before the subscription is set up, a Web service is created to catch the events. In Beta 3, the BisSubscribe utility can be found in the C:\Program Files\Microsoft Visual Studio 2005 Team Foundation Server\TF Setup folder.

BisSubscribe.exe allows you to define the types of events that you'll subscribe to and defines the exact signature and attributes of the Web service method used to catch the events. A proxy generated for the Notify Web method and modified to execute some work is shown here:

<SoapDocumentMethod(Action:="http://schemas.microsoft.com/
TeamFoundation/2005/06/Services/Notification/02/Notify", _
RequestNamespace:="http://schemas.microsoft.com/TeamFoundation/2005/06/
Services/Notification/02"), WebMethod()> _
Public Sub Notify(ByVal eventXml As String)
    ' implement event functionality
    ThreadPool.QueueUserWorkItem(AddressOf BuildProject, eventXml)
End Sub
To create the subscription, you would run this command:
BisSubscribe /eventType CheckinEvent /userId PICCOLO\TFSSetup /address
TeamFoundation:8080/Continuous/Build.asmx /deliveryType Soap /domain
TeamFoundation

The address parameter is worthy of note. This is the location of the Web service that contains the Notify Web method. The easiest location is to create an additional Virtual Root beneath the Web site that contains the Team Foundation Server Web services (this is because the account that the Team Foundation Server application pool runs as already has permission to read the source and kick off a build; if you set up a different Web service, you'll need to ensure that it runs with credentials to do everything that needs to happen during a build). The physical location of these Web services is in the C:\Program Files\Microsoft Visual Studio 2005 Team Foundation Server\Web Services folder. As seen by the command-line options, I have added a virtual root called Continuous to service these requests.

The Notify Web method accepts an XML document in the eventXml parameter. This document contains all of the metadata for a particular event. Figure 8 shows the check-in event XML where I have checked in an intentional error in MathLibrary.cs.

Figure 8 Check-In Event XML (Close Figure 8)
<CheckinEvent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Artifacts>
    <Artifact xsi:type="ClientArtifact"
      ArtifactType="Changeset" ServerItem="">
      <Url>http://TeamFoundation:8080//VersionControl/Changeset.aspx?
           artifactMoniker=90&webView=true</Url>
    </Artifact>
    <Artifact xsi:type="ClientArtifact" ArtifactType="VersionedItem"
      Item="MathLibrary.cs"
      Folder="$/MSDN Calculator/Msdn.MathLibrary/Msdn.MathLibrary"
      TeamProject="MSDN Calculator" ItemRevision="111" ChangeType="edit"
      ServerItem="$/MSDN Calculator/Msdn.MathLibrary/
                  Msdn.MathLibrary/MathLibrary.cs">
    </Artifact>
  </Artifacts>
  <Title>MSDN Calculator Check-in 111: Added intentional error</Title>
  <ContentTitle>Check-in 111: Added intentional error</ContentTitle>
  <Owner>PICCOLO\\bwaldron</Owner>
  <Committer>PICCOLO\\bwaldron</Committer>
  <Number>90</Number>
  <CreationDate>10/15/2005 6:06:01 PM</CreationDate>
  <Comment>Added intentional error</Comment>
  <TimeZone>Pacific Daylight Time</TimeZone>
  <TimeZoneOffset>-07:00:00</TimeZoneOffset>
  <TeamProject>MSDN Calculator</TeamProject>
  <PolicyOverrideComment />
</CheckinEvent>

From the Notify Web method, I call another method that will build the project and report results to the development team. In ref="fig9">Figure 9, I extract the needed information from the event XML. Creating a build requires two steps. First, I need to create the parameters that are passed to the build controller to start a build. The minimum parameters are shown in the example. The build controller needs the Team Foundation Server name, project name, build server name, and the directory on the build machine to use for compilation. A predefined build type must be specified as well. In this case I supply the Continuous Integration build type that was defined previously in the team project.

Figure 9 Starting a Continuous Integration Build (Close Figure 9)
Private Sub BuildProject(ByVal state As Object)
    Dim eventXml As String = CType(state, String)

    ' parse info from Event XML
    Dim xmlDocument As XmlDocument = New XmlDocument()
    xmlDocument.LoadXml(eventXml)
    Dim teamProject As String = _
        xmlDocument.DocumentElement("TeamProject").InnerText
    Dim requestor As String = _
        xmlDocument.DocumentElement("Committer").InnerText
        Dim changeTitle As String = _
            xmlDocument.DocumentElement("ContentTitle").InnerText
        Dim offset As Integer = requestor.IndexOf("\"c)

        If offset > -1 Then
            requestor = requestor.Substring(offset + 1)
        End If

        ' create the build parameters for StartBuild method
        Dim buildParameters As _
            Microsoft.TeamFoundation.Build.Proxy.BuildParameters = _
            New Microsoft.TeamFoundation.Build.Proxy.BuildParameters()
        buildParameters.TeamFoundationServer = FoundationServer
        buildParameters.TeamProject = teamProject
        buildParameters.BuildDirectory = BuildDirectoryPath
        buildParameters.BuildMachine = BuildServer
        buildParameters.BuildType = BuildType

        Dim status As BuildConstants.BuildStatusIconID = _
            BuildConstants.BuildStatusIconID.Default
        Try
            ' create the build controller, start build and record
            Dim buildController As BuildController = _
             BuildProxyUtilities.GetBuildControllerProxy(FoundationServer)
            Dim uri As String =
                buildController.StartBuild(buildParameters)

            Dim buildData As _
                Microsoft.TeamFoundation.Build.Proxy.BuildData = Nothing
            Dim buildStore As BuildStore = _
                BuildProxyUtilities.GetBuildStoreProxy(FoundationServer)

            ' wait until the build completes
            While status = BuildConstants.BuildStatusIconID.Default
                'wait a second to check status
                Thread.Sleep(1000)
                buildData = buildStore.GetBuildDetails(uri)
                status = CType(buildData.BuildStatusId, _
                    BuildConstants.BuildStatusIconID)
            End While

            If status = BuildConstants.BuildStatusIconID.BuildFailed Then
                ' create a workitem for the developer who checked in
                AssignWorkItem(requestor, changeTitle, buildData)
            End If
            NotifyGroup(requestor, buildData)
        Catch soapEx As SoapException
            ' TODO: e-mail build administrator
        End Try
    End Sub

After the build parameters are defined, I obtain a proxy to the Build Controller. This is a Web service proxy that calls the StartBuild method with the defined parameters. A unique Uniform Resource Identifier (URI) is returned from the StartBuild method that will allow me to check the status of the build. After starting the build, another proxy is created to connect to the Build Store, the object where I check the status of the build and its details. I use a while loop to contact the Build Store periodically, passing in the URI from the StartBuild method, in order to determine whether the build has completed.

Assigning a Work Item

I check the status of the build after it completes. If the build has failed, I call the AssignWorkItem method to assign a case to the developer who checked in the offending code. To do this I retrieve another Web service proxy, this time to the Work Item Store. This allows me to create and save a new work item, as shown in the code in Figure 10.

Figure 10 Assign a New Work Item (Close Figure 10)
Private Sub AssignWorkItem(ByVal requestor As String, _
    ByVal changeTitle As String, ByVal buildData As _
    Microsoft.TeamFoundation.Build.Proxy.BuildData)

    Dim server As TeamFoundationServer = _
        TeamFoundationServerFactory.GetServer(FoundationServer)
    Dim store As WorkItemStore = _
        CType(server.GetService(GetType(WorkItemStore)), WorkItemStore)

    Dim workItemTypes As WorkItemTypeCollection = _
        store.Projects(buildData.TeamProject).WorkItemTypes

    ' Enter the work item as a bug
    Dim workItemType As WorkItemType = workItemTypes("bug")
    Dim workItem As WorkItem = New WorkItem(workItemType)

    workItem.Title = String.Format("Build {0} broken because " & _
        "check-in changes.  Investigate and correct: {1} ", _
        buildData.BuildNumber, changeTitle)
    workItem.Fields("System.AssignedTo").Value = requestor
    workItem.Fields("Microsoft.VSTS.Common.Priority").Value = 1
    workItem.Save()
End Sub

The work item will show up as a task within the My Work Items list for the developer who caused the build to fail. The work item assigned based on my intentional failure is shown in Figure 11. This task gives the details necessary for the developer to fix the error, and also tracks when the issue is completed. Many metrics can be derived from these work items, including how many times a build was broken during an iteration and the amount of time it took to fix the error. All of this information gives transparency to the development effort.

Figure 11 Automatically Assigned Work Item
Figure 11 Automatically Assigned Work Item

In addition to assigning a work item, an e-mail is sent out to the development team alerting them of the build status. An e-mail can be sent out even if the build is successful. A good extension point would be to integrate the ability to send an instant message or another alerting mechanism. An example of the e-mail for a failed build is shown in Figure 12.

Figure 12 Automated E-Mail Sent to Development Team
Figure 12 Automated E-Mail Sent to Development Team

The e-mail provides a link to the drop of the newly built project and to the build log. The build log contains information needed to determine the cause of a failed build or a failed test case. This entire workflow is automated without a single person having to check on the status or manually assign work items, enabling the development team to concentrate on the most important item: building quality software.

Conclusion

Visual Studio 2005 Team System and Team Foundation Server have many features that support agile development methodologies. With unit testing and work items ingrained into the product, using a TDD approach has never been easier within Visual Studio. As I've demonstrated, extending Team Foundation Server to support practices like continuous integration is as easy as catching events and connecting to the server's infrastructure.

Ben Waldron is the founder of Piccolo Technology, a .NET training and solutions company, and is lead architect at Learning.com where he is developing technical education solutions. He is based in Portland, OR and can be contacted at bwaldron@PiccoloTechnology.com.


Reporting Services: Deliver SQL Server Reports To SharePoint To Enhance Team Collaboration

Reporting Services

Deliver SQL Server Reports To SharePoint To Enhance Team Collaboration

Ed Hild
This article discusses:
  • The architecture of a report collaboration solution
  • Creating a SharePoint Web service to save reports
  • Building a report delivery extension
  • Deploying the solution to a SharePoint site
This article uses the following technologies:
SQL Server Reporting Services, Windows SharePoint Services
Code download available at: ReportingServices.exe (204KB)
Contents
Architecture
Building the Web Service
Building the Delivery Extension
Deployment
SQL Server 2000 and SQL Server 2005
Conclusion

I
magine you work for a company that has a critical database application that continually records new incoming information. This database could be storing order information, sales contacts, or just about any other kind of data. Now consider that a group of individuals within your company rely on reports about this data—perhaps trend analysis of monthly billing statements—for making key business decisions. These reports are created on a schedule: daily, weekly, or monthly. Sound familiar? Chances are your company has similar systems in place. Just about any type of company can relate to this.

Microsoft® SQL Server™ Reporting Services simplifies this scenario by allowing users to subscribe to a report and have it automatically delivered to them. But while this is a useful feature, it only addresses one aspect of the larger objective. Once users receive a report, they'll likely want to do more than just read through the data. Perhaps they'll need to create and assign tasks to employees, or maybe they will want to allow group members to have a constructive discussion about the data, or they may simply need to incorporate the data into a Microsoft PowerPoint® presentation.

The problem is that the system delivers a copy of the report to each individual by e-mail or sticking it on an internal company file share—the two out-of-the-box delivery options for subscriptions. Neither case supports collaboration regarding the report within the group. A more interesting solution would be to deliver the report to a Microsoft SharePoint® document library where discussion and collaboration features are readily available to be used with the report. This article details how to create a custom SQL Server Reporting Services delivery extension that will store the report in a Windows® SharePoint Services (WSS) document library.

Architecture

In the scenario I've described, the time of the report is critical. The report must be identical for all users regardless of when they view it. Therefore, I cannot rely on a model where each user runs the report on demand when they happen to visit the SharePoint site. (An on-demand solution could easily be accomplished using the new report viewer web part in SQL Server 2000 Reporting Services Service Pack 2, however the collaboration features of SharePoint would not be applicable.) Instead, the custom delivery extension must run the report at a scheduled time and deliver it in a completely automated fashion. Since SharePoint is capable of storing documents, my delivery extension will render the report as a PDF file and store it in a designated document library.

So how does the file, once it's been generated, make its way to the document library? The solution includes a custom SharePoint Web service that provides an interface to accept the report file and, through the WSS object model, stores the file in the appropriate library. Using a custom Web service adds flexibility, letting you include additional tasks, other than simply storing the file.

To facilitate the inclusion of a binary file in the request, the Web service will use the Web Service Enhancements (WSE) Direct Internet Message Encapsulation (DIME) with WS-Attachment functionality. I selected this method over base64 encoding of the binary data within the XML message since, in a Unicode character format, the base64 encoding method inflates the size of the request. Sending attachments with DIME moves the file outside of the SOAP envelope. Figure 1 shows this architecture at a high level.

Figure 1 Solution Architecture Overview
Figure 1 Solution Architecture Overview

The beauty of this architecture is that the end result is just a file in a document library. Users can, therefore, use all of the regular functionality that SharePoint has to offer, including collaboration and SharePoint alerts. If someone needs to confirm a report is valid and sign off on it before other users are able to view it, she can turn on content approval. If the content of the reports must be searchable, simply install a PDF IFilter.

Building the Web Service

The custom SharePoint Web service has just one method, UploadDocument. This method only accepts SOAP Web service requests that have a file attachment. To receive a set of parameters from the client, the Web service exposes a DocInfo class that has been specifically designed to provide flexibility as well as version resiliency (see Figure 2).

Figure 2 DocInfo Class (Close Figure 2)
<XmlType(Namespace:="http://sample.microsoft.com/DocumentUtil")> _
Public Class DocInfo
    ' Fields
    Public FileName As String
    Public Folder As String
    Public Version As String

    <XmlAnyElement()> _
    Public UnknownElements As XmlElement()
    <XmlAnyAttributeAttribute()> _
    Public UnknownAttributes As XmlAttribute()
End Class

At a minimum, this Web service must receive two strings: the names of the file and the library. However, the solution must remain flexible with the ability to support additional parameters in the future. In fact, this Web service is likely to add capabilities to your SharePoint implementation that go well beyond merely storing reports. For more information on building a class so that its serialization is open-ended, see Doug Purdy, a Program Manager at Microsoft, discuss this topic on the MSDN® TV episode "Loosely Coupled Web Services".

The element and attribute arrays in the DocInfo class allow me to add fields in the future without breaking backwards compatibility. Meanwhile, the version identifier enables the Web service method to discover what level of client is making the request so that it can act accordingly.

Once the UploadDocument Web service method has confirmed that the request has an attachment, it uses a BinaryReader to read the attachment into a byte array (see Figure 3). In the sample code (available in both C# and Visual Basic in the download), you will see the use of RequestSoapContext. This class provides access to the WSE-specific extensions of the Web service. Through its Attachments collection, you can access the actual PDF report file that the delivery extension is sending. With the attached report received, the UploadDocument method continues its processing using the SharePoint object model to determine which "web" the client is targeting:

Dim site As SPWeb = SPControl.GetContextWeb(Me.Context)

Figure 3 Accessing the Attached Report (Close Figure 3)
Public Sub UploadDocument(ByVal info As DocInfo)
    'Reject any requests which are not valid SOAP requests
    If (RequestSoapContext.Current Is Nothing) Then
        Throw New ApplicationException( _
            "Only SOAP requests are permitted.")
    End If

    If (RequestSoapContext.Current.Attachments.Count = 0) Then
        Throw New ApplicationException( _
            "No attachments were sent with the message.")
    End If

    Dim reader As BinaryReader
    Try
        Dim fileData As Byte() = Nothing
        'retrieve the attachment
        reader = New BinaryReader( _
            RequestSoapContext.Current.Attachments(0).Stream)
        fileData = reader.ReadBytes(CType(reader.BaseStream.Length, _
            Integer))
        ...

In this context, "web" is a generic SharePoint object model term. It can refer to a Windows SharePoint Services (WSS) site or a SharePoint Portal Server (SPS) area. The correct web is determined using the SharePoint context object, which is based on the URL used by the client to make the request. The SharePoint site's Web services are deployed in such a way that they can be referenced through the vti_bin folder of any web. To gain this functionality you must make sure to deploy your Web service in the same manner as the out-of-the-box services. I explore this topic in more detail in the deployment section of this article.

Once the UploadDocument method determines the web in which the report should be stored, the GetFolder method is used to retrieve the correct document library. The library then adds the byte array to its files collection using the given file name:

Dim folder As SPFolder = site.GetFolder(info.Folder)
Dim file As SPFile = folder.Files.Add(info.FileName, fileData, True)

This process saves the report to the SharePoint document library. The True argument specified in this call specifies that the current file will overwrite any existing file with the same name. If the user has enabled versioning on the SharePoint library, the new file will actually update the existing file. As described later in this article, the delivery extension also has an option to append a timestamp into the file name. This is handy if a user does not want to enable versioning of the library, but would still like to keep all the report instances.

Building the Delivery Extension

The out-of-the-box delivery options I mentioned earlier, delivery via e-mail and delivering the file to a file share, each constitute a delivery extension. To get the report to a SharePoint library, I must build a custom delivery extension. This involves coding three classes: one that stores the subscription settings, one that renders the user interface, and one that actually delivers the report. When a user sets up a subscription to deliver a report to SharePoint, they need to provide four pieces of information: the URL of the WSS site or SPS area, the name of the document library, the file name to use for the report, and whether a timestamp should be added to the file name. Each of these values is stored in a SubscriptionData class.

The solution also includes an SPSLibraryDeliveryUIProvider class that generates the user interface used to capture this information from the user. This class renders the forms fields that are specific to the SharePoint library delivery extension. Figure 4 shows the subscription entry form for the custom delivery extension. The SPSLibraryDeliveryUIProvider is only responsible for the three textboxes and one dropdown list that match these settings.

Figure 4 Setting Up a Subscription (Close Figure 4)

Most of the work of a delivery extension involves coding the class that actually delivers the report. The class I created is named SPSLibraryDeliveryProvider. It implements the IExtension and IDeliveryExtension interfaces. This class is automatically instantiated by the Reporting Services engine when a report is ready to be delivered and the user has selected my custom solution as the method of delivery.

In the class, the Deliver method is where the action begins. This method receives a notification parameter that is used to retrieve the user-specific settings for delivery as well as to communicate progress or failure to the host service. The Deliver method, in turn, calls SaveReport, which is where the delivery extension calls the custom SharePoint Web service. SaveReport begins by rendering the report into a PDF file (see Figure 5).

Figure 5 SaveReport Implementation (Close Figure 5)
Private Sub SaveReport(ByVal notificationObj As Notification, _
    ByVal data As SubscriptionData)

    Dim files() As _
        Microsoft.ReportingServices.Interfaces.RenderedOutputFile = _
            notificationObj.Report.Render("PDF", "<DeviceInfo/>")

    If (files(0).Data.Length > 0) Then
        ' WEBSERVICEREF is a literal string constant
        ' whose value is "/_vti_bin/DocumentUtil.asmx"

        Dim serviceProxy As DocumentUtilService.DocumentUtilWse = _
            New DocumentUtilService.DocumentUtilWse
        serviceProxy.Url = data.siteUrl + Me.WEBSERVICEREF

        ' establish endpoint for wse addressing
        Dim virtualServerUrl As String = CreateVSUrl(data.siteUrl)
        Dim dest As EndpointReference = New EndpointReference( _
            New Uri(virtualServerUrl + Me.WEBSERVICEREF))
        serviceProxy.RequestSoapContext.Addressing.Destination = dest

        'credentials
        serviceProxy.Credentials = _
            System.Net.CredentialCache.DefaultCredentials

        'build delivery Info
        Dim info As DocumentUtilService.DocInfo = _
            New DocumentUtilService.DocInfo
        info.FileName = GenerateFileName(data)
        info.Folder = data.libraryName

        ' build the attachment
        Dim attachment As DimeAttachment = _
            New DimeAttachment("application/pdf", _
            TypeFormat.MediaType, files(0).Data)
        serviceProxy.RequestSoapContext.Attachments.Add(attachment)

        'deliver
        serviceProxy.UploadDocument(info)
    End If
End Sub

The next section of the SaveReport method configures the proxy class that Visual Studio® created when a Web reference was added to the project. In the code, the proxy class is DocumentUtilService.DocumentUtilWse. To initialize the proxy, I must set its Url property to the URL provided by the user, but with /_vti_bin/DocumentUtil.asmx appended. This will ensure that the SharePoint context is properly established for the code within the Web service method. Since SharePoint actually modifies the requests that come into a site's _vti_bin directories and routes them to the _vti_bin at the root of the virtual server, the proxy must be configured with a destination endpoint reference. By setting the destination endpoint reference, the resulting request will pass the validation enforced by the WS-Addressing logic within WSE. The URL of the endpoint reference is constructed using the CreateVSUrl function, which determines the URL of the virtual server from the URL of the site that the user provided.

Lastly, the proxy's security credentials are set. The Web service call will be made with the security context of the account that was assigned to the ReportServer service. However, this is done only for the example. Microsoft does not recommend running the service with elevated privileges, as it can open your network to elevation of privilege attacks. Rather, you should gather specific credentials in the delivery extension settings and use those to make the requests.

The last section of the SaveReport method actually populates the request with data. A DocInfo class is created and filled with the file name of the report and the name of the library in which it should be stored. The GenerateFileName function called here takes the user specified file name and adds the .pdf file extension and, optionally, the chosen timestamp type. The binary representation of the report is attached to the request using the RequestSoapContext.Attachments collection. Finally, the delivery extension sends its information to the custom SharePoint Web service.

Deployment

Key deployment items are described in more detail in the instructions that come packaged with the sample code, but are noted here to ensure that you will know to keep them in mind.

The ReportServer Windows service that is installed by SQL Server Reporting Services should be configured with an Active Directory® domain account. By assigning it a domain account, administrators will be able to assign the delivery extension rights to specific SharePoint libraries.

The custom SharePoint Web service must be deployed as described in the MSDN article, "Writing Custom Web Services for SharePoint Products and Technologies". This involves creating .aspx files from the .wsdl and .disco files and placing them in the C:\Program Files\Common Files\Microsoft Shared\Web server extensions\60\ISAPI folder.

The delivery extension assembly must be deployed to the ReportManager and ReportServer directories of SQL Server Reporting Services. The delivery extension must also be added to the RSWebApplication and RSReportServer configuration files. Lastly, there are code access security (CAS) configurations that need to be addressed in the Rssrvpolicy and Rsmgrpolicy files. For detailed information on this, see the MSDN article "Code Access Security in SQL Server 2000 Reporting Services".

SQL Server 2000 and SQL Server 2005

I originally wrote the solution described in this article using SQL Server 2000. The good news is that for SQL Server 2005, the interfaces on which I'm relying remain unchanged and, with the exception of an upgrade/recompile of the delivery extension project, everything went fine.

For SQL Server 2005, the Reporting Services directory is located at C:\Program Files\Microsoft SQL Server\MSSQL.3\Reporting Services (if you install SQL Server, Analysis Services, and Reporting Services together). Follow the same instructions for modifying the configuration and policy files while placing the upgraded SPSLibraryDelivery.dll in the bin folders.

For my test, I also installed Windows SharePoint Services Service Pack 1, extended a new IIS Web site, and created a top-level team site. The deployment instructions for the custom Web service remained unchanged.

Conclusion

There are many different methods of generating reports to provide users with critical information. The challenge for most organizations is not producing such reports, but rather delivering them in a timely and effective manner. With the custom delivery extension described here, reports can be captured and placed within SharePoint document libraries, allowing users to view and analyze the reports in a more robust manner that supports collaboration and other useful features.

Ed Hild is a technology architect at the Microsoft Technology Center in Virginia. He delivers architecture design sessions, proofs of concept, and strategy briefings to customers with emphasis on the Microsoft collaboration platform. He can be reached at edhild@microsoft.com or visit his blog at blogs.msdn.com/edhild.


The Perfect Host: Create And Host Custom Designers With The .NET Framework 2.0

The Perfect Host

Create And Host Custom Designers With The .NET Framework 2.0

Dinesh Chandnani
This article discusses:
  • Understanding designers and services
  • Creating a hosting form
  • Building and using a toolbox
  • Loading and managing designers
This article uses the following technologies:
Windows Forms, Visual Studio 2005
Code download available at: DesignerHosting.exe (229KB)
Contents
Extensibility with Services
DesignSurface and DesignSurfaceManager
The Hosting Form
The Toolbox
Multiple DesignSurfaces
DesignerLoaders
CodeDomDesignerLoader
Conclusion

V
ersion 1.0 of the Microsoft® .NET Framework provided a very flexible design-time architecture, but offered virtually no implemented code to actually create and host designers. All of the hosting logic was implemented in Visual Studio® .NET, requiring third parties to rewrite all of this complex logic. This has now changed; the .NET Framework 2.0 introduces a set of classes that can be used to host designers right out of the box.

Figure 1 Runtime
Figure 1 Runtime

To understand how .NET Framework designers work, it is important to know how designers are used. A designer is an object that only exists at design time and that connects to an object that normally exists at run time. The framework connects these two objects and provides a conduit for the design-time object to augment the behavior of the run-time object. At run time, a form and a button on that form are connected only through the parent/child relationship between the two controls (see Figure 1). There is no other object that controls the lifetime of these controls.

Figure 2

The picture looks a little more complex at design time. Both the form and the button have designers associated with them. Both objects are also connected to a host container (see Figure 2), which owns these two objects. The host container also provides services—such as selection services to select one or more objects at design time, and UI services for displaying messages, invoking help, and interacting with the development environment—that the objects and designers may use.

The host container has many responsibilities. It creates components, binding them to designers and providing services to the components and designers it maintains. It loads designers from some sort of persistent state and saves them back to that state. The host container provides logic for undo, clipboard functions, and many other services that the designers rely upon to deliver a robust design-time environment.

Figure 3 Designer Hosting
Figure 3 Designer Hosting

The host container can also persist the state of the designer. To do this, it would use a designer loader. The designer loader may, in turn, use serializers to serialize the components, as shown in Figure 3.

Extensibility with Services

The .NET Framework designer architecture is extensible. Critical to this extensibility, services provide the ability to enhance the functionality available to the various designers. A service is any object that can be queried by a type. Typically you would define some abstract class or interface that represents the service and then provide an implementation of that service. You can add services to and remove services from an object called a service container. IDesignerHost, the primary host interface for designers, is a service container. A service is a feature that may be shared between components written by different parties. Because of this, you must follow certain rules when using and creating services.

Services are not guaranteed. Whenever you ask for a service by calling the GetService method, you must always check to see whether GetService returned a valid object. Not all services are available on all platforms, and services that were once available may not be available in the future. Thus, your code should be written to degrade gracefully, usually by disabling the feature that required the service, in case a service does become unavailable.

If you add a service, remember to remove it when your designer is disposed. The designer host may destroy and recreate your designer from time to time. If you fail to remove a service, the old designer may be left in memory.

DesignSurface and DesignSurfaceManager

The .NET Framework 2.0 introduces two classes that are used for hosting designers and providing services to the designers: DesignSurface and DesignSurfaceManager. DesignSurface is what the user perceives as a designer; it is the UI the user manipulates to change design-time features. DesignSurface may be used as a standalone designer or it may be coupled with DesignSurfaceManager to provide a common implementation for an application that hosts multiple DesignSurfaces.

DesignSurface provides several design-time services automatically (see Figure 4). Most of these can be overridden in the service container. It is illegal to replace the non-replaceable services because their implementations all depend on each other. Note that all services added to the service container that implement IDisposable will be disposed when the design surface is disposed.

Figure 4 Default DesignSurface Design-Time Services (Close Figure 4)

Service | Replaceable | Description
IComponentChangeService | No | Raises events as changes are made to components.
IDesignerHost | No | Master interface for designers. Controls access to types, services, and transactions.
IContainer | No | Each designer has an IContainer that owns the set of components that are being designed.
IServiceContainer | No | Derives from IServiceProvider and provides a way to add and remove services from the designer.
IExtenderProviderService | Yes | Allows objects that are not part of the container's Components collection to provide their own extender providers. The IExtenderProviderService allows clients to register IExtenderProviders. An IExtenderProvider provides new custom properties on existing objects such as Modifiers.
IExtenderListService | Yes | Used by TypeDescriptor to get a list of extender providers. This allows extender providers to live outside of the container.
ITypeDescriptorFilterService | Yes | Primary interface for metadata filtering. This provides designer metadata hooks. You can use this, for example, to hide properties from the Property window that should not be displayed or to change other attributes on components.
ISelectionService | Yes | Provides a way to select components in the designer.
IReferenceService | Yes | Provides a way to get a name for objects, even when those objects are not sited.
IPropertyValueUIService | Yes | Provides a simple property/UI dictionary that can be used by property windows and designers to add glyphs to properties.

In addition to the default services, DesignSurface also provides IDictionaryService, available through a component's site. This service, which provides a generic dictionary of key/value pairs that can be used to store arbitrary data about a component, is unique to each component. It is not possible to replace these services because there is no way to replace services on a per-site basis.

DesignSurfaceManager is intended to be a container of designers. It provides common services that handle event routing between designers, property windows, and other global objects. Using DesignSurfaceManager is optional, but recommended if you intend to have several designer windows.

DesignSurfaceManager also provides several design-time services automatically (see Figure 5). Each of these can be overridden by replacing their value in the protected ServiceContainer property. As with DesignSurface, all DesignSurfaceManager services added to the service container that implement IDisposable will be disposed when the designer application is disposed.

Figure 5 DesignSurfaceManager Design-Time Services (Close Figure 5)

Service | Description
IUIService | Provides a way for components to show a UI such as error messages and dialog boxes
IDesignerEventService | Provides a global eventing mechanism for designer events

IDesignerEventService is a particularly useful service. It allows an application to know when a designer becomes active. IDesignerEventService provides a collection of designers and is a single place where global objects, such as the Property window, can listen to selection change events.

The Hosting Form

To demonstrate how simple it is to host a designer, I wrote the following sample code to create a basic Windows® Forms designer and display it:

// Create the DesignSurface and load it with a form
DesignSurface ds = new DesignSurface();
ds.BeginLoad(typeof(Form));

// Get the View of the DesignSurface, host it in a form, and show it
Control c = ds.View as Control;
Form f = new Form();
c.Parent = f;
c.Dock = DockStyle.Fill;
f.Show();

In this code snippet, I have loaded the DesignSurface with a Form. Similarly, you can load the DesignSurface with any component that has a root designer available for it. For example, you could load a UserControl or a Component instead.

The sample provided in the code download for this article hosts four different root components: Form, UserControl, Component, and MyTopLevelComponent (a graph designer). When you run the sample, a shell UI will open. This interface includes a toolbox, a properties browser, a tab control for hosting designers, an output window, and a Solution Explorer, as shown in Figure 6. Selecting File | New | Form from the menu opens a new designer host with Windows Forms Designer. This essentially uses the code I've just shown you to load the designer. Rather than loading a Form, the sample app illustrates how to load a UserControl or Component.

Figure 6 Hosting Windows Forms Designer
Figure 6 Hosting Windows Forms Designer

To create a root component, create a designer that implements IRootDesigner and then associate this designer with your component. The View property of the root component specifies the view that will be presented to the user.

One of the main services provided by DesignSurface is IDesignerHost. It is the master interface used to provide designers and controls access to types, services, and transactions. It can also be used to create and destroy components. To add a button to the Windows Forms designer I created earlier, all I need to do is get the IDesignerHost service from the DesignSurface and use it to create the button as shown in Figure 7.

Figure 7 Create a Button with IDesignerHost (Close Figure 7)
// Add a Button to the Form
IDesignerHost idh = (IDesignerHost)ds.GetService(typeof(IDesignerHost));
Button b = (Button)idh.CreateComponent(typeof(Button));

// Set the Parent of this Button to the RootComponent (the Form)
b.Parent = (Form)idh.RootComponent;

// Use ComponentChangeService to announce changing of the
// Form's Controls collection */
IComponentChangeService icc = (IComponentChangeService)
    idh.GetService(typeof(IComponentChangeService));
    icc.OnComponentChanging(idh.RootComponent,
    TypeDescriptor.GetProperties(idh.RootComponent)["Controls");

IToolboxUser specifies that the designer supports adding controls from the toolbox. This means that if you do have a toolbox that implements ToolboxService, you can use the IToolboxUser interface to add controls to the root component. For instance:

/* Add a Button to the Form using IToolboxUser */
IDesignerHost idh = (IDesignerHost)ds.GetService(typeof(IDesignerHost));
IToolboxUser itu = (IToolboxUser)idh.GetDesigner(idh.RootComponent);
itu.ToolPicked(new ToolboxItem(typeof(Button)));
When controls are added to the custom RootDesigner in the sample application by double-clicking the items in the toolbox, the view of the RootDesigner updates to display a pie chart as shown in Figure 8. Clicking on the GraphStyle link changes the view to a bar graph.

Figure 8 Custom RootDesigner Updates
Figure 8 Custom RootDesigner Updates

The Toolbox

MyRootDesigner implements the IToolboxUser interface. This has two methods: GetToolSupported and ToolPicked. You can use GetToolSupported to filter the items that can be added onto the designer. ToolPicked eventually calls into the CreateComponents method (which, as the name implies, is responsible for creating the components) of ToolboxItem.

Now that I have added the controls and components to the designer, let's take a closer look at how to implement a toolbox. First, your toolbox needs to implement IToolboxService—this service is added to the service container and can be accessed by anyone who needs to use it. The main functions of IToolboxService are shown in Figure 9.

Figure 9 IToolboxService Members (Close Figure 9)

Method or Property | Description
GetSelectedToolboxItem | Returns the selected toolbox item
SerializeToolboxItem | Creates a binary serialized object (DataObject) from ToolboxItem
DeserializeToolboxItem | Returns the ToolboxItem from a binary serialized object (DataObject)

To allow the items from the toolbox to be added onto the designer using the mouse or keyboard, the toolbox in the sample hooks onto the KeyDown and MouseDown events. For the Enter key or a mouse double-click, IToolboxUser.ToolPicked is called. The sample shows how to serialize the ToolboxItem into DataObject and DoDragDrop for when a mouse single-click event occurs. IToolboxService.SerializeToolboxItem will be called on mouse up and the item will be added to the designer.

When a new control or component is added to the designer, you can provide a custom name for the control by implementing INameCreationService. The sample application shows an example of this service in action using CreateName, ValidateName, and IsValidName.

Multiple DesignSurfaces

When managing multiple DesignSurfaces, it is a good idea to use a DesignSurfaceManager. This makes it easier to manage these DesignSurfaces. (Note that the services of DesignSurfaceManager are also available to DesignSurface.)

Calling DesignSurfaceManager.CreateDesignSurface calls into CreateDesignSurfaceCore. You can override this function to create a custom DesignSurface and add the services. The sample app creates a custom HostSurface by overriding this function in the HostSurfaceManager class:

protected override DesignSurface CreateDesignSurfaceCore(
    IServiceProvider parentProvider)
{
    return new HostSurface(parentProvider);
}
You can then hook into the ActiveDesignSurfaceChanged event and update the output window in the HostSurfaceManager class, as shown here:
void HostSurfaceManager_ActiveDesignSurfaceChanged(
    object sender, ActiveDesignSurfaceChangedEventArgs e)
{
    ToolWindows.OutputWindow o =
        this.GetService(typeof(ToolWindows.OutputWindow)) as
        ToolWindows.OutputWindow;
    o.RichTextBox.Text += "New host added.\n";
}

DesignerLoaders

So far I've created DesignSurfaces, hosted designers, added controls, implemented a toolbox, and added and accessed services like OutputWindow. The next step is to persist the designer. The designer loader is, as you'd expect, responsible for loading the designer from some persistent state. Simple and flexible, designer loaders have very few requirements. In fact, you can create an instance of the Windows Forms designer with a one-line designer loader that simply creates an instance of System.Windows.Forms.Form.

In addition to loading a form design, a designer loader is also responsible for saving the design. Because saving is an optional behavior, a designer loader listens to change events from the designer host and automatically saves state according to these events.

The .NET Framework 2.0 introduces two new classes for writing custom loaders: BasicDesignerLoader and CodeDomDesignerLoader. The sample app illustrates implementations of both loader types. Earlier I demonstrated loading the DesignSurface with a root component by passing in the type of the component. However, if you are using a loader, it should be used to load the design surface. The BeginLoad code snippet you'll need when using loaders should look something like this:

// Load it using a Loader
ds.BeginLoad(new MyLoader());

The DesignerLoader is responsible for loading the root component in the DesignSurface and creating any components. When creating a new form or any other root component, the loader simply loads it. In comparison, when loading from a code file or some other store, the loader is responsible for parsing the file or store and recreating the root component along with any other necessary components.

The .NET Framework defines an abstract base class called DesignerLoader that is used to load and save designers from persistent storage. The base class is abstract so that any type of persistence model can be used. This, however, adds to the complexity of implementing the class.

BasicDesignerLoader provides a complete and common implementation of a designer loader minus any information related to the persistence format. Like DesignerLoader, it is abstract, not dictating anything about the persistence format. BasicDesignerLoader does, however, handle the standard work of knowing when to save, knowing how to reload, and tracking change notifications from designers. Its features include support for multiple load dependencies, tracking of the modified bit to indicate a need to save changes, and deferred idle time reload support.

The services shown in Figure 10 are added to the designer host's service container by BasicDesignerLoader. As with other services, you can change replaceable services by editing their value in the protected LoaderHost property.The sample app implements a BasicDesignerLoader that persists the state in an XML format. To see how it works, select File | Type | BasicDesignerLoader. Then create a new Form by selecting File | New | Form. To see the XML code that is generated, select View | Code | XML. The XML code that is generated by the app will look something like this:

<Object type="System.Windows.Forms.Form, System.Windows.Forms,
    Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
    name="Form1" children="Controls">
  <Property name="Name">Form1</Property>
  <Property name="DataBindings">
    <Property name="DefaultDataSourceUpdateMode">OnValidation</Property>
  </Property>
  <Property name="ClientSize">292, 273</Property>
</Object>
PerformFlush and PerformLoad are the two abstract functions of the BasicDesignerLoader that you need to implement for serializing and deserializing, respectively.

Figure 10 BasicDesignerLoader Services (Close Figure 10)

Service | Replaceable | Description
IDesignerLoaderService | Yes | Allows objects to request that the designer reload itself at idle.
IDesignerSerializationManager | No | Used to serialize and deserialize objects. The serialization manager is added as a service so anyone who needs to perform serialization can utilize any serialization providers that were added to the serialization manager.
CodeDomDesignerLoader

Design-time serialization is handled by generating source code. One of the challenges of any code-generation scheme is the handling of multiple languages. The .NET Framework is designed to work with a variety of languages, so I also want the designers to emit to several languages. There are two ways to address this. The first is to require each language vendor to write the code-generation engine for their language. Unfortunately, no language vendor can anticipate the wide variety of code-generation requirements that third-party component vendors may need. The second approach is to require each component vendor to provide code generation for each language they want to support. This is equally bad, because the number of languages supported is not fixed.

To solve this problem, the .NET Framework defines an object model called the Code Document Object Model (CodeDOM). All source code can essentially be broken down into primitive elements and the CodeDOM is an object model for those elements. When code adheres to CodeDOM, the generated object model can later be sent to a code generator for a particular language to render the appropriate code.

The .NET Framework 2.0 introduces CodeDomDesignerLoader, which inherits from BasicDesignerLoader. The CodeDomDesignerLoader is a full loader that works by reading and writing CodeDOM. It is a turnkey designer loader, so all you need to do is provide the CodeDOM.

In the sample app you can select File | Type | CodeDomDesigner-Loader to see an example of the CodeDOM in action. Create a new form by selecting File | New | Form—this creates a DesignSurface and loads it using a CodeDomDesignerLoader. To check the code, select View | Code | C# to see the C# version of the form's code, or View | Code | VB to see the Visual Basic® version.

To generate the code, the sample app uses CSharpCodeProvider and VBCodeProvider. It also uses the code providers to compile the code and run the executable (see Figure 11).

Figure 11 Generate, Compile, and Run (Close Figure 11)
CompilerParameters cp = new CompilerParameters();
AssemblyName[] assemblyNames =
    Assembly.GetExecutingAssembly().GetReferencedAssemblies();

foreach (AssemblyName an in assemblyNames)
{
    Assembly assembly = Assembly.Load(an);
    cp.ReferencedAssemblies.Add(assembly.Location);
}

cp.GenerateExecutable = true;
cp.OutputAssembly = executable;

cp.MainClass = "DesignerHostSample." +
    this.LoaderHost.RootComponent.Site.Name;

// Compile CodeCompileUnit using CodeProvider
CSharpCodeProvider cc = new CSharpCodeProvider();
CompilerResults cr = cc.CompileAssemblyFromDom(cp, codeCompileUnit);

if (cr.Errors.HasErrors)
{
    string errors = string.Empty;
    foreach (CompilerError error in cr.Errors)
    {
        errors += error.ErrorText + "\n";
    }
    MessageBox.Show(errors, "Errors during compile.");
}

ITypeResolutionService, which is required when using the CodeDomDesignerLoader, is responsible for resolving a type. One scenario, for example, where this service is called to resolve types is when adding a control from the toolbox into the designer. The sample application resolves all types from the System.Windows.Forms assembly so that you can add controls from the Windows Forms tab of the toolbox.

Conclusion

As you've seen, the .NET Framework provides a powerful and flexible designer hosting infrastructure. Designers provide straightforward extensibility that help to address very specific needs or more advanced scenarios than were supported by previous versions of Visual Studio. Of course, designers can be hosted easily outside Visual Studio, as well. Be sure to download the sample application, so you can play around with the code and start implementing your own custom designers today.

Dinesh Chandnani is a Software Design Engineer in Test for the .NET Client Team at Microsoft, working on designer hosting and other designer features. He started working for Microsoft in 2002 after completing his masters' degree in Computer Science at the University of Arizona.


Editor's Note: When Did Clients Get So Smart?

Editor's Note
When Did Clients Get So Smart?

This issue marks the sixth anniversary of our merger, when Microsoft Systems Journal and Microsoft Internet Developer combined to become MSDN Magazine. We've been around to witness the birth and growth of the Microsoft .NET Framework and Web services, and the evolution of the smart client model of software design. The changes in application models have been surprisingly brisk over the past six years, as the combination of richer programming tools and more reliable connectivity have enabled leaps in what is possible.

For years, we programmed along happily, creating our standalone apps that ran on Windows. They did a lot of stuff. They ran the workflow for the accounting department (which consisted of one chain-smoking guy who wore the same Santa Claus tie every day of the year). They collected data (the number of widgets the accounting department lost track of this week). They even made it possible for the accounting department to run a mail merge for its holiday card shipments.

Then someone had a bright idea. Why not take all those applications and put them online? We could give it a snappy name like the "Web," and you would only need a single piece of software ever again—let's call it the "browser." That sounds good. From then on, all of your important software would run off a server somewhere in North Dakota, and boy wouldn't that be easy!

But here's the problem. You know those sites that make you watch a two-minute Flash intro of dancing utensils before a "Click to Skip" button slowly fades in just as the last spoons are rolling off the screen? The graphics are completely orthogonal to the actual functionality of the site's application. More importantly, the Click to Skip button is in a different place every time. And that's the big downfall of the thin client in our eyes: every one of them tried to reinvent the user experience, ignoring much of what we know about human behavior.

Another element of human behavior is the simple inability to trust that things will always work properly. Ironically, you're more likely to lose work stored locally than data stored remotely in a redundant server, but the feeling of holding a copy of what you've done is elemental.

But "regular" applications had their own issues. Many of you will recall some program you got a decade ago that showed up on a stack of 3.5" floppies. You'd insert disk 1 to start the installation, then disk 2, then disk 5...then back to disk 1. Installation was a nightmare. The program would take it upon itself to overwrite your DLLs with its own older versions, because they were the ones it had been tested against. On the other hand, the program didn't need a high-speed connection to work properly.

So in between traditional apps and thin clients, smart clients started to grow. A smart client sits on the client machine, but at the same time it makes use of the connectivity that most of us have to exchange data and services from elsewhere. Outlook is a good example of a smart client. You connect to your server, work on your mail, and when you disconnect, the experience is seamless. You can keep working, using your data as before.

Just before MSDN Magazine was created, MIND was covering SOAP, a new XML protocol that helped drive the very idea of the .NET Framework. Our coverage history over the past six years has been rich with .NET, but in a larger sense it's been all about smart clients. SQL Server, BizTalk, mobile platforms, and Windows Forms have all played a part in the move to smart clients. Think back to just six years ago. An application that could download its own patches, or that could work seamlessly online or offline, was a stretch. In 2006, this is expected behavior for any serious program.

If the past half decade (plus a year) has provided the changeover from applications to smart clients, what will the next six years bring? It could be that in 2012, smart clients will be known by a different name: applications.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Mark Boulter, Shawn Burke, Rob Caron, Eric Carter, Joe Duffy, Jessica Fosler, Ken Henderson, Eric Jarvi, Ashok Kamath, Jason Lucas, Doug Neumann, Lukasz Pawlowski, Matt Powell, Lutz Roeder, Jason Shirk, Dharma Shukla, Brian Welcker, and David R. Williamson.

Active Directory, Excel, InfoPath, IntelliSense, Microsoft, MSDN, Outlook, PowerPoint, SharePoint, Visual Basic, Visual C++, Visual C#, Visual SourceSafe, Visual Studio, Windows, Windows NT, Win32, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Automated Build Process, Real-World E-Mail Tasks, and More

Toolbox
Automated Build Process, Real-World E-Mail Tasks, and More
Scott Mitchell

Contents
Manage the Project Build Process
Sending E-Mail
The Bookshelf

Manage the Project Build Process

While the Visual Studio® Build menu's Build Solution option is the usual way developers compile their current work on a project, the process of building the complete solution for testing, deployment, or production often requires many more steps. The build process varies from company to company, but typically involves checking out the latest version of the source code, running automated tests such as unit tests, performing the actual compilation, and completing any peripheral tasks such as creating Web sites, applications, or virtual directories in IIS, building the database schema and inserting any needed initial or test data, and so on.

The build process is an ideal candidate for automation. Even for small projects, having an automated build process can greatly improve efficiency by reducing the time needed to process the build steps manually and handle any mistakes that may result from performing a build by hand.

There are several open-source and commercial build manager products that can help with automation. NAnt is a popular open-source build manager. With the .NET Framework 2.0, Microsoft has released their own build manager called MSBuild. Both NAnt and MSBuild are designed to be build manager engines, processing build commands from an XML file. Neither engine has an integrated GUI environment through which build scripts can be managed or through which a build's progress can be examined.

A build manager with a very slick, built-in graphical front end is Visual Build Pro by Kinook Software. Visual Build Pro allows the build master to graphically create the build script, selecting a task from the left pane and adding it to the collection of tasks in the right pane. When starting the build process, the interface shows the current step being executed and the status of past steps. Visual Build Pro is not just eye candy. The easy-to-use interface sits atop a powerful and highly customizable build manager engine that can also be accessed through the command line.

Visual Build Pro models common steps in the build process as actions. There are actions to work with Visual SourceSafe®, Current Version System (CVS), Secure File Vault, Subversion, and other popular source control providers. There are also actions to integrate with automated testing frameworks. The actual compilation process of the checked out, tested source code can be performed with actions designed to compile using Visual Studio 6.0, Visual Studio .NET, MSBuild, NAnt, or other tools.

The build process can be a mundane and lengthy task and is best to be automated. Visual Build Pro's interface makes creating and managing builds as easy as dragging and dropping. A wide array of built-in actions make performing common build tasks a snap, and Visual Build Pro's scripting capabilities allow for customized actions. If you're looking for a tool for managing your builds, consider Visual Build Pro.

Price: $295 for a single license.

www.visualbuild.com

Sending E-Mail

For both desktop and Web-based applications, e-mail is commonly used for communication. Domain-specific apps as well as programs in beta testing often include a "Send Feedback" feature that provides the user with a way to directly communicate with the software vendor.

Virtually all Web sites utilize e-mail in some form or another, whether through a Web-based feedback form, a way for one user to contact another while keeping their e-mail addresses private, or an automated way to alert a user that some event has transpired, such as a particular stock price having exceeded a user-defined price per share.

aspNetEmail, by Advanced Intellect LLC, provides the same core functionality as the mail-related classes in the .NET Framework 2.0, but includes a bevy of additional features that greatly ease the process of programmatically sending e-mail. For example, with the .NET Framework classes, you can only set the body of the e-mail message to a string; with aspNetEmail, the body can be a string, the path to a local file, a remote URL, or the HTML output of an ASP.NET server control. aspNetEmail also includes complete vCard and iCalendar support and exposes an event model that allows you to tap into various stages in the mail sending process.

I've used aspNetEmail in two past projects and my favorite feature is its blindingly fast and easy mail-merge function. Rather than send a single message to a list of recipients, mail merges let each message be individu-alized for the recipient. Simply add placeholders in the message body, then call the SendMailMergeList method, passing in a DataSet, DataReader, CSV file, or collection, and the method will automatically enumerate the datasource, fill in the placeholders accordingly, and blast off a personalized message for each record.

Simply sending an e-mail message from a .NET-based application is a walk in the park with the Framework's mail-related classes. To accomplish more interesting, real-world e-mail tasks, however, you'll want to turn to a third-party e-mail component. With its affordable price, ample documentation, and top-notch support, aspNetEmail is a good addition to any developer's toolbox.

Price: $129 for a single server license.

www.aspnetemail.com

The Bookshelf

If you've worked on Microsoft®-based component-oriented systems prior to the .NET Framework, then you're acutely familiar with COM's limitations: component registration that led straight to DLL Hell, the lack of side-by-side execution for different component versions, and the loose coupling between a component and its security semantics, to name a few. While many of these issues were addressed with DCOM, the added complexity limited adoption. Fortunately, many of the ills of COM and DCOM have been righted with .NET. In the .NET Framework, the key principles of component-oriented development—location transparency, concurrency management, versioning, component security, and so on—are baked into the Framework and can be easily utilized by both developers and administrators in ways not possible with previous technologies.

In Programming .NET Components, 2nd Edition (O'Reilly, 2005), Juval Lowy illustrates component-oriented features in .NET. This book is geared to those who are well-versed in component-oriented development, and it includes many sidebars comparing various features in .NET with COM implementations. However, even developers who are not domain experts will find many gems that can prove useful in arenas other than component development.

Lowy begins with a wonderful explanation of the differences between object-oriented development and component-oriented development, along with an examination of the hallmarks of component-oriented development. In the remainder of the book, he spends a chapter on each of the key principles of component-oriented development. Chapter 3 delves into interface-based programming, with some great tips on how to best factor interfaces. Chapter 5 examines versioning, a topic important to any developer who ships a .NET-based library or app. Additional chapters look at events, asynchronous calls, multithreading, and remoting. The book concludes with a lengthy chapter on security.

There are discussions on features specific to the .NET Framework 2.0 and Visual Studio 2005, features that are pertinent to component-oriented development. For example, Lowy examines how generics affect interface design, and explores Visual Studio 2005 capabilities for refactoring. While the book does explore 2.0-specific features, those still using .NET 1.x will find plenty of useful information.

If you are a component developer moving into .NET, work in a component-oriented design shop, or are just interested in learning more about component design in .NET, this is a great place to start.

Price: $44.95.

www.oreilly.com

Computer trade books typically fall into one of two camps: they either focus on a very specific topic or attempt to give a broad overview of a more general topic. The sheer size and weight of Pro ASP.NET 2.0 (Apress, 2005), by Matthew MacDonald and Mario Szpuszta, clearly indicates that it's in the latter camp. Spanning more than 1,200 pages, Pro ASP.NET 2.0 aims to be the one-stop spot for mastering ASP.NET 2.0, covering the wide swath of features and technologies that encompass the dynamic Web development framework. (There are two versions of this book: one with source code in C# and one in Visual Basic®—which was written by Matthew MacDonald and Lawrence Moroney.)

To the authors' credit, Pro ASP.NET 2.0 does a good job of balancing the task of covering a wide array of topics and covering them in depth. The central ASP.NET ideas are explored in detail, with sections on the core concepts, data access, and security each examined over roughly 200 pages. The more peripheral subjects—building custom server controls, advanced Web services, dynamic graphics, and client-side callbacks, among others—are covered in just an overview. This approach strikes the perfect balance between the breadth and the depth of the material.

Pro ASP.NET 2.0 focuses on real-world issues and concerns. Rather than just talking about the power of the provider model, the authors provide working examples of plausible custom providers for membership, profiles, and site navigation. Furthermore, the book covers the trade-offs that accompany various design decisions. For example, in the profiles chapter there's a discussion on when to store data using the default profile provider versus using a custom provider, or simply moving the data to a separate, specific table.

This book is written for intermediate to advanced developers, and assumes that its readers have Web development experience. If you are interested in learning the ins and outs of ASP.NET version 2.0, consider picking up a copy of Pro ASP.NET 2.0.

Price: $59.99.

www.apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.net.


Advanced Basics: Set Word Document Properties Programmatically

Advanced Basics
Set Word Document Properties Programmatically
Ken Getz

Code download available at: AdvancedBasics0603.exe (131KB)
Contents
Creating the Settings
Creating the Form
Using the BackgroundWorker Component
Automating Microsoft Word

At the beginning of another lovely day of writing courseware in mad pursuit of unrealistic deadlines, I received a frantic call from a business partner. He was at the end of a long consulting project and had several hundred Microsoft® Word documents, all of which required their document properties to be set identically, except the Title property of the document, which was to be based on the document file name, minus the .doc extension. He didn't have time to set all the properties manually. Could I write a little utility to do the work for him?

Of course, I conjectured that he could probably set the properties manually in the time that it would take me to write a utility. But part of me was rejoicing: article fodder! So here we are.

In case you didn't know, Microsoft Word documents (and a large group of Windows® documents, including each of the Microsoft Office document types) maintain a set of properties that can be viewed in a number of ways. Within Windows Explorer, you can right-click a document, and select Properties from the context menu. Figure 1 shows the Windows dialog box.

Figure 1 View Document Properties
Figure 1 View Document Properties

From within Word, you can select File | Properties to view the same information. You could, of course, simply open each document in turn and manually change the settings. That would be rather tedious, though. A better solution would be to iterate through a selected folder, and possibly the subfolders within the folder, and update the properties programmatically for each file you find. And this is, of course, the solution I chose and the one I'll describe here. (For more information about the built-in document properties, browse to BuiltInDocumentProperties Property).

Word (along with Microsoft Excel® and Microsoft PowerPoint®) provides a repository for common document information, including Title, Subject, Author, and more. (Actually, it's the Office core functionality that makes all this possible, but each application exposes the capability through its own object model.) In addition, each Office application allows you to add custom properties, although this wasn't on my list of agenda items, so I didn't deal with it in my application. You could add support for modifying custom document properties if you want. Think of it as a homework assignment.

Figure 2 Users Can Iterate Through Folders
Figure 2 Users Can Iterate Through Folders

In satisfying the needs of my business partner, I used Visual Studio® 2005 to create the application shown in Figure 2. This application satisfies a number of project requirements:

  • The user must be able to select a folder in which to search, with the option of searching in subfolders of the selected folder.
  • The user must be able to specify text for any of the visible document properties or to leave the property blank to clear any existing value.
  • The application must allow the user to leave selected properties unchanged.
  • The application must allow the user to cancel the operation, preventing further documents from being modified but not reverting changes already made.
  • The application must allow the user to supply a fixed title for each document. The user should also be able to allow the application to insert the document's file name as the title.

As with any weekend project, there's a lot of room for improvement. You may find it instructive to add the features I purposefully left out. For instance, error handling is somewhat vague. You might want to provide better reporting on error conditions. Reporting on modified files would be helpful as well. It might be nice to add support for displaying a list of files that were successfully updated.

In its current state, the application supports only Word. It would be useful, and not terribly difficult, to modify it so that it supports other Office applications as well. Plus, it supports only a subset of the built-in document properties. You might decide to add support for more built-in or user-defined properties.

Although the sample application isn't terribly complicated, it does demonstrate several interesting technologies, including COM interoperability to automate Word, My.Settings to store user selections, object data binding, and the BackgroundWorker component. In the following sections I'll describe some issues involved in creating the app, and describe the interesting code. By the way, the sample is available in this month's download.

Creating the Settings

Because the application might be run several times, for different groups of documents, it seemed useful to me that the application maintain choices across sessions. It was a perfect occasion to take advantage of application settings! I created a setting corresponding to each selection within the form—each textbox, checkbox, and option has a corresponding setting. To create the settings, I right-clicked on the project within the Solution Explorer window, selected Properties from the context menu, and selected the Settings tab. Figure 3 shows the settings I created for the application.

Figure 3 Settings for Every User Option
Figure 3 Settings for Every User Option

Now I wanted to be able to bind controls on the form directly to these settings so I wouldn't have to write the code to move the values into and out of controls on the form myself. Visual Studio 2005 makes this easy—I simply added a new data source to the project, using the Data | Add New Data Source menu. Choose to add an Object data source and you can select as a data source the My.MySettings class (which Visual Studio creates for you once you add all your settings). The settings then will show up in the Data Sources window (see Figure 4). For another discussion of using and binding to settings, see my December 2005 column (Advanced Basics: What's My IP Address?).

Figure 4 MySettings
Figure 4 MySettings

Creating the Form

Once you've created the data source corresponding to the MySettings class, you can drag any setting from the Data Sources window onto a form. Doing this creates a BindingSource object named MySettingsBindingSource on the form, which provides the connection between the settings and the form. I bound each checkbox and textbox control on the form to an individual setting in this manner.

The form includes a FolderBrowserDialog control (fbd, in the code), and clicking the Select Input Folder button runs the following code:

' Select the folder containing the document files to be modified.
fbd.ShowNewFolderButton = False
fbd.SelectedPath = folderLabel.Text
If fbd.ShowDialog() = Windows.Forms.DialogResult.OK Then
    folderLabel.Text = fbd.SelectedPath
    My.Settings.Folder = fbd.SelectedPath
End If

Note that direct binding to the folder label didn't work because I modified the value in code. The code sets the My.Settings.Folder value directly, once the user selects a path.

The settings for the form also include Size and Location values, so that the user can move the form around and the application persists the information. The Form.FormClosing event handler runs the following code, which saves the current information:

My.Settings.Size = Me.Size
My.Settings.Location = Me.Location
My.Settings.Save()
The Form.Load event handler runs the following code, which initializes the data binding, and restores the locations:
MySettingsBindingSource.DataSource = My.Settings

' Binding the form's properties to these settings
' doesn't work as you might expect.
If Not My.Settings.Location.IsEmpty Then
    Me.Location = My.Settings.Location
End If
If Not My.Settings.Size.IsEmpty Then
    Me.Size = My.Settings.Size
End If

In order to allow the user to be able to opt out of overwriting property values, the form includes checkboxes along the left edge. The code overwrites any value for which the checkbox is selected. For those that aren't selected, the form simply skips over the property. The Form.Load event handler hooks up the binding between the checkbox's Checked property and the corresponding textbox's Visible property for each checkbox on the form—if the user unchecks a checkbox, the corresponding textbox disappears:

   ' Set up control property bindings:
   titleTextBox.DataBindings.Add("Visible", _
         useTitleCheckbox, "Checked")
   subjectTextBox.DataBindings.Add( _
         "Visible", useSubjectCheckBox, "Checked")
   ' and so on, for each pair of controls...

(The form could also have handled the CheckedChanged event for each checkbox, but I like making this type of interaction declarative, using the DataBindings collection of each control in the Load event handler.)

Figure 5 Snap Lines Help You Align Controls
Figure 5 Snap Lines Help You Align Controls

The single form within the application, shown in Figure 5, includes a number of labels, textboxes, and checkboxes that needed to be carefully aligned. The Windows Forms designer in Visual Studio 2005 provides snap lines, which make it easier to create good-looking forms—even ones this simple. Figure 5 shows the snap lines in use, aligning a Label control with an edge and a text baseline.

Using the BackgroundWorker Component

I've written about the BackgroundWorker component, in a very similar application, in my March 2005 column (Advanced Basics: Doing Async the Easy Way), so I won't repeat that discussion. The sample form includes event handlers for the component's DoWork, ReportProgress, and RunWorkerCompleted events. The DoWork event handler begins the search process and manages the document modifications. The ProgressChanged event handler simply updates a ToolstripStatusLabel control within the form's status bar, like this:

statusLabel.Text = "Updating " & e.UserState.ToString

The RunWorkerCompleted event handler updates the Enabled properties of the two buttons at the bottom of the form and attempts to update the status bar with information about the success of the search, as shown in the following:

   Dim statusText As String = "Ready"
   If e.Cancelled Then
     statusText &= _
       ". Previous operation cancelled."
   End if
     statusLabel.Text = statusText
   makeChangesButton.Enabled = True
   cancelButton.Enabled = False

The form also includes code to ensure that you don't click the Make Changes button again until the code running from the previous click has completed. The BackgroundWorker component cannot run more than one task at a time, so the form disables the Make Changes button as soon as you click it (enabling the Cancel button concurrently). When the BackgroundWorker completes its work, whether you canceled or not, the code reverses the state of the buttons.

Automating Microsoft Word

Finally, I'll address the heart of the matter—making Word do stuff. The application has a COM reference set to the Word 11.0 Object Library. When I installed Office 2003, I elected to install the entire package, including the Office 2003 Primary Interop Assemblies (PIAs). These assemblies provide a fixed set of interop assemblies for you to program against, using the Microsoft .NET Framework. If you don't have these installed, Visual Studio generates the interop assembly for you when you set the reference, but this isn't the optimal solution. Instead, if you're going to program Word, make sure you do install the appropriate PIAs from the Office 2003 installation media.

The form's code includes class-wide references for the Word Application object and reference to a Word Document object:

Private app As Word.Application
Private doc As Word.Document
To make it easier to program against the Word object model, I generally add Imports statements to the top of each code file, as shown in the following:
Imports Office = Microsoft.Office.Core
Imports Word = Microsoft.Office.Interop.Word
Without those Import statements, the previous code snippet would look like this:
Private app As Microsoft.Office.Interop.Word.Application
Private doc As Microsoft.Office.Interop.Word.Document

After you've set all the options and have clicked Make Changes, the code iterates (recursively, if you've requested subfolders to be searched) through the selected folder or folders and, as it finds each file, calls the OnFileFound procedure in the code. This procedure, after reporting the progress to the BackgroundWorker, opens the document:

' Open the document.
doc = app.Documents.Open(fi.FullName)
The code uses the BuiltInDocumentProperties property of the document to set each property. In each case, the code verifies whether you want to overwrite the property and, if so, places the value from the TextBox control into the property, using the procedure shown in the following code:
Private Sub SetProperty( _
        ByVal chk As CheckBox, _
        ByVal prop As Word.WdBuiltInProperty, _
        ByVal txt As TextBox)

    ' If the supplied CheckBox control is checked,
    ' then set the Word property to the supplied value.
    If chk.Checked Then
        doc.BuiltInDocumentProperties(prop).Value = txt.Text
    End If
End Sub

The OnFileFound procedure calls the SetProperty procedure for each property, like this:

SetProperty(useSubjectCheckBox, wdPropertySubject, subjectTextBox)
SetProperty(useAuthorCheckBox, wdPropertyAuthor, authorTextBox)
' and so on...
The code handles the Title property in a special manner, because you may have opted to replace the Title property with the document's file name, or you may have supplied a fixed value:
If useTitleCheckbox.Checked Then
    If useDocNameCheckBox.Checked Then
        doc.BuiltInDocumentProperties(wdPropertyTitle).Value = _
           Path.GetFileNameWithoutExtension(doc.Name)
    Else
        doc.BuiltInDocumentProperties(wdPropertyTitle).Value = _
           titleTextBox.Text
    End If
End If

I tested this code on several hundred documents of my own before sending it off to my partner, for whom it immediately failed. It took a few minutes to diagnose, but he had a document in which someone had set the ReadOnlyRecommended property to True. (If this property is set, Word prompts you to open the document as read-only, but allows you to override the suggestion.) Unfortunately, by the time you open the document programmatically, it's too late to change the setting, and the dialog box halts the application.

A search on the Web turned up a tacky, if workable solution: you must save the document without the ReadOnlyRecommended property set—you cannot change the setting for the document you're attempting to work with. Therefore, I added sub-optimal code to the OnFileFound procedure, and it's called immediately after the code opens the document (see Figure 6).

Figure 6 ReadOnlyRecommended (Close Figure 6)
If doc.ReadOnlyRecommended Then
    Dim docName As String = fi.FullName & "x"

    ' Save the doc with a new name,
    ' with ReadOnlyRecommended set to False.
    doc.SaveAs(docName, , , , , , False)
    doc.Close()

    ' Open the new document.
    doc = app.Documents.Open(docName)
    doc.SaveAs(fi.FullName)
    doc.Close()
    File.Delete(docName)

    doc = app.Documents.Open(fi.FullName)
End If

The code in Figure 6 does exactly what the suggestion I found indicated: it checks the property, and if it's set, saves the document using a new name and having the property set to False. It then deletes the original document and renames the new document to the original document's name. Not pretty, but it solves the problem. If you have a better solution, please let me know.

After using this utility for a while (I've found it quite useful in my own work), I've added a few more settings that help clean up documents before I submit them—I delete all comments and accept all revisions in the OnFileFound procedure, as well:

' Accept revisions:
doc.Revisions.AcceptAll()

' Delete all comments:
For Each cmt As Word.Comment In doc.Comments
    cmt.Delete()
Next
All that's left for the OnFileFound procedure to do is to close the document and save changes:
doc.Close(True)

If you investigate the BackgroundWorker's DoWork event handler, you'll see that this block of code handles the entire operation:

' Open Word, and handle the documents in the specified folder.
Try
    app = New Word.Application
    HandleDocs(folderLabel.Text, e)
Catch ex As Exception
    MessageBox.Show(ex.Message)
Finally
    app.Quit()
    System.Runtime.InteropServices.Marshal.ReleaseComObject(app)
End Try
This code creates the instance of Word and calls the HandleDocs procedure to iterate through folders and files, as requested. Once the work has completed, the DoWork event handler quits Word and releases its reference using the ReleaseComObject method. (You may not agree with this design—I could have left Word running until you quit the application, but I hate to leave applications running in the background unless they're actually doing something.)

The HandleDocs procedure does the grunt work: iterating through the file system, as you see in Figure 7. The only interesting things about HandleDocs are that it checks for cancellations from the user, and it calls itself recursively if you ask it to search subfolders.

Figure 7 HandleDocs Procedure (Close Figure 7)
Private Sub HandleDocs(ByVal path As String, _
        e as System.ComponentModel.DoWorkEventArgs)

    Try
        If bgw.CancellationPending Then
            e.Cancel = True
            Exit Sub
        End If

        ' Handle subfolders, if required.
        Dim diLocal As New DirectoryInfo(path)
        If LookInSubfoldersCheckBox.Checked Then
            For Each di As DirectoryInfo In diLocal.GetDirectories
                HandleDocs(di.FullName, e)
            Next
        End If

        ' Search for matching file names.
        For Each fi As FileInfo In diLocal.GetFiles("*.doc")
            ' Cancellation pending? Cancel and get out.
            If bgw.CancellationPending Then
                e.Cancel = True
                Exit Sub
            End If
            OnFileFound(fi)
        Next

    Catch ex As UnauthorizedAccessException
        ' Don't do anything at all, just quietly get out. This means you
        ' weren't meant to be in this folder. All other exceptions will
        ' bubble back out to the caller.
    End Try
End Sub

As with any project, I had other options. I considered two: creating a COM add-in for Word, and using Visual Studio 2005 Tools for the Office 2003 System, but neither seemed appropriate. COM add-ins are useful when you want to run code while you're using the Office application, but aren't helpful outside the running application. Visual Studio Tools for Microsoft Office is great when you want to create solutions using Excel, Word, Outlook®, or InfoPath® that use managed code, but in each case, the solutions are document-centric. After a few moments thought, it became clear that a simple Windows-based application using COM Automation was the best approach.

I'm sure you may be wondering why I didn't use the ill-fated COM component that allowed you to modify document properties without actually loading the document into its host application. That component appears to have been removed from MSDN® (where I originally found it) and to be honest, it never worked completely correctly. Some properties could be changed using the component, but others wouldn't change or would cause your application to fail in violent sorts of ways (sometimes requiring Windows to restart). Enough said about that option.

I've been spending much time recently writing examples using the new XML-based file format for Office 12 documents. This new file format will make tasks such as the one undertaken here much more efficient. Using the new format, there's no reason to load Word to perform the work, and there's not even the need to load the entire document. Because of the partitioning of the data using this new file format, you can load just the properties part of the document, make your changes, and save that tiny piece. Watch this column—when the public beta arrives, I'll revisit this application and kick the tires of the new file format.

Send your questions and comments to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies and a courseware author for AppDev. He is coauthor of ASP.NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2002), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com. Ken would like to thank Brian Randell for providing the inspiration for the utility described in this column.


CLR Inside Out: Ensuring .NET Framework 2.0 Compatibility

CLR Inside Out
Ensuring .NET Framework 2.0 Compatibility
Jesse Kaplan

Contents
What is Compatibility?
Side-by-Side Execution and Backwards Compatibility
Testing
Wrap-Up

If we learned only one thing about compatibility in the past few years, it is that compatibility is much more than avoiding breaking changes. On the Microsoft®.NET Framework and Visual Studio® teams, we do our part to ensure that the products we build are stable platforms that developers can truly rely on. At the same time, developers working on these platforms need to do their part to ensure that their applications can withstand just a little shaking. I'll take you on a behind-the-scenes tour of our compatibility efforts and discuss what we learned in the process.

What is Compatibility?

So what exactly is compatibility, and how many ways can an application be compatible? Well, first there's platform side-by-side (SxS) compatibility. This refers to the ability to have multiple versions of a platform installed and running at the same time, allowing each application to run on the version of the platform on which it was built. For example, on a machine running Windows Vista™ with versions 1.1 and 2.0 of the .NET Framework installed, apps built against the .NET Framework 2.0 will run against the 2.0 version, while any older apps will still run against the 1.1 version.

Platform-backwards compatibility is the ability for an application built on one version of the platform to run properly on later versions of that platform, while platform-forward compatibility means that an application built on one version of the platform can run properly on earlier versions of that platform.

With the .NET Framework 2.0 release, both side-by-side execution and backwards compatibility are supported. The 1.1 release supported forward compatibility as well, but too many features were added and improved in version 2.0 for it to be likely that you could develop an application built on the .NET Framework 2.0 that didn't use them. While it may not be a surprise that we no longer support forward compatibility, we did want to continue to support both side-by-side and backwards compatibility.

Side-by-Side Execution and Backwards Compatibility

On the .NET Framework and Visual Studio teams, the most important of our compatibility goals mandated that installing the .NET Framework should not break existing applications, regardless of the fact that most applications would be relying on side-by-side execution in this scenario and simply run against the version they were built on. Next in order of importance was that we wanted to ensure that applications would run properly if only the latest runtime were installed. There was a bit more leeway here because we expected side-by-side to be the typical configuration, but we also knew that plenty of version 1.1-based applications would be running on version 2.0.

Finally, we wanted applications built on Visual Studio .NET 2002 or Visual Studio .NET 2003 to be easily upgraded to use Visual Studio 2005. We decided that a developer should be able to convert a large, multiproject solution in a couple of hours.

Of course, no matter how careful we were going to be, there would always be some applications that would break when run on a later version of the Framework. When trying to determine if the value of a breaking change outweighs the impact of the break, we would look at these areas of impact, in order: security, standards compliance, reliability or determinism, and correctness.

At the same time we looked at the following criteria for determining the impact of the break: Does it break side-by-side execution for some applications? Will many applications break in a scenarios using only version 2.0? Will it be easy to determine the problem and fix the application?

As you might expect, sometimes applications break for reasons beyond our control. For example, the change that causes the most applications to break with each release is the increment in the version number. There are also many applications that will fail because they have race conditions that will be exposed once the performance characteristics of the runtime have changed. Still other applications will fail because they took dependencies on the runtime that are far beyond the scope of our API's public contracts (for example, using private reflection to read and write private members of Framework classes). These are the types of application breaks that make our guidance on writing compatible programs so vital for application developers and where it's their turn to do their part to ensure continued compatibility.

Case 1: Visual Studio Tools for Office 2003

Of all the compatibility issues discovered in the past few years, this one has far and away generated the most internal controversy. This was one of those cases where there was no actual breaking change in the .NET Framework and yet on the very first build of the 2.0 version, 100 percent of all applications built using Visual Studio Tools for the Microsoft Office System 2003 would be broken simply by installing the same bits with their new version number. We spent a long time discussing solutions, trying to find anything we could change in the .NET Framework to get these applications running, but in the end we had to get creative and at the same time make some very difficult decisions. I'll go into details about the solution and its impact in a bit, but first some details on the actual break.

Technical Details

The model used for Visual Studio Tools for Office 2003 applications is that they are included as part of Word or Excel documents: they get downloaded alongside the .doc or .xls file and are run automatically when the document is opened. To prevent this from becoming the path to a new generation of macro viruses, Visual Studio Tools for Office 2003 utilized Code Access Security (CAS) and ensured that simply downloading these documents would not give their associated applications permission to run; an administrator would have to explicitly grant them permissions on the .NET Framework CAS security store.

This approach left the machine in a very secure state but left these applications open to a subtle, but dangerous, versioning problem: the CAS security store, in the form of the security.config file, is tied to a particular version of the runtime and thus each version gets, and only reads from, its own. But because Visual Studio Tools for Office 2003 applications are loaded inside of Microsoft Word and Microsoft Excel®, they automatically get the latest version of the runtime currently installed on the machine. This collision between a dependency on a version-specific .config file and roll-forward runtime selection caused every Visual Studio Tools for Office 2003 solution to fail on the very first build with the new version number. The applications themselves didn't have any problems and the managed code in Office that hosted them worked as well, but the application had stored its security permissions in the version 1.1 security.config and the runtime itself was, by design, only loading the 2.0 version.

We initially attempted to find a change that we could make to the .NET Framework 2.0 because fixing it there might have prevented end users and developers from ever noticing that there was a problem. In the end, though, we were unable to find a solution confined to the Framework.

The Fix: A Two-Pronged Solution

While we initially wanted to make the change directly in the runtime, the only viable solution was to fix the component in Office that was managing the Visual Studio Tools for Office 2003 applications and validating their security settings. This component, otkloader.dll, needed to be changed so that it would always look in the security.config file that corresponds to the version of the runtime the application was built against. This solution was easy, but it had two important drawbacks. The first is that a Visual Studio Tools for Office 2003 solution built against the .NET Framework 1.1 will always require the version 1.1 runtime to be installed on the machine even if the application is otherwise compatible with a later version. The same would hold true for Visual Studio Tools for Office 2003 applications based on the .NET Framework 2.0.

The second drawback to fixing the break directly into the otkloader leads us to the second prong in this fix. By placing the fix inside of an Office component, rather than the .NET Framework, it becomes possible, and even probable, that the .NET Framework 2.0 will get installed on a given machine before the Office Update does. This means that installing the .NET Framework 2.0 on a machine before the Office Update to the otkloader gets installed will cause all Visual Studio Tools for Office 2003 applications to fail, going against one of the central pillars of our release: "Installing the .NET Framework cannot break existing applications." So we took a step back, restated the problem, and found the solution.

We simply needed to prevent version 2.0 from being loaded until we detected that the otkloader update was present and only then allow the Word and Excel processes to float forward as they normally would. This led us to develop the "Lockback Shim" which forces a process that would normally get the latest runtime to run with a previous version instead. Internally, we affectionately call these Little Blue Switches in contrast to our Big Red Switch which, for testing purposes, forces all processes on the machine to the latest version installed.

Now, by adding Word and Excel to the lockback list, we are able to keep Word and Excel on version 1.1 until we detect that the version 2.0-compatible otkloader has been installed. This was a very hard choice for us to have to make. It means that even on machines that do not have Visual Studio Tools for Office 2003 applications we will refuse to load the .NET Framework 2.0 for Word or Excel add-ins that may require it until we detect the fix for Visual Studio Tools for Office 2003 applications. This might hamper the adoption of version 2.0 as a development framework for Office but we felt that the impact of breaking all Visual Studio Tools for Office 2003 applications would be too great. We have taken many steps to mitigate this—the fix is included as part of the Visual Studio Tools for Office 2003 2.0 runtime, it is available as an automatic update on Office Update, and it will be available for inclusion in Visual Studio setup projects, but it was still a tough call.

Testing

For the releases of the .NET Framework 2.0 and Visual Studio 2005 we decided that we needed to be 100 percent compatible with the previous releases. We assembled a set of applications of various types, sizes, and complexity to test against the latest builds. On day one, build one, our pass percentage was very high but as check-ins were made the number started to drop.

Initially, most of the failing applications were blocked by simple bugs—the type that tend to creep in at the beginning of a major development cycle. As these bugs were fixed and others opened, our passing percentages began to trend upwards. We soon reached some of the first intentional breaking changes of the product and it became clear that our goal of 100 percent compatibility was in direct conflict with some of our plans for new features. Thus the Developer Division Compatibility Council (DDCC) was formed.

DDCC was a central group of three senior members of teams from across the division whose job was to make the trade-offs between compatibility and other quality metrics. We retained our target compatibility goal of 100 percent but added a * next to the number with the footnote: "except for DDCC-approved breaking changes." After this, when teams wanted to make a breaking change or when one was discovered, the team would go to DDCC for a decision. The pass percentages for our applications continued to rise and were very high by the time we shipped Beta 1.

Soon after we shipped Beta 1, we received MSDN® Feedback reports indicating that the high passing percentages we saw in our test lab were not realized in the real world. So we released an instrumented build of Beta 1 in the hope that developers would use it and tell us which of the .NET Framework APIs they were depending on. This would help us determine the possible impact of our breaking changes. We did everything we could to publicize this release. Still, in the end we received approximately five uploads of code coverage information. It was clear we needed a new way to reach customers.

As a result, we held the first of many Compatibility Developer Labs. Here we discovered that when configured to run on the .NET Framework 2.0, 100 percent of the version 1.0 and version 1.1 applications tested failed. Except for two Microsoft Office Add-ins, every single application passed in the default side-by-side configuration (with both the .NET Framework versions 1.1 and 2.0 installed).

With the realization that our previous success metrics were insufficient, we took a step back and regrouped. We retained our goal of 100 percent passing except for DDCC-approved breaking changes. In addition, we added absolute metrics detailing what percentage of various types of apps must pass in each of the compatibility scenarios, regardless of reason for failure.

Next we started an extensive search for external applications from many different channels. In a couple of months we expanded our internal test suite from fewer than 50 applications to more than 250 that included everything from Microsoft server applications, to small Web downloads, to enterprise-level inventory and personnel software, and even two robotics applications. With this expanded suite we were able to validate our compatibility against a wide range of applications and could finally identify the issues that were impacting multiple applications.

In addition to expanding our in-house suite of applications, we expanded our first Compatibility Developer Lab and started bringing more developers on site so that we could assist them in testing their applications on the latest builds. We were able to test apps we would never be able to bring in-house and get results for scenarios that we usually can't test.But some apps couldn't be moved from one environment to another and some customers couldn't make it to Redmond. So we went to them.

Case 2: Enterprise Library

Enterprise Library was created by the Microsoft patterns & practices group as a set of sample libraries that provide functionality commonly required in enterprise applications. The libraries are released in source form and could be considered "functional guidance" that gives enterprise developers a solid base to build upon. Depending on how a developer chooses to use the Application Blocks, they can be compiled alongside the application, modified and extended to fit the needs of a particular application or environment, or just consulted as guidance in developing an independent solution. The most common use case is to simply compile it with the application—extensive customization and modification is much less common.

Enterprise Library and its application blocks are exceptionally popular. In our series of compatibility tests in India, we estimate that fully half of the line-of-business (enterprise) applications tested were built using Enterprise Library. This popularity means that any changes to the .NET Framework that impact Enterprise Library will be magnified as every application built with it will be impacted as well.

Technical Details

One of the goals of the Enterprise Library Application Blocks is to implement their functionality while following the best practices for developing on the .NET Framework. So, when developing the Configuration Application Block, the Enterprise Library team, in the January 2005 release, was diligent in not only properly reading the information from the configuration files, but also in verifying that the configuration files were properly formed.

In particular the Configuration Application Block verified that all the attributes in the section declaration element were understood, there were no unregistered configuration sections, and no configuration sections used a reserved name (names starting with ".config").

For the application's .config file these are good things to validate, but because of a subtlety in the way the configuration system works this verification code was running not just on app.config but on machine.config as well.

As it turns out, all three of these validation steps caused problems when running these applications on the .NET Framework 2.0.

We added a new feature that let partially trusted applications read information from certain sections in the configuration section (a new "requirePermission" attribute in the section declaration). We added a new implicit section that had no explicit section handler but was understood by the version 2.0 configuration system. And finally, we used a reserved name (configProtectedData) for that section in order to avoid clashing with any user-defined sections.

These are changes that we would generally not consider breaking but were unfortunately added late in the product cycle (post Beta 2) and cause all applications built with the January release of Enterprise Library to fail when running on the latest version of the .NET Framework. The June release of Enterprise Library did not have these problems, but the ratio for downloads of the January versus June releases was more than 5 to 1.

Possible Fixes

The fact that the changes that exposed these problems in the Configuration Block were added so late that the July CTP was the first public release that contained them meant that when we discovered them in early August, we only had a few weeks to make any fixes to the .NET Framework before we entered the pre-release stabilization period. Complicating things further was the fact that the second failure (unregistered section) actually obscured the third failure (section using a reserved name) and gave us even less time to react to it.

If there were only a couple applications out there that ran into this problem, we might have shrugged it off and worked with the application vendor to fix the problem. But because it was our guidance and sample code that caused applications to fail and because so many were built that include the failure, a fix was essential. In less than a week the System.Configuration team came up with a private build that contained the fixes for the two known breaks.

  • They removed the requirePermission attribute from all elements in the default machine.config file and added them in programmatically at run time.
  • They also added a section for configProtectedData and registered a fake section handler for it.

It was only once we had a build with these fixes that we were able to find out that the Configuration Application Block was validating that no reserved sections were used. Then we realized what trouble we were in. The only way we could fix this would be to rename this section so that it didn't start with "config". But we were only two weeks from entering the stabilization period and renaming the section would require finding all the places in Visual Studio and SQL Server™ that relied on this section (there were many) and giving them the new name. Then we would have had to update all of our documentation and samples (in all languages and localizations) to reflect this new name.

All the changes involved with renaming the section would have taken so long that we would have had to significantly slip the release date of Visual Studio 2005 and SQL Server 2005. Then one of our team members proposed an alternate solution that, if implemented, may have gone down in history as one of the ugliest hacks ever: rather than rename the section so that the validation code no longer failed on it, hack the String.StartsWith method so that it in this specific situation it lied and returned false rather than true. We would detect that the application had just called "configProtectedData".StartsWith("config") and return false. We even designed a series of optimization to make this check impact other applications as little as possible:

if (result == true && input.Length == 6 && this.Length == 19 &&
    this.Equals("configProtectedData") &&
    MethodInfo.GetCurrentMethod().Name.Equals("VerifySectionName"))
{
    return false;
}
But making that change to the String class so late in the product cycle was extremely risky.

Without fixing the reserved section name issue the other two fixes were useless. We were left with three options. We could rename the section and significantly delay the release of Visual Studio and SQL Server, hack String.StartsWith to return false when it should otherwise return true, or leave customers of the January 2005 Enterprise Library broken and work with the Enterprise Library team to distribute guidance on how to fix their applications.

In the end we went with the third option. We couldn't slip these products and we couldn't introduce such an ugly hack into such a low-level API as the String class without having sufficient time for testing its impact. Plus, we had serious misgivings about polluting the implementation of such an important API. The changes required to update an application depending on Enterprise Library are minimal and we've made sure to provide several good options for developers doing such an update. We have yet to see an application built with Enterprise Library fail simply by installing the newer framework, but this is the one break we left in the product that hurt the most.

Wrap-Up

In the end, these efforts had a profound impact. By expanding our internal coverage, bringing customers to us, and visiting them directly, we were able to find classes of compatibility issues that we had simply never encountered before. The result was that the vast majority of applications will not be impacted by the installation of the .NET Framework 2.0, most applications will work just fine on machines that only have the latest version, and, with the late-breaking changes to ASP.NET project migration due to feedback from our labs, most developers will have no problems migrating large, multiproject solutions, from Visual Studio .NET 2003 to Visual Studio 2005, in the course of an afternoon.

For a look at two actual breaking changes and how we decided to handle them, see the case studies sidebars (Case 1: Visual Studio Tools for Office 2003 and Case 2: Enterprise Library). They were selected because we expect that they are really the only two compatibility issues that are going to impact more than a few isolated apps. They both also have interesting stories behind them and will impact developers and their applications in different ways.

Send your questions and comments to clrinout@microsoft.com.

Jesse Kaplan is a Program Manager on the CLR team at Microsoft and, among other things, is responsible for application compatibility. You can contact him at jessek@microsoft.com.


Data Points: Using XQuery, New Large DataTypes, and More

New information has been added to this article since publication.
Refer to the Editor's Update below.

Data Points
Using XQuery, New Large DataTypes, and More
John Papa

Code download available at: DataPoints0603.exe (116KB)

SQL Server 2005 introduces a lot of new features, but it also enhances the popular and oft-used Transact-SQL (T-SQL) language. Changes include the introduction of new datatypes to store large values using the MAX indicator, the integration of enhanced XML querying and data modification with XQuery, and the new XML datatype.

In this month's column I will address these and other features based on some of my most frequently received questions regarding SQL Server™ 2005 and its T-SQL enhancements. All sample code, which runs on the AdventureWorks database that ships with SQL Server 2005, is included in the download file. I also included some extra examples in the downloadable code for your extended enjoyment.

Q I know I can store XML in the new XML datatype in SQL Server 2005, but how can I query parts of the XML without pulling all of it out of the database using an XmlDataReader? For example, how do I filter or pull a subset of XML nodes?

A The XML datatype is one of the brand new datatypes introduced to SQL Server 2005. It not only makes storing XML documents and fragments easier and cleaner than in previous versions of SQL Server, but it also provides mechanisms to interact with the XML data. SQL Server 2005 accomplishes the querying and modification of XML data by allowing the XML datatype's methods to use XQuery. XQuery is to XML what SQL is to relational data. (The XQuery language specification is located at www.w3.org/TR/xquery.)

The XML datatype has at its disposal a subset of XQuery already built in. The XQuery methods accept XPath expressions that can be used to navigate the XML. There are five built-in functions (shown in Figure 1) that interact with the XML datatype. In fact, the five functions are accessible right from an XML variable using the following sample format, where @myXml is an XML variable:

@myXml.Query(@myXPathExpression)

Figure 1 XML Datatype Functions (Close Figure 1)

Function | Description
Query | Gets a set of nodes from an XML document/fragment (returns XML)
Value | Gets a single value from an element or attribute of an XML document/fragment (returns scalar value)
Exist | Returns a Boolean value indicating if the XQuery expression returns values
Modify | Changes values in an XML document/fragment
Nodes | Gets a context to a node based upon the XQuery expression

Some examples would help illuminate the use of these new features. In the next few examples I will use a few of the AdventureWorks database's tables, which contain XML columns, for that purpose. First, let's assume that you want to grab all of the résumés from a set of prospective job candidates who have experience with computers. You can retrieve values from an XML document (resumes) stored in the XML datatype using the query method, as shown here:

SELECT
    Resume.query('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
    Resume";
    data(/Resume/Name/Name.First)') FirstName
FROM
    HumanResources.JobCandidate
WHERE
    Resume.exist('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    /Resume/Employment/Emp.OrgName[contains(.,"World")]') = 1

This example accesses the HumanResources.JobCandidate table's Resume column, which contains the candidate's résumé in XML format. Notice that the SELECT clause refers to the Resume column and issues the query method directly from it. The query method accepts an XPath expression that in this case evaluates to use a specific XML Schema Definition (XSD) as its default namespace. (This XSD happens to be included in the SQL Server 2005 AdventureWorks database.) By using this XSD as the default namespace there is no need to prefix all of the nodes in the XPath expression with the namespace prefix.

This XPath is referring to the first name node and selecting it to be returned. By using the data clause, I am telling the statement to return only the contents of the selected nodes. The WHERE clause in this example uses the exist method to evaluate if the company name has the word "World" in its title. The exist method returns either a 0 or a 1 to represent false or true, respectively. This statement will return a list of all of the first names where the person previously worked at a company that had the word "World" in its title:

   FirstName
   ----------------
   Stephen
   Peng
Removing the word "data" from this example would change the results to return the entire node and not just its contents:
FirstName
----------------
<Name.First xmlns="http://schemas.microsoft.com/sqlserver/2004/07/
adventure-works/Resume">Stephen</Name.First>
<Name.First xmlns="http://schemas.microsoft.com/sqlserver/2004/07/
adventure-works/Resume">Peng</Name.First>
So far I have touched on ways to query XML using the query and exist methods. You also can use the value method if all you are looking to retrieve is a single node's contents. The following is an example where I retrieve the list of e-mail addresses from the résumé pool:
SELECT
    Resume.value(        'declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    (/Resume/EMail)[1]', 'VARCHAR(100)') as email
FROM
    HumanResources.JobCandidate
WHERE
    LEN(Resume.value('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    (/Resume/EMail)[1]', 'VARCHAR(100)')) > 1
Notice the use of the value method in this example. I could have used the query method, but since I only want a single value and only its contents, the value function offers a bit cleaner and more efficient option in this case. For example, notice that the data keyword is not needed when using value since you are already referring to the contents of the nodes. The value method expects a singleton to be returned. In other words, it expects a single value to be returned so you need to specify which instance of the e-mail node you want to grab. If you have only a single row, it is most common to just code [1] to indicate the first instance. The value method also expects to be told to what datatype to convert the value. In this case I convert the e-mail address to a varchar(100). The query just shown returns the following results:
Shai@Example.com
Max@Wingtiptoys.com
Krishna@TreyResearch.net
Stephen@example.com
Tai@Example.com

You can also use the for, where, return, and order by statements to retrieve data. Figure 2 shows a command that embeds a for/return statement inside the query method's expression. In this case, the for statement iterates through the /Resume/Employment nodes. For each iteration, the current node is referenced by the variable $employer. Within the for statement you can add a where clause to filter the nodes you want to return data from as well. In this example I simply iterate through each Employment node and return the contents of its Emp.OrgName child node.

Figure 2 XQuery Using For and Return (Close Figure 2)
SELECT
    JobCandidateID,
    Resume.query('declare default element namespace
"http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/Resume";
    for $employer in /Resume/Employment
    return data($employer/Emp.OrgName)
    ') Company
FROM
    HumanResources.JobCandidate
WHERE
    Resume.exist('declare default element namespace
"http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/Resume";
    /Resume/Employment/Emp.OrgName[contains(.,"World")]') = 1

Taking a closer look at the WHERE clause of the SQL statement in Figure 2, you will notice that I use the exist method to limit the rows. Here I use the contains statement to look for any /Resume/Employment/Emp.OrgName node whose content contains the word "World".

There are several variations of these commands that you can use in combination. The SQL Server Books Online has a complete list of the functions and statements available to the new XQuery capabilities of SQL Server 2005.

Q How can I modify parts of the XML in an XML datatype without overwriting it?

A SQL Server 2005 exposes a modify method on the XML datatype which provides the ability to insert, update, or delete XML data within an XML datatype. There are three different statements that can be used with the modify method: Insert, which is used to insert new nodes into the XML, Replace Value Of, which is used to change the value of one or more nodes in the XML, and Delete, which is used to delete nodes from the XML.

Keep in mind that all of these statements are issued in a SQL UPDATE statement. For example, when inserting data, you are actually inserting more XML into an existing row's XML datatype column in the database. All three of these methods manipulate data within an XML column. Their usage is quite similar to the querying methods (nodes, exist, value, and query) in that they accept an XPath expression to locate the nodes to manipulate. The insert syntax is as follows:

xmlColumn.modify(
'insert your-content
{as first | as last}
into | before | after
your-xpath-expression')
The insert keyword is followed by the content or XML nodes that you want to insert. Then, optionally you can specify whether the content should be inserted first or last with respect to the XPath expression. Next, specify whether the content will be inserted into, before, or after the node referred to by the XPath expression. Finally, specify the XPath expression that locates the node that you want to refer to for the insert.

It's time for an example again. In this case let's examine some code that inserts a telephone extension node and value for the job candidate named Shai Bassli. Figure 3 shows the code that will insert a telephone extension node and value of 555 within the first Telephone node for Shai Bassli.

Figure 3 Insert a Node (Close Figure 3)
DECLARE @ext int
SET @ext = 555

UPDATE    HumanResources.JobCandidate
SET    Resume.modify('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    insert <Tel.Extension>{sql:variable("@ext")}</Tel.Extension>
into(/Resume/Address/Addr.Telephone/Telephone)[1]')
WHERE
    Resume.exist('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works
        Resume";
    /Resume/Name/Name.Last[. = "Bassli"]') = 1
The syntax to modify a node is as follow:

xmlColumn.modify(
'replace value of
your-xpath-expression
with
your-new_value
')

If you look closely at the code in Figure 3 you will see that the extension's value is set using a variable like sql:variable("@ext"). This is a way to embed a local variable or parameter defined in the T-SQL within the XQuery statement. In this case I am passing the value of the extension. This feature makes XQuery much more flexible than if it didn't allow parameters.

The replace value of keywords are followed by an XPath expression that is evaluated and resolved to a node whose value you want to replace. For example, I could locate and replace the e-mail address of a person. Following the XPath expression is the with statement and then the new value. Figure 4 shows an example of how to locate Shai Bassli's telephone record and replace the extension of 555 with 777.

Figure 4 Replace a Node (Close Figure 4)
DECLARE @ext int
SET @ext = 555
DECLARE @newExt int
SET @newExt = 777

UPDATE    HumanResources.JobCandidate
SET    Resume.modify('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    replace value of
(/Resume/Address/Addr.Telephone/Telephone/Tel.Extension[.=sql:variable("@ext")])[1]
    with xs:int(sql:variable("@newExt"))')
WHERE
    Resume.exist('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    /Resume/Name/Name.Last[. = "Bassli"]') = 1

So far I have shown how to insert a telephone extension and modify its value. Now I will demonstrate how to delete the telephone extension from the XML without affecting the rest of the XML document, effectively returning the XML to its original state. The Delete syntax is as follows:

xmlColumn.modify('delete your-xpath-expression')

The syntax is pretty straightforward here; all you have to specify is the delete keyword and the XPath expression that resolves to the node that you want to remove. Be careful that you specify the correct node to remove because the delete command will remove the node you specify and all of its children. For example, if you specify /Resume as your XPath expression you will be deleting the entire résumé. OK, maybe that is obvious to you, but it is always good to be extra careful with deletes anyway.

The code in Figure 5 shows an example of how to delete (or remove) a node from an XML document (or fragment). Notice that I am deleting the node that has a telephone extension value of 777 and the last name of Bassli.

Figure 5 Delete a Node (Close Figure 5)
DECLARE @newExt int
SET @newExt = 777

UPDATE    HumanResources.JobCandidate
SET    Resume.modify('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    delete
(/Resume/Address/Addr.Telephone/Telephone/Tel.Extension[.=sql:variable("@newExt")])')
WHERE
    Resume.exist('declare default element namespace
    "http://schemas.microsoft.com/sqlserver/2004/07/adventure-works/
        Resume";
    /Resume/Name/Name.Last[. = "Bassli"]') = 1

Q I've heard that in SQL Server 2005 I might not have to store my large string data fields in columns using the text datatype because of the new large datatypes. Where have these new datatypes been all my life?

A When SQL Server 7.0 was introduced, it raised the limit on VARCHAR fields to 8000 bytes. There were still issues, however, as many large string-based fields exceeded 8000 characters and had to either be broken into multiple rows, stored in a binary field (for example TEXT, NTEXT, or IMAGE) in the database, or even stored in a document in a file system. SQL Server 2005 blows the doors off the variable length fields' limits! The new MAX specifier can be used with the VARCHAR, NVARCHAR, and VARBINARY datatypes to declare variables that can store vast quantities of data with fewer limitations than the binary large object datatypes.

A variable declared as VARCHAR(MAX) or VARBINARY(MAX) can store 231 (or about 2 billion) characters. A variable declared as NVARCHAR(MAX) can store 230or about 1 billion) characters. Unlike the binary large object datatypes, these new MAX datatypes can be passed as parameters to stored procedures, just like the new XML datatype, and they can be indexed or even parsed by string functions. Now, of course, I won't go crazy and define all of my string columns as VARCHAR(MAX) in my tables. However, these new fields can be great for note-based columns. For example, assuming your application needs to store a diary of daily events where the entries are all text and can be a quick sentence or seven pages of text. In the past you could solve this problem several ways, such as by storing this data in a TEXT column, limiting the notes to 8000 characters in a VARCHAR(8000), or even splitting the text up into multiple rows behind the scenes—like the text nvarchar(4000) column in the syscomments table. None of these solutions were great. With SQL Server 2005 you can now define a column to be a VARCHAR(MAX) and it can store the entire text in it, and even be indexed.

Ah, but you might remember that in SQL Server 2000 the reason for the limitation was page size. This meant that table rows could not exceed 8060 bytes (in SQL Server 7.0 and SQL Server 2000). You could define multiple columns on a table that exceeded this limit, but then you would get a warning indicating that if the data later exceeded the limit, you would get an error. Often, this meant DBAs would try to split up tables so they would not hit the limit. In SQL Server 2005, rows can span multiple pages, so the 8060 limit is no longer applicable, and you can define several columns on a single table as MAX types, if needed.

[ Editor's Update - 11/21/2006: Rows cannot span pages in SQL Server 2005, however portions of the row may be moved off the row's page so that the row can be very large. See msdn2.microsoft.com/ms190969.aspx for more information.]

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a Senior .NET Consultant with ASPSOFT and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences such as VSLive or blogging at codebetter.com/blogs/john.papa.


Cutting Edge: Windows Workflow Foundation

Cutting Edge
Windows Workflow Foundation
Dino Esposito

Code download available at: CuttingEdge0603.exe (663KB)
Contents
A Common Business Scenario
Applying Windows Workflow Foundation
The Helpdesk Solution
The Human Factor
The Helpdesk Front End
State Persistence
Resuming the Workflow Instance
Conclusion

In the January 2006 issue, Don Box and Dharma Shukla introduced Windows® Workflow Foundation and discussed the overall architecture of the framework and its constituent components (see WinFX Workflow: Simplify Development With The Declarative Model Of Windows Workflow Foundation). It inspired me to take this topic one step further and discuss how you can use Windows Workflow Foundation to address a common business scenario where automatic processes intersect with human activity. It provides a framework for developing and executing a wide range of applications based on complex processes. Typical examples are document management, business-to-business, and consumer applications. You can use Visual Studio® 2005 to help design the underlying workflow in addition to the top-level applications and assemblies involved.

A Common Business Scenario

Organizations typically have a number of internal processes for tasks such as order processing, purchase requests, travel expenses, and the like. Workflow brings order to these independent processes in a transparent, dynamic, and robust fashion.

Let's consider a typical helpdesk workflow process. It begins when the helpdesk employee gets a customer call and opens a ticket recording the name of the customer, the time of the call, and a brief description of the issue. Once the ticket has been created, the employee forgets about it and waits for another incoming call. At the end of the day, he logs off the computer and goes home. At the same time, in another department, perhaps in another city, a bunch of technicians are alerted to open issues. Each active technician picks up a request and either solves it or escalates it to a second level of help. How would you write code to implement this process?

You might have a Windows Forms application that collects input data about the call and creates a record in a database—the ticket, with time, description, status, and a unique ID. Users of a second Windows Forms application will see a real-time list of pending requests and pick one up. The operator then does whatever is in his power to resolve the issue (call back the customer, retrieve requested information, send e-mail, or perform some remote activity) and indicates whether the problem was solved or will need to be investigated further. This decision can be represented with an imperative action such as clicking a button to update the ticket in the same underlying database. Finally, if there's another category of users to involve, a custom made front end will give them a chance to indicate that the issue has been successfully closed or aborted.

Although this process clearly represents a workflow with some human-driven decision points, it can be easily implemented with traditional sequential code written with classic programming languages and databases.

Applying Windows Workflow Foundation

When you have a workflow-based system composed of activities, like you have with Windows Workflow Foundation, you can implement an application using a powerful mix of imperative code and declarative maps of activities and the declarative rules that bind them. The major benefit is that you can model (even visually) the solution and have Windows Workflow embed a run-time server to interpret your graph and proceed along the links that you defined among building blocks. The more complex the process, the simpler it is to devise and implement a flow for it. The more dynamically changeable the process, the less code you'll need to write and maintain. Let's see how to implement a Windows Workflow Foundation solution for the helpdesk scenario.

The Helpdesk Solution

The helpdesk workflow I've built begins by creating a ticket and then stop while waiting for a response from a connected user or technician. Whether the ticket is closed or escalated, the workflow gets an external event and updates the internal state of the application to track the event. In light of this, the workflow requires an interaction with the outside world. These kinds of asynchronous activities are one of the problems inherent in real-world workflow processes that Windows Workflow Foundation addresses. Because interaction with an entity outside the system is required, the host application and the workflow can define a contract for any data exchange that proves necessary. The IHelpDeskService interface you see here describes the communication interface that is established between the workflow and its host:

[DataExchangeService]
public interface IHelpDeskService
{
    event EventHandler<HelpDeskTicketEventArgs> TicketClosed;
    event EventHandler<HelpDeskTicketEventArgs> TicketEscalated;

    void CreateTicket(
        string description, string userRef, string createdBy);
    void CloseTicket(string ticketID);
    void EscalateTicket(string ticketID);
}

Now let's begin by writing a service class named HelpDeskService, as shown in Figure 1. The HelpDeskService class is added to the workflow runtime and represents the point of contact between the host application and the workflow. The host application invokes the public methods on the class to fire external events to the workflow. Fired events signal domain-specific events that will guide the flow of operations. Methods on the IHelpDeskService interface represent the only pieces of code that a workflow can invoke on external components through the InvokeMethod activity. Factored out in this way, the logic of the HelpDeskService adds a lot of flexibility to the workflow as it basically lets the workflow call back the host for the actual implementation of basic actions. By including a class that implements the IHelpDeskService interface different hosts can create, solve, or escalate tickets using different algorithms and storage media while keeping the workflow logic intact. You can compile the interface and the class in a distinct assembly or just keep these files inside the same assembly where the workflow is defined.

Figure 1 Implementing Workflow with HelpDeskService (Close Figure 1)
using System;
using System.Threading;
using System.Workflow.ComponentModel;
using System.Workflow.Runtime;
using System.Workflow.Runtime.Messaging;

namespace MySamples
{
    public class HelpDeskService : IHelpDeskService
    {
        // Implement events
        public event EventHandler<HelpDeskTicketEventArgs> TicketClosed;
        public event EventHandler<HelpDeskTicketEventArgs>
            TicketEscalated;

        public void RaiseCloseTicketEvent(Guid instanceId)
        {
            // Raise the event to the workflow
            HelpDeskTicketEventArgs args =
                new HelpDeskTicketEventArgs(instanceId, "");
            if (TicketClosed != null) TicketClosed(this, args);
        }

        public void RaiseEscalateTicketEvent(Guid instanceId)
        {
            // Raise the event to the workflow
            HelpDeskTicketEventArgs args =
                new HelpDeskTicketEventArgs(instanceId, "");
            if (TicketEscalated != null) TicketEscalated(this, args);
        }

        void IHelpDeskService.CreateTicket(
            string description, string userRef, string createdBy)
        {
            // Fill up a ticket: same ID as the workflow
            string ticketID =
                WorkFlowEnvironment.CurrentInstanceId.ToString();

            // Create ticket on the DB
            HelpDeskHelpers.CreateTicket(
                ticketID, description, userRef, createdBy);
        }

        void IHelpDeskService.CloseTicket(string ticketID)
        {
            // Update ticket on the DB
            HelpDeskHelpers.UpdateTicket(ticketID, TicketStatus.Closed);
        }

        void IHelpDeskService.EscalateTicket(string ticketID)
        {
            // Update ticket on the DB
            HelpDeskHelpers.UpdateTicket(ticketID,
                TicketStatus.Escalated);
        }
    }
}

HelpDeskWorkflow is a sequential workflow hosted by a Windows Forms app. Figure 2 shows its graphical model. It begins with an InvokeMethod activity that causes the app to create the ticket based on information provided by the user. After that, the ticket sits in a database waiting for a technician to pick it up and either solve it or escalate it.

Figure 2 Helpdesk Workflow in Visual Studio 2005
Figure 2 Helpdesk Workflow in Visual Studio 2005

After creating the ticket, the workflow waits, potentially for a long time—even hours or days. What happens during this time? Should the workflow instance stay loaded in memory? If the workflow becomes idle, you can unload it from memory—a process also known as passivation. A local service, such as the helpdesk service, remains the main point of contact between the host application and the sleeping workflow.

The Human Factor

When the human factor interacts with the host application and does something to wake up the workflow, the local service posts a request to the runtime to resume the passivated workflow. The Windows Workflow Foundation toolbox contains an activity, called Listen, that just idles the workflow and listens for incoming wake-up calls. The block labeled WaitForSolution in Figure 2 is an instance of the Listen activity.

The Listen activity is basically useless without one or more child branches, each representing a possible event that can be hooked up at this point. In the helpdesk example, the Listen activity contains two branches—ticket closed and ticket escalated. Each branch is an EventDriven activity. Both Listen and EventDriven activities require no special settings and are mere containers of child activities. To catch an external event, the workflow needs an EventSink component.

In Figure 2, the TicketClosed block is an EventSink bound to the TicketClosed event in the local helpdesk service. Figure 3 lists the EventSink properties set here. To begin, you select the interface of the service that exposes the event. Setting the InterfaceType property is a requirement; in addition, you can only select an interface decorated with the [DataExchangeService] attribute, as in Figure 1.

Figure 3 EventSink Properties
Figure 3 EventSink Properties

Once the data exchange interface is set, the EventName property is prefilled with all events found on the interface. You pick up the event of choice and set its parameters. If you want a further notification about when the event has been processed, you set the Invoked property, too.

The EventSink returns control to the workflow after a period of idleness due to waiting for human intervention. Following the event, you proceed with any further activity that suits the workflow. For example, in the helpdesk application I call another method on the local service to update the status of the ticket in the ticket database.

In a real-world scenario, there's at least one database the workflow needs to interact with, directly or indirectly. In this example, I'm using a SQL Server™ 2000 table, Tickets, that stores a row for each ticket being processed. By design, the ID of the ticket matches the ID of the workflow that is used to manage it.

The Helpdesk Front End

Once compiled, the workflow is nothing more than a reusable assembly and thus can be referenced by any type of .NET-based application. Let's now imagine a front-end application for helpdesk operators. The application is a Windows Forms program that allows users to fill in a form and create a ticket to initiate the workflow, as shown in Figure 4.

Figure 4 HelpDesk Front-End App
Figure 4 HelpDesk Front-End App

The workflow calls into the CreateTicket method of the local service and has the service add a new record to the tickets database. After the operator has opened a new ticket, he continues answering phone calls and replying to e-mails while the workflow is started and becomes idle waiting for a human intervention. Each ticket is represented with a different instance of the workflow. For simplicity, the ID of the workflow is also the ID of the ticket.

State Persistence

At the end of the day, a workflow is a tree of activities, so how is its lifetime managed? The Windows Workflow Foundation run-time engine manages the execution of any workflow and allows workflows to remain active for a long time and even survive machine reboots. The run-time engine is powered by pluggable services that provide an overall rich execution environment—transactions, persistence, tracking, timer, and threading.

It's unlikely that you want your workflow to stay in the memory of the host process until some intervention causes it to proceed. As mentioned, many real-world human-based workflows may take hours or longer for this to happen. As in the preceding example, the helpdesk operator starts the workflow and loads the workflow class inside the front-end application process. That operator may then shut down his machine and go home. However, the ticket must remain active and available to other operators, even from within another application. For this to happen, the workflow must support serialization to a durable medium.

A workflow may be a long-running operation and it is impractical for that object to remain active in memory for days. First of all, a host process can't generally afford to cache all the objects that accumulate in days of continued activity; second, the host process might shut down or reboot more than once.

The solution is to configure the runtime to unload workflows as they go idle. In this case, the host program will use the following code to initialize the runtime:

WorkflowRuntime wr = new WorkflowRuntime();
wr.StartRuntime();

You can also set a few event handlers on the WorkflowRuntime class to run some code when the workflow is idle, persisted, or unloaded. Configured in this way, the runtime will automatically serialize the workflow to a storage medium as the workflow encounters a Listen activity or enters a wait state. Another alternative is to instruct the host program to programmatically persist the workflow at a well-known point of execution. In this case, you call the EnqueueItemOnIdle method on the WorkflowInstance class. You get an instance of the WorkflowInstance class as the return value of the CreateWorkflow method on the run-time class:

WorkflowInstance inst = theRuntime.CreateWorkflow(type);
...
inst.EnqueueItemOnIdle();
inst.Unload();

Just remember though that a call to EnqueueItemOnIdle doesn't necessarily persist the workflow immediately. Persistence is immediate only if the workflow is idle or suspended. Otherwise, the runtime takes note and persists the workflow instance later when the workflow is suspended or idle. Note that EnqueueItemOnIdle is limited to saving the state of the workflow instance and doesn't unload it from memory. To unload a workflow instance, you need to call the Unload method. One of the standard run-time services supported by the workflow runtime is the workflow persistence service. You turn it on through the code in Figure 5.

Figure 5 Workflow Persistence Service (Close Figure 5)
private WorkflowRuntime InitWorkflowRuntime()
{
    // Get a new workflow runtime
    WorkflowRuntime wr = new WorkflowRuntime();

    // Add custom HelpDesk service
    theHelpDeskService = new HelpDeskService();
    wr.AddService(theHelpDeskService);

    // Add system SQL state service
    SqlWorkflowPersistenceService stateService =
        new SqlWorkflowPersistenceService("Data Source=localhost;" +
        "Initial Catalog=WFState;UID=...;");
    wr.AddService(stateService);

    // Start
    wr.StartRuntime();
    return wr;
}

private void btnCreate_Click(object sender, EventArgs e)
{
    // Fill the Parameters collection for this instance of the workflow
    Dictionary<string, object> parameters =
        new Dictionary<string, object>();
    parameters.Add("TicketDescription", txtDesc.Text);
    parameters.Add("TicketUser", txtUser.Text);
    parameters.Add("TicketCreatedBy", txtCreatedBy.Text);

    // Get the type of the workflow
    Type type = typeof(MySamples.HelpDeskWorkflow);

    // Start the workflow instance
    WorkflowInstance inst =
        theWorkflowRuntime.CreateWorkflow(type, parameters);

    // Feedback to the user
    string msg = String.Format("The ticket '{0}' has been successfully " +
        "created!", inst.InstanceId);
    MessageBox.Show(msg, "HelpDesk", MessageBoxButtons.OK,
        MessageBoxIcon.Information);
    FillGrid();
}

The WorkflowPersistence service adds serialization capabilities to all the instances of workflows that execute inside a given run-time engine. The workflow service core functionality is represented by the WorkflowPersistenceService class. Windows Workflow Foundation provides an implementation of it through the SqlWorkflowPersistenceService. This class saves workflow data to a SQL Server 2000 or SQL Server 2005 database with a well-known schema. There are a couple of scripts and dialog-based utilities that you can use to easily create the target database. You can download everything you might need from Windows Workflow Foundation Web. An instance of the SQLWorkflowPersistenceService must be created and registered with the runtime before the runtime is actually started. The constructor of the persistence service takes the connection string as its sole argument.

As mentioned, run-time services are a pluggable part of the overall Windows Workflow Foundation architecture, so you can create your own services and replace standard services with your own.

Resuming the Workflow Instance

In the scenario being discussed, another operator—the technician—is expected to pick up any open tickets and do whatever he can to solve or escalate them. The technician will use another application or, at least, another instance of the app shown in Figure 4. In both cases, another instance of the workflow runtime must be created with persistence support and the right workflow state must be loaded and resumed. As you can see, this can only happen if the ID of the idled workflow has been saved. In the helpdesk sample app, by design, the ticket ID matches the workflow ID and each ticket corresponds to an instance of the workflow created to handle it.

Figure 6 The Problem Solver
Figure 6 The Problem Solver

The Problem Solver application (see Figure 6) initializes the workflow runtime using nearly the same code as the HelpDesk front-end application in Figure 4. The only minor difference is that in the Problem Solver application I register handlers for the WorkflowLoaded and WorkflowCompleted events on the runtime:

WorkflowRuntime wr = new WorkflowRuntime();
wr.WorkflowLoaded += OnWorkflowLoaded;
wr.WorkflowCompleted += OnWorkflowCompleted;

The operator selects a ticket from the displayed list and works with the user who raised it. When done, she clicks to solve or escalate the ticket, thus terminating the workflow. The event handler should retrieve the saved instance of the workflow based on the ticket ID and wake it up from the idle state. Here's the code you need:

string workflowID = (string)ticketList.SelectedValue;
Guid id = new Guid(workflowID);
Type type = typeof(MySamples.HelpDeskWorkflow);
try {
  WorkflowInstance inst = theWorkflowRuntime.GetLoadedWorkflow(id);
  theHelpDeskService.RaiseCloseTicketEvent(inst.InstanceId);
} catch { ... }

The core part of the preceding code is in the call to the GetLoadedWorkflow method. GetLoadedWorkflow takes the GUID that represents the workflow instance and retrieves it from the configured durable medium, if it is not currently in memory. In this case, the workflow is loaded into memory and scheduled for execution. This happens even if the workflow instance has been previously aborted. The WorkflowLoaded event fires when the workflow is loaded back into memory.

GetLoadedWorkflow returns an object of type WorkflowInstance that you can use for inspecting the current status of the workflow as well as its activities. At this point, you raise one of the events the workflow was waiting for. The corresponding EventSink activity will trap the event and proceed until the end of the workflow is reached or a subsequent wait point is encountered.

When the workflow completes, it is automatically removed from the Windows Workflow Foundation persistence database. The WorkflowCompleted event fires when the workflow has reached its end. In the helpdesk scenario, the workflow completes after the operator clicks to solve or escalate (see Figure 6).

From a development perspective, it is essential to note two things. First, to raise an event to the workflow, you have to post the request to a pooled thread:

public void RaiseCloseTicketEvent(Guid instanceId) {
    ThreadPool.QueueUserWorkItem(JustCloseTheTicket,
        new HelpDeskTicketEventArgs(instanceId, "Jim"));
}
public void JustCloseTheTicket(object o) {
    HelpDeskTicketEventArgs args = o as HelpDeskTicketEventArgs;
    if (TicketClosed != null) TicketClosed(null, args);
}

Second, the WorkflowCompleted event may not be raised on the main Windows Forms thread, so you cannot directly update any of the UI controls. (This is due to the fact that Windows Forms controls are not thread-safe.) Updating Windows Forms controls from within a thread different from the one which created them is definitely possible, but requires an indirect call to the method that updates controls:

private delegate void UpdateListDelegate();
private void UpdateList() {
   this.Invoke(new UpdateListDelegate(FillList));
}

You call UpdateList from within the WorkflowCompleted event handler. The Invoke method on the Form class guarantees that FillList gets called on the right thread so that any control can be safely refreshed.

Conclusion

Windows Workflow Foundation allows you to visually design complex algorithms to solve business problems and model processes. A workflow is a tool used to describe a flow of data and actions. Therefore, any scenario where an IF or a WHILE statement are required can be a workflow. However, nobody would ever use a workflow for just an IF statement; the workflow runtime does have a cost that can be amortized as the complexity of the flow grows beyond a given threshold.

Where's the boundary that delimits the cost effective use of a workflow? The helpdesk scenario, as implemented in this column, is probably too simple for a workflow and could have been implemented in an equivalent way by employing a tickets database—even the same tickets database that the workflow deals with through the helpdesk service.

The real benefit of a workflow-based solution consists of making complex processes simpler to model and implement, and, more importantly, evolve and extend. Windows Workflow Foundation provides a managed execution environment for Windows Workflow programs. It provides durability, robustness, suspension/resumption, and compensation characteristics to the programs. In a sense, activities are analogous to intermediate language (IL) opcodes or program statements, but contain domain-specific wisdom. In short, Windows Workflow Foundation makes your program semantics declarative and explicit, and it lets you model your applications close to the real-world process. It's the right tool for the job. You wouldn't write a front-end visual application using IL, but rather you would use a RAD development tool and a more human-readable language. The Windows Workflow Foundation SDK provides an extensible programming language designed for modeling complex business processes, especially when these processes may evolve over time. In this case, the key benefit is that you add ad hoc activities to address the flow of work and use built-in behavioral activities to control that flow. You focus on tasks, the runtime does the rest. If you want to further explore Windows Workflow Foundation or look for other ideas, visit Windows Workflow Foundation.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Test Run: Determining .NET Assembly and Method References

Test Run
Determining .NET Assembly and Method References
James McCaffrey

Code download available at: TestRun0603.exe (133KB)
Contents
.NET Assembly Dependencies
Outputting .NET Assembly Dependencies
.NET Method Dependencies
Comments and Discussion

Before you can test any software system effectively, you must understand the system under test. If the system includes the Microsoft® .NET Framework, understanding the system under test includes understanding its assembly and method dependencies. This is not particularly enjoyable or challenging from a conceptual point of view, but it is absolutely indispensable.

Many tools are available to uncover .NET assembly and method dependencies, but the majority are GUI-based and not particularly automation-friendly. And these tools are not very customizable. Colleagues of mine who write software test automation in a .NET-based environment use two custom-built tools to programmatically determine assembly and method references. When I teach .NET test automation classes to software engineers, I often pull out these tools, AssemblyRefs.exe and AnalyzeMethodCalls.exe. So in this month's column I'll present two very lightweight, automation-friendly, highly customizable tools for finding .NET assembly dependencies and .NET method dependencies.

Figure 1 Finding .NET Assembly Dependencies
Figure 1 Finding .NET Assembly Dependencies

If you can't wait to get started, take a look at Figure 1. The sample run analyzes an application named RunScenario.exe for assembly dependencies and saves output in XML. AssemblyRefs.exe's source is short (less than one page of code) and can easily be modified to suit your needs. Figure 2 shows a tool that finds .NET method dependencies. This sample run analyzes a class library named CombinationsLib.dll and sends output directly to the command shell. AnalyzeMethodCalls.exe is also small (again, less than one page of code) and is easily extended.

Figure 2 Finding .NET Method Dependencies
Figure 2 Finding .NET Method Dependencies

There is nothing especially difficult about these two tools but they raise some very interesting design issues. And while this column is aimed primarily at those with beginning and intermediate skills, even if you have advanced coding skills I bet you'll find some interesting and useful new techniques.

.NET Assembly Dependencies

Let's start by writing a small, flexible tool that determines .NET assembly dependencies. Assemblies are the fundamental building blocks of all .NET-based applications. An assembly can be located in one physical file (an .exe or .dll), or it may span several physical files. Assemblies contain several pieces of data, including an assembly manifest (which contains metadata about the assembly, such as version number), type metadata (which contains information about classes, enumerations, and so forth), the actual intermediate language code that implements the assembly, and a set of resources (typically configuration settings, images, and so on). As the MSDN® documentation points out, functionally, assemblies form a security boundary, a type boundary, a reference-scope boundary, and a version boundary. Therefore, if you are testing code, you must be able to determine which assemblies are referenced by the assembly under test. Without this information you cannot create meaningful test cases. This is also useful for debugging complex systems that will not build properly.

Now that you are motivated, let's write the tool. The key to a .NET assembly reference tool is the Assembly.GetReferencedAssemblies method in the System.Reflection namespace. GetReferencedAssemblies returns an array of type System.Reflection.AssemblyName that contains information about each of the assemblies referenced by the calling Assembly object. An AssemblyName object is a wrapper around the full display name of an Assembly object. For example, for the assembly RunScenario.exe shown in Figure 1, the AssemblyName's string representation is:

RunScenario, Version=1.0.1880.30104, Culture=neutral, PublicKeyToken=null

At first glance it seems as if the problem of generating all assembly references is solved because you can makes calls along the lines of the following:

   Assembly a = Assembly.LoadFrom(
      pathToAssembly);
   AssemblyName [] an =
      a.GetReferencedAssemblies();
   foreach (AssemblyName name in an)
      Console.WriteLine(name.ToString());

This is true up to a point, but there are two problems. First, each of the referenced assemblies can reference other assemblies, which may in turn reference other assemblies, and so forth. The second problem is that this approach is limited to the type of output you can produce (which makes it automation-unfriendly). What you need is code that will get all levels of assembly references, and also will provide a way to easily produce multiple output formats.

Figure 3 Assembly Dependencies Tree
Figure 3 Assembly Dependencies Tree

The solution comes right out of a CS101 class. The assembly references form a tree structure with the assembly under test as the root node. Figure 3 shows a hypothetical set of .NET assembly dependencies. The assembly under test is node A. Node A references assemblies B, C, and D, and so forth.

You need to traverse the tree structure, storing each assembly reference. How? You could write a recursive routine. But recursion has certain disadvantages—I will have more to say about this when I discuss Microsoft hiring interviews later in this column. A better approach is to use a non-recursive algorithm that stores assembly references on a stack:

create empty stack
push root assembly reference(s) onto stack
while (stack is not empty)
{
    pop the top assembly reference
    display info about the popped item
    get all assembly references of the popped item
    foreach reference
        push the reference onto the stack, "right-to-left"
}

This standard algorithm will traverse the reference tree in a "pre-order" traversal. If the algorithm is applied to the tree structure in Figure 3, the output will be: A, B, C, E, F, I, J, D, G, H. This output format doesn't really convey the hierarchical structure of the assemblies, so you need to store additional information that indicates which level each assembly node is on. If you define the top-level node A to be at level 0, then nodes B, C, and D are at level 1; nodes E, F, G, and H are at level 2, and nodes I and J are at level 3. You can use a tiny Info class to represent each assembly's name and level:

class Info
{
    public readonly string Name;
    public readonly int Level;

    public Info(string name, int level)
    {
        this.Name = name; this.Level = level;
    }
} // class Info
Since this class just wraps data fields, you can use a struct instead of a class if you like. To make the tool more general, instead of popping items off the stack and displaying them directly, you can write a routine that stores the assembly Info objects in an ArrayList or a List<Info> in a pre-order fashion. With this approach you can iterate through the resulting objects and display the assembly dependencies as text, or save the info to an XML file, or use another method that meets your needs. Figure 4 shows a short method that accepts as input a path to a .NET assembly file and returns an ArrayList that holds Info objects (assembly names and levels) of referenced assemblies in pre-order fashion.

Figure 4 Storing Reference Dependencies Information (Close Figure 4)
static ArrayList StoreAssemblyReferenceInfo(string targetAssemblyPath)
{
    Hashtable found = new Hashtable();   // used to avoid dupes
    ArrayList results = new ArrayList(); // resulting info
    Stack stack = new Stack(); // stack of Assembly Info objects

    // store root assembly (level 0) directly into results list
    Assembly assembly = Assembly.LoadFrom(targetAssemblyPath);
    stack.Push(new Info(assembly.ToString(), 0));

    // do a preorder, non-recursive traversal
    while (stack.Count > 0)
    {
        Info info = (Info)stack.Pop(); // get next assembly info
        if (!found.ContainsKey(info.Name))
        {
            found.Add(info.Name, info.Name);
            results.Add(info); // store it to results ArrayList

            Assembly child = Assembly.Load(info.Name);
            AssemblyName[] subchild = child.GetReferencedAssemblies();
            for (int i = subchild.Length-1; i >= 0; --i)
            {
                stack.Push(
                    new Info(subchild[i].ToString(), info.Level+1));
            }
        }
    }
    return results;
}

The StoreAssemblyReferenceInfo routine in Figure 4 uses an auxiliary Hashtable object to keep track of assemblies that have already been explored in order to avoid duplication. The loading routine begins by loading the assembly under test from its path supplied as the input argument, using the Assembly.LoadFrom method. You get this root assembly object's name, create an Info object with level 0 from the name, and add it to the stack of Info objects to be processed.

Now that the stack of Info objects has been seeded, you follow the traversal algorithm I described earlier. You pop the top assembly Info object off the stack, load it into memory using the Assembly.Load method, get all of its referenced dependencies using the GetReferencedAssemblies method, and push their associated Info objects onto the stack. This process continues until all unique instances of referenced assemblies in the assembly dependency tree have been accounted for. For the dependency tree in Figure 3, after StoreAssemblyReferenceInfo finishes, the return ArrayList object will hold {A 0}, {B 1}, {C 1}, {E 2}, {F 2}, {I 3}, {J 3}, {D 1}, {G 2}, {H 2}.

Figure 5 Graph Structure
Figure 5 Graph Structure

.NET assembly references can actually form a graph structure rather than a tree (see Figure 5). Because the StoreAssemblyReferenceInfo method does not store duplicate values, it will create a virtual tree structure rather than a graph structure. In practice, this approach has proven to be useful and effective. If you need to, you can easily modify the code presented here to store assembly dependencies as a graph structure. Instead of using an Info object that contains just an assembly name and the assembly tree level, you would create a Node object that stores the assembly name (as a string) and all assembly references (as an ArrayList for example). Because graph structures do not have standard traversal orderings, you can simply store all the Node objects in an ArrayList object and iterate through the list.

Here it's assumed that all referenced assemblies can be successfully loaded. The Assembly.Load(assemblyString) method will throw one of the following exceptions if the assembly cannot be loaded:

ArgumentNullException - assemblyString is a null reference.
FileNotFoundException - assemblyString is not found.
BadImageFormatException - assembly assemblyString is not a valid
                          assembly.
SecurityException - The caller does not have the required permission.

You may want to catch these exceptions, print a warning message, and continue execution even though your resulting dependencies output will not be accurate.

Outputting .NET Assembly Dependencies

With the assembly dependencies information (name and level) saved in pre-order fashion in an ArrayList object, you have great flexibility in output. Suppose you just want a simple text display to the command shell. You can write this method:

static void DisplayReferencedAssembliesAsText(string targetAssemblyPath)
{
    ArrayList results = StoreAssemblyReferenceInfo(targetAssemblyPath);
    foreach(Info info in results)
    {
        Console.WriteLine(info.Name.PadLeft(
            info.Name.Length + info.Level * 2, ' '));
    }
}
And call it along the lines of:
string path = @"C:\Here\There\TargetAssembly.exe";
DisplayReferencedAssembliesAsText(path);

You simply use the assembly-level information to format output, indenting twice as many spaces as the level number. If applied to Figure 3, the output on the console shell will resemble:

A
  B
  C
    E
    F
      I
      J
  D
    G
    H

Suppose you want more sophisticated output. SaveReferencedAssemblies (see Figure 6) takes an ArrayList of assembly Info objects stored in pre-order fashion and produces an XML file.

Figure 6 Outputting Assembly Dependencies as XML (Close Figure 6)
static void SaveReferencedAssembliesAsXML(
    string targetAssemblyPath, string fileName)
{
    ArrayList results = StoreAssemblyReferenceInfo(targetAssemblyPath);
    XmlTextWriter xtw = new XmlTextWriter(fileName, Encoding.Unicode);
    xtw.Formatting = Formatting.Indented;
    xtw.WriteStartDocument();
    xtw.WriteStartElement("Assemblies"); // write XML-required root node
    int pendingEndTags = 0;

    for (int i = 0; i < results.Count-1; ++i) // all but last node
    {
        Info curr = (Info)results[i];
        Info next = (Info)results[i+1];
        if (next.level == curr.level) // sibling up ahead
        {
            xtw.WriteStartElement("Assembly");
            xtw.WriteAttributeString("Name", curr.name);
            xtw.WriteEndElement();
        }
        else if (next.level > curr.level) // child ahead
        {
            xtw.WriteStartElement("Assembly");
            xtw.WriteAttributeString("Name", curr.name);
            ++pendingEndTags;
        }
        else if (next.level < curr.level) // last child
        {
            xtw.WriteStartElement("Assembly");
            xtw.WriteAttributeString("Name", curr.name);
            xtw.WriteEndElement();
            for (int j = 1; j <= curr.level - next.level; ++j)
            {
                xtw.WriteEndElement();
                --pendingEndTags;
            }
        }
    }

    // take care of pesky last node
    Info last = (Info)results[results.Count-1];
    xtw.WriteStartElement("Assembly");
    xtw.WriteAttributeString("Name", last.name);
    xtw.WriteEndElement();
    for (int j = 1; j <= pendingEndTags; ++j)
        xtw.WriteEndElement();

    xtw.WriteEndElement(); // write end of required XML root node
    xtw.Close();
}

The SaveReferencedAssemblies routine traverses through the ArrayList of assembly Info objects looking at the current node and the next node in the list. If the level of the next node is equal to the level of the current node, that means the current node has a sibling, so you can emit the complete node as XML with a start tag and an end tag. If the level of the next node is greater than the level of the current node, the current node has children, so you emit only the start node of an XML element. And if the level of the next node is less than the level of the current node, that means the current node is the last node in a group of siblings, so you need to emit the entire node and also emit end element tags. The number of XML end element tags necessary will be equal to the difference between the levels of the current and next nodes.

The routine finishes up by taking care of the very last node in the ArrayList of assembly Info objects. Because it is the last node, it will be a leaf node (no children). You can emit the node as XML with start and end element tags. Now there could be pending XML end element tags. The routine keeps track of this with the local variable pendingEndTags, and you emit that number of end element tags if required. The output from SaveReferencedAssemblies is illustrated in Figure 1.

Note that the techniques I've demonstrated up to this point are based on metadata. Only assemblies that are statically referenced will be found by these traversal techniques. So, for example, if a configuration file stored the name or location of an assembly to be loaded at run time and used in a late-bound fashion, these techniques would not account for those assemblies.

.NET Method Dependencies

Now I'll show you a quick, automation-friendly, flexible, and easy way to determine .NET method dependencies. Knowing what methods are called by your system allows you to create meaningful test cases, and also allows you to identify "dead code"—methods that are not called at all. Manually reading through source code is usually not a viable option because in any realistic system there will just be too much code. My goal here is to list all the methods in a .NET assembly unit, and for each method list the methods it calls. At first thought this would seem to be a pretty easy task. I wanted to produce an especially lightweight (less than one page of code) and easily customizable tool.

One obvious approach you might try is to parse through the source code of the system under test. But if you've ever tried to do this, you'll know it is very tough. Another approach is to attempt to write a tool that directly parses the binary assembly under test file. This turns out to be very, very difficult. Yet another approach several of my colleagues have tried is to use the ildasm.exe tool, point it to the system under test, instruct it to produce text file output using the /OUT command-line switch, and then parse through the resulting intermediate language (IL). This is possible but unfortunately the text output from ildasm.exe was not designed for automation and can be difficult to parse.

There is another way. We could manually parse through the IL stored in the assembly, but, as noted, that would be difficult.

However, others have already done this task for us. A great way to achieve easy method dependencies output is the elegant .NET Reflector tool, written by Lutz Roeder. It is available at www.aisto.com/roeder/dotnet. The Reflector tool is widely known and used. In fact, reflector.exe and ildasm.exe are two of the best-known GUI tools for .NET system discovery. What is not well-known or well-documented is that Reflector contains a great API set that dramatically simplifies working with .NET assemblies. The code that produced the output shown in Figure 2 is listed in Figure 7.

Figure 7 Finding Method Dependencies with Reflector (Close Figure 7)
static void Main(string[] args)
{
    ArrayList listMethods = new ArrayList();
    try
    {
        Console.WriteLine("\nStart method reference analysis\n");
        IServiceProvider serviceProvider = new Application(null);
        IAssemblyLoader assemblyLoader = (IAssemblyLoader)
            serviceProvider.GetService(typeof(IAssemblyLoader));
        IAssembly rAssembly =
            assemblyLoader.LoadFile("..\\..\\..\\CombinationsLib.dll");
        Console.WriteLine("Assembly: " + rAssembly.ToString());

        foreach (IModule module in rAssembly.Modules)
        {
            foreach (ITypeDeclaration typeDeclaration in module.Types)
            {
                foreach (IMethodDeclaration methodDeclaration in
                    typeDeclaration.Methods)
                {
                    Console.WriteLine("Method: " + methodDeclaration);
                    IMethodBody body = methodDeclaration as IMethodBody;
                    if (body != null)
                    {
                        foreach (IInstruction inst in body.Instructions)
                        {
                            switch(inst.Code.ToString())
                            {
                                case "call":
                                case "callvirt":
                                case "calli":
                                case "newobj":
                                Console.WriteLine("  Calls to: " +
                                    inst.Operand.ToString());
                            }
                        }
                    }
                }
            }
        }

        assemblyLoader.Unload(rAssembly);
    }
    catch(Exception ex)
    {
        Console.WriteLine("Fatal error: " + ex.Message);
    }
    Console.WriteLine("\nDone");
}

The program in Figure 7 analyzes a .NET class library named CombinationsLib.dll. The associated using statements are:

using System;
using System.IO;
using System.Reflection;
using System.Collections;
using Reflector;
using Reflector.CodeModel;
The first four of these six namespaces are part of the .NET Framework and should be familiar. The Reflector and Reflector.CodeModel namespaces are part of the Reflector API. To call the API you must download the reflector.exe file from Lutz's Web site and add a reference to it. Visual Studio® 2005 allows you to add references to .exe files. Unfortunately, Visual Studio .NET 2003 does not. If you're using Visual Studio .NET 2003, you can rename the file to reflector.dll and then add a reference to it. An alternative is to use the command-line compiler csc.exe with the /reference switch set to reflector.exe; this will work with both the .NET Framework 1.x and the .NET Framework 2.0.

The code in Figure 7 is straightforward. You instantiate a Reflector IServiceProvider interface, then use that interface to instantiate an IAssemblyLoader object. Next call the LoadFile method to load an assembly for use by the Reflector API. Notice that LoadFile returns a Reflector IAssembly rather than a System.Reflection.Assembly object. Once the Reflector assembly has been loaded into memory, you can iterate through the assembly using the Reflector.CodeModel Modules, Types, Methods, and Instructions collections. The Reflector.CodeModel IInstruction objects are direct analogs of intermediate language instructions. Here are some Reflector-style IL instructions:

L0021: conv.i8
L0022: beq.s 47
L0024: ldstr Array length does not equal k
L0029: newobj Exception..ctor(String) : Void
L002E: throw
L002F: ldarg.0
L0030: ldarg.1
L0031: stfld Combination.n : Int64
L0036: ldarg.0

Each Reflector instruction includes an address followed by the colon character, an operator/instruction like "ldstr" (load string) or "call" (call a method), and an optional list of operands/arguments. Informally, when the term "IL instruction" is used, it often means either an opcode like 0x3B or its assembly language equivalent (bre—branch if equal), either with or without an operand. The Reflector API uses a slightly different terminology. If the variable instruction's ToString results in

L0123: ldstr "Hello"
then in Reflector terminology, instruction.Code.ToString is "ldstr" and instruction.Operand.ToString is "Hello".

Now because you are looking for .NET method calls, you need to identify exactly which IL instructions make method calls. This depends on exactly how you define a method call. The ECMA Common Language Infrastructure (CLI) specification document says in section 12.4.1.2 that IL, also called Common Intermediate Language (CIL) or Microsoft intermediate language (MSIL), has three call instructions that are used to transfer argument values to a destination method:

call Used when the destination address is fixed at the time the CIL is linked.

calli Used when the destination address is calculated at run time.

callvirt Uses the class of an object, known only at run time, to determine the method to be called.

If you examine the code in Figure 7 you'll see that I check for these three instructions. I also check for the "newobj" IL instruction, which calls a constructor. Whether or not you consider calls to constructors and properties as method calls depends on what you mean by a method. One of the benefits of lightweight tools like this one is that you can modify them to meet your own needs.

Lutz's slick Reflector code does all the work. Behind the scenes, the Reflector API code disassembles raw .NET assemblies, peeling away all file header information, and translates the remaining binary information to human-readable format. In theory, you could do this from scratch by using the information in the ECMA CLI specification document, but it is an awful lot of work.

You can extend my method dependencies tool in several ways. One interesting possibility is to modify it so that it not only keeps track of called methods, but also called-by methods. Testers are usually most concerned with which methods are called by a particular method so they can create meaningful test cases. But developers often want to know which methods call a particular method.

A second modification I've seen is a call tree structure very similar to the assembly references tree structure. The tool as presented just lists one level of method calls. But you can easily produce multilevel method call dependencies.

Comments and Discussion

The two lightweight tools I've presented here are easily modifiable and automation-friendly, but don't neglect GUI-based tools. Let me call your attention to the terrific CLR Profiler 2.0 tool. The CLR Profiler tool gives a very powerful way to analyze .NET assemblies during their run time. Figure 8 shows CLR Profiler output.

Figure 8 CLR Profiler Output
Figure 8 CLR Profiler Output

The .NET assembly dependencies tool and the .NET method dependencies tool I've presented are often used by some of my colleagues at Microsoft as the basis for interview questions. When faced with a tree traversal problem in a hiring interview, new college graduates tend to immediately jump to a recursive solution. Interviewers are often looking for at least a mention of the possible problems with a recursive solution (memory allocation issues, debugging difficulty, and so forth) and a mention of a nonrecursive solution. SDET interview questions at Microsoft rarely have one correct answer—most coding questions are chosen so that they have multiple approaches, each with pros and cons. Just like coding problems in a real production environment.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com. Thanks to Lutz Roeder for his help with this column.


Concurrent Affairs: Build a Richer Thread Synchronization Lock

Concurrent Affairs
Build a Richer Thread Synchronization Lock
Jeffrey Richter

Code download available at: ConcurrentAffairs0603.exe (115KB)
Contents
What Is an Optex?
Synchronization Fairness
Avoid Convoys: Building a Better Optex
Critical Regions

In my last column, I showed the various thread synchronization mechanisms employed by the Microsoft® .NET Framework (see Concurrent Affairs: Performance-Conscious Thread Synchronization). I then examined the performance characteristics of all these mechanisms and determined that the Interlocked methods performed the best because the calling thread never has to transition to kernel mode. I then looked at how to build some simple locks using the Interlocked methods.

While the Interlocked methods are fast, they are, unfortunately, very limited—they can only manipulate 32-bit and 64-bit values atomically. Frequently, an object may have several fields in it that you need to modify as a single unit atomically. While you cannot use the Interlocked methods to modify multiple fields as a single atomic unit, you can leverage the Interlocked methods to build a more sophisticated thread synchronization mechanism. Here, I'll build a thread synchronization lock, which I call an Optex (Optimized mutual exclusive lock).

What Is an Optex?

An Optex works much like a Win32 CRITICAL_SECTION or the common language runtime (CLR) System.Threading.Monitor class. The lock grants one thread at a time access to the resource that is logically guarded by the Optex. However, unlike a CRITICAL_SECTION and the Monitor class, my Optex lock doesn't support recursion, so a thread cannot acquire the lock multiple times without releasing the lock. This lack of recursion, however, does allow one thread to acquire the lock and a different thread to release the lock—a useful feature in some scenarios.

As you'll see, my Optex lock executes really fast when there is no contention on the lock. In fact, when there is no contention, the thread attempting to acquire the lock will execute some quick field assignments and return; the thread will execute entirely in user mode. The thread will not transition into kernel mode and it will not wait on a kernel object. On the other hand, when there is contention for the lock, the lock becomes much slower because threads will transition to kernel mode and they will wait on a semaphore kernel object. Also, this lock will be pretty equitable, waking threads that wait on the lock using a FIFO algorithm. However, this fair and equitable approach is not guaranteed. Figure 1 shows the code for my Optex lock, and Figure 2 demonstrates how to define a class that uses an Optex object in order to allow only one thread at a time access to any instance of the SomeResource class.

Figure 2 Define Class That Uses Optex (Close Figure 2)
public class SomeResource {
   private Optex m_optex = new Optex();

   public void AccessResource() {
      try {
         m_optex.Enter();
         // Access the resource...
      }
      finally {
         m_optex.Exit();
      }
   }
}
Figure 1 Optex Lock (Close Figure 1)
public sealed class Optex : IDisposable {
   private Int32 m_Waiters = 0;
   private Semaphore m_WaiterLock = new Semaphore(0, Int32.MaxValue);

   public Optex() { }
   public void Dispose() {
      if (m_WaiterLock != null)
      {
         m_WaiterLock.Close();
         m_WaiterLock = null;
      }
   }

   public void Enter() {
      Thread.BeginCriticalRegion();
      // Add ourself to the set of threads interested in the Optex
      if (Interlocked.Increment(ref m_Waiters) == 1) {
         // If we were the first thread to show interest, we got it.
         return;
      }

      // Another thread has the Optex, we need to wait for it
      m_WaiterLock.WaitOne();
      // When WaitOne returns, this thread now has the Optex
   }

   public void Exit() {
      // Subtract ourself from the set of threads interested in the Optex
      if (Interlocked.Decrement(ref m_Waiters) > 0) {
         // Other threads are waiting, wake 1 of them
         m_WaiterLock.Release(1);
      }
      Thread.EndCriticalRegion();
   }
}

As you can see, an Optex has just two data fields inside of it: m_Waiters and m_WaiterLock. The m_Waiters field is initialized to zero indicating that no threads own the lock.

When a thread comes along and calls Enter, Interlocked.Increment is used to add 1 to the m_Waiters field. If m_Waiters is incremented to 1, then it was 0 before. Therefore, the calling thread can own the lock and Enter returns immediately. In this case Enter executes very quickly. On the other hand, if m_Waiters is incremented to a value greater than 1, then other threads are trying to enter or have entered the Optex, and the Optex waits on the semaphore. In this case, Enter is slow because the thread transitions to kernel mode. But this isn't really a problem because the thread needs to wait anyway until the lock is available.

Optex's Exit method is fairly straightforward. First, a thread never sleeps or enters a wait state when releasing a lock. When Exit is called, it subtracts one from m_Waiters. If the result becomes zero, then no other threads are waiting on the Optex and Exit simply returns; here performance is very good. If, however, decrementing m_Waiters results in a value that is greater than zero, then at least one other thread is waiting for the lock via the semaphore. In this case, the Exit method releases one of these waiting threads by signaling the semaphore and then returns immediately. Here performance is good but not great because the thread that's calling Exit must transition from user mode to kernel mode and back to signal the semaphore.

The Optex class is very simple and yet quite effective. It would be easy to add additional features to Optex if you so desired. For example, you could add a TryEnter method which would allow a thread to do something else if the lock is not available. TryEnter could even accept a timeout value instead of just assuming zero. It would also be possible to modify the Enter method: if contention is detected, the thread could spin in user mode several times trying to get the lock before transitioning to kernel mode to wait on the semaphore. This could improve performance if a thread happens to call Enter at just the wrong time.

In addition, you could easily add recursion support so that a single thread could acquire the lock multiple times without deadlocking, making the lock behave exactly like a Win32 CRITICAL_SECTION or the CLR Monitor class. The possibilities are endless.

Synchronization Fairness

In Windows®, threads waiting on a kernel object (mutex, semaphore, event, file handle, and so forth) are, for the most part, processed fairly. This means that when the kernel object becomes signaled, Windows wakes up the waiting threads using a FIFO algorithm. This algorithm, by the way, does not take thread priority into account. So, if a lower-priority thread starts waiting on a mutex and then a higher-priority thread later waits on the same mutex, Windows will wake the lower-priority thread since it started waiting first.

I say that waiting is mostly fair because there are times when this fairness is violated. For example, when debugging a process, breakpoints cause all threads in the process to be suspended. If any of these threads happen to be waiting on a thread synchronization object, then Windows pulls these threads out of the FIFO queue so they are not waiting on the thread synchronization object anymore. When the execution resumes after the breakpoint, the threads are resumed and they race back to re-enter the FIFO queue. But, these threads enter the FIFO queue from the back like all other threads, so they lose their original position in line. Furthermore, if two threads were waiting and they both race back to the queue, they may get back into the queue in a different order than they were in before.

In Windows there are other ways that waiting threads can be prevented from being processed in a FIFO fashion. For example, waiting threads can be prematurely awakened due to asynchronous procedure calls (APCs), especially if the threads are in an alertable state. It turns out that the CLR makes fairness even less likely to occur. The culprit of this added unfairness is the garbage collector.

When the CLR wants to start a garbage collection, it will determine which threads are currently executing managed code and which are currently executing unmanaged code. After making this determination, the CLR will suspend the threads that are executing managed code. Threads that are currently executing unmanaged code will hijack themselves when they attempt to return back to managed code. It doesn't happen frequently, but every once in a rare while there is a small window of time in which a thread meets these conditions: it is in managed code, the CLR thinks it needs to suspend this thread, the thread calls into the unmanaged Win32 WaitForSingleObject or WaitForMultipleObjects functions, and the CLR suspends the thread while in one of these functions.

In addition, the CLR always puts waiting threads in an alertable state, which makes it even more likely that the threads can be awakened and pulled out of the queue. The CLR does this to handle APIs like Thread's Interrupt method. The point of all this is to say that locks in the CLR and in Windows are not fair. And, in fact, it is entirely possible (but very unlikely) that a thread waiting on a lock will not get it at all if there is a steady stream of threads that are attempting to enter the lock, especially if these threads are also allocating objects causing frequent garbage collections to occur.

Fortunately, most applications access locks infrequently so fairness is not an issue. In fact, most applications will run fine without fair thread synchronization, but at least you should be aware of this issue.

Avoid Convoys: Building a Better Optex

As it turns out, trying to have a lock treat its waiting threads fairly causes a problem known as the convoy problem. Simply stated, the convoy problem results when the lock remains in contention for a long time, causing waiting threads to constantly transition to kernel mode. Here the locks exhibits worse performance than if you had just used a kernel object (such as a mutex) in the first place. Since the Optex class shown earlier tries to treat its waiting treads fairly, it can cause the convoy problem. Now, let me explain the convoy problem in more detail.

You'll notice that when an Optex lock is under contention, the lock never becomes free. That is, the m_Waiters field never goes to zero (indicating that the lock is free) until there are no threads waiting on the lock. So, when a thread calls Enter, if the lock is already owned, the calling thread transitions to kernel mode to wait. When the thread that owns the lock calls Exit, it detects that another thread is waiting and it also transitions to kernel mode to set the semaphore. Unfortunately, the way that the Optex class is currently written, the thread leaving the lock actually owns the lock longer than necessary because it transitions to kernel mode to release the semaphore. And, releasing the semaphore causes Windows to do a thread reschedule in order to context-switch to another thread. All of this—holding the lock longer, rescheduling, context switching—ends up hurting performance significantly. It also means that the lock tends to remain under contention for a longer time and this means that threads calling Enter continuously call WaitOne, causing them to transition to kernel mode. Since the lock rarely becomes free, threads keep transitioning to kernel mode and the lock performs worse than if you used a kernel object to begin with.

Figure 3 shows a new version of the Optex class that fixes the convoy problem and improves performance. In this new version, the m_LockState field contains two pieces of information: whether the lock is free or owned (bit 0) and the number of threads waiting on the lock (bits 1 through 31). I also defined a few constants in the Optex class to help manipulate the m_LockState field. Naturally, the bits of this Int32 field will be manipulated using various interlocked methods. In fact, I needed to define a few helper methods (using the techniques discussed in my previous column) such as InterlockedAdd, InterlockedOr, and IfThen to make the code easier to write and more maintainable. You can see these methods implemented towards the bottom of the Optex class.

Figure 3 Optex.cs (Close Figure 3)
using System;
using System.Diagnostics;
using System.Threading;

namespace MsdnMag.Threading {
   public sealed class Optex : IDisposable {
      // Bit     0: 0=Lock is free, 1=Lock is owned
      // Bits 1-31: Number of waiters
      private Int32 m_LockState = c_lsFree;

      private const Int32 c_lsFree  = 0x00000000;
      private const Int32 c_lsOwned = 0x00000001;
      private const Int32 c_1Waiter = 0x00000002;

      private Semaphore m_WaiterLock = new Semaphore(0, Int32.MaxValue);

      public Optex() { }
      public void Dispose() {
         if (m_WaiterLock != null)
         {
            m_WaiterLock.Close();
            m_WaiterLock = null;
         }
      }

      public void Enter() {
         Thread.BeginCriticalRegion();
         while (true) {
            // Turn on the "owned" bit
            Int32 ls = InterlockedOr(ref m_LockState, c_lsOwned);

            // If lock was free, this thread got it, return
            if ((ls & c_lsOwned) == c_lsFree) return;

            // Another thread owned the lock, add 1 waiter
            if (IfThen(ref m_LockState, ls, ls + c_1Waiter)) {
               // If successfully added 1, wait for lock
               m_WaiterLock.WaitOne();
            }
            // We weren't able to add 1 waiter or waiter woke,
            // attempt to get the lock
         }
      }

      public void Exit() {
         // Pre-condition:  Lock's state must be Owned
         // Post-condition: Lock's state must become Free
         //                 (the lock is never passed)

         // Phase 1: Free the lock
         Int32 ls = InterlockedAnd(ref m_LockState, ~c_lsOwned);
         if (ls == c_lsOwned) {
            // If no waiters, nothing to do, we can just return
         } else {
            // Phase 2: Possibly wake waiters
            // If lock is free, try to subtract 1 from the
            // number of waiters
            ls &= ~c_lsOwned;
            if (IfThen(ref m_LockState, ls & ~c_lsOwned, ls - c_1Waiter))
            {
               // We sucessfully subtracted 1, wake 1 waiter
               m_WaiterLock.Release(1);
            } else {
               // Lock's state changed by other thread,
               // other thread will deal with it
            }
         }
         Thread.EndCriticalRegion();
      }

      #region Interlocked Helpers
      private static Int32 InterlockedAnd(ref Int32 target, Int32 with) {
         Int32 i, j = target;
         do {
            i = j;
            j = Interlocked.CompareExchange(ref target, i & with, i);
         } while (i != j);
         return j;
      }

      private static Int32 InterlockedOr(ref Int32 target, Int32 with) {
         Int32 i, j = target;
         do {
            i = j;
            j = Interlocked.CompareExchange(ref target, i | with, i);
         } while (i != j);
         return j;
      }

      private static Boolean IfThen(ref Int32 val, Int32 @if, Int32 then)
      {
         return (Interlocked.CompareExchange(ref val, @then, @if) == @if);
      }

      private static Boolean IfThen(ref Int32 val, Int32 @if,
            Int32 then, out Int32 prevVal) {
         prevVal = Interlocked.CompareExchange(ref val, @then, @if);
         return (prevVal == @if);
      }
      #endregion
   }
}

This version of the Enter method contains a while loop that continues until the calling thread acquires the lock. Inside the loop, InterlockedOr is called to make sure the lock is taken by any thread. If InterlockedOr returns c_lsFree (0), then the lock was previously unowned, the calling thread acquired it, and it can return from Enter immediately; the result is very good performance.

If InterlockedOr doesn't return c_lsFree, then another thread owns the lock and the calling thread must increment the count of waiting threads and then wait on the semaphore. When this thread wakes, it will loop around and attempt to acquire the lock just like any other thread. This is a bit inefficient and has two potential problems, which I'll address shortly. For now, let's look at the new version of the Exit method.

To avoid the convoy problem, you'll want the Exit method to release the lock entirely in user mode. Then, if another thread comes along, it can acquire the lock without contention and without transitioning to kernel mode, thereby executing with great performance. However, another thread could have tried to enter the lock earlier, had contention, and gone into kernel mode to wait until the lock was released. This thread does need to be awakened but I'm going to do this differently than in the previous version of the Optex class.

When Exit is called you know that the lock must be owned (m_LockState must have the c_lsOwned bit set) and you want the lock to be free before the method returns. The first thing Exit does is call InterlockedAnd to turn off bit 0 making the lock free. If InterlockedAnd returns c_lsOwned, then this thread owned the lock and no other threads are waiting on the lock. We expect this no-contention situation to be very common, and in this case Exit simply returns; again, performance is very good.

If InterlockedAnd doesn't return c_lsOwned, it means that some other bits are turned on—indicating that 1 or more threads are waiting on the lock. Now, if the lock is still free, I will try to decrement the count of waiting threads by 1 (by calling my IfThen method subtracting c_1Waiter from the count). If Exit successfully subtracts one waiter, then I'll wake up one waiting thread by releasing the semaphore. If the call to IfThen fails to subtract 1 from the count of waiting threads, then another thread must have snuck in and taken the lock; therefore, I will not wake the waiting thread. The thread that snuck in will wake the thread when it calls Exit to leave the lock.

Now, let me address the two problems I spoke of earlier. The first problem is that this lock is unfair because it is possible that a waiting thread got awakened, but the lock was given to another thread that snuck in by calling Enter. However, I discussed fairness in the previous section and in most situations this unfairness is not an issue worth worrying about. The second problem is that Exit woke up a waiting thread which looped around and then simply went back to waiting on the lock again. This, of course, hurts performance. However, it's all a numbers game. For most applications, the performance of this technique—which eliminates the convoy problem—outperforms the other technique of passing the lock from thread to thread. It is unlikely that the awakened thread will miss the lock and have to go back to wait on it again; usually, the awakened thread will get the lock, improving performance. The big performance win, of course, is that most threads that call Enter will not see contention on the lock and they will not transition to kernel mode.

In all my testing, this new version of the Optex class definitely outperforms the previous version and I would never use the first version in any production code. By the way, the Monitor class's Enter and Exit methods are implemented to be unfair in order to avoid the convoy problem. The result is improved performance, so feel free to use the Monitor Enter/Exit methods in your own code.

Critical Regions

You'll notice that in my SpinWaitLock code in my last column and in the two versions of the Optex code here, I have all the Enter methods call Thread's BeginCriticalRegion method and I have Exit call Thread's EndCriticalRegion method. Let me explain what these calls are about.

The Microsoft SQL Server™ team really needed the CLR to be much more robust and reliable than versions 1.0 or 1.1. Towards this end, version 2.0 now supports critical regions. A critical region is a block of code that might be dependent on state shared between multiple threads. Certainly, the fields of a thread synchronization lock object qualify as shared state. The resource that is being protected by the lock itself would be considered shared state as well.

The CLR would like to know when a thread has entered into a critical region in case an unhandled exception is thrown from within the region. You see, if an unhandled exception occurs, then it is likely that state shared between threads is corrupt, and allowing other threads in the AppDomain to run and access the shared state is just asking for trouble. Usually a host process (like SQL Server) will want to know that an unhandled exception occurred in a critical region and the host will unload the entire AppDomain.

Code can notify the CLR that a critical section is starting by calling Thread's BeginCriticalRegion method. Internally, all this method does is increment a counter associated with the calling thread by 1. Code notifies the CLR that it is leaving a critical region by calling Thread's EndCriticalRegion method, which internally decrements the thread's counter by 1. A host application (such as SQL Server) can determine when a thread executing in an AppDomain has experienced an unhandled exception. The host can then determine if the thread was in the process of accessing shared state (inside a critical region) and the host can decide how to handle this situation. For example, the host could unload the AppDomain (along with all its state) or the host could shut down the CLR entirely within the host's process.

Microsoft has updated the Monitor and Mutex classes to call BeginCriticalRegion and EndCriticalRegion appropriately. However, if you are building your own locks as I've done in this column, then you must call the methods manually. Or, if you're accessing shared state without locks, then you should be calling these methods manually as well in order to properly support hosts such as SQL Server.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), a training and consulting firm. He is the author of several books, including Applied Microsoft .NET Framework Programming (Microsoft Press, 2002). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


Bugslayer: Strengthening Visual Studio Unit Tests

Bugslayer
Strengthening Visual Studio Unit Tests
John Robbins

Code download available at: Bugslayer0603.exe (169KB)
Contents
Watch Out for Hardcoded Paths
Working with GenericTest and EXEs
Create Unit Test...
Using NUnit
TimeOutAttribute
Code for You
Names and Places
A Better MSTEST.EXE
Wrap Up

Visual Studio 2005 brought so many new features to the table that it can seem almost overwhelming. One of the most exciting additions is the new unit testing features found in the Test menu on the main menu bar. While we've had some excellent unit testing tools in .NET Framework-based development for a while, these new unit testing features provide clean integration and an emphasis that was missing before.

Nevertheless, the unit testing tools do have a few rough spots. Here, I will discuss the pitfalls and problems I've encountered while working with the new testing tools. I've included one of the real modules I've been working on along with its unit test so you can have a useful example at hand. Note that all of these problems are fixable, and you can confidently implement these testing tools immediately in your development shops.

While using the unit testing tools, I've noticed a couple areas where extending the environment was necessary, so I've provided the code I used to make my testing easier. With this added help, the testing features now work the way I think they should.

For this column, I'm assuming you've read the "Writing Quality Code" and "Working with Unit Tests" sections of the documentation. Those papers provide a good overview, and I'll cover the practical aspects here. If you've got the Visual Studio® Team System or Team Edition for Developers, you'll also have the tools geared for testers. The Team Edition for Testers documentation covers those tools and has the all-important Class Library documentation for extending or interacting with the testing environment.

Watch Out for Hardcoded Paths

Adding tests to a new project is fantastically easy using the Unit Test Wizard. This handy feature can save you hundreds of hours of typing time (the developers at Microsoft deserve a lot of credit for this). However, something happens behind the scenes that can cause you a lot of grief: paths are hardcoded! This is a problem, for example, when you move a test to another machine or directory. I hope that a service pack for Visual Studio 2005 will allow you to set a starting path for tests and the hardcoded paths will use relative paths. For now, however, the easy way to solve the hardcoded path problem is to have a benevolent dictator on the team decree all drives and paths for all time. If it's a small enough project, this solution may be fine. Another, more practical workaround is to reference the %testdeploymentdir% environment variable in your paths. It is set when a test runs.

The first place where hardcoded paths appear is in the VSMDI file, which is a big wrapper around simple lists of tests. When you open a VSMDI file and it can't find the test assemblies or TESTRUNCONFIG files, you're prompted for locations for those items. What I found interesting was that the next time I opened that same VSMDI file, it found everything. Obviously, the updated paths had to have been stored somewhere, but the VSMDI file itself was never changed. I found a hidden file with the name <file name>.VSMDI.OPTIONS in the directory where the VSMDI file resides.

When I opened the VSMDI.OPTIONS file, which is a ubiquitous XML file, I shook my head in frustration. As you can see in Figure 1, there is obviously support for path searching with VSMDI files, but there's no way in the Visual Studio user interface to set the search paths. (Furthermore, there's no reason for these VSMDI files to be hidden.) So, when I want to use the VSMDI file on a different directory structure, I copy an example VSMDI.OPTIONS file to the appropriate directory and manually edit it, adding the paths to the assemblies and TESTRUNCONFIG files. I've included an example file in the BaseVSMDIOPTIONSFile directory for this column's source code.

Figure 1 VSMDI.OPTIONS File (Close Figure 1)
<?xml version="1.0" encoding="utf-8"?>

<Tests>
  <edtdocversion branch="retail" build="50727" revision="42" />
  <SearchPaths
      type="Microsoft.VisualStudio.TestTools.Common.SearchPaths">
    <TestSearchPath type="System.Collections.ArrayList">
      <element type="System.String">
        C:\Dev\Column\39\SourceCode\Debug
      </element>
    </TestSearchPath>
    <RunConfigSearchPath type="System.Collections.ArrayList">
      <element type="System.String">
        C:\Tests\Bugslayer.Utility.Tests
      </element>
    </RunConfigSearchPath>
  </SearchPaths>
</Tests>

The main reason for using a VSMDI file is so you can specify lists of tests to run—this is useful during active development as it allows you to run just the specific tests related to the bits of code you're working on. To execute all the tests in a test assembly, you can use the console test runner, MSTEST.EXE. There are numerous command-line options, but you'll only need the /testcontainer: option, which specifies the assembly containing the tests. Optionally, you can use /resultsfile to specify the name of the results file. When using MSTEST.EXE, you don't have to worry about the VSMDI files. In the next section, I'll discuss an MSBuild.EXE task I put together to automatically run all appropriate tests found in a directory structure so you can avoid using VSMDI files altogether.

The second place you'll find hardcoded paths is in the TESTRUNCONFIG files; the assemblies to instrument and strong key files all have hardcoded paths. The best solution I've figured out to address this issue is to use a different TESTRUNCONFIG file when you run tests on different directory structures. As there's no way to create a new TESTRUNCONFIG, you'll need to copy one of your existing files over to the machine location and edit it with Visual Studio. If the drives and upper directories are different, the IDE and MSTEST.EXE will handle TESTRUNCONFIG files with relative paths but, again, you'll have to edit them by hand.

As it's not the testing tools themselves doing the code coverage instrumentation, there's nothing stopping you from manually performing the code coverage for your compiled assemblies yourself from the command line. If you're working on ASP.NET bits, you'll need to do code coverage from the Visual Studio IDE to create the appropriate assembly on the fly and get it instrumented.

Doing manual code coverage is a three-step process. The first is to instrument your assemblies so they have the code coverage hooks in them. The second is to start the monitor process and tell it where to write the coverage file. Any instrumented binaries that load while the monitor process is running will have their coverage data added to the output file. The final step is to shut down the monitor to write the COVERAGE file. I have whipped up a Coverage.Targets file you can use with MSBuild to automate this process. Figure 2 shows pieces of the file; the full file is available for download from the MSDN®Magazine Web site.

Figure 2 Coverage.Targets (Close Figure 2)
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <!-- The task to wrap VSPerfMon.EXE since it has trouble with
         console I/O redirection.-->
    <UsingTask TaskName="VSPerfMonTask" AssemblyFile=
        "$(BUGSLAYERBUILDTASKSDIR)\Bugslayer.Build.Tasks.DLL" />

    <!-- **************** Properties Used By All ******************* -->
    <!-- These are here in case you need to set them from the command
         line. -->
    <PropertyGroup>
        <BASEDIR>$(VSINSTALLDIR)\Team Tools\Performance Tools\</BASEDIR>
        <VSINSTR>&quot;$(BASEDIR)VSINSTR.EXE&quot;</VSINSTR>
        <VSPERFCMD>&quot;$(BASEDIR)VSPERFCMD.EXE&quot;</VSPERFCMD>
    </PropertyGroup>

    <!-- ************* Instrument and Monitor Target *************** -->
    <Target Name=»InstrumentAndMonitor» DependsOnTargets=
        «CodeCoverageInstrumentTarget;StartCoverageMonitorTarget» />

    <!-- **************** Instrumentation Target ******************* -->
    <Choose>
        <When Condition=»$(StrongNameFile) != ''»>
            <PropertyGroup>
                <OffVsInstrWarnings>2001;2013</OffVsInstrWarnings>
            </PropertyGroup>
        </When>
        <Otherwise>
            <PropertyGroup>
                <OffVsInstrWarnings>2013</OffVsInstrWarnings>
            </PropertyGroup>
        </Otherwise>
    </Choose>

    <!-- The target for instrumenting. -->
    <Target Name=»CodeCoverageInstrumentTarget»
            Condition="'$(VSINSTALLDIR)' != ''"
            Inputs="@(InputCoverageBinaries)"
            Outputs="@(OutputCoverageBinaries->
                     $(OutputDir)\%(filename)%(extension)">
        <!-- Make the output directory if it doesn't exist. -->
        <MakeDir Condition =»!Exists('$(OutputDir)')»
                 Directories =»$(OutputDir)» />
        <!-- Run VSINSTR.EXE on the individual binary. -->
        <Exec Command=»$(VSINSTR) /NOWARN:$(OffVsInstrWarnings)
                       /COVERAGE /OUTPUTPATH:$(OutputDir)
                       %(InputCoverageBinaries.Identity)»/>
        <!-- If a strong name key file was specified, run SN.EXE to
             resign it.-->
        <Exec Condition=»$(StrongNameFile) != ''»
              Command=»sn -q -R $(OutputDir)\%
                       (InputCoverageBinaries.filename)%
                       (InputCoverageBinaries.extension)
                       $(StrongNameFile)»/>
    </Target>

    <!-- ***************** Start Monitor Target ******************** -->
    <Target Name=»StartCoverageMonitorTarget»
            Condition="'$(VSINSTALLDIR)' != ''">
        <VSPerfMonTask OutputFile=»$(OutputCoverageFile)»
            MonitorType=»Coverage» User=»$(User)»/>
    </Target>

    <!-- ***************** Stop Monitor Target ********************* -->
    <Target Name=»StopCoverageMonitorTarget»
            Condition=»'$(VSINSTALLDIR)' != ''»>
        <Exec Command=»$(VSPERFCMD) /SHUTDOWN»/>
    </Target>
</Project>

As you poke through Figure 2, you'll see that I wrote a task in Bugslayer.Build.Tasks.DLL to run the monitoring process. Figure 3 shows it in Visual Studio.

Figure 3 Visual Studio Editor (Close Figure 3)

When I first wrote Coverage.TARGETS, I used the Exec task to execute VSPERFCMD /START:Coverage /OUTPUT:$(OutputCoverageFile) but there was a severe problem in doing that: MSBuild.EXE completely hung on the call. VSPERFCMD.EXE spawns off the actual monitor process, VSPERFMON.EXE. If you run VSPERFMON.EXE from a command prompt, the process sits there spitting out connections and other information active in the process, so you can't just call it directly from the project file.

The problem turns out to be in MSBuild.EXE, and stems from the fact that VSPERFMON.EXE is spawned from the VSPERFCMD.EXE process with the bInheritHandles flag to CreateProcess set to true. Any process started with inherit handles will hang under MSBuild.EXE. Consequently, I have to call Process.Start from the ITask.Execute method in my task to make everything happy under MSBuild.EXE.

Working with GenericTest and EXEs

If you've got an existing test system based upon EXE programs, the discussion of the GenericTest type in the documentation probably piqued your curiosity. While working on a project that relied on a batch file that ran nine EXEs as the unit test, I was able to use the GenericTest type to quickly wrap some of my existing code in automation goodness. There were a few catches though. The first small obstacle was that GenericTest allows zero as a successful return value from the EXE. That's not too big a deal, but considering the advanced features in GenericTest, I'm dismayed to see something as simple as a field for acceptable exit codes was left out.

The bigger problem with GenericTest is that it's a bastion of hardcoding. Fortunately, it's relatively easy to figure out the relative path location. If your GenericTest resides in C:\FOO, the test is actually started from C:\FOO\TestResults\<User>_<Machine>_<TimeStamp>\Out. Thus, if your EXE file executed by GenericTest is in C:\FOO as well, you can use .\.\.\<name>.EXE as the program to execute. Unfortunately, nearly everything else in GenericTest is hardcoded from the drive on down. What's interesting is that the Out directory is where all your binaries are copied to each time they run. That way, even if you change the code, you can rerun the test's previous versions easily in order to reproduce any problems.

A handy feature, the GenericTest type will capture anything that goes to standard output, giving you a log of the run in the result file. Unfortunately, there currently seems to be a problem with the capture where pumping a lot of information will cause the test driver process to hang. But most test applications aren't pumping out 100 plus lines of output in a few milliseconds.

Create Unit Test...

When it comes to testing, the real magic of Visual Studio is the wonderful Create Unit Test... option you get when right-clicking on a method in the text editor. The feature works well and makes it easy to quickly add unit tests. But I have a small philosophical problem with the fact that it lets you create unit tests that reach directly into the class and access private methods.

The main argument for allowing the testing tool to call private or protected methods directly is that it eases testing (there is less code to write) and helps broaden code coverage. These arguments are seductive, but I fall in the camp that thinks unit testing should only come through the public interfaces. The unit test is the first use of the code and you want to gear the testing towards how others will use it. If there are private methods that you can't sufficiently test without short circuiting and calling them directly, I have to wonder if the code might need to be redesigned. To keep from accidentally creating a unit test that directly calls a private method, go to the Create Unit Test dialog, click on the Filter dropdown in the upper-right corner, and uncheck Display Non-public Items.

By no means am I an absolutist. I'm sure there are cases where it would help immensely to call private methods. However, just because the tool allows you to do something doesn't mean you should rely on it. Unit testing is the first stage of testing and it's the first place where you start the white box testing.

Using NUnit

I have projects where we've already made a major investment in a test system built out of NUnit. (A new version that works with the .NET Framework 2.0 was recently released.) In one case, we wanted the code to be portable between NUnit and the Visual Studio test systems, giving us the best of both worlds. When planning this, I stumbled upon something extremely cool that would require minimal code changes exactly once and allow the code to work with both NUnit and Visual Studio.

The Microsoft patterns & practices group has released the very interesting Composite UI Application Block. They include their unit tests in the installation and, when reading through the code, I noticed this brilliant bit of code at the top of all the tests:

#if !NUNIT
using Microsoft.VisualStudio.TestTools.UnitTesting;
#else
using NUnit.Framework;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestMethod = NUnit.Framework.TestAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
#endif
All I had to do was change my methods that used NUnit's Test attribute to be TestMethod and I had a test harness code that worked both ways.

TimeOutAttribute

The documentation for most of the attributes is quite good. But one of the more important attributes, TimeOutAttribute, isn't covered except in the API documentation. While the TESTRUNCONFIG file allows you to specify the overall timeout value of the unit test, the TimeOutAttribute lets you specify the maximum number of milliseconds an individual test can take. I find TimeOutAttribute invaluable on those test methods that hit the database so I can keep an eye on those queries. Keep in mind that the timing values include some of the test runner's time. In addition, the speed and capabilities of the machine will affect the time. Be sure to experiment with your tests to see how the timing works on your system.

The TestContext class, which is also the TestContext property added by the Unit Test Wizard, is also only briefly touched upon. The main discussion is on using the TestContext property to get the data rows in when you are using the DataSourceAttribute. The TestContext class has a lot more to offer. The documentation shows the TestContext class as being abstract, but the derived type actually handed to your unit test is UnitTestAdapterContext, from down deep in Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll, which you'll find in <Visual Studio .NET install dir>\Common7\IDE\PrivateAssemblies. You may want to look at UnitTestAdapterContext with .NET Reflector to see how it works.

Probably the most important method supported by this class is WriteLine, which you can use to add additional output to the individual test results. All the writes show up in the Additional Information section of the report. To find out what test is running or what directory the tests started from you can use the TestContext property fields TestName and TestDir, respectively. Finally, if you want a timer for all or part of your tests, call TestContext.BeginTimer and TestContext.EndTimer. The timer statistics are output to the test run results in the Standard Console Output section.

Code for You

As I mentioned, I've included one of my real units along with its unit test. Bugslayer.Utility.DLL is a collection of utility code I've been dragging around between projects. The ArgParser class is a command-line argument parsing class based on the class in the old .NET Framework SDK WordCount sample. The SystemMenuForm class is a Windows Form that allows you to append items to the System Menu and respond to clicks as normal events.

I wrote GlobalMessageBox because I was tired of seeing the Code Analysis error, Specify MessageBoxOptions, every time I used a message box. The rule states that when using a message box, you need to look through the parent classes and see if the RightToLeft property is set toYes. If it is, you must call the appropriate overload of MessageBox.Show to pass in the MessageBoxOptions flags. GlobalMessageBox takes care of this for me, suppressing the error and allowing my code to work correctly on right-to-left language systems.

The full unit test for Bugslayer.Utility.DLL is in the Bugslayer.Utility\Tests\Bugslayer.Utility.Tests directory. There are 39 tests in the various CS files providing over 92 percent code coverage. As this is a Windows Forms-focused unit that that brings up message boxes and controls, you can't run it unattended, but it can still run in under 15 seconds.

The other chunk of code is in the Bugslayer.Build.Tasks.DLL assembly, and is for use with MSTEST.EXE. While running tests in the IDE is great for debugging them, I love that I can easily execute my tests outside the IDE. MSTEST.EXE is likely to become very important in your life, as it will be the runner for all your big smoke and regression tests.

Names and Places

The oddest part of MSTEST.EXE is its output naming convention. If you use the /RUNCONFIG option to use a TESTRUNCONFIG file, the output file will use the naming convention specified in that file. If you don't use /RUNCONFIG, or you leave it set to the default, all the output is written to .\TestResults\<user>_<machine> <timestamp>. I recommend using names that are more quickly identifiable.

MSTEST.EXE offers the /RESULTSFILE option, but this causes the output file name to lose the timestamp. Additionally, MSTEST.EXE fails if the file name specified to /RESULTSFILE exists. What I want is to specify a name that refers to the particular focus I am working on, but without having to add the timestamp manually.

You might be thinking that a possible solution is to use the VSMDI test metadata files that you're used to seeing in the Test Manager window. In fact, MSTEST.EXE does have a /TESTMETADATA option to load and run the tests. The problem is that you can only specify one VSMDI file.

One possible solution is to create a separate VSMDI file that imports all the other VSMDI files in your code. That will certainly work, but it presents yet another maintenance task to remember every time you add new tests to your code.

Also worth mentioning is that you can't tell the IDE or MSTEST.EXE when running VSMDI files where to place the output. The output goes to a directory where the VSMDI file resides. I recommend keeping the tests in a directory below the source code in version control so that if you share the project all the test code goes with it.

With VSMDI files as part of each test and no way to centralize the output, the output will be scattered all over your source code. This isn't a big deal, but it does mean you have to clean up the source tree manually. After having dealt with the results of many test runs, I decided I wanted an easier way to handle this.

A Better MSTEST.EXE

Given this discussion, I saw there were four features I wanted to add to MSTEST.EXE. The first was a way to dynamically find all your unit tests and run them just like a mini smoke test. The second was an easy way to identify similar test runs other than by just reading the timestamp. The third was to ensure all the test output went to a single place. And finally, I wanted a very easy way to get rid of extraneous test runs no matter where they were in your source tree.

These requirements scream for MSBuild. The MSTestTask in Bugslayer.Build.Tests.DLL wraps MSTEST.EXE so you've got all the control in the world. As you look at the code, you'll notice that it's derived from a ToolTask class, which comes out of the Microsoft.Build.Utilities.DLL assembly. ToolTask is what you want to use when writing build tasks that wrap a command-line tool because it does most of the heavy lifting.

For many tools, all you need to do is define your unique properties and override three methods and one property. The property is ToolName, which returns the executable name of the tool. The GenerateFullPathToTool method returns the complete drive, path, and file name to the tool itself. To validate the parameters, you override the ToolTask.ValidateParameters method and return true if everything works. To build the actual command line to the tool, override ToolTask.GenerateCommandLineCommands and use either the CommandLineBuilder class or my small extension to it, ExtendedCommandLineBuilder.

Run MSTEST.EXE /? for all the possible command-line parameters. The ResultsFile is required as it specifies the output file name. You also need to set either the TestMetaData parameter or the TestContainer parameter to indicate the metadata file or test containers, respectively.

The really interesting work is in MSTestTask.ValidateParameters because that's where I crack open the TESTRUNCONFIG file and do a little XPath jujitsu to make sure the unique name I generate matches the formatting you may have set in TESTRUNCONFIG. You might be thinking that it would have been easier to just write this task in a TARGETS file using the Exec task. In fact, while you would not have had the ability to look in the TESTRUNCONFIG file to get the name, you could do the rest.

However, my long-term plan for MSTestTask is to extend the Tests property to allow wildcards to be passed for names of tests to execute. That would allow you to easily execute tests matching only those with specific prefixes. The work would simply be a bunch of reflecting through the assemblies passed in the TestContainer property, looking for the classes that have the TestClassAttribute on them and methods with the TestMethodAttribute matching the regular expression passed in.

The other action part of MSTestTask comes in the RunTests.Targets file, which you'll find in the .\Build directory with the source code. This contains the very cool ExecuteAllTests target, which starts at a directory you specify and looks for all unit tests, GenericTests, WebTests, and OrderedTests in the entire hierarchy and automatically executes them. You can think of the ExecuteAllTests target as an automatic regression test for your unit test. As you add new tests, it will execute them automatically. The code for RunTests.Targets, which is included in the code download, makes judicious use of excluding files to get just the ones we want. To see an example of RunTests.Targets in use, look at SmokeTest.proj, which shows the smoke test for all of this column's code.

The final TARGETS file in the .\Build directory is MSTestCleanUp.Targets. As the name implies, its job is to find all those directories that have TestResults as one of the paths and delete them. It's a good example of using transforms as well as the wonderful RemoveDuplicates task in MSBuild. With MSTestClean-Up.Targets at your side, you won't sully your source directories with extra files.

Wrap Up

If you can't tell, I'm extremely excited about the new unit testing tools in Visual Studio 2005. Things like ASP.NET 2.0 and DataGrids get all the attention in the press, but the testing tools will have a much bigger impact when you're trying to get your application out the door on time. I can guarantee that the more time you spend playing with the testing tools, the better your code will be!

Tip 73 You can control the default programming language for a unit test and exactly what items are put in a new unit test when it's created from the Unit Test Wizard. Go into the Options dialog, expand the Test Tools node, and go to the Test Project property page. There you'll see the Default test project type combobox and the default file selections for each language type. If you're like me, you'll uncheck the "About Test Projects" introduction file after you create your second unit test.

Tip 74 Try to keep all the unit tests for an assembly in a single test assembly. That one-to-one mapping is easy from a maintenance standpoint. However, as the assembly grows, the number of tests can get quite large. I like to place prefixes on the test method names related to the feature they are testing. This allows for easy grouping. For example, in the Bugslayer.Utility.Tests.DLL assembly, the tests related to the GlobalMessageBox class all start with "GMB_".

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in .NET and Windows. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at www.wintellect.com.


.NET Matters: Abortable Thread Pool

.NET Matters
Abortable Thread Pool
Stephen Toub

Code download available at: NETMatters0603.exe (118KB)

Q In your July 2005 column (.NET Matters: StringStream, Methods with Timeouts), you demonstrated how to use Thread.Abort to cancel a long-running operation. I know you warned against this approach for reliability reasons, but I think the risks are acceptable for my situation. I've queued a bunch of work items to be executed in the thread pool. Later on, I may need to cancel one of these operations (in effect, a manual timeout), dequeuing it from the pool if it hasn't already executed, and aborting it if it's currently in the process of executing. The ThreadPool doesn't seem to expose any functionality like this. Is my only option to write my own ThreadPool?

A That's a great question. To answer the first part, the ThreadPool does not expose the ability to dequeue an item from its internal queue, though there are alternatives, as you'll see shortly. The second part of your question comes up frequently in a much more generalized form: how can you determine which thread a queued work item will execute on? The answer, of course, is that you don't know until it's already executing. After all, the whole point of having a pool of threads is to allow any one of them to complete the requested work. This means that the best way to discover which thread a work item is executing on is to have the work item tell you as soon as it starts running. For example, the first thing the work item's method could do is make some sort of callback, passing along information about the current Thread object. In your scenario, with the work item's Thread in hand, you could then easily abort it.

This solution is perilous, however. By the time the notification is received and the thread is aborted, the original work item could have already finished, such that the pool's thread already moved on to another work item. You would be aborting that secondary work item rather than the desired one, opening yourself up to a world of hurt. Plus, this requires you to have the appropriate callback logic in every method you queue to the pool.

Instead, consider a thread pool implementation that handed you back a cookie of sorts for a queued work item. The pool could then also provide a Cancel method that would take one of these cookies and cancel the associated work item, removing it from the queue or aborting the executing work item as appropriate. Implementing a custom thread pool for this task probably isn't the best idea, but there are other alternatives. (Note that it is possible to implement custom thread pools; for an example of doing so, see my February 2005 column at .NET Matters: File Copy Progress, Custom Thread Pools.)

In my October, November, and December 2004 columns, I showed how to implement static wrappers for ThreadPool, each of which accomplished a different task (waiting for queued work to complete, prioritizing certain queued items, and throttling pool usage). That same technique can be used here to create a wrapper for the thread pool that allows work items to be canceled. Figure 1 shows a high-level diagram of the process involved and Figure 2 shows its implementation.

Figure 2 Implementation (Close Figure 2)
public sealed class WorkItem
{
  private WaitCallback _callback;
  private object _state;
  private ExecutionContext _ctx;

  internal WorkItem(WaitCallback wc, object state, ExecutionContext ctx)
  {
    _callback = wc;
    _state = state;
    _ctx = ctx;
  }

  internal WaitCallback Callback { get { return _callback; } }
  internal object State { get { return _state; } }
  internal ExecutionContext Context { get { return _ctx; } }
}

public enum WorkItemStatus
{
  Completed, Queued, Executing, Aborted}

public static class AbortableThreadPool
{
  private static LinkedList<WorkItem> _callbacks =
    new LinkedList<WorkItem>();
  private static Dictionary<WorkItem, Thread> _threads =
    new Dictionary<WorkItem, Thread>();

  public static WorkItem QueueUserWorkItem(
    WaitCallback callback, object state)
  {
    if (callback == null) throw new ArgumentNullException("callback");
    WorkItem item = new WorkItem(
      callback, state, ExecutionContext.Capture());
    lock (_callbacks) _callbacks.AddLast(item);
    ThreadPool.QueueUserWorkItem(new WaitCallback(HandleItem));
    return item;
  }

  private static void HandleItem(object ignored)
  {
    WorkItem item = null;
    try
    {
      lock (_callbacks)
      {
        if (_callbacks.Count > 0)
        {
          item = _callbacks.First.Value;
          _callbacks.RemoveFirst();
        }
        if (item == null) return;
        _threads.Add(item, Thread.CurrentThread);
      }

      ExecutionContext.Run(item.Context,
        delegate { item.Callback(item.State); }, null);
    }
    finally
    {
      lock (_callbacks)
      {
        if (item != null) _threads.Remove(item);
      }
    }
  }

  public static WorkItemStatus Cancel(WorkItem item, bool allowAbort)
  {
    if (item == null) throw new ArgumentNullException("item");

    lock (_callbacks)
    {
      LinkedListNode<WorkItem> node = _callbacks.Find(item);
      if (node != null)
      {
        _callbacks.Remove(node);
        return WorkItemStatus.Queued;
      }
      else if (_threads.ContainsKey(item))
      {
        if (allowAbort)
        {
          _threads[item].Abort();
          _threads.Remove(item);
          return WorkItemStatus.Aborted;
        }
        else return WorkItemStatus.Executing;
      }
      else return WorkItemStatus.Completed;
    }
  }
}
Figure 1 High-Level Process
Figure 1 High-Level Process

To use this class, you simply substitute AbortableThreadPool anywhere you're using ThreadPool to queue work items you want to be able to cancel. It's fine to still make calls to ThreadPool.QueueUserWorkItem while using AbortableThreadPool, but those work items won't be cancelable. A typical call sequence might look like the following:

WorkItem item = AbortableThreadPool.QueueUserWorkItem(
  new WaitCallback(YourWorkMethod));
... // do other stuff here
WorkItemStatus status = AbortableThreadPool.Cancel(item, false);
Console.WriteLine("Status from Cancel: " + status);

As with the previous wrappers, for this to work I need to be able to store the information supplied to the ThreadPool.QueueUserWorkItem method, and for that I've created the WorkItem class (also shown in Figure 2). An instance of this class stores three pieces of information, including the two pieces of information supplied by the user (the WaitCallback to be executed and the object state to be passed as an argument to that method). The third piece of information is the ExecutionContext captured at the time QueueUserWorkItem is invoked.

ExecutionContext is a most welcome addition to the Microsoft® .NET Framework 2.0, serving as a container for important data about the current logical thread of execution. This includes the thread's security context, including its compressed stack, consisting of code access security permission grants and stack modifiers, and the current Windows® identity; its synchronization context, which allows synchronous and asynchronous operations to behave in accordance with the appropriate synchronization model, for example using Control.Invoke or Control.BeginInvoke in a Windows Forms application; its logical call context; and its host execution context. What makes ExecutionContext shine is that you can capture the ExecutionContext for the current thread, and then later on use that ExecutionContext while running arbitrary code, even if that code isn't running on the thread for which the context was originally captured.

Why is that a good thing? Think about what happens when you queue a work item to the thread pool, or use Control.Invoke to marshal a call to the GUI thread of a Windows Forms application, or call Thread.Start to execute a method on a new thread, or invoke a delegate asynchronously. In essence, one thread is asking another thread to perform work on its behalf. Since the two threads don't share any thread-based state in common, if nothing about the source thread is transferred to the target thread, the disconnect that occurs can lead to a variety of problems. Some of these problems lead to functional bugs, such as data stored in the source's thread-local storage not being available during execution, and others lead to security vulnerabilities, such as the execution happening under a different Windows identity.

By allowing the capture and flow of execution context from one thread to another, the .NET Framework 2.0 solves this disconnect. In fact, in most scenarios you'll never have to deal with ExecutionContext directly as it is automatically flowed across thread transfer points (ThreadPool.QueueUserWorkItem, Thread.Start, delegate asynchronous invocation, and so on.) You only have to deal with it directly when some action you perform causes this automatic flow to break. With that said, the reason why WorkItem stores an ExecutionContext should become apparent shortly.

The AbortableThreadPool class shown in Figure 2 maintains several collections of information. One collection, a linked list of WorkItem instances, contains work items queued using AbortableThreadPool.QueueUserWorkItem that haven't had a chance to execute yet. The other collection is a dictionary, mapping WorkItem instances to Thread objects; this collection is used to track on which pool thread a work item is currently executing.

AbortableThreadPool.QueueUserWorkItem is the simplest of the methods on this class. It creates a WorkItem that stores the supplied WaitCallback and user state object, as well as the current ExecutionContext. This WorkItem is then enqueued into the linked list of work items waiting to be executed. In addition, a placeholder callback is inserted into the actual .NET ThreadPool. This placeholder contains no information about the actual work to be performed and simply points back to the AbortableThreadPool.HandleItem method. As one delegate for HandleItem is queued when and only when the AbortableThreadPool,QueueUserWorkItem method is called, every work item in the linked list has a corresponding queued work item in the actual thread pool. The WorkItem, which serves as the cookie discussed earlier in this column, is returned to the user.

When HandleItem is called, it removes the next WorkItem from the linked list of work items. This WorkItem is then inserted along with the current Thread instance (as retrieved from Thread.CurrentThread) into the table of running work items (the WorkItem serves as the key into the dictionary). At this point, the ExecutionContext is retrieved from the WorkItem and is used to execute the queued delegate, supplying it with the relevant user state object. When the delegate finishes executing, the WorkItem-to-Thread mapping is removed from the running items table.

This system makes it possible to cancel a work item regardless of its current status in the pool. A user armed with the WorkItem returned from QueueUserWorkItem can pass the instance to the Cancel method. The Cancel method first looks in the linked list of WorkItems; if the WorkItem has not begun execution, it'll be in this list. If the item is found, it's simply removed from the queue. This causes the number of WorkItems in the linked list and the number of queued work items in the managed thread pool to become disconnected, but that's OK. When HandleItem attempts to retrieve the next item from the queue, it returns gracefully if no more items are available.

However, this scenario is one of the reasons why you must store the ExecutionContext in the WorkItem and execute the delegate under that context. By default, execution context flows from the thread that queues a delegate to the pool to the execution of that delegate. But by canceling a work item in this fashion—something the managed thread pool is completely unaware of—you cause the work item and flowed contexts to get out of sync. As a result, one work item will execute under the context flowed for a different work item.

Even if you couldn't cancel items, it would still be necessary to store the execution context with the work item. Each time HandleItem is called from the ThreadPool, it'll pick off the next WorkItem in our queue. But the fact that the thread pool executes the HandleItem methods in the order you queued them doesn't guarantee anything about the order in which various instructions within those HandleItem calls will execute. It's perfectly possible for one HandleItem call to start before another, but for the second call to remove one of the waiting WorkItems first. If that happens—again, there will be a disconnect in the flow of the execution context—you must do it manually.

If the Cancel method does not find the WorkItem returned from QueueUserWorkItem in the linked list, there are only two other possibilities. First, the work item could be executing currently, in which case it would show up as a key in the dictionary, mapping WorkItem instances to Thread instances. Moreover, if it is in that table you can easily retrieve the Thread on which that delegate is executing, thus allowing you to abort it if requested. If the WorkItem is not a key in that dictionary, it must have already finished executing, and thus there is nothing to be canceled. The locks that bookend the execution of the work item and that surround the call to Abort prevent the incorrect thread from being aborted; this would not be preventable if the Thread object for the pool thread were simply returned to the user of the AbortableThreadPool.

As with my method timeouts example in the July 2005 issue, I'll caution you against using Thread.Abort to control the lifetime of a thread, especially if you have little knowledge about the implementation of the code you're aborting. An abort inside of a critical region could spell disaster for an entire AppDomain, and while a polite abort does signficantly reduce the window of disaster to very tiny slivers, that one time out of 1,000 could lead to a deadlock or resource leak or worse. Depending on your workload and abort rate, this may or may not snowball over time. For more information on why this is dangerous and for ways to deal with it, see my article on reliability in the October 2005 issue (High Availability: Keep Your Code Running with the Reliability Features of the .NET Framework).

That said, and as with the method timeouts example, there are times when it's nice to have the power of Thread.Abort. For those situations, AbortableThreadPool could come in handy. And since I've implemented the Cancel method to accept a parameter that indicates whether a currently executed work item should be aborted, you can, of course, choose to use this code simply to cancel pending work items rather than also canceling work items that have already started execution.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


C++ at Work: Event Programming, Part 2

C++ At Work
Event Programming, Part 2
Paul DiLascia

Code download available at: CAtWork0603.exe (3,178KB)

Last month I answered a question about implementing native events in C++ (see C++ At Work: Event Programming). I discussed events in general and showed how to use an interface to define event handlers for your class that clients must implement to handle events. My implementation had some shortcomings that I promised to resolve in the sequel, so here I am to finish the story.

Figure 1 Calculating Prime Numbers
Figure 1 Calculating Prime Numbers

But first, a brief review of the previous episode. I wrote a program, PrimeCalc (Figure 1), with a class CPrimeCalculator that calculates prime numbers. CPrimeCalculator raises two events: Progress and Done. As it searches for prime numbers, CPrimeCalculator raises the Progress event to report how many primes it's found so far. When it's finished, it raises the Done event. The events are defined by an interface, IPrimeEvents:

class IPrimeEvents {
public:
  virtual void OnProgress(UINT nPrimes) = 0;
  virtual void OnDone() = 0;
};

Clients that want to handle prime events must derive from IPrimeEvents, implement the handler functions, and call CPrimeCalculator::Register to register their interface. CPrimeCalculator::Register adds the client object/interface to its internal list. When it's time to raise a Progress event, CPrimeCalculator calls a helper function NotifyProgress:

void CPrimeCalculator::NotifyProgress(UINT nFound)
{
  list<IPrimeEvents*>::iterator it;
  for (it=m_clients.begin(); it!=m_clients.end(); it++) {
    (*it)->OnProgress(nFound);
  }
}

NotifyProgress walks the client list, calling each client's OnProgress handler. As a programmer using CPrimeCalculator, writing the code to handle events is straightforward and easy—just derive from IPrimeEvents and implement the handlers. But as a programmer implementing a class like CPrimeCalculator that raises events, the mechanism is tedious. You have to implement a NotifyFoo function for every Foo event, even though each one has the same familiar pattern. And the event-raising code is split across two classes, the event interface (IPrimeEvents) and the event source class (CPrimeCalculator). What if you want to use the same event interface with different event sources? IPrimeEvents is pretty generic. I might rename it IProgressEvents and use it with any worker class that reports Progress in the form of an integer and Done when it's finished. But each class that raises progress events has to (re)implement the notification functions that raise the event. Ideally, all the event code should reside in a single class.

Since the notification functions follow a predictable pattern, it's natural to ask: Isn't there some way to implement them generically? Shouldn't I be able to encapsulate the entire event mechanism in a single class or template or macro or something any event source can use? The answers are yes and yes. I'll show you how to create an event system that uses macros and templates to reduce event coding to the barest minimum. Our journey requires flying high into the C++ stratosphere of nested templates and functor classes.

I'll implement the system in several steps. The goal is to write a template that implements the notification functions NotifyProgress and NotifyDone. Each function has a similar but not identical pattern:

// NotifyFoo — raise Foo event
list<IPrimeEvents*>::iterator it;
for (it=m_clients.begin(); it!=m_clients.end(); it++) {
  (*it)->OnFoo(/*args*/);
}

That is, iterate the client list, and call OnFoo for each client, passing the event parameters. How can I write this as a template? I can parameterize the interface IPrimeEvents as a type T, but how do I parameterize the event handler functions OnFoo, which can have any name and signature a programmer chooses?

Whenever you need to parameterize a function, you should think: function class. Also known as a functor, a function class is a C++ sleight-of-hand trick that converts a function to a class, so instead of passing a pointer to a callback function, you pass an instance of the function class. Functors abound in the Standard Template Library (STL), which implements several algorithms that use functors. In particular, the for_each algorithm is useful here:

for_each(m_clients.begin(), m_clients.end(),
  NotifyProgress(nFound));
The for_each algorithm iterates the container from beginning to end and invokes the function object NotifyProgress for each element. But what exactly is this "function object"? It's not a function; it's an object. The class looks like this:
class NotifyProgress {
protected:
  UINT m_nFound;
public:
  NotifyProgress(UINT n) : nFound(n) { }
  void operator()(IPrimeEvents* obj)
  {
    obj->OnProgress(nFound);
  }
};

NotifyProgress implements the function operator()(IPrimeEvents*), which is what the for_each algorithm needs. In general, if you have a collection of objects of type T, the for_each algorithm expects a functor that implements operator()(T). It calls your operator with each T object in the collection. So in this case the function operator takes a pointer to IPrimeEvents and returns void—because the client list is a list of pointers-to-IPrimeEvents. To pass additional parameters, the constructor stores them as data members. Calling NotifyProgress(nFound) is calling the constructor to create a stack instance initialized with m_nFound=nFound. So the general pattern for any Foo functor that raises the Foo event is:

class NotifyFoo {
protected:
  ARG1 m_arg1; // whatever, as many as needed
public:
  NotifyProgress(ARG1 a1, ...) : m_arg1(a1) { }
  void operator()(IMyEvents* obj)
  {
    obj->OnFoo(m_arg1, ...);
  }
};

The constructor saves the event parameters as data members and the function operator passes them to the object's event handler function. The upshot of all this—and of all functors in general—is that I've converted a function OnFoo into a class NotifyFoo. And why is that useful? Because now I can write a template. But before I do that, there's one little thing I want to mention. To earn your gold star, you really should derive your functor from an STL class called unary_function:

class NotifyProgress :
  public unary_function<IPrimeEvents*, void>
{
  .
  . // as before
  .
};
This says that NotifyProgress is a unary function whose function operator takes one argument, a pointer-to-IPrimeEvents, and returns void. The unary_function makes your functor class "adaptable," which lets you combine it with STL adapters like not1, bind2nd, and so on. But even if, as is the case with my event handlers, you never plan to use adapters, unary_function is still a good idea because it announces to the world, "this is a functor." It's a way of documenting your code. For a full discussion of adapters, see Effective STL: 50 Specific Ways to Improve Your Use of the Standard Template Library (Addison-Wesley, 2001) by Scott Meyers.

STL gurus may be wondering why I don't use the mem_fun adapter to convert IPrimeEvents::OnProgress directly to a function object. It's because OnProgress is virtual and you can't adapt a virtual function. If you do, it'll get sliced to the base class. If you're using the Boost library, you could use its bind adapter to directly convert virtual event handlers like OnFoo to functors without writing a functor. If you have no idea what I'm talking about, don't fret, just keep reading.

Of course, I also need a NotifyDone for the Done event. Since the Done event has no parameters, the constructor has none either:

class NotifyDone :
  public unary_function<IPrimeEvents*, void>
{
public:
  NotifyDone() { }
  void operator()(IPrimeEvents* obj)
  {
    obj->OnDone();
  }
};

So now that I have my functor classes I can use for_each instead of manually iterating the client list. Where should I put them? The functors belong with the event specification, so I'll put them inside IPrimeEvents, as nested classes. Figure 2 shows the code. Astute readers will notice I've made a couple of other minor modifications as well. Instead of naming the functor NotifyProgress, I called it Progress. You'll see how this makes the code more readable later. And instead of declaring all my event handlers pure, I've defined them with empty implementations. IPrimeEvents has only two events, but for a general event mechanism, it seems unfriendly to make programmers implement every event handler when they may be interested in handling only a few. So now each handler gets a default do-nothing implementation. To make the base class abstract, I've declared the destructor pure. This is a standard C++ trick when you want to make a class abstract without having any pure virtual functions. The only catch is you have to define the dtor. A pure virtual function has no definition—unless it's the destructor. Since each derived class dtor calls its base class dtor, the base class needs an implementation even though it's pure:

inline IPrimeEvents::~IPrimeEvents() { }
Now with my functors defined, CPrimeCalculator raises the Progress event, like so:
// in CPrimeCalculator:
void NotifyProgress(UINT nFound)
{
  for_each(m_clients.begin(), m_clients.end(),
    IPrimeEvents::Progress(nFound));
}

Figure 2 Prime 1 (Close Figure 2)
/////////////////
// Interface for prime events. This class defines the signatures
// of the event handler functions as well as nested functor
// classes to use with for_each. Clients that handle prime events must
// derive from this and override the event handlers for events they
// handle.
//
class IPrimeEvents {
protected:
    IPrimeEvents() { }
    virtual ~IPrimeEvents() = 0;

public:
    // Virtual event handler fns. Override the ones you handle.
    virtual void OnProgress(UINT /* nPrimes */) { }
    virtual void OnDone() { }

    // Nested functor class for Progress event. Note how the ctor requires
    // the same parameters as the event handler and stores them in the
    // object.
    class Progress : public unary_function<IPrimeEvents*, void> {
    protected:
        UINT nFound;
    public:
        Progress(UINT n) : nFound(n) { }
        void operator()(IPrimeEvents* obj)
        {
            obj->OnProgress(nFound);
        }
    };

    // Nested functor class for Done event. This time there are no event
    // args.
    class Done : public unary_function<IPrimeEvents*, void> {
    public:
        Done() { }
        void operator()(IPrimeEvents* obj)
        {
            obj->OnDone();
        }
    };
};

inline IPrimeEvents::~IPrimeEvents() { }

// List (vector) of prime numbers.
typedef vector<UINT> PRIMELIST;

//////////////////////////////////////////////////////////////////
// Class to calculate 1st n prime numbers.
//
class CPrimeCalculator
{
protected:
    list<IPrimeEvents*> m_clients;     // list of client objects to notify
    PRIMELIST m_primes;            // list (vector) of primes numbers found

    // Raise events. Use for_each with function class.
    void NotifyProgress(UINT nFound)
    {
        for_each(m_clients.begin(), m_clients.end(),
            IPrimeEvents::Progress(nFound));
    }

    void NotifyDone()
    {
        for_each(m_clients.begin(), m_clients.end(),
            IPrimeEvents::Done() );
    }

    ...
};

So far, I've introduced functor classes Progress and Done, and now NotifyProgress and NotifyDone can use STL's for_each algorithm. What next? Remember, my goal is to get rid of the NotifyFoo functions entirely—or rather, implement them as templates so programmers creating events don't have to write these boilerplate functions for every event they define. Converting the for loop to a for_each algorithm is only the first step.

The functors set the stage for templates by converting the virtual OnFoo member function to a Foo functor type. (Functors are a bit like .NET delegates in this respect.) Now that my notification functions vary by type instead of function names, I can parameterize them. And when I do that, I can move the whole event implementation out of the source class CPrimeCalculator and into a new template class CEventMgr that's totally generic. Figure 3 shows the result. CEventMgr<I> holds a list of I* pointers. It has Register and Unregister methods to add and remove clients from its list, and it has a template member function Raise that raises an event:

template <typename I>
class CEventMgr
{
  ...
  template <typename F>
  void Raise(F fn)
  {
    for_each(m_clients.begin(), m_clients.end(), fn);
  }
};

Figure 3 EventMgr.h (Close Figure 3)
////////////////////////////////////////////////////////////////
// Generic event manager for native C++.
//
#pragma once

#include <list>
#include <algorithm>
using namespace std;

//////////////////
// Generic event manager. Holds list of client objects. Template class
// parameterized by the event interface.
//
// - Instantiate in event source class.
//
// - Source calls Raise to raise an event, passing functor object
// initialized with event params.
//
// - Clients derive from event interface and override event handlers.
// Clients call Register/Unregister to add/remove themselves.
//
template <typename I>
class CEventMgr
{
protected:
    list<I*> m_clients; // list of registered client objects
public:
    CEventMgr() { }
    ~CEventMgr() { }

    // Register: Add client to list.
    void Register(I* client)
    {
        m_clients.push_back(client);
    }

    // Unregiser: Remove client from list.
    void Unregister(I* client)
    {
        m_clients.remove(client);
    }

    // Nested template member function! This fn calls the function object
    // F for each registered client. It merely passes F to for_each. Use
    // the DEFINE_EVENT macros to generate the functors. See IPrimeEvents
    // in Prime.h for example.
    template <typename F>
    void Raise(F fn)
    {
        for_each(m_clients.begin(), m_clients.end(), fn);
    }
};

// Macro to declare an event interface. Generates declarations for ctor/dtor.
#define DECLARE_EVENTS(name)                                            \
protected:                                                              \
    name() { }                                                          \
    virtual ~name() = 0;                                                \

// Macro to implement event interface. Defines dtor
#define IMPLEMENT_EVENTS(name)                                          \
    inline name::~name() { }                                            \

// Macros to define events that use 0, 1 or 2 parameters. If you need more
// than two, you should define a struct/class and pass a pointer to it.

// Event with no args: Declare OnFoo handler and Foo functor.
#define DEFINE_EVENT0(iface,name)                                 \
virtual void On##name() { };                                      \
class name : public unary_function<iface*, void> {                \
public:                                                           \
    name() { }                                                    \
    void operator()(iface* obj)                                   \
    {                                                             \
        obj->On##name();                                          \
    }                                                             \
};                                                                \

// Event with one arg: Declare OnFoo handler and Foo functor.
#define DEFINE_EVENT1(iface,name,T1)                              \
virtual void On##name(T1) { }                                     \
class name : public unary_function<iface*, void> {                \
protected:                                                        \
    T1 m_arg1;                                                    \
public:                                                           \
    name(T1 a1) : m_arg1(a1) { }                                  \
    void operator()(iface* obj)                                   \
    {                                                             \
        obj->On##name(m_arg1);                                    \
    }                                                             \
};                                                                \

// Event with two args: Declare OnFoo handler and Foo functor.
#define DEFINE_EVENT2(iface,name,T1,T2)                           \
virtual void On##name(T1, T2) { }                                 \
class name : public unary_function<iface*, void> {                \
protected:                                                        \
    T1 m_arg1;                                                    \
    T2 m_arg2;                                                    \
public:                                                           \
    name(T1 a1, T2 a2) : m_arg1(a1), m_arg2(a2) { }               \
    void operator()(iface* obj)                                   \
    {                                                             \
        obj->On##name(m_arg1, m_arg2);                            \
    }                                                             \
};                                                                \

Holy cow, Batman! How often do you get to write a template-within-a-template? But that's just what's required here. Now to raise an event I can write:

void NotifyProgress(UINT nFound)
{
  m_eventmgr.Raise(IPrimeEvents::Progress(nFound));
}

That's more like it! No for loop, not even a for_each. All the details are encapsulated in CEventMgr, and raising an event is one line of code that knows nothing about the mechanics. I could even dispense with NotifyProgress entirely and simply call CEventMgr::Raise when I want to raise an event—but good coding ethics prevent me because I'd rather encapsulate Raise in a function in case I ever change CEventMgr or expose the event-raising function to clients. Since NotifyProgress is inline, there's no performance lost.

In case templates make your brain hurt, let me spell out what's going on. CEventMgr is a template class parameterized by the event interface I. So CEventMgr<IPrimeEvents> instantiates an event manager based on IPrimeEvents. It holds a data member m_clients, which is a list of IPrimeEvents pointers, list<IPrimeEvents*>. Within CEventMgr is a template member function, Raise<F>, which passes its functor argument F to for_each. So when you write

m_eventmgr.Raise(IPrimeEvents::Progress(nFound));
the compiler sees you're trying to call CEventMgr::Raise with an argument of type IPrimeEvents::Progress, so it uses the template to generate the member function CEventMgr::Raise(IPrimeEvents::Progress). The implementation passes the functor instance to for_each, which invokes the functor's function operator() for each I* object in the client list. The functor calls the object's OnProgress handler—just what I wanted! Aren't templates cool?

We've reached third base, we're almost home. The functors let me parameterize the event methods and use for_each, but they're still several lines long and I so hate typing. So the final step is to introduce some macros to spare my carpal tunnels. Here's the final condensed definition for IPrimeEvents:

class IPrimeEvents {
  DECLARE_EVENTS(IPrimeEvents);
public:
  DEFINE_EVENT1(IPrimeEvents, Progress, UINT /*nFound*/);
  DEFINE_EVENT0(IPrimeEvents, Done);
};
IMPLEMENT_EVENTS(IPrimeEvents);

Figure 4 shows the full code—you can see I'd be hard pressed to condense it any further. Only the essential information appears: the names and signatures for each event handler. The macros assume the name of the Foo event handler is OnFoo. Some programming purists hate macros, but not me. Why not use every tool at your disposal? DECLARE_EVENTS declares the constructor and destructor; IMPLEMENT_EVENTS implements the inline dtor. The macros DEFINE_EVENT0, DEFINE_EVENT1, and DEFINE_EVENT2 declare/define the OnFoo event handlers and Foo functors for events with zero, one, or two parameters. If you need more than two parameters, you can define a struct and pass a pointer to it as a single event parameter:

MumbleArgs args;
args.a = 1;
args.b = 2;
// etc.
m_eventMgr.Raise(IMyEvents::Mumble(&args));

Figure 4 Prime.h (Close Figure 4)
// Now the whole thing is defined with macros.
//
class IPrimeEvents {
    DECLARE_EVENTS(IPrimeEvents);
public:
    DEFINE_EVENT1(IPrimeEvents, Progress, UINT /*nFound*/);
    DEFINE_EVENT0(IPrimeEvents, Done);
};

IMPLEMENT_EVENTS(IPrimeEvents);

// List (vector) of prime numbers.
typedef vector<UINT> PRIMELIST;

//////////////////////////////////////////////////////////////////
// Class to calculate 1st n prime numbers.
//
class CPrimeCalculator
{
protected:
    CEventMgr<IPrimeEvents> m_eventmgr;     // instantiate general event manager
    PRIMELIST m_primes;         // list (vector) of primes numbers found

    ...

    // Raise events. The notification functions simply call
    // CEventMgr::Raise with the appropriate functor object.
    void NotifyProgress(UINT nFound)
    {
        m_eventmgr.Raise(IPrimeEvents::Progress(nFound));
    }

    // Notify clients I'm Done.
    void NotifyDone()
    {
        m_eventmgr.Raise(IPrimeEvents::Done());
    }

public:

    // Register client to receive events
    void Register(IPrimeEvents* client)
    {
        m_eventmgr.Register(client);
    }

    // Unregister client from receiving events.
    void Unregister(IPrimeEvents* client)
    {
        m_eventmgr.Unregister(client);
    }
};

Alternatively, you could implement DEFINE_EVENT3. But remember: functor objects are passed by value, so they should be small. Why copy a ton of parameters back and forth across the heap and stack when you can pass a pointer instead? You can also use a struct if your event handlers need to return something. I made the handler functions return void in order to simplify my life.

Programmers sometimes ask if functors introduce a lot of extra overhead. In fact, functors are usually more efficient than functions. The reason is inlining. When you pass a pointer-to-function in C++, it really is a pointer, even if you define your function inline. You can't pass a function by value. But when you pass an object instance to a template function, the compiler can generate everything inline if you define your functions that way. In the case of events, there's still only one function call through a pointer, which happens when the function operator invokes the virtual OnFoo handler.

Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: Legal Doesn’t Think the Way You Do

{ End Bracket }
Legal Doesn’t Think the Way You Do
Don McGowan

If you develop software for a large company, you've undoubtedly experienced the joys of dealing with "Legal." The meetings, the endless explanations, the long waits. I'm here to report from the other side. No, I'm not going to apologize or make excuses either for myself or my colleagues, but if I tell you what's in our heads, maybe you can address some issues in advance, and maybe you can avoid dealing with us at all. (And if you don't have a Legal department, you might think it's time to stop reading. Nothing could be further from the truth. Fact is, everyone has a Legal department. Yours is called "the jury.")

Legal Is Full of Pessimists There's a simple reason behind this thinking. Lawyers are paid to do what comes naturally to pessimists: tell you everything that can go wrong. We're like testers, only better-dressed (sometimes). So handle us just like testers. Gather your team together and think of everything that could possibly go wrong with your project. Assume a collapse of the global telecoms systems at the exact time you deploy your Web-based app. Imagine the possibility that one of your team members stole their entire codebase from some rootkit author living in Kazakhstan. Or, perhaps more realistic, if you've promised delivery on a certain date and one of your team members gets sick or married, what will you do? And it's OK if you don't have answers to all of these questions. Showing that you've thought of them is half the battle. Business is about managing risks, and Legal's job is to find those risks and explain them.

Legal Isn't Paid to Fix Anything More true than you know, but worth bearing in mind. Once Legal has found all the problems it can think of, its work is complete. No one expects Legal to figure out the solutions to these problems. Of course, a good lawyer should help with finding solutions, but law school doesn't prepare you to fix bugs. Lawyers shouldn't be writing your code. They shouldn't be telling you who has the right skillset to be on your team. Many of us would be perfectly happy to help out with fixing the problems, but we have no idea where to start. And that's because...

Legal Doesn't Think the Way You Do Have you ever noticed that Legal doesn't have project managers? And that if you ask two lawyers the same question you'll never get the same answer. Or that sometimes you can't get an answer at all. And if you do get an answer, it's often "No."

There are very few black and white situations in law. When there's a statute that says "writing spyware is illegal" then anyone can tell you that if you do it, you're breaking the law. But what if the software you're writing needs to be updated regularly and you write a feature that phones home for updates. Is that "spyware" under these laws and, if so, what do you need to do to make it legal? That's a surprisingly difficult question. If you ask Legal to tell you how to guarantee that your software won't be "spyware," then Legal has to tell you not to let your software phone home. If you want certainty, then "No" is the only answer where Legal is guaranteed never to be proven wrong. You've asked for a yes-no answer to an either-or question.

By the way, it's not like Legal is the only place this happens. Would you feel comfortable saying that your app is unhackable? Or just that no one has found a way to hack it (yet)? Again, you can fix this. Don't ask for black or white. Ask how dark a shade of grey you're dealing with. Of course, then you're the one who has to take the risk, but...

Legal Takes Risks Too, but They're Different Risks You call the final shots—not Legal. I come across lots of people who want to let Legal tell them whether or not they should do something, and they get really upset when we won't. It may not be obvious, but every time we say "yes" we leave ourselves open to the chance of having been wrong, that we took a shades-of-grey question and answered black or white. That's why it can take five lawyers to answer one question—because we're all paranoid about missing something. Again, it's like your own experience. We have hackers constantly looking to attack our work, they're just called "other lawyers" and they do it in front of a jury where there's an audience and everyone knows which lawyer got the company into this mess. So that's why we won't take your risks, too—we have enough of our own.

So, it turns out we're people just like you. If we're a necessary part of your life, you can make it less painful to deal with us. Understand that we're paid to find problems. Know that we usually can't give you a straight answer. And be ready for us to tell you that the decision is yours. Sorry, there's no magic bullet. But then again, silver bullets kill werewolves, not bloodsucking vampires.

Don McGowan is an attorney for Microsoft Corporation. He tries to be less useless than this column probably makes him sound.