 MSDN Magazine January 2006

MSDN Magazine January 2006

WinFX Workflow: Simplify Development With The Declarative Model Of Windows Workflow Foundation

Windows Workflow Foundation allows you to write workflow-based programs in terms of domain-specific activities that are implemented in CLR-based programming languages such as C# and Visual Basic. Here Don Box and Dharma Shukla get you started.

Don Box and Dharma Shukla Dazzling Graphics: Top Ten UI Development Breakthroughs In Windows Presentation Foundation

The Windows Presentation Foundation provides new techniques for UI-based developing applications and makes better use of current hardware and technologies. In this article, Ian Griffiths and Chris Sells explain 10 of the most significant advances that make WPF superior to its Win32 predecessors.

Ian Griffiths and Chris Sells Talking Windows: Exploring New Speech Recognition And Synthesis APIs In Windows Vista

Windows Vista includes a built-in speech recognition engine exposed through a number of new APIs that will let your users interact with your app using speech rather than a keyboard or mouse. Here Robert Brown explains speech recognition and introduces you to the APIs to use in your upcoming Windows Vista applications.

Robert Brown XPS Documents: A First Look at APIs For Creating XML Paper Specification Documents

Windows Vista includes improved document technology called the XML Paper Specification that is designed to provide users with a consistent document appearance regardless of where and how the document is viewed, solving the age-old problem of document portability and display consistency. Here Bob Watson explains.

Bob Watson Visual Studio 2005: Create Reusable Project And Item Templates For Your Development Team

Visual Studio 2005 introduces a new model that lets you define your own project templates and starter kits, something developers have been requesting for some time. Here Matt Milner shows you how to consume, create, and customize these templates.

Matt Milner Unit Testing Tips: Write Maintainable Unit Tests That Will Save You Time And Tears

Everybody's talking about unit testing and you want to get into the game too. But you don't want the tests to take up all your development time and energy. Fortunately Roy Osherove is here to dispel some testing myths and put you on the road to efficient unit testing.

Roy Osherove Editor's Note: Back to the Future

As we head into the cold winter months, we've got great content to keep you warm! You might wonder why I started this Editor's Note with such a tired intro, but we have a reason. We're covering a lot of fresh, exciting new material this month, and we didn't want to upstage the good news.

Toolbox: Code Generation, A Charting Component, and More

Developers have a number of tools at their disposal, but which of these components do you really need to complete your project? Each month we'll analyze the latest software and books to see if they merit inclusion in your toolbox.

Scott Mitchell Dev Q&A: DataGridView

After receiving a late-breaking news bulletin at magazine headquarters early this month stating that the Web is just a fad that will never amount to anything, we have reluctantly decided to retire our venerable Web Q&A column and replace it with one we like to call Dev Q&A.

Edited by Nancy Michell CLR Inside Out: Base Class Library Performance Tips and Tricks

The common language runtime (CLR) sits at the very heart of managed code. Indeed, it is the heart of managed code, so to understand managed code you need to understand the CLR.

Kit George Data Points: SqlConnectionStringBuilder, DataView, and More

Several significant enhancements have been made to ADO. NET 2. 0 in the areas of improved performance, increased flexibility, and added features. In my last column (Data Points: DataSet and DataTable in ADO.

John Papa Advanced Basics: The Sound of Music

So you've got a burning desire to accompany your application's splash screen with a rousing chorus of "Funky Cold Medina" (that's Tone Loc for those of you who aren't children of the 80s). Prior to working with Visual Studio® 2005, adding even simple tunes and system sounds to your application could be a challenge.

Brad McCabe Cutting Edge: Custom Data Control Fields

In ASP. NET 2. 0, the GridView and DetailsView controls are designed to work together. They don't merely provide complementary services, they also share a number of helper classes and components. The output of the GridView control consists of a sequence of rows, each with a fixed number of columns.

Dino Esposito Service Station: All About ASMX 2.0, WSE 3.0, and WCF

The release of the Microsoft® . NET Framework 2. 0 reshapes the Web services landscape in several interesting, and perhaps confusing, ways. So this month I'm going to field some of the most common questions related to ASP.

Aaron Skonnard Extreme ASP.NET: Codebehind and Compilation in ASP.NET 2.0

As I write this column, the release candidates of the Microsoft® .NET Framework 2.0 and Visual Studio® 2005 have just come out, and by the time you read this, they will both already be on the shelves. It feels like it's been a long time coming.

Fritz Onion Security Briefs: Encrypting Without Secrets

Do you have a Web site or other system that deals in secrets of any sort? It seems like every time I give a security talk, people ask how to deal with the sticky problem of storing secrets. Connection strings with passwords are an obvious problem.

Keith Brown .NET Matters: Iterating NTFS Streams

Stephen Toub C++ at Work: Installing a Hook, Strings in Managed C++, and More

Paul DiLascia { End Bracket }: Transactions for Memory

Injecting parallelism into your app is easy—managed code offers things like explicit threading and a thread pool for that. Ensuring your code remains correct when run in parallel, on the other hand, is not quite so simple.

Joe Duffy


WinFX Workflow: Simplify Development With The Declarative Model Of Windows Workflow Foundation

WinFX Workflow

Simplify Development With The Declarative Model Of Windows Workflow Foundation

Don Box and Dharma Shukla
This article is based on a prerelease version of WinFX. All information contained herein is subject to change.
This article discusses:
  • Windows Workflow Foundation
  • What's inside a workflow
  • Composite activities and execution
  • Context and state management
This article uses the following technologies:
WinFX, XAML, C#, Visual Studio 2005
Contents
Workflow 101
Inside a Workflow
Composite Activities
Inside Activity Execution
Normal Activity Execution
Context and State Management
Where Are We?

W
inFX includes a technology called Windows Workflow Foundation that allows programs to be expressed as declarative, long-running processes called workflows. Unlike traditional Microsoft® .NET Framework programs, workflow-based programs are typically specified in a declarative Extensible Application Markup Language (XAML) document that specifies the structure of the program in terms of domain- specific activities. These activities are typically implemented in a traditional common language runtime (CLR)-based programming language such as C# or Visual Basic®.

WinFX™ provides a set of general-purpose activities that cover most control flow constructs, but users are free to ignore them and write an entirely new set of activities that are precisely tailored to the problem domain they are working in. More commonly, a workflow program will use WinFX-provided activities for basic control flow and program structure, and will use custom user-defined activities for domain-specific functionality.

In addition to supporting a XAML-based compositional approach to creating programs, workflow-based programs also benefit from a richer set of runtime services than traditional CLR-based programs. The WinFX workflow runtime can be hosted in any CLR AppDomain. The runtime allows workflows to be removed from memory (a technique called passivation) and later reloaded and resumed without making developers write explicit state management logic. The workflow runtime also provides common facilities for handling errors and compensating transactions to allow either automatic or customized undo logic to be specified for long-running units of work. In addition, you can take advantage of management services that allow the state of a given workflow program to be inspected either through eventing, tracking, or querying the workflow state.

Workflow 101

In WinFX, a workflow can be expressed in either declarative XAML or in imperative code using any .NET-targeted language. The use of XAML allows for workflows to be developed and modified in visual designers without requiring traditional C# or Visual Basic code. A workflow can be defined either purely in XAML or with XAML plus a codebehind file that contains C# or Visual Basic code. Pure XAML workflows have the advantage of being directly loadable at run time without a separate compilation step. Just to prove the point, in this article we'll use XAML wherever possible. In all cases, the XAML we show could be expressed in C# or Visual Basic, albeit more verbosely.

Creating a workflow program is a matter of producing the right XML description of the program. For a certain class of users, creating and editing XML in a text editor is an adequate (and sometimes ideal) solution. For the large number of users who prefer a visual designer, WinFX includes a visual workflow designer that can be embedded into any Windows-based application (shown in Figure 1). Again, for exposition purposes, this article shows workflow programs in their underlying XAML form, but in all cases, the XAML can be created and edited using the workflow designer.

Figure 1 Visual Workflow Designer (Close Figure 1)

To allow the developer using C# or Visual Basic to create and edit workflows, the workflow designer can be used within Visual Studio® 2005, including integration with the Visual Studio project system and debugger. Integration with the Visual Studio debugger allows you to use the familiar F5/F10/F11 key bindings to debug through both the workflow visualization and the underlying C# or Visual Basic code in a single debugging session (shown in Figure 2).

Figure 2 Visual Workflow Debugger (Close Figure 2)

Inside a Workflow

Fundamentally, a workflow is a tree of domain-specific program statements called activities. The concept of an activity is central to the workflow architecture—you should think of activities as domain specific opcodes, and workflows as programs that are written in terms of those opcodes.

WinFX provides a family of broadly applicable activities, several of which are discussed in this article. WinFX also allows you to write your own domain-specific activities either in XAML or in a CLR-compatible language. A WinFX activity is simply a CLR type that derives (directly or indirectly) from System.Workflow.ComponentModel.Activity. For example, the following C# code defines a working (yet useless) WinFX activity:

using System;
using System.Workflow.ComponentModel;

// bind an XML namespace to our CLR namespace for XAML
[assembly: XmlnsDefinition(
    "http://schemas.example.org/MyStuff", "MyStuff.Activities")]
namespace MyStuff.Activities
{
    public class NoOp : Activity {}
}
To use this activity, we can write a simple workflow in XAML like the following line of code:
<my:NoOp xmlns:my="http://schemas.example.org/MyStuff" />
This workflow consists of a simple tree that contains exactly one activity. Because our activity does nothing, when we load and run this workflow, nothing happens. More accurately, the workflow runtime loads the activity tree and calls the Execute method on the root activity. Because our activity didn't override the Execute method, nothing happens.

To get a better handle on how workflow-based programs work, let's define a somewhat more interesting activity, shown in Figure 3. This activity has two noteworthy characteristics. For one, it defines a public property named Text that can be initialized in the XAML-based workflow. More importantly, it overrides the Execute virtual method and responds by printing the Text property to the console. With this activity in place, we can now write a workflow that does something:

<my:WriteLine Text="Hello, world from WinFX."
    xmlns:my="http://schemas.example.org/MyStuff"
/>
When this is run, the workflow program prints out the string "Hello, world from WinFX."

Figure 3 A Simple Activity (Close Figure 3)
namespace MyStuff.Activities
{
    public class WriteLine : Activity
    {
        string text;

        public string Text
        {
            get { return text; }
            set { text = value; }
        }

        protected override ActivityExecutionStatus  Execute(
            ActivityExecutionContext aec)
        {
            Console.WriteLine(this.Text);
            return ActivityExecutionStatus.Closed;
        }
    }
}

Like all workflow programs, the previous workflow did not specify what environment to run in. Rather, it is up to the hosting environment (for example, SharePoint®, ASP.NET, a custom application server, or the Windows® shell) to load and run the workflow. The workflow runtime is embeddable in any environment. The runtime is exposed to hosting environments through the System.Workflow.Runtime.WorkflowRuntime class. Causing the workflow to activate is a two phase process: First, you load the program in memory by instantiating WorkflowRuntime and calling the CreateWorkflow method on it, passing either the XAML definition as XML or a compiled activity type. WorkflowRuntime.CreateWorkflow returns an object of type WorkflowInstance, which is a handle to the workflow program instance in memory. Next, in order to begin the actual workflow execution, you simply call WorkflowInstance.Start. The following code shows how to host the workflow runtime and run a workflow from its XAML definition:

// create and start an instance of the workflow runtime
WorkflowRuntime runtime = new WorkflowRuntime();
runtime.StartRuntime();
// get an XML reader to the XAML-based workflow definition
XmlReader xaml = XmlTextReader.Create("myworkflow.xaml");
// create a running instance of our workflow
WorkflowInstance instance = runtime.CreateWorkflow(xaml);
instance.Start();

The WorkflowInstance.Start call creates the initial data structures needed to run the workflow and then returns control to the hosting environment. The actual work that is represented by the workflow will be scheduled asynchronously by the runtime using either host-provided threads or the CLR thread pool.

Composite Activities

The WriteLine activity shown previously is an example of an atomic activity—that is, the activity contains all of its execution logic as opaque code. The WinFX workflow programming model also supports the notion of a composite activity, which is an activity that implements its execution logic in terms of one or more child activities. Composite activities are derived from the CompositeActivity class and have a well-known property (Activities) that contains the child activities. In XAML, these child activities are expressed as child elements of the composite.

WinFX includes a set of commonly used composite activities that simplify building workflows. The first one we'll look at is SequenceActivity, as shown in the following:

<SequenceActivity
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/workflow"
xmlns:my="http://schemas.example.org/MyStuff"
>
    <my:WriteLine Text="One"/>
    <my:WriteLine Text="Two"/>
    <my:WriteLine Text="Three"/>
    <my:WriteLine Text="Four"/>
</SequenceActivity>
When run, this workflow writes the following console output:
One
Two
Three
Four
The SequenceActivity executes every child activity in sequence. To support conditional execution, WinFX provides an activity for conditional execution called IfElseActivity.

The IfElseActivity contains one or more child activities, each of which binds a Boolean expression to a sequence of activities that execute if (and only if) the Boolean expression evaluates to true. Figure 4 shows a simple example using IfElseActivity.

Figure 4 Using IfElseActivity (Close Figure 4)
XAML for MyWorkflow
<!-- MyWorkflow.xaml -->
<SequenceActivity
x:Class="MyNamespace.MyWorkflow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/workflow"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:my="http://schemas.example.org/MyStuff"
>
    <IfElseActivity>
        <IfElseBranchActivity>
            <IfElseBranchActivity.Condition>
                <CodeCondition Condition="Is05"/>
            </IfElseBranchActivity.Condition>
            <my:WriteLine Text="Circa-Whidbey"/>
        </IfElseBranchActivity>
        <IfElseBranchActivity>
            <IfElseBranchActivity.Condition>
                <CodeCondition Condition="Is06"/>
            </IfElseBranchActivity.Condition>
            <my:WriteLine Text="Circa-Vista"/>
        </IfElseBranchActivity>
        <IfElseBranchActivity>
            <my:WriteLine Text="Unknown Era"/>
        </IfElseBranchActivity>
    </IfElseActivity>
</SequenceActivity>
Codebehind for MyWorkflow
using System;
using System.Workflow.ComponentModel;
using System.Workflow.Activities;

namespace MyNamespace
{
    public partial class MyWorkflow : SequenceActivity
    {
        void Is05(object s, ConditionalEventArgs e)
        {
            e.Result = (DateTime.Now.Year == 2005);
        }

        void Is06(object s, ConditionalEventArgs e)
        {
            e.Result = (DateTime.Now.Year == 2006);
        }
    }
}

This code corresponds to the visual designer view shown in Figure 5. Note that this XAML file uses the x:Class attribute to indicate that the file is defining a new type (MyNamespace.MyWorkflow) rather than an instance of an existing type. The use of x:Class allows us to have a codebehind file that contains C# or Visual Basic code, also shown in Figure 4.

Figure 5 Visual Representation of MyWorkflow.xaml
Figure 5 Visual Representation of MyWorkflow.xaml

In this example, we've defined two conditional expressions (Is05 and Is06) that are referenced by the workflow definition using the standard XAML notation for wiring up methods based on symbolic name.

The semantics of the IfElseActivity are simple. Each subordinate IfElseBranchActivity has a condition expression that is evaluated to determine which branch to execute. The first IfElseBranchActivity whose condition expression evaluates to true is selected to run. If no condition expressions evaluate to true, then the last branch will be selected provided it had no condition (which was the case in our workflow). The workflow just shown is equivalent to the following C#:

if (DateTime.Now.Year == 2005) Console.WriteLine("Circa-Whidbey");
else if (DateTime.Now.Year == 2006) Console.WriteLine("Circa-Vista");
else Console.WriteLine("Unknown era");

Though the workflow-based version of the program is more verbose, it is also more transparent, easier to change (especially by non-C# programmers), and can take advantage of the services of the workflow runtime such as program suspension, dehydration, and compensation.

The workflow example using IfElseActivity used CodeConditions for its Boolean expressions. Expressions can also be written in pure XAML without relying on a separate codebehind file (the syntactic details are beyond the scope of this article). Using XAML-based expressions allows the conditions used in the workflow to be expressed in a declarative format that is amenable to visual designers rather than a separate textual code editor that is specific to a given programming language.

In addition to sequencing and conditionals, WinFX includes an activity that models iteration: WhileActivity. Like IfElseBranchActivity, WhileActivity has a Condition property and a set of activity children. Figure 6 shows a simple workflow that uses WhileActivity as well as the codebehind file for this activity.

Figure 6 Using WhileActivity (Close Figure 6)
XAML for MyWorkflow
<!-- MyWorkflow.xaml -->
<SequenceActivity x:Class="MyNamespace.MyWorkflow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/workflow"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:my="http://schemas.example.org/MyStuff"
>
    <WhileActivity>
        <WhileActivity.Condition>
            <CodeCondition Condition="LoopTest"/>
        </WhileActivity.Condition>
        <my:WriteLine Text="Hello, WinFX"/>
        <CodeActivity ExecuteCode="IncrementCounter" />
    </WhileActivity>
</SequenceActivity>\
Codebehind for MyWorkflow
using System;
using System.Workflow.ComponentModel;
using System.Workflow.Activities;
namespace MyNamespace
{
    public partial class MyWorkflow : SequenceActivity
    {
        int count;

        void LoopTest(object s, ConditionalEventArgs e)
        {
            e.Result = count < 10;
        }

        void IncrementCounter(object s, EventArgs e)
        {
            count++;
        }
    }
}

Using this code, our workflow would print out the message "Hello, WinFX" ten times. Note that in this example, we've used the WinFX-provided CodeActivity to cause the underlying IncrementCounter method to execute at the end of our while loop's body.

Beyond the basic three composite activities that mimic traditional imperative programming languages, WinFX provides a range of more exotic activities that support forward-chaining rule evaluation, event-condition-action (ECA) workflows, and parallelism. The latter is expressed most directly using the WinFX ParallelActivity. Consider this XAML fragment that creates two parallel branches of execution:

<ParallelActivity
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/workflow"
xmlns:my="http://schemas.example.org/MyStuff"
>
    <SequenceActivity>
        <my:WriteLine Text="One"/>
        <my:WriteLine Text="Two"/>
    </SequenceActivity>
    <SequenceActivity>
        <my:WriteLine Text="Three"/>
        <my:WriteLine Text="Four"/>
    </SequenceActivity>
</ParallelActivity>

When run, the two sequential branches are scheduled to run in parallel, with the execution logic of ParallelActivity determining the order. Given this XAML, the following would be legal output:

One
Three
Two
Four
This would also be legal output:
Three
One
Four
Two

ParallelActivity makes no promise about the exact order of execution across branches. However, ParallelActivity will not complete until all branches are finished executing. This means that, given the XAML fragment shown in Figure 7, the initial line that is output will always be "Zero" and the sixth and final line output will always be "Five." The intervening output is again nondeterministic.

Figure 7 Using ParallelActivity (Close Figure 7)
<SequenceActivity
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/workflow"
xmlns:my="http://schemas.example.org/MyStuff"
>
    <my:WriteLine Text="Zero"/>
    <ParallelActivity>
        <SequenceActivity>
            <my:WriteLine Text="One"/>
            <my:WriteLine Text="Two"/>
       </SequenceActivity>
       <SequenceActivity>
           <my:WriteLine Text="Three"/>
           <my:WriteLine Text="Four"/>
       </SequenceActivity>
    </ParallelActivity>
    <my:WriteLine Text="Five"/>
</SequenceActivity>

Inside Activity Execution

One of the key architectural principles of the Windows Workflow Foundation is that specific program semantics are always the responsibility of the individual activities and are never the responsibility of the workflow runtime. The architecture anticipates that developers as well as domain experts will define custom activities that are tailored to given application domains. To keep domain specific semantics out of the runtime, activities and the workflow runtime communicate via a well-defined contract that is expressed using virtual methods on the Activity base class. The workflow runtime does not treat any of the activities shipped as a part of WinFX any differently than the ones you create. Rather, the workflow runtime uses the same contract to interact with both SequenceActivity and a custom activity like the WriteLine activity we've used throughout this article.

To understand how activities and the runtime relate, let's look at the state machine that each activity must implement. That state machine is reflected by the ActivityExecutionStatus enumeration and is shown in Figure 8. Additionally, the Activity base class exposes a CLR event for each state that is raised whenever a transition occurs.

Figure 8 Activity State Machine
Figure 8 Activity State Machine

Each activity is in exactly one of the six states at any given point in its lifetime—initialized, executing, canceling, faulting, compensating, and closed. The dotted transitions represent the final transitions to the closed state beyond which the activity will not transition any further. In general, activity state transition can be influenced directly by the workflow runtime or by the parent activity requesting the workflow runtime to schedule the transition for its child. In either case, the workflow runtime coordinates (and enforces) the transition of activities from one state to another.

The contract between the workflow runtime and an activity is expressed in terms of the following protected virtual methods on the Activity base class:

void Initialize(IServiceProvider provider)
ActivityExecutionStatus Execute(ActivityExecutionContext aec)
ActivityExecutionStatus HandleFault(ActivityExecutionContext aec)
ActivityExecutionStatus Cancel(ActivityExecutionContext aec)

Additionally, as a part of the workflow runtime-activity contract, compensatable activities are required to implement ICompensatableActivity which has one method called Compensate:

namespace System.Workflow.ComponentModel {
    public interface ICompensatableActivity {
       ActivityExecutionStatus.Compensate(ActivityExecutionContextaec)
    }
}
Except for the Initialize method (which is called synchronously when the workflow is first started by the host), all of the methods return the state that the activity is in when it returns control to the runtime. For operations that can execute quickly and synchronously, the activity will return the enumeration value for the next state (typically closed):
protected override ActivityExecutionStatus Execute(
    ActivityExecutionContext aec)
{
    DoWorkSynchronously();
    // indicate that we're now in the Closed state
    return ActivityExecutionStatus.Closed;
}
In contrast, activities that enqueue asynchronous work must indicate that they are still executing:
protected override ActivityExecutionStatus Execute(
    ActivityExecutionContext aec)
{
    EnqueueAsynchronousWork();
    // indicate that we're still executing
    return ActivityExecutionStatus.Executing;
}
It is then the responsibility of the activity to inform the runtime when it transitions to the closed state. This is typically done in response to some event raised by the runtime itself:
// Activity's event handler that is registered with the runtime
void LastStageOfAsyncWork(object sender, EventArgs e)
{
    // grab a reference to the runtime
    ActivityExecutionContext aec = (ActivityExecutionContext)sender;
    // inform the runtime that we're now
    // closed
    aec.CloseActivity();
}

This event handler demonstrates an important aspect of activity development. Each time the runtime calls into your activity, it provides a reference to itself of type ActivityExecutionContext. This reference is the only API into the runtime for activities to use. More importantly, the reference that is passed to you is only valid for as long as your CLR method is executing. Once your method returns control to the runtime, the context reference is no longer valid. The runtime will enforce this with an exception if you make the mistake of caching the reference in a field for later use.

Despite the inherently asynchronous nature of the runtime/activity contract, the runtime will never make concurrent calls to a given activity. Rather, any given activity is guaranteed to have at most one method invocation executing at a given instant. This vastly simplifies the task of writing activities in languages like C# or Visual Basic.

Normal Activity Execution

Under normal circumstances, an activity experiences two transitions: from initialized to executing and from executing to closed. The first transition (from initialized to executing) occurs when the workflow runtime is able to assemble the required resources for the activity's execution and schedules the Execute method on the activity. The second transition (from executing to closed) happens either synchronously when the activity's Execute method returns ActivityExecutionStatus.Closed or, if the Execute method returns ActivityExecutionStatus.Executing, asynchronously when a subsequent event handler on the activity runs and calls ActivityExecutionContext.CloseActivity to communicate the state transition to the runtime.

The Execute method of a composite activity typically asks the workflow runtime to schedule the execution of its child activities, and it subscribes to their status changes asynchronously. It then remains in the executing state at least while one of its children is in the executing state.

To better understand the normal activity execution protocol, we can look at the implementation of the SequenceActivity. For simplicity, we will just focus on the normal execution path of activity (initialized to executing to closed), and thus only look at the Execute method.

In the implementation of the Execute method, SequenceActivity subscribes to the closed event of its first child and requests the workflow runtime to schedule the execution of its first child by calling the ExecuteActivity method on the runtime context, as you can see in Figure 9.

Figure 9 SequenceActivity's Execute Implementation (Close Figure 9)
namespace System.Workflow.Activities
{
    public class SequenceActivity : CompositeActivity
    {
        protected override ActivityExecutionStatus Execute(
            ActivityExecutionContext aec)
        {
            // if we have no children, then transition to closed
            if (this.EnabledActivities.Count == 0)
                return ActivityExecutionStatus.Closed;

            // otherwise, schedule the first child after
            // registering for its Closed event
            Activity child = this.EnabledActivities[0];
            child.Closed += OnChildClosed;
            aec.ExecuteActivity(child);

            return ActivityExecutionStatus.Executing;
        }

        ...
    }
}

The call to ExecuteActivity causes the scheduler to run the child activity using the standard initialized-to-executing-to-closed sequence. When the child gets scheduled, the workflow runtime will place the child in the executing state and call the Execute method of the child. Eventually, when the child goes to the closed state, the event handler for the Closed event (SequenceActivity.OnChildClosed) will get called by the runtime, passing a new ActivityExecutionContext as the first parameter. Figure 10 shows the definition of that event handler for SequenceActivity.

Figure 10 SequenceActivity Closed Event Handler (Close Figure 10)
void OnChildClosed(object sender, ActivityStatusChangeEventArgs e)
{
    ActivityExecutionContext aec = (ActivityExecutionContext)sender;
    Activity child = e.ForActivity;

    // unregister event handler on child
    child.Closed -= OnChildClosed;

    // find index of child
    int index = EnabledActivities.IndexOf(e.ForActivity);

    // if this was the last child, transition to Closed
    if (index == EnabledActivities.Count—1)
    {
        aec.CloseActivity();
    }
    else
    {
        // otherwise schedule the next one
        child = EnabledActivities[index + 1];
        child.Closed += OnChildClosed;
        aec.ExecuteActivity(child);
    }
}

In this event handler, we check to see if there are any remaining children. If there are, we schedule the next one, using the same event handler method we used before as the completion routine. If there are no more children to run, then we inform the runtime that we are ready to transition to the Closed state by calling the CloseActivity method.

Context and State Management

Astute readers will notice that activities interact with the runtime through the ActivityExecutionContext (AEC). Conceptually, the AEC acts as a serializable execution environment in which activities get executed and the object state of the activities is automatically managed in the environment. The workflow runtime allows activities to create new persistent execution environments during normal activity execution. Activities that require their children to execute more than once need to create new AECs for each execution. The canonical example of this is WhileActivity, which mimics the standard C while loop.

Figure 11 shows the multiple AECs spawned for each of the iterations of a WhileActivity. Assuming that the WhileActivity in the program iterates three times, three instances of SequenceActivity within the WhileActivity are executed dynamically in distinct AECs, each with its own state. By giving each iteration of the while loop its own AEC, the runtime has the ability to undo the execution of each iteration when running compensation logic.

Figure 11 Activity Execution Contexts Spawned for Iteration
Figure 11 Activity Execution Contexts Spawned for Iteration

Workflows can run for long periods of time. At any given instant, a workflow instance may be considered idle, which means that the runtime has no runnable work for any of the workflow's activities. When a workflow instance is idle, its state may be serialized to a host-provided store and removed from memory. This process is called passivation. The serialized state of the workflow instance can be used to reactivate the instance in memory, at which point the instance is restored to its state and is available to do work. Reactiviation typically happens when the host detects that new work is available, usually by noticing that an external event has occurred.

Consider the following simple workflow program that uses the built-in DelayActivity to cause the workflow to go idle in the middle of the sequence:

<SequenceActivity
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/workflow"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:my="http://schemas.example.org/MyStuff"
>
    <my:WriteLine Text="One"/>
    <my:WriteLine Text="Two"/>
    <DelayActivity TimeoutDuration="00:10:00" />
    <my:WriteLine Text="Three"/>
    <my:WriteLine Text="Four"/>
</SequenceActivity>
In these snippets, when DelayActivity executes, it sets up a subscription for a timer and returns ActivityExecutionStatus.Executing. At this point, the workflow instance has no runnable work and is therefore considered idle. At idle-time, the workflow runtime will attempt to persist the state of the workflow instance using a host-provided persistence service. The host environment may provide an implementation of the abstract WorkflowPersistenceService type when initializing the runtime. If the host provides a concrete implementation of this type (through the WorkflowRuntime.AddService method), then the runtime will call the host's SaveWorkflowInstanceState method, giving the host a chance to save the state to a persistent store. Each workflow instance is identified by a unique runtime-generated GUID, which the store is expected to use to identify the persisted state of the instance for later retrieval.

The runtime persists the workflow instance at idle-time for reliability purposes. By default, the instance is retained in memory. The host can cause the workflow instance to be evicted from memory either by setting the runtime-wide UnloadOnIdle property to true or by calling the WorkflowInstance.Unload method on a specific instance. The host can detect when an instance goes idle by registering for the WorkflowIdled event on the runtime, like so:

WorkflowRuntime runtime = new WorkflowRuntime();
runtime.StartRuntime();
runtime.WorkflowIdled += delegate(object s, WorkflowEventArgs e)

{
...
};

To force a saved workflow to be reloaded into memory from the host's persistence service, you can call the WorkflowRuntime.GetWorkflow method, passing in the instance's unique GUID.

Where Are We?

We barely scratched the surface of the rich workflow programming model and capabilities provided by Windows Workflow Foundation. Windows Workflow Foundation provides a way to declaratively and more naturally express the semantics of your application in terms of activities, including the program control flow, transactions, concurrency, synchronization, exception handling, and interactions with other applications. It also provides a rich set of services beyond those available in the base CLR, including automatic program persistence, compensating transactions, and runtime inspection of the program state. The workflow runtime can be hosted in any CLR app domain and thus can be embedded in any application or application container. You can download the WinFX beta from Windows Vista and WinFX Beta and start writing activities today.

Don Box is an architect in the Microsoft Connected Systems Division where he works on programming models and plumbing to support building programs that talk to other programs. Prior to joining Microsoft, Don led a small but multi-national cult of IUnknown worshippers. Reach Don at www.pluralsight.com/blogs/dbox.

Dharma Shukla is a development lead on the Windows Workflow Foundation team at Microsoft responsible for the workflow programming model and developer tools. He is also working on a Windows Workflow book. Reach Dharma at www.dharmashukla.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Dazzling Graphics: Top Ten UI Development Breakthroughs In Windows Presentation Foundation

New information has been added to this article since publication.
Refer to the Editor's Update below.

Dazzling Graphics

Top Ten UI Development Breakthroughs In Windows Presentation Foundation

Ian Griffiths and Chris Sells
This article is based upon a prerelease version of WinFX. All information contained herein is subject to change.
This article discusses:
  • Declarative programming and data-driven UI
  • The drawing object model and the WPF content model
  • Rich text support, adaptable layout, and lookless controls
This article uses the following technologies:
WinFX, Visual Studio 2005
Contents
10. Advanced Graphics
9. Drawing Object Model
8. Rich Application Text
7. Adaptable UI Layout
6. Flexible Content Model
5. Lookless Controls
4. Data-Driven UI
3. Consistent Styles
2. Triggers
1. Declarative Programming
Where Are We?

T
he current Win32-based Windows UI graphics subsystem, found in Windows® XP, has been around for nearly 20 years. It's aging and limited, and as a result, user interface development has been somewhat, well, constrained at best.

The Windows Presentation Foundation (WPF), which is built on the .NET Framework, provides new techniques for developing applications, and makes better use of current hardware and technologies. In this article, we'll show you 10 of the most significant advances that make WPF superior to its Win32® forebears. While some of the specific details are likely to change in the final release of WPF, we expect the key concepts to remain the same. For details on how you can start working with WPF today, see the sidebar "Getting the Bits."

10. Advanced Graphics

Over the years, drawing functionality in Windows has improved steadily, though incrementally. Prior to WPF, GDI+ was the pinnacle of 2D graphics in Windows. GDI+ provides fully scalable drawing primitives including Bézier paths, cardinal splines, text, and various shapes. It supports a range of advanced gradient and texture fill styles. And it offers full support for partial transparency and anti-aliasing.

Enhanced Drawing Capabilities Based on just its basic drawing capabilities, WPF looks like a major step forward. It makes considerably better use of the underlying graphics hardware, so it performs much faster than GDI+. It offers as rich a set of drawing features, and brings a few new capabilities to the table. For example, while GDI+ allows arbitrary clip regions to be applied when drawing, WPF takes this feature even further: the opacity mask feature not only lets you clip output to any shape, but also lets you modify the opacity of any arbitrary content.

Figure 1 Repeating Pattern for Use as an Opacity Mask
Figure 1 Repeating Pattern for Use as an Opacity Mask

For example, consider the repeating graphical pattern in Figure 1. You can create a brush, using some Extensible Application Markup Language (XAML) code, that lets you paint with this pattern. (XAML, by the way, is an XML-based language for building trees of .NET objects. It provides a compact and convenient way of creating WPF objects.) The following shows some XAML code that defines such a brush:

<DrawingBrush Viewbox="0,0,1,1" ViewboxUnits="Absolute"
        Viewport="0,0,0.07,0.2" TileMode="Tile">
    <DrawingBrush.Drawing>
        <GeometryDrawing Brush="HorizontalGradient #f000 #0000">
            <GeometryDrawing.Geometry>
                <RectangleGeometry Rect="0,0,1,0.6" />
            </GeometryDrawing.Geometry>
        </GeometryDrawing>
    </DrawingBrush.Drawing>
</DrawingBrush>

This brush can be used as the OpacityMask for any user interface element. Figure 2 shows some yellow text on top of a graduated fill background, using this mask to modulate the opacity.

Figure 2 Text with an Opacity Mask Applied
Figure 2 Text with an Opacity Mask Applied

Fewer Control Constraints These new drawing capabilities are nice to have, but WPF makes some deeper changes. A more profound difference is that drawings are no longer confined to a single control. Consider any Win32-based application—any Windows-based application built using Windows Forms, MFC, Windows Template Library, or Visual Basic®, as well as any application that uses the Win32 API directly.

Getting the Bits

If you want to start playing with WPF today, you can download and install the WinFX™ runtime and SDK to your machine. This is prerelease software and, as such, has not been subjected to the usual level of testing given to shipping Microsoft software. Thus, do not install this on a machine you need to use for day-to-day work.

This article is based upon the September 2005 community technology preview (CTP) release of WinFX, a post-Beta 1 release. The September CTP WinFX runtime can be downloaded from September CTP and the corresponding WinFX SDK can be downloaded from WinFX SDK. In addition, you can also download the Visual Studio Extensions for WinFX from Visual Studio Extensions.

Be aware that the September 2005 CTP of WinFX is only compatible with Visual Studio 2005 Beta 2. It will not run with more recent builds. For more information about how to get up and running with the CTP release of WinFX, see Rob Relyea's longhornblogs.com post.

[ Editor's Update - 10/05/2006: The current version of the .NET Framework 3.0, formerly known as WinFX, is available from http://msdn.microsoft.com/windowsvista/downloads/products/getthebeta/default.aspx. The version now available differs slightly from the version on which this article is based.]

With Win32, and the associated GDI and GDI+ drawing APIs, each control in the UI owns its part of the window exclusively. Windows are carved up by the controls into a set of disjointed regions that are usually, but not necessarily, rectangular. If you pick any single pixel on a window, there will be exactly one control responsible for drawing that pixel.

The problem with this approach is that it makes certain UI effects impossible. A commonly asked question about Windows Forms is: "How do I draw something so that it appears to be on top of all the controls on the form?" Because each control has exclusive ownership of its part of the screen, there is no straightforward way to achieve this effect. While there are some viable workarounds in the GDI+ environment, these tend not to work if you want to use partially transparent drawing effects. GDI+ offers partial transparency and anti-aliasing for drawing operations, but only when performed within the confines of a single control.

WPF breaks free from this limitation. No single control or UI element has exclusive ownership of any part of the window in a WPF application. You can create partially transparent UI elements, or controls that are not visually constrained to their logical rectangle. For example, it's possible for a control to cast a shadow that falls outside of its logical region.

Take Figure 3, for example. This image shows two buttons. The top button is semi-transparent and the bottom button is visible through the top button. There is also a partially transparent ellipse mixed in. This sort of overlapping design simply isn't possible with Windows Forms or other Win32-based systems. (Windows Forms offers pseudo-transparency, which can work around some of these limitations, but it only works for very basic designs.)

Figure 3 Controls and Transparency
Figure 3 Controls and Transparency

In effect, the whole window becomes a single drawing surface that each of the parts of the UI contribute to. This helps enable another important WPF graphics feature: resolution independence.

Resolution Independence The resolution of flat-panel displays has been gradually increasing. 150 dpi displays are commonplace, 200 dpi displays are readily available, and resolutions are likely to continue increasing. Scalability is crucial to enable the use of high-dpi displays. Without the ability to scale an application's UI, the UI simply becomes smaller as the resolution increases, typically becoming unusable at anything much higher than 150 dpi.

Although GDI+ enables resolution-independent drawing, as did GDI, it is very difficult to write visually scalable applications using these technologies. If you apply a scaling or rotation transform with these APIs, it only affects the drawings. There is no effect on the sizes and positions of controls in the window. Technically, the problem is that while GDI and GDI+ are scalable, USER32 isn't. You can scale drawings, but you can't easily scale the way that a window's space is carved up by controls.

Because WPF composes all of the controls in the window as a single drawing, rather than segregating each control to its own region, it is easy to apply scaling or rotation transformations to the UI as a whole. Thus, since we can scale any WPF UI up or down, WPF applications are effectively resolution-independent. Images remain sharp and clear, as opposed to the blur you would expect from simple bitmap scaling.

Built-In Animation and Video WPF provides built-in support for animation and video, allowing you to build dynamic UIs more easily. Though you can animate parts of your UI with GDI or GDI+, the process requires you to set up a timer and write code to update the display regularly. With WPF, you can instruct the system to animate elements automatically.

Support for animation is comprehensive, letting you animate any part of the user interface. This example shows a rectangle whose width is repeatedly animated from 20 to 200 every five seconds:

<Rectangle Fill="Blue" Height="100">
  <Rectangle.Triggers>
    <EventTrigger
        RoutedEvent="Rectangle.Loaded">
      <BeginStoryboard>
        <Storyboard TargetProperty=
            "(Rectangle.Width)">
          <DoubleAnimation From="20" To="200"
              Duration="0:0:5"
              RepeatBehavior="Forever" />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>
(This example also illustrates another new feature—the Rectangle element uses WPF's object model for representing drawings.)

WPF also lets you incorporate video with a single element, as shown here:

<MediaElement Name="myVideo"
    Source="C:\WINDOWS\system32\oobe\images\
        intro.wmv"
    Width="450" Height="400" >
This is a user interface element like any other user interface element. You can put a MediaElement anywhere you might put a button or an ellipse. You can apply all the usual visual operations, such as scaling, rotation, or clipping. It's just a UI element that happens to show video clips.

9. Drawing Object Model
Figure 4 Repainting in Win32
Figure 4 Repainting in Win32

The WPF graphics system offers a completely new programming style. With Win32-based UI technologies, if you want to draw customized graphics, you must write code that responds to repaint requests, drawing the details on demand. This code effectively paints directly to the screen. If that part of the screen becomes obscured by another window and then uncovered, Windows sends a message requesting that this be redrawn. If you want to change the appearance of your graphics, you need to instruct Windows to invalidate the relevant areas of the screen in order to initiate a repaint. Figure 4 illustrates this system.

Figure 5 Repainting in WPF
Figure 5 Repainting in WPF

While WPF still supports this model, it offers a simpler alternative. In WPF, you can create objects that represent the drawing primitives you wish to use and add these to your UI. You can use either XAML or traditional application code to do this. Once these objects are in the user interface element tree, they are rendered by WPF—you do not need to write any code to handle repaint requests. (Figure 5 shows this system.) Moreover, when you change a property of such an object, such as the Width property of a Rectangle, the display is updated automatically. In other words, you can use drawing primitives in the same way that you use controls—just create them and set their properties, leaving them to take care of the business of drawing themselves onto the screen.

The code in Figure 6 creates 10 Rectangle elements and adds them to a Window. Whenever the Window receives mouse wheel events, it adjusts the Width of all the Rectangle elements (see Figure 7).

Figure 6 Drawing Rectangles (Close Figure 6)
public partial class Window1 : Window
{
    private Rectangle[] rectangles = new Rectangle[10];

    public Window1()
    {
        InitializeComponent();

        Canvas c = new Canvas();
        this.Content = c;

        Random r = new Random();
        byte[] colors = new byte[3];
        for (int i = 0; i < rectangles.Length; ++i)
        {
            rectangles[i] = new Rectangle();
            r.NextBytes(colors);
            rectangles[i].Fill = new SolidColorBrush(Color.FromArgb(
                255, colors[0], colors[1], colors[2]));
            rectangles[i].Width = 200;
            rectangles[i].Height = 100;
            Canvas.SetLeft(rectangles[i], i*20);
            Canvas.SetTop(rectangles[i], i*10);

            c.Children.Add(rectangles[i]);
        }
    }

    protected override void OnMouseWheel(MouseWheelEventArgs e)
    {
        int diff = e.Delta / 10;
        for (int i = 0; i < rectangles.Length; ++i)
        {
            if (rectangles[i].Width + diff > 0)
            {
                rectangles[i].Width += diff;
            }
        }
    }
}

Notice that the OnMouseWheel method does not explicitly trigger any redrawing. In fact, there isn't a repaint or redraw handler in sight. In a Win32-based program, it would be necessary to provide such a handler and to invalidate the window in order to get the display updated. In WPF, this is unnecessary.

Figure 7 Adjusting Rectangle Width
Figure 7 Adjusting Rectangle Width

The drawing object model makes it much easier than before to integrate graphical elements into applications. But good presentation isn't just about graphics—the quality of your textual presentation is also important.

8. Rich Application Text

Text in Windows applications has long fallen short of the text capabilities offered by Web applications. HTML and cascading style sheets (CSS) provide Web applications with a host of text layout and formatting features. Windows applications, on the other hand, have had little support.

Of course, it is possible to display richly formatted text using Win32—after all, Microsoft® Internet Explorer and Microsoft Word are both Windows applications—but the programming task is monumental. Alternatively, you can embed the MSHTML control and use it to display text. However, this technique does not fully integrate document-quality text rendering directly into your application, meaning certain application areas, such as text styles within a button label, can't benefit from MSHTML. Unfortunately, you're left with a sharp contrast between the document display managed by MSHTML and the rest of your application.

WPF makes it easier to integrate richly formatted text into the user interface. Even the simplest text-handling element, the TextBlock control, lets you use any mixture of font style, size, color, and typeface. Here's an example consisting of a single TextBlock element:

<TextBlock FontFamily="Calibri" FontSize="11pt">
    Hello, <Bold>world!</Bold>
    <Span FontFamily="Old English Text MT"
        FontSize="24pt">This</Span>
    is a
    <Italic>single</Italic>
    <Span FontFamily="Consolas" Foreground="Blue">
    &lt;<Span Foreground="DarkRed">TextBlock</Span>/&gt;
    </Span>
    <Hyperlink>element</Hyperlink>.
</TextBlock>

As you can see in Figure 8, even within this one element, we can achieve a wide variety of styles. Compared to the humble Label control in Windows Forms, this is pretty powerful stuff. Although, even if all you want is the plain text that you can get with Label, TextBlock is still very easy to use. To forgo the fancy formatting, you can just give it a simple string to display.

Figure 8 TextBlock with Rich Formatting
Figure 8 TextBlock with Rich Formatting

For slightly larger volumes of text, WPF offers the TextFlow control, which provides document-like features. For example, you can include numbered or bulleted lists and floating figures. TextFlow offers a feature set similar to HTML, but unlike HTML, it is designed to integrate seamlessly with WPF controls.

WPF removes the hard boundary between document display and other application features, making it easy to build applications with much more advanced use of text and typography. Since the TextFlow element is an integrated part of WPF, you can place this element anywhere in your UI. And you're free to mix user interface elements, for example, placing buttons right in the middle of text (see Figure 9).

Figure 9 Mixing Controls and Text
Figure 9 Mixing Controls and Text

Furthermore, WPF offers advanced text layout features for applications that need to display large volumes of text (these are especially useful where text presentation is the primary purpose of the application). For instance, the DocumentViewer control displays text documents, using advanced text layout heuristics to optimize the readability of text for the available space and resolution. This text layout technology incorporates the fruits of extensive research Microsoft has performed regarding on-screen readability.

Getting your text to look right is crucial to building a great-looking application overall. But more important still is making sure that all the parts of your application's visuals appear in the right place at the right size, regardless of how the user may choose to arrange her windows.

7. Adaptable UI Layout

Most application designers don't have the luxury of knowing the size at which their programs will be run. There is enormous variation in screen size from one PC to another. Moreover, some users run applications maximized—filling the entire screen—while others use smaller windows so they can see several applications at once.

This doesn't stop some designers from designing a UI that only works at one size (a practice that, for some reason, is particularly rife on the Web). However, if you want to serve the needs of your users, you should produce a UI with a layout that can adapt itself to the space available. Such adaptability is also useful if you want your application to be accessible and localizable. A sufficiently adaptable UI won't need a redesign to accommodate rendering at a low screen resolution or translation into a language where all the words are twice as long.

WPF offers a powerful toolbox of layout elements that make it easy to build a UI with an adaptable layout. The simple docking and anchoring techniques familiar to Windows Forms developers are available. It is also possible to use a flow-style layout, where UI elements are arranged just as words would be arranged in text: left to right, and then flowing onto the start of the next line once the current line is full. WPF also offers a Grid. This is an extremely powerful layout element which enables grid-like or table-like layouts—these will be familiar to anyone who has abused the HTML table element. Because of its general-purpose applicability, the Grid is probably used more than all the other layout panels put together.

As an example of one simple use, Figure 10 shows the code for a grid with two rows and two columns. Inside the Grid element, the ColumnDefinition and RowDefinition elements define the properties of the columns and rows in the Grid. Each UI element then has a row and column property that designates where it appears in the Grid. Figure 11 shows the results.

Figure 10 Building a Grid (Close Figure 10)
<Grid>

    <Grid.ColumnDefintions>
        <ColumnDefinition Width="*" />
        <ColumnDefinition Width="5*" />
    </Grid.ColumnDefintions>

    <Grid.RowDefintions>
        <RowDefinition Height="Auto" />
        <RowDefinition Height="*" />
    </Grid.RowDefintions>

    <TextBlock Grid.Row="0" Grid.Column="0"
        Grid.ColumnSpan="2"
        Background="Cyan" FontSize="18pt"
        TextAlignment="Center">

        Title
    </TextBlock>

    <ListBox Grid.Row="1" Grid.Column="0">
        <ListBoxItem>List</ListBoxItem>
        <ListBoxItem>On</ListBoxItem>
        <ListBoxItem>Left</ListBoxItem>
        <ListBoxItem>Hand</ListBoxItem>
        <ListBoxItem>Side</ListBoxItem>
    </ListBox>

    <Rectangle Grid.Row="1" Grid.Column="1"
        Fill="LimeGreen" />
    <Ellipse Grid.Row="1" Grid.Column="1"
        Fill="White" Margin="10" />
</Grid>
Figure 11 The Rendered Grid
Figure 11 The Rendered Grid

This example also shows that items do not need to be constrained to a single cell in the grid. The Title TextBlock uses the Grid.ColumnSpan property to span both columns. This is how it manages to fill the entire width of the grid.

Additionally, this example illustrates the use of automatic and proportional sizing. The first row's Height has been set to Auto, which tells the grid to make the row exactly high enough to hold that row's contents and no higher. The second row ends up filling the remaining available space. The column widths use a different technique; they have been set to * and 5*. The * syntax indicates that rows or columns should share the available space. The number indicates the proportions used to allot the space. The column with a width of 5* gets five times as much space as the one with *. (This techique is not unlike assigning percentages in HTML tables.) You can also use decimals, such as 1.2*.

Layout elements can be nested, making it possible to produce extremely complex layouts. Moreover, if you need something not available in the built-in layout types, you can easily write new layout element types. While Windows Forms offered significantly better automatic layout facilities than its predecessors (especially in the .NET Framework 2.0), WPF takes UI layout capabilities to a whole new level.

6. Flexible Content Model

Many controls provide placeholders for some kind of application-specific content. Buttons, for example, usually have a label to let the user know what the button is for. Some controls have several placeholders: tab controls allow a caption to be placed in each tab, as well as content in the tab body. Most UI systems, however, are somewhat restrictive about the content that can be used as captions and labels. Often, only plain text is allowed. In some cases, a control will go as far as letting you display a bitmap alongside or in place of the text.

WPF offers a radically different approach. Instead, its content model allows any content to be used liberally in such a placeholder. For example, the Button control doesn't require you to supply text—you can put anything you want in there. You could put a bitmap, a drawing, or even a layout panel containing several drawings, as shown in this code:

<Button>
    <StackPanel Orientation="Horizontal">
        <Ellipse Fill="Navy" Stroke="Yellow"
            Width="30" Height="20" />
        <TextBlock VerticalAlignment="Center">
            You can put anything in a button!
        </TextBlock>
    </StackPanel>
</Button>

It's even technically possible to use a menu or a textbox as the content of a Button, although that would be a crime against usability. Figure 12 shows some of the possibilities, although this particular instance illustrates some poor choices.

Figure 12 Abusing the Content Model
Figure 12 Abusing the Content Model

Even though this example is not useful, it does illustrate the flexibility of the content model. Its lack of restrictions enables a host of visual design possibilities that, in the past, would have typically required owner-draw code to be written. This new model provides a great deal of power for visual designers—designers can now prototype designs that once would have required input from developers.

5. Lookless Controls

While the content model lets you customize the contents of controls, WPF's lookless approach to controls enables you to customize a control's entire appearance. This sort of task has traditionally required a custom control to be written, because appearance has always been an integral part of the control. In many UI technologies, the very definition of a control is essentially a visible component. But this is not the case in WPF.

WPF takes the radical step of disassociating the appearance from the control. Most WPF controls are lookless, meaning they just provide behavior. The appearance is supplied separately in the form of a template. A control typically provides a built-in template so it has a default appearance. Better yet, it can supply several templates—perhaps one to match the XP Luna theme, another to blend in with Aero, and a third to match the classic Windows 2000 theme. However, the template that defines a control's default appearance can be replaced in order to change the look of the control.

Figure 13 shows an example of replacing the template of a button. The ContentPresenter element is a placeholder that tells WPF where the button's content should be added into the results (and without this, the button's caption wouldn't appear).

Figure 13 Customizing Button Templates (Close Figure 13)
<Button>
    <Button.Template>
    <ControlTemplate>
        <Grid>
        <Rectangle Fill="VerticalGradient White Red" Stroke="DarkRed"
            RadiusX="10" RadiusY="10" />
        <ContentPresenter Content="{TemplateBinding Button.Content}"
            Margin="{TemplateBinding Padding}"
            HorizontalAlignment="{TemplateBinding
                HorizontalContentAlignment}"
            VerticalAlignment="{TemplateBinding
                VerticalContentAlignment}" />
        </Grid>
    </ControlTemplate>
    </Button.Template>
    Click me!
</Button>

When developing a reusable UI element, you need to think carefully about what type of element it is. If you are used to using Windows Forms, you might be naturally inclined to make all visible objects derive from Control. This, however, is often the wrong choice in WPF. If the purpose of your component is to provide a particular interaction—say, the checkable behavior of a Button or the display and selection behavior of a ListBox—then it should be a control. But if its job is to provide a particular visual appearance with no intrinsic behavior, it should not be a control; instead, it should derive from FrameworkElement, like Ellipse and Rectangle do by way of the Shape class.

Controls have behavior and (usually) a replaceable appearance. This makes WPF controls much more flexible than Windows Forms or Win32 UI components—you can change anything about how the control looks without losing any of its functionality.

4. Data-Driven UI

We've already discussed how the WPF content model allows you to place whatever content you want inside certain kinds of controls. In our example, we simply embedded some fixed content. But hard-coding content is pretty dull. Wouldn't it be much more exciting to create an application that dynamically changes what it displays.

It turns out you can do this, too. Controls that support the WPF content model allow you to use anything as the content, even non-UI objects. That means you can write the sort of code shown in Figure 14. This may seem like a peculiar thing to do; you probably expect a button's content to consist of either text or some UI elements. But in this code, we are setting the button to be an instance of a custom class.

Figure 14 Object as Content (Close Figure 14)
public class Person
{
    private string m_name;
    private DateTime m_dateOfBirth;

    public string Name
    {
        get { return m_name; }
        set { m_name = value; }
    }

    public DateTime DateOfBirth
    {
        get { return m_dateOfBirth; }
        set { m_dateOfBirth = value; }
    }
}

...

Person ian = new Person();
ian.Name = "Ian";
ian.DateOfBirth = new DateTime(1973, 10, 14);
myButton.Content = ian;

Admittedly, the default behavior isn't terribly useful, as shown in Figure 15. WPF didn't know what to do with the object, so it just called ToString. This may not feel like progress. However, even though WPF doesn't know what to do with this object by default, we can tell it what to do by supplying a data template.

Figure 15

A data template is essentially a set of instructions that tells WPF how to display an object. You can explicitly assign a data template in a particular context. For example, in this case you could set the Button's ContentTemplate property to use the Name property from the Person object, as follows:

<Button x:Name="myButton">
    <Button.ContentTemplate>
        <DataTemplate>
            <TextBlock Text="{Binding Name}" />
        </DataTemplate>
    </Button.ContentTemplate>
</Button>

You don't have to set the data template on a particular control. If you prefer, you can associate a data template with a particular type, indicating that WPF should use the specified template whenever it encounters that particular type as content. This is putting the template into the resources section (see Figure 16). Note that this template is a little more ambitious. It displays all of the information from the source data, as shown in Figure 17.

Figure 16 Applying the Data Template as a Resource (Close Figure 16)
<?Mapping XmlNamespace="local" ClrNamespace="AvalonApplication1"?>
<Window x:Class="AvalonApplication1.Window1"
    xmlns="http://schemas.microsoft.com/winfx/avalon/2005"
    xmlns:x="http://schemas.microsoft.com/winfx/xaml/2005"
    xmlns:local="local"
    Title="AvalonApplication1">
    <Window.Resources>
    <DataTemplate DataType="{x:Type local:Person}">
        <StackPanel Orientation="Horizontal">
            <TextBlock FontWeight="Bold">Name:</TextBlock>
            <TextBlock Text="{Binding Name}" Margin="2,0,10,0" />
            <TextBlock FontWeight="Bold">Date of birth:</TextBlock>
            <TextBlock Text="{Binding DateOfBirth}" Margin="2,0,10,0" />
        </StackPanel>
    </DataTemplate>
</Window.Resources>

Anything in the window that uses a Person object as its content will pick up this data template by default. If you were to set the contents of a listbox to an array of Person objects, each line of the list would use this template.

Figure 17 Richer Data Template
Figure 17 Richer Data Template

Data templates act as a bridge between objects and the UI. They allow designers great flexibility to manage the way data is presented. Achieving the same effects in Win32-based technologies requires a great deal of developer effort.

3. Consistent Styles

If you've ever used a word processor, you're probably familiar with the concept of styles. A style is a set of properties that can be applied to some part of your document in order to get a particular appearance. Styles in WPF are very similar, but can be applied to any element in a user interface, not just to parts of a document. Moreover, a WPF style can be used to set any property. For example, you can create a style that sets the Maximum property of a Slider control to 200.

Although styles can be used to set any property, the usual reason for using styles is to ensure visual consistency across your application. If you want your application to convey an old-world look, you can create a style to consistently apply the appropriate properties. This code, for instance, defines a style called oldeWorlde that sets the font to the Old English typeface:

<Window.Resources>
    <Style x:Key="oldeWorlde">
        <Setter Property="Control.FontFamily"
            Value="Old English Text MT" />
        <Setter Property="Control.FontSize" Value="18" />
    </Style>
</Window.Resources>
This style can then be applied to any control you like:
<Button Style="{StaticResource oldeWorlde}">Click me!</Button>

The oldeWorlde style is a named style and will only be applied to controls that explicitly request it in their Style property. It is also possible to define a style that will be applied by default to a designated type of control. The following style will automatically apply to all buttons in a window:

<Window.Resources>
    <!-- all Buttons get this style -->
    <Style TargetType="{x:Type Button}">
    <Setter Property="Control.FontFamily"
        Value="Old English Text MT" />
    <Setter Property="Control.FontSize" Value="18" />
    <Setter Property="Background"
        Value="VerticalGradient Red Black" />
    <Setter Property="Foreground"
        Value="VerticalGradient Yellow Orange" />
    </Style>
</Window.Resources>
...
<!-- no need to set the style manually -->
<Button>Click me!</Button>

Notice that styles are really just lists of properties and values. This means that styles can be used to apply templates—after all, Template is just another property. This provides an easy way to change how a particular control looks throughout the entire window or application. For example, earlier we defined a custom template for a button. In practice, we would probably want to apply that template to more than just one button—our application would look pretty odd with one customized button set among numerous standard buttons. With a style, we can apply the template to all of our buttons, as shown in Figure 18.

Figure 18 Universal Button Template (Close Figure 18)
<Window.Resources>
    <Style TargetType="{x:Type Button}">
    <Setter Property="Button.Template">
        <Setter.Value>
        <ControlTemplate>
            <Grid>
            <Rectangle Fill="VerticalGradient White Red"
                Stroke="DarkRed"
                RadiusX="10" RadiusY="10" />
            <ContentPresenter
                Content="{TemplateBinding Button.Content}"
                Margin="{TemplateBinding Padding}"
                HorizontalAlignment="{TemplateBinding
                    HorizontalContentAlignment}"
                VerticalAlignment="{TemplateBinding
                    VerticalContentAlignment}" />
            </Grid>
        </ControlTemplate>
        </Setter.Value>
    </Setter>
    </Style>
</Window.Resources>

This will cause all of the buttons in the window to use this custom template. To change all buttons in all windows, just put the style inside the application-level resources. When you create a WPF project in Visual Studio® 2005, the project template includes a MyApp.xaml file, which contains an Application.Resources section. Any styles you define in this section will be applied to all the windows in your application.

Styles provide a useful mechanism for managing the look of an application independently of its behavior and structure. This separation allows developers to get on with the business of defining the application's behavior, while designers get on with the business of making it look good.

2. Triggers

A trigger is a declarative mechanism that specifies how a control should respond to certain stimuli. Triggers can be declared inside either styles or templates. For example, you could use a trigger to indicate that buttons should always turn green when the mouse is over them:

<Style TargetType="{x:Type Button}">
    <Style.Triggers>
    <Trigger Property="IsMouseOver" Value="True">
        <Setter Property="Background" Value="Green" />
    </Trigger>
    </Style.Triggers>
</Style>

Triggers can also be used to fire animations. For example, you might want a button to fade to a different color while the mouse hovers over it. Animations are different in nature from properties in that they are temporal. While we can set a property to a value, we cannot set an animation to a value—instead, animations are set to start and stop. So animation triggers are event-driven, rather than property-driven.

Figure 19 illustrates how you can use EventTrigger elements to launch animations. This example indicates that when the Mouse.MouseEnter event occurs, the Opacity property must be animated from its current value to the value 1. Similarly, when the Mouse.MouseLeave event occurs, the property is animated from its current value back down to 0.5.

Figure 19 Using an Animation Trigger (Close Figure 19)
<Rectangle Fill="Blue" Height="100" Width="200" Opacity="0.5">
    <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Mouse.MouseEnter">
        <BeginStoryboard>
            <Storyboard TargetProperty="(Rectangle.Opacity)">
                <DoubleAnimation To="1" Duration="0:0:1" />
            </Storyboard>
        </BeginStoryboard>
    </EventTrigger>
    <EventTrigger RoutedEvent="Mouse.MouseLeave">
        <BeginStoryboard>
            <Storyboard TargetProperty="(Rectangle.Opacity)">
                <DoubleAnimation To="0.5" Duration="0:0:1" />
            </Storyboard>
        </BeginStoryboard>
    </EventTrigger>
    </Rectangle.Triggers>
</Rectangle>

Triggers are an important part of the declarative approach of WPF. This is yet another feature that puts more power in the hands of designers, allowing them to design a UI that responds to actions without first needing code from developers.

1. Declarative Programming

Perhaps the most striking architectural theme in WPF is the use of declarative programming. In declarative programming, emphasis is placed on "what" rather than "how". You declare what you want the system to do, rather than listing the set of operations it must execute in order to achieve this goal. SQL is the canonical example—you construct SQL queries that declare what information you want to retrieve; it's up to the database to figure out how to execute that query.

We've already shown you a few examples that rely on the declarative approach. To provide custom graphics, you can create objects that represent our drawings and WPF then handles the process of rendering these drawings to the screen. These include updating the screen when properties are changed. Likewise, we explained how you can create objects that declare how to animate parts of the UI, and WPF will then perform the actual animation. You've also seen how the layout elements can be used to declare how the UI should be arranged—as opposed to positioning the elements by hand or writing code to determine how they are placed and sized.

There are numerous benefits to a declarative style. Foremost is that it usually results in simpler programs that are easier to read. Code that simply declares what needs to be done is usually a lot shorter than code that performs the actual work. Consider the Grid example shown earlier, which started with the row and column definitions, as shown here:

<Grid.ColumnDefintions>
    <ColumnDefinition Width="*" />
    <ColumnDefinition Width="5*" />
</Grid.ColumnDefintions>

<Grid.RowDefintions>
    <RowDefinition Height="Auto" />
    <RowDefinition Height="*" />
</Grid.RowDefintions>
This is a compact expression of our layout requirements. It tells the Grid element that we require two rows and two columns. It says that the columns should be sized proportionally, with the second column getting five times as much of the total available width as the first column. As for the rows, it tells the grid to pick the height for the first row automatically based on its contents and to make the second row fill all the remaining available space. Imagine how much code this would require if you were to write this manually.

It's usually easier to work out what declarative code is trying to do because it explicitly states its intent. Thus, there is no need for maintenance developers to expend a lot of effort trying to infer the intent of the code. Declarative code also enables design tools to do a much better job. A design tool knows what effect you are trying to achieve because you explicitly state your intent. In addition, with declarative programming, the framework is allowed latitude in how it performs the work. This can potentially enable significant optimizations.

Since WPF allows many aspects of a UI's behavior to be defined declaratively, it is fairly straightforward to express these behaviors in XAML. Even though layout and animation are aspects of an application's run-time behavior, you can define these behaviors declaratively with XAML, ultimately providing this power to designers, even if they're not programmers.

Where Are We?

WPF is a radical departure from the established way of building Windows-based apps. It offers significant advances in graphical capabilities and has a huge impact on how application UIs are built. The drawing object model and text handling capabilities make it much easier to exploit the visual potential offered by modern computers. Layout services allow designers to build applications that arrange the content in a way that adapts to the user's requirements. The content model provides great freedom of where we can make use of the textual and graphic facilities. The separation of design and code enabled by lookless controls, styles, data templates, and triggers allows designers to remain in control of an application's appearance while developers maintain control of its behavior. And the declarative programming that runs through all of these facilities reduces the amount of work you have to do to build great apps.

Ian Griffiths is an independent consultant and an instructor for Pluralsight. He has written various books, including Programming Windows Presentation Foundation: Beta Edition (O'Reilly Media, 2003), written with Chris Sells. See his blog at www.interact-sw.co.uk/iangblog.

Chris Sells is a Program Manager for the Connected Systems Division at Microsoft where he is working on next-gen application development techniques. More info about Chris and his various projects can be found at www.sellsbrothers.com.


Talking Windows: Exploring New Speech Recognition And Synthesis APIs In Windows Vista

Talking Windows

Exploring New Speech Recognition And Synthesis APIs In Windows Vista

Robert Brown
This article is based on a prerelease version of WinFX. All information contained herein is subject to change.
This article discusses:
  • Introduction to speech recognition and synthesis
  • The Windows Vista speech recognition experience
  • Windows Vista Speech APIs
  • Controlling pronunciation and creating grammars
This article uses the following technologies:
Windows Vista, WinFX
Contents
Elements of Speech
Talking to Windows Vista
Windows Vista Speech APIs
System.Speech.Synthesis
System.Speech.Recognition
Telephony Applications
Conclusion

M
icrosoft has been researching and developing speech technologies for over a decade. In 1993, the company hired Xuedong (XD) Huang, Fil Alleva, and Mei-Yuh Hwang—three of the four people responsible for the Carnegie Mellon University Sphinx-II speech recognition system, which achieved fame in the speech world in 1992 due to its unprecedented accuracy. Right from the start, with the formation of the Speech API (SAPI) 1.0 team in 1994, Microsoft was driven to create a speech technology that was both accurate and accessible to developers through a powerful API. The team has continued to grow and over the years has released a series of increasingly powerful speech platforms.

In recent years, Microsoft has placed an increasing emphasis on bringing speech technologies into mainstream usage. This focus has led to products such as Speech Server, which is used to implement speech-enabled telephony systems, and Voice Command, which allows users to control Windows Mobile® devices using speech commands. So it should come as no surprise that the speech team at Microsoft has been far from idle in the development of Windows Vista™. The strategy of coupling powerful speech technology with a powerful API has continued right through to Windows Vista.

Windows Vista includes a built-in speech recognition user interface designed specifically for users who need to control Windows® and enter text without using a keyboard or mouse. There is also a state-of-the-art general purpose speech recognition engine. Not only is this an extremely accurate engine, but it's also available in a variety of languages. Windows Vista also includes the first of the new generation of speech synthesizers to come out of Microsoft, completely rewritten to take advantage of the latest techniques.

On the developer front, Windows Vista includes a new WinFX® namespace, System.Speech. This allows developers to easily speech-enable Windows Forms applications and apps based on the Windows Presentation Framework. In addition, there's an updated COM Speech API (SAPI 5.3) to give native code access to the enhanced speech capabilities of the platform. For more information on this, see the "New to SAPI 5.3" sidebar.

Elements of Speech

The concept of speech technology really encompasses two technologies: synthesizers and recognizers (see Figure 1). A speech synthesizer takes text as input and produces an audio stream as output. Speech synthesis is also referred to as text-to-speech (TTS). A speech recognizer, on the other hand, does the opposite. It takes an audio stream as input, and turns it into a text transcription.

Figure 1 Speech Recognition and Synthesis (Close Figure 1)

A lot has to happen for a synthesizer to accurately convert a string of characters into an audio stream that sounds just as the words would be spoken. The easiest way to imagine how this works is to picture the front end and back end of a two-part system.

The front end specializes in the analysis of text using natural language rules. It analyzes a string of characters to figure out where the words are (which is easy to do in English, but not as easy in languages such as Chinese and Japanese). This front end also figures out details like functions and parts of speech—for instance, which words are proper nouns, numbers, and so forth; where sentences begin and end; whether a phrase is a question or a statement; and whether a statement is past, present, or future tense.

All of these elements are critical to the selection of appropriate pronunciations and intonations for words, phrases, and sentences. Consider that in English, a question usually ends with a rising pitch, or that the word "read" is pronounced very differently depending on its tense. Clearly, understanding how a word or phrase is being used is a critical aspect of interpreting text into sound. To further complicate matters, the rules are slightly different for each language. So, as you can imagine, the front end must do some very sophisticated analysis.

The back end has quite a different task. It takes the analysis done by the front end and, through some non-trivial analysis of its own, generates the appropriate sounds for the input text. Older synthesizers (and today's synthesizers with the smallest footprints) generate the individual sounds algorithmically, resulting in a very robotic sound. Modern synthesizers, such as the one in Windows Vista, utilize a database of sound segments built from hours and hours of recorded speech. The effectiveness of the back end depends on how good it is at selecting the appropriate sound segments for any given input and smoothly splicing them together.

If this all sounds vastly complicated, well, it is. Having these-text- to speech capabilities built into the operating system is very advantageous, as it allows applications to just use this technology. There's no need to go create your own speech engines. As you'll see later in the article, you can invoke all of this processing with a single function call. Lucky you!

Speech recognition is even more complicated than speech synthesis. However, it too can be thought of as having a front end and a back end. The front end processes the audio stream, isolating segments of sound that are probably speech and converting them into a series of numeric values that characterize the vocal sounds in the signal. The back end is a specialized search engine that takes the output produced by the front end and searches across three databases: an acoustic model, a lexicon, and a language model. The acoustic model represents the acoustic sounds of a language, and can be trained to recognize the characteristics of a particular user's speech patterns and acoustic environments. The lexicon lists a large number of the words in the language, along with information on how to pronounce each word. The language model represents the ways in which the words of a language are combined.

Neither of these models is trivial. It's impossible to specify exactly what speech sounds like. And human speech rarely follows strict and formal grammar rules that can be easily defined. An indispensable factor in producing good models is the acquisition of very large volumes of representative data. An equally important factor is the sophistication of the techniques used to analyze that data to produce the actual models.

Of course, no word has ever been said exactly the same way twice, so the recognizer is never going to find an exact match. And for any given segment of sound, there are very many things the speaker could potentially be saying. The quality of a recognizer is determined by how good it is at refining its search, eliminating the poor matches, and selecting the more likely matches. A recognizer's accuracy relies on it having good language and acoustic models, and good algorithms both for processing sound and for searching across the models. The better the models and algorithms, the fewer the errors that are made, and the quicker the results are found. Needless to say, this is a difficult technology to get right.

While the built-in language model of a recognizer is intended to represent a comprehensive language domain (such as everyday spoken English), any given application will often have very specific language model requirements. A particular application will generally only require certain utterances that have particular semantic meaning to that application. Hence, rather than using the general purpose language model, an application should use a grammar that constrains the recognizer to listen only for speech that the application cares about. This has a number of benefits: it increases the accuracy of recognition, it guarantees that all recognition results are meaningful to the application, and it enables the recognition engine to specify the semantic values inherent in the recognized text. Figure 2 shows one example of how these benefits can be put to use in a real-world scenario.

Figure 2 Using Speech Recognition for Application Input
Figure 2 Using Speech Recognition for Application Input

Talking to Windows Vista

Accuracy is only part of the equation. With the Windows Vista speech recognition technology, Microsoft has a goal of providing an end-to-end speech experience that addresses key features that users need in a built-in desktop speech recognition experience. This includes an interactive tutorial that explains how to use speech recognition technology and helps the user train the system to understand the user's speech.

The system includes built-in commands for controlling Windows—allowing you to start, switch between, and close applications using commands such as "Start Notepad" and "Switch to Calculator." Users can control on-screen interface elements like menus and buttons by speaking commands like "File" and "Open." There's also support for emulating the mouse and keyboard by giving commands such as "Press shift control left arrow 3 times."

Windows Vista speech technology includes built-in dictation capabilities (for converting the user's voice into text) and edit controls (for inserting, correcting, and manipulating text in documents). You can correct misrecognized words by redictating, choosing alternatives, or spelling. For example, "Correct Robot, Robert." Or "Spell it R, O, B, E, R as in rabbit, T as in telephone." You can also speak commands to select text, navigate inside a document, and make edits—for instance, "Select 'My name is,'" "Go after Robert," or "Capitalize Brown."

The user interface is designed to be unobtrusive, yet to keep the user in control of the speech system at all times (see Figure 3). You have easy access to the microphone state, which includes a sleeping mode. Text feedback tells the user what the system is doing, and provides instructions to the user. There's also a user interface used for clarifying what the user has said–when the user utters a command that can be interpreted in multiple ways, the system uses this interface to clarify what was intended. Meanwhile, ongoing use allows the underlying models to adapt continually improve accuracy over time.

Figure 3 Speech UI in Windows Vista
Figure 3 Speech UI in Windows Vista

To enable built-in speech functionality, from the Start Menu choose All Programs | Accessories | Accessibility and click Speech Recognition. The first time you do this, the system will step you through the tutorial, where you'll be introduced to some basic commands. You also get the option of enabling background language model adaptation, by which the system will read through your documents and e-mail in the background to adapt the language model to better match the way you express yourself. There are a variety of things the default settings enable. I recommend that you ask the system "what can I say" and then browse the topics.

But you're a developer, so why do you care about all this user experience stuff? The reason this is relevant to developers is that this is default functionality provided by the operating system. This is functionality that your applications will automatically get. The speech technology uses the Windows accessibility interfaces to discover the capabilities of each application; it then provides a spoken UI for each. If a user says the name of an accessible element, then the system will invoke the default function of that element. Hence, if you have built an accessible application, you have by default built a speech-enabled application.

Windows Vista Speech APIs

Windows Vista can automatically speech-enable any accessible application. This is fantastic news if you want to let users control your application with simple voice commands. But you may want to provide a speech-enabled user interface that is more sophisticated or tailored than the generic speech-enabled UI that Windows Vista will automatically supply.

There are numerous examples of why you might need to do this. If, for example, your user has a job that requires her to use her hands at all times. Any time she needs to hold a mouse or tap a key on the keyboard is time that her hands are removed from the job—this may compromise safety or reduce productivity. The same could be true for users who need their eyes to be looking at something other than a computer screen. Or, say your application has a very large number of functions that get lost in toolbar menus. Speech commands can flatten out deep menu structures, offering fast access to hundreds of commands. If your users ever say "that's easier said than done," they may be right.

In Windows Vista, there are two speech APIs:

  • SAPI 5.3 for native applications
  • The System.Speech.Recognition and System.Speech.Synthesis namespaces in WinFX
Figure 4 illustrates how each of these APIs relates to applications and the underlying recognition and synthesis engines.

Figure 4 Speech APIs in Windows Vista
Figure 4 Speech APIs in Windows Vista

The speech recognition engine is accessed via SAPI. Even the classes in the System.Speech.Recognition namespaces wrap the functionality exposed by SAPI. (This is an implementation detail of Windows Vista that may change in future releases, but it's worth bearing in mind.) The speech synthesis engine, on the other hand, is accessed directly by the classes in System.Speech.Synthesis or, alternatively, by SAPI when used in an unmanaged application.

Both types implement the SAPI device driver interface (DDI), which is an API that makes engines interchangeable to the layers above them, much like the way device driver APIs make hardware devices interchangeable to the software that uses them. This means that developers who use SAPI or System.Speech are still free to use other engines that implement the SAPI DDI (and many do).

Notice in Figure 4 that the synthesis engine is always instantiated in the same process as the application, but the recognition engine can be instantiated in another process called SAPISVR.EXE. This provides a shared recognition engine that can be used simultaneously by multiple applications. This design has a number of benefits. First, recognizers generally require considerably more run-time resources than synthesizers, and sharing a recognizer is an effective way to reduce the overhead. Second, the shared recognizer is also used by the built-in speech functionality of Windows Vista. Therefore, apps that use the shared recognizer can benefit from the system's microphone and feedback UI. There's no additional code to write, and no new UI for the user to learn.New to SAPI 5.3

SAPI 5.3 is an incremental update to SAPI 5.1. The core mission and architecture for SAPI are unchanged. SAPI 5.3 adds performance improvements, overall enhancements to security and stability, and a variety of new functionality, including:

W3C Speech Synthesis Markup Language SAPI 5.3 supports the W3C Speech Synthesis Markup Language (SSML) version 1.0. SSML provides the ability to mark up voice characteristics, speed, volume, pitch, emphasis, and pronunciation so that a developer can make TTS sound more natural in their application.

W3C Speech Recognition Grammar Specification SAPI 5.3 adds support for the definition of context-free grammars using the W3C Speech Recognition Grammar Specification (SRGS), with these two important constraints: it does not support the use of SRGS to specify dual-tone modulated-frequency (touch-tone) grammars, and it only supports the expression of SRGS as XML—not as Augmented Backus-Naur Form (ABNF).

Semantic Interpretation SAPI 5.3 enables an SRGS grammar to be annotated with JScript® for semantic interpretation, so that a recognition result may contain not only the recognized text, but the semantic interpretation of that text. This makes it easier for apps to consume recognition results, and empower grammar authors to provide a full spectrum of semantic processing beyond what could be achieved with name-value pairs.

User-Specified "Shortcuts" in Lexicons This is the ability to add a string to the lexicon and associate it with a shortcut word. When dictating, the user can say the shortcut word and the recognizer will return the expanded string.

As an example, a developer could create a shortcut for a location so that a user could say "my address" and the actual data would be passed to the application as "123 Smith Street, Apt. 7C, Bloggsville 98765, USA". The following code sets up the lexicon shortcut:

CComPtr<ISpShortcut> cpShortcut;
HRESULT hr = cpShortcut.CoCreateInstance(CLSID_SpShortcut);
if (SUCCEEDED(hr))
{
    hr = cpShortcut.AddShortcut(L"my address", 1033,
        L"123 Smith Street, Apt. 7C, Bloggsville 98765, USA",
        SPSHT_OTHER);
}

When this code is used, the shortcut is added to the speech lexicon. Every time a user says "my address," the actual address is returned as the transcribed text.

Discovery of Engine Pronunciations SAPI 5.3 enables applications to query the Windows Vista recognition and synthesis engines for the pronunciations they use for particular words. This API will tell the application not only the pronunciation, but how that pronunciation was derived.

System.Speech.Synthesis

Let's take a look at some examples of how to use speech synthesis from a managed application. In the grand tradition of UI output examples, I'll start with an application that simply says "Hello, world," shown in Figure 5. This example is a bare-bones console application as freshly created in Visual C#®, with three lines added. The first added line simply introduces the System.Speech.Synthesis namespace. The second declares and instantiates an instance of SpeechSynthesizer, which represents exactly what its name suggests: a speech synthesizer. The third added line is a call to SpeakText. This is all that's needed to invoke the synthesizer!

Figure 5 Saying Hello (Close Figure 5)
using System;
using System.Speech.Synthesis;

namespace TTS_Console_Sample_1
{
    class Program
    {
        static void Main(string[] args)
        {
            SpeechSynthesizer synth = new SpeechSynthesizer();
            synth.SpeakText("Hello, world!");
        }
    }
}

By default, the SpeechSynthesizer class uses the synthesizer that is nominated as default in the Speech control panel. But it can use any SAPI DDI-compliant synthesizer.

The next example (see Figure 6) shows how this can be done, using the old Sam voice from Windows 2000 and Windows XP, and the new Anna and Microsoft® Lili voices from Windows Vista. (Note that this and all remaining System.Speech.Synthesis examples use the same code framework as the first example, and just replaces the body of Main.) This example shows three instances of the SelectVoice method using the name of the desired synthesizer. It also demonstrates the use of the Windows Vista Chinese synthesizer, Lili. Incidentally, Lili also speaks English very nicely.

Figure 6 Hearing Voices (Close Figure 6)
SpeechSynthesizer synth = new SpeechSynthesizer();

synth.SelectVoice("Microsoft Sam");
synth.SpeakText("I'm Sam.");
synth.SpeakText("You may have heard me speaking to you in Windows XP.");
synth.SpeakText("Anna will make me redundant.");

synth.SelectVoice("Microsoft Anna");
synth.SpeakText("I am the new voice in Windows.");
synth.SpeakText("Sam belongs to a previous generation.");
synth.SpeakText("I sound great.");

synth.SelectVoice("Microsoft Lili");
synth.SpeakText("我是在北京被研究开发的 我使用了专业播音员的声音。每
个听到过我说话的人都说我是中文语音合成中最棒的！"); // Requires MS Mincho and SimSun fonts to view
/* "I was developed in Beijing, using recordings of a professional news reader.
Everybody who hears me talk says that I am the best synthesized Chinese
voice they have ever heard!" */

In both of these examples, I use the synthesis API much as I would a console API: an application simply sends characters, which are rendered immediately in series. But for more sophisticated output, it's easier to think of synthesis as the equivalent of document rendering, where the input to the synthesizer is a document that contains not only the content to be rendered, but also the various effects and settings that are to be applied at specific points in the content.

Much like an XHTML document can describe the rendering style and structure to be applied to specific pieces of content on a Web page, the SpeechSynthesizer class can consume an XML document format called Speech Synthesis Markup Language (SSML). The W3C SSML recommendation (www.w3.org/TR/speech-synthesis) is very readable, so I'm not going to dive into describing SSML in this article. Suffice it to say, an application can simply load an SSML document directly into the synthesizer and have it rendered. Here's an example that loads and renders an SSML file:

SpeechSynthesizer synth = new SpeechSynthesizer();
PromptBuilder savedPrompt = new PromptBuilder();
savedPrompt.AppendSsml("c:\\prompt.ssml");
synth.Speak(SavedPrompt);

A convenient alternative to authoring an SSML file is to use the PromptBuilder class in System.Speech.Synthesis. PromptBuilder can express almost everything an SSML document can express, and is much easier to use (see Figure 7). The general model for creating sophisticated synthesis is to first use a PromptBuilder to build the prompt exactly the way you want it, and then use the Synthesizer's Speak or SpeakAsync method to render it.

Figure 7 Using PromptBuilder (Close Figure 7)
//This prompt is quite complicated
//So I'm going to build it first, and then render it.
PromptBuilder myPrompt = new PromptBuilder();

//Start the main speaking style
PromptStyle mainStyle = new PromptStyle();
mainStyle.Rate = PromptRate.Medium;
mainStyle.Volume = PromptVolume.Loud;
myPrompt.StartStyle(MainStyle);

//Alert the listener
myPrompt.AppendAudio(new Uri(
    "file://c:\\windows\\media\\notify.wav"), "Attention!");
myPrompt.AppendText("Here are some important messages.");

//Here's the first important message
myPrompt.AppendTextWithPronunciation("WinFX", "wɪnɛfɛks");
myPrompt.AppendText("is a great platform.");

//And the second one
myPrompt.AppendTextWithHint("ASP", SayAs.Acronym);
myPrompt.AppendText(
    "is an acronym for Active Server Pages. Whereas an ASP is a snake.");

myPrompt.AppendBreak();

//Let's emphasise how important these messages are
PromptStyle interimStyle = new PromptStyle();
interimStyle.Emphasis = PromptEmphasis.Strong;
myPrompt.StartStyle(interimStyle);
myPrompt.AppendText("Please remember these two things.");
myPrompt.EndStyle();

//Then we can revert to the main speaking style
myPrompt.AppendBreak();
myPrompt.AppendText("Thank you");

myPrompt.EndStyle();
//Now let's get the synthesizer to render this message
SpeechSynthesizer synth = new SpeechSynthesizer();
synth.Speak(myPrompt);

Figure 7 illustrates a number of powerful capabilities of the PromptBuilder. The first thing to point out is that it generates a document with a hierarchical structure. The example uses one speaking style nested within another. At the beginning of the document, I start the speaking style I want used for the entire document. Then about halfway through, I switch to a different style to provide emphasis. When I end this style, the document automatically reverts to the previous style.

The example also shows a number of other handy capabilities. The AppendAudio function causes a WAV file to be spliced into the output, with a textual equivalent to be used if the WAV file can't be found. The AppendTextWithPronunciation function allows you to specify the precise pronunciation of a word. A speech synthesis engine already knows how to pronounce most of the words in general use in a language, through a combination of an extensive lexicon and algorithms for deriving the pronunciation of unknown words. But this won't work for all words, such as some specialized terms or brand names. For example, "WinFX" would probably be pronounced as "winfeks". Instead, I use the International Phonetic Alphabet to describe "WinFX" as "wɪnɛfɛks", where the letter "ɪ" is Unicode character 0x026A (the "i" sound in the word "fish", as opposed to the "i" sound in the word "five") and the letter "ɛ" is Unicode character 0x025B (the General American "e" sound in the word "bed").

In general, a synthesis engine can distinguish between acronyms and capitalized words. But occasionally you'll find an acronym that the engine's heuristics incorrectly deduce to be a word. So you can use the AppendTextWithHint function to identify a token as an acronym. There are a variety of nuances you can introduce with the PromptBuilder. My example is more illustrative than exhaustive.

Another benefit of separating content specification from run-time rendering is that you are then free to decouple the application from the specific content it renders. You can use PromptBuilder to persist its prompt as SSML to be loaded by another part of the application, or a different application entirely. The following code writes to an SSML file with PromptBuilder:

using(StreamWriter promptWriter = new StreamWriter("c:\\prompt.ssml"))
{
    promptWriter.Write(myPrompt.ToXml());
}
Another way to decouple content production is to render the entire prompt to an audio file for later playback:
SpeechSynthesizer synth = new SpeechSynthesizer();
synth.SetOutputToWaveFile("c:\\message.wav");
synth.Speak(myPrompt);
synth.SetOutputToNull();
Whether to use SSML markup or the PromptBuilder class is probably a matter of stylistic preference. You should use whichever you feel more comfortable with.

One final note about SSML and PromptBuilder is that the capabilities of every synthesizer will be slightly different. Therefore, the specific behaviors you request with either of these mechanisms should be thought of as advisory requests that the engine will apply if it is capable of doing so.

System.Speech.Recognition

While you could use the general dictation language model in an application, you would very rapidly encounter a number of application development hurdles regarding what to do with the recognition results. For example, imagine a pizza ordering system. A user could say "I'd like a pepperoni pizza" and the result would contain this string. But it could also contain "I'd like pepper on a plaza" or a variety of similar sounding statements, depending on the nuances of the user's pronunciation or the background noise conditions. Similarly, the user could say "Mary had a little lamb" and the result would contain this, even though it's meaningless to a pizza ordering system. All of these erroneous results are useless to the application. Hence an application should always provide a grammar that describes specifically what the application is listening for.

In Figure 8, I've started with a bare-bones Windows Forms application and added a handful of lines to achieve basic speech recognition. First, I introduce the System.Speech.Recognition namespace, and then instantiate a SpeechRecognizer object. Then I do three things in Form1_Load: build a grammar, attach an event handler to that grammar so that I can receive the SpeechRecognized events for that grammar, and then load the grammar into the recognizer. At this point, the recognizer will start listening for speech that fits the patterns defined by the grammar. When it recognizes something that fits the grammar, the grammar's SpeechRecognized event handler is invoked. The event handler itself accesses the Result object and works with the recognized text.

Figure 8 Ordering a Pizza (Close Figure 8)
using System;
using System.Windows.Forms;
using System.ComponentModel;
using System.Collections.Generic;
using System.Speech.Recognition;

namespace Reco_Sample_1
{
    public partial class Form1 : Form
    {
        //create a recognizer
        SpeechRecognizer _recognizer = new SpeechRecognizer();

        public Form1() { InitializeComponent(); }

        private void Form1_Load(object sender, EventArgs e)
        {
           //Create a pizza grammar
           Choices pizzaChoices = new Choices();
           pizzaChoices.AddPhrase("I'd like a cheese pizza");
           pizzaChoices.AddPhrase("I'd like a pepperoni pizza");
           pizzaChoices.AddPhrase("I'd like a large pepperoni pizza");
           pizzaChoices.AddPhrase(
               "I'd like a small thin crust vegetarian pizza");
           Grammar pizzaGrammar =
               new Grammar(new GrammarBuilder(pizzaChoices));

           //Attach an event handler
           pizzaGrammar.SpeechRecognized +=
               new EventHandler<RecognitionEventArgs>(
                   PizzaGrammar_SpeechRecognized);

           _recognizer.LoadGrammar(pizzaGrammar);
        }

        void PizzaGrammar_SpeechRecognized(
            object sender, RecognitionEventArgs e)
        {
            MessageBox.Show(e.Result.Text);
        }
    }
}

The System.Speech.Recognition API supports the W3C Speech Recognition Grammar Specification (SRGS), documented at www.w3.org/TR/speech-grammar. The API even provides a set of classes for creating and working with SRGS XML documents. But for most cases, SRGS is overkill, so the API also provides the GrammarBuilder class that suffices nicely for our pizza ordering system.

The GrammarBuilder lets you assemble a grammar from a set of phrases and choices. In Figure 8 I've eliminated the problem of listening for utterances I don't care about ("Mary had a little lamb"), and constrained the engine so that it can make much better choices between ambiguous sounds. It won't even consider the word "plaza" when the user mispronounces "pizza". So in a handful of lines, I've vastly increased the accuracy of the system. But there are still a couple of problems with the grammar.

The approach of exhaustively listing every possible thing a user can say is tedious, error prone, difficult to maintain, and only practically achievable for very small grammars. It is preferable to define a grammar that defines the ways in which words can be combined. Also, if the application cares about the size, toppings, and type of crust, then the developer has quite a task to parse these values out of the result string. It's much more convenient if the recognition system can identify these semantic properties in the results. This is very easy to do with System.Speech.Recognition and the Windows Vista recognition engine.

Figure 9 shows how to use the Choices class to assemble grammars where the user says something from a list of alternatives. In this code, the contents of each Choices instance are specified in the constructor as a sequence of string parameters. But you have a lot of other options for populating Choices: you can iteratively add new phrases, construct Choices from an array, add Choices to Choices to build the complex combinatorial rules that humans understand, or add GrammarBuilder instances to Choices to build increasingly flexible grammars (as demonstrated by the Permutations part of the example).

Figure 9 Using Choices to Assemble Grammars (Close Figure 9)
private void Form1_Load(object sender, EventArgs e)
{
    //[I'd like] a [<size>] [<crust>] [<topping>] pizza [please]

    //build the core set of choices
    Choices sizes = new Choices("small", "regular", "large");
    Choices crusts = new Choices("thin crust", "thick crust");
    Choices toppings = new Choices("vegetarian", "pepperoni", "cheese");

    //build the permutations of choices...

    //choose all three
    GrammarBuilder sizeCrustTopping = new GrammarBuilder();
    sizeCrustTopping.AppendChoices(sizes, "size");
    sizeCrustTopping.AppendChoices(crusts, "crust");
    sizeCrustTopping.AppendChoices(toppings, "topping");

    //choose size and topping, and assume thick crust
    GrammarBuilder sizeAndTopping = new GrammarBuilder();
    sizeAndTopping.AppendChoices(sizes, "size");
    sizeAndTopping.AppendChoices(toppings, "topping");
    sizeAndTopping.AppendResultKeyValue("crust", "thick crust");

    //choose topping only, and assume the rest
    GrammarBuilder toppingOnly = new GrammarBuilder();
    toppingOnly.AppendChoices(toppings, "topping");
    toppingOnly.AppendResultKeyValue("size", "regular");
    toppingOnly.AppendResultKeyValue("crust", "thick crust");

    //assemble the permutations
    Choices permutations = new Choices();
    permutations.AddGrammarBuilders(sizeCrustTopping);
    permutations.AddGrammarBuilders(sizeAndTopping);
    permutations.AddGrammarBuilders(toppingOnly);

    //now build the complete pattern...
    GrammarBuilder pizzaRequest = new GrammarBuilder();
    //pre-amble "[I'd like] a"
    pizzaRequest.AppendChoices(new Choices("I'd like a", "a"));
    //permutations "[<size>] [<crust>] [<topping>]"
    pizzaRequest.AppendChoices(permutations);
    //post-amble "pizza [please]"
    pizzaRequest.AppendChoices(new Choices("pizza", "pizza please"));

    //create the pizza grammar
    Grammar pizzaGrammar = new Grammar(pizzaRequest);

    //attach the event handler
    pizzaGrammar.SpeechRecognized +=
        new EventHandler<RecognitionEventArgs>(
            PizzaGrammar_SpeechRecognized);

    //load the grammar into the recognizer
    _recognizer.LoadGrammar(pizzaGrammar);
}

void PizzaGrammar_SpeechRecognized(object sender, RecognitionEventArgs e)
{
    StringBuilder resultString = new StringBuilder();
    resultString.Append("Raw text result: ");
    resultString.AppendLine(e.Result.Text);
    resultString.Append("Size: ");
    resultString.AppendLine(e.Result.Semantics["size"].Value.ToString());
    resultString.Append("Crust: ");
    resultString.AppendLine(e.Result.Semantics["crust"].Value.ToString());
    resultString.Append("Topping: ");
    resultString.AppendLine(
        e.Result.Semantics["topping"].Value.ToString());
    MessageBox.Show(resultString.ToString());
}

Figure 9 also shows how to tag results with semantic values. When using GrammarBuilder, you can append Choices to the grammar, and attach a semantic value to that choice, as can be seen in the example with statements like this:

AppendChoices(toppings, "topping");

Sometimes a particular utterance will have an implied semantic value that was never uttered. For example, if the user doesn't specify a pizza size, the grammar can specify the size as "regular", as seen with statements like this:

AppendResultKeyValue("size", "regular");

Fetching the semantic values from the results is done by accessing RecognitionEventArgs.Result.Semantics[<name>].

Telephony Applications
Resources

This article only touches on the capabilities of the new speech technology included in Windows Vista. Here are several resources to help you learn more:

  • WinFX SDK
  • Microsoft Speech Server
  • Windows Accessibility
  • The Microsoft speech newsgroups: microsoft.public.speech_tech and microsoft.public.speech_tech.sdk

The following members of the Microsoft speech team have active blogs that you may find useful:

  • Jay Waltmunson
  • Jen Anderson
  • Philipp Schmid
  • Richard Sprague
  • Rob Chambers
  • Robert Brown

One of the biggest growth areas for speech applications is in speech-enabled telephony systems. Many of the principles are the same as with desktop speech: recognition and synthesis are still the keystone technologies, and good grammar and prompt design are critical.

There are a number of other factors necessary for the development of these applications. A telephony application needs a completely different acoustic model. It needs to be able to interface with telephony systems, and because there's no GUI, it needs to manage a spoken dialog with the user. A telephony application also needs to be scalable so that it can service as many simultaneous calls as possible without compromising performance.

Designing, tuning, deploying, and hosting a speech-enabled telephony application is a non-trivial project that the Microsoft Speech Server platform and SDK have been developed to address. For more on the Microsoft Speech Server, take a look at the Speech Server Web site.

Conclusion

Windows Vista contains a new, more powerful desktop speech platform that is built into the OS. The intuitive UI and powerful APIs make it easy for end users and developers to tap into this technology. If you have the latest Beta build of Windows Vista, you can start playing with these new features immediately.

The Windows Vista Speech Recognition Web site, should be live by the time you're reading this. For links to other sources of information about Microsoft speech technologies, see the "Resources" sidebar.

Robert Brown is a Lead Program Manager on the Microsoft Speech & Natural Language team. Robert joined Microsoft in 1995 and has worked on VOIP and messaging technologies, Speech Server, and the speech platform in Windows Vista. Special thanks to Robert Stumberger, Rob Chambers and other members of the Microsoft speech team for their contribution to this article.


Visual Studio 2005: Create Reusable Project And Item Templates For Your Development Team

Visual Studio 2005

Create Reusable Project And Item Templates For Your Development Team

Matt Milner
This article discusses:
  • Consuming existing templates
  • Creating new project and item templates
  • Customizing templates through XML metadata
  • Extending the new project wizard
This article uses the following technologies:
Visual Studio 2005, XML
Code download available at: CodeTemplates.exe (139KB)
Contents
Consuming an Existing Template
Creating Templates
Customizing Templates
Templates for Multiple Projects
Providing Guidance
Parameter Replacement
Extending the Wizard
Conclusion

O
ver my years of working with clients on building Microsoft® .NET Framework-based applications, I have often heard a common feature request for Visual Studio® .NET: "It would be great if we could create our own project types and item types so that all developers in our organization would have access to them." Many orgs want to create shared templates for projects such as Web sites or for project items, such as a default Web page or forms. Visual Studio 2005 introduces a new model for defining templates and starter kits for projects and items that make this not only possible, but relatively simple. I'll take a look at how to consume, create, and customize these templates.

In the past, users have been able to create project or item templates, but it took a lot of inside knowledge of arcane text file formats, JavaScript files, and folder naming conventions, as well as a little bit of magic, to make it work. With the new model, a user can create a simple ZIP file of the items to be included in the template and an XML metadata file that describes the template and its contents. This new model supports creating item templates such as a Web page, a C# or Visual Basic® code file, or a configuration file. It also supports single-project templates like Web sites, class libraries, or smart client applications, as well as multiproject solutions. In fact, most of the built-in project and item templates that you use to create new solutions in Visual Studio 2005 are based on this same template mechanism.

Consuming an Existing Template

The easiest way to understand how these templates work is to first take a look at how to consume a template that has already been created. In Visual Studio 2005, you can configure the locations to search for templates, and these locations can use any Universal Naming Convention (UNC) path. In the Options dialog box, under the Projects and Solutions settings node, you can configure the user-specific locations to look for project or item templates (see Figure 1). The default values for these settings point to template folders in your My Documents folder. The templates included with Visual Studio 2005 are located under the Visual Studio installation directory and are included in the project setup dialog boxes by default.

Figure 1 Template Location Options
Figure 1 Template Location Options

By dropping a ZIP file with the appropriate metadata file in the user-configured template location, the new project or item template will appear in the respective dialog boxes. The sample code included with this article includes sample project item and project templates that can be copied into these directories for testing purposes. After adding the ZIP file, start Visual Studio 2005, choose File | New | Project, and notice that the new project template is included under the My Templates section of the dialog box (see Figure 2). Selecting this project template will create a new project with the items included in the ZIP file.

Figure 2 Your Templates Appear in the New Project Dialog Box
Figure 2 Your Templates Appear in the New Project Dialog Box

There are several benefits gained by being able to consume templates in this way. First, because the user locations are configurable, development groups can define a file share on a common server where templates are deployed for the group. All developers in the group can configure their environment to look to that file share for templates. When a new template is deployed to the server by copying the ZIP file to the share, all developers immediately have access to it from within Visual Studio. After the initial deployment, changes to the template can be made in one place instead of having to be deployed to each developer's workstation.

In addition, because the template is used as a true template, it remains unchanged and can be used repeatedly to create new projects. This is a vast improvement over the old "copy and paste" approach where one developer would create a file or project and hand it off to another. Without careful management, the original was quickly lost and modified with no chance to get back to a known good configuration. Because the project and item template locations are configured independently, an organization can choose to have only project templates centralized and allow developers to create their own item templates.

In addition to templates, Visual Studio 2005 includes a starter kit concept. Starter kits enable broad consumption of samples through community sites from which templates can be downloaded. From the New Project dialog box, one of the options is Search Online Templates. Choosing this option allows you to search for starter kits online by keyword. When a starter kit is found, you can choose to download and install it into the appropriate directory so that you can then use it as a template for adding items or projects. As the name implies, starter kits are generally used as learning tools or samples to help convey knowledge or provide a starting point for larger projects.

Creating Templates

Consuming templates is a great start, but most developers will want to know how to create their own templates. Visual Studio 2005 makes it extremely easy to create your own templates right in the IDE. Once they are created, you have the ability to extend and change them.

The simplest type of template to create is an item template. To do so, simply open a project that includes the file you want to use as a template and then chooses File | Export Template to run the Export Template Wizard shown in Figure 3.

Figure 3 Exporting an Item Template
Figure 3 Exporting an Item Template

The wizard will prompt you to indicate the item to export and any assembly references to include. Adding references allows you to ensure that assemblies your template depends on are included in the project when the item is added. For example, if an item template uses the System.Data.SqlXml assembly to access SQL Server™ XML functionality, it is important to make sure this assembly gets referenced by the project when the template is added. The references available to choose from in the wizard include the default assembly references for the current project type as well as any references that have been added to the current project before running the wizard.

In addition to the assembly references, the export wizard is intelligent enough to export resource files required by the item be exported. For example, if an ASP.NET Web page is the item being exported, the C# or Visual Basic codebehind file and resource file for that page will also get exported and included when the template is used to create new items. The wizard also edits these files to add placeholders for class names and namespaces so that when they are added to a new project they can be named differently and show up in the correct code namespace. For example, instead of having a class that's always named BusinessObjectClass when you add your template, the user can specify a name for the new code file, and this will be used as the name for the class in your template.

When the parameters have been selected for the item template—including a name, a description, and the icon to use when displaying the template—you have the option of installing the template. When exporting the template, it gets created in your My Documents\Visual Studio 2005\My Exported Templates folder. Templates in this folder are not included in the New Item or New Project dialog boxes by default. In order to make the template available for use, it must be installed by copying it to the configured directory for templates. Checking the box to install the template simply means that the ZIP file will get copied to the folder you configured for templates in addition to the exported templates folder.

Not installing the template allows you to edit the metadata directly or add other files to the ZIP file before installing the template into the appropriate folder. For example, if you created a Master Page template for use in Web sites, you might want to include a CSS file that includes the styles used on the Master Page to ensure that the styles used are also included when the Master Page template is selected. In this case, you would not want to install the template at this point until you have had a chance to add the stylesheet file to the ZIP file and update the metadata file to include the stylesheet when installing the template. I will look at the steps required to customize a template like this later in the article.

Creating a project template works in much the same way with the exception that assembly references are not needed as they are inferred from the current project configuration. The main difference between the project template and an item template is that a project template provides the means to create a new project, whereas the item template can only be added to an existing project. Both template types, however, allow for multiple items to be added to a project when the wizard is run.

Customizing Templates

As you can see, creating a project or item template has been made extremely simple in Visual Studio 2005. However, like many wizards, the default template created may not be exactly what you want. Fortunately, because the design of the templates involves a ZIP file and a metadata file, it is not difficult to extend a template with new features once it's created.

The template metadata file is an XML file that describes the template and indicates the files and resources to be included in with the project or item. A sample metadata file created by using the wizard to export an item template is shown in Figure 4.

Figure 4 Metadata File for an Item Template (Close Figure 4)
<VSTemplate Version="2.0.0"
  xmlns="http://schemas.microsoft.com/developer/vstemplate/2005"
  Type="Item">
    <TemplateData>
        <Icon>__TemplateIcon.ico</Icon>
        <DefaultName>MasterPage.master</DefaultName>
        <Name>Example Org Master Page</Name>
        <Description>
            The default master page for Example Org web sites
        </Description>
        <ProjectType>Web</ProjectType>
        <ProjectSubType>CSharp</ProjectSubType>
    </TemplateData>
    <TemplateContent>
        <References>
            <Reference>
                <Assembly>System, Version=2.0.0.0,
                    Culture=neutral, PublicKeyToken=B77A5C561934E089
                </Assembly>
            </Reference>
            <Reference>
                <Assembly>mscorlib, Version=2.0.0.0,
                    Culture=neutral, PublicKeyToken=B77A5C561934E089
                </Assembly>
            </Reference>
        </References>
        <ProjectItem TargetFileName="$fileinputname$.master"
            ReplaceParameters="true">master~1.mas</ProjectItem>
        <ProjectItem TargetFileName="$fileinputname$.master.cs"
            ReplaceParameters="true">master~1.cs</ProjectItem>
    </TemplateContent>
</VSTemplate>

The first thing to notice is the general structure of this metadata, which includes a root VSTemplate node and two child nodes: TemplateData and TemplateContent. The VSTemplate node simply acts as a wrapper for the other two items and indicates the type of the template. Because of the XML namespace declaration in the file, any developer editing this file in Visual Studio 2005 will automatically get IntelliSense® support for the items that are allowed, based on the schema for template files.

The TemplateData element contains information that is used to provide details about the template in the New Project or New Item dialog box. The name, description, and icon elements are all displayed directly in the dialog box as you would expect. The DefaultName element is used as the root of the default name provided for the new item or project. The ProjectType element indicates the type of project that the item can be included in and affects the location in the dialog box. The example shown in Figure 4 will cause this item template to be available only for C# Web projects.

Figure 5 shows some other optional elements that can be used in the TemplateData section of the metadata file, as well as options for the items already mentioned.

Figure 5 TemplateData Elements for an Item (Close Figure 5)

Element Name | Description
ProjectType | Indicates the type of project that this item can be added to. Possible values include CSharp, VisualBasic, VisualC, JSharp, and Web.
SupportsMasterPage | For items being added to Web projects only, this Boolean option allows you to indicate that the user should be able to select a master page to go along with your item. This is useful for custom Web page item templates that you create.
SupportsCodeSeparation | For items being added to Web projects only, this Boolean option allows you to indicate that the user should be able to choose if they want their code separated from their presentation. Again, this is most useful for custom Web page item templates that you might create.
SupportsLanguageDropdown | If a template is identical for multiple languages, providing a value of true in this element will allow the user to choose a language for the template. This is only used in Web projects where the template may come from different languages such as the WebForm template. When the language is changed, the metadata file from the new template location is used and the dialog updated. For example, if you switch the language from C# to Visual J# when adding a Web form, the code separation option is disabled because the Visual J# template does not support code separation.

The TemplateContent element is where you define the files and references for your item template. The TemplateContent element references any assemblies to be included with the item or project and generally will not need to be edited since you can specify these in the export wizard.

Each item in the ZIP file that is to be included in the project is called out with a ProjectItem element. In addition to identifying the source files, the attributes of the ProjectItem element allow you to indicate whether an item has a subtype, whether parameters in the file should be replaced, and whether the target file name differs from the source file name. The subtype attribute can include values such as Form or Component, which provide Visual Studio with hints about how to display the item in the correct designer when it is opened.

Replacing parameters allows the New Item wizard to create namespaces, class names, and similar items dynamically so that the template is not entirely static. Without specifying true for this attribute, the template files will appear in the new project exactly as they are in the ZIP file.

The TargetFileName attribute allows you to specify either a constant or parameterized value for the file name as it should appear in the new project. This allows you to reuse a template file to create multiple instances of the item in the same project. For example, a class file could be added to the same project twice with different TargetFileName attributes. When the item is added, two new classes will be added to the project based on the same template file. Allowing for this type of reuse within the template makes it much easier to manage the files involved in creating a template.

When creating project templates, the process is very similar to that used for item templates, but there are a few slight differences in the metadata file. First, because this is a project, all ProjectItem elements are included under a Project element, which indicates the project file name and whether parameters should be replaced when naming the project file. An abbreviated project template file is shown in Figure 6.

Figure 6 Metadata for a Project Template (Close Figure 6)
<VSTemplate Version="2.0.0"
   xmlns="http://schemas.microsoft.com/developer/vstemplate/2005"
    Type="Project">
    <TemplateData>
        ...
    </TemplateData>
    <TemplateContent>
        <References>
            ...
        </References>
        <Project TargetFileName="WinFormTemplate.csproj"
            File="WinFor~1.csp" ReplaceParameters="true">
            <ProjectItem SubType="Code"
                TargetFileName="$fileinputname$.cs"
                ReplaceParameters="true">Class1.cs</ProjectItem>
            <Folder Name="OtherClasses">
                <ProjectItem SubType="Code">Class2.cs</ProjectItem>
            </Folder>
        </Project>
    </TemplateContent>
</VSTemplate>

Notice that in addition to the Project element that identifies the project information, this example also uses the Folder element to indicate that the ZIP file contains a folder containing the project items indicated. The folder will be included in the project structure when the template is used. The project template also provides different options for the TemplateData section to influence the user interaction with the New Project dialog box. These items are shown in Figure 7.

Figure 7 TemplateData Elements for a Project (Close Figure 7)

Element Name | Description
LocationField | Options include Hidden, Disabled, or Enabled. This allows you to indicate whether the user should be able to specify a location for their new project. Defaults to Enabled.
EnableLocationBrowseButton | Indicates if the user should be able to browse the file system for a location to choose for the new project. Defaults to True.
CreateNewFolder | Indicates that the wizard should create a new folder for the project at the location selected. Defaults to True.
AllowCreationWithoutSave | Indicates if this project template can be an in-memory temporary project. This element provides support for a new feature in Visual Studio 2005 which allows a user to create a new project and work on it, but if the project is not saved, it is discarded. This allows for quick testing or prototyping without having to fill up the My Projects folder with a lot of projects that will never get used.
ProjectSubType | Used by internal templates to further restrict the location within the dialog to certain project subtypes such as Windows, Office, or Database. Used by Web projects to specify the language type.

Web projects are configured in a slightly different manner as they use a different wizard for project creation. Instead of using a language for the project type such as Visual Basic and then a ProjectSubType such as Windows, a Web project uses Web as the project type, and the language as the ProjectSubType.

Templates for Multiple Projects

In addition to being able to configure a template for a single project, it is often useful to create an entire solution as a template or starting point. For example, a development group might not simply want a single Web project as the starting point for a new Web solution as they may want to include business object and data access projects. One resolution would be to create a template for each project and allow the developer to add each project they needed individually. However, a much simpler approach, and one that provides more consistency, is to create a template for the entire solution and give a developer all of the projects they need to get started.

Multi-project templates still involve only one ZIP file, but they include one master metadata file and an individual metadata file for each project in the solution. The project metadata files are no different than what has been discussed so far, but the solution template file now provides the template data to the wizard through the TemplateData element and its children and provides pointers to the other project metadata files through the TemplateContents element and its children. The abbreviated sample in Figure 8 shows a solution metadata file that includes three projects: a Web site, a business logic layer, and a data access layer.

Figure 8 Metadata for a Solution (Close Figure 8)
<VSTemplate Version="2.0.0"
  xmlns="http://schemas.microsoft.com/developer/vstemplate/2005"
  Type="ProjectGroup">
    <TemplateData>
        ...
    </TemplateData>
    <TemplateContent>
        <ProjectCollection>
            <ProjectTemplateLink Projectname="DataAccessLayer">
            .\DAL\DAL.vstemplate
            </ProjectTemplateLink>
            <ProjectTemplateLink ProjectName="BusinessLogic">
            .\BAL\BAL.vstemplate
            </ProjectTemplateLink>
            <ProjectTemplateLink ProjectName="WebSite">
            .\WebSite\WebSite.vstemplate
            </ProjectTemplateLink>
        </ProjectCollection>
    </TemplateContent>
</VSTemplate>
Figure 9 Solution Folder
Figure 9 Solution Folder

Notice that the Type for the template is a ProjectGroup and that the template content is simply a collection of links to the other project templates. A friendly name can be given to the projects, and this is how they will be named in the Solution Explorer. The paths to the other projects are based on the folder structure of the items in the ZIP file. The ZIP file used with the example template metadata in Figure 8 would look something like Figure 9. Each project in the solution has its own folder and its own vstemplate metadata file, and the links in the example provide relative paths to the project template files based on the folder structure in the ZIP file.

Providing Guidance

In addition to providing templates, it is often helpful to provide guidance or extra information about how to use the template. One way to do this is to provide well-commented template code files, but this requires a user to dig through your template code to understand how to use it. Visual Studio 2005 templates provide a better mechanism for not only delivering the documentation your users will need, but for making sure they are presented with the documentation after adding an item or project from your template.

In both project and item templates, the ProjectItem element can have several attributes applied to it that allow for displaying items to the user. The OpenInEditor Boolean attribute allows a project item to be opened in the default editor based on the file type of that item. For example, an XML file included in your project and marked with OpenInEditor set to true will be opened in the XML editor. The OpenInWebBrowser attribute allows a text or HTML file to be opened in a Web browser window. This is an easy way to distribute help or guidance with your templates. Simply create an HTML file providing your guidance and include it in the template. Users will view your help file whenever they use your template so you can be sure they have the information they need to properly use your template files once installed.

In addition to having files open in the browser or editor windows, you can control the order in which the items appear when you have multiple items opening. You should use the OpenOrder attribute on the ProjectItem element in order to control the order of the displayed items in Visual Studio. The lower the number supplied for OpenOrder, the higher the priority and the closer to the front the item will be displayed.

Parameter Replacement

In many of the examples so far, parameter replacement has been used to allow for the dynamic creation of item and project names and namespaces. Parameter replacement involves using placeholders in the metadata file or in the included source files to indicate where a certain parameter value should be inserted. When you create a new item or project from a template, Visual Studio inserts actual values for the placeholders. There are several built-in parameters that can be used; it is also possible to add your own parameters to be passed to the wizard for replacement.

For parameter replacement to occur, the ReplaceParameters attribute must be set to true on either the Project element or the ProjectItem element. These are the two main locations where parameter replacement is used. When the wizard processes the project files and the included items, for each item where ReplaceParameters is set to true, the wizard searches through a dictionary of values to see if it has any known keys that match the parameters in the file. Parameters with a match are replaced with the stored values. Figure 10 shows the built-in parameters that are available to be used in metadata files and source files.

Figure 10 Built-In Replacement Parameters (Close Figure 10)

Parameters | Description
itemname | The user-provided name from the dialog. This value is often used for naming classes and files.
safeitemname | The same as itemname but with all unsafe characters removed.
safeitemrootname | The name of the root item, which can be used for multi-item templates. For example, if you have partial classes, or a code-beside or code-behind scenario, you can use this item in the secondary files when you need the safeitemname of the main item being created.
projectname | The user-provided name of the project for a project template.
safeprojectname | The user-provided name of the project for a project template with all unsafe characters removed.
rootnamespace | The root namespace of the project for an item template, which can be used to replace the namespace in a project item source file.
guid[1-10] | A GUID that can be used as a unique identifier for uses such as project GUID in the project file. You can specify up to 10 different GUID parameters using the syntax: guidx, where x is a number between 1 and 10.
time | The current time in the format DD/MM/YYYY HH:MM:SS.
year | The current four-digit year.
username | The Windows username of the logged-in user.
userdomain | The Windows domain of the logged-in user.
machinename | The name of the machine where the template is being used.
clrversion | The version of the common language runtime being used.
registered-organization | The registered organization of the user based on the system settings.
wizarddata | A single string that can be any XML or string data included in the WizardData element in a template metadata file.

Custom parameters can also be added to the template and are then available in the item templates and metadata file as well. To add custom parameters, you add a CustomParameters section within the TemplateContent element of the metadata file. For each parameter you want to add, you can specify a CustomParameter element with a name and the value (see Figure 11). Name your values using the $parametername$ syntax in this file in order to use the same naming convention in the source files.

Figure 11 Using the CustomParameter Element (Close Figure 11)
<VSTemplate Version="2.0.0"
  xmlns="http://schemas.microsoft.com/developer/vstemplate/2005"
  Type="Item">
    <TemplateData>
        ...
    </TemplateData>
    <TemplateContent>
        ...
        <CustomParameters>
            <CustomParameter
                Name="$mycustomparameter$"
                Value="ExampleValue" />
        </CustomParameters>
    </TemplateContent>
</VSTemplate>

Extending the Wizard

If all the extensibility and parameter replacement is not enough to meet your needs and you really have to write some code to provide the dynamic data you need for your template, you can write a wizard extension and configure it in the template metadata. An example of when you might need this type of extension is a case where you have several items in a template and need to be able to include or exclude some at run time. With a wizard extension, you can augment the standard New Project or New Item wizard with your own code and make decisions about what items to include or exclude as well as adding dynamic custom parameters for replacement in the project files.

In order to create your own wizard extension, you need to create a .NET library project in your managed language of choice and add a reference to the Microsoft.VisualStudio.TemplateWizardInterface.dll and EnvDTE.dll assemblies. Create a class in your library project and implement the Microsoft.VisualStudio.TemplateWizardInterface.IWizard interface. The IWizard interface members are listed in the following code:

public interface IWizard
{
    void BeforeOpeningFile(EnvDTE.ProjectItem projectItem) ;
    void ProjectFinishedGenerating(EnvDTE.Project project);
    void ProjectItemFinishedGenerating(EnvDTE.ProjectItem projectItem);
    void RunFinished();
    void RunStarted(object automationObject,
        Dictionary<string, string> replacementsDictionary,
        WizardRunKind runKind,
        object[] customParameters);
    bool ShouldAddProjectItem(string filePath);
}

The first three methods provide access to the IDE extensibility model for the items that have been added and might be opening. The RunFinished and RunStarted methods provide a means for running initialization and clean-up code. RunStarted is a common method to provide an implementation for and to write logic that sets flags used by the ShouldAddProjectItem method; these flags will determine whether an item from the template should be added to the project.

In order to configure the extension, you must modify the VSTemplate element in the template metadata file to add a WizardExtension element that defines the assembly information for your custom wizard extension. The abbreviated template file in Figure 12 shows how this configuration works. When the wizard is run, the wizard extension will be loaded and its methods called at the appropriate time to augment the wizard processing.

Figure 12 Using the WizardExtension Element (Close Figure 12)
<VSTemplate Version="2.0.0"
  xmlns="http://schemas.microsoft.com/developer/vstemplate/2005"
  Type="Item">
    <TemplateData>
        ...
    </TemplateData>
    <TemplateContent>
        ...
    </TemplateContent>
    <WizardExtension>
        <Assembly>MSDNMagazine.TemplateWizardExtension.dll</Assembly>
        <FullClassName>
        MSDNMagazine.TemplateWizardExtension.ExampleOrgWebPageExtension
        </FullClassName>
    </WizardExtension>
</VSTemplate>

Conclusion

Visual Studio 2005 has greatly improved the template capabilities and the ease with which you can create reusable, dynamic templates to provide robust starting points for projects and project items. The use of a simple ZIP file and an XML metadata file make customizing templates straightforward. In addition, the export features in Visual Studio make starting a template as simple as opening an existing project. The samples available for download provide several simple illustrations of the items discussed in this article put to use. See the Visual Studio Template Schema Reference in the MSDN® Library for additional details about adding customization to your template metafiles.

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Web services, XML, and BizTalk Server. As an instructor for Pluralsight, Matt teaches courses on Web services, BizTalk Server and ASP.NET. Matt lives in Minnesota with his wife Kristen and his son Max.


Unit Testing Tips: Write Maintainable Unit Tests That Will Save You Time And Tears

Unit Testing Tips

Write Maintainable Unit Tests That Will Save You Time And Tears

Roy Osherove
This article discusses:
  • The truth about unit testing
  • Testing the right things
  • Creating maintainable tests
  • Creating readable tests
This article uses the following technologies:
Visual Studio 2005, Visual Basic
Contents
The Truth About Unit Testing
Test the Right Thing
Creating Maintainable Tests
Creating Readable Tests
Avoid Partially Relevant Code in Your Setup Method
Parting Words

T
here's a lot of talk these days about unit testing and how one should go about writing unit tests for their applications under different scenarios (for starters, see my June 2005 MSDN®Magazine article on testing your data layer, available at Know Thy Code: Simplify Data Layer Unit Testing using Enterprise Services). That means there are a lot of developers who say to themselves (and to their teams) "Hey, we should start writing tests, too!" And so they begin writing unit test upon unit test until they reach a point where the tests themselves become a problem. Perhaps maintaining them is too hard and takes too long, or they are not readable enough to make sense, or maybe they have bugs.

It is at that point that developers are forced to make a tough decision: dedicate precious time to improving their tests or ignore the problem, effectively throwing away their hard work. The cause of this problem is simply inexperience writing unit tests.

In this article, I'll try to bring you some of the most important practices I've learned over the years while developing and consulting, and while training developers. These tips should help you write effective, maintainable, and robust unit tests. And I hope this advice helps you to avoid huge amounts of wasted time and effort.

The Truth About Unit Testing

In this section I'll outline some of the most common beliefs about the benefits to be gained by using many unit tests and explain why these beliefs are not always necessarily true. And then I'll help you make these beliefs hold true for your projects.

Tracking Bugs is Easier Well, not necessarily. How do you know that your tests are correct? That they fail when something actually breaks down? How do you know that you're covering enough code in your tests to ensure that if anything is wrong in your production code, that some test, somewhere, will break?

What happens if you have bugs in your unit tests? You'll suddenly start getting a lot of false positives—essentially, a bug will be found, but the problem won't be in your code under test. Your test's logic has a bug and therefore the test fails. These bugs are the most annoying and hardest to find because you're usually looking in the wrong place, checking your application instead of checking your tests. In Section I, I'll show you how to ensure that having a lot of unit tests does in fact make tracking bugs easier.

Code is Easier to Maintain Considering the last point, you're probably inclined to think this belief isn't necessarily true either. And you're right. Let's say that for each logical method in your code you have at least one test method. (Realistically, you'll probably have even more.) In projects with good test coverage, as much as 60 percent of the code can be unit tests. Now consider that the tests have to be maintained as well. What happens if you have 20 tests against a complex logical method and you add a parameter to the method? The tests won't compile. The same thing happens when you change constructors of classes. Suddenly you find yourself needing to change a lot of tests just to make sure your application still works. And that takes lots of time.

For this belief to be true, you need to make sure your tests are easy to maintain. Write them while keeping the DRY rule in mind: Don't Repeat Yourself. I'll look at this issue more closely later.

Code is More Understandable This is a benefit of unit tests that people don't usually expect at first. Think about changing code (say, a specific class or a method) in a project you've never seen before. How do you approach the code? You probably go around all the project code looking for places where this specific class or method is being used. Not surprisingly, unit tests are a great place to find such examples. And, when written correctly, unit tests can provide a handy set of API documentation for the project, easing the process of documenting and understanding code behavior written by old and new developers on the team.

However, this is only true if the tests are readable and understandable, a rule that many unit test developers don't follow. I'll expand on this belief, and show you how to write readable unit tests in the Readable Tests section of this article.

Test the Right Thing

One of the most common mistakes made by newcomers to Test Driven Development (TDD) is that they often confuse the "Fail first" requirement with "Fail by testing something illogical." For example, you might start with a method requirement with the spec:

' returns the sum of the two numbers
Function Sum(ByVal a As Integer, ByVal b As Integer) As Integer
So you might write a failing test like this:
<TestMethod()> _
Public Sub Sum_AddsOneAndTwo()
    Dim result As Integer = Sum(1, 2)
    Assert.AreEqual(4, result, "bad sum");
End Sub
While at first glance this approach might look like a good way to write a failing test, it totally misses the point of why you initially set out to write a failing test.

A failing test proves that there is something wrong with the production code. That is, the test should pass when the feature you're testing is done. With the current example, however, the test will fail even if the production code is complete because the test is not logically correct. Making it pass requires a change to the test itself—not a change to the production code. (Knowing when a change to the production code is required is the intent with test-first programming.) In short, this test does not reflect the end result you'd like when the production code is complete; thus it is not a good test.

A good test in TDD will require you to change the code to make it work as desired, rather than forcing you to reflect on the current reality or a desired result that is not logical with the requirements—for example, where 1+1 returns 0 just to fail the test. The example shown earlier is similar to this situation. In practice, the tests should reflect the results you'd expect if the current requirement would have worked. Then you can adjust the reality of your code to pass the test.

As a rule, a passing test should never be removed because passing tests serve as the regression tests for maintenance work. They are there to ensure that when you change code, you don't break any existing functionality that's already working. This is also why you shouldn't change a passing test unless the change is merely to make it more readable (in other words, refactoring the test).

When a Test Fails Incorrectly Sometimes you might encounter failing tests even though the change you made to the code was absolutely reasonable. This usually means you've encountered conflicting requirements. Commonly, this is when a new requirement (a changed feature) conflicts with an old requirement that may no longer be valid. There are two possible routes to go here:

  1. Delete the failing test after verifying that it is no longer valid—essentially since the old requirement is either invalid or is tested elsewhere.
  2. Change the old test so you test the new requirement (essentially using a new test), and test the old requirement under new settings (the test logic stays the same, but the initialization function may change).

Sometimes a test is still valid even if it uses invalid techniques to accomplish its task. For example, say you have a Person class with method Foo that behaves a certain way, and it is tested via Test X. Years later, another requirement comes along and the method's logic is enhanced to throw an exception when a new initialization feature in the object is missing. Suddenly, Test X fails, even though it has nothing to do with the method other than its use of the same class in order to test its functionality. The test fails because it is missing some initialization step before calling the method.

This doesn't mean you should remove Test X. You'd lose testing of some important functionality that should still work assuming correct initialization takes place. Instead, you might want to change the creation of the class to be initialized properly in your old test so that you can keep using it for its intended purpose.

Of course, if you have 200 tests failing just because they use that old constructor, you've got a problem maintaining your tests. This is why you should always remove duplication in your tests just as you should in production code.

Test Coverage and Testing Angles How do you know if you have good coverage for your new code? Try removing a line or a constraint check. If all tests still pass, you don't have enough code coverage and you probably need to add another unit test.

The best way to make sure you are adding the correct test is to not uncomment that line or check until you produce a test that fails until you do uncomment it. This may be hard, but if you can't think of a way to make this code fail, you probably don't have a good reason for writing that line of code in the first place.

You never know when the next developer will try to play with your code. He may try to optimize it or wrongly delete some essential line. If you don't have a test that will fail, other developers may never know they made a mistake.

You might also want to try replacing various usages of parameters that are passed into your method with constants. For example, take a look at this method:

Public Function Sum(ByVal x As Integer, ByVal y As Integer, _
        ByVal allowNegatives As Boolean) As Integer
    If Not allowNegatives Then Throw New Exception()
    Return x + y
End Function
You can mess with the code to test for coverage. Here are some variations on how to test for it:
' Try this...
If Not True Then ' replace flag with const
    If x < 0 OrElse y < 0 Then Throw New Exception()
End If

' Or this...
If Not allowNegatives Then
    ' replace check with const
    If False OrElse y < 0 Then Throw New Exception()
End If

If all the tests still pass, you're missing a test. Another red flag is when you have only one test that checks for various equality to values, such as with the following:

Assert.AreEqual(3, retval)
Seeing this only once (in one test) in relation to some method usually means you can safely return 3 as a value and all the tests for this method will still pass. This, of course, means you're missing a test. If you're doing code reviews on unit tests, this is an easy one to look out for.

Make sure that your tests are written as simply as possible. A unit test should not contain an if switch or any other logical statements. If you do find yourself writing something like a logical statement in your test, there's a good chance you're testing more than one thing. In doing so, you're making your test harder to read and maintain, while increasing the possibility of bugs. The KISS principle (Keep it simple, stupid) plays a large role in unit tests, as well as in production code. Keep your tests simple and you just might find bugs in your production code rather than in your unit tests.

Making Tests Easy to Run If your tests aren't easy to run, people won't trust them. Your application will most likely have two different kinds of tests:

  • Tests that can run smoothly without any configuration. (With this sort of test, I could go to any machine, get the latest version of your code and tests from source control, and run them all without a hitch.)
  • Tests that need some configuration before they can be run.
The first type is what you're after. The second type is what you often end up with, especially if you're new to unit tests. If you find yourself with tests that have special needs, that's okay for now. But it is important that you separate the two groups of tests so they can be run individually.

The idea is that any developer should be able to make a change and run some tests without having to do any special configurations to enable the tests. If there are some tests that need special attention before running, the developer needs to know about them, so she can spend time enabling those tests. Because a lot of developers are lazy by nature (not you, of course), you should assume that they won't do the necessary configurations. Instead, they'll let the tests fail because they have better things to do.

When people let tests fail, they begin to think they can't trust the tests. It's hard to tell if that test might have caught a real bug this time or if it is just firing another false positive. The developers may not even understand why the tests are failing in the first place. Once they don't trust your tests, the developers will stop running them. This, in turn, will result in undiscovered bugs, and this will start you down a dark path. Bugs lead to frustration. Frustration leads to anger. Anger leads to the Dark Side.

To avoid the Dark Side, make sure you always have a group of tests that are ready to go—tests that will always run safely and can always be trusted. Put the tests that belong in the configuration-challenged group in a different folder, tree, or project with specific instructions about what needs to be done to run them. By doing this, developers will have tests they can run (and trust) without investing time configuring them. And when they do have time and desire, they can configure and run the more involved tests.

Creating Maintainable Tests

Try to avoid testing private/protected members. This issue can get a little religious for some people, but I strongly believe that 99 percent of the time you can fully test a class by writing unit tests against its public interfaces alone. Testing private members can make your tests more brittle if some internal aspect of the class being tested changes. You should be able to test private functionality by invoking some public functionality elsewhere in the code. Testing only public members leads to tests that can withstand constant code refactorings and internal implementation changes, while still making sure the overall functionality stays the same.

Reuse your creation, manipulation, and assertion code when possible. Don't create instances of classes directly inside a unit test. If you see the word "new" in front of any class that is not part of the unit test framework, you should consider putting that creation code in a special factory method that creates the object instance for you. You can then reuse that method to get fresh instances of your class in other tests. This helps to keep the tests maintainable across time and guards your tests from unforeseen changes to the code under test. As an example, Figure 1 shows a couple of simple tests that use a Calc class.

Figure 1 Testing the Calc Class (Close Figure 1)
<TestMethod()> _
Public Sub Parse_EmptyString_ReturnsZero()
    Dim c As New Calc
    Const ZERO As Integer = 0
    Dim emptyStringParsedResult As Integer = c.Parse("")
    Assert.AreEqual(ZERO, emptyStringParsedResult)
End Sub

<TestMethod()> _
Public Sub Parse_StringWithOneDigit_ReturnsSameDigit()
    Dim c As New Calc
    Const SAME_DIGIT As Integer = 1
    Dim digitParseResult As Integer = c.Parse("1")
    Assert.AreEqual(SAME_DIGIT, digitParseResult)
End Sub

Suppose you have 20, or maybe even 100, tests against the Calc class, all looking surprisingly similar to these. Now a design change forces you to remove the default Calc constructor and use a different constructor that takes some parameters. Immediately, all your tests break. You might be able to fix this using a simple find and replace—or you might not. The main issue is that you'll waste valuable time fixing your tests. This isn't the case, though, if you use a factory method to create Calc instances in your test classes, as shown in Figure 2.

Figure 2 Using a Factory Method (Close Figure 2)
<TestMethod()> _
Public Sub Parse_EmptyString_ReturnsZero()
    Dim c As Calc = Factory_CreateDefaultCalc()
    Const EXPECTED_ZERO As Integer = 0
    Dim emptyInput As String = String.Empty
    Verify_Parse_Result(EXPECTED_ZERO, emptyInput, c)
End Sub

<TestMethod()> _
Public Sub Parse_StringWithOneDigit_ReturnsSameDigit()
    Dim c As Calc = Factory_CreateDefaultCalc()
    Const EXPECTED_SAME_DIGIT As Integer = 1
    Dim oneDigitInString As String = "1"
    Verify_Parse_Result(EXPECTED_SAME_DIGIT, oneDigitInString, c)
End Sub

Private Function Factory_CreateDefaultCalc() As Calc
    Return New Calc()
End Function

Private Sub Verify_Parse_Result(ByVal expected As Integer, _
        ByVal input As String, ByVal c As Calc)
    Dim result As Integer = c.Parse(input)
    Assert.AreEqual(expected, result)
End Sub

I've made a couple of changes to the tests to make them more maintainable. First, I moved the creation code into a reusable factory method. This means I would only have to change one simple method to make all the tests in this test class work with a new constructor. Another simple solution for the creation problem is to move the creation into the <TestInitialize()> method of the test class. Unfortunately, this works well only when you want to reuse an object as a local class variable in many tests. If you only use it for some of the tests (partially relevant members), you might as well instantiate them in the test itself to make it more readable.

By the way, notice that I've named the method Factory_CreateDefaultCalc. I like to name any helper methods in my test class with special prefixes so that I know what they are used for. This can help with readability.

My second change was to reuse the assertion code in the test by moving this code into a verification method. A verification method is a reusable method in your test class that contains an Assert statement but that can take different inputs and verify something on them. You use verification methods when you are asserting the same thing over and over again with varying inputs or initial state. The nice thing about this is that even though the Assert is located in a different method, if the Assert fails you'll still get an assert exception and the original calling test will be shown in the test failure output window.

I'm also sending in the Calc instance instead of using a local variable, so I know I always send an instance that has been initialized by the calling test. You may want to do the same thing when changing object state—for instance, when configuring specific objects under test or objects that will be sent to the tests using specific Configure_XX methods. Those methods should explain what they configure an object to be used for. The code in Figure 3 shows an example of this.

Figure 3 Tests with Object Configuration (Close Figure 3)
<TestMethod()> _
Public Sub IsLoginOK_BadPassword_ReturnsFalse()
    Dim userData As New UserData("user")
    userData.SetPassword("password")
    loginManager.AddUser(userData)
    Dim result As Boolean = loginManager.IsLoginOK("user", "badPassword")
    Assert.IsFalse(result, "Login should have failed for a bad password")
End Sub

<TestMethod()> _
Public Sub IsLoginOK_GoodPassword_ReturnsTrue()
    Dim userData As New UserData("user")
    userData.SetPassword("password")
    loginManager.AddUser(userData)
    Dim result As Boolean = loginManager.IsLoginOK("user", "password")
    Assert.IsTrue(result, _
        "Login should have succeeded for a good password")
End Sub

This test has a lot of setup code that deals with adding initial state to the LoginManager object, which is a member in this test class. There is certainly some repetition here. Figure 4 shows how this example looks after refactoring out the initialization code.

Figure 4 Refactoring Repeated Initialization (Close Figure 4)
<TestMethod()> _
Public Sub IsLoginOK_BadPassword_ReturnsFalse()
    Configure_InitLoginManagerWithUser(loginManager, "user", "password")
    Dim result As Boolean = loginManager.IsLoginOK("user", "badPassword")
    Assert.IsFalse(result, "Login should have failed for a bad password")
End Sub

<TestMethod()> _
Public Sub IsLoginOK_GoodPassword_ReturnsTrue()
    Configure_InitLoginManagerWithUser(loginManager, "user", "password")
    Dim result As Boolean = loginManager.IsLoginOK("user", "password")
    Assert.IsTrue(result, _
        "Login should have succeeded for a good password")
End Sub

Private Sub Configure_InitLoginManagerWithUser( _
        ByVal login As LoginManager, ByVal user As String, _
        ByVal pass As String)
    Dim userData As New UserData(user)
    userData.SetPassword(pass)
    login.AddUser(userData)
End Sub

The revised tests are much more readable and maintainable. Just be careful not to refactor your tests so much that they end up being a single, unreadable line of code. Note that I could have also used a Verify_XX method here, but that's not really what I set out to illustrate here.

Avoid Dependencies Between Tests A test should be able to stand on its own. It should not rely on any other test, nor should it depend on tests being run in a specific order. I should be able to take all the tests you've written, run all or just some of them, in any order, and know that they will always behave the same. If you don't enforce this rule, you will end up with tests that only behave as expected when run in specific situations. This, of course, is problematic when you're under a deadline and you want to make sure you didn't introduce any new bugs into the system. You may get confused and think that there's something wrong with your code when, in reality, the problem is simply the order in which your tests are running. As a result, you may start to lose faith in your tests and write less and less of them. This is a long and slippery road.

If you call out from one test to another test, you create a dependency between them. You essentially test two things in one test (I'll explain why this is a problem in the next section). If, on the other hand, you have Test B and it depends on a state created by Test A, you fall into the "ordering" trap. If you or someone else were to change Test A, Test B would break and you wouldn't know why. Troubleshooting this failure can steal a lot of time.

Using <TestInitialize()> and <TestCleanup()> methods is essential to obtaining better test isolation. Make sure that your test always uses fresh, new instances of objects under test, and that all state is known in advance and will always be the same no matter where or when your test is run.

Avoid Multiple Asserts in a Single Unit Test Consider Assert failures as symptoms of a disease and Asserts as indication points or blood checks for the body of the software. The more symptoms you can find, the easier the disease will be to diagnose and treat. If you have multiple Asserts in one test, only the first failing Assert will reveal itself as failed by throwing an exception. Consider the test illustrated in the following code:

<TestMethod()> _
Public Sub Sum_AnyParamBiggerThan1000IsNotSummed()
    Assert.AreEqual(3, Sum(1001, 1, 2)
    Assert.AreEqual(3, Sum(1, 1001, 2) ' Assert fails
    Assert.AreEqual(3, Sum(1, 2, 1001) ' This line never executes
End Sub
You lose sight of other possible symptoms from that line onwards. After a failure, subsequent Asserts aren't executed. These unused Asserts could provide valuable data (or symptoms) that would help you quickly narrow your focus and discover the underlying problem. So running multiple Asserts in a single test adds complexity with little value. Additional Asserts should be run in separate, self-contained unit tests so that you have a good opportunity to see what fails.

Creating Readable Tests

If you've written unit tests before, do all of your unit tests have a good message on the Assert line? Probably not. Most developers don't bother writing a good Assert message because they are more concerned with writing the test.

Assume you're the new developer on a team and you're trying to read a unit test. Like this one:

<TestMethod()> _
Public Sub TestCalcParseNegative()
    Dim c As New Calc
    Assert.AreEqual(1000, c.Parse("-1, -1000")
End Sub
As a simple exercise, see if you understand the usage case of the Parse method of Calc in this case. You probably have a good guess, but this could easily be any number of usage cases that output the result of 1000:
  • Return the largest negative number in the group as a positive
  • Ignore the first number if negative and return sum of rest as positive
  • Return the numbers multiplied by each other
Now consider this small change in the unit test:
<TestMethod()> _
Public Sub Parse_NegativeFirstNum_ReturnsSumOfTheRestAsPositive()
    Dim c As New Calc
    Dim parsedSumResult As Integer = c.Parse("-1", "-1000")
    Const SUM_WITH_IGNORED_FIRST_NUM As Integer = 1000
    Assert.AreEqual(SUM_WITH_IGNORED_FIRST_NUM, parsedSumResult)
End Sub
Isn't this much easier to understand? When the Assert message is gone, the best place to express intent is in the test name. If you use it wisely, you'll find you don't need to read the test code to understand what the code tests. In fact, you often won't need to write any comments at all because the code, such as with this example, is self documenting.

The name contains three parts: the name of the method under test (Parse), the state or rule under test (sending in a string with a first negative number), and the expected output or behavior (the sum of the rest of the numbers are returned as a positive). Notice that I removed the words Test and Calc from the name. I already know this is a test by the attribute so there's no need to repeat this info. I also know this is a test on the Calc class because test classes are usually written for one specific class (this class would probably have been called CalcTests).

The name is long, but who cares? It reads much like a sentence in standard English and makes it easy for a newcomer to understand the test. More so, when this test fails, I'll know what the problem is and maybe even without debugging the code.

Notice that I've gone ahead and separated the actual act of parsing from the act of asserting on the result by creating a result variable on a different line. There are at least two reasons for this. First, you can assign a readable name to a variable that contains the result, which makes your Assert line very understandable and easy to read. Second, the invocation against the object under test may be very long and might make your Assert line stretch all the way beyond the edge of the screen, forcing the test reader to scroll to the right. Personally, that's one of the things I find most annoying.

I use a lot of constants in my tests to make sure my Asserts read like a book. In the previous example, you could read the Assert to say "make sure that the parsed sum is equal to the sum with the first number ignored." Good naming for your variables can sometimes make up for a badly named test.

Of course, sometimes an Assert message is the best way to convey intent in a unit test. A good Assert message should always explain either what should have happened or what happened and why it's wrong. For example, "Parse should have ignored the first number if it is a negative," "Parse did not ignore the first negative number," and "X called object Y even though flag was false" are all useful Assert messages that clearly describe the resulting situation.

Avoid Partially Relevant Code in Your Setup Method

A <TestInitialize()> method is a great place to instantiate member variables that will be used by your tests. All your tests. Avoid variables that are only used by some of the tests. Those can be local variables within the test itself. If you create partially relevant instances as class members simply to avoid duplication of creation in the tests, you should use factory methods as explained earlier in this article. Using partially relevant variables makes both your code and the setup method less readable. Only variables to be used in each and every test should be member variables and used in the <TestInitialize()> method.

Figure 5 shows a class that has two member variables for testing, but one of them (cxNum) is only partially used. Figure 6 shows how you might replace the code in the tests to make it more readable.

Figure 6 Rewritten Code for Better Readability (Close Figure 6)
Dim calc As Calc

<TestInitialize()> _
Public Sub Setup()
    Calc = New Calc()
End Sub

<TestMethod()> _
Public Sub Parse_EmptyString_ReturnsZero()
    Dim result As Integer = calc.Parse("")
    Assert.AreEqual(0, result)
End Sub

<TestMethod()> _
Public Sub ParseComplex_BadComplexNumber_ReturnsZero()
    Dim badCxNum As ComplexNumber = Factory_CreateBadComplexNumber()
    Dim result As Integer = calc.ParseComplex(badCxNum)
    Assert.AreEqual(0, result)
End Sub

Private Function Factory_CreateBadComplexNumber() As ComplexNumber
    Return New ComplexNumber(1, 2)
End Function
Figure 5 Written with Partially Relevant Code (Close Figure 5)
Dim calc As Calc
Dim cxNum As ComplexNumber

<TestInitialize()> _
Public Sub Setup()
    Calc = New Calc()
    cxNum = New ComplexNumber(1, 2)
End Sub

<TestMethod()> _
Public Sub Parse_EmptyString_ReturnsZero()
    Dim result As Integer = calc.Parse("")
    Assert.AreEqual(0, result)
End Sub

<TestMethod()> _
Public Sub ParseComplex_BadComplexNumber_ReturnsZero()
    Dim result As Integer = calc.ParseComplex(cxNum)
    Assert.AreEqual(0, result)
End Sub

Parting Words

As you can see, writing unit tests is not a trivial task. Approached correctly, unit tests can yield amazing results for developer productivity and quality of code. It can help you to create applications with far fewer errors, while also giving other developers insight into your code. But it takes a commitment upfront, making sure to follow some simple rules. When approached poorly, unit tests can achieve the opposite results, stealing valuable time and complicating the testing process.

Roy Osherove is the Principal of Team Agile, a consultancy firm devoted to agile software development and .NET architecture. Roy also maintains a blog with related information at www.iserializable.com. You can reach him at Roy@TeamAgile.com.


Editor's Note: Back to the Future

Editor's Note
Back to the Future

As we head into the cold winter months, we've got great content to keep you warm! You might wonder why I started this Editor's Note with such a tired intro, but we have a reason. We're covering a lot of fresh, exciting new material this month, and we didn't want to upstage the good news.

We're starting our 21st volume of MSDN Magazine, and we're always looking for ways to freshen our publication to keep it interesting and useful for you. This month, we're adding a new column, relaunching two others, and we've even added some great authors to help recreate some old favorites.

Dev Q&A In 1996, we launched Microsoft Internet Developer Magazine, which was later incorporated into MSDN Magazine. In the first issue of MIND, we introduced Geek to Geek, a column written by Robert Hess. Later, Nancy Michell ably took over the compilation and we renamed the column Web Q&A. Recently, we've decided that the original concept—pulling expert questions and answers—would best be applied to a variety of thematic programming. Nancy will continue to gather the best questions and answers from Microsoft professionals under a broader banner: Dev Q&A.

Toolbox In the same first issue of MIND, we introduced New Stuff, a monthly look at new products for developers and their friends. After 10 years reporting what's new, the column, renamed Toolbox, will dive deeper into the tools you need to get your job done. Scott Mitchell, a Microsoft MVP and longtime developer, will lend his experience to the column to critically evaluate new products and bring a unique viewpoint to offerings in the marketplace.

CLR Inside Out The Java language was introduced just 10 years ago. Five years later, Microsoft began discussing the vision for .NET and the CLR. The .NET Framework has now been with us for longer than the Java language had been around when .NET was introduced! There's so much to cover in the CLR, especially with version 2.0 arriving, that we're devoting a column to it. CLR Inside Out comes to you from the source—the Microsoft team who designed the CLR.

We also have some big names stepping in to write columns this month. Brad McCabe makes an appearance in Advanced Basics to discuss how you can make noises with the .NET Framework 2.0, and Fritz Onion lends his pen to Extreme ASP.NET to talk about codebehind and compilation in ASP.NET 2.0.

And there's so much more. Now that Windows Vista is available in beta form to MSDN subscribers, we want to give you a head start learning about some of the technologies you'll be using over the next several years.

WinFX is the managed code model for Windows Vista, building upon and extending the .NET Framework. If you haven't checked this out yet, you should—it's really cool. In this issue, we cover several technologies exposed through WinFX:

Windows Presentation Foundation Do you dream of a better graphical subsystem than what's available today? The Windows Presentation Foundation will offer you some remarkable new capabilities. It will be made available for Windows XP as well.

The XML Paper Specification This technology, known as XPS, lets users create documents that have a consistent appearance independent of where or how the document is viewed.

WinFX Speech API You really get a lot out of WinFX, and that includes improved speech APIs. This month, we'll take a look at how to use them to your advantage.

Windows Workflow Foundation The Windows Workflow Foundation will be the programming model for building apps on Windows that include both system and human workflow features.

None of this is way-out stuff—you can get your hands on all of it today if you're an MSDN subscriber. We're quite pleased that some technologies, like Windows Presentation Foundation, are being designed for Windows Vista, but will be available for Windows XP and Windows Server 2003 as well. That's pretty cool.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Ryan Byington, Rob Chambers, Kevin Cogger, Prasadi de Silva, Mike Downen, David Ebbo, David Fetterman, Mark Fussell, Brian Grunkemeyer, David Gutierrez, Daniel Herling, Eric Jarvi, Ravi Krishnaswamy, Jesse McGatha, Shaun Miller, Bradley Millington, Anthony Moore, Rob Relyea, Chris Sells, Payam Shodjai, Andrey Shur, Robert Stumberger, and Ravinder Vuppula.

JScript, Microsoft, MSDN, SharePoint, Visio Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Win32, WinFX, Windows Mobile, Windows NT, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Code Generation, A Charting Component, and More

Toolbox
Code Generation, A Charting Component, and More
Scott Mitchell

Contents
Save Time with Code Generation
Build Snazzy Charts Quickly and Easily
Turn Images into Icons
The Bookshelf

Developers have a number of tools at their disposal, but which of these components do you really need to complete your project? Each month we'll analyze the latest software and books to see if they merit inclusion in your toolbox.

Save Time with Code Generation

Last week your boss handed you the technical specs for your company's latest and greatest data-driven application. Over the past several days you've spent countless hours meticulously crafting the database tables, and the myriad of C# or Visual Basic® business objects needed to model these database entities in code. Great! You're now ready to start working on the fun stuff, the business logic and the front-end.

Your enthusiasm quickly wanes when you receive an e-mail from your boss informing you that the specs weren't as rock solid as he had thought: some of the database tables need to be removed, some new tables need to be added, and many others require one or more new fields.

Anytime there's a low-level change to the system's data model, there's bound to be a lot of work to update the system to reflect the change. While changes to the back-end can radically affect the front-end portion of the application, much of the middle-tier architecture's changes are rather tedious and therefore ripe for automation. Why write the code yourself when a program can write it for you?

CodeSmith, by CodeSmith Tools, LLC, is a code generation tool that can help remove the tedium of any oft-repeated programming task through the use of templates. A template can be executed, which outputs text that contains the automated code, stored procedures, SQL scripts, or whatever else you customize the template for. A template in CodeSmith is composed of three parts: properties, source code that's executed at runtime, and a mix of static and dynamic text. Armed with this template you could create a business object for any database table with the click of a button.

While you can write your own CodeSmith templates from scratch, a number of popular templates ship with the software. There's also an active user-driven template community available on CodeSmith's Web site where enthusiasts share their customized templates. CodeSmith also offers integrated Visual Studio® support, allowing template creation and code generation straight from the IDE. CodeSmith templates can also be executed from the command line, making it possible to integrate code generation within the build process.

Automated code generation works best if it's embraced as early as possible in a project. While you can create and customize templates to fit into an existing application, it might not be worth the time and effort if the application is fairly stable. If, however, you work on an ever-changing application or are beginning a new project, consider using CodeSmith to assist in churning out the application's tedious code.

Price: $299 for the Professional Version.

www.codesmithtools.com

Build Snazzy Charts Quickly and Easily

A picture is worth a thousand words, and nowhere is this more true than in reporting. While tables of numerical data may be interesting, nothing can replace the value of a well-designed, professional-looking chart. Pictorial representations of data better highlight trends and enable the viewer to more quickly grasp the "big picture."

In my experience I have found that charts should most definitely be provided, but only if they're done right. While charts can be constructed programmatically from the System.Drawing classes, crafting anything but the simplest of charts will likely eat away more time than forecast, and, even then, will likely look amateurish. In short, buy a quality charting component, and use it!

When evaluating a charting component, I look for three things:

Ease of Use How long does it take to go from nothing to a professional-looking chart?

How Well Does it Work with My Data My data may be in a database, an XML file, or a custom business object. How easy is it to get my data, in whatever form it may be, onto the charting surface?

Bells and Whistles What charts are available, can the end user interact with the chart, how easy is it to customize the colors, fonts, and appearance of the chart, and so on.

The charting component I'm particularly fond of is Dundas Charts, from Dundas Software, which meets and exceeds all three of my charting component criteria. With Dundas Charts I can go from nothing to a fully functional, impressive-looking chart in a matter of minutes. The wizard, which is fully integrated with Visual Studio, makes it easy to customize the chart's type and style settings. The object model allows you to bind any enumerable data source—DataSets, DataReaders, arrays, lists, and so on—to any one of a number of chart properties. Put these two features together and you can create the chart, customize its appearance, and bind it to data in a matter of minutes.

What is especially impressive about Dundas Charts is the quality of the charts. A variety of attractive chart styles make it easy to create eye-pleasing charts, even for someone as artistically challenged as I am. Furthermore, there are plenty of bells and whistles with Dundas Charts. Want to make that chart 3D? Simply click a checkbox. Want to have the chart animated? That's another checkbox. Want to add a trend line, based on one of a number of statistical models? That's one line of code.

When displaying charts online, you can render the charts as image files (JPG, PNG, BMP, and so on), as Scalable Vector Graphics (SVG), or as a Macromedia Flash file. There's also support for image maps. All of the chart rendering is done server-side, so there's no dependencies required on the client (unless you render as SVG or Flash, in which case the browser must have the appropriate software installed).

While there is both an ASP.NET version and a Windows® Forms version, my experience has been solely with the ASP.NET version. Dundas Charts has made creating snazzy-looking charts an easy, fast, and fun task; highly recommended!

Price: $699 for the ASP.NET or Windows Forms Professional Version.

www.dundas.com

Turn Images into Icons

If you are a micro-ISV, operate as an independent consultant, or work in a small company, you likely don't have a full-time resource that you can tap to meet your graphic and icon needs. You may have an artistic friend or know a freelance graphic artist you can use when the need arises, but you hate to bug your friend, and that graphic artist isn't cheap. What's a graphically impaired developer to do?

If you simply need icons for your application, check out Frontbase Image to Icon (created by Frontbase, Ltd.). It is designed to do one thing—- take an existing image and convert it into an icon. You can now grab your logo or that image the graphics designer already whipped up, and convert it into an icon on your own.

Frontbase Image to Icon is a cinch to use. Simply select an image file—BMP, GIF, JPG, PNG, or other formats—- and you've got yourself an icon! You can specify the icon's dimensions and color depth, modify the icon's transparency pixel-by-pixel, and adjust the contrast, brightness, and colors of the resulting icon. Image to Icon also supports multiple images per icon file.

Frontbase Image to Icon is the perfect solution for the graphically challenged developer who needs to be able to quickly generate icons from existing image files.

Price: $19.95.

www.image2icon.com

The Bookshelf

If you're a hard-core .NET coder, you no doubt spend the majority of your day using Visual Studio. While most developers use Visual Studio as a means for writing and debugging code, the IDE offers much more functionality underneath the surface. You can monitor and tweak your servers, manage your database's structure and data, and integrate third-party software through the use of Add-Ins.

Visual Studio Hacks (O'Reilly, 2005), by James Avery, provides 100 quick lessons about Visual Studio. (Note: I authored five of the hacks.) The 100 hacks span a variety of useful topics, such as working with the editor, mastering debugging, customizing Visual Studio, and extending and enhancing Visual Studio through Add-Ins. One of my main concerns with this book was what version of Visual Studio it would focus on. Would it look solely at Visual Studio 2005 hacks, or would there be information for developers still using previous versions? The good news is that the book is fairly balanced. Most of the hacks discussed apply to both Visual Studio .NET 2003 and Visual Studio 2005, with any version-related differences noted in the text.

Due to the vast amount of time I spend using Visual Studio, what interested me most was information that could improve my efficiency with the IDE. There are two chapters in Visual Studio Hacks that achieved this aim: Speed Hacks and Customizing Visual Studio.

The Speed Hacks chapter includes 15 hacks for decreasing the time it takes to get Visual Studio to do what you want. This includes tips on creating and using macros, quickly creating connection strings, and inserting commonly used code automatically. I found this chapter the most useful of all the 13 in the book. Customizing Visual Studio also proved helpful, as it shows how to tailor the Visual Studio environment to match your personality and work style.

The book wasn't written to be an end-all reference for using Visual Studio. If you are new to the IDE, or need to become an in-depth Visual Studio guru, there are better books available. If, however, you use Visual Studio on a regular basis and want some quick, interesting tips and tidbits of information on how to customize and get the most out of Visual Studio, this book is for you.

Price: $24.95, 500 pages.

www.oreilly.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

All prices were confirmed at press time and are subject to change.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


Dev Q&amp;A: DataGridView

Dev Q&A
DataGridView
Edited by Nancy Michell

After receiving a late-breaking news bulletin at magazine headquarters early this month stating that the Web is just a fad that will never amount to anything, we have reluctantly decided to retire our venerable Web Q&A column and replace it with one we like to call Dev Q&A. Seriously, Web programming has grown far beyond HTML-formatted text to encompass just about any kind of programming you can imagine. Limiting the scope of this column to Web-only questions seems a little superficial. Dev Q&A will take a look at questions related to many aspects of programming in Visual Basic®, C#, ASP.NET, and the like. Compiled from questions submitted internally and answered by Microsoft developers, the column will present useful solutions to commonly encountered coding dilemmas. This month we'll discuss the Windows® Forms DataGridView control.

Q How do I get a DataGridView to scroll programmatically to a given row index? I have tried to define the following helper in my grid class derived from DataGridView:

public void ScrollToRow ( int rowIndex )
{
  OnScroll(new ScrollEventArgs(
    ScrollEventType.LargeIncrement, rowIndex));
}
However, it does not work. I'm using the Microsoft® .NET Framework 2.0 Beta 2.

A There are a number of properties and methods that can be used in order to scroll the rows or columns programmatically. Among others, they include FirstDisplayedScrollingColumnIndex, FirstDisplayedScrollingRowIndex, FirstDisplayedScrollingColumnHiddenWidth, HorizontalScrollingOffset, and VerticalScrollingOffset. The DataGridView control also exposes the VerticalScrollBar and HorizontalScrollBar protected properties, which you can use in order to display a particular row or column and control the built-in scrollbars.

Q I am trying to add rows dynamically to a Windows Forms DataGrid bound to an ArrayList, but the new row is not displayed in the DataGrid control. How can I show the newly appended row? Here's my code:

private void addButton_Click(object sender, System.EventArgs e)
{
  XmlAttribute attribute = GetAttributeToAdd();
  this.attributeArray.Add(attribute);

  this.attributeDataGrid.DataBindings.Clear();
  this.attributeDataGrid.DataSource = attributeArray;
  this.attributeDataGrid.SetDataBinding(this.attributeArray, null);
  this.attributeDataGrid.Refresh();
}

A You need to wrap the code that adds items to the array list in a currencyManager.SuspendBinding/ResumeBinding block. By the way, you don't need to set the Data source on the DataGrid each time you add an item. Without that, the added code is a bit simpler, as you see here:

// PRECONDITION: the DataGrid's Data source is already set.
private void addButton_Click(object sender, System.EventArgs e)
{
  this.BindingContext[this.attributeDataGrid.DataSource,
    this.attributeDataGrid.DataMember].SuspendBinding();

  XmlAttribute attribute = GetAttributeToAdd();
  this.attributeArray.Add(attribute);

  this.BindingContext[this.attributeDataGrid.DataSource,
    this.attributeDataGrid.DataMember].ResumeBinding();
}

Note that in the .NET Framework 2.0, there are better data binding mechanisms than ArrayList, such as BindingList<T>. BindingList<T> is a generic implementation of IBindingList and is very useful for data binding.

Q I have a DataGridView problem: when dragging column headers to different locations, the column header flickers. I derive from the DataGridView to provide grouping support, but I have no custom logic for handling column moves. Can you help?

A In your derived class, have you set DoubleBuffer to be true? Doing so could cause flickering when moving columns around. To fix this, you could turn off DoubleBuffer or write code for custom painting when moving columns.

A better solution, so you don't have to write code for custom painting or sacrifice the advantages of double buffering, is shown in Figure 1. The code in this solution looks to see if you are on the column header and, if so, turns off DoubleBuffer, but only while manipulating the column headers.

Figure 1 Eliminating Flicker While Moving Columns (Close Figure 1)
protected override void OnMouseDown(MouseEventArgs e)
{
    base.OnMouseDown(e);

    DataGridView.HitTestInfo hitTestInfo = this.HitTest(e.X, e.Y);
    if (hitTestInfo.Type == DataGridViewHitTestType.ColumnHeader)
    {
        this.DoubleBuffered = false;
    }
}

protected override void OnMouseUp(MouseEventArgs e)
{
    base.OnMouseUp(e);
    if (!this.DoubleBuffered)
    {
        this.DoubleBuffered = true;
    }
}

Q In my Windows-based application, I have a DataGridView. A BindingList<BusinessObject> contains my business objects, and I've assigned it as the grid's data source. Now, sorting does not work. Do you have a solution?

A There are two ways to sort with the DataGridView: one is in an unbound DataGridView, where you can use the SortCompare method. The other is if you're bound to a list that implements sorting. The BindingList has hooks so that you can support sorting and searching, but neither of these is built in. To support sorting, you should inherit a new list type from BindingList<BusinessObject> and override the sorting methods/properties (there are several, including SupportsSortingCore, IsSortedCore, SortPropertyCore, SortDirectionCore, ApplySortCore, and RemoveSortCore).

Q Do you know of a property that will allow a user to type a new value into a DataGridView ComboBox?

A There is no property to enable this for a DataGridView ComboBox, but you can write some code yourself to do it. First, you need to handle the EditingControlShowing event in order to change the DropDownStyle of the ComboBox control to DropDown. Then, in the CellValidating event, add the FormattedValue value to the ComboBox list if it doesn't already exist. This has been implemented in Figure 2.

Figure 2 Interacting with the DataGridView ComboBox (Close Figure 2)
private void dataGridView1_EditingControlShowing(
    object sender, DataGridViewEditingControlShowingEventArgs e)
{
    ComboBox c = e.Control as ComboBox;
    if (c != null) c.DropDownStyle = ComboBoxStyle.DropDown;
}

private void dataGridView1_CellValidating(
    object sender, DataGridViewCellValidatingEventArgs e)
{
    if (e.ColumnIndex == comboBoxColumn.Index)
    {
        object eFV = e.FormattedValue;
        if (!comboBoxColumn.Items.Contains(eFV))
        {
            comboBoxColumn.Items.Add(eFV);
        }
    }
}

Q Do ComboBox controls get reused across columns, or does each column get its own ComboBox editing control?

A ComboBox controls can be reused across columns. The editing control type is cached and if the types are the same, then the control is reused. However, a lot of properties are reset in the process (including DropDownStyle).

Q I've used the columns editor in Visual Studio 2005 to define the set of columns that I'd like to show in the DataGridView, but it displays all of the properties, not just the ones I define. How do I limit the display to just the ones I want?

A By default, at design time the designer adds all public properties it finds that aren't attributed as Browsable(false). You can then use the columns editor to modify the column set (add/remove).

At run time you can set the AutoGenerateColumns property to false and manually add columns (using the DataGridView.Columns.Add method) to the DataGridView. You set the column's DataPropertyName to identify if the column is bound and what field it is bound to.

Q Can you tell me the best way to bind XML data to a DataGridView control?

A The simplest possible solution is to load the XML into a DataSet and bind to that. Another solution would be to use XML serialization to create an object graph from the XML and bind to that. To bind to the raw XML, you would need to create wrapper classes that implement ITypedList, IBindingList, and ICustomTypeDescriptor.

Q In a DataGridView with columns of type DataGridViewComboBoxColumn, if the user selects a new value in the cell using the ComboBox dropdown, a new row is automatically added to the end of the grid if there are no more empty rows. I need the user to be able to type inside the cells, so I handle the EditingControlShowing event and change the DropDownStyle of the ComboBox. However, the new row is not added if the user does not use the dropdown to enter the data.

A This behavior is by design. The grid doesn't know that any value has changed, since it only listens to the SelectedIndexChanged event. As shown in Figure 3, you can manually listen to the ComboBox's TextChanged event and then call the DataGridView's NotifyCurrentCellDirty method so that the new row gets added the moment the user types in the new value.

Figure 3 Using NotifyCurrentCellDirty (Close Figure 3)
this.m_dataGridView.EditingControlShowing +=
    new DataGridViewEditingControlShowingEventHandler(
        this.m_dataGridView_EditingControlShowing);

void m_dataGridView_EditingControlShowing(object sender, DataGridViewEditingControlShowingEventArgs e)
{
    ComboBox comboBox = e.Control as ComboBox;
    if (comboBox != null)
    {
        comboBox.DropDownStyle = ComboBoxStyle.DropDown;
        comboBox.TextChanged += new EventHandler(comboBox_TextChanged);
    }
}

void comboBox_TextChanged(object sender, EventArgs e)
{
    this.m_dataGridView.NotifyCurrentCellDirty(true);
}

Q I have switched recently to the .NET Framework 2.0. I am now considering using DataGridView instead of DataGrid in my app. I need a checkbox column that has checked and unchecked states, and a disabled state. How can I do this?

A The DataGridView does not have built-in support for disabled controls, so you will have to implement your own checkbox cell/column. The documentation includes a sample for creating a disabled button cell/column.

Q How do I make a ComboBox appear only for the row that has current focus in a DataGridView, not just for the cell that has focus?

A You need to set the DataGridViewComboBoxColumn.DisplayStyle property to Nothing for all ComboBox columns. Finally, use the DataGridView RowEnter event and RowLeave events to toggle the DataGridViewComboBoxCell.DisplayStyle property from Nothing to ComboBox for the specific row. Figure 4 shows an example that assumes the ComboBox column is the first column.

Figure 4 Showing a ComboBox for the Current Row (Close Figure 4)
private void dataGridView1_RowEnter(
    object sender, DataGridViewCellEventArgs e)
{
    DataGridViewComboBoxCell comboBoxCell = (DataGridViewComboBoxCell)
        this.dataGridView1.Rows[e.RowIndex].Cells[0];
    comboBoxCell.DisplayStyle = DataGridViewComboBoxDisplayStyle.ComboBox;
}

private void dataGridView1_RowLeave(
    object sender, DataGridViewCellEventArgs e)
{
    DataGridViewComboBoxCell comboBoxCell = (DataGridViewComboBoxCell)
        this.dataGridView1.Rows[e.RowIndex].Cells[0];
    comboBoxCell.DisplayStyle = DataGridViewComboBoxDisplayStyle.Nothing;
}

Q I need to display a specific set of images in a cell based upon the integer value of the cell. For example, when the cell's value is three, I need three dots to appear in the cell. How do I do this?

A You'll need to create a new cell class that inherits from the DataGridViewImageCell. By default, the DataGridViewImageCell's formatted value and real value types are both images. In your custom image cell you need to set the ValueType to be integer (since your values are integers, not images). Next you need to override the GetFormattedValue property where you provide an integer to image mapping. This code can be as complex as necessary, but you should avoid it being slow, since the formatted value is used many times during painting.

In your case you can create three static images that contain dots for each integer value you want. In the GetFormattedValue code, you will return the correct image based upon the integer. Finally, if the grid's AllowUserToAddRows is true then you need to override the DefaultNewRowValue property and return a default integer value. This is necessary because the default DataGridViewImageCell returns a red "X" image, but you need the default value to be an integer. Creating a custom image column as well will make it easy to use your custom image cell. An example is shown in Figure 5.

Figure 5 Custom Image Column for the DataGridView (Close Figure 5)
public class CustomImageColumn : DataGridViewImageColumn
{
    public CustomImageColumn()
    {
        this.CellTemplate = new CustomImageCell();
        this.ValueType = typeof(int);
    }
}

public class CustomImageCell : DataGridViewImageCell
{
    static Image [] dotImages = ... // load up custom dot images

    public CustomImageCell()
    {
        this.ValueType = typeof(int);
    }

    protected override object GetFormattedValue(
        object value, int rowIndex, ref DataGridViewCellStyle cellStyle,
        TypeConverter valueTypeConverter,
        TypeConverter formattedValueTypeConverter,
        DataGridViewDataErrorContexts context)
    {
        return dotImages[(int)value];
    }

    public override object DefaultNewRowValue
    {
        get { return 0; }
    }
}

Thanks to the following Microsoft developers for their technical expertise: Harshit Agarwal, Abhinaba Basu, Scott Berry, Mark Boulter, Regis Brid, Mark Feinholz, Pankaj Garg, Daniel Herling, Kavitha Jonnakuti, Appireddy Kikkuru, Mark Rideout, Chris Sells, Ketaanh Shah, Gagandeep Singh, Saritha Suryadevara, Sunil Swami, and Vijaya Bhaskar Yerrapureddy.


CLR Inside Out: Base Class Library Performance Tips and Tricks

CLR Inside Out
Base Class Library Performance Tips and Tricks
Kit George

Contents
The Base Class Library
String Concatenation
Parsing Base Types
Collections
Reading from Files
General I/O Tips
Regular Expression Compilation
Local vs. UTC DateTimes
String Comparisons
PerformanceCounter Lifetime
Resource Lookup
Lock-Free Tracing
Conclusion

The common language runtime (CLR) sits at the very heart of managed code. Indeed, it is the heart of managed code, so to understand managed code you need to understand the CLR.

The CLR team continues to develop technologies that improve the Microsoft® .NET Framework and help you make full use of the managed environment. Over the next year, the CLR team will, in this column, provide insights into the core of managed code, share best practices, explain some of the more obscure functionalities, and present personal tips and suggestions. Everything from the String class to just-in-time (JIT) compilation is up for discussion, with each column focusing on a specific functional area of the CLR. Expect the odd foray into more general topics, such as .NET Framework design guidelines, as well.

The Base Class Library

The base class library (BCL) contains such fundamental types that everyone expects them to work under all conditions. Of course, this simply can't be the case. Like any other part of the Framework, BCL types are appropriate in some situations and not in others. This month I'm going to tackle performance tips and tricks for BCL types. While it would be nice to give you some absolute rules, such rules rarely work well for all scenarios. Plus, even if one API has a small performance benefit over another, it often comes at the cost of something else, like code simplicity. Where one style or pattern gives you more control, allowing you to tweak your performance, the other more general pattern requires less code but sacrifices performance. Thus, you shouldn't underestimate the impact of tweaking your use of BCL types. Using them properly can really affect your application speed.

With the introduction of the .NET Framework 2.0 comes generics and a wealth of other new features, including more base types exposing TryParse. If you want to maximize performance, it's a good idea to become familiar with the various patterns and best practices.

String Concatenation

It's well known that StringBuilders are an efficient way to concatenate strings. However, they are not always the best option. I've seen code use StringBuilders for every concatenation operation, and this is certainly not a wise choice.

StringBuilders are most efficient when you're concatenating a lot of strings, when the number of strings is unknown, and when the strings are long. However, if you're only concatenating a few strings or they're relatively short, standard concatenation is often a better choice. For this you can use either your language's chosen concatenation construct or String.Concat; these approaches will provide comparable performance. StringBuilders have associated overhead that standard strings don't have. Plus, at the end of the day, you have to turn the StringBuilder back into a string, which always incurs a cost.

Figure 1 Processing Time with 25-Character Strings
Figure 1 Processing Time with 25-Character Strings

So, just as with any performance optimization, you should test to see if the changes you're making are worth it. Consider a scenario in which you're appending a variable number of strings, each of which is 25 characters long. The graph in Figure 1 shows the results of a simple test I ran to determine how long it takes to append the specified number of 25-character strings to a StringBuilder and to a string. You'll notice that for this scenario, StringBuilder is clearly better after seven concatenations.

Parsing Base Types

The .NET Framework 2.0 introduces a pattern across all the relevant base types called TryParse, which mimics Parse. The difference in behavior is that Parse will throw an exception when the data check fails, while TryParse simply goes to an else statement.

The code patterns for Parse and TryParse are very similar (see Figure 2). The key thing to note is that if and when the someString value is successfully parsed, the performance is exactly the same for both patterns. Therefore, you're not paying any kind of penalty for using Parse when you have a string you know will be of the type you're parsing into.

Figure 2 Parse vs. TryParse (Close Figure 2)

Parse
Try
    Dim i As Integer = Int32.Parse(someString)
    ' code which deals with the valid integer
Catch fe As FormatException
    ' code which deals with an invalid input
End Try
TryParse
Dim iVal As Integer
If Int32.TryParse(someString, iVal) Then
    ' code which deals with the valid integer
Else
    ' code which deals with an invalid input
End If

Performance takes a hit, however, when you strike a single string that cannot successfully be parsed. In the case of Parse, you'll notice it goes to an exception block. Exceptions are notoriously slow, but this is generally okay because, after all, you're meant to be in an exceptional state (which shouldn't happen frequently)! The point is that for this particular test, it's not really an exceptional state: you're simply checking if this data is like you want it to be. You end up paying a lot in performance for that exceptional case. In contrast, TryParse deals with this scenario through the simple use of an if-else block. There is no performance hit if it has to go to the else block.

TryParse will always perform equal to or better than Parse, and sometimes the performance difference can be dramatic. At its extreme (if every string being parsed cannot be turned into the underlying type making the attempt), Parse can take 150 times longer than TryParse. For example, the first time an exception is caught there's a huge performance hit. After that, cached information speeds things up, but it's still a lot slower. At the other end, if you're parsing 100 strings and only one of those strings won't successfully be parsed, then TryParse is only about 30 percent faster. Only! You may as well use TryParse from now on.

There's a good basic lesson here: make sure you don't use exceptions as a mechanism to detect logic flow. It might seem like a failed parse is an exceptional condition, but it's generally not. After all, you typically parse to find out if a string represents data of a specific type.

Of course, while TryParse is preferable when you need to determine whether a string is another data type, there are still situations where you should use Parse instead. One such occasion is when the parse is not expected to fail, and thus a failed parse is in fact an exceptional condition, and performance is not an issue. The other important case is when you need to distinguish between the reasons for a failed parse, such as an overflow versus a format exception.

In both of these examples, Parse is very appropriate. The FormatException that will result from an invalid Parse indicates that the failed Parse was unexpected and therefore by definition is an exceptional case.

As a historic note for those of you who have observed that Double.TryParse existed in previous versions of .NET, the reason is that the Visual Basic IsNumeric function was using Double.Parse and the Visual Basic team noted that the performance in the failure case was terrible.

Collections

Collections are a fundamental aspect to programming applications in .NET. It's critical to know which collection is best for your scenario. With the introduction of generics, you not only have to decide between conceptually different collections, but you now have two different patterns to choose from: generic and non-generic. It's a hint about what we on the BCL team consider to be the future of collections: there will be little ongoing work and additions in the non-generics collection space, although feedback is always considered.

The good news is that generics are on par with or better than their non-generic counterparts. Take the numbers in Figure 3 as a rough guide. Each of the specified operations was run many times on an ArrayList of Int32, on a List<Int32>, on an ArrayList of String objects, and on a List<String>. The critical differences compared here are a value type (Int32) and a reference type (string) in ArrayList and List<T>, for a variety of operations. The numbers show the relative performance of the generic operation compared to the non-generic operation (a value of 100 would mean they performed identically, a value of 200 would mean the generic version fared twice as well, and a value of 50 would mean the non-generic version was twice as fast as the generic version).

Figure 3 Comparing Generic and Non-Generic Collections (Close Figure 3)

Method | Relative Performance of List<Int32> Compared to ArrayList (percent) | Relative Performance of List<String> Compared to ArrayList (percent)
Add | 241.84 | 106.05
Contains | 267.25 | 89.21
ctor(ICollection) | 241.13 | 191.99
Enumeration | 226.19 | 205.21
Indexer | 106.33 | 109.77
IndexOf | 669.42 | 108.52
Insert | 107.89 | 99.71
Remove | 765.85 | 102.20
Sort | 1579.49 | 85.51

You can see that List<T> is always faster for value types. For reference types it is sometimes slower, but never by much. The gains far outweigh the losses, so it's pretty easy to say "use generics!" Of course, if you're performing intensive operations involving sorts, then you might want to try out both options to see which does better with your code.

SortedDictionary, a new generic counterpart to the SortedList collection, uses a tree-based implementation, so insertions, removals, and lookups should all be O(log N). In contrast, SortedList uses a sorted array-based implementation, so insertions and removals are O(N) and lookups should be O(log N). Generally then, SortedDictionary is the better choice, but it does have some drawbacks. Since each item in the tree is wrapped in a Node, and since a Node is a reference type, using a SortedDictionary will create a lot of objects on the garbage-collected heap, which may negatively impact the performance of the garbage collector (GC). Also, since the items in SortedDictionary are linked together through references and are not guaranteed to be near each other in memory, there is a greater chance that one of the items will not be found in the processor cache causing extra trips to memory and possibly extra page faults. The result of this will be performance that is worse than that of SortedList.

Reading from Files

The .NET Framework 2.0 introduces new APIs on the File class which enable simple reading and writing of an entire file. Rather than having to write your own logic to consume the contents of a file, File.ReadAllLines (and ReadAllBytes and ReadAllText) does this for you in one easy call. Performance-wise, if you actually want to retrieve and hold onto all of that data, these operations are great. For example, if you simply want to open a file and dump its contents into a Textbox (say your recreating notepad), ReadAllText is magic. Let's take a look at the code in Figure 4 to make sure we're on the same page. It shows how you'd use ReadToEnd with a StreamReader to get the data from a file versus how you'd use File.ReadAllText. For ReadAllText, I don't have to wrap the call in a try block, since the important task of closing the file is performed for me. These patterns are comparable from a performance perspective, although clearly ReadAllText is a little simpler. Note that there are a variety of ways to write the ReadLine code, and this is but one option. In this particular case, ReadAllText actually outperforms the code that reads a line at a time.

Figure 4 StreamReader vs. File.ReadAllText (Close Figure 4)

Getting the data from a file using StreamReader
StringBuilder sb = new StringBuilder();
using (StreamReader sr = new StreamReader(filename))
{
    while (sr.Peek() >= 0)
    {
        sb.AppendLine(sr.ReadLine());
    }
}
textBox1.Text = sb.ToString();
Getting the data from a file using ReadAllText
textBox1.Text = File.ReadAllText(filename);

Of course, the new APIs on File (ReadAllLines, ReadAllText, ReadAllBytes) do read the entire file in one go and retain that information in memory. As you retrieve the data, if you need to perform some transformation logic or you're parsing the data into different types, then it may well be better to read a line at a time and release the underlying data as you go.

To demonstrate this case, imagine you're doing something with each line or segment of data as it comes in (see Figure 5). The Person class you're using has a Parse method, which creates a Person object from the line of data. For File.ReadAllLines, the data from the file is sitting around in memory the whole time, but this isn't the case for StreamReader.ReadLine. So for this scenario, ReadLine is an option to consider. Note that it would take a pretty sizable file to see the kind of difference that's going to make you prefer the ReadLine alternative. You'll notice as well that in this case, the coding pattern for File.ReadAllLines isn't much less complex than the coding pattern for StreamReader.ReadLine. Given this, either alternative seems fair.

Figure 5 StreamReader vs. File.ReadAllLines (Close Figure 5)

Getting and parsing the data from a file using StreamReader.ReadLine
Dim lp As New List(Of Person)
Using sr As New StreamReader(filename)
    Do While sr.Peek() >= 0
        Dim p as Person
        If Person.TryParse(sr.ReadLine(), p) Then lp.Add(p)
    Loop
End Using
Return lp
Getting and parsing the data from a file using File.ReadAllLines
Dim lp As New List(Of Person)
For Each nextPerson As String In File.ReadAllLines(filename)
    Dim p as Person
    If Person.TryParse(nextPerson, p) Then lp.Add(p)
Next
Return lp

General I/O Tips

Here's a handful of things to remember when doing I/O operations. FileStream has an internal buffer which defaults to 4KB. When you're writing data to a file, if you're attempting to write more than 4KB, you actually bypass that buffer automatically—good news from a performance perspective. But now, imagine that you're trying to tweak performance of your application. You increase the size of FileStream's buffer to 32KB, so that a write isn't performed until at least that much data is available. The NTFS file system also has an internal buffer, and its size is 64KB. The 32KB of data you write from FileStream will be captured in that buffer. To bypass the NTFS buffer and go directly to the I/O device that you're writing to, increase the buffer of FileStream to above 64KB. This can result in a performance boost and provide good control over I/O operations.

If you have long-running applications that write a file and expect it to be sequential on disk, you may be surprised. When you first create the file to write to, a certain amount of space is reserved. If another file is created in the meantime (a likely scenario), then when you later add data to your original file it may be reserved in another location, resulting in disk fragmentation of your file and poorer performance than you expect. If you find yourself in this situation (writing data to a file in a long-running application) remember to reserve plenty of space in the file when you first create it. You can do this by using FileStream.SetLength.

When using the new compression APIs available in version 2.0 of the .NET Framework (the System.IO.Compression namespace), it's fundamentally important to consider buffering your data when writing out your file. When you write to the compressed stream, the algorithm for compressing the data will perform its operations based on the bits it can currently see. So if you're passing only a small segment of bits, the savings will be small or even negative. Wrapping the compressed stream in a buffered stream, so that the writes are performed in larger chunks, gives the compression algorithm more of an opportunity to spot good, positive savings for you.

Regular Expression Compilation

In the Regular Expression (Regex) space, there is the option to specify that a regular expression is compiled through use of the RegexOptions.Compiled setting. This switch hints at an underlying aspect of regular expressions in .NET: they have three modes. Let's look at each of those modes and discuss the relative performance trade-offs of each.

By default, regular expressions in .NET are interpreted. That is, you create a Regex with no compiled switch. Interpreted Regexes have the smallest impact to the startup performance of an application, but they have the slowest run-time speed since all of the processing remains to be conducted at run time. They are a good option if your Regex will be used rarely, since a rarely used expression won't impact overall run-time speed much (by definition), and taking any kind of startup hit probably isn't worth it:

Regex r = new Regex("abc*");
Regex.Match("1234bar", @"(\d*)bar");

The second option is regular expressions that are compiled on the fly. This is simple to achieve by passing in RegexOptions.Compiled when performing your Regex operation on the Regex (or by passing it into the Regex constructor). In this scenario, the regular expression engine does initial work to parse the expression into opcodes. Then, the engine turns those opcodes into Microsoft intermediate language (MSIL) using Reflection.Emit. Basically, it front loads as much work as it can to the startup of your app, effectively trading run-time speed for startup speed. In practice, compilation takes about an order of magnitude longer to start up, but on average yields 30 percent better run-time performance. This is a good solution if you have a regular expression you're going to be performing often. The overall run-time savings will add up and become significant compared to the hit you have to take at startup time.

There is a caution here, however. Emitting MSIL with Reflection.Emit loads a lot of code and uses a lot of memory, and that's not memory you'll ever get back. Generated MSIL cannot be unloaded, or at least, it couldn't in the .NET Framework 1.x. The only way to unload code is to unload an entire application domain. This is a general rule, and is simply the sacrifice you have to make. The good news is that part of this problem is solved in the .NET Framework 2.0—regular expressions are compiled using Lightweight Code Generation, which allows the generated MSIL to be garbage collected. But even then, you may need to release more of the functionality faster. Therefore, the general approach to compiled Regexes should be to only use this mode for a finite set of expressions that you know will be used repeatedly. Even more specifically, avoid this mode except for one or two key Regexes, and for the rest use either interpreted or, the final option, precompiled. Here's how RegexOptions looks:

Regex r = new Regex("abc*", RegexOptions.Compiled);
Regex.Match("1234bar", @"(\d*)bar", RegexOptions.Compiled);

Regex precompilation solves many of the problems associated with compiling on the fly, and retains all of the performance benefits. Precompilation means you do all of the work of parsing and generating MSIL when you compile your app, ending up with a custom class derived from Regex. Run-time performance of this technique is identical to what you get when you compile on the fly.

There is, of course, a trade-off: the whole meaning of precompiled is that the Regex is put into its own assembly before you run your application. You therefore can't have the code for constructing the Regex inline, and you'll have to write other code (a tool perhaps) to construct your precompiled Regex ahead of time. You can do this with Regex.CompileToAssembly:

Dim rci As RegexCompilationInfo = New RegexCompilationInfo( _
    "abc*", RegexOptions.None, "standard", "Sample.Regex", True)
Regex.CompileToAssembly( _
    New RegexCompilationInfo() {rci}, New AssemblyName("foo.dll"))

Once in its own assembly, your calling code can reference this Regex directly. There's the startup hit of loading the assembly, but the performance benefits of not having to interpret or compile the expression at run time are preserved. This option has an additional benefit for performance. Your startup time reduces to loading and JITing your class, which should be comparable to the startup cost of interpreted mode, making this option the best of both worlds.

Overall, I strongly suggest you consider precompilation of Regexes, particularly for expressions you use a lot, and from multiple applications or assemblies.

Local vs. UTC DateTimes

One of the easiest things to note is that UtcNow (basically, culture-neutral time) performs better than Now. And not a little bit better, a lot better.

Of course, this piece of information isn't that helpful when you're showing data to users. After all, at the point that you want to show someone a DateTime, more often than not it has to be turned into some form of localized useful data. But if you're comparing two DateTimes to see if one event occurred before or after another, or you're performing other calculations where culture is not required, then UtcNow is the clear choice. Of course, this means that it's better to keep your DateTimes as UTC values until you need to retrieve them in some localized format (performance aside, it's a good idea to keep your DateTimes as UTC values, as this allows you to compare data from multiple time zones accurately and without being adversely affected by Daylight Savings Time).

Try out the following example code to see the performance results for yourself. Simply change the call to UtcNow to calls to Now to try the culture-aware version:

DateTime dt = DateTime.UtcNow;
Stopwatch sw = new Stopwatch();
for (int i = 0; i < 10; i++) {
    sw.Start();
    for (int j = 0; j < 100000; j++) {
        if (DateTime.UtcNow == dt) { /* do action */ }
    }
    Console.WriteLine(sw.ElapsedTicks);
}

String Comparisons

Strings in .NET are fairly powerful in that they can be compared in culture-sensitive ways, or with no regard for culture at all. By default, many, or even most, operations are culture sensitive. Unless you need to compare strings in a culture-sensitive fashion, then always use ordinal comparisons to compare strings. Of course, the same goes for sorting strings as well. As with UtcNow, the savings by sorting/comparing this way are dramatic. There are specific APIs on String which highlight the value of this approach (String.CompareOrdinal, for example). This kind of comparison is as first-class as its culture-aware counterpart.

When you want to keep case in mind (as many string comparisons do) but you still want the performance benefit, then StringComparer.OrdinalIgnoreCase is the way to go. This is also substantially faster than respecting culture in comparisons, while leaving you with the one concept that many sorts/comparisons typically want: the ability to be case-insensitive.

PerformanceCounter Lifetime

PerformanceCounters aren't free to instantiate—there is a cost overhead. Therefore, don't recreate PerformanceCounters if you can avoid it. Instead, instantiate an instance of the counter for your entire application and keep it around. A good way to achieve this, of course, is to have a static variable for each counter you want to represent and simply hold onto it in your application:

Shared ReadOnly myCounter1 As PerformanceCounter = _
    New PerformanceCounter("category", "counter1", "instance")
Shared ReadOnly myCounter2 As PerformanceCounter = _
    New PerformanceCounter("category", "counter2", "instance")

Resource Lookup

The NeutralResourcesLanguage attribute was added in the .NET Framework 2.0 solely for performance reasons, so it's a good choice for this column. When resource probing is performed, it's done in a very structured manner. If your culture is es-MX (Spanish in Mexico), for example, then resources are searched for in the order of es-MX, es, and finally, the fallback (default) resources in the main assembly.

It's expected that the fallback resources will be in the culture for which the app is most frequently distributed. Even for that culture the extra lookups are performed anyway. That is, if your fallback resources are the same as those used for en-US (English in US), then a search is still performed for en-US and en. The NeutralResourcesLanguageAttribute can help you bypass those checks. By applying it to an assembly (your Main assembly) you can tell the resources engine to avoid those lookups for the specified culture. Don't underestimate the cost of those lookups. They're good to avoid:

[C#]
[assembly: NeutralResourcesLanguageAttribute("en")]

[Visual Basic]
<Assembly: NeutralResourcesLanguageAttribute("en")>

Lock-Free Tracing

Lock-Free Tracing is a new feature in the .NET Framework 2.0. In previous versions of the Framework, two traces were not allowed to happen at the same time. All tracing was always locked internally, guaranteeing atomicity of the given operation, and that tracing would occur in the right order.

This is quite expensive, however, and impacts tracing performance considerably. In version 2.0, there's a new way to specify that you don't want to lock a particular listener—a new UseGlobalLock property on the Trace class. This property is set to true by default, reflecting the version 1.x behavior. But you can set it to false programmatically or in the config file:

<system.diagnostics>
    <trace useGlobalLock="false" />
</system.diagnostics>

By turning off global locking, you're giving each listener the opportunity to specify whether it should be locked. Note that if a listener says it isn't thread safe, global locking is used anyway. This is basically to ensure that global locking is only used if it's absolutely clear that it's what you intended to do.

In order to take advantage of this feature, make your own custom TraceListener (derived from the TraceListener class). Ensure that critical operations, such as WriteLine, are atomic and that you set IsThreadSafe to true. If you take these actions, the TraceListener code will take advantage of the UseGlobalLock setting, resulting in a faster TraceListener.

Conclusion

In this month's installment, I've discussed myriad ways in which you can improve the performance of your application. The broad advice always remains: pay attention to the techniques you're using, and ensure you're considering the optimal performance for that scenario. If you do trade performance for functionality, at least do it consciously, not by accident. You can continually improve performance using new features or by using APIs the way they're designed to be used. So keep an eye out for ways to maximize performance when using the BCL.

Send your questions and comments to clrinout@microsoft.com.

Kit George is a Program Manager for the base classes within the Common Language Runtime team. David Gutierrez, Anthony Moore, Ravi Krishnaswamy, Brian Grunkemeyer, Ryan Byington, and David Fetterman, also from the CLR team, contributed to this column.


Data Points: SqlConnectionStringBuilder, DataView, and More

Data Points
SqlConnectionStringBuilder, DataView, and More
John Papa

Code download available at: DataPoints0601.exe (130KB)
Contents
Binary Serialization
DataView Enhancements
Building Connections
What are Batch Updates?
Choose Carefully
Wrapping It Up

Several significant enhancements have been made to ADO.NET 2.0 in the areas of improved performance, increased flexibility, and added features. In my last column (Data Points: DataSet and DataTable in ADO.NET 2.0), I discussed the improvements made to the DataTable class, the effect of the powerful new LoadOption enumerator, how to change a row's state, and the new DataTableReader class. In this installment of Data Points, I will continue to explore significant enhancements that are introduced with ADO.NET 2.0, such as the new SqlConnectionStringBuilder, improvements to the DataView, batch updates, and improved DataSet serialization. I will also demonstrate how some of its new features perform when they are put to the test.

Binary Serialization

ADO.NET 1.x provides the means to serialize a DataSet. The bad news is that the XML format used for DataSets can cause performance problems when a large DataSet is serialized and passed over the network. In ADO.NET 2.0, you can serialize a DataSet or a DataTable in true binary format, resulting in smaller consumption of memory and network bandwidth.

Let's take a look at how to implement binary serialization of a DataTable. The RemotingFormat property of the DataTable is a new property introduced in ADO.NET 2.0 that can be used to set the serialization format. The RemotingFormat property expects one of the SerializationFormat enumeration values: SerializationFormat.Xml or SerializationFormat.Binary. Once RemotingFormat has been set, you can create an instance of the BinaryFormatter class to serialize the DataTable. The example in Figure 1 loads a list of customers into a DataTable and serializes it into the binary format.

Figure 1 Binary Serialization of a DataTable (Close Figure 1)
string cnStr = @"server=.;database=northwind;integrated security=true;";
using (SqlConnection cn = new SqlConnection(cnStr))
{
    string sql = "SELECT o.OrderID, o.CustomerID, o.OrderDate, " +
                 "od.ProductID, p.ProductName, od.UnitPrice, " +
                 "od.Quantity FROM Orders o " +
                 "INNER JOIN [Order Details] od " +
                 "ON o.OrderID = od.OrderID INNER JOIN Products p ON " +
                 " p.ProductID = od.ProductID ";
    SqlCommand cmd = new SqlCommand(sql, cn);
    SqlDataAdapter adpt = new SqlDataAdapter(cmd);
    DataTable dtCustomers = new DataTable("Customers");
    adpt.Fill(dtCustomers);

    BinaryFormatter bin = new BinaryFormatter();
    using (StreamWriter sw = new StreamWriter(@"c:\customers.bin"))
    {
        dtCustomers.RemotingFormat = SerializationFormat.Binary;
        bin.Serialize(sw.BaseStream, dtCustomers);
    }
}

A DataTable that is serialized in binary format is generally smaller in size than the same DataTable serialized in XML format. But how much smaller?

Being the skeptic that I am, I tried a few tests to see how different the size of a DataSet would be when serialized in XML versus binary format. I tried several different sample sets using several different powers of 10. For my tests, I created a table called MyOrders and I added over 100,000 rows to it. Then I ran the code in Figure 2, which produced the results shown in Figure 3. (Your results may vary depending on your data.)

Figure 3 XML vs. Binary Serialization Payloads (Close Figure 3)

Rows | Binary KB | XML KB | Binary KB/ Row | XML KB/Row
1 | 9.1 | 2.15 | 9.1 | 2.15
10 | 9.67 | 5.38 | 0.967 | 0.538
25 | 10.5 | 10.6 | 0.42 | 0.424
100 | 14.9 | 37.3 | 0.149 | 0.373
1,000 | 68.2 | 389 | 0.0682 | 0.389
10,000 | 602 | 3596 | 0.0602 | 0.3596
100,000 | 5934 | 36137 | 0.05934 | 0.36137
Figure 2 Serializing a DataTable (Close Figure 2)
string cnStr = @"server=.;database=northwind;integrated security=true;";
for (int power = 0; power <= 5; power++)
{
    using (SqlConnection cn = new SqlConnection(cnStr))
    {
        double top = (Math.Pow(10, power));
        string sql = "SELECT TOP " + top +
                     " OrderID, CustomerID, OrderDate, " +
                     "ProductID, ProductName, UnitPrice, Quantity" +
                     "FROM MyOrders ";
        SqlCommand cmd = new SqlCommand(sql, cn);
        SqlDataAdapter adpt = new SqlDataAdapter(cmd);
        DataTable dtMyOrders = new DataTable("MyOrders");
        adpt.Fill(dtMyOrders);

        BinaryFormatter bin = new BinaryFormatter();
        using (StreamWriter sw = new StreamWriter(
            @"c:\myorders" + top + ".bin"))
        {
            dtMyOrders.RemotingFormat = SerializationFormat.Binary;
            bin.Serialize(sw.BaseStream, dtMyOrders);
        }
        using (StreamWriter sw = new StreamWriter(
            @"c:\myorders" + top + ".xml"))
        {
            dtMyOrders.RemotingFormat = SerializationFormat.Xml;
            bin.Serialize(sw.BaseStream, dtMyOrders);
        }
    }
}

One thing that stands out is that the tests I ran for 1 and for 10 rows shows that the binary files are actually larger than the XML files. To figure out the number of rows where the crossover occurred, I ran my test an additional 100 times for rows 1 through 100. For my data sampling, the binary file was smaller than the XML file whenever there were 25 or more rows, so I added the results for 25 rows to the grid in Figure 3. The bottom line here is that the binary format can significantly decrease the amount of network bandwidth and time that is needed to pass a DataTable or a DataSet across a network.

Taking this one step further, I added two additional columns to the table in Figure 3 to represent the amount of memory consumed per row. Looking at the XML serialization data and starting at 100 rows and greater, notice how the amount of space required per row doesn't vary all that much (0.375 versus 0.389). However, the memory required for each row by the binary serialization method decreases significantly as the number of rows increase (0.149KB at 100 rows all the way down to 0.059KB at 100,000 rows).

It is important to keep in mind that it is still good practice to pass the fewest rows necessary in your application. If you are passing data to a data layer, you should only pass the changed data (inserts, updates, and deletes) using the GetChanges method of the DataTable, since there is rarely a need to pass down data that did not change. However, if your application requires that you pass an entire stateful set of data between tiers or possibly to another business system, binary serialization can come in handy.

DataView Enhancements

A DataView is a great tool for filtering, sorting, and in general just presenting data to the user in a more user-friendly format. In ADO.NET 2.0, the DataView class provides several new features that truly enhance its value. Let's assume, for example, that in ADO.NET 1.x you have a DataTable filled with a list of customers that came from a database. The DataTable is being used to store the list of customers and then a DataView is used on top of the DataTable to present the user with the list of customers in a sorted grid. In this example, the DataView is being used as a filter to present data. Going a step further, you might want to grab a distinct list of cities from the customer data to load another grid or a dropdown list. But if you also wanted to persist the DataView of customers or the DataView of the cities, you could not do this directly from the DataView. That is no longer the case in ADO.NET 2.0.

Through the DataView.ToTable method, the DataView now allows you to create a DataTable directly from the DataView. The new DataTable is a separate and distinct object that is not linked to the original DataTable. For example, let's assume I have a DataTable called dtCust and I create a DataView called view that displays a filtered and sorted list of the customers in the dtCust. If I then invoke the ToTable method on view to create a second DataTable called dtCities, I now have two DataTable objects that each store their own separate copies of their data. So if I change a customer in dtCust, it does not affect the data in dtCities whatsoever.

The code in Figure 4 creates a DataTable called dtCust and loads it with a list of customers from a database. It then creates a DataView called vwCust that only contains a distinct list of the cities of the USA customers. The DataView's ToTable method is then used to create a new DataTable called dtCities. This example uses the ToTable method's overloaded signature which accepts the name of the new DataTable to create, a Boolean value to indicate whether or not distinct values should be grabbed, and a string array containing the list of columns to grab from the DataView and add to the new DataTable.

Figure 4 Using ToTable To Create a DataTable from a DataView (Close Figure 4)
string cnStr = @"server=.;database=northwind;integrated security=true;";
string sql = "SELECT CustomerID, CompanyName, City, Region, Country " +
             "FROM Customers ORDER BY CustomerID";

DataTable dtCust = new DataTable();
using(SqlConnection cn = new SqlConnection(cnStr))
{
    SqlCommand cmd = new SqlCommand(sql, cn);
    SqlDataAdapter adpt = new SqlDataAdapter(cmd);
    adpt.Fill(dtCust);
}

DataView view = new DataView(dtCust);
view.RowFilter = "Country='USA'";
view.Sort = "City";
DataTable dtCities = view.ToTable("CustomerCities",
        true, "City", "Country");

Building Connections

ADO.NET 2.0 introduces a brand new class called SqlConnectionStringBuilder that can be used to help build a connection string. All you have to do is instantiate it and set the properties that are appropriate, and it builds the connection string for you (revealed through its ConnectionString property). The following code sample shows how the SqlConnectionStringBuilder works:

SqlConnectionStringBuilder cnBldr = new SqlConnectionStringBuilder();
cnBldr.DataSource = "MyServer";
cnBldr.InitialCatalog = "Northwind";
cnBldr.IntegratedSecurity = true;
using (SqlConnection cn = new SqlConnection(cnBldr.ConnectionString))
{
    ...
}

In this example, I set the server name via the DataSource property, the database name via the InitialCatalog property, and I tell it to use integrated security. IntelliSense® makes this approach much easier than attempting to hand code a connection string, and the strongly typed nature of the class helps eliminate harmful typos. There is no longer a need to try to remember the name of the attribute for the connection string. The Microsoft® .NET Framework also comes with the System.Data.OleDb.OleDbConnectionStringBuilder, System.Data.Odbc.OdbcConnectionStringBuilder, and System.Data.OracleClient.OracleConnectionStringBuilder classes. All of these are strongly typed classes that inherit from the DBConnectionStringBuilder class. Of course, if your database provider does not offer a strongly typed connection builder class, you could create your own class to build connections by inheriting from the DBConnectionStringBuilder class.

What are Batch Updates?

One of the best new features of ADO.NET is batch updating. Batch updating is easy to implement and reduces the number of network round-trips between your data layer and a database. To demonstrate the value of batch updates, I'll walk through a common update scenario that uses ADO.NET 1.x without batch updates and then discuss one that uses ADO.NET 2.0 with the new batch updating functionality.

First, I'll create a DataTable and load it with 100 rows from a database and present it to a user via a grid. I'll then add 12 rows, modify 9 rows, and delete 11 rows from the DataTable. At this point I have a DataTable with 70 unchanged rows, 9 modified rows, 11 deleted rows, and 12 added rows. I then use the GetChanges method to grab a new DataTable containing only the deltas (the modified, added, and deleted rows). I only want the deltas because I see no reason to pass all of the rows to my data layer throughout the layers of my application. In the data layer, I then create a DataAdapter and assign to the InsertCommand, UpdateCommand, and DeleteCommand properties the appropriate SQL stored procedures. When the DataAdapter's Update method is executed against the delta DataTable, the UpdateCommand is executed for each of the modified rows found using that row's data. Likewise, when it sees a deleted row or an added row, the DeleteCommand and InsertCommand are used, respectively. The DataAdapter.Update method executes 30 stored procedure calls (1 for each dirty row) and makes 30 separate round-trips between the data layer and the database.

In this example, using ADO.NET 1.1 techniques, each execution of a stored procedure packages the stored procedure call and sends it to the database, waits for any returning information, and then executes the next stored procedure in succession. If my data layer and my database are on separate physical servers, I have just introduced 30 round-trips across the network. If I had 100 changed rows, there would be 100 round-trips. You get the idea.

This same example with ADO.NET 2.0 reduces the network round-trips by using batch updates. Just prior to executing the DataAdapter.Update method, I set the UpdateBatchSize property of the DataAdapter to 30. This tells the DataAdapter to send 30 command executions at a time to the database. This time when the DataAdapter.Update method executes and iterates through the delta rows, it prepares the stored procedures to be executed until it has 30 of them prepared. Then it sends the 30 commands to the database in a single RPC request. In this example, all 30 stored procedure calls are sent across the network and then executed individually on the database server. Thus, only one network round-trip is required which streamlines this scenario.

For demonstration purposes, I built a consolidated example of a single batch of code (shown in Figure 5) that emulates the scenario I just described. In a real-world scenario, I would not recommend putting all of this code into a single method, of course. In this example, I retrieve some customers, modify one of them, and add a new customer to the DataTable. Then I set up the DataAdapter and its commands using SQL statements. At this point I set the UpdateBatchSize to 10, which in effect tells the DataAdapter to pool its commands and send them in groups of 10 (or less in case there are less than 10 operations to be performed).

Figure 5 Implementing Batch Updates (Close Figure 5)
string cnStr = @"server=.;database=northwind;integrated security=true;";
string sql = "SELECT CustomerID, CompanyName, City, Region, Country " +
             "FROM Customers ORDER BY CustomerID";
using (SqlConnection cn = new SqlConnection(cnStr))
{
    SqlCommand selCmd = new SqlCommand(sql, cn);
    SqlDataAdapter adpt = new SqlDataAdapter(selCmd);
    DataTable dtCustomers = new DataTable("Customers");
    adpt.Fill(dtCustomers);
    dtCustomers.PrimaryKey =
        new DataColumn[] { dtCustomers.Columns["CustomerID"] };

    // Add a customer row
    DataRow newRow = dtCustomers.NewRow();
    newRow["CustomerID"] = "FOO";
    newRow["CompanyName"] = "The Foo Company";
    newRow["City"] = "Fooville";
    dtCustomers.Rows.Add(newRow);

    // Change a customer row
    DataRow row = dtCustomers.Rows.Find("ALFKI");
    row["City"] = "Here"; /// was Berlin

    // Set up the Insert Command
    string insSql = "INSERT Customers (CustomerID, CompanyName, City) " +
                    "VALUES (@CustomerID, @CompanyName, @City)";
    SqlCommand insCmd = new SqlCommand(insSql, cn);
    insCmd.UpdatedRowSource = UpdateRowSource.None;
    insCmd.Parameters.Add("@CustomerID", SqlDbType.NChar,
        10, "CustomerID");
    insCmd.Parameters.Add("@CompanyName", SqlDbType.NChar,
        80, "CompanyName");
    insCmd.Parameters.Add("@City", SqlDbType.NChar, 30, "City");
    adpt.InsertCommand = insCmd;

    // Set up the Update Command
    string updSql = "UPDATE Customers SET CompanyName = @CompanyName, " +
                    "City = @City WHERE CustomerID = @CustomerID";
    SqlCommand updCmd = new SqlCommand(updSql, cn);
    updCmd.UpdatedRowSource = UpdateRowSource.None;
    updCmd.Parameters.Add("@CompanyName", SqlDbType.NChar,
        80, "CompanyName");
    updCmd.Parameters.Add("@City", SqlDbType.NChar, 30, "City");
    updCmd.Parameters.Add("@CustomerID", SqlDbType.NChar,
        10, "CustomerID");
    adpt.UpdateCommand = updCmd;

    // Set the Batch Size. 0 means all; default is 1.
    adpt.UpdateBatchSize = 10;

    // Send the new rows and the changed rows to the database
    adpt.Update(dtCustomers.GetChanges());
}

So the idea is to group multiple commands and package them for transmission across the network to the database server. By default the UpdateBatchSize is set to 1, which means it acts just like it did in ADO.NET 1.1, where each delta row causes a single command to be sent to the database on its own, one at a time. When using an explicit value, such as 10, the DataAdapter packages 10 commands and sends them in a single RPC call to the database server. If there are 15 modified, added, and deleted rows in total and if the UpdateBatchSize is set to 10, there will be 2 separate packages transmitted: the first with 10 rows and the second with the remaining 5 rows. If the UpdateBatchSize is set to 0, the DataAdapter will attempt to send all of the command executions in a single package. If batch updates are not supported, then the UpdateBatchSize is automatically reverted back to 1.

Batch updates are implemented by packaging everything into a single RPC request. So if there are six commands to execute against the database contained in the batch, they will all be sent in a single request. The RPC request will contain each SQL command separated by an RPC marker. This is important because if you watch SQL Profiler after running the example in Figure 5 you will see two commands executing against the database individually. This technique is also better than some other techniques that limit the number of parameters or the SQL command length. The first time I tried batch updates, I thought everyone was nuts because I watched SQL Profiler and saw all of the separate SQL commands being executed. It wasn't until I used a network sniffer that I could clearly see that only one round-trip was taking place.

Choose Carefully

There is a time and place for everything, and batch updates are no exception. Batch updates do not support the use of return or output parameters, so you will want to avoid using batch updates if you need return or output parameters. Batch updates also affect the RowUpdated event because they will cause the RowUpdated event to fire only once for each batch even though so many updates were made. Figure 6 shows some differences in the number of times the RowUpdated event handler would execute based upon the UpdateBatchSize when 25 rows are updated.

Figure 6 RowUpdated Event Execution on 25 Rows (Close Figure 6)

UpdateBatchSize | Number of Times RowUpdated Fires | Comments
0 | 1 | Once for the entire batch
1 | 25 | Once per row
4 | 7 | Once for every 4 rows: Ceiling(25/4)
10 | 3 | Once for every 10 rows: Ceiling(25/10)
25 | 1 | Once for the entire batch
30 | 1 | Once for the entire batch

When using batch updates, keep in mind that the Row property that is exposed in the RowUpdated event handler will represent the last row in the batch. If the count of actual rows in the batch is smaller than the UpdateBatchSize, the Rows property will return null.

Wrapping It Up

The release of ADO.NET 2.0 brings several new enhancements to its core classes along with new classes, some of which bring better performance and others which yield greater functionality and flexibility. With all the changes you'll find in ADO.NET 2.0, it's a good idea to try the features out a little at a time so you can refine your database toolbox and be ready to get the best performance at the best price for your data-driven apps. Now go out and reap the benefits of ADO.NET 2.0.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a Senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences, such as VSLive, or blogging at codebetter.com/blogs/john.papa.


Advanced Basics: The Sound of Music

Advanced Basics
The Sound of Music
Brad McCabe

So you've got a burning desire to accompany your application's splash screen with a rousing chorus of "Funky Cold Medina" (that's Tone Loc for those of you who aren't children of the 80s). Prior to working with Visual Studio® 2005, adding even simple tunes and system sounds to your application could be a challenge. There are numerous new classes and namespaces that have been added to the Microsoft® .NET Framework 2.0 to help you do just that. I'm going to take a look at one of them, the System.Media namespace, in this month's column.

Let's start by bringing this application into the 1960s with a simple beep, reminiscent of Pong. To do this, simply use the SystemSounds class, which has five public properties: Asterisk, Beep, Exclamation, Hand, and Question. Each of these properties returns an object of type SystemSound, which in turn exposes a Play method. So to make the computer beep, use System.Media.SystemSounds.Beep.Play. Pretty impressive, huh?

What's nice about the SystemSounds class is that if the user has modified his system sounds or applied any themes, they will be reflected in your application. Now, with one line of code before displaying a message box, you can play a sound to match the urgency of the message, such as the system exclamation or question sound, to get a user's attention based on the content of your message:

System.Media.SystemSounds.Exclamation.Play()
MessageBox.Show("Visual Basic!", "Advanced Basics", _
    MessageBoxButtons.OK, MessageBoxIcon.Exclamation)

Playing system sounds is useful, but what if you want to further customize your application with more complex sounds? In System.Media, you'll find a SoundPlayer class for controlling the playback of .wav files. The SoundPlayer supports playing sounds from a file, a URL, a stream, or an embedded WAV resource.

The first step to playing a file is to load it. You are required to explicitly load the file first if you are going to play it from a stream or a URL. If you are playing from an embedded resource or a file, the Play method will load the file for you if you have not already done so. As with so many other aspects of .NET, you have flexibility in your approach: you can load the files synchronously or asynchronously, depending on the needs of your application.

If you are loading the .wav file close to the time when you will play it back, and you cannot guarantee that the file will be loaded by the time you call the Play method, you should choose a synchronous approach. The downside to loading in this manner is that you will block any further execution of your application on the main thread and lock up the user interface during the load.

The asynchronous approach is effective if you are loading from a slower source, such as a URL, or if you have a large file. You should call the asynchronous load method at an early stage in your execution. Unlike its synchronous counterpart, the asynchronous approach allows application code to continue to execute while your file is loaded. You will be notified that the file has loaded with a LoadCompleted event in case you want to do any additional processing or take any other action. In addition, you can check the IsLoadCompleted property at any time to determine the status of the load. The drawback to this approach is that it takes a bit more code to capture the events, and it adds a slight extra bit of complexity to your code. To load a file asynchronously, your code would look similar to this:

Dim Player As New SoundPlayer
Player.SoundLocation = "C:\Program Files\Messenger\newemail.Wav"
Player.LoadAsync()

Now, you are ready to play your .wav file. Like with loading, you have two choices here: either synchronously or asynchronously. Notice a pattern? The same pros and cons apply. Synchronous playing requires the user to listen to your file completely before the application logic continues executing, and you don't have to worry about any other code or pesky user interaction. But once again your user is locked out and stuck until your file has finished playing. If your sound is only a second or two long this might not be an issue, but any longer and your user will swear that your application is too slow and will tell everyone how awful it is. (Isn't perception just grand?)

In fact, the SoundPlayer.Play method will play the file asynchronously. The designers of the Framework require you to go out of your way here to block the UI thread and lock up your application code:

Dim Player As New SoundPlayer
Player.SoundLocation = "C:\Program Files\Messenger\newemail.Wav"
Player.LoadAsync()
If Player.IsLoadCompleted Then Player.Play()

If you want to play your file synchronously, you will have to call SoundPlayer.PlaySync. Remember, if your file is large, your application's performance will be visibly affected.

What if you want to continue playing your .wav file until the user either hits a stop button or you programmatically end it? To do this, you can call the PlayLooping method and the Framework will asynchronously loop your file. Stopping any asynchronously playing file is as simple as calling the SoundPlayer.Stop method.

Now let's take a second to talk about the My namespace. It was designed as a "speed dial" into the .NET Framework and to give a boost to developer productivity. For example, almost everything discussed so far can be done in a line or two of code using My.Computer.Audio. To play a file, you can use My.Computer.Audio.Play("C:\Program Files\Messenger\newemail.Wav"). No need to worry about manually creating the SoundPlayer object or loading files, even for streams or URLs.

So why bother writing more code than that? Well, besides learning how to perform the functionality directly against the Framework (a useful skill to have if you ever need to code something in a language that does not easily benefit from My, like C#), you also need to work directly with the SoundPlayer class if you want to attach to and work with any of the three main events it will raise.

First, the LoadCompleted event that I briefly mentioned earlier will be fired at the end of an asynchronous file load. If you are loading your .wav file from a URL, this is an important event to capture, as you will want to check the event argument to see if the file was successfully loaded. In fact, even if you are loading asynchronously from a local file, it is a good idea to always code defensively and verify that the file is indeed loaded:

Private Sub Player_LoadCompleted(ByVal sender As Object, _
        ByVal e As ComponentModel.AsyncCompletedEventArgs) _
        Handles Player.LoadCompleted
    If e.Error IsNot Nothing Then
        ' Handle error
        isLoaded = False
    Else
        isLoaded = True
    End If
End Sub

Next, the SoundLocationChanged event is raised when a new audio source has been set for the SoundPlayer. When the location has been changed or updated, you might want to start loading the .wav file asynchronously, stop playing the previous file if one is looping, or perform any other action your application needs to take.

The third and final major event that you may need to interact with is the StreamChanged event. This event is similar in use to the SoundLocationChanged event and would be used if you were loading your .wav file from a MemoryStream or FileStream.

Making your computer beep, or playing .wav files, is hardly blasting out the tunes. If you want more than the ability to play .wav files, you'll need to reach outside the .NET Framework. One option is to make use of Windows Media® Player (you can download and install the player SDK). With the Windows® Media Player SDK installed and a reference to Windows Media Player libraries set up, you have access to most major file formats, including MP3 and WMA.

Just like the simple design of the System.Media namespace, playing files with Windows Media Player is very straightforward. Three lines of code and you are up and running:

Dim WMP As New WMPLib.WindowsMediaPlayer
WMP.URL = "C:\My Music\Funky Cold Medina.wma"
WMP.controls.play()

Now you have three ways to add sounds and media to your application. You can use the new System.Media namespace for a lightweight approach to system sounds and .wav files, you can use the Visual Basic® My namespace to have a more productive but slightly limited approach to accessing the System.Media namespace, or you can use Windows Media Player to support files like WMA and MP3. To simplify this, you can build an extension to the My namespace to combine these and pull together everything I have shown you (see Figure 1). An application that makes use of this is shown in Figure 2. See Joe Binder's article from the July 2005 edition of MSDN Magazine for more information on extending My (Visual Basic: Simplify Common Tasks by Customizing the My Namespace).

Figure 1 Extending My with Better Sound Support (Close Figure 1)
Namespace My
    Partial Class MyComputer
        Overloads ReadOnly Property Audio() As Audio
            Get
                Return New Audio
            End Get
        End Property
    End Class
End Namespace

Public Class Audio
    Inherits Microsoft.VisualBasic.Devices.Audio

    Private Shared fileType As String = ""
    Private Shared WMP As New WMPLib.WindowsMediaPlayer

    '  You need to overload Play to switch between System.Media and WMP.
    Public Overloads Sub Play(ByVal location As String)
        ' If anything is already playing then stop it
        If fileType <> "" Then [Stop]()

        ' Find out what type of file you want to play and play it.
        Select Case New IO.FileInfo(location).Extension.ToLower
            Case ".wma", ".mp3"
                WMP.URL = location
                WMP.controls.play()
                fileType = "WMP"

            Case ".wav"
                MyBase.Play(location)
                fileType = "WAV"

            Case Else
                Throw New Exception("Invalid File Type")
        End Select
    End Sub

    ' While you are overloading let's just combine system sounds
    ' into one big play function
    Public Overloads Sub Play( _
            ByVal systemSound As System.Media.SystemSound)
        My.Computer.Audio.PlaySystemSound(systemSound)
    End Sub

    '  You need to overload the Stop function to allow you to stop WMP
    Public Overloads Sub [Stop]()
        Select Case fileType
            Case "WMP"
                WMP.controls.stop()
            Case "WAV"
                MyBase.Stop()
            Case Else
                Throw New Exception("Unknown File Type Currently Playing")
        End Select

        fileType = ""
    End Sub
End Class
Figure 2 App with Extended My Capabilities
Figure 2 App with Extended My Capabilities

To summarize, adding sound to your application can help build a connection with the user and provide valuable non-visual feedback when used properly. Prior to the introduction of the System.Media namespace with the 2.0 release of the .NET Framework, adding the ability to quickly play system sounds and .wav files was not very straightforward and therefore was not utilized to its potential. Now developers using Visual Basic have the ability to choose either to use the productive "speed dial" of My.Computer.Audio or call the System.Media classes directly.

Now for the common sense portion of the column. Remember that while you can easily add sound to your applications, do so wisely and sparingly. While you may think the best UI is accompanied by a blaring Metallica riff, there will always be users who much prefer something just a bit more subtle—Green Day perhaps! A good test is "would my Mom find this useful?"

Send your questions and comments to basics@microsoft.com.

Brad McCabe is a Program Manager at Microsoft and, among other things, is responsible for the Visual Basic Developer Center on MSDN.


Cutting Edge: Custom Data Control Fields

Cutting Edge
Custom Data Control Fields
Dino Esposito

Code download available at: CuttingEdge0601.exe (131KB)
Contents
The DataControlField Class
Designing the CalendarField Component
Implementing CalendarField
Using the CalendarField Component
Designing a DropDownField Component
Styling Control Fields
Conclusion

In ASP.NET 2.0, the GridView and DetailsView controls are designed to work together. They don't merely provide complementary services, they also share a number of helper classes and components. The output of the GridView control consists of a sequence of rows, each with a fixed number of columns. Each table column maps to a data column in the bound data source. The DetailsView control has a fixed number of rows (one for each column in the bound data source) and a constant number of columns (header and value).

GridView and DetailsView can be used together to form master/details pages, as I demonstrated in my last column (see Cutting Edge: Flexible Custom Data Views). Both represent data columns in the bound data source. In fact, a GridView's column and a DetailsView's row are the same kind of object and are represented in ASP.NET 2.0 by the same class—DataControlField.

DataControlField is an abstract class and cannot be used for actual data binding. A bunch of child field classes are derived for binding data to GridView and DetailsView controls. In my last column I covered some of these field types. In particular, I showed how to use the TemplateField class to add validation and special editing user interface capabilities to bound fields. The TemplateField class works as a generic container of markup. It is ideal for rendering custom types of data and any combination of existing fields that you might need. The TemplateField type can also be used to render foreign-key columns through dropdown lists or dates through calendars.

But what if you wanted a specialized, more compact calendar field class—a CalendarField type that would render the bound data as a string in view mode and employ a more effective Calendar control in edit and insert mode? As I mentioned, you can achieve the same result using a TemplateField, but that would be like using late-bound versus early-bound objects. A data-bound page based on CalendarField will be easier to read and even a bit faster to process at run time.

I'll discuss the design and implementation of custom data field types and demonstrate how to actually code them using a few examples—CalendarField to render date values and DropDownField to render foreign key values defined on an external data source.

The DataControlField Class

The DataControlField class in ASP.NET 2.0 is a parallel for the DataGridColumn type used extensively with DataGrid controls in ASP.NET 1.x and is used only by the GridView and DetailsView controls. However, if you happen to write a custom data-bound control that requires binding to columns of data, you can use the DataControlField type to bind data columns to your control's user interface.

As I mentioned, DataControlField is an abstract class that inherits directly from System.Object. It implements the IStateManager interface. Figure 1 lists the methods and properties on the class.

Figure 1 Members of the DataControlField Class (Close Figure 1)

Property | Description
AccessibleHeaderText | Gets or sets text that is rendered as the AbbreviatedText property value in some controls for accessibility reasons.
ControlStyle | Gets the style properties for any input control that the field uses in edit or insert mode.
FooterStyle | Gets the style properties for the footer of the field. This is only applicable to the GridView control.
FooterText | Indicates the text displayed in the footer of the field. This is only applicable to the GridView control.
HeaderImageUrl | Gets or sets the URL reference to an image to display instead of text on the header of this field.
HeaderStyle | Gets the style properties for the header of this field.
InsertVisible | Indicates whether the field is displayed when the associated control is in Insert mode. This is only applicable to the DetailsView control.
ItemStyle | Gets the style properties for an item of this field.
ShowHeader | Indicates whether the field header is visible. This is only respected by the DetailsView control.
SortExpression | Indicates the expression used when this field is used to sort the bound data source. This is only applicable to the GridView control.
Visible | Indicates whether the field is visible.
Method | Description
ExtractValuesFromCell | Extracts the value from the control(s) in the field cell and adds it to the dictionary passed to this method, using a unique key name for this field.
Initialize | Initializes the field.
InitializeCell | Initializes a cell in the field.
ValidateSupportsCallback | Indicates that the controls that are contained by the field support callbacks.

As you can see, the class supports a few style properties—ItemStyle, HeaderStyle, FooterStyle, and ControlStyle. In ASP.NET, style properties are not persisted in the view state through the usual ViewState container, as most simple type properties are. The Style class from which all style properties derive implements IStateManager to take care of view state persistence. To utilize this specific capability, classes that expose style properties typically implement IStateManager themselves to delegate view state persistence to the methods defined on the objects that represent style properties. I'll return to this later.

Types derived from DataControlField are listed in Figure 2. Although the DataControlField class is marked abstract, it contains a lot of code and many concrete members. The only abstract method is CreateField, defined as follows:

Protected MustOverride Function CreateField() As DataControlField

Figure 2 Data Control Field Classes (Close Figure 2)

Class | Description
BoundField | Renders a simple string-based output for each element. It uses a TextBox control to capture input in edit/insert mode.
ButtonField | Renders a command button for each element. Not supported in edit/insert mode.
CheckBoxField | Renders a checkbox control for each element. A CheckBox control is used both in view and edit/insert mode.
CommandField | Renders command buttons to perform selecting, editing, inserting, or deleting operations.
HyperLinkField | Renders a hyperlink control for each element. Not supported in edit/insert mode.
ImageField | Renders an image for each element from the bound URL. The field is predisposed to display BLOB data containing images. In edit/insert mode, it uses a TextBox control to edit the image URL.
TemplateField | Renders some custom UI for each element. It allows you to define a template for the edit mode and one for the insert mode.

This method is expected to return an instance of the particular data control field object. Other methods should also be overridden in derived classes because their built-in implementation is empty (though not abstract). Those methods are ExtractValuesFromCell and InitializeCell. You must override them to create a significant new field type. Depending on what kind of new functionality you want to build into the field, overriding additional members might be required as well.

Designing the CalendarField Component

The CalendarField component inherits from DataControlField and adds three new properties—DataField, ReadOnly, and DataFormatString. DataField is a string property and indicates the name of the data source column that will be bound to the field. ReadOnly is a Boolean property to indicate whether the contents of the field are editable. To control whether the field is to be added in the insert mode UI, you can use the base InsertVisible Boolean property. Finally, DataFormatString indicates the desired format of the date. The contents of the DataFormatString property will be passed to the ToString method of the DateTime class, meaning that {0:d} and similar expressions are invalid. Here's an example of the CalendarField persistence format:

<msdn:CalendarField DataField="OrderDate"
      HeaderText="Date" ShowHeader="true"
      DataFormatString="dd MMM, yyyy" ReadOnly="False" />

The code that follows shows the implementation of the DataField property (keep in mind that the other custom properties follow the same implementation scheme):

Public Overridable Property DataField As String
      Get
         Dim o As Object = MyBase.ViewState("DataField")
         If (Not o Is Nothing) Then Return CType(o, String)
         Return String.Empty
      End Get
      Set(ByVal value As String)
         MyBase.ViewState("DataField") = value
         OnFieldChanged()
      End Set
End Property

The OnFieldChanged method is a protected virtual method defined on the parent class DataControlField and meant to signal to the host control (the GridView) that the value of a property on the field has changed. Each control field component knows about its host control through the Control property defined on DataControlField. The Control property is protected and set when the host control initializes the field component.

The Initialize method (see Figure 1) is called by the host control (GridView or DetailsView) to initialize the field. The initialization takes place before the field is added to the Fields collection of GridView and DetailsView. GridView and DetailsView call the Initialize method from within their CreateChildControls method. The Initialize method is given its basic behavior in the DataControlField base class and there's no real need for you to override it in a derived class:

Public Overridable Function Initialize( _
        sortingEnabled As Boolean, ctl As Control) As Boolean

Initialize receives a Boolean value that indicates if sorting is supported. It also receives a reference to the host control (GridView or DetailsView) to be stored in the internal Control property. The return value tells the host control if a call to DataBind is required to complete the initialization of the field type.

Another method on DataControlField that you might want to override in some cases is ValidateSupportsCallback. The default implementation of the method in DataControlField simply throws an exception that the field type doesn't intend to support script callbacks. In a derived class you can override this method to prevent the exception from being thrown in some or all cases. For example, you can supply an empty method implementation to say that a field supports callbacks or throws an exception based on the value of a field-specific property:

Public Overrides Sub ValidateSupportsCallback()
   If Not SupportsCallback Then
      Throw New NotSupportedException("Callbacks Not Supported")
   End If
End Sub

InitializeCell and ExtractValuesFromCell will need overrides in any significant control. InitializeCell lets you take control of the contents of cell where the field value is displayed. ExtractValuesFromCell extracts values from the cell to pursue updates or insert operations. Before taking the plunge into these methods and their overrides for the CalendarField component, let's have a look at Figure 3 where other protected overridable methods are listed.

Figure 3 Protected Overridable Members of DataControlField (Close Figure 3)

Overridable Method | Description
CopyProperties | Receives an object reference that represents a copy of the current field object and adds class-specific properties.
CreateField | Abstract method, returns an instance of the field type.
LoadViewState | Restores the class style properties from the view state.
OnFieldChanged | Fires the FieldChanged event defined on the DataControlField class.
SaveViewState | Saves the class style properties to the view state.
TrackViewState | Enables view state tracking for the style properties.

Implementing CalendarField

The CalendarField class provides the three aforementioned properties—DataField, ReadOnly, and DataFormatString—and overrides InitializeCell, ExtractValuesFromCell, CopyProperties, and CreateField.

CreateField follows a common scheme and gets the same form of implementation in all built-in field types:

Protected Overrides Function CreateField() As DataControlField
   Return New CalendarField()
End Function
Overriding this method is mandatory if you derive from DataControlField; it is optional, but recommended, if you derive from existing field types.

In InitializeCell you begin by calling the base method to import much of the logic that deals with header and footer rendering. Next, you add some code to customize the current cell, as shown here:

Sub InitializeCell(cell As DataControlFieldCell, _
        cellType As DataControlCellType, _
        rowState As DataControlRowState, rowIndex As Integer)

     ' Call the base method
     MyBase.InitializeCell(cell, cellType, rowState, rowIndex)

     ' Initialize the contents of the cell
     If cellType = DataControlCellType.DataCell Then
          InitializeDataCell(cell, rowState)
     End If
End Sub
The InitializeCell method receives four arguments: a reference to the cell object, the type of the cell, and the state and index of the row being rendered. Most of the existing field components delegate their rendering tasks to another internal method that is usually named InitializeDataCell. Figure 4 shows the implementation of InitializeDataCell for CalendarField.

Figure 4 Initializing the Cell for a CalendarField Component (Close Figure 4)
Protected Overridable Sub InitializeDataCell( _
        ByVal cell As DataControlFieldCell, _
        ByVal rowState As DataControlRowState)
    Dim ctrl As Control = Nothing

    ' If in edit/insert mode...
    Dim state As DataControlRowState = rowState & DataControlRowState.Edit
    If ((Not Me.ReadOnly And (state <> DataControlRowState.Normal)) Or _
        rowState = DataControlRowState.Insert) Then

        Dim cal As New Calendar
        cal.ToolTip = HeaderText
        cell.Controls.Add(cal)

        ' Save the control to use for binding (edit/insert mode)
        If DataField.Length > 0 Then ctrl = cal

        _inInsertMode = (rowState = DataControlRowState.Insert)

    ElseIf (DataField.Length > 0) Then
        ' Save the control to use for binding (view mode)
        ctrl = cell
    End If

    ' If the column is visible, trigger the binding process
    If Not ctrl Is Nothing And Visible Then
        AddHandler ctrl.DataBinding, New EventHandler(OnBindingField)
    End If
End Sub

Protected Overridable Sub OnBindingField( _
        ByVal sender As Object, ByVal e As EventArgs)
    Dim target As Control = CType(sender, Control)

    ' If in view mode ...
    If TypeOf (target) Is TableCell Then
        Dim t As TableCell = CType(target, TableCell)
        t.Text = LookupValueForView(target.NamingContainer)
    ElseIf TypeOf (target) Is Calendar Then
        Dim cal As Calendar = CType(target, Calendar)
        Dim dt As DateTime = LookupValueForEdit(target.NamingContainer)
        cal.SelectedDate = dt
        cal.VisibleDate = dt
    End If
End Sub

At its core, the InitializeDataCell method determines which control to bind to data—the cell for view mode or a Calendar control for edit or insert modes. The control is then attached to a DataBinding event handler to actually retrieve and display data. In the handler, you determine the working mode, retrieve the value to display, and configure the control as appropriate.

In view mode, the LookupValueForView helper method determines the value to display. It takes a reference to the naming container of the control (table cell or calendar) that currently forms the field's UI. The naming container is passed to the DataBinder.GetDataItem method to obtain a reference to the row object:

Dim dataItem As Object = DataBinder.GetDataItem(container)
Dim dt As DateTime = _
    CType(DataBinder.GetPropertyValue(dataItem, DataField), DateTime)

Next, DataBinder.GetPropertyValue extracts the requested field from the row object. For a CalendarField control, this value is a DateTime object and can be further formatted before display. In edit or insert mode, no significantly different behavior is required, except that you might want to distinguish between edit and insert and provide a default value in the latter case:

Protected Overridable Function LookupValueForEdit( _
        container As Control) As DateTime
    If Not _inInsertMode Then
        Dim dataItem As Object = DataBinder.GetDataItem(container)
        Dim value As Object = DataBinder.GetPropertyValue( _
            dataItem, DataField)
        Return CType(value, DateTime)
    End If
    Return DateTime.Now
End Function

GridView and DetailsView controls scan their own list of bound fields to collect input values when an update or insert command is selected. In doing so, they call the ExtractValuesFromCell method on the field types. Figure 5 details the implementation of ExtractValuesFromCell in the CalendarField component.

Figure 5 ExtractValuesFromCell Method (Close Figure 5)
Public Overrides Sub ExtractValuesFromCell( _
        ByVal dictionary As IOrderedDictionary, _
        ByVal cell As DataControlFieldCell, _
        ByVal rowState As DataControlRowState, _
        ByVal includeReadOnly As Boolean)

    Dim selectedValue As Object

    If cell.Controls.Count > 0 Then
        Dim cal As Calendar = CType(cell.Controls(0), Calendar)

        If cal Is Nothing Then
            Throw New InvalidOperationException( _
                "CalendarField could not extract control.")
        Else
            selectedValue = cal.SelectedDate
        End If
    End If

    ' Add the value to the dictionary
    If dictionary.Contains(DataField) Then
        dictionary(DataField) = selectedValue
    Else
        dictionary.Add(DataField, selectedValue)
    End If
End Sub

The ExtractValuesFromCell method is invoked when the field is in edit or insert mode. You locate the input control and capture any significant values it may contain. Based on the code shown in Figure 4, the CalendarField's input control is the first control in the Controls collection of the cell. (Note that the position of the input control is arbitrary and depends on the structure of the cell you create in InitializeCell.)

Once you hold the currently selected value of the input control, you add it to the dictionary object that was passed to the ExtractValuesFromCell method. The dictionary contains name/value pairs where the name entry is the name of the DataField property.

Using the CalendarField Component

A custom data control field must be registered with the page (or the application) as any other custom server control. To do this, use the @ Register directive like so:

<%@ Register Namespace="Samples.CustomFields"
    TagPrefix="expo" Assembly="HelperFields" %>
Add the following code to the <fields> element of a DetailsView or GridView:
<msdn:CalendarField DataField="OrderDate" HeaderText="Date" />
Turn the page in design mode and work through the Visual Studio® 2005 menus and windows to get what's shown in Figure 6.

The sample DetailsView has two custom data fields: CalendarField, used for the Date row, and a DropDownField, used for the posted-by row. I'll say more about the DropDownField control shortly. As you can see, the two control fields have a slightly different user interface. This is due to the DataBinding event handler. Basically, you check the DesignMode property and output some markup if the output is going to be shown in Visual Studio 2005:

If DesignMode Then
   Return "<select><option>Databound Date</option>           </select>"
End If

Figure 6 DetailsView
Figure 6 DetailsView

DesignMode is a protected property of the DataControlField class. It wraps the DesignMode property of the host control. The code you just saw returns a dropdown-like user interface and characterizes the custom control field. Although the custom data control fields have to be manually coded in the editor, you have full visual support as far as properties editing is concerned. Smart tags and the Properties dialog box show custom fields correctly.

Figure 7 shows a sample master/detail page in action. The DetailsView control uses the CalendarField component to display the date of a data-bound order.

Figure 7 Custom Data Fields in Action—Edit Mode
Figure 7 Custom Data Fields in Action—Edit Mode

Designing a DropDownField Component

The DropDownField data field is used with foreign-key fields, that is with data columns that contain indexes to data living in another table. Imagine a table of orders with columns to track the customer who made the order and the employee who physically placed the order. In the Orders table, you won't typically store the full name of employee and customer; rather, you'd store a pointer to another table. In the end, a row in the Orders table contains numbers instead of plain names. This might be great for database management systems, but it's not great for human consumers. When displaying data, you need to transform those numbers into readable names. This is only half the task that you might expect from a DropDownField component. When in edit or insert mode, in fact, the data field should be able to display a list of possible options—that is, all possible customers or employees.

I designed the DropDownField component to expose DataTextField and DataValueField properties. The former indicates the data column to use for display purposes; the latter indicates the data column to use for actual I/O data exchange. The DataTextField is useful only in one particular scenario: when your query contains joined columns. Consider the following SQL command:

SELECT o.*, e.lastname FROM orders o
INNER JOIN employees e on o.employeeid=e.employeeid
WHERE o.orderid=@id

A row from this query that is bound to a DetailsView control contains both the ID of the employee who placed the order (table Orders) and her last name (joined table Employees). In this case, you set DataValueField to employeeid and DataTextField to lastname. If you don't have joined columns, you just leave DataTextField blank.

The DropDownField component also counts three more properties. DataSourceIDForEdit indicates the data source control providing the options to list in edit/insert mode. DataValueFieldForEdit denotes the field in the data source used to determine the value of the items in the dropdown list. Finally, DataTextFieldForEdit denotes the field in the data source used to determine the display text of items in the dropdown list.

To a large extent, the code for DropDownField is the same as for CalendarField. A key difference exists when the data binding is made. To prepare the calendar control, you only need to set its SelectedDate property with the value you obtain from the bound row. With a DropDownField, you first need to populate the dropdown list and select the item that corresponds to the DataValueField's value in the bound row. The dropdown list control is bound to the list of feasible data items through the DataSourceID property. For data binding, this would work just fine. However, setting DataSourceID on a data-bound control doesn't automatically start the binding process. If you try to access the Items collection immediately after setting DataSourceID, you'll get a null reference exception. If you call DataBind to force binding, you get a stack overflow exception because you're calling DataBind from within a DataBinding event, which is triggered by a call to DataBind. The trick is to attach a DataBound event handler to the dropdown list control:

Dim dd As DropDownList = CType(target, DropDownList)
dataValue = LookupValueForEdit(target)
AddHandler dd.DataBound, AddressOf OnDropDownDataBound
dd.DataTextField = DataTextFieldForEdit
dd.DataValueField = DataValueFieldForEdit
dd.DataSourceID = DataSourceIDForEdit

The DataBound event is new in ASP.NET 2.0 for data-bound controls and is fired when the data binding process is complete. In this way, the event handler for DataBound fires when the Items collection is fully populated and you can select the item safely:

Sub OnDropDownDataBound(sender As Object sender, e As EventArgs)
   Dim dd As DropDownList = CType(sender, DropDownList)
   Dim li As ListItem = dd.Items.FindByValue(dataValue)
   li.Selected = True
End Sub

A key goal of the DropDownField component is to be able to use the bound value, DataValueField, to find a related field, possibly in another data source. This operation implies a database query, an INNER JOIN operation, or an access to the ASP.NET cache where related data is cached. Running a real query against a database is simply out of the question for very good reasons: for performance and because it would break consistency and force you to inject connection string and SQL information in a data control field. The only reasonable approach entails retrieving the related field (for example, the last name of the employee knowing the ID) from the data source used to populate the dropdown list in edit mode (you populate the dropdown list when the data control field is initialized). To minimize the performance hit, the data source should be accessed only when you enter into edit/insert mode. Caching should be enabled on the data source control and a cache key dependency mechanism should be used to refresh the cached data in case of changes.

What if the row bound to the DetailsView contains a joined field? In this case, the required information is already available. That's why I defined an optional DataTextField property. If it is set, and the specified field really exists, the data source for edit is loaded only when the host control really enters in edit/insert mode. If DataTextField is not available, then the data source is accessed at initialization time, and the input dropdown list control is created earlier and made available to internal methods through a private property. Here's the markup code that binds a DropDownField field to the employeeid column of the Orders table and the lastname column in the Employees table:

<msdn:DropDownField
     DataValueField="employeeid"
     HeaderText="Posted by"
     DataSourceIDForEdit="EmployeeDataSource"
     DataValueFieldForEdit="employeeid"
     DataTextFieldForEdit="lastname" />
EmployeeDataSource is defined in the following:
<asp:SqlDataSource ID="EmployeeDataSource" runat="server"
   ConnectionString='<%$ ConnectionStrings:LocalNWind %>'
   SelectCommand="SELECT employeeid, lastname FROM
   employees" />
The full source code of DropDownField and CalendarField is available with the download for this issue of MSDN®Magazine.

As a side note, remember that the DropDownField component that is discussed here is optimized for data-driven sources. However, the DropDownField component could easily be used to allow users to select any value out of a fixed list—for example, an enum type or any enumerable collection of data. To support these scenarios, you might want to add a DataSource property and bind its contents to the DataSource property of the dropdown list control programmatically.

Styling Control Fields

The DataControlField type defines a few style properties for header, footer, and data items. In addition, the ControlStyle property lets you style the input controls used by the field component:

<asp:BoundField DataField="CompanyName" HeaderText="Company">
     <ControlStyle BackColor="red" />
</asp:BoundField>

The preceding code snippet generates a bound field to view or edit the CompanyName data column. The internal textbox of the field will have a red background. When writing custom data control fields, you can use the ControlStyle property to style input controls in a polymorphic way. Both the Calendar and the dropdown list controls, in fact, can be styled through the ControlStyle property. By assigning default values to the property, you can also set new default settings for the input controls.

Depending on characteristics and capabilities of the input controls, you may need to define custom style properties. A new style property can be of type TableItemStyle or a new custom type derived from TableItemStyle. Finally, any new style property you add must be properly persisted in the view state. You need to override LoadViewState, SaveViewState, and TrackViewState methods. Figure 8 shows how to proceed for a sample YourStyle property.

Figure 8 Handling ViewState (Close Figure 8)
Protected Overrides Sub LoadViewState(savedState As Object)
   If Not savedState Is Nothing Then
      Dim state As Object() = CType(savedState, Object())
      If Not state(0) Is Nothing Then
         MyBase.LoadViewState(state(0))
      End If
      If Not state(1) Is Nothing Then
         YourStyle.LoadViewState(state(1))
      End If
   End If
End Sub

Protected Overrides Sub TrackViewState()
   MyBase.TrackViewState()
   YourStyle.TrackViewState()
End Sub

For the sample CalendarField field, custom style properties can be defined for any of the calendar style you want to control from the page level. The ControlStyle property in this case is of little help because it can only address the style of the calendar core element. In order to style internal parts of the helper calendar control, you need to expose some of the internal styles of the Calendar through new properties.

Conclusion

DataControlField is the base class for data fields—helper components that GridView and DetailsView controls use to build their own data-bound user interface. DataControlField has several derived classes, ImageField, CheckBoxField, and the more popular TemplateField and BoundField, that can be used in pages and ASP.NET applications. In addition, you can create custom data fields. Two canonical examples are DropDownField for foreign keys and CalendarField for dates. In addition to the description of these two components given here, the documentation for DataControlField contains another sample field—the RadioButtonField. In the documentation code, the RadioButtonField inherits from CheckBoxField and provides an alternate user interface for Boolean choices. Look there if you want to explore this topic further.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Service Station: All About ASMX 2.0, WSE 3.0, and WCF

Service Station
All About ASMX 2.0, WSE 3.0, and WCF
Aaron Skonnard

Code download available at: ServiceStation0601.exe (115KB)

The release of the Microsoft® .NET Framework 2.0 reshapes the Web services landscape in several interesting, and perhaps confusing, ways. So this month I'm going to field some of the most common questions related to ASP.NET Web Services (ASMX) 2.0, Web Services Enhancements (WSE) 3.0, and Windows® Communication Foundation (WCF), which is part of WinFX®.

Q Can you explain how the new ASMX 2.0 support for interfaces works? Which of the attributes go on the interface and which go on the class?

A The ASMX framework in the .NET Framework 2.0 introduces a significant programming model improvement: the ability to define your Web service contracts in .NET interface definitions. This feature allows you to annotate .NET interface definitions with the usual System.Web.Services attributes ([WebServiceBinding], [WebMethod], [SoapDocumentMethod], and so forth). Then you can implement the .NET interface on a .NET class to effectively implement the service contract.

This approach is very similar to the one in WCF (the main difference being the attribute names). It allows you to decouple the service contract from the implementation code, thereby making things easier to manage and reuse over time, not to mention making the code much easier to read. Figure 1 shows an example of this approach.

Figure 1 QuoteService ASMX Definitions (Close Figure 1)
using System.Web.Services;

[WebServiceBinding(
    Name = "QuoteService",
    Namespace = "http://example.org/stocks",
    ConformsTo = WsiProfiles.BasicProfile1_1,
    EmitConformanceClaims=true)]
public interface IQuoteService
{
    [WebMethod]
    StockQuote GetQuote(string symbol);
    [WebMethod]
    List<StockQuote> GetQuotes(List<string> symbols);
}

[WebService(Namespace="http://example.org/stocks")]
public class QuoteService : IQuoteService
{
    public StockQuote GetQuote(string symbol)
    {
        ... // retrieve and return new StockQuote
    }
    public List<StockQuote> GetQuotes(List<string> symbol)
    {
        ... // retrieve and return List<StockQuote>
    }
}

As illustrated in Figure 1, [WebServiceBinding] and [WebMethod] go on the interface while [WebService] goes on the class. When employing this model, all attributes that affect the service contract definition must be placed on the interface definition and not on the class. This includes [SoapDocumentService], [SoapDocumentMethod], and the various System.Xml.Serialization attributes that you could apply to method signatures. You cannot mix and match these attributes across the interface and class definitions, and ASMX helps to ensure that you don't.

For example, once you've derived a class from an interface that contains these service contract attributes, ASMX will inspect the class at run time to make sure that you haven't used any additional attributes that might further change the service contract. If ASMX finds one, it will display an error indicating you've violated the model (see Figure 2).

Figure 2 ASMX Service Attribute Usage Error
Figure 2 ASMX Service Attribute Usage Error

Although this minimizes the number of attributes you'll need to use on the derived class, there are still a few that come into play for configuring the service endpoint. You'll use [WebService] on the class to specify the <wsdl:service> details of the endpoint. This makes sense because you could have multiple services implementing the same service contract. Likewise, if you're using WSE 3.0, you'll use [Policy] on the class to apply a security policy configuration to the specific endpoint.

However, in some instances, you might also need to use [WebMethod] on the derived methods when you need to configure local processing behaviors through the various [WebMethod] properties (BufferResponse, CacheDuration, EnableSession, TransactionOption). For example, in the code in Figure 3 I'm using [WebMethod] to configure caching of the responses.

Figure 3 Configuring WebMethod Caching (Close Figure 3)
[WebService(Namespace="http://example.org/stocks")]
public class QuoteService : IQuoteService
{
    [WebMethod(CacheDuration=60)]
    public StockQuote GetQuote(string symbol)
    {
        ... // retrieve and return new StockQuote
    }
    [WebMethod(CacheDuration=60)]
    public List<StockQuote> GetQuotes(List<string> symbol)
    {
        ... // retrieve and return List<StockQuote>
    }
}

The use of [WebMethod] here is not affecting the service contract. It's simply configuring local processing behavior. When using [WebMethod] in this context, you can only use these behavioral properties and must avoid anything that could modify the service contract.

Q How can I host an ASMX 2.0 class in a custom application (without using IIS)?

A In my column in the December 2004 issue of MSDN®Magazine (Service Station: Run ASMX Without IIS), I showed how to take advantage of HTTP.sys and the new HttpListener class in the .NET Framework 2.0 to host the ASP.NET pipeline in your own application. This was a non-trivial task but the only way to reuse your ASMX classes outside of IIS in your own applications. WSE 3.0 changes all of this.

WSE 3.0 merges the new and improved ASMX 2.0 programming model with its messaging layer and hosting model. This makes it possible to host ASMX endpoints within your own applications over any WSE-supported transport such as TCP. This means I can now host the ASMX code shown in Figure 1 in a Windows service, a Windows Forms application, or even a console application. The code is simple—you just call SoapReceivers.Add and supply your ASMX class, as illustrated here:

class Program {
    static void Main(string[] args) {
        Uri uri = new Uri("soap.tcp://localhost:9393/quoteservice");
        SoapReceivers.Add(uri, typeof(QuoteService));
        Console.WriteLine("Listening...");
        Console.ReadLine();
    }
}

This is very similar to the WCF model where you use ServiceHost to host service types in any host environment. However, before you get too excited, I should point out that you cannot supply an HTTP address to SoapReceivers.Add like you can with ServiceHost in WCF. SoapReceivers doesn't provide direct integration with HTTP.sys, which is what would be required in this case. So to host your ASMX services over HTTP, you must revert to an <httpHandler> mapping or write the HTTP.sys integration code yourself.

Q What features does WCF provide for developers that ASMX 2.0 and WSE 3.0 don't?

A ASMX 2.0 and WSE 3.0 provide many key features that some developers mistakenly believe are unique to WCF. For example, both stacks provide a similar attribute-based programming model where you can author service contracts on .NET interface definitions (see the first question and answer in this column). Both stacks provide transport-neutral SOAP implementations that ship with multiple transport channels and custom transport hooks. Both stacks allow for hosting services in any Windows-based application (through SoapReceivers in WSE or ServiceHost in WCF). Both stacks provide multiple message encodings, including the two most widely supported: XML 1.0 and Message Transmission Optimization Mechanism (MTOM). And both stacks provide support for message-based security, configurable via simple configuration elements (turnkey security profiles). The table in Figure 4 summarizes how the stacks compare.

Figure 4 ASMX 2.0 Plus WSE 3.0 Compared to WCF (Close Figure 4)

Feature | ASMX 2 .0 plus WSE 3. 0 | WCF
Hosting | IIS/ASP.NET (.asmx) SoapReceivers | IIS/ASP.NET (.svc) ServiceHost<T> WAS
Programming Model | [WebService], [WebMethod], and so on (supports interfaces, generics, and the like) | [ServiceContract], [OperationContract], and so on (supports interfaces, generics, and so on)
Message Exchange Patterns (MEP) | One-way Request-response Custom (using WSE API) | One-way Request-response First/last-operation Duplex Custom
XML Serialization | System.Xml .Serialization | System.Runtime .Serialization System.Xml.Serialization (you can choose)
Encodings | XML 1.0 MTOM DIME Custom | XML 1.0 MTOM Binary Custom
Transports | HTTP TCP Custom | HTTP TCP Named pipes MSMQ P2P Custom
Protocols | Security | Security Reliable messaging Transactions
Behaviors (enabled via attributes or configuration | Local DTC transactions HTTP buffering HTTP caching HTTP sessions Custom (via SoapExtensions, WSE filters) | Concurrency Instancing Throttling Thread-binding Exception handling and faults Impersonation Session management Transaction behaviors Custom (via behavior types)

The main area where the stacks differ is in their support for the various WS-* specifications. WSE 3.0 only supports the security framework while WCF supports the security, reliable messaging, and transaction frameworks. And as the table illustrates, WCF also provides more local processing behaviors and customization hooks. Virtually every layer in the WCF object model is extensible via code, attributes, or configuration.

Ultimately, WCF offers a more complete development framework that has been designed from the ground up around the various Web services protocols. Still, you can have a very similar experience today if you use ASMX 2.0 plus WSE 3.0. And the outlook for future migration looks promising.

Q Will it be difficult to port ASMX code to Windows Communication Foundation?

A Not unless you consider changing attribute names difficult. Moving ASMX code forward to WCF is a mechanical process, one which you could even automate to some degree if desired. This process entails converting the various System.Web.Services and System.Web.Services.Protocols attributes to the equivalent System.ServiceModel attributes. Figure 5 shows an example of how the ASMX code in Figure 1 would port to WCF.

Figure 5 Porting the QuoteService to WCF (Close Figure 5)
using System.ServiceModel;

[ServiceContract(
    Name = "QuoteService",
    Namespace = "http://example.org/stocks",
    FormatMode = ContractFormatMode.XmlSerializer)]
public interface IQuoteService
{
    [OperationContract]
    StockQuote GetQuote(string symbol);
    [OperationContract]
    List<StockQuote> GetQuotes(List<string> symbols);
}

public class QuoteService : IQuoteService
{
    public StockQuote GetQuote(string symbol)
    {
        ... // retrieve and return new StockQuote
    }
}

Both frameworks support annotating either the class definition or a separate interface definition, so additional refactoring isn't necessary, but may be desired. And by using FormatMode=ContractFormatMode.XmlSerializer, you don't have to port all of your message types over to the new [DataContract] model if you don't want to. Microsoft will publish more detailed migration details as WCF nears release.

Q When should I use WSE with ASMX and when should I use ASMX alone? I'd like to be clearer on this issue.

A ASMX 2.0 provides support for the WS-I Basic Profile 1.1 and SOAP 1.2. This means support for XML 1.0, XML Schema Definition (XSD), Web Services Description Language (WSDL), SOAP 1.1 and SOAP 1.2, and basic profile conformance validation at compile time. WSE 3.0 picks up where ASMX 2.0 leaves off by providing support for some of the more advanced WS-* protocols. Today, you use WSE 3.0 when you need to enhance your ASMX services with one or more of the following features:

  • Message-base security (WS-Security)
  • Efficient binary data transfer (MTOM)
  • Alternative hosting environments
  • Custom declarative policy pipeline

Message-based security and MTOM are the number one and number two reasons people turn on WSE in their ASMX projects, and the good news is that you can do so without modifying the service logic. WSE 3.0 taps into ASMX through its provided extensibility points. This makes it possible for you to take advantage of various WSE 3.0 features without having to learn a new programming model—all you have to do is run the WSE Settings Tool and select the options you want.

The other reason to enable WSE 3.0 in your ASMX projects is to take advantage of its policy pipeline extensibility model. WSE makes it possible to inject filters that perform pre- and post-processing on the SOAP messages entering and leaving the endpoint. This extensibility point is easier to work with than the ASMX SoapExtension framework as it provides the ability to create custom declarative policies that can be combined with the existing WSE-supplied declarative security policies. You may decide to "turn on" WSE just to have this extensibility option, or at the very least to use the improved SOAP message diagnostic tracing feature.

With WSE 2.0, most developers turned away from ASMX when they needed to do custom hosting. Now that WSE 3.0 makes it possible to do custom hosting of ASMX types, this is no longer the case. The main scenario left that would lead you towards using the WSE messaging API standalone is the need to implement alternative message exchange patterns, such as duplex communications, multicast, or pub/sub scenarios. The messaging classes, SoapSender and SoapReceiver, make it possible to implement any type of advanced messaging pattern.

Q When I enable WSE 3.0 in my Web site, I no longer see the WSE SoapExtension in the config file. Instead I see a SoapServerProtocolFactory. What is that?

A By default, ASMX endpoints can be invoked using a few different server protocols, including HttpSoap, HttpSoap12, HttpGet, HttpPost, HttpPostLocalhost, and Documentation. You can enable or disable any of these protocols in Web.config. The default ASMX configuration in the .NET Framework 2.0 is shown here:

<webServices>
   <protocols>
      <clear />
      <add name="HttpSoap12" />
      <add name="HttpSoap" />
      <add name="HttpPostLocalhost" />
      <add name="Documentation" />
   </protocols>
   ...
</webServices>

Each of these protocols expects a different type of HTTP message and implements a different ASMX invocation protocol. For example, in the case of HttpSoap, the incoming HTTP payload will always contain a <soap:Envelope> whereas with HttpGet the data comes across in the query string and the HTTP payload is empty. The former returns a SOAP envelope in the response, while the latter just returns plain XML.

In order to deal with these differences, the ASMX handler (System.Web.Services.Protocols.WebServiceHandler) uses what's called a ServerProtocol class to process the incoming message. There are several classes that derive from ServerProtocol, including SoapServerProtocol and HttpGetServerProtocol. These classes map to the different protocols that you see in the configuration file. The ASMX handler inspects the incoming HTTP message and chooses the appropriate ServerProtocol type. From that point on, the ServerProtocol instance manages the process of invoking the operation. The SoapExtension framework is one of the things it manages during the invocation process.

The SoapExtension framework is an extensibility point for developers to inject code between the ASMX handler and the target WebMethod. It essentially provides the mechanism for transforming incoming and outgoing message streams. When you need to modify messages in a SoapExtension, you must provide a new stream that it will hand to the next extension in the chain (this is wired up with SoapExtension.ChainStream in the beginning). Hence, a SoapExtension receives a stream, parses it, does its work, and writes to a stream that the next SoapExtension will read from, and this process repeats itself.

WSE 2.0 was implemented entirely as a SoapExtension. One of the problems with implementing WSE at this level is that it led to a great deal of serialization/deserialization overhead at the stream level. Moving to WSE 3.0, the WSE team wanted to optimize their SOAP processing model by moving up an abstraction layer and writing their own SoapServerProtocol implementation. So they asked the ASMX team if they would make it possible to replace the default SoapServerProtocol via configuration in ASMX 2.0. The ASMX developers provided a new configuration element called <soapServerProtocolFactory> that anyone can use to provide their own implementation (although this is not encouraged). The ASMX handler reads this type and uses it to create instances of the corresponding SoapServerProtocol type to process new requests. WSE 3.0 uses this hook and replaces the default SoapServerProtocol class with Microsoft.Web.Services3.WseProtocol.

WseProtocol manages both the SoapExtension mechanism and the WSE pipeline within the same class, thereby reducing some stream friction. WSE 3.0 still uses a SoapExtension (called WseProtocolExtension) but it never shows up in the configuration file because WseProtocol injects it dynamically at startup (to ensure it's always there). The WseProtocolExtension is responsible only for recognizing and parsing MTOM and XML 1.0 streams; everything else is done by the WseProtocol implementation.

So to make a long story short, the <soapServerProtocolFactory> is how WSE 3.0 wires itself into ASMX 2.0. WSE 3.0 still uses a SoapExtension, it's just not where it decided to do the bulk of its work in this version.

Q When hosting ASMX classes in my own app, what happens to HttpContext.Current? What about SoapExtensions?

A Unfortunately, the ASMX design is tightly coupled with ASP.NET. For example, when Visual Studio® creates new ASMX classes, it derives them from System.Web.Services.WebService by default. This exposes the various System.Web.HttpContext properties through properties on the new ASMX class, thereby encouraging their use throughout WebMethods. Some developers are also accustomed to explicitly using the HTTP context via HttpContext.Current.

When you host an ASMX class using SoapReceivers in WSE 3.0, this type of code no longer works. In this case, your code is executing outside of IIS and the ASP.NET HTTP pipeline. HttpContext.Current will always be null, so this may require you to modify your existing WebMethod logic. In general, you'll want to avoid depending on anything in the HTTP pipeline throughout your WebMethod code if you plan to reuse that type across hosting environments outside of ASP.NET.

Also, when using SoapReceivers as a host, you can no longer depend on HttpModules or SoapExtensions to do the work for you. These intermediaries are no longer in play since you're now executing outside of the ASMX handler.

Q I can't find the filters tab in the WSE 3.0 settings tool. How can I access it?

A The Filters tab is no longer in the WSE 3.0 Settings Tool because the <filters> element has been removed from the WSE 3.0 configuration section. In WSE 2.0, you used the <filters> element to define how you wanted the pipeline to be constructed with filters. One of the filters included by default was the policy filter. The policy filter looked at another configuration file (the policy file) to determine how to process incoming and outgoing messages according to its assertions (which were similar to filters).

In WSE 3.0, these two concepts have been unified. So now policies consist of assertions and each assertion injects WSE filters into the pipeline. You place assertions in a policy to control how the filter pipeline is built and organized at run time. So if you want to inject a custom filter into the pipeline, you write the filter class, write a policy assertion class that uses the new filter, use the assertion class in a policy, and apply the policy to a service. In WSE 3.0, it's the policy framework that drives the pipeline so there is no longer a need for the <filters> section.

Q What transports are supported by WSE 3.0? Is it hard to write a custom transport?

A WSE 3.0 supports HTTP and TCP. WSE provides a custom transport framework to allow third parties to plug in their own transport implementations. Since the release of WSE 2.0, several developers have provided sample WSE transports to illustrate how this can be done. There is one available for Microsoft Message Queue Server (MSMQ) (see SoapMSMQ Transport), SMTP (see soap.smtp://), and UDP (see Soap.Udp 0.1). All of these transports were written for WSE 2.0, but they port over to WSE 3.0 without many issues.

This code provides an example of using the SMTP transport to host a service in a console application:

SoapReceivers.Add(new Uri("soap.smtp://test@skonnard.com"),
    typeof(ExpenseReportService));
Console.ReadLine();
Then on the client side you simply need to specify the SMTP address before invoking the method, as shown here:
ExpenseReportServiceWse svc = new ExpenseReportServiceWse();
svc.Url = "soap.smtp://test@skonnard.com";
svc.Submit(report);

Now the client will send messages to the test@skonnard.com e-mail address, and the receiver will pick them up from that same address. One of the interesting things about this example is that the client and receiver no longer have to be running at the same time. Notice how the developer experience is unaffected when using a custom transport. The only thing you have to do is tell WSE how to process the "soap.smtp" protocol scheme by adding a <transport> mapping to the WSE configuration section, as illustrated here:

<microsoft.web.services3>
  <messaging>
    <transports>
      <add scheme="soap.smtp" type="SoapSmtpTransport,SoapSmtp"/>
    </transports>
  </messaging>
  ...
</microsoft.web.services3>
This element maps a protocol scheme to an ISoapTransport-derived class. When you call SoapReceivers.Add, it looks up the type that maps to the specified protocol scheme from the configuration section. Then it instantiates the specified type and uses the ISoapTransport methods to produce the input and output communication channels. The same thing happens on the client side. These channels hide all transport implementation details from the rest of the API.

Hence, writing a custom WSE transport entails implementing three classes: one that models the transport (ISoapTransport) and two for modeling input/output channels (ISoapInputChannel and ISoapOutputChannel). The transport serves up channels and manages resources shared across all channels, while the channels focus on sending/receiving messages via the transport. You're dealing with low-level transport details here, so I wouldn't say it's easy. The best way to get your head around how this works is to walk through the existing WSE transport samples I mentioned previously.

Q Can I use "Add Web Reference" to generate a proxy for a custom hosted WSE 3.0 service?

A If you have a service that is hosted in a Windows service using TCP for communications, the only way to retrieve the WSDL is by using the WseWsdl3.exe tool. This tool sends a SOAP message to the service requesting its metadata and generates the proxy from the WSDL returned in the SOAP response message.

However, if you have the same service also hosted via ASP.NET (HTTP), you can use the typical "Add Web Reference" command in Visual Studio (or wsdl.exe) to generate a proxy (make sure you have WSE enabled in the client application before doing this so you get a WSE proxy). The generated WSE proxy can be used with any WSE transport, not just with HTTP. Simply change the address of the proxy to the TCP address using either the Url or Destination property and it will work just like a proxy generated via WseWsdl3.exe.

Q How should I organize my solution to share ASMX types across different hosts?

A When you need to share ASMX types across different hosts, you should put your ASMX classes in a shared class library. This seems kind of odd at first, since your ASMX code has probably always existed in Web projects, but there's no reason they can't be factored out into a library. I would even go a step further and organize all of my message serialization classes into a separate class library of their own, as you may want to share these types with clients and code generators (such as using a schema importer extension).

With this configuration, simply add a reference to your class libraries in each host project, including the Web site host. Then your Web site host will no longer need the service code in App_Code. Instead your .asmx files will refer to the shared service classes by name. And your custom hosts can supply the same service type when calling SoapReceivers.Add.

Q My team developed a common library of XSD serialization classes that we want all of our developers to use when implementing or consuming our services. When they run "xsd.exe /c" or "wsdl.exe", the code generator should use our types instead of generating new ones. Is this possible?

A Yes, by implementing what's called a schema importer extension—a new extensibility point offered by the .NET Framework 2.0. Here's the gist of it. You write a class that derives from SchemaImporterExtension and override ImportSchemaType. In your implementation of this method, you define how the incoming XSD types should map to your existing .NET types. Figure 6 shows an example.

Figure 6 Implementing a SchemaImporterExtension (Close Figure 6)
class CommonTypesSIE : SchemaImporterExtension
{
    public override string ImportSchemaType(
        string name, string ns, XmlSchemaObject context,
        XmlSchemas schemas, XmlSchemaImporter importer,
        CodeCompileUnit compileUnit,
        CodeNamespace mainNamespace,
        CodeGenerationOptions options,
        CodeDomProvider codeProvider)
    {
        if (name.Equals("StockQuote") &&
            ns.Equals("http://example.org/stocks"))
        {
            mainNamespace.Imports.Add(
               new CodeNamespaceImport("CommonTypes"));
            return "CommonTypes.StockQuote";
        }
        return null;
    }
}

Then you configure your schema importer extension to be used by xsd.exe or wsdl.exe. You can do this in the <schemaImporterExtensions> element of the <system.xml.serialization> configuration section. Or you can do it by supplying the list of schema importer extensions to use when running xsd.exe or wsdl.exe (via the /parameters option).

Q Can you tell me what the file format is for /parameters in xsd.exe and wsdl.exe?

A Both xsd.exe and wsdl.exe introduce a new switch, /parameters, for packaging up all of the command-line options you want to use into a single XML file. The format of the XML file is defined by an XML Schema definition that is available in the download for this column. The file must contain a top-level <wsdlParameters> element, which contains all of the options. Figure 7 shows an example.

Figure 7 Options.xml (Close Figure 7)
<!-- options.xml -->
<wsdlParameters xmlns='http://microsoft.com/webReference/'>
    <language>c#</language>
    <sharetypes>true</sharetypes>
    <webReferenceOptions
         xmlns="http://microsoft.com/webReference/">
        <codeGenerationOptions>newAsync
           </codeGenerationOptions>
        <verbose>true</verbose>
        <schemaImporterExtensions>
            <type>CommonTypesSIE, CommonTypesSIE</type>
        </schemaImporterExtensions>
    </webReferenceOptions>
</wsdlParameters>

You would then pass that file to wsdl.exe, as shown here:

C:\demos> wsdl.exe /parameters:options.xml
  http://localhost/stocks/quoteservice.asmx

This approach helps ensure a consistent set of options each time one of these code-generation tools is used. It's interesting to note that some of the xsd.exe and wsdl.exe features are only accessible via this mechanism.

Q I've heard speculation that version 3.0 might be the last WSE release. Is this true?

A After WCF ships, WSE will most likely stop evolving with new features and move into support mode (bug fixes, service packs, and so forth). WCF is positioned as the new .NET Web services platform, which completely replaces the need for the functionality WSE has provided up to this point. And since WCF is getting closer to its release date, it's very likely that WSE 3.0 will be the last version to ship. The WSE development team has already been restructured within the WCF team to consolidate the efforts of the developers across the teams.

However, it's still likely that something like WSE will be used to extend and "enhance" WCF after it hits the streets, although who knows what they will call it. Such a vehicle could be used to incorporate new protocols and behaviors as they become available using the various WCF extensibility points.

Be sure to check the following documents for more on WSE: "Developing .NET Web Services with Beta 2", "What's New in WSE 3.0", and the Web Services page.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services 2.0, Applied BizTalk Server 2006, and Introducing Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


Extreme ASP.NET: Codebehind and Compilation in ASP.NET 2.0

Extreme ASP.NET
Codebehind and Compilation in ASP.NET 2.0
Fritz Onion

Contents
Codebehind
Compilation
Assembly Generation
Conclusion

As I write this column, the release candidates of the Microsoft® .NET Framework 2.0 and Visual Studio® 2005 have just come out, and by the time you read this, they will both already be on the shelves. It feels like it's been a long time coming.

I remember sitting in a room on the Microsoft campus in August of 2003 listening to Scott Guthrie and others (including my fellow columnist, Rob Howard) present the wide array of new features coming in ASP.NET 2.0. They astounded us with one demo after another of features that greatly simplified Web development, and did so in a pluggable and extensible fashion so that changes could be made at any level as needed during the development process.

Quite a bit has changed in the subsequent beta releases, mostly in the form of refinements, bug fixes, and control additions. However, one feature—the codebehind model—has changed rather dramatically since that first preview, primarily in response to customer feedback. Now on the cusp of the release, I thought I would take this opportunity to describe this new codebehind model, the rationale behind it, and how you as a Web developer will use it. I will also cover some of the potentially unexpected side effects of this model and how to plan for them in your designs. Note that the ASP.NET 2.0 runtime fully supports the 1.x model, so applications written for 1.x can run without modification.

Codebehind

Although the codebehind model is different in 2.0, its syntax has changed little. In fact, the change is so subtle that you may not even notice it unless you look really closely. Figure 1 shows the new codebehind syntax.

Figure 1 Syntax in ASP.NET 2.0 (Close Figure 1)

Default.aspx
<%@ Page Language="C#" AutoEventWireup="true"
    CodeFile="Default.aspx.cs" Inherits="MsdnMag.Default" %>
Default.aspx.cs
namespace MsdnMag
{
    public partial class Default : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
        }
    }
}

There are two differences between this model and the previous 1.x model—the introduction of the CodeFile attribute in the @ Page directive and the declaration of the codebehind class as a partial class. As you start building the page, you will notice another difference—server-side controls no longer need to be explicitly declared in your codebehind class, but you still have complete access to them programmatically. For example, the form in Figure 2 has several server-side controls that are used programmatically in the codebehind file, but notice the absence of any explicit control declarations in the codebehind class.

Figure 2 Implicit Server-Side Control Access (Close Figure 2)

Default.aspx
<%@ Page Language="C#" AutoEventWireup="true"
    CodeFile="Default.aspx.cs" Inherits="MsdnMag.Default" %>

<!DOCTYPE html PUBLIC "..." "...">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Untitled Page</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
     Enter your name:
     <asp:TextBox ID="_nameTextBox" runat="server" /><br />
     <asp:Button ID="_enterButton" runat="server"
                 Text="Enter" OnClick="_enterButton_Click"/> <br />
     <asp:Label ID="_messageLabel" runat="server" />
    </div>
    </form>
</body>
</html>
Default.aspx.cs
namespace MsdnMag
{
  public partial class Default : System.Web.UI.Page
  {
    protected void _enterButton_Click(object sender, EventArgs e)
    {
      _messageLabel.Text = "Hello there " + _nameTextBox.Text + "!";
    }
  }
}

The reason this works has to do with the partial keyword applied to your codebehind class. In addition to turning your .aspx file into a class definition with methods for rendering the page, as it has always done, ASP.NET now also generates a sibling partial class for your codebehind class that contains protected control member variable declarations. Your class is then compiled together with this generated class definition and used as the base class for the class generated for the .aspx file. The end result is that you essentially write codebehind classes the way you always have, but you no longer have to declare (or let the designer declare for you) member variable declarations of server-side controls. This was always a somewhat fragile relationship in 1.x, since if you ever accidentally modified one of the control declarations so that it no longer matched the ID of the control declared on the form, things suddenly stopped working. Now the member variables are declared implicitly and will always be correct. Figure 3 shows an example set of classes involved.

Figure 3 Class Generation with Codebehind (Close Figure 3)

Class for ASPX file generated by ASP.NET
namespace ASP
{
    public class default_aspx : MsdnMag.Default
    {
       ...
    }
}
Sibling partial class generated by ASP.NET
namespace MsdnMag
{
  public partial class Default : IRequiresSessionState
  {
    protected TextBox  _nameTextBox;
    protected Button   _enterButton;
    protected Label    _messageLabel;
    private   HtmlForm form1;
    ...
  }
}
Codebehind partial class that you write
namespace MsdnMag
{
  public partial class Default : Page
  {
    void _enterButton_Click(object sender, EventArgs e)
    {
       _messageLabel.Text = "Hello there " + _nameTextBox.Text + "!";
    }
  }
}

Note that the partial class model is only used if you use the CodeFile keyword in your @ Page directive. If you use the Inherits keyword without CodeFile (or with the src attribute instead), ASP.NET resorts to the 1.x codebehind style and simply places your class as the sole base class for the .aspx file. Also, if you have no codebehind at all, the class generation acts very much the same as it does in 1.x. Since ASP.NET 2.0 is backwards compatible with 1.x, there is now a range of codebehind options at your disposal.

Visual Studio 2005 will use the new partial class codebehind model for any Web Forms, and it will also happily convert Visual Studio .NET 2003 projects to use the new model as well if you use the conversion wizard. It is best, if possible, to convert all files to the new codebehind model, since some of the new features of ASP.NET 2.0 depend on it (if you're using Visual Studio, converting is pretty much the only option, since Visual Studio 2005 won't open unconverted 1.x projects). For example, strongly typed access to the Profile property bag is added to the sibling partial class for codebehind classes in 2.0, but if you use the 1.x codebehind model, that strongly typed accessor is added directly to the .aspx generated class definition, and will be unavailable to your codebehind class. This is also true for strongly typed Master Page and previous page access.

Compilation

At this point, you may be wondering why the ASP.NET team bothered to use inheritance at all with this new codebehind model. ASP.NET could easily generate all of the control variable declarations in addition to the rendering methods from the .aspx file as a partial class which could then be merged with your simplified codebehind class. This is exactly how Windows Forms works in the .NET Framework 2.0. All of the designer-generated code is placed into a sibling partial class which is then merged with your application logic and event handlers into a single Form-derived class, creating a clean separation between machine-generated code and developer code without resorting to inheritance.

Well, it turns out that the original implementation of codebehind in ASP.NET 2.0 did exactly this—the codebehind class was just a partial class that was merged with the parsed .aspx file class definition. It was simple and effective, but unfortunately, not flexible enough. The problem with this model was that it was no longer possible to deploy the codebehind files in precompiled binary assemblies along with intact .aspx files since they now had to be compiled at the same time (a restriction when using partial classes is that all partial pieces of a class must be merged during a single compilation, and class definitions cannot span assemblies). This restriction was unacceptable to many developers as they were already used to being able to deploy binary codebehind assemblies along with intact .aspx files which could then be updated in place without having to recompile. This is, in fact, the exact model used by default in Visual Studio .NET 2003, and is thus very prevalent in practice.

As a result of reintroducing the inheritance model and shifting the partial class into the base class, .aspx files can now be deployed and compiled independently from the codebehind class. To complete the picture, you need some way to generate the sibling partial classes containing control variable declarations during compilation or deployment since this was always done in the past on demand in response to requests. Enter the ASP.NET compiler.

The ASP.NET compiler (aspnet_compiler.exe) was originally introduced in ASP.NET 2.0 as a way of completely precompiling an entire site, making it possible to deploy nothing but binary assemblies (even .aspx and .ascx files are precompiled). This is compelling because it eliminates any on-demand compilation when requests are made, eliminating the first postdeployment hit seen in some sites today. It also makes it more difficult for modifications to be made to the deployed site (since you can't just open .aspx files and change things), which can be appealing when deploying applications that you want to be changed only through a standard deployment process. The compiler that ships with the release version of ASP.NET 2.0 supports this binary-only deployment model, but it has also been enhanced to support an updatable deployment model, where all source code in a site is precompiled into binary assemblies, but all .aspx and .ascx files are left basically intact so that changes can be made on the server (the only changes to the .aspx and .ascx files involve the CodeFile attribute being removed and the Inherits attribute being modified to include the assembly name). This model is possible because of the reintroduction of inheritance in the codebehind model, so that the sibling partial classes containing control declarations can be generated and compiled independently of the actual .aspx file class definitions.

Figure 4 Binary Deployment with aspnet_compiler.exe
Figure 4 Binary Deployment with aspnet_compiler.exe

Figure 4 shows an invocation of the aspnet_compiler.exe utility using the binary deployment option, and the resulting output to a deployment directory. Note that the .aspx files in the deployment directory are just marker files with no content. They have been left there to ensure that a file with the endpoint name is present in case the "Check that file exists" option for the .aspx extension in an IIS app is set. The PrecompiledApp.config file is used to keep track of how the app was deployed and whether ASP.NET needs to compile any files at request time. To generate the "updatable" site, you would add a -u to the command line, and the resulting .aspx files would contain their original content (and not be empty marker files). Note that this functionality can also be accessed graphically through the Build | Publish Web Site menu item of Visual Studio 2005, as you can see in Figure 5. Both the command-line tool and Visual Studio rely on the ClientBuildManager class of the System.Web.Compilation namespace to provide this functionality.

Figure 5 Build | Publish Web Site Tool in Visual Studio 2005
Figure 5 Build | Publish Web Site Tool in Visual Studio 2005

With the aspnet_compiler utility in hand, you can work on your application without worrying about how it will be deployed for the most part, since any site can now be deployed in any of three ways—all source, all binary, or updatable (source code in binary and .aspx files in source)—without any modification to page attributes or code files used in development. This was not possible in previous releases of ASP.NET since you had to decide at development time whether to use the src attribute to reference codebehind files or to precompile them and deploy the assemblies to the /bin directory. Complete binary deployment was not even an option.

Assembly Generation

Now that compilation into assemblies can happen in one of three places (either explicitly by the developer, using aspnet_compiler.exe, or during request processing), understanding the mapping of files into assemblies becomes even more important. In fact, depending on how you write your pages, you can actually end up with an application that works fine when deployed as all source or all binary, but which fails to compile when deployed using the updatable switch.

The model ASP.NET generally uses creates separate assemblies for the contents of the App_Code directory as well as the global.asax file (if present), and then compiles all of the .aspx pages in each directory into a separate assembly. (If pages in the same directory are authored in different languages or if they have dependencies on each other through an @ Reference directive, they could also end up in separate assemblies.) User controls and Master Pages are also typically compiled independently from .aspx pages. It is also possible to configure the App_Code directory to create multiple assemblies if, for example, you wanted to include both Visual Basic® and C# source code in a project. There are some subtleties in the details of assembly creation, depending on which mode of deployment you have chosen. Figure 6 describes the components of your Web site that compile into separate assemblies based on the deployment mode you are using. (Note that I am ignoring the resource, theme, and browser directories since they don't contain code, although they are compiled into separate assemblies as well. The target assembly can also differ based on language variance and reference dependencies, as mentioned previously.)

Figure 6 Assembly Generation (Close Figure 6)

 | Deployment Mode
 | All Source | All Binary | Updatable (mixed)
What compiles into a unique assembly | App_Code directory global.asax .ascx and associated codebehind file (separate assembly for each user control) .master and associated codebehind file (separate assembly for each master page) All .aspx files and their codebehind files in a given directory (separate assembly per directory) | App_Code directory global.asax .ascx and .master files and their associated codebehind files All .aspx files and their codebehind files in a given directory (separate assembly per directory) | App_Code directory (D) global.asax (R) .ascx and .master files (R) codebehind files for .ascx and .master files (D) All .aspx files in a given directory (separate assembly per directory) (R) All codebehind files associated with .aspx files in a given directory (separate assembly per directory) (D)
When it's compiled | Request time | Deployment time | (R) = Compiled at request time (D) = Compiled at deployment time

The only other twist in the assembly generation picture is that you can use the -fixednames option of aspnet_compiler to request that each .aspx file be compiled into a separate assembly whose name remains the same across different invocations of the compiler. This can be useful if you want to be able to update individual pages without modifying other assemblies on the deployment site. It can also generate a large number of assemblies for sites of any significant size, so be sure to test your deployment before depending on this option.

If this sounds complicated, the good news is that most of the time you shouldn't have to think about which files map to separate assemblies. Your .aspx files are always compiled last, and always include references to all other generated assemblies, so typically things will just work no matter what deployment model you choose.

One of the key differences in deployment that may actually affect the way you author code in your pages is the split in compilation when using updatable deployments. When you deploy an updatable site, the codebehind files are compiled into separate assemblies prior to deployment. The classes generated from the .aspx files are not compiled until a request is actually made for a file in a directory. This is in contrast to binary deployment, in which all files are compiled prior to deployment, and to source deployment, in which all files are compiled at request time. As a simple example of how this can cause problems, consider the user control (.ascx file) in Figure 7 with an embedded property, and an associated page that uses the control and sets the property from its codebehind class.

Figure 7 User Control (Close Figure 7)

MyUserControl.ascx
<%@ Control Language="C#"%>

<script runat="server">
    public string Color
    {
        get { return (string)ViewState["color"] ?? "white"; }
        set { ViewState["color"] = value; }
    }

    protected override void OnLoad(EventArgs e)
    {
        _mainPanel.BackColor = System.Drawing.Color.FromName(Color);
        base.OnLoad(e);
    }
</script>

<asp:Panel runat="server" ID="_mainPanel">
<h2>Some text</h2>
Other text
</asp:Panel>
Default.aspx
<%@ Page Language="C#" AutoEventWireup="true"  CodeFile="Default.aspx.cs" Inherits="_Default" %>

<%@ Register Src="MyUserControl.ascx" TagName="MyUserControl" TagPrefix="uc1" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Untitled Page</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <uc1:MyUserControl id="MyUserControl1" runat="server">
        </uc1:MyUserControl></div>
    </form>
</body>
</html>
Default.aspx.cs
using System;
using System.Web.UI;

public partial class _Default : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        MyUserControl1.Color = "purple";
    }
}

The page in Figure 7 will compile and run in either source or binary deployment mode, but will fail to compile when deployed as an updatable site since the definition of the Color property of the user control is unavailable at deployment time (this limitation also existed in the 1.x model). You can typically avoid issues like this by keeping all code in codebehind files or, at the other extreme, not using codebehind files at all and leaving code directly in .aspx and .ascx files.

Another thing to keep in mind when considering the file-to-assembly mapping is that the use of the internal keyword to prevent external assemblies from accessing methods in your classes may work in some deployment scenarios and not others, because of the different assembly mapping options. Unless you plan ahead of time which deployment option you will be using, it is probably best to avoid internal methods in your pages and stick to the type-scoped protection keywords: public, protected, and private.

Conclusion

The new codebehind model in ASP.NET 2.0 seems both familiar and foreign to ASP.NET developers. It's familiar because it still uses inheritance to relate codebehind classes with their .aspx generated class definitions, and yet foreign elements like partial classes and the implicit generation of control member variable declarations are fundamental shifts. In practice, you will probably not notice much difference in usage, but it will be important to understand the class relationships and assembly mappings outlined here whenever you are doing something out of the ordinary, like creating a common base Page class or mixing codebehind and inline code models.

Send your questions and comments for Fritz to xtrmasp@microsoft.com.

Fritz Onion is a cofounder of Pluralsight, a premier Microsoft .NET training provider, where he heads the Web development curriculum. Fritz is the author of Essential ASP.NET (Addison Wesley, 2003) and the upcoming Essential ASP.NET 2.0 (Addison Wesley, 2006). Reach him at pluralsight.com/fritz.


Security Briefs: Encrypting Without Secrets

Security Briefs
Encrypting Without Secrets
Keith Brown

Code download available at: SecurityBriefs0601.exe (150KB)
Contents
Thinking About Encryption
Using Public Key Cryptography in Managed Code
How Smart is Your Key Container?
Key Containers and the Key Manager
Key Size
The Encryptor and Decryptor
The Sample Encrypting Agent (Take Orders)
The Sample Decrypting Agent (Process Orders)
Moving Ahead

Do you have a Web site or other system that deals in secrets of any sort? It seems like every time I give a security talk, people ask how to deal with the sticky problem of storing secrets. Connection strings with passwords are an obvious problem. You're better off simply using integrated security to get rid of those secrets, at least with SQL Server™, or an Oracle database. But what about credit card numbers and other financial or personal information? Can encryption help?

Thinking About Encryption

If you're considering encryption, keep in mind that it's a secret compression or transfer technique. Encryption doesn't get rid of secrets, but it's good at turning big secrets (secret documents and so on) into smaller secrets called keys. But there's still a key somewhere! The trick is figuring out how to separate the key from the data to reduce the chances that an attacker could get both the encrypted data and the decryption key.

Figure 1 Encrypting and Decrypting Agents
Figure 1 Encrypting and Decrypting Agents

Many systems that deal in secrets (credit card numbers, personal information, and so on) can be factored to allow the most highly exposed machines, such as Web servers, to accept secret data and encrypt it without ever having a secret key. Once the ciphertext is stored in a database or queue, a business process running on a dedicated and highly secure machine can retrieve the ciphertext and decrypt it. Figure 1 shows an example of factoring the encrypting and decrypting agents onto separate machines. The encrypting agent holds a public key, and the decrypting agent holds the corresponding private key. Here I'll present a sample application that uses this technique to protect a user's financial information while it's waiting to be processed.

Using Public Key Cryptography in Managed Code

There are a couple of ways to use public key cryptography in the Microsoft® .NET Framework. If you're working with a public key infrastructure (PKI), you can make use of the rich support for certificates in the version 2.0 base class library (BCL). But not everyone wants to manage a PKI, and this isn't really required for the ideas I'm presenting here. For those simpler cases, the BCL includes a couple of low-level classes that allow you to use public key cryptography directly, without the overhead of certificates. The class I'll be using here is RSACryptoServiceProvider. This class allows you to generate public/private key pairs and store the key material in CryptoAPI (CAPI) key containers on a file system or smart card. And, of course, you can also use this class to encrypt and decrypt data.

One thing you should keep in mind with public keys is that they are very good at encrypting small bits of data, such as secret keys or hashes. But they are not designed for encrypting bulk data. So my solution will use a typical hybrid cryptosystem: I'll generate a random secret key and use Advanced Encryption Standard (AES) to encrypt the actual data I want to protect. Then I'll use RSACryptoServiceProvider to encrypt the AES key.

How Smart is Your Key Container?

RSACryptoServiceProvider is a thin veneer over CAPI, where key material is managed in something called a key container. This container abstraction allows you to use a variety of crypto providers, including some that use hardware devices, like smart cards, to store key material. Smart cards are a great way to store private keys because the key material never leaves the card. The card has its own embedded memory and a microprocessor, so it can generate the key material you need, store it, and use it to encrypt and decrypt data without ever disclosing the private key to the main computer's memory. If there's ever any downtime on your decrypting agent (that is, if it doesn't run 24?7), an administrator can simply pull out the smart card to reduce the window of exposure to attack. When the private key is not available anywhere on the system, your encrypted data is much safer than when it is!

My solution will use the built-in cryptographic service provider (CSP) so that you won't need any special hardware to use the code in this column. But it wouldn't be difficult to tweak the code to support a smart card, and smart cards are not at all expensive. You can get a smart card reader and a set of five cards for less than 100 bucks, so don't let the fear of hardware costs prevent you from considering them.

Key Containers and the Key Manager

Before you can encrypt data using a public key, you need to generate a key pair, and that means learning a bit about key containers. Each CAPI key container can have up to two public/private key pairs. One is used for "key exchange," which is a fancy term that means that the key pair is used to encrypt and decrypt secret keys like the AES key I'll use to protect sensitive data. The other set is used for creating digital signatures, which I won't be using in this column.

You can have as many key containers as you need, limited only by the storage space on the device you're using. While technically you only need one key pair to be able to encrypt and decrypt data, my encrypting agent assumes that it may be asked to use different keys to encrypt data. It therefore tags each bit of encrypted data with a key name so that the decrypting agent knows which key to use for decrypting. This simple key versioning scheme allows you to easily swap out keys from time to time, in case a key is compromised or as a best practice since the longer a single key is used, the weaker and more risky that key becomes.

Since key management is essential, the first program I've written is a Windows® Forms app called Key Manager (see Figure 2), which allows an administrator to enumerate, create, and delete key containers on a machine. (This program could be tweaked to manage key containers on a smart card.) Each time you create a new key container, Key Manager generates a new RSA public/private key pair in that container and displays the public key details in a format that's easy to cut and paste into the <appSettings> section of the config file of the encrypting agent. These settings include not only the public key itself, but also the name of the key container, which the other parts of my solution use for key versioning (as you'll see later).

Figure 2 Key Manager Application
Figure 2 Key Manager Application

Using Key Manager is easy. Just run it on the machine where you will host your decrypting agent. Select a key size and a name for your key container, like "MyApp-1" (you can use the hyphenated number for key versioning). Once you've created the key container for the decrypting agent, the private key is installed and ready to process data encrypted with the MyApp-1 key. By default, administrators on the machine as well as the user who created the key container will have permission to use the key. This is enforced by the access control list (ACL) on the key container file. Press the "Security..." button and Key Manager will launch the file properties dialog to allow you to configure this ACL to grant read access to the user account that your decrypting agent will use. Be sure to copy the public key configuration data that Key Manager provides and paste it into the encrypting agent's config file, which contains the key container name and the public key details. This is all the encrypting agent needs in order to encrypt data. By the way, if you have ever wondered where file system-based key containers are stored, take a look inside the following hidden directory:

\Documents and Settings\All Users\Application
Data\Microsoft\Crypto\RSA\MachineKeys

CAPI also supports per-user containers, but given that you can control the ACL on each machine-level container, I have kept things simple by using machine-level key containers.

Key Size

The size of the key you generate impacts the level of security and inversely, the performance you'll get from the cryptosystem. Also, the longer the key you choose, the longer it'll take to generate in Key Manager. In a Virtual PC environment running on a reasonably fast machine, a 4096-bit key takes several seconds to generate, while a 16384-bit key takes several minutes! At the moment, Key Manager creates these keys on the user interface thread, which was expedient for a sample app, but an obvious improvement would be to generate the key asynchronously so the UI doesn't freeze up. Regardless of how long it takes to create the key pair, what you should care about is the security and performance trade-off at run time, so I ran some tests to time encryption and decryption using the five key sizes that Key Manager supports (see Figure 3).

Figure 3 Key Size and Runtime Performance
Figure 3 Key Size and Runtime Performance

Pay attention to the units in Figure 3. Encryption is measured in milliseconds, but decryption is measured in seconds. Private key operations are not cheap in RSA. If performance is critical, you can purchase specialized hardware to offload encryption and/or decryption from the main CPU, dramatically increasing performance without sacrificing security. This will require an alternate CSP, which will require a bit of tweaking of my code, but that's a small price to pay for the performance benefit you'll see. From a security perspective, current recommendations are to stick with RSA keys that are at least 2048 bits long as an absolute minimum; 4096 bits seems to be the sweet spot in the performance/security curve today. And in the future, you can go even higher. As you can see from Figure 3, my own tests include keys up to 16384 bits long.

When I set out to write this application, I tried to use RSACryptoServiceProvider exclusively because it is the official managed interface for using CAPI key containers. But, it wasn't quite up to snuff for what I needed, so I performed a bit of P/Invoke magic to get to CAPI directly. This allowed me to enumerate key containers so you could quickly see all the key containers for the built-in RSA provider on your machine. The two important CAPI functions I rely upon are CryptAcquireContext and CryptGetProvParam. I've wrapped all these details in a managed class called KeyContainerManager, which you may find useful on its own.

The Encryptor and Decryptor

Besides KeyContainerManager, there are two other core classes in my solution, Encryptor and Decryptor, which you'll use in the encrypting and decrypting agents, respectively. While Decryptor needs the private key installed on the machine in a key container, Encryptor only needs the public key details provided via its constructor. An easy way to configure this is using the encrypting agent's configuration file. In my sample application, the encrypting agent is an ASP.NET Web application, so I simply configure the public key information in the Web.config file.

RSACryptoServiceProvider grew a couple of new methods in version 2.0 that allow you to easily access key information:

string ToXmlString(bool includePrivateParameters)
void FromXmlString(string xmlParameters)
In fact, Key Manager calls ToXmlString(false) to extract the public key parameters as a string of XML, which it then encodes and puts into an <appSettings> configuration entry that you can paste into the encrypting agent's config file. My Encryptor class must be constructed with two bits of information: a key name and the public key parameters as an XML string:
public Encryptor(string keyName,
string publicKeyXmlParams) {
    this.keyName = keyName;
    rsa.FromXmlString(publicKeyXmlParams);
}

The keyName argument is how I've implemented key versioning. Whatever string you provide here will be added as a prefix to the ciphertext produced when you encrypt data. The Decryptor will pick off this prefix and use it as the key container it binds to when decrypting the data. You'll want this to be easily configured in the encrypting agent, which is why Key Manager also includes this configuration item. In the sample application's Web.config file, you'll find an <appSettings> element that looks something like this:

<appSettings>
    <add key="keyName" value="TestKeyContainer"/>
    <add key="publicKeyData" value="&lt;RSAKey..."/>
</appSettings>

When the sample app needs to encrypt some data, it constructs a new Encryptor with these two pieces of data from Web.config and then calls Encrypt. When you want to change keys, you just need to update the Web.config file to indicate the new key name and public key data, and the application will immediately start using the new key with no downtime. And since each piece of encrypted data includes the key name, the decryptor always knows which key it should use for any given ciphertext. You just need to ensure that the machine hosting the decrypting agent has all the necessary keys installed.

The Encryptor.Encrypt method UTF8 encodes the string you pass in, then generates a random 256-bit AES secret key and a random 128-bit initialization vector (IV). It uses that to encrypt the data you pass in via the AES class called RijndaelManaged. (Rijndael is the name of the algorithm that won the contest to become the AES standard, so you can think of it as a synonym for AES. Its name is based on the surnames of the cryptographers who invented it and is pronounced something like "Rain-Doll".)

Figure 4 Data Format
Figure 4 Data Format

The Encryptor now encrypts the AES key with the RSA public key and forms the data structure shown in Figure 4, a length-prefixed, encrypted AES key followed by the IV and the encrypted data. This binary blob of data is base64-encoded into a string, to which is prepended the key name that identifies which RSA key should be used to decrypt the AES key.

If all of this sounds very complicated, you'll be happy to know that using the Encryptor class, you can simply pass in a string to be encrypted, and you'll get back a string that contains all the information needed for Decryptor.Decrypt to do its job (well, everything except the private key material, of course). This makes the interface to the Encryptor and Decryptor very easy to use:

string Encryptor.Encrypt(string plaintext);
string Decryptor.Decrypt(string ciphertext);

Of course, this isn't the only way to package the encrypted blob. You may find it more convenient to separate out the key name from the blob, or deal in byte arrays instead of strings, but those are easy changes to make once you understand the overall architecture of this solution. Another thing you should consider up front in a real application is format versioning. Don't wait until version 2 to put a version number on those encrypted blobs!

Decryptor.Decrypt unwinds the data structure by first peeling off the key container name and constructing an RSACryptoServiceProvider instance bound to that key container. It then decodes the base64-encoded payload and peels off the encrypted AES key, decrypting it using RSACryptoServiceProvider.Decrypt. That step is likely the most CPU-intensive part of the entire process if you're encrypting relatively small strings. At this point an AES key, represented by an instance of the RijndaelManaged class, is initialized with the decrypted secret key and the IV in the payload, and the encrypted data is decrypted. Finally, the decrypted byte array is decoded into a string via UTF8 and returned as the result of Decryptor.Decrypt.

The cipher mode my solution uses is cipher block chaining (CBC), a very popular scheme that happens to be the default in the .NET Framework. This cipher mode does not provide integrity protection on the payload. That is, when you decrypt the data, you aren't guaranteed to be notified if the ciphertext was tampered with. While integrity protection is not a goal of this particular solution, based on my experience a lot of people believe they get integrity protection automatically when they encrypt data. Often, when you decrypt CBC-encrypted ciphertext that has been modified, you'll get an exception when the padding on the end isn't what was expected, but this isn't a real integrity guarantee. It won't stop a determined attacker.

If you're interested in learning more about cipher modes, there's a very good article on the topic at Wikipedia: Block cipher modes of operation. And, of course, my favorite book for understanding how cryptography fits together is Practical Cryptography by Niels Ferguson and Bruce Schneier (Wiley, 2003).

The Sample Encrypting Agent (Take Orders)

I've pulled these classes together into a sample application you can build with Visual Studio® 2005, and I encourage you to try it out. You've already seen the first part, Key Manager, which is used to administer the application's keys. I've also constructed a sample encrypting agent called Take Orders, a Web application that accepts a few pieces of private data in a simple user interface: an e-mail address, credit card number, and expiration date. Take Orders pops this data into an XML string and uses an instance of Encryptor to encrypt the entire string.

Figure 5 Orders Waiting to be Processed
Figure 5 Orders Waiting to be Processed

Now I could have pushed each encrypted order into a message queue or database, but I wanted to make the application trivial for you to deploy, so I chose simply to write the data to a file and drop it into a directory of your choice. You'll need to configure a directory for Take Orders to use. Be sure to set the ACL on the directory so that the Web application has permission to write into it. The Web.config file for Take Orders includes an <appSettings> element called dataDir which you should point to this directory. Figure 5 shows what this directory looks like after a bit of use, as well as the innards of one of the files (you should recognize the key name at the front, separated from the base64-encoded blob by a pipe symbol). Note that each file is named with a GUID for uniqueness.

The Sample Decrypting Agent (Process Orders)

Figure 6 shows the Process Orders application, a Windows Forms app that displays each order based on the date the file was created. Selecting an order shows the contents of the encrypted file. The file can then be decrypted by pressing the Decrypt Order button, which uses an instance of the Decryptor class to unwind the order file and decrypt the message. Process Orders then displays the details of the order including the e-mail address, credit card number, and expiration date supplied by the user.

Figure 6 Process Orders Application
Figure 6 Process Orders Application

At this point you can press the Process Order button to send an e-mail to indicate that the order has been processed. It also deletes the encrypted order file from disk, making this system behave very much like a queue. If you use your own e-mail address and SMTP server, you'll actually get an e-mail. This little extra uses the newly refactored SmtpClient class that you can find in the System.Net.Mail namespace, new to the .NET Framework 2.0.

Moving Ahead

I encourage you to think about novel ways of dealing with sensitive data. By factoring your system into agents that encrypt and decrypt data independently of one another, it's possible to significantly limit the exposure of sensitive data. Of course, the threat of information disclosure is never going to go away completely. If your decrypting agent is compromised, so is your sensitive data (or at least data that's encrypted using keys that the decrypting agent currently holds). Keep in mind that a smart card makes it easier to completely remove private key material when the decrypting agent is not active, limiting exposure even further. So take this application, play with it, and think of ways to put the ideas behind it to use.

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is a cofounder of Pluralsight, a Microsoft .NET training provider. Keith is the author of Pluralsight's Applied .NET Security course, as well as several books, including The .NET Developer's Guide to Windows Security, which is available both in print and on the Web. Learn more at www.pluralsight.com/keith.


.NET Matters: Iterating NTFS Streams

.NET Matters
Iterating NTFS Streams
Stephen Toub

Code download available at: NETMatters0601.exe (131KB)

Q I read in your December 2005 column how to enumerate files using interop to access the file management functions in Kernel32.dll (see .NET Matters: BigInteger, GetFiles, and More). I don't need to enumerate files, but rather I need to enumerate the NTFS Alternate Data Streams within a particular file. Do you know any way I can accomplish this?

A First, nothing in the Microsoft® .NET Framework provides this functionality. If you want it, plain and simple you'll need to do some sort of interop, either directly or using a third-party library.

If you're using Windows Server™ 2003 or later, Kernel32.dll exposes counterparts to FindFirstFile and FindNextFile that provide the exact functionality you're looking for. FindFirstStreamW and FindNextStreamW allow you to find and enumerate all of the Alternate Data Streams within a particular file, retrieving information about each, including its name and its length. The code for using these functions from managed code is very similar to that which I showed in my December column, and is shown in Figure 1.

Figure 1 Using FindFirstStreamW and FindNextStreamW (Close Figure 1)
[SecurityPermission(SecurityAction.LinkDemand, UnmanagedCode=true)]
public sealed class SafeFindHandle : SafeHandleZeroOrMinusOneIsInvalid
{
    private SafeFindHandle() : base(true) { }

    protected override bool ReleaseHandle() {
        return FindClose(this.handle);
    }

    [DllImport("kernel32.dll")]
    [return: MarshalAs(UnmanagedType.Bool)]
    [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
    private static extern bool FindClose(IntPtr handle);
}

public class FileStreamSearcher
{
    private const int ERROR_HANDLE_EOF = 38;
    private enum StreamInfoLevels { FindStreamInfoStandard = 0 }

    [DllImport("kernel32.dll", ExactSpelling=true,
        CharSet=CharSet.Auto, SetLastError=true)]
    private static extern SafeFindHandle FindFirstStreamW(
        string lpFileName, StreamInfoLevels InfoLevel,
        [In, Out, MarshalAs(UnmanagedType.LPStruct)]
        WIN32_FIND_STREAM_DATA lpFindStreamData, uint dwFlags);

    [DllImport("kernel32.dll", ExactSpelling=true,
        CharSet=CharSet.Auto, SetLastError=true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static extern bool FindNextStreamW(
        SafeFindHandle hndFindFile,
        [In, Out, MarshalAs(UnmanagedType.LPStruct)]
        WIN32_FIND_STREAM_DATA lpFindStreamData);
    [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
    private class WIN32_FIND_STREAM_DATA
    {
        public long StreamSize;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst=296)]
        public string cStreamName;
    }

    public static IEnumerable<string> GetStreams(FileInfo file)
    {
        if (file == null) throw new ArgumentNullException("file");
        WIN32_FIND_STREAM_DATA findStreamData =
            new WIN32_FIND_STREAM_DATA();

        SafeFindHandle handle = FindFirstStreamW(
            file.FullName, StreamInfoLevels.FindStreamInfoStandard,
            findStreamData, 0);
        if (handle.IsInvalid) throw new Win32Exception();
        try
        {
            do { yield return findStreamData.cStreamName; }
            while (FindNextStreamW(handle, findStreamData));
            int lastError = Marshal.GetLastWin32Error();
            if (lastError != ERROR_HANDLE_EOF)
                throw new Win32Exception(lastError);
        }
        finally { handle.Dispose(); }
    }
}

You simply call FindFirstStreamW, passing to it the full path to the target file. The second parameter to FindFirstStreamW dictates the level of detail you want in the returned data; currently, there is only one level (FindStreamInfoStandard), which has a numerical value of 0. The third parameter to the function is a pointer to a WIN32_FIND_STREAM_DATA structure (technically, what the third parameter points to is dictated by the value of the second parameter detailing the information level, but as there's currently only one level, for all intents and purposes this is a WIN32_FIND_STREAM_DATA). I've declared that structure's managed counterpart as a class, and in the interop signature I've marked it to be marshaled as a pointer to a struct. The last parameter is reserved for future use and should be 0.

If a valid handle is returned from FindFirstStreamW, the WIN32_FIND_STREAM_DATA instance contains information about the stream found, and its cStreamName value can be yielded back to the caller as the first stream name available. FindNextStreamW accepts the handle returned from FindFirstStreamW and fills the supplied WIN32_FIND_STREAM_DATA with information about the next stream available, if it exists. FindNextStreamW returns true if another stream is available, or false if not.

As a result, I continually call FindNextStreamW and yield the resulting stream name until FindNextStreamW returns false. When that happens, I double check the last error value to make sure that the iteration stopped because FindNextStreamW ran out of streams, and not for some unexpected reason.

Unfortunately, if you're using Windows® XP or Windows 2000 Server, these functions aren't available to you, but there are a couple of alternatives. The first solution involves an undocumented function currently exported from Kernel32.dll, NTQueryInformationFile. However, undocumented functions are undocumented for a reason, and they can be changed or even removed at any time in the future. It's best not to use them. If you do want to use this function, search the Web and you'll find plenty of references and sample source code. But do so at your own risk.

Another solution, and one which I've demonstrated in Figure 2, relies on two functions exported from Kernel32.dll, and these are documented. As their names imply, BackupRead and BackupSeek are part of the Win32® API for backup support:

BOOL BackupRead(HANDLE hFile, LPBYTE lpBuffer,
                DWORD nNumberOfBytesToRead,
                LPDWORD lpNumberOfBytesRead, BOOL bAbort,
                BOOL bProcessSecurity, LPVOID* lpContext);

BOOL BackupSeek(HANDLE hFile, DWORD dwLowBytesToSeek,
                DWORD dwHighBytesToSeek, LPDWORD lpdwLowByteSeeked,
                LPDWORD lpdwHighByteSeeked, LPVOID* lpContext);

Figure 2 Using BackupRead and BackupSeek (Close Figure 2)
public enum StreamType
{
  Data = 1, ExternalData = 2, SecurityData = 3,
  AlternateData = 4, Link = 5, PropertyData = 6,
  ObjectID = 7, ReparseData = 8, SparseDock = 9
}

public struct StreamInfo
{
  public StreamInfo(string name, StreamType type, long size)
  {
    Name = name;
    Type = type;
    Size = size;
  }
  public readonly string Name;
  public readonly StreamType Type;
  public readonly long Size;
}

public class FileStreamSearcher
{
  [DllImport("kernel32.dll")]
  [return: MarshalAs(UnmanagedType.Bool)]
  private static extern bool BackupRead(
    SafeFileHandle hFile, IntPtr lpBuffer,
    uint nNumberOfBytesToRead, out uint lpNumberOfBytesRead,
    [MarshalAs(UnmanagedType.Bool)] bool bAbort,
    [MarshalAs(UnmanagedType.Bool)] bool bProcessSecurity,
    ref IntPtr lpContext);

  [DllImport("kernel32.dll")]
  [return: MarshalAs(UnmanagedType.Bool)]
  private static extern bool BackupSeek(SafeFileHandle hFile,
    uint dwLowBytesToSeek, uint dwHighBytesToSeek,
    out uint lpdwLowByteSeeked, out uint lpdwHighByteSeeked,
    ref IntPtr lpContext);

  public static IEnumerable<StreamInfo> GetStreams(FileInfo file)
  {
    const int bufferSize = 4096;
    using (FileStream fs = file.OpenRead())
    {
      IntPtr context = IntPtr.Zero;
      IntPtr buffer = Marshal.AllocHGlobal(bufferSize);
      try
      {
        while (true)
        {
          uint numRead;
          if (!BackupRead(fs.SafeFileHandle, buffer,
             (uint)Marshal.SizeOf(typeof(Win32StreamID)), out numRead,
             false, true, ref context)) throw new Win32Exception();

          if (numRead > 0)
          {
            Win32StreamID streamID = (Win32StreamID)
              Marshal.PtrToStructure(buffer, typeof(Win32StreamID));
            string name = null;
            if (streamID.dwStreamNameSize > 0)
            {
              if (!BackupRead(fs.SafeFileHandle, buffer,
                   (uint)Math.Min(bufferSize, streamID.dwStreamNameSize),
                   out numRead, false, true, ref context))
                       throw new Win32Exception();
              name = Marshal.PtrToStringUni(buffer, (int)numRead / 2);
            }

            yield return new StreamInfo(name,
              streamID.dwStreamId, streamID.Size);

            if (streamID.Size > 0)
            {
              uint lo, hi;
              BackupSeek(fs.SafeFileHandle, uint.MaxValue,
                int.MaxValue, out lo, out hi, ref context);
            }
          }
          else break;
        }
      }
      finally
      {
        Marshal.FreeHGlobal(buffer);
        uint numRead;
        if (!BackupRead(fs.SafeFileHandle, IntPtr.Zero, 0, out numRead,
             true, false, ref context)) throw new Win32Exception();
      }
    }
  }
}

The idea behind BackupRead is that it can be used to read data from a file into a buffer, which can then be written to the backup storage medium. However, BackupRead is also very handy for finding out information about each of the Alternate Data Streams that make up the target file. It processes all of the data in the file as a series of discrete byte streams (each Alternate Data Stream is one of these byte streams), and each of the streams is preceded by a WIN32_STREAM_ID structure. Thus, in order to enumerate all of the streams, you simply need to read through all of these WIN32_STREAM_ID structures from the beginning of each stream (this is where BackupSeek becomes very handy, as it can be used to jump from stream to stream without having to read through all of the data in the file).

To begin, you first need to create a managed counterpart for the unmanaged WIN32_STREAM_ID structure:

typedef struct _WIN32_STREAM_ID
{
    DWORD dwStreamId;
    DWORD dwStreamAttributes;
    LARGE_INTEGER Size;
    DWORD dwStreamNameSize;
    WCHAR cStreamName[ANYSIZE_ARRAY];
}
WIN32_STREAM_ID;

For the most part, this is like any other structure you'd marshal through P/Invoke. However, there are a few complications. First and foremost, WIN32_STREAM_ID is a variable-sized structure. Its last member, cStreamName, is an array with length ANYSIZE_ARRAY. While ANYSIZE_ARRAY is defined to be 1, cStreamName is just the address of the rest of the data in the structure after the previous four fields, which means that if the structure is allocated to be larger than sizeof (WIN32_STREAM_ID) bytes, that extra space will in effect be part of the cStreamName array. The previous field, dwStreamNameSize, specifies exactly how long the array is.

While this is great for Win32 development, it wreaks havoc on a marshaler that needs to copy this data from unmanaged memory to managed memory as part of the interop call to BackupRead. How does the marshaler know how big the WIN32_STREAM_ID structure actually is, given that it's variable sized? It doesn't.

The second problem has to do with packing and alignment. Ignoring cStreamName for a moment, consider the following possibility for your managed WIN32_STREAM_ID counterpart:

[StructLayout(LayoutKind.Sequential)]
public struct Win32StreamID
{
  public int dwStreamId;
  public int dwStreamAttributes;
  public long Size;
  public int dwStreamNameSize;
}

An Int32 is 4 bytes in size and an Int64 is 8 bytes. Thus, you would expect this struct to be 20 bytes. However, if you run the following code, you'll find that both values are 24, not 20:

int size1 = Marshal.SizeOf(typeof(Win32StreamID));
int size2 = sizeof(Win32StreamID); // in an unsafe context

The issue is that the compiler wants to make sure that the values within these structures are always aligned on the proper boundary. Four-byte values should be at addresses divisible by 4, 8-byte values should be at boundaries divisible by 8, and so on. Now imagine what would happen if you were to create an array of Win32StreamID structures. All of the fields in the first instance of the array would be properly aligned. For example, since the Size field follows two 32-bit integers, it would be 8 bytes from the start of the array, perfect for an 8-byte value. However, if the structure were 20-bytes in size, the second instance in the array would not have all of its members properly aligned. The integer values would all be fine, but the long value would be 28 bytes from the start of the array, a value not evenly divisible by 8. To fix this, the compiler pads the structure to a size of 24, such that all of the fields will always be properly aligned (assuming the array itself is).

If the compiler's doing the right thing, you might be wondering why I'm concerned about this. You'll see why if you look at the code in Figure 2. In order to get around the first marshaling issue I described, I do in fact leave the cStreamName out of the Win32StreamID structure. I use BackupRead to read in enough bytes to fill my Win32StreamID structure, and then I examine the structure's dwStreamNameSize field. Now that I know how long the name is, I can use BackupRead again to read in the string's value from the file. That's all well and dandy, but if Marshal.SizeOf returns 24 for my Win32StreamID structure instead of 20, I'll be attempting to read too much data.

To avoid this, I need to make sure that the size of Win32StreamID is in fact 20 and not 24. This can be accomplished in two different ways using fields on the StructLayoutAttribute that adorns the structure. The first is to use the Size field, which dictates to the runtime exactly how big the structure should be:

[StructLayout(LayoutKind.Sequential, Size = 20)]

The second option is to use the Pack field. Pack indicates the packing size that should be used when the LayoutKind.Sequential value is specified and controls the alignment of the fields within the structure. The default packing size for a managed structure is 8. If I change that to 4, I get the 20-byte structure I'm looking for (and as I'm not actually using this in an array, I don't lose efficiency or stability that might result from such a packing change):

[StructLayout(LayoutKind.Sequential, Pack=4)]
public struct Win32StreamID
{
  public StreamType dwStreamId;
  public int dwStreamAttributes;
  public long Size;
  public int dwStreamNameSize;
  // WCHAR cStreamName[1];
}

With this code in place, I can now enumerate all of the streams in a file, as shown here:

static void Main(string[] args) {
    foreach (string path in args) {
        Console.WriteLine(path + ":");
        foreach (StreamInfo stream in
            FileStreamSearcher.GetStreams(new FileInfo(path)))
        {
            Console.WriteLine("\t{0}\t{1}\t{2}",
               stream.Name != null ? stream.Name : "(unnamed)",
               stream.Type, stream.Size);
        }
     }
}

You'll notice that this version of FileStreamSearcher returns more information than the version that uses FindFirstStreamW and FindNextStreamW. BackupRead can provide data on more than just the primary stream and Alternate Data Streams, also operating on streams containing security information, reparse data, and more. If you only want to see the Alternate Data Streams, you can filter based on the StreamInfo's Type property, which will be StreamType.AlternateData for Alternate Data Streams.

To test this code, you can create a file that has Alternate Data Streams using the echo command at the command prompt:

> echo ".NET Matters" > C:\test.txt
> echo "MSDN Magazine" > C:\test.txt:magStream
> StreamEnumerator.exe C:\test.txt
test.txt:
        (unnamed)               SecurityData    164
        (unnamed)               Data            17
        :magStream:$DATA        AlternateData   18
> type C:\test.txt
".NET Matters"
> more < C:\test.txt:magStream
"MSDN Magazine"

So, now you're able to retrieve the names of all Alternate Data Streams stored in a file. Great. But what if you want to actually manipulate the data in one of those streams? Unfortunately, if you attempt to pass a path for an Alternate Data Stream to one of the FileStream constructors, a NotSupportedException will be thrown: "The given path's format is not supported."

To get around this, you can bypass FileStream's path canonicalization checks by directly accessing the CreateFile function exposed from kernel32.dll (see Figure 3). I've used a P/Invoke for the CreateFile function to open and retrieve a SafeFileHandle for the specified path, without performing any of the managed permission checks on the path, so it can include Alternate Data Stream identifiers. This SafeFileHandle is then used to create a new managed FileStream, providing the required access. With that in place, it's easy to manipulate the contents of an Alternate Data Stream using the System.IO namespace's functionality. The following example reads and prints out the contents of the C:\test.txt:magStream created in the previous example:

string path = @"C:\test.txt:magStream";
using (StreamReader reader = new StreamReader(CreateFileStream(
    path, FileAccess.Read, FileMode.Open, FileShare.Read)))
{
    Console.WriteLine(reader.ReadToEnd());
}

Figure 3 Using P/Invoke for CreateFile (Close Figure 3)
private static FileStream CreateFileStream(string path,
    FileAccess access, FileMode mode, FileShare share)
{
    if (mode == FileMode.Append) mode = FileMode.OpenOrCreate;
        SafeFileHandle handle = CreateFile(path, access, share,
        IntPtr.Zero, mode, 0, IntPtr.Zero);
    if (handle.IsInvalid)
        throw new IOException("Could not open file stream.", new Win32Exception());
    return new FileStream(handle, access);
}

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
private static extern SafeFileHandle CreateFile(string lpFileName,
    FileAccess dwDesiredAccess, FileShare dwShareMode,
    IntPtr lpSecurityAttributes, FileMode dwCreationDisposition,
    int dwFlagsAndAttributes, IntPtr hTemplateFile);

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


C++ at Work: Installing a Hook, Strings in Managed C++, and More

C++ At Work
Installing a Hook, Strings in Managed C++, and More
Paul DiLascia

Code download available at: CAtWork0601.exe (146KB)

Q I want to call SetWindowsHookEx to set a WH_CBT hook to do something. But I know MFC also installs this hook. This means in one thread WH_CBT is installed twice. Can I do this?

Ken Dang

A The short answer is yes. You can install several hooks of the same type as long as you follow the proper procedure. Windows® hooks are designed to operate in a daisy-chain fashion similar to subclassing. To install a hook, you call SetWindowsHookEx with the type of hook and a pointer to your hook procedure. SetWindowsHookEx returns a handle to the old hook:

HHOOK hOldHook; // global
...
hOldHook = SetWindowsHookEx(WH_CBT, MyCbtProc, ...);

Now Windows calls your hook procedure whenever one of the hook events happens. When your procedure is done handling the event, it's supposed to call the next hook by calling—what else?—CallNextHookEx:

LRESULT CALLBACK
MyCbtProc(int code, ...)
{
   if (code==/* whatever */) {
      // do something
   }
   return CallNextHookEx(hOldHook, code, ...);
}

Of course, there's no one forcing you to call CallNextHookEx, but if you don't, your application is likely to break. In the case of the computer-based training (CBT) hook, MFC uses it to trap window creation. Windows calls the CBT hook with HCBT_CREATEWND whenever it creates a window. MFC handles HCBT_CREATEWND by subclassing the window and attaching it to its CWnd object. The details are a bit complex, but the simplified version goes something like this:

// simplified from wincore.cpp
LRESULT _AfxCbtFilterHook(int code, WPARAM wp, ...)
{
   if (code==HCBT_CREATEWND) {
      CWnd* pWndInit = pThreadState->m_pWndInit;
      HWND hWnd = (HWND)wp;
      pWndInit->Attach(hWnd);
      SetWindowLongPtr(hWnd, GWLP_WNDPROC, &AfxWndProcafxWndProc);
   }
   return CallNextHookEx(...);
}

I've omitted the grungies to highlight what's important. MFC attaches the window object to its HWND and subclasses it by installing AfxWndProc. This is how MFC connects C++ window objects to their HWNDs. AfxWndProc is the procedure that routes WM_XXX messages (via a very circuitous path) to your message map handlers.

When Windows calls the CBT hook, it passes the HWND as WPARAM. But how does MFC know which CWnd-derived object to attach? By setting a global. To create a window, you must call CWnd::Create or CWnd::CreateEx. The former calls the latter, so all paths go through CWnd::CreateEx. Before creating the window, CWnd::CreateEx installs the CBT hook and sets a global. In code it looks something like this:

// simplified from wincore.cpp
BOOL CWnd::CreateEx(...)
{
   AfxHookWindowCreate(this);
   ::CreateWindowEx(...);
   AfxUnhookWindowCreate();
   return TRUE;
}

AfxHookWindowCreate installs the CBT hook _AfxCbtFilterHook. It also saves a pointer to the window object in the thread's state as pThreadState->m_pWndInit:

void AFXAPI AfxHookWindowCreate(CWnd* pWnd)
{
   _AFX_THREAD_STATE* pThreadState = _afxThreadState.GetData();
   pThreadState->m_hHookOldCbtFilter = ::SetWindowsHookEx(
      WH_CBT, _AfxCbtFilterHook, NULL, ::GetCurrentThreadId());
   pThreadState->m_pWndInit = pWnd;
}

Think of the thread state as a place to hold thread-wide globals. So now the action goes like this. Your app calls CWnd::Create or CWnd::CreateEx. CWnd::CreateEx installs a CBT hook, sets a global pointer to the CWnd being created, and finally calls ::CreateWindowEx to actually create the window. After it creates the window—but before sending any messages like WM_CREATE or WM_GETMINMAXINFO—Windows calls the CBT hook with HCBT_CREATEWND. Then _AfxCbtFilterHook gets control and subclasses the window and connects it to its CWnd. MFC knows which CWnd to use because it previously stored the CWnd pointer in pThreadState->m_pWndInit. Pretty clever, eh?

After _AfxCbtFilterHook returns control to Windows, Windows sends WM_GETMINMAXINFO and WM_CREATE messages to your window, which MFC now processes in the normal way, by routing control to your OnGetMinMaxInfo and OnCreate handlers. This is only possible because the HWND is now attached to its CWnd object. When ::CreateWindowEx returns control to CWnd::CreateEx, CWnd::CreateEx calls AfxUnhookWindowCreate to remove the CBT hook and set pThreadState->m_pWndInit = NULL. The only reason for the CBT hook is to trap window creation so MFC can connect the CWnd to its HWND. The hook exists only for the duration of the call to ::CreateWindowEx.

Astute readers may wonder why MFC goes through all that trouble—why not just attach and subclass the window in CWnd::CreateEx? That would work fine, except for one problem. The CWnd object would miss any messages sent from ::CreateWindowEx—such as WM_GETMINMAXINFO, WM_NCCREATE and WM_CREATE. The problem is ::CreateWindow Ex doesn't let you specify a window proc when you create your window. You have to subclass it afterwards. But by then, several messages have already gone by. In order to handle all the messages, including the early creation messages, MFC has to connect the window object before the messages are sent. The only way to do that is using a CBT hook, because Windows calls the CBT hook just after it creates the window, but before sending it any messages. So the whole point and purpose of the CBT hook is to trap window creation in order to connect CWnd objects to their HWNDs before the window receives any messages.

That's way more than you need to know in answer to your question, but it's good to understand when, where, and why MFC uses its CBT hook. I also wanted to show you how MFC uses the thread global m_pWndInit to pass the CWnd to the hook function. Often you'll need to do something similar. SetWindowsHookEx has no void* param you can pass along to your hook procedure. If your hook proc needs information in the form of data, the only way to pass it is through a global. In most situations a normal static will do; you don't need a thread-specific global unless your data is thread-specific. MFC uses the thread state because it maintains a separate window map for each thread. The window map holds all the CWnds for a thread; it's what links each CWnd with its HWND.

But as far as multiple hooks are concerned, you can install as many as you like as long as you remember to call ::CallNextHook. You won't interfere with MFC.

Q I'm in the process of converting an existing C++ class library to the Managed Extensions so I can expose it to my .NET Framework-based clients. Some of my code calls API functions that require the HINSTANCE of the current running module. I don't want to use the HINSTANCE of my DLL; I want the caller to supply the HINSTANCE of the EXE calling my DLL. I can declare the HINSTANCE parameter as an IntPtr, but how do my .NET-based clients get the app's HINSTANCE to pass to my function? For example, how would they do it in C#?

Hunter Gerson

A Good question! This one stumped me for about 15 minutes. In MFC you can call AfxGetInstanceHandle(). MFC stores the HINSTANCE in its Application object, in CWinApp::m_hInstance. So if you're using the Microsoft® .NET Framework, you might think that you should take a look in the Application object for Application.HInstance or an Application property or some such thing. Why not? Because it's not there!

If you search the Framework documentation for "hinstance," you'll discover there's a method Marshal.GetHINSTANCE. Well, that sounds promising. This static Marshal method requires the module whose HINSTANCE you want to obtain:

// In C#
Module m;
...
IntPtr h = Marshal.GetHINSTANCE(m);

Now you know how to get the HINSTANCE—but where do you get the module? Again, you might think to look in the Application class for something like Application.GetModule. Or perhaps Application is derived from module, that would make sense. Alas, neither is true. Well, maybe there's a Module property. Nope. Well, not quite. There is a Module property, but it's not a property of the Application, it's a Type property. In the .NET Framework, every object has a Type and every Type has a Module. Type.Module represents whichever module implements the type. So to get the HINSTANCE of the calling module, your clients can write:

Type t = myObj.GetType();
Module m = t.Module;
IntPtr h = Marshal.GetHINSTANCE(m);

You can also use typeof (--typeof in C++)to get the type without an object instance, as in typeof(MyApp). Just tell your customers to make sure they use a type that's implemented in the calling module. If you use some other type—for example, one of the Framework types like String—you'll get the wrong module.

Figure 1 shows a simple C# console app, ShowModule, that illustrates this. Figure 2 shows it running. ShowModule displays module information including the HINSTANCE for two types: the MyApp class defined in the application itself and the String[] (String array) type defined in mscorlib.

Figure 1 Getting an HINSTANCE in C# (Close Figure 1)
using System;
using System.Reflection;
using System.Runtime.InteropServices;

class MyApp {
   // main entry point
   static void Main(string[] args) {
      ShowModule(typeof(MyApp));   // show module for MyApp (me)
      ShowModule(args.GetType());  // show module for String[] (CLR)
   }

   // Helper to display module info and HINSTANCE given Type.
   static void ShowModule(Type t)
   {
      Module m = t.Module;
      Console.WriteLine("Module info for type {0}:", t);
      Console.WriteLine(" Name = {0}", m.Name);
      Console.WriteLine(" FullyQualifiedName = {0}",
         m.FullyQualifiedName);
      Console.WriteLine(" HINSTANCE = 0x{0:x}\n",
         (int)Marshal.GetHINSTANCE(m));
   }
}
Figure 2 Module Information
Figure 2 Module Information

Q How do I convert an MFC CString to a String in managed C++? I have the following C++ function:

int ErrMsg::ErrorMessage(CString& msg) const
{
   msg.LoadString(m_nErrId);
   msg += _T("::Error");
   return -1;
}
How do I rewrite this using managed C++ and replacing CString in the parameter list with String? I can't figure out how to declare the parameters, what to do with the const, and how to load a managed String from a resource file. I read that Strings can't be modified because they're immutable, but I want to modify the string passed.

Sumit Prakash

A There are several things going on here, so let's tackle them one at a time. First, the const declaration. There's no notion of const methods in the .NET Framework, so you can forget about that. Sorry, that's just the way it is.

Next, how to declare your new function. You know you want to change CString to String, but what's the exact syntax? Your function modifies the CString passed, which is why you used a reference. In .NET, it's true that Strings are immutable. You can't modify the contents of a String. All the methods that seem to modify a String actually return a new String. For example:

String* str = S"hello";
str = str->ToUpper();

String::ToUpper returns a new String, which you can assign to str. If you want to modify the String itself, you need a different class, StringBuilder. But you don't need StringBuilder here because you're not actually modifying the String, you're modifying the variable that references it. In order to see this, consider what your function would look like in C#:

int ErrorMessage(ref string msg)
{
   msg = ...;
   return -1;
}

The msg parameter is declared as ref, which means that when ErrorMessage alters msg, it alters the variable passed, not the String object itself, as shown here:

string str = "";
err.ErrorMessage(ref str);

Now instead of referencing the empty string, str references whatever string ErrorMessage set it to. So in C#, you'd use a ref parameter. But, there's no ref keyword in C++, nor is there any managed __ref keyword. C++ doesn't need one, because it already has references! And the compiler is smart enough to make them work with managed code. All you have to do is remember that in C++, managed objects are always pointers or handles. Just replace CString with String* (or String^ if you're using Visual Studio® 2005 with C++/CLI). The new declaration looks like this:

int ErrMsg::ErrorMessage(String*& msg){
   msg = "foo";
   return -1;
}

That is, the new function takes a reference to a (managed) String pointer. If you want to be really explicit, you can even use __gc, as in ErrorMessage(String __gc * __gc & msg). In the real world, you don't need __gc because the compiler knows String is a managed class. If you're using C++/CLI, you'd use a tracking reference-to-handle as in: ErrorMessage(String^% msg). Amazingly, it all makes perfect sense. But that's not the end of the story, because there's yet another way you can declare ErrorMessage. You can use a pointer-to-pointer, like so:

int ErrMsg::ErrorMessage(String** msg){
   *msg = "foo";
   return -1;
}

Even in C++, the difference between a reference and a pointer is slight. The main difference is that a reference must always be initialized and can't be NULL. Otherwise the difference is mainly syntactic—whether you use . or -> to dereference the object. Internally, references are implemented as pointers. In .NET, there are no pointers. Everything is a reference. Or everything is a pointer, if you peer under the hood.So whether you use a reference-to-pointer or pointer-to-pointer, either way your String parameter looks to the outside Framework world like a ref parameter. I wrote a C# program, RefTest, that shows this (see Figure 3 and Figure 4).

Figure 4 C# Consumer of C++ Library (Close Figure 4)
using System;
using MyLib; // C++ lib

class MyApp {
   // main entry point
   static int Main(string[] args) {
      ErrMsg em = new ErrMsg();
      String str = null;

      // Test declaration 1: reference-to-pointer
      em.Message1(ref str);
      Console.WriteLine("Message1: str = {0}", str);

      // Test declaration 2: pointer-to-pointer
      em.Message2(ref str);
      Console.WriteLine("Message2: str = {0}", str);

      // Test old-style resource strings: There are only 2.
      for (int i=1; i<=3; i++) {
         em.GetString(ref str, i);
         Console.WriteLine("Resource String #{0} = {1}", i, str);
      }

      return 0;
   }
}
Figure 3 Library that Uses Pass-By-Reference Parameters (Close Figure 3)

Mylib.cpp
#using <mscorlib.dll>
#include <atlstr.h> // CString

using namespace System;
using namespace System::Runtime::InteropServices;

namespace MyLib {

public __gc class ErrMsg {
public:

   // reference-to-pointer
   int Message1(String*& str)
   {
      str = "Hello, world #1";
      return 0;
   }

   // pointer-to-pointer
   int Message2(String** str)
   {
      *str = "Hello, world #2";
      return 0;
   }

   // Load string from DLL resource file. Note explicit module handle.
   int GetString(String*& str, int id)
   {
      CString s;
      if (s.LoadString(::GetModuleHandle(_T("MyLib.dll")), id)) {
         str = s;  // Copy to caller's string. Compiler knows what to do.
         return 0;
      }
      // Note: need to box id sinve String::Format expects an Object.
      str = String::Format("Error: String {0} not found.", __box(id));
      return -1;
   }
};

}
MyLib.rc
// Resource file has two strings
STRINGTABLE
BEGIN
   1 "Resource string #1"
   2 "Resource string #2"
END

RefTest uses a library class ErrMsg written in C++. ErrMsg is a managed class with two methods, Message1 and Message2, which set their String parameter to a message "Hello, world #1" and "Hello, world #2", respectively. One uses String** and the other uses String*&. Either way, C# callers must use the ref keyword when calling either Message1 or Message2.

The "ref" is required in both cases. If you remove it, you'll get "error CS1503: Argument '1': cannot convert from 'string' to 'ref string'." Note that it's legal to call Message1 with str=null. To your C++ function, str isn't NULL, it's a reference to null. You should be aware of this if your function accesses the String passed. For example:

int ErrMsg::Message1(String*& str)
{
   int len = str->Length;
   ...
}
This compiles fine, but it throws an exception if the caller passes str=null. You should rewrite your code to gracefully handle the case of str=null, as shown in the following:
int ErrMsg::Message2(String*& str)
{
   if (str==NULL)
      return -1;
   ...
}

So, which should you use—pointer or reference? It doesn't much matter. Personally, I prefer the reference (&) because it mirrors ref, looks cleaner, and requires less typing to dereference the object.

So much for the declaration. Now to the final point. How do you load a resource string? As you discovered, there's no LoadString method in the String class. That's because the .NET Framework doesn't do resources the Windows way. The .NET Framework takes an entirely different approach, one I described in my November 2002 MSDN®Magazine article (see .NET GUI Bliss: Streamline Your Code and Simplify Localization Using an XML-Based GUI Language Parser) as "infinitely flexible, but tedious for small tasks."

The .NET way is to use satellite assemblies of text or XML resource (.resx) files. In .NET, there are two kinds of resources: strings and objects. For strings, all you have to do is create a .txt file with name=value pairs and run resgen.exe. Your app then calls ResourceManager.GetString to fetch the string. For everything else, you have to write a program that serializes the object to a .resx file, then call ResourceManger.GetObject to load it at run time. For details, read the documentation or my GUI article. In it, I showed how to write a FileRes class and FileResGen program that vastly simplify the process for file-based resources like image files (.BMP, .GIF, .JPG, and so on).

The advantage of doing resources the .NET way is they're easier to localize. Just translate the text/resource and store it in a subfolder with the same name as the locale—en for English, fr for French, or kv for Komi. The Framework automatically loads the proper assembly based on the user's CultureInfo.CurrentUICulture (MFC does something similar with satellite DLLs based on GetSystemDefaultUILanguage.) So if you want to be a full-fledged .NET citizen, you should probably rewrite your library to use satellites and ResourceManager. But if localization isn't important (perhaps you're loading internal error messages not seen by users) or time is short, you can still load old-style string resources from your .RC file. But you have to call ::LoadString or use CString internally to load the string, then copy it to the caller's String object. That's the wonderful thing about writing in C++! You can call Windows directly without explicitly using P/Invoke, and use your favorite ATL/MFC classes as usual. The only trick here is that since you want to load the string from your DLL, not the calling application, you must explicitly tell LoadString to use your DLL's HINSTANCE:

CString s;
HINSTANCE h = ::GetModuleHandle(_T("MyLib.dll")); // use DLL's handle
s.LoadString(h, id);

Figure 3 shows the full working code. Figure 5 shows the results if you compile and run RefTest. As always, you can download RefTest and all the other code from the MSDN Magazine Web site.

Figure 5 RefTest in Action
Figure 5 RefTest in Action

Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: Transactions for Memory

{ End Bracket }
Transactions for Memory
Joe Duffy

Injecting parallelism into your app is easy—managed code offers things like explicit threading and a thread pool for that. Ensuring your code remains correct when run in parallel, on the other hand, is not quite so simple. And if you manage to make it correct, tuning it to effectively exploit parallel hardware is even harder. Let's face it: concurrency is not a walk in the park.

History is often quick to offer unique insight into new problems. Concurrency is no different. In fact, database vendors have faced and solved similar problems for years. When a client app generates some SQL, the client programmer seldom thinks in terms of managing concurrent access to individual tables, rows, and columns. Database server software takes care of correctness through the use of transactions, ensuring atomicity, consistency, isolation, and durability (ACID). Databases also manage parallel scale-out through intelligent data- and query-parallelization algorithms, resulting in efficient utilization across many processors, cores, and in some cases over a cluster of machines.

This sounds promising. What might transactions for memory bring to our concurrent software?

Locks Are Implementation Details If you take a close look at lock-based programming, a reasonable conclusion is that locking is just a tool, not a programming model:

lock (...) {
    try {
        // (1) 'atomic' updates
    } catch {
        // (2) ensure that invariants hold
        throw;
    }
}

The intent of this code is that updates to data inside (1) are atomic and consistent, provided that those accessing the data follow the locking protocol. As written, this is rather clumsy. It requires that the programmer hand-author the compensation code in (2) to roll back inconsistencies in the face of failure. But what if she could write the following instead?

atomic { /* (1) atomic updates */ }
Thread-safety, isolation, and compensation are implied.

How Might Atomic Blocks be Implemented? One approach is transactional memory. The block runs inside a transaction and all memory reads and writes are isolated within a thread-private log. When the atomic block completes, conflicts are detected and, if none are found, the writes are committed to memory and the effects become visible to the entire program. Otherwise, the transaction log is discarded and the block is reexecuted. If an exception escapes the atomic block unhandled, all updates made by the block are automatically thrown away. It all just works, much like working with a database as I described earlier.

Declarative Concurrency Permits the Runtime to Optimize Serializing access to data with locks involves trading correctness for parallelism and scalability. Clever locking schemes can be used to reap the benefits of both, but require complex code and engineering. If you're crafting a highly reusable, scientific-quality library, this amount of work might be par for the course. But if you're writing an application, the tax is simply too high.

Locks employ a technique known in database circles as pessimistic concurrency, while transactional memory prefers optimistic concurrency. The former prohibits concurrent access to data structures while the transaction executes, while the latter detects (and sometimes even tolerates) problematic conflicts at commit time. If two concurrent tasks work on the same data structure and do not conflict, for example, serialization is not necessary; yet a naïve lock-based algorithm might impose it. Permitting the transaction manager to choose an appropriate execution strategy using runtime heuristics can lead to highly scalable code.

Moreover, in-memory transactions can eliminate hard-to-debug "heisenbugs" such as deadlocks, which often result in program hangs and are hard to provoke, and priority inversion and lock convoys, both of which can lead to fairness and scalability problems. Because the transaction manager controls locking details, it can ensure fair and deadlock-free commit protocols.

An Eye Towards the Future There are several interesting programming models that seek to simplify mainstream concurrent programming, with transactional memory at the front of the pack. Whether it will win out remains an open question, but one thing is certain: with ever-increasing parallelism coming to the desktop- and server-machines near you, easier concurrency programming models will evolve alongside. In fact, some might say we must do for concurrent programming what garbage collection did for memory management. Transactions for memory is a step in that direction. Information from MSR on transactional memory may be found at research.microsoft.com/~tharris.

Joe Duffy is a Program Manager on the CLR Team at Microsoft, where he focuses on concurrency programming models for managed code. Keep an eye out for Joe's upcoming book, Professional .NET Framework 2.0 (Wrox, 2006).