 MSDN Magazine February 2008

MSDN Magazine February 2008

Roll Your Own: Create a Language Compiler for the .NET Framework

Joel Pobar presents an introduction to how compilers work and how you can write your own compiler to target the .NET Framework.

Joel Pobar WinUnit: Simplified Unit Testing for Native C++ Applications

Maria Blees introduces WinUnit, a handy new unit testing tool for native C++ projects.

Maria Blees Silverlight: Create a Custom Expression Encoder Publishing Plug-In

Laurence Moroney demonstrates how to employ the Expression Encoder plug-in APIs by building and debugging a simple FTP upload plug-in.

Laurence Moroney PIAB And WCF: Integrating the Policy Injection Application Block with WCF Services

Using the PIAB you can enhance WCF services with policies such as validation, performance monitoring, authorization and caching without having to change a line of code.

Hugh Ang and David San Filippo WF How-To: Building State Machines with Windows Workflow Foundation

State Machine workflows represent a different way of visualizing program logic. Rather than flowing from activity to activity like sequential workflows, State Machines transition from state to state. Learn about uses for State machines and see how to design and build a State Machine workflow with Windows Workflow Foundation.

Keith Pijanowski Editor's Note: Overcoming Language Barriers with Elegant Design

This month Howard Dierking explains why the choice of a programming language can be a personal one.

Howard Dierking Toolbox: Powerful Text Editing, Web Testing in .NET, Extended Unit Testing, and More

Powerful Text Editing, Web Testing in .NET, Extended Unit Testing, and More

James Avery CLR Inside Out: Dynamic Languages and Silverlight

DLRConsole gives you an interactive environment for playing with dynamic languages in Silverlight.

Jimmy Schementi Basic Instincts: XML Programming in Visual Basic 9.0

With Visual Basic 9.0, working with XML gets much easier for developers. Here's a look at some of the new features, including LINQ support, XML literals, embedded expressions, XML properties, and XML Schema IntelliSense.

Jonathan Aneja Data Points: Designing an Entity Data Model

With the Entity Framework, developers are given more flexibility by being able to design around a conceptual model rather than a relational data model. To get started, you need to know how to design an Entity Data Model. John Papa walks you through that process.

John Papa Cutting Edge: Customize Controls with AJAX Extenders, Part 2

This month Dino looks at AJAX control extenders again, adding more advanced features including masked editing and autocompletion.

Dino Esposito Office Space: Security Programming in SharePoint 2007

This month Ted Pattison presents an overview of programming security and permissions for Windows SharePoint Services 3.0.

Ted Pattison Test Run: The Microsoft UI Automation Library

James McCaffrey shows you how to get started with UI test automation using the new Microsoft UI Automation library.

Dr. James McCaffrey Foundations: What's New for WCF in Visual Studio 2008

Visual Studio 2008 and the .NET Framework 3.5 provide new tools and support that extends Windows Communication Foundation (WCF). Visual Studio 2008 also automates a number of manual WCF tasks for the developer as well.

Juval Lowy .NET Matters: Stream Pipeline

This month Stephen Toub explains how to make the most of dual processors when running encryption and compression tasks.

Stephen Toub { End Bracket }: Finding the Best of the Best

Passion, intelligence, and the love of learning exemplify the ideal Microsoft candidate. Howard Dierking explores these and other recruiting philosophies.

Howard Dierking


Roll Your Own: Create a Language Compiler for the .NET Framework

Roll Your Own

Create a Language Compiler for the .NET Framework

Joel Pobar
This article discusses:
  • Language definition
  • The phases of a compiler
  • The CLR abstract stack
  • Tools for getting your IL right
This article uses the following technologies:
.NET Framework
Code download available at: CompilerWriting2008_02.exe (158KB)
Contents
Language Definition
High-Level Architecture
The Scanner
The Parser
Targeting the .NET Framework
Tools for Getting Your IL Right
The Code Generator
Wrapping Up ... Almost
Dynamic Method Invocation
Using LCG to Perform Quick Late-Binding
The Dynamic Language Runtime

C
ompiler hackers are celebrities in the world of computer science. I've seen Anders Hejlsberg deliver a presentation at the Professional Developers Conference and then walk off stage to a herd of men and women asking him to sign books and pose for photographs. There's a certain intellectual mystique about individuals who dedicate their time to learning and understanding the ins and outs of lambda expressions, type systems, and assembly languages. Now, you too can share some of this glory by writing your own compiler for the Microsoft® .NET Framework.

There are hundreds of compilers for dozens of languages that target the .NET Framework. The .NET CLR wrangles these languages into the same sandpit to play and interact together peacefully. A ninja developer can take advantage of this when building large software systems, adding a bit of C# and a dabble of Python. Sure, these developers are impressive, but they don't compare to the true masters—the compiler hackers, for it is they who have a deep understanding of virtual machines, language design, and the nuts and bolts of these languages and compilers.

In this article, I will walk you through the code for a compiler written in C# (aptly called the "Good for Nothing" compiler), and along the way I will introduce you to the high-level architecture, theory, and .NET Framework APIs that are required to build your own .NET compiler. I will start with a language definition, explore compiler architecture, and then walk you through the code generation subsystem that spits out a .NET assembly. The goal is for you to understand the foundations of compiler development and get a firm, high-level understanding of how languages target the CLR efficiently. I won't be developing the equivalent of a C# 4.0 or an IronRuby, but this discussion will still offer enough meat to ignite your passion for the art of compiler development.

Language Definition

Software languages begin with a specific purpose. This purpose can be anything from expressiveness (such as Visual Basic®), to productivity (such as Python, which aims to get the most out of every line of code), to specialization (such as Verilog, which is a hardware description language used by processor manufacturers), to simply satisfying the author's personal preferences. (The creator of Boo, for instance, likes the .NET Framework but is not happy with any of the available languages.)

Once you've stated the purpose, you can design the language—think of this as the blueprint for the language. Computer languages must be very precise so that the programmer can accurately express exactly what is required and so that the compiler can accurately understand and generate executable code for exactly what's expressed. The blueprint of a language must be specified to remove ambiguity during a compiler's implementation. For this, you use a metasyntax, which is a syntax used to describe the syntax of languages. There are quite a few metasyntaxes around, so you can choose one according to your personal taste. I will specify the Good for Nothing language using a metasyntax called EBNF (Extended Backus-Naur Form).

It's worth mentioning that EBNF has very reputable roots: it's linked to John Backus, winner of the Turing Award and lead developer on FORTRAN. A deep discussion of EBNF is beyond the scope of the article, but I can explain the basic concepts.

The language definition for Good for Nothing is shown in Figure 1. According to my language definition, Statement (stmt) can be variable declarations, assignments, for loops, reading of integers from the command line, or printing to the screen—and they can be specified many times, separated by semicolons. Expressions (expr) can be strings, integers, arithmetic expressions, or identifiers. Identifiers (ident) can be named using an alphabetic character as the first letter, followed by characters or numbers. And so on. Quite simply, I've defined a language syntax that provides for basic arithmetic capabilities, a small type system, and simple, console-based user interaction.

Figure 1 Good for Nothing Language Definition (Close Figure 1)
<stmt> := var <ident> = <expr>
        | <ident> = <expr>
        | for <ident> = <expr> to <expr> do <stmt> end
        | read_int <ident>
        | print <expr>
        | <stmt> ; <stmt>

<expr> := <string>
        | <int>
        | <arith_expr>
        | <ident>

<arith_expr> := <expr> <arith_op> <expr>
<arith_op> := + | - | * | /

<ident> := <char> <ident_rest>*
<ident_rest> := <char> | <digit>

<int> := <digit>+
<digit> := 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

<string> := " <string_elem>* "
<string_elem> := <any char other than ">

You might have noticed that this language definition is short on specificity. I haven't specified how big the number can be (such as if it can be bigger than a 32-bit integer) or even if the number can be negative. A true EBNF definition would precisely define these details, but, for the sake of conciseness, I will keep my example here simple.

Here is a sample Good for Nothing language program:

var ntimes = 0;
print "How much do you love this company? (1-10) ";
read_int ntimes;
var x = 0;
for x = 0 to ntimes do
   print "Developers!";
end;
print "Who said sit down?!!!!!";

You can compare this simple program with the language definition to get a better understanding of how the grammar works. And with that, the language definition is done.

High-Level Architecture

A compiler's job is to translate high-level tasks created by the programmer into tasks that a computer processor can understand and execute. In other words, it will take a program written in the Good for Nothing language and translate it into something that the .NET CLR can execute. A compiler achieves this through a series of translation steps, breaking down the language into parts that we care about and throwing away the rest. Compilers follow common software design principles—loosely coupled components, called phases, plugged together to perform translation steps. Figure 2 illustrates the components that perform the phases of a compiler: the scanner, parser, and code generator. In each phase, the language is broken down further, and that information about the program's intention is served to the next phase.

Figure 2 The Compiler's Phases
Figure 2 The Compiler's Phases(Click the image for a smaller view)
Figure 2 The Compiler's Phases
Figure 2 The Compiler's Phases (Click the image for a larger view)

Compiler geeks often abstractly group the phases into front end and back end. The front end consists of scanning and parsing, while the back end typically consists of code generation. The front end's job is to discover the syntactic structure of a program and translate that from text into a high-level in-memory representation called an Abstract Syntax Tree (AST), which I will elaborate on shortly. The back end has the task of taking the AST and converting it into something that can be executed by a machine.

The three phases are usually divided into a front end and a back end because scanners and parsers are typically coupled together, while the code generator is usually tightly coupled to a target platform. This design allows a developer to replace the code generator for different platforms if the language needs to be cross-platform.

I've made the code for the Good for Nothing compiler available in the code download accompanying this article. You can follow along as I walk through the components of each phase and explore the implementation details.

The Scanner

A scanner's primary job is to break up text (a stream of characters in the source file) into chunks (called tokens) that the parser can consume. The scanner determines which tokens are ultimately sent to the parser and, therefore, is able to throw out things that are not defined in the grammar, like comments. As for my Good for Nothing language, the scanner cares about characters (A-Z and the usual symbols), numbers (0-9), characters that define operations (such as +, -, *, and /), quotation marks for string encapsulation, and semicolons.

A scanner groups together streams of related characters into tokens for the parser. For example, the stream of characters " h e l l o w o r l d ! " would be grouped into one token: "hello world!".

The Good for Nothing scanner is incredibly simple, requiring only a System.IO.TextReader on instantiation. This kicks off the scanning process, as shown here:

public Scanner(TextReader input)
{
    this.result = new Collections.List<object>();
    this.Scan(input);
}

Figure 3 illustrates the Scan method, which has a simple while loop that walks over every character in the text stream and finds recognizable characters declared in the language definition. Each time a recognizable character or chunk of characters is found, the scanner creates a token and adds it to a List<object>. (In this case, I type it as object. However, I could have created a Token class or something similar to encapsulate more information about the token, such as line and column numbers.)

Figure 3 The Scanner's Scan Method (Close Figure 3)
private void Scan(TextReader input)
{
  while (input.Peek() != -1)
  {
    char ch = (char)input.Peek();

    // Scan individual tokens
    if (char.IsWhiteSpace(ch))
    {
      // eat the current char and skip ahead
      input.Read();
    }
    else if (char.IsLetter(ch) || ch == '_')
    {
      StringBuilder accum = new StringBuilder();

      input.Read(); // skip the '"'

      if (input.Peek() == -1)
      {
        throw new Exception("unterminated string literal");
      }

      while ((ch = (char)input.Peek()) != '"')
      {
        accum.Append(ch);
        input.Read();

        if (input.Peek() == -1)
        {
          throw new Exception("unterminated string literal");
        }
      }

      // skip the terminating "
      input.Read();
      this.result.Add(accum);
    }

    ...
  }
}

You can see that when the code encounters a " character, it assumes this will encapsulate a string token; therefore, I consume the string and wrap it up in a StringBuilder instance and add it to the list. After scan builds the token list, the tokens go to the parser class through a property called Tokens.

The Parser

The parser is the heart of the compiler, and it comes in many shapes and sizes. The Good for Nothing parser has a number of jobs: it ensures that the source program conforms to the language definition, and it handles error output if there is a failure. It also creates the in-memory representation of the program syntax, which is consumed by the code generator, and, finally, the Good for Nothing parser figures out what runtime types to use.

The first thing I need to do is take a look at the in-memory representation of the program syntax, the AST. Then I'll take a look at the code that creates this tree from the scanner tokens. The AST format is quick, efficient, easy to code, and can be traversed many times over by the code generator. The AST for the Good for Nothing compiler is shown in Figure 4.

Figure 4 AST for the Good for Nothing Compiler (Close Figure 4)
public abstract class Stmt
{
}

// var <ident> = <expr>
public class DeclareVar : Stmt
{
    public string Ident;
    public Expr Expr;
}

// print <expr>
public class Print : Stmt
{
    public Expr Expr;
}

// <ident> = <expr>
public class Assign : Stmt
{
    public string Ident;
    public Expr Expr;
}

// for <ident> = <expr> to <expr> do <stmt> end
public class ForLoop : Stmt
{
    public string Ident;
    public Expr From;
    public Expr To;
    public Stmt Body;
}

// read_int <ident>
public class ReadInt : Stmt
{
    public string Ident;
}

// <stmt> ; <stmt>
public class Sequence : Stmt
{
    public Stmt First;
    public Stmt Second;
}

/* <expr> := <string>
 *  | <int>
 *  | <arith_expr>
 *  | <ident>
 */
public abstract class Expr
{
}

// <string> := " <string_elem>* "
public class StringLiteral : Expr
{
    public string Value;
}

// <int> := <digit>+
public class IntLiteral : Expr
{
    public int Value;
}

// <ident> := <char> <ident_rest>*
// <ident_rest> := <char> | <digit>
public class Variable : Expr
{
    public string Ident;
}

// <arith_expr> := <expr> <arith_op> <expr>
public class ArithExpr : Expr
{
    public Expr Left;
    public Expr Right;
    public BinOp Op;
}

// <arith_op> := + | - | * | /
public enum ArithOp
{
    Add,
    Sub,
    Mul,
    Div
}

A quick glance at the Good for Nothing language definition shows that the AST loosely matches the language definition nodes from the EBNF grammar. It's best to think of the language definition as encapsulating the syntax, while the abstract syntax tree captures the structure of those elements.

There are many algorithms available for parsing, and exploring all of them is beyond the scope of this article. In general, they differ in how they walk the stream of tokens to create the AST. In my Good for Nothing compiler, I use what's called an LL (Left-to-right, Left-most derivation) top-down parser. This simply means that it reads text from left to right, constructing the AST based on the next available token of input.

The constructor for my parser class simply takes a list of tokens that was created by the scanner:

public Parser(IList<object> tokens)
{
    this.tokens = tokens;
    this.index = 0;
    this.result = this.ParseStmt();

    if (this.index != this.tokens.Count)
        throw new Exception("expected EOF");
}

The core of the parsing work is done by the ParseStmt method, as shown in Figure 5. It returns a Stmt node, which serves as the root node of the tree and matches the language syntax definition's top-level node. The parser traverses the list of tokens using an index as the current position while identifying tokens that are subservient to the Stmt node in the language syntax (variable declarations and assignments, for loops, read_ints, and prints). If a token can't be identified, an exception is thrown.

Figure 5 ParseStmt Method Identifies Tokens (Close Figure 5)
private Stmt ParseStmt()
{
    Stmt result;

    if (this.index == this.tokens.Count)
    {
        throw new Exception("expected statement, got EOF");
    }

    if (this.tokens[this.index].Equals("print"))
    {
        this.index++;
        ...
    }
    else if (this.tokens[this.index].Equals("var"))
    {
        this.index++;
        ...
    }
        else if (this.tokens[this.index].Equals("read_int"))
    {
        this.index++;
        ...
    }
    else if (this.tokens[this.index].Equals("for"))
    {
        this.index++;
        ...
    }
    else if (this.tokens[this.index] is string)
    {
        this.index++;
        ...
    }
    else
    {
        throw new Exception("parse error at token " + this.index +
            ": " + this.tokens[this.index]);
    }
    ...
}

When a token is identified, an AST node is created and any further parsing required by the node is performed. The code required for creating the print AST node is as follows:

// <stmt> := print <expr>
if (this.tokens[this.index].Equals("print"))
{
    this.index++;
    Print print = new Print();
    print.Expr = this.ParseExpr();
    result = print;
}

Two things happen here. The print token is discarded by incrementing the index counter, and a call to the ParseExpr method is made to obtain an Expr node, since the language definition requires that the print token be followed by an expression.

Figure 6 shows the ParseExpr code. It traverses the list of tokens from the current index, identifying tokens that satisfy the language definition of an expression. In this case, the method simply looks for strings, integers, and variables (which were created by the scanner instance) and returns the appropriate AST nodes representing these expressions.

Figure 6 ParseExpr Performs Parsing of Expression Nodes (Close Figure 6)
// <expr> := <string>
// | <int>
// | <ident>
private Expr ParseExpr()
{
  ...
  if (this.tokens[this.index] is StringBuilder)
  {
    string value = ((StringBuilder)this.tokens[this.index++]).ToString();
    StringLiteral stringLiteral = new StringLiteral();
    stringLiteral.Value = value;
    return stringLiteral;
  }
  else if (this.tokens[this.index] is int)
  {
    int intValue = (int)this.tokens[this.index++];
    IntLiteral intLiteral = new IntLiteral();
    intLiteral.Value = intValue;
    return intLiteral;
  }
  else if (this.tokens[this.index] is string)
  {
    string ident = (string)this.tokens[this.index++];
    Variable var = new Variable();
    var.Ident = ident;
    return var;
  }
  ...
}

For string statements that satisfy the "<stmt> ; <stmt>" language syntax definition, the sequence AST node is used. This sequence node contains two pointers to stmt nodes and forms the basis of the AST tree structure. The following details the code used to deal with the sequence case:

if (this.index < this.tokens.Count && this.tokens[this.index] ==
    Scanner.Semi)
{
    this.index++;

    if (this.index < this.tokens.Count &&
        !this.tokens[this.index].Equals("end"))
    {
        Sequence sequence = new Sequence();
        sequence.First = result;
        sequence.Second = this.ParseStmt();
        result = sequence;
    }
}

The AST tree shown in Figure 7 is the result of the following snippet of Good for Nothing code:

Figure 7 helloworld.gfn AST Tree and High-Level Trace
Figure 7 helloworld.gfn AST Tree and High-Level Trace(Click the image for a smaller view)
Figure 7 helloworld.gfn AST Tree and High-Level Trace
Figure 7 helloworld.gfn AST Tree and High-Level Trace (Click the image for a larger view)

var x = "hello world!";
print x;

Targeting the .NET Framework

Before I get into the code that performs code generation, I should first step back and discuss my target. So here I will describe the compiler services that the .NET CLR provides, including the stack-based virtual machine, the type system, and the libraries used for .NET assembly creation. I'll also briefly touch on the tools that are required to identify and diagnose errors in compiler output.

The CLR is a virtual machine, meaning it is a piece of software that emulates a computer system. Like any computer, the CLR has a set of low-level operations it can perform, a set of memory services, and an assembly language to define executable programs. The CLR uses an abstract stack-based data structure to model code execution, and an assembly language, called Intermediate Language (IL), to define the operations you can perform on the stack.

When a computer program defined in IL is executed, the CLR simply simulates the operations specified against a stack, pushing and popping data to be executed by an instruction. Suppose you want to add two numbers using IL. Here's the code used to perform 10 + 20:

  ldc.i4    10
  ldc.i4    20
  add

The first line (ldc.i4 10) pushes the integer 10 onto the stack. Then the second line (ldc.i4 20) pushes the integer 20 onto the stack. The third line (add) pops the two integers off the stack, adds them, and pushes the result onto the stack.

Simulation of the stack machine occurs by translating IL and the stack semantics to the underlying processor's machine language, either at run time through just-in-time (JIT) compilation or before hand through services like Native Image Generator (Ngen).

There are lots of IL instructions available for building your programs—they range from basic arithmetic to flow control to a variety of calling conventions. Details about all the IL instructions can be found in Partition III of the European Computer Manufacturers Association (ECMA) specification (available at msdn2.microsoft.com/aa569283).

The CLR's abstract stack performs operations on more than just integers. It has a rich type system, including strings, integers, booleans, floats, doubles, and so on. In order for my language to run safely on the CLR and interoperate with other .NET-compliant languages, I incorporate some of the CLR type system into my own program. Specifically, the Good for Nothing language defines two types—numbers and strings—which I map to System.Int32 and System.String.

The Good for Nothing compiler makes use of a Base Class Library (BCL) component called System.Reflection.Emit to deal with IL code generation and .NET assembly creation and packaging. It's a low-level library, which sticks close to the bare metal by providing simple code-generation abstractions over the IL language. The library is also used in other well-known BCL APIs, including System.Xml.XmlSerializer.

The high-level classes that are required to create a .NET assembly (shown in Figure 8) somewhat follow the builder software design pattern, with builder APIs for each logical .NET metadata abstraction. The AssemblyBuilder class is used to create the PE file and set up the necessary .NET assembly metadata elements like the manifest. The ModuleBuilder class is used to create modules within the assembly. TypeBuilder is used to create Types and their associated metadata. MethodBuilder and LocalBuilder deal with adding methods to types and locals to methods, respectively. The ILGenerator class is used to generate the IL code for methods, utilizing the OpCodes class, which is a big enumeration containing all possible IL instructions. All of these Reflection.Emit classes are used in the Good for Nothing code generator.

Figure 8 Reflection.Emit Libraries Used to Build a .NET Assembly
Figure 8 Reflection.Emit Libraries Used to Build a .NET Assembly(Click the image for a smaller view)
Figure 8 Reflection.Emit Libraries Used to Build a .NET Assembly
Figure 8 Reflection.Emit Libraries Used to Build a .NET Assembly (Click the image for a larger view)

Tools for Getting Your IL Right

Even the most seasoned compiler hackers make mistakes at the code-generation level. The most common bug is bad IL code, which causes unbalance in the stack. The CLR will typically throw an exception when bad IL is found (either when the assembly is loaded or when the IL is JITed, depending on the trust level of the assembly). Diagnosing and repairing these errors is simple with an SDK tool called peverify.exe. It performs a verification of the IL, making sure the code is correct and safe to execute.

For example, here is some IL code that attempts to add the number 10 to the string "bad":

ldc.i4    10
ldstr    "bad"
add

Running peverify over an assembly that contains this bad IL will result in the following error:

[IL]: Error: [C:\MSDNMagazine\Sample.exe : Sample::Main][offset 0x0000002][found ref 'System
.String'] Expected numeric type on the stack.

In this example, peverify reports that the add instruction expected two numeric types where it instead found an integer and a string.

ILASM (IL assembler) and ILDASM (IL disassembler) are two SDK tools you can use to compile text IL into .NET assemblies and decompile assemblies out to IL, respectively. ILASM allows for quick and easy testing of IL instruction streams that will be the basis of the compiler output. You simply create the test IL code in a text editor and feed it in to ILASM. Meanwhile, the ILDASM tool can quickly peek at the IL a compiler has generated for a particular code path. This includes the IL that commercial compilers emit, such as the C# compiler. It offers a great way to see the IL code for statements that are similar between languages; in other words, the IL flow control code generated for a C# for loop could be reused by other compilers that have similar constructs.

The Code Generator

The code generator for the Good for Nothing compiler relies heavily on the Reflection.Emit library to produce an executable .NET assembly. I will describe and analyze the important parts of the class; the other bits are left for you to peruse at your leisure.

The CodeGen constructor, which is shown in Figure 9, sets up the Reflection.Emit infrastructure, which is required before I can start emitting code. I begin by defining the assembly name and passing that to the assembly builder. In this example, I use the source file name as the assembly name. Next is the ModuleBuilder definition—for one module definition, this uses the same name as the assembly. I then define a TypeBuilder on the ModuleBuilder to hold the only type in the assembly. There are no types defined as first class citizens of the Good for Nothing language definition, but at least one type is necessary to hold the method, which will run on startup. The MethodBuilder defines a Main method to hold the IL that will be generated for the Good for Nothing code. I have to call SetEntryPoint on this MethodBuilder so it will run on startup when a user runs the executable. And I create the global ILGenerator (il) from the MethodBuilder using the GetILGenerator method.

Figure 9 CodeGen Constructor (Close Figure 9)
Emit.ILGenerator il = null;
Collections.Dictionary<string, Emit.LocalBuilder> symbolTable;

public CodeGen(Stmt stmt, string moduleName)
{
  if (Path.GetFileName(moduleName) != moduleName)
  {
    throw new Exception("can only output into current directory!");
  }

  AssemblyName name = new
    AssemblyName(Path.GetFileNameWithoutExtension(moduleName));
  Emit.AssemblyBuilder asmb =
    AppDomain.CurrentDomain.DefineDynamicAssembly(name,
      Emit.AssemblyBuilderAccess.Save);
  Emit.ModuleBuilder modb = asmb.DefineDynamicModule(moduleName);
  Emit.TypeBuilder typeBuilder = modb.DefineType("Foo");

  Emit.MethodBuilder methb = typeBuilder.DefineMethod("Main",
    Reflect.MethodAttributes.Static,
    typeof(void),
    System.Type.EmptyTypes);

  // CodeGenerator
  this.il = methb.GetILGenerator();
  this.symbolTable = new Dictionary<string, Emit.LocalBuilder>();

  // Go Compile
  this.GenStmt(stmt);

  il.Emit(Emit.OpCodes.Ret);
  typeBuilder.CreateType();
  modb.CreateGlobalFunctions();
  asmb.SetEntryPoint(methb);
  asmb.Save(moduleName);
  this.symbolTable = null;
  this.il = null;
}

Once the Reflection.Emit infrastructure is set up, the code generator calls the GenStmt method, which is used to walk the AST. This generates the necessary IL code through the global ILGenerator. Figure 10 shows a subset of the GenStmt method, which, at first invocation, starts with a Sequence node and proceeds to walk the AST switching on the current AST node type.

Figure 10 Subset of GenStmt Method (Close Figure 10)
private void GenStmt(Stmt stmt)
{
    if (stmt is Sequence)
    {
        Sequence seq = (Sequence)stmt;
        this.GenStmt(seq.First);
        this.GenStmt(seq.Second);
    }

    else if (stmt is DeclareVar)
    {
        ...
    }

    else if (stmt is Assign)
    {
        ...
    }
    else if (stmt is Print)
    {
        ...
    }
}

The code for the DeclareVar (declaring a variable) AST node is as follows:

else if (stmt is DeclareVar)
{
    // declare a local
    DeclareVar declare = (DeclareVar)stmt;
    this.symbolTable[declare.Ident] =
        this.il.DeclareLocal(this.TypeOfExpr(declare.Expr));

    // set the initial value
    Assign assign = new Assign();
    assign.Ident = declare.Ident;
    assign.Expr = declare.Expr;
    this.GenStmt(assign);
}

The first thing I need to accomplish here is to add the variable to a symbol table. The symbol table is a core compiler data structure that is used to associate a symbolic identifier (in this case, the string-based variable name) with its type, location, and scope within a program. The Good for Nothing symbol table is simple, as all variable declarations are local to the Main method. So I associate a symbol with a LocalBuilder using a simple Dictionary<string, LocalBuilder>.

After adding the symbol to the symbol table, I translate the DeclareVar AST node to an Assign node to assign the variable declaration expression to the variable. I use the following code to generate Assignment statements:

else if (stmt is Assign)
{
    Assign assign = (Assign)stmt;
    this.GenExpr(assign.Expr, this.TypeOfExpr(assign.Expr));
    this.Store(assign.Ident, this.TypeOfExpr(assign.Expr));
}

Doing this generates the IL code to load an expression onto the stack and then emits IL to store the expression in the appropriate LocalBuilder.

The GenExpr code shown in Figure 11 takes an Expr AST node and emits the IL required to load the expression onto the stack machine. StringLiteral and IntLiteral are similar in that they both have direct IL instructions that load the respective strings and integers onto the stack: ldstr and ldc.i4.

Figure 11 GenExpr Method (Close Figure 11)
private void GenExpr(Expr expr, System.Type expectedType)
{
  System.Type deliveredType;

  if (expr is StringLiteral)
  {
    deliveredType = typeof(string);
    this.il.Emit(Emit.OpCodes.Ldstr, ((StringLiteral)expr).Value);
  }
  else if (expr is IntLiteral)
  {
    deliveredType = typeof(int);
    this.il.Emit(Emit.OpCodes.Ldc_I4, ((IntLiteral)expr).Value);
  }
  else if (expr is Variable)
  {
    string ident = ((Variable)expr).Ident;
    deliveredType = this.TypeOfExpr(expr);

    if (!this.symbolTable.ContainsKey(ident))
    {
      throw new Exception("undeclared variable '" + ident + "'");
    }

    this.il.Emit(Emit.OpCodes.Ldloc, this.symbolTable[ident]);
  }
  else
  {
    throw new Exception("don't know how to generate " +
      expr.GetType().Name);
  }

  if (deliveredType != expectedType)
  {
    if (deliveredType == typeof(int) &&
        expectedType == typeof(string))
    {
      this.il.Emit(Emit.OpCodes.Box, typeof(int));
      this.il.Emit(Emit.OpCodes.Callvirt,
        typeof(object).GetMethod("ToString"));
    }
    else
    {
      throw new Exception("can't coerce a " + deliveredType.Name +
        " to a " + expectedType.Name);
    }
  }
}

Variable expressions simply load a method's local variable onto the stack by calling ldloc and passing in the respective LocalBuilder. The last section of code shown in Figure 11 deals with converting the expression type to the expected type (called type coercion). For instance, a type may need conversion in a call to the print method where an integer needs to be converted to a string so that print can be successful.

Figure 12 demonstrates how variables are assigned expressions in the Store method. The name is looked up via the symbol table and the respective LocalBuilder is then passed to the stloc IL instruction. This simply pops the current expression from the stack and assigns it to the local variable.

Figure 12 Store Expressions to a Variable (Close Figure 12)
private void Store(string name, Type type)
{
  if (this.symbolTable.ContainsKey(name))
  {
    Emit.LocalBuilder locb = this.symbolTable[name];

    if (locb.LocalType == type)
    {
      this.il.Emit(Emit.OpCodes.Stloc, this.symbolTable[name]);
    }
    else
    {
      throw new Exception("'" + name + "' is of type " +
        locb.LocalType.Name + " but attempted to store value of type " +
        type.Name);
    }
  }
  else
  {
    throw new Exception("undeclared variable '" + name + "'");
  }
}

The code that is utilized to generate the IL for the Print AST node is interesting because it calls in to a BCL method. The expression is generated onto the stack, and the IL call instruction is used to call the System.Console.WriteLine method. Reflection is used to obtain the WriteLine method handle that is needed to pass to the call instruction:

else if (stmt is Print)
{
  this.GenExpr(((Print)stmt).Expr, typeof(string));
  this.il.Emit(Emit.OpCodes.Call,
    typeof(System.Console).GetMethod("WriteLine",
    new Type[] { typeof(string) }));
}

When a call to a method is performed, method arguments are popped off the stack last on, first in. In other words, the first argument of the method is the top stack item, the second argument the next item, and so on.

The most complex code here is the code that generates IL for my Good for Nothing for loops (see Figure 13). It is quite similar to how commercial compilers would generate this kind of code. However, the best way to explain the for loop code is to look at the IL that is generated, which is shown in Figure 14.

Figure 14 For Loop IL Code (Close Figure 14)
// for x = 0
IL_0006:  ldc.i4     0x0
IL_000b:  stloc.0

// jump to the test
IL_000c:  br         IL_0023

// execute the loop body
IL_0011:  ...

// increment the x variable by 1
IL_001b:  ldloc.0
IL_001c:  ldc.i4     0x1
IL_0021:  add
IL_0022:  stloc.0

// TEST
// load x, load 100, branch if
// x is less than 100
IL_0023:  ldloc.0
IL_0024:  ldc.i4     0x64
IL_0029:  blt        IL_0011
Figure 13 For Loop Code (Close Figure 13)
else if (stmt is ForLoop)
{
    // example:
    // var x = 0;
    // for x = 0 to 100 do
    //   print "hello";
    // end;

    // x = 0
    ForLoop forLoop = (ForLoop)stmt;
    Assign assign = new Assign();
    assign.Ident = forLoop.Ident;
    assign.Expr = forLoop.From;
    this.GenStmt(assign);
    // jump to the test
    Emit.Label test = this.il.DefineLabel();
    this.il.Emit(Emit.OpCodes.Br, test);

    // statements in the body of the for loop
    Emit.Label body = this.il.DefineLabel();
    this.il.MarkLabel(body);
    this.GenStmt(forLoop.Body);

    // to (increment the value of x)
    this.il.Emit(Emit.OpCodes.Ldloc, this.symbolTable[forLoop.Ident]);
    this.il.Emit(Emit.OpCodes.Ldc_I4, 1);
    this.il.Emit(Emit.OpCodes.Add);
    this.Store(forLoop.Ident, typeof(int));

    // **test** does x equal 100? (do the test)
    this.il.MarkLabel(test);
    this.il.Emit(Emit.OpCodes.Ldloc, this.symbolTable[forLoop.Ident]);
    this.GenExpr(forLoop.To, typeof(int));
    this.il.Emit(Emit.OpCodes.Blt, body);
}

The IL code starts with the initial for loop counter assignment and immediately jumps to the for loop test using the IL instruction br (branch). Labels like the ones listed to the left of the IL code are used to let the runtime know where to branch to the next instruction. The test code checks to see if the variable x is less than 100 using the blt (branch if less than) instruction. If this is true, the loop body is executed, the x variable is incremented, and the test is run again.

The for loop code in Figure 13 generates the code required to perform the assignment and increment operations on the counter variable. It also uses the MarkLabel method on ILGenerator to generate labels that the branch instructions can branch to.

Wrapping Up ... Almost

I've walked through the code base to a simple .NET compiler and explored some of the underlying theory. This article is intended to provide you with a foundation to the mysterious world of compiler construction. While you'll find valuable information online, there are some books you should also check out. I recommend picking up copies of: Compiling for the .NET Common Language Runtime by John Gough (Prentice Hall, 2001), Inside Microsoft IL Assembler by Serge Lidin (Microsoft Press®, 2002), Programming Language Pragmatics by Michael L. Scott (Morgan Kaufmann, 2000), and Compilers: Principles, Techniques, and Tools by Alfred V. Oho, Monica S. Lam, Ravi Sethi, and Jeffrey Ullman (Addison Wesley, 2006).

That pretty well covers the core of what you need to understand for writing your own language compiler; however, I'm not quite finished with this discussion. For the seriously hardcore, I'd like to look at some advanced topics for taking your efforts even further.

Dynamic Method Invocation

Method calls are the cornerstone of any computer language, but there's a spectrum of calls you can make. Newer languages, such as Python, delay the binding of a method and the invocation until the absolute last minute—this is called dynamic invocation. Popular dynamic languages, such as Ruby, JavaScript, Lua, and even Visual Basic, all share this pattern. In order for a compiler to emit code to perform a method invocation, the compiler must treat the method name as a symbol, passing it to a runtime library that will perform the binding and invocation operations as per the language semantics.

Suppose you turn off Option Strict in the Visual Basic 8.0 compiler. Method calls become late bound and the Visual Basic runtime will perform the binding and invocation at run time.

Rather than the Visual Basic compiler emitting an IL call instruction to the Method1 method, it instead emits a call instruction to the Visual Basic runtime method called CompilerServices.NewLateBinding.LateCall. In doing so, it passes in an object (obj) and the symbolic name of the method (Method1), along with any method arguments. The Visual Basic LateCall method then looks up the Method1 method on the object using Reflection and, if it is found, performs a Reflection-based method invocation:

   Option Strict Off

Dim obj
obj.Method1()

IL_0001:  ldloc.0
IL_0003:  ldstr      "Method1"
...
IL_0012:  call       object CompilerServices.NewLateBinding::LateCall(object, ... , string, ...)

Using LCG to Perform Quick Late-Binding

Reflection-based method invocation can be notoriously slow (see my article "Reflection: Dodge Common Performance Pitfalls to Craft Speedy Applications" at msdn.microsoft.com/msdnmag/issues/05/07/Reflection). Both the binding of the method and the method invocation are many orders of magnitude slower than a simple IL call instruction. The .NET Framework 2.0 CLR includes a feature called Lightweight Code Generation (LCG), which can be used to dynamically create code on the fly to bridge the call site to the method using the faster IL call instruction. This significantly speeds up method invocation. The lookup of the method on an object is still required, but once it's found, a DynamicMethod bridge can be created and cached for each repeating call.

Figure 15 shows a very simple version of a late binder that performs dynamic code generation of a bridge method. It first looks up the cache and checks to see if the call site has been seen before. If the call site is being run for the first time, it generates a DynamicMethod that returns an object and takes an object array as an argument. The object array argument contains the instance object and the arguments that are to be used for the final call to the method. IL code is generated to unpack the object array on to the stack, starting with the instance object and then the arguments. A call instruction is then emitted and the result of that call is returned to the callee.

The call to the LCG bridge method is performed through a delegate, which is very quick. I simply wrap the bridge method arguments up into an object array and then call. The first time this happens, the JIT compiler compiles the dynamic method and performs the execution of the IL, which, in turn, calls the final method.

This code is essentially what an early bound static compiler would generate. It pushes an instance object on the stack, then the arguments, and then calls the method using the call instruction. This is a slick way of delaying that semantic to the last possible minute, satisfying the late-bound calling semantics found in most dynamic languages.

The Dynamic Language Runtime

If you're serious about implementing a dynamic language on the CLR, then you must check out the Dynamic Language Runtime (DLR), which was announced by the CLR team in late April. It encompasses the tools and libraries you need to build great performing dynamic languages that interoperate with both the .NET Framework and the ecosystem of other .NET-compliant languages. The libraries provide everything that someone creating a dynamic language will want, including a high-performing implementation for common dynamic language abstractions (lightning-fast late-bound method calls, type system interoperability, and so on), a dynamic type system, a shared AST, support for Read Eval Print Loop (REPL), and more.

A deep look into the DLR is well beyond the scope of this article, but I do recommend you explore it on your own to see the services it provides to dynamic languages. More information about the DLR can be found on Jim Hugunin's blog (blogs.msdn.com/hugunin).

Joel Pobar is a former Program Manager on the CLR team at Microsoft. He now hangs out at the Gold Coast in Australia, hacking away on compilers, languages, and other fun stuff. Check out his latest .NET ramblings at callvirt.net/blog.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


WinUnit: Simplified Unit Testing for Native C++ Applications

WinUnit

Simplified Unit Testing for Native C++ Applications

Maria Blees
This article discusses:
  • Unit testing philosophy
  • Setting up WinUnit
  • Building test fixtures
  • Implementation and customization
This article uses the following technologies:
C++, Visual Studio
Code download available at: WinUnit2008_02.exe (1438KB)
Contents
Start Testing Today
Getting Started with WinUnit
Fixtures: Setup and Teardown
Running WinUnit
Implementation Details
Going Further...

T
hese days it can be hard not to feel downright oppressed as a native code developer—it seems like the developers using the Microsoft® .NET Framework get all the cool tools!

I've always been interested in good engineering practices, but I've been frustrated by so-called "engineering experts" who extol the virtues of unit testing, yet can offer little more than hand waving when asked for tool recommendations for native code. What I really wanted, and what I thought would be the easiest to integrate into an automated build system, was the equivalent of NUnit for native code. That is, I wanted to be able to make a DLL with only tests in it, and have an external test-runner that would run those tests and take care of the reporting and logging. I also wanted to be able to declare each test only once, and to have a minimum of extra code in my test DLL.

So I built a native code unit testing tool I call WinUnit. I'll go into more detail later, but here's a little preview of how easy it is to create and run a test DLL using WinUnit. To start, create a single CPP file—let's call it DummyTest.cpp:

#include "WinUnit.h"

BEGIN_TEST(DummyTest)
{
   WIN_ASSERT_TRUE(3 > 4, _T("You should see this error message."));
}
END_TEST

Build it into a DLL with this command:

cl /EHsc /LD DummyTest.cpp

Then execute WinUnit on it:

>WinUnit DummyTest.dll
Processing [DummyTest.dll]...
(DummyTest)
DummyTest.cpp(5): error : WIN_ASSERT_TRUE failed: "3 > 4". You should see this error message.
FAILED: DummyTest.
[DummyTest.dll] FAILED.  Tests run: 1; Failures: 1.

There were errors.
Tests run: 1; Failed: 1.

Changing 3 > 4 to the true expression 4 > 3 will of course get rid of the failure.

WinUnit will accept any number of DLLs or directories, process them, report the total results, and return with an exit code. By the way, if you haven't guessed, WinUnit only works on Windows®.

I concede that the world was not entirely devoid of options for C++ unit testing before I wrote my own tool. The problem I found with existing offerings was that they generally placed a higher priority on portability than on being easy to understand and use out of the box. CppUnitLite is my favorite among the portable set, but because it's designed to be very customizable, there was more overhead than I would have liked in getting it set up. The line between test-runner and tests is blurry due to the fact that it's all done within the same binary, and the code that actually invokes the test-running functionality needs to be written and put somewhere. In addition, the tricky macro implementation was hard to explain to my coworkers and thus made adoption difficult. However, if you require portability to other operating systems, CppUnitLite is definitely worth taking a look at.

Other options that have been available for C++ unit testing for quite some time are the same options available for testing .NET code, most notably NUnit and the Visual Studio® Team Test unit testing framework. Since both will execute any properly attributed .NET assembly, you can use these tools with tests written in managed C++ (now C++/CLI). The main inconvenience I find with this approach is that the tools are highly geared toward .NET development and not toward native code. This should not be a surprise. But it is hard to use most of the provided assert methods with native C++ for they expect .NET-style objects, and as a developer you will need to have at least a passing familiarity with C++/CLI (or Managed Extensions for C++) to write the tests. You'll find more details about limitations with this approach in the footnote on the Visual Studio 2005 Product Feature Comparisons page (msdn2.microsoft.com/vstudio/aa700921.aspx). As mentioned previously, I find the "ease of convincing my coworkers to use it" factor important, and using the managed tools provides an extra hurdle in this area.

Start Testing Today
Code Coverage and Convenience

I'd like to briefly address a feature of Visual Studio Team System (VSTS) that deserves a mention for its relevance to unit testing: its set of code coverage tools. While the exact significance of code coverage numbers is debatable, reviewing your coverage after you've carefully constructed thoughtful tests can be, if nothing else, an excellent sanity check.

Unfortunately, VSTS does not make it easy to use (or even to know about) the code coverage tools for native code. Along with the code for this project, I've included a set of macros that will make it easy for you to do code coverage on native projects—in conjunction with WinUnit as well as with any other scenarios you have for exercising your code.

To use these macros, simply load WinUnit.vsmacros into Macro Explorer in Visual Studio. In the _Variables module you'll need to set variables for your particular local machine, which includes the path to the Visual Studio performance tools (including the code coverage tools) and the paths to WinUnit.exe and WinUnit.h. You'll find more details in readme.txt as well as the _Readme module in the macro project.

The CodeCoverage module includes macros for starting and stopping coverage data collection, instrumenting binaries (you can specify a list of binaries to be instrumented for each project), and launching a .coverage file with the results.

The RunningTests module includes macros for running the test containing the cursor in the currently opened document, all tests in the currently opened file, and all tests in the selected project. Other macros provide shortcuts for changing relevant project settings and for running different sets of tests with code coverage turned on.

Before getting into WinUnit usage, I want to discuss my philosophy about unit testing in general and unit testing native code in particular. If you're like me, you may have noticed that the clean, elegant, and modular classes shown in unit testing examples seem to have nothing in common with the blob of legacy code you typically have to work with. If you think that you must rework your entire code base before you can unit test, I can tell you those tests are going to be a long time coming. The good news is that a testable unit doesn't have to be a clean, elegant class. It's simply the smallest unit of code that can be tested in isolation. If that's your whole application, well, there's no better time to start than now!

Seriously, if your application consists of exactly one executable and its only interface is through a GUI, you may have to do a bit of refactoring to make the non-GUI part automatable. However, it can certainly be done incrementally. I recommend reading the whitepaper "The Humble Dialog Box," by Michael Feathers, for a sensible approach to making a GUI application more testable (objectmentor.com/resources/articles/TheHumbleDialogBox.pdf).

Your ultimate goal should be to have automated developer tests running as part of your build process, making it as easy as possible for developers to add tests as they fix bugs or write new code. Once the infrastructure is set up, you can work on refactoring existing code to allow better test coverage in problem areas. The book Working Effectively with Legacy Code, also by Michael Feathers (Prentice Hall, 2004), is an excellent resource for guiding you through this process (and many of the examples are in C++).

If your application is already split up into static libraries with clearly defined interfaces, you can link these directly into test DLLs and you'll be able to get at the individual classes to test them. Consider having one test DLL per library to maintain clear interface boundaries.

If your application contains separate DLLs, or is itself a library provided in DLL form, there are two ways you can access the exports to exercise them in tests. If you use import libraries, you can link those into test DLLs and have transparent access to the production functions for testing. If you do not use import libraries, use GetProcAddress in conjunction with LoadLibrary and FreeLibrary to get the DLL exports to call directly in your tests.

Note that if your DLL is a COM DLL, it doesn't need to be registered to be tested. Windows XP and later offer a mechanism called "Registration-Free COM," which you could use to get around the usual registration requirement for COM DLLs. You'll find details at msdn2.microsoft.com/ms973913. As a bonus, once you've gotten it working for tests, you can use the feature in your production code.

If your application is a monolithic executable, you probably want to split it up so you have at least one library. But you might also want to test the executable in its entirety to check for proper exit codes, for example. In my unit tests for WinUnit itself, I included some tests that exercised the entire executable. You can find these within the TestWinUnit project—in the MainTest.cpp file—in the accompanying code download.

At risk of coming across as the style police, there's one point about physically organizing production code that I would be remiss if I did not mention. In my years at Microsoft working on large C++ projects, I've found that one thing that made modularity (and testability) exceedingly difficult was having multiple classes jumbled together across arbitrary .cpp and .h files. It's up to you how strictly you adhere to this practice in your production code, but I find associating one and only one class with a similarly named pair of .cpp and .h files to be helpful. As a bonus, it simplifies keeping your tests organized, with one matching test file for each class. You can see some examples in my projects WinUnitLib and TestWinUnit.

There is one other book I recommend reading if the whole concept of unit testing has just passed you by, or if you're a developer in need of a refresher on how to think like a tester: Pragmatic Unit Testing with C# in NUnit, by Andrew Hunt and David Thomas, (Pragmatic Bookshelf, 2006) is an excellent primer. (There's also a Java counterpart.) The meat of the book is actually language-independent, despite the language-specific title. A cute mnemonic from the book is that unit tests should be A-TRIP: Automatic, Thorough, Repeatable, Independent, and Professional. Repeatable means each test produces the same results any time it is run. Independent means tests should be able to be run in any order, with no dependencies on each other. This point will be important to remember later.

Getting Started with WinUnit

The first thing you'll need to do is build WinUnit and place WinUnit.exe and WinUnit.h in known locations on your machine. You can refer to the readme.txt included with the code for more information. Next, to get started writing tests, create a test project. As I demonstrated earlier, this is just a regular C++ DLL. For information on creating a native C++ DLL project in Visual Studio, see the walkthrough at msdn2.microsoft.com/ms235636.

As with other unit testing frameworks, several assert macros are provided with WinUnit for your test-writing convenience, such as the WIN_ASSERT_TRUE you saw earlier. (I'll discuss the other WIN_ASSERT macros you can use with WinUnit later in this section.) The asserts perform their magic using C++ exceptions, so you should keep the Visual Studio 2005 default compiler option of /EHsc when building your test DLL. However, there's no requirement for your production code to use C++ exception handling, even if it's linked into your test DLL. (Note that the Visual C++® 2005 toolset or later is required to use the provided assert macros.) You'll also want to add WinUnit's Include directory to your test project's include path as well (see Figure 1), or to your global include paths.

Figure 1 Setting Project Include Path
Figure 1 Setting Project Include Path(Click the image for a smaller view)
Figure 1 Setting Project Include Path
Figure 1 Setting Project Include Path (Click the image for a larger view)

I designed WinUnit to work well from the command line, which means you can set up Visual Studio to run WinUnit on your test project after every build. To do so, go to the project Properties of your test project, and under Configuration Properties | Build Events, select Post-Build Event (see Figure 2). For Command Line, type the full path to where you copied WinUnit.exe followed by "$(TargetPath)" (include the double quotes). For Description, type "Running WinUnit..." (without the quotes). If you add the folder containing WinUnit.exe to your global executables path, you don't need to specify the full path here.

Figure 2 Configuring WinUnit as a Post-Build Event
Figure 2 Configuring WinUnit as a Post-Build Event(Click the image for a smaller view)
Figure 2 Configuring WinUnit as a Post-Build Event
Figure 2 Configuring WinUnit as a Post-Build Event (Click the image for a larger view)

Alternatively (or in addition), you may wish to add WinUnit.exe to your Tools menu in order to run your tests. To accomplish this, go to Tools | External Tools and click Add. For Title, type "&WinUnit". For Command, browse to WinUnit.exe. For Arguments, type "$(TargetPath)" (include the double quotes). For Initial Directory, type $(TargetDir). Uncheck Close on Exit and check Use Output Window (see Figure 3). At this point in the process you can select WinUnit from the Tools menu, and it will execute on whichever project is currently selected.

Figure 3 Adding WinUnit to External Tools
Figure 3 Adding WinUnit to External Tools

Finally, to set up your project for debugging, go to project Properties | Configuration Properties | Debugging, and type the full path to WinUnit.exe in the Command textbox. For Command Arguments, type "$(TargetPath)" (include the double quotes).

To verify everything in your project is set up correctly, you can add DummyTest.cpp to your project and build. The project should build, but if you've typed in the false assertion line as presented, you'll clearly see a test failure on that line.

Now that the setup is out of the way, it's time to look at the different ways of using WinUnit features to write tests. You may want to switch to WinUnitComplete.sln for the moment to follow along with my examples. The projects containing the examples are SampleLib and TestSampleLib. SampleLib is a static library that happens to contain exactly one class, BinaryNumber. TestSampleLib is a test DLL that links with SampleLib.lib and includes tests for the BinaryNumber class along with a few other examples.

General WinUnit test functions always begin with BEGIN_TEST(TestName) and end with END_TEST. Within each test, one or more WIN_ASSERT macros are used in order to verify various bits of functionality.

My example BinaryNumber class has two constructors. One takes an unsigned short, and one takes a string comprising the characters 1 and 0. I intend for the constructors to result in equivalent objects when passed equivalent values. So I may have a test like this:

BEGIN_TEST(BinaryNumberConstructorsShouldBeEquivalent)
{
  unsigned short numericValue = 7;
  BinaryNumber bn1(numericValue);
  BinaryNumber bn2("111");
  WIN_ASSERT_EQUAL(bn1.NumericValue, bn2.NumericValue,
    _T("Both values should be %u."), numericValue);
  WIN_ASSERT_STRING_EQUAL(bn1.StringValue, bn2.StringValue);
}
END_TEST

This test will fail if either of the two assert lines is false. Notice that the WIN_ASSERT_EQUAL macro in this example is passed the two values being compared, plus two extra arguments. These comprise an informational message that will be shown if the assert fails. All WIN_ASSERT macros take an optional printf-style format string and arguments for this purpose.

Since I implemented operator '==' for the BinaryNumber class, I can also use the following construction (which is what I have in my sample file, BinaryNumberTest.cpp):

BEGIN_TEST(
  BinaryNumberConstructorsShouldBeEquivalent)
{
  BinaryNumber bn1(7);
  BinaryNumber bn2("111");
  WIN_ASSERT_EQUAL(bn1, bn2);
}
END_TEST

Here's a different type of assert, WIN_ASSERT_THROWS. Suppose I use exceptions for my error handling in my production code, and I want to force an error condition in my test to ensure that the proper exception is thrown. I might make a test like this:

BEGIN_TEST(
  BinaryNumberPlusRecognizesIntegerOverflow)
{
  unsigned short s = USHRT_MAX / 2 + 1;

    BinaryNumber bn1(s);
    BinaryNumber bn2(s);

    WIN_ASSERT_THROWS(bn1 + bn2,
      BinaryNumber::IntegerOverflowException)
  }
  END_TEST

I know that my BinaryNumber class only holds an unsigned short. I also know that the operator '+' should detect when I'm trying to add two numbers together that are too big. WIN_ASSERT_THROWS takes an expression that should throw a C++ exception, along with the exception type itself. The test fails if the exception is not thrown.

Figure 4 lists the WIN_ASSERT macros available in WinUnit.h, as well as one non-assert, WIN_TRACE, that can be used to provide tracing through the OutputDebugString API function in your tests.

Figure 4 Assert and Trace Macros in WinUnit.h (Close Figure 4)

Macro | Description
WIN_ASSERT_EQUAL(expected, actual, ...) | Compares expected and actual using ==; fails if they are not equal.
WIN_ASSERT_NOT_EQUAL(notExpected, actual, ...) | Compares notExpected and actual using !=; fails if they are equal.
WIN_ASSERT_STRING_EQUAL(expected, actual, ...) | Does a case-sensitive string comparison of expected and actual. Strings to be compared can be either wide-character or not, but both must be the same "wideness". (This does not affect the "wideness" of the optional message.)
WIN_ASSERT_ZERO(zeroExpression, ...) | Compares zeroExpression to 0; fails if they are not equal.
WIN_ASSERT_NOT_ZERO(nonzeroExpression, ...) | Compares nonzeroExpression to 0; fails if they are equal.
WIN_ASSERT_NULL(nullExpression, ...) | Compares nullExpression to NULL; fails if they are not equal. Only works with pointers.
WIN_ASSERT_NOT_NULL(notNullExpression, ...) | Compares notNullExpression to NULL; fails if they are equal. Only works with pointers.
WIN_ASSERT_FAIL(message, ...) | Always fails; informational message is required.
WIN_ASSERT_TRUE(trueExpression, ...) | Succeeds if trueExpression evaluates to true.
WIN_ASSERT_FALSE(falseExpression, ...) | Succeeds if !falseExpression evaluates to true.
WIN_ASSERT_WINAPI_SUCCESS(trueExpression, ...) | Succeeds if trueExpression evaluates to true. Use this with Windows functions whose documentation says to call GetLastError for more error information on failure. This macro/function calls GetLastError and includes the string associated with the error code as part of the message.
WIN_ASSERT_THROWS(expression, exceptionType, ...) | Succeeds if expression throws a C++ exception of type exceptionType.
WIN_TRACE(message, ...) | Used to output informational messages for debugging purposes. Its use does not cause a test failure. "message" is a printf-style format string, followed by any arguments.

Note that for either WIN_ ASSERT_EQUAL or WIN_ASSERT_NOT_EQUAL, if a numeric literal is passed in as one of the values and an unsigned number as the other, you'll get a signed/unsigned mismatch, as numeric literal integers always template-match to int. To get around this, postfix numeric literals with 'U' so they will match as unsigned.

All asserts take an optional printf-style format string, plus arguments. If _UNICODE is defined, these message strings will be wchar_t*, so use the _T ("") macro around the format strings, or L"" if you're building Unicode-only.

You may have noticed that my test names begin with the name of the class being tested, followed by the name or description of the method being tested, followed by a partial sentence describing what the test is supposed to show. This convention serves two purposes. First, it is an easy way to make clear exactly what you're trying to test, which can be useful when looking at test output. Second, it's a way to group tests to be run together at varying levels of granularity. WinUnit does not inherently have the concept of groups of tests, as do its .NET Framework-based equivalents. The way to tell WinUnit to run a subset of tests in a project is with the -p (prefix) option. You specify a prefix string, and WinUnit will run all the tests whose names start with that string. By naming your tests with words ordered least to most specific, you will be able to easily run related groups of tests from the command line.

Fixtures: Setup and Teardown

In real life, the things you're trying to test are often not as simple as the examples I've shown so far. There may be cases where several setup steps are required to get to the point where you can actually execute the functionality you want to verify. For example, if I were testing a function that deletes all the files in a directory, I might first need to create a directory and put files in it. Or I might want to set an environment variable required for the functionality I'm testing.

To maintain the independence and repeatability of tests, it's important that whatever setup work you do at the beginning of a test is undone at the end of the test. WinUnit supports the concepts of single-test fixtures. These are setup and teardown function pairs that will be executed at the start and finish of each test associated with them. This is especially helpful if you have several tests that require the same setup and cleanup.

Figure 5 shows an example of a fixture, with setup and teardown functions implemented and two tests associated with it. Note that this example also shows that you can also use the various WIN_ASSERT macros even in fixtures so you can report failure if the fixture you define does not work correctly.

Figure 5 Setup and Teardown Test Fixture (Close Figure 5)
//  DeleteFileTest.cpp

#include "WinUnit.h"

#include <windows.h>

// Fixture must be declared.
FIXTURE(DeleteFileFixture);

namespace
{
    TCHAR s_tempFileName[MAX_PATH] = _T("");
    bool IsFileValid(TCHAR* fileName);
}

// Both SETUP and TEARDOWN must be present.
SETUP(DeleteFileFixture)
{
    // This is the maximum size of the directory passed to GetTempFileName.
    const unsigned int maxTempPath = MAX_PATH - 14;
    TCHAR tempPath[maxTempPath + 1] = _T("");
    DWORD charsWritten = GetTempPath(maxTempPath + 1, tempPath);
    // (charsWritten does not include null character)
    WIN_ASSERT_TRUE(charsWritten <= maxTempPath && charsWritten > 0,
        _T("GetTempPath failed."));

    // Create a temporary file
    UINT tempFileNumber = GetTempFileName(tempPath, _T("WUT"),
        0, // This means the file will get created and closed.
        s_tempFileName);

    // Make sure that the file actually exists
    WIN_ASSERT_WINAPI_SUCCESS(IsFileValid(s_tempFileName),
        _T("File %s is invalid or does not exist."), s_tempFileName);
}

// TEARDOWN does the inverse of SETUP, as well as undoing
// any side effects the tests could have caused.
TEARDOWN(DeleteFileFixture)
{
    // Delete the temp file if it still exists.
    if (IsFileValid(s_tempFileName))
    {
        // Ensure file is not read-only
        DWORD fileAttributes = GetFileAttributes(s_tempFileName);
        if (fileAttributes & FILE_ATTRIBUTE_READONLY)
        {
            WIN_ASSERT_WINAPI_SUCCESS(
                SetFileAttributes(s_tempFileName,
                    fileAttributes ^ FILE_ATTRIBUTE_READONLY),
                    _T("Unable to undo read-only attribute of file %s."),
                    s_tempFileName);
        }

        // Since I'm testing DeleteFile, I use the alternative CRT file
        // deletion function in my cleanup.
        WIN_ASSERT_ZERO(_tremove(s_tempFileName),
            _T("Unable to delete file %s."), s_tempFileName);
    }

    // Clear the temp file name.
    ZeroMemory(s_tempFileName,
        ARRAYSIZE(s_tempFileName) * sizeof(s_tempFileName[0]));
}

BEGIN_TESTF(DeleteFileShouldDeleteFileIfNotReadOnly, DeleteFileFixture)
{
    WIN_ASSERT_WINAPI_SUCCESS(DeleteFile(s_tempFileName));
    WIN_ASSERT_FALSE(IsFileValid(s_tempFileName),
        _T("DeleteFile did not delete %s correctly."),
        s_tempFileName);
}
END_TESTF

BEGIN_TESTF(DeleteFileShouldFailIfFileIsReadOnly, DeleteFileFixture)
{
    // Set file to read-only
    DWORD fileAttributes = GetFileAttributes(s_tempFileName);
    WIN_ASSERT_WINAPI_SUCCESS(
        SetFileAttributes(s_tempFileName,
        fileAttributes | FILE_ATTRIBUTE_READONLY));

    // Verify that DeleteFile fails with ERROR_ACCESS_DENIED
    // (according to spec)
    WIN_ASSERT_FALSE(DeleteFile(s_tempFileName));
    WIN_ASSERT_EQUAL(ERROR_ACCESS_DENIED, GetLastError());
}
END_TESTF

namespace
{
    bool IsFileValid(TCHAR* fileName)
    {
        return (GetFileAttributes(fileName) != INVALID_FILE_ATTRIBUTES);
    }
}

In this example, I'm pretending I was the writer of the Windows DeleteFile function and I'm writing some test functions for it. In my setup function, I first create a temporary file. In my test functions, I exercise the DeleteFile functionality by trying to delete the temporary file. Although I expect the file to have been deleted by the end of one of the tests, I still check whether the file exists and delete it in the teardown function. The teardown should undo whatever the setup did and should not count on the tests succeeding. This ensures that your tests are independent and repeatable.

As you can see, there's a slightly different syntax for tests that use fixtures. Instead of the BEGIN_TEST and END_TEST construction, BEGIN_TESTF and END_TESTF are used, with the fixture name following the name of the test in BEGIN_TESTF. The setup and teardown functions are indicated by the all-caps SETUP and TEARDOWN, with the name of the fixture in parentheses.

You'll notice I've used the WIN_ASSERT_WINAPI_SUCCESS macro a few times in this example. As indicated in Figure 4, this macro is to be used in conjunction with Windows functions whose documentation explicitly says to call GetLastError for more information in the case of failure. The first parameter to the assert should be an expected true statement related to the function call. (It can be the function call itself, if it returns a Boolean.) If the function itself is not part of the expression, it's a good idea to include a message indicating the name of the function that has failed if this assert fires, to make it easier to see at a glance from the output exactly what happened. The assert implementation calls GetLastError and retrieves the message string associated with the error code and adds that to the test results.

To see a fixture used in a different way, see BinaryNumberTest.cpp in the TestSampleLib project. There I used it to open and close a data provider from which I read rows of test data to be used in my tests. In this case, the data provider is just a text file, and each row is a line, but it could just as easily be an XML file or even a database table. You should consider using a similar approach if you are testing functionality that can benefit from large amounts of data being run through it.

One difference between WinUnit fixtures and those of other unit test frameworks is that they are never automatically associated with tests—the association must be explicit. This also means that it is possible to have more than one fixture in a single file.

Running WinUnit

Now let's take a look at how WinUnit works. WinUnit uses one of the only reflection-like features available for native code on Windows: discovering the exports of a DLL. There was a great two-part article by Matt Pietrek in MSDN® Magazine in February and March 2002 ("Inside Windows: An In-Depth Look into the Win32® Portable Executable File Format"), which I referred to extensively in figuring out how to do this. It works for both 32-bit and 64-bit executables, but be sure to use a 64-bit build of WinUnit to run 64-bit test binaries (32-bit executables can't run 64-bit ones).

The command line syntax for WinUnit is:

WinUnit [options] {dllName | directoryName}+

The tool takes one or more DLLs or directories over which all contained DLLs will be enumerated. It discovers the exports and executes the ones whose name—which must be undecorated—starts with TEST_. This restriction is intended to ensure that only functions you meant to run as tests will be run, since the function prototype is assumed. The expected function prototype is as follows:

bool __cdecl TEST_TestName(wchar_t* errorBuffer, size_t bufferSize);

The errorBuffer parameter receives error output from the test; bufferSize is the size of the buffer in wide characters. If the test function returns false (or throws a structured exception handling (SEH) exception), it's considered a failed test, and whatever output was put in the errorBuffer is displayed.

Besides DLLs and directory names, WinUnit takes several optional command-line arguments, which are shown in Figure 6. By default the output of WinUnit goes to the console. Informational messages go to stdout; error messages go to stderr. After WinUnit has processed the specified DLLs, it exits with a code of 0 for success and non-zero otherwise (see Figure 7).

Figure 7 WinUnit Exit Codes (Close Figure 7)

Code | Description
0 | No errors (any tests that were run succeeded).
1 | One or more tests failed.
2 | An unhandled exception caused premature termination.
-1 | Usage error.
Figure 6 WinUnit Command-Line Arguments (Close Figure 6)

Parameter | Description
-q | Do not output to console.
-n | No interactive UI (error dialogs are turned off as much as possible). Recommended for automated builds.
-b | Output to debugger (by default, output goes only to console).
-p string | Only run tests starting with the prefix string (it's case-insensitive).
-x | Ignore the requirement for the TEST_ prefix of exports--process all exports regardless of name. (This only works in conjunction with the -s option, to avoid the danger of running functions with the wrong function prototype.)
-s | Show test names only. Default is to run them.
-o filename | Put the output in the given file. This is unrelated to the -q and -b options—output can go to any or all of the console, a file, or the debugger.
-v number | Run at the given verbosity. 1 is the default. 0 will show you only failing tests and total results.
-l customLoggerString | The custom logger string is the name of a DLL in which you've implemented a custom logger, followed optionally by a colon and an initialization string to be passed to the logger's Initialize function.
--variable value | This option will set environment variable variable to value value for the lifetime of the WinUnit process. This can be used for passing data to tests via the command line.
-e ( exactTestName* ) | (Note: This option is included for completeness but is intended only for automation purposes. Use –p for regular use.) Run only tests specified by their full names (case-sensitively), space-delimited, and enclosed in parentheses.

The BEGIN_TEST macros used in the examples prepend TEST_ to the test names and use 'extern "C" __declspec(dllexport)' to export the function names undecorated. This is the equivalent of putting them in a .def file for the DLL.

The WIN_ASSERT macros work via C++ exceptions; however, all exceptions are caught within the test functions themselves, so WinUnit doesn't care whether exceptions were used or not. I used exceptions because they make it easy to exit out of any block at any point and still ensure that proper cleanup occurs. You could implement your own test functions that would work just as well with WinUnit entirely without macros by using the function prototype shown earlier, ensuring that the names started with TEST_, and exporting them in undecorated form from a DLL.

The BEGIN_TEST and END_TEST macros declare the function and set up a try/catch block. The WIN_ASSERT macros throw an AssertException class exception on failure, which holds a message string describing the error. If a thrown AssertException is caught, the message is copied into the buffer and the function returns false.

Implementation Details

I initially thought it would be elegant to have no parameters to the test functions, and instead just throw exceptions directly out of the functions and catch and process them in the test runner. This would have meant there was no need for an END_TEST macro. Unfortunately, stack unwinding did not work properly when exceptions were thrown across the DLL boundary, so it ended up working out better to have the exceptions as an implementation detail and catch them within the tests themselves.

Implementing the fixture concept was another challenge. My first idea was to rely on C++ automatic storage and declare an object at the top of the test function whose constructor and destructor would do setup and teardown. However, I soon discovered firsthand why you're not supposed to throw exceptions in destructors. Part of the C++ specification states that when an exception is thrown by a destructor function during stack unwind, the terminate function is called. This state could occur if I implemented fixtures as described and had a fixture object with exception-throwing asserts in its destructor. If the asserts in the fixture object's destructor fired during the stack unwind due to an exception thrown in the body of the function, terminate would be called.

The remedy for this was to ensure that the fixture object's destructor would never be called during a stack unwind by putting it alone in its own try/catch block outside the main try/catch for the function. This necessitated the special fixture syntax using the BEGIN_TESTF and END_TESTF pair to house two try/catch blocks and a fixture object declaration.

Running across the unpleasant terminate behavior made me think of something else I also wanted to make sure to handle: I wanted to provide the option of running in non-interactive mode (currently the –n command line option). WinUnit generally runs non-interactively, but I noticed that when the terminate function was called, it put up a dialog box. I thought dialog boxes like this would be inconvenient in automation scenarios because they would potentially block further automated tasks.

You can see what I did to disable such error messages in ErrorHandler.cpp in the WinUnit project. Interestingly, some of the dialogs that can come up in error conditions are associated with the C Runtime (CRT). This means that setting a global variable in the CRT to turn them off isn't going to do any good if the variable is set in the test-running tool (WinUnit) and the test binary is using a different CRT. If you want to ensure that CRT-specific error dialogs will be turned off, you will need to be sure you are using the same Runtime Library option for building both WinUnit and your test binaries, and that it's one of the DLL options. This option can be found in project Properties | Configuration Properties | C/C++ | Code Generation. By default, it is /MD for release builds and /MDd for debug builds. If you build both WinUnit and your test binaries in Visual Studio 2005 with the same option here, you will have the same CRT instance.

I wanted to have some trace functionality to aid in debugging tests. I decided to take advantage of the fact that my tool is meant to run on Windows and that I know the PE format. I therefore hook (override) OutputDebugStringA and OutputDebugStringW in the test DLL being executed, sending the output wherever I want. I borrowed this part heavily from the code that comes with the book Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press®, 2003), by John Robbins, who kindly gave me permission to use it in this project. I then provided a WIN_TRACE macro which funnels its arguments to OutputDebugString.

I also wanted to have some way to pass information from the command line into the tests. I thought this might be useful if, for example, you were using the tool as part of an automated build, and there was some known location that contained test data files, and you wanted the tests to know about that location. I decided on environment variables. Arbitrary variables can be set for the process using the "--" command-line option; they can be retrieved within the tests using WinUnit::Environment::GetString (a wrapper on GetEnvironmentVariable), GetEnvironmentVariable itself, or getenv_s/_wgetenv_s.

Finally, I wanted to offer the ability for a user to write a custom logger. Internally, a logger chain is used, so you can opt to send output to console, to OutputDebugString, to a file, or to any combination of those. A fourth option is to provide your own custom logger, which I'll talk about next.

Going Further...

WinUnit should solve your basic native C++ unit testing needs, but there are a few things you may want to consider adding to it.

You may have noticed the –l command-line option in Figure 4 for passing in a custom logger. You might want to use this feature to send WinUnit output to an XML file or even to a database for use in a reporting system. Also, the default loggers do not write Unicode; the –o option creates an ANSI text file. If you want your output in Unicode, you will need to write a custom logger for that. To implement a custom logger feature, simply create a DLL that implements one or more of a set of ten logger-related functions, and then pass the path to the DLL to WinUnit via the –l option. You can see my project SampleLogger.cpp for more details.

You might also wish to add your own custom asserts, using the existing ones as examples. You could make an additional header file or a static library to be linked into your test DLLs. It might be interesting to implement a more in-depth set of string asserts, similar to the StringAssert class that comes with the Visual Studio unit testing framework for .NET-based code, which includes string comparisons with regular expressions. A FileAssert class could be useful as well—comparing entire files or attributes of files. You'll notice that I used an Assert class with static methods for the implementation of the asserts, and the WIN_ASSERT macros are thin wrappers around them. I would have liked not to have used macros at all, but I wanted to be able to easily include the file and line number. Of course, macros pollute the global namespace, so if you're going to make your own it would be a good idea to start them with a different prefix.

Another handy Visual Studio add-in would be one that allowed you to run tests individually (with or without code coverage) by right-clicking on them, similar to how TestDriven.NET works for unit tests in the .NET Framework. See the sidebar "Code Coverage and Convenience" for a short discussion of code coverage tools.

My goal has been to show that unit testing native C++ can be easy, fun, and you can get started right now. To give you an idea of the power of WinUnit, take a look through the TestWinUnit project, where I used WinUnit to test itself. Those examples are completely real world and will show you advanced usage you can apply to your own unit tests. If you've been struggling with your native C++ unit testing, WinUnit makes it easy—and any time you can make testing easy, you're far more likely to actually do it.

Maria Blees has been a developer at Microsoft for 10 years and has a special fondness for native code and engineering excellence. She can be reached at the address listed in the code and welcomes bug reports and suggestions for this tool.


Silverlight: Create a Custom Expression Encoder Publishing Plug-In

Silverlight

Create a Custom Expression Encoder Publishing Plug-In

Laurence Moroney
This article discusses:
  • Expression Encoder plug-in basics
  • Building a boilerplate plug-in
  • Adding FTP upload functionality
  • Building, installing, and debugging
This article uses the following technologies:
WPF, Visual Studio, Expression Encoder
Contents
Expression Encoder and Plug-Ins
Creating a Plug-In
Building the FTP Upload Plug-In
Implementing the Publish Class
Saving Configuration Settings
Compile, Deploy, and Debug

A
recently released tool, Microsoft® Expression® Encoder helps you encode, enhance, and publish rich media experiences that can be rendered using Microsoft SilverlightTM. Expression Encoder has an extensible plug-in architecture, giving you the ability to write plug-ins that allow for the publication of Expression Encoder output to different sources.

In this article, I will step through the building of a plug-in for Expression Encoder using Microsoft Visual Studio® 2008 Beta 2. This plug-in will fit within the standard workflow of an Expression Encoder session, and, at the end of an encoding session, it will upload the resulting output to an FTP server.

Expression Encoder and Plug-Ins

Figure 1 shows Expression Encoder. The left side of the workspace contains the video preview and a control bar that allows you to set the portion of the video to re-encode. On the right-hand side of the screen, there are three tabs: Settings, for configuring up the encoding session; Metadata, for applying new information to the media; and Output, for taking post-encode actions.

Figure 1 Expression Encoder
Figure 1 Expression Encoder(Click the image for a smaller view)
Figure 1 Expression Encoder
Figure 1 Expression Encoder (Click the image for a larger view)

When a publishing plug-in is present in <Installation Directory>\Plugins and is loaded correctly, the Output tab will contain a Publish pane that lists the available publishing options. Each type requires a separate plug-in. Figure 2 shows that the FTP plug-in is available.

Figure 2 Selecting the FTP Plug-In
Figure 2 Selecting the FTP Plug-In

The top part of the Publish pane is fixed for all plug-ins. The Publish To list is populated from the names of the valid plug-ins detected in the plug-ins directory. The Auto Start Publish checkbox is always available and, if checked, will kick off the publication automatically at the end of an encoding.

The Publish button is grayed out until encoding is complete, at which point it becomes available. By clicking the button, you can then publish all the files associated with the encoding at once. (This could be a single file if no template is used, or all files needed for the video player if a template is used.) The Settings section lets the user specify the details necessary for publishing the output. In the case of an FTP output, the server address, user name, and password are necessary. Plug-ins also provide the facility for an advanced settings area that is not initially visible but can be expanded using the pane expander button.

The Settings pane is free-format, so there's a lot of flexibility available to implement whatever your plug-in needs using XAML and Windows® Presentation Foundation (WPF).

Creating a Plug-In

The first step in constructing a plug-in is to create a new Microsoft .NET Framework 3.0 class library. Then you add the required references that allow you to compile and execute WPF and encoder applications. These are listed in Figure 3.

Figure 3 Required Encoder References (Close Figure 3)

Reference | Location
EncoderTypes | C:\Program Files\Microsoft Expression\Encoder 1.0\EncoderTypes.dll
PresentationCore | C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\PresentationCore.dll
PresentationFramework | C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\PresentationFramework.dll
WindowsBase | C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.0\WindowsBase.dll

A plug-in is simply a standard .NET class that derives from the PublishPlugin base class and is decorated with the EncoderPluginAttribute attribute, which adds the metadata that describes your plug-in. Its constructor takes two strings, the plug-in name and a lengthier description.

Here's the required code:

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Expression.Encoder.Plugins.
  Publishing;

namespace TestPlugin
{
    [EncoderPlugin("Boiler Plate",
      "Boiler Plate App")]
    public class Class1 : PublishPlugin
    {
    }
}

To successfully compile the plug-in, you need to implement the three member functions of the PublishPlugin class. CreateStandardSettingsEditor creates a standard settings dialog and returns an object that describes the UI for it. CreateAdvancedSettingsEditor creates the advanced settings dialog and returns an object that describes the UI for the dialog. PerformPublish is called when the Publish button is pressed. It takes two parameters: root and filesToPublish. Root contains the directory of the files to be published. filesToPublish is an array of paths to the files that Expression Encoder will publish.

Figure 4 shows an example of a boiler plate class that implements these three functions. This class will compile into a DLL. If you copy this DLL to the Plugins directory, it will thereafter be available as a publish plug-in.

Figure 4 Boiler Plate Plug-In (Close Figure 4)
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Expression.Encoder.Plugins.Publishing;

namespace TestPlugin
{
    [EncoderPlugin("Boiler Plate", "Boiler Plate App")]
    public class Class1 : PublishPlugin
    {
        public override void PerformPublish(string root,
            string[] filesToPublish)
        {
        }
        #region IPublishPlugin Members
        public override object CreateStandardSettingsEditor()
        {
            return "Standard Settings";
        }
        public override object CreateAdvancedSettingsEditor()
        {
            return "Advanced Settings";
        }
        #endregion
    }
}

The attribute's constructor arguments define the list entry in the Publish To dropdown list, and the objects returned by the CreateStandardSettingsEditor and CreateAdvancedSettingsEditor functions define the UI for the Settings panes.

Building the FTP Upload Plug-In

Let's start with the simple boilerplate plug-in (which doesn't actually do anything) and expand it into a functional FTP upload plug-in. The standard and advanced parameters panes are implemented in WPF XAML, but they require their data to be implemented in a class that inherits from INotifyPropertyChanged in order for the plug-in to detect changes the user makes to the settings. A single class (in this case PublishData) is sufficient for both parameter panes.

Five properties are implemented for the FTP upload plug-in. UseProxy determines whether system proxy should be used or bypassed. ServerAddress provides the address of the FTP server to upload to. UserName is the login ID for the server, and UserPassword is, of course, the associated password for the login ID. Finally, ServerDirectory is the directory on the server that will receive your uploaded files. Figure 5 shows the complete class that implements these properties.

Figure 5 Properties for the FTP Plug-In (Close Figure 5)
using System;
using System.Collections.Generic;
using System.Text;
using System.Windows;
using System.Windows.Threading;
using System.Threading;
using System.ComponentModel;

namespace FTPPublishPlugin
{
    public class PublishData : INotifyPropertyChanged
    {
        public event PropertyChangedEventHandler PropertyChanged;

        private bool _UseProxy = false;
        public bool UseProxy
        {
            get { return _UseProxy; }
            set
            {
                _UseProxy = value;
                PropertyChanged(this,
                  new PropertyChangedEventArgs("UseProxy"));
            }
        }

        private string _ServerAddress = "";
        public string ServerAddress
        {
            get { return _ServerAddress; }
            set
            {
                _ServerAddress = value;
                PropertyChanged(this,
                  new PropertyChangedEventArgs("ServerAddress"));
            }
        }

        private string _UserName = "";
        public string UserName
        {
            get { return _UserName; }
            set
            {
                _UserName = value;
                PropertyChanged(this,
                  new PropertyChangedEventArgs("UserName"));
            }
        }

        private string _UserPassword = "";
        public string UserPassword
        {
            get { return _UserPassword; }
            set
            {
                _UserPassword = value;
                PropertyChanged(this,
                  new PropertyChangedEventArgs("UserPassword"));
            }
        }

        private string _ServerDirectory = "";
        public string ServerDirectory
        {
            get { return _ServerDirectory; }
            set
            {
                _ServerDirectory = value;
                PropertyChanged(this,
                  new PropertyChangedEventArgs("ServerDirectory"));
            }
        }
    }

}

The structure of this class is straightforward. First, the class has to implement INotifyPropertyChanged to receive property notifications from the UI. It then implements a standard property handler design pattern whereby a private member variable stores the value of the property and is encapsulated by a getter and a setter accessor method. When the getter is called, the value of the private member variable is returned. When the setter is called, the private member is updated and the PropertyChanged event is raised. The PropertyChanged event is initialized with an argument containing the name of the property to be changed. The UI uses XAML data binding to hook into this code, causing user actions to call the getter and setter. An example of the XAML that is used in the dialog for the Server Address property is shown here:

<TextBox Grid.Column="1" Grid.Row="0" Text="{Binding Path=ServerAddress, Mode=TwoWay}" Margin="0,3,0,3" />

The parameter pane's UI controls are implemented using standard WPF XAML user controls. Note that the x:Class for these controls must be set to use the namespace you have been using thus far in your code; for example, they should be FTPPublishPlugin.StandardParams and FTPPublishPlugin.AdvancedParams for the standard and advanced panes, respectively.

These are databound to their associated properties within the PublishData class. This is achieved by setting the PublishData class to be the data context for the user control. When the plug-in instantiates the StandardParams user control, it passes an instance of the PublishData class to the control. The user control then uses this instance to set up the data context, as you see here:

public StandardParams(PublishData data)
{
    this.InitializeComponent();
    this.DataContext = data;

}

Thus, to implement the label and textbox for the Server Address, you simply use the following XAML in the user control:

<Label Grid.Column="0" Grid.Row="0" Margin="0,3,0,3">Server Address
</Label>
<TextBox Grid.Column="1" Grid.Row="0" Text=
"{Binding Path=ServerAddress, Mode=TwoWay}" Margin="0,3,0,3" />

As you can see, the TextBox's text property is bound to the ServerAddress property; the binding mode is two-way, allowing the property to be set and get automatically by WPF.

Handling a password entry field is a little more difficult, as you cannot databind to it for security reasons. In addition, the automatic styling of labels and textboxes that the encoder provides is not available to the password box, so you have to set the styling yourself. Thus, in order to implement a password box, you use the XAML shown in Figure 6 to implement the UI.

Figure 6 Password Box XAML (Close Figure 6)
<PasswordBox PasswordChar="*" Grid.Column="1" Grid.Row="2"
    x:Name="pass" PasswordChanged="HandlePW" Margin="0,3,0,3">
    <PasswordBox.Style>
        <Style TargetType="{x:Type PasswordBox}" >
            <Setter Property="BorderBrush"
                Value="{DynamicResource BackgroundBrush}"/>
            <Setter Property="Foreground"
                Value="{DynamicResource Text1Brush}"/>
            <Setter Property="Background"
                Value="{DynamicResource BackgroundBrush}"/>
            <Setter Property="FontFamily"
                Value="{DynamicResource
                {x:Static SystemFonts.MessageFontFamilyKey}}"/>
            <Setter Property="FontSize"
                Value="{DynamicResource
                {x:Static SystemFonts.MessageFontSizeKey}}"/>
            <Setter Property="FontWeight"
                Value="{DynamicResource
                {x:Static SystemFonts.MessageFontWeightKey}}"/>
        </Style>
    </PasswordBox.Style>
</PasswordBox>

This specifies the event handler for the password change to be HandlePW. You use this event handler to determine the typed password and to store it in the PublishData class that is the current data context. Here's an example of this function:

public void HandlePW(object sender, RoutedEventArgs e)
{
    PublishData d = (PublishData) this.DataContext;
    d.UserPassword = pass.Password;
}

Implementing the Publish Class

Now that the foundations are laid, creating the publish class for an FTP server is straightforward. The first thing you'll need to do is to ensure that an instance of the PublishData class is created with the plug-in. The natural place to do this is in the constructor for the plug-in:

private StandardParams _standardParams;
private PublishData _publishData;
public Publish() : this(new PublishData())
{
}
public Publish(PublishData publishData)
{
    _publishData = publishData;
    _standardParams = new StandardParams(_publishData);
}

As you can see, this also initializes the StandardParams user control, so it will display when the plug-in is rendered. It passes the instance of the PublishData class (called _publishData) to the user control; this will be used to set the data context, as outlined in the previous step.

The next thing you do is to implement the required functions for creating the standard- and advanced-settings editor panes. If you recall the boilerplate example, these returned strings that were then rendered. Now you want to render the user controls for the respective settings panes:

public override object CreateStandardSettingsEditor()
{
    return _standardParams;
}
public override object CreateAdvancedSettingsEditor()
{
    return new AdvancedParams(this, _publishData);
}

The final step is to implement the Publish function, which gets called when the user requests publication of the output, either by pressing the Publish button or by selecting the Auto Publish option. In this case, you are going to publish to FTP, so the code for connecting to the server is implemented here. Figure 7 shows an example.

Figure 7 Connecting to the FTP Server (Close Figure 7)
public override void PerformPublish(string root, string[] filesToPublish)
{
    try
    {
        int nFileCount = filesToPublish.Length;
        int nFileProgressSize = 100 / nFileCount;
        int nCurrentProgress = 0;
        WebClient wc = new WebClient();
        if (!_publishData.UseProxy)
            wc.Proxy = null;
        wc.BaseAddress = "ftp://" + _publishData.ServerAddress + "/";
        string strURI = "";
        wc.Credentials =
        new NetworkCredential(_publishData.UserName,
            _publishData.UserPassword);

        foreach (string theFile in filesToPublish)
        {
            if (!m_Cancelled)
            {
                strURI = theFile.Substring(theFile.LastIndexOf("\\") + 1);
                if (_publishData.ServerDirectory != String.Empty)
                    strURI = _
                        publishData.ServerDirectory + "\\" + strURI;
                nCurrentProgress += nFileProgressSize;
                OnProgress(strURI, nCurrentProgress);
                wc.UploadFile(strURI, theFile);
            }
        }

        if (m_Cancelled)
        {
            PublishingHost.ShowMessageBox("Upload Cancelled",
                System.Windows.MessageBoxButton.OK,
                System.Windows.MessageBoxImage.Error);
        }
    }

    catch (Exception ex)
    {
        string strIE = ex.Message;
        if (ex.InnerException != null)
            strIE += "\n: " + ex.InnerException.Message;

        throw new Exception(strIE);
    }

    protected override void CancelPublish()
    {
        m_Cancelled = true;
        base.CancelPublish();
    }
}

As you can see, this is fairly straightforward C# code for managing the upload of a set of files to an FTP server. The required data for the FTP server is taken from the _publishData class, which in turn is set by the controls discussed earlier.

One function of note is OnProgress, which is part of the Plugins API and is not necessary to implement. You simply call this function with a string and a number from 0 to 100. It then renders the string and a bar representing the value as a percentage of progress. In this case, the progress is determined by the number of files uploaded, so if, for example, there are 10 files, each is considered 10 percent of the progress regardless of the file size.

Also take note of the CancelPublish function. The API provides this function for you to override. It will be fired when the user clicks the Cancel button that is automatically generated by the UI when you are publishing. In this case, it simply sets the m_Cancelled Boolean to true and cancels publication.

Note that as this code is just a sample application, it has a single exception handler that captures all exceptions and throws them up to the user. In a real application, you should provide more detailed error information for the various exceptions your application may encounter.

Saving Configuration Settings

In order to keep the user from constantly having to enter credentials for the FTP server being published to, you can override the SaveJobSettings and LoadJobSettings methods from the PublishPlugin base class. These methods will be called on the plug-in instance by Expression Encoder when the user calls Save Job from the file menu, and they allow you to serialize and restore your configuration properties. For example, to save your configuration properties, you would write the following:

public override void SaveJobSettings(XmlWriter writer)
{
  writer.WriteStartElement("FtpSample");
  writer.WriteElementString("UseProxy",
      _publishData.UseProxy.ToString());
  writer.WriteElementString("ServerAddress",
      _publishData.ServerAddress);
  writer.WriteElementString("UserName", _publishData.UserName);
  writer.WriteElementString("ServerDirectory",
      _publishData.ServerDirectory);
  writer.WriteEndElement();
}

One thing to note here is that you must nest your configuration properties within a top level XML Element. This is because your properties are serialized into your encoding job's XML job file, which contains lots of additional data. You can restore your saved properties with the following:

public override void LoadJobSettings(XmlReader reader)
{
  reader.ReadStartElement("FtpSample");
  _publishData.UseProxy = bool.Parse(xr.ReadElementString("UseProxy"));
  _publishData.ServerAddress = xr.ReadElementString("ServerAddress");
  _publishData.UserName = xr.ReadElementString("UserName");
  _publishData.ServerDirectory = xr.ReadElementString("ServerDirectory");
  reader.ReadEndElement();
}

Compile, Deploy, and Debug

Should everything compile happily, you will have three files in your projects bin\debug directory: your plug-in DLL, a .pdb file for your DLL, and EncoderTypes.dll. Copy only the plug-in DLL file to the Plugins folder. Now, when you launch Expression Encoder, the plug-in should be loaded correctly.

After loading Expression Encoder, return to Visual Studio and select Debug | Attach to Process. The Attach to Process dialog will appear, and from there select MediaEncoderUI.exe.

If you want to inspect the publication process, you can set a breakpoint in the Publish function. Then, after completing an encoding job, if you press the Publish button in Expression Encoder, you will hit your breakpoint and will be able to step through and see what is happening as Expression Encoder publishes your files.

Laurence Moroney is a Senior Technology Evangelist with Microsoft, specializing in Silverlight. He is the author of many books on computing topics, including Silverlight, AJAX, interoperability, and security. His blog is at blogs.msdn.com/webnext.


PIAB And WCF: Integrating the Policy Injection Application Block with WCF Services

PIAB And WCF

Integrating the Policy Injection Application Block with WCF Services

Hugh Ang and David San Filippo
Code download available at: WCFandPIAB2008_02.exe (196KB)
This article discusses:
  • Integrating the Policy Injection Application Block and WCF
  • Creating Custom WCF Behaviors
  • Applying behaviors using .NET attributes or configuration
  • Integration without changing any code
This article uses the following technologies:
WCF, PIAB
Contents
A Real-World Example
WCF Architecture Overview
Dispatcher
Extending IEndpointBehavior and IContractBehavior
The Configuration Approach
The Attribute Approach
The Sample Code

O
ne of the most important software design principles is the separation of different responsibilities within our applications. In service-oriented design, we separate applications into services and operations. In real life, however, implementation concerns tend to leak into the services.

This is a problem not only in service-oriented design, but in object-oriented design as well. Enterprise Library 3.0 introduced the Policy Injection Application Block (PIAB) to solve this problem in object-oriented design. At the time of this writing, however, the latest version of Enterprise Library (3.1) doesn't directly support the integration of PIAB with Windows® Communication Foundation (WCF) services, so using PIAB to separate concerns in service-oriented applications might seem daunting. But, as you'll see, it's easy enough with the right techniques. Let's get started.

To leverage the PIAB, you must have control over how your objects are instantiated. With service frameworks like WCF that abstract object instantiation away from the developer, this creates a problem when trying to integrate the PIAB. However, WCF provides a variety of extensibility points through behaviors. Here we will show you how to leverage custom WCF behaviors in order to apply PIAB at the WCF service point without requiring additional code. With WCF service successfully integrated, PIAB can truly become a ubiquitous framework for separating service logic from cross-cutting implementation details.

A Real-World Example

To better understand this leakage problem, imagine you were building a simple service to find a custom customer object using a customer ID. You write a simple method in Visual Studio® 2005 to invoke a data access object and return a customer object:

public Customer GetCustomer(string customerID)
{
    return CustomerDAO.GetCustomer(customerID);
}

Then you remember the customer ID field cannot be empty. So you add a few lines for validation. You also add a few tracing statements so you can make sure the correct values are being passed to and returned from the data access object. Finally you add exception handling to wrap any exceptions thrown from the data access object with a custom exception. Next thing you know, you have 15 lines of code. And it could have been worse. Authorization logic, caching logic, or logic to update performance counters would have added even more code bloat.

Yet none of these additions has anything to do with retrieving a customer object. Ultimately, what you get for all this work is code that is more difficult to understand, maintain, and reuse. Unfortunately, examples like this are common in the real world, especially when business logic spans more than one line of code.

Using the PIAB, you would only need a single line of code in your method. The code for validation, exception handling, and so forth would be introduced through the creation of policies that define the implementation details needed at run time. Policies can be applied to objects through configuration or by decorating classes with attributes. Each policy has a set of matching rules to determine where it should be applied, and a collection of handlers that determine which behaviors should be introduced when invoking methods on the object. The PIAB comes with a number of useful handlers for validation, logging, exception handling, managing performance counters, authorization, and caching. Many of these handlers leverage Enterprise Library's other application blocks.

In order to consume an object that uses these policies, you must create an instance of the class using the PolicyInjectorFactory. Instead of instantiating the class using the "new" operator, you would pass the type to the Factory's "Create" method. The factory will then determine which policies apply to that type using the matching rules. PIAB supports matching rules based on types, assemblies, member names, method signatures, namespaces, parameter types, property types, return types, custom attributes, tags (PIAB gives you a tag attribute that can be set to a value for matching), and your own custom matching rules.

If no matching rules apply, the factory will simply instantiate the type and return it; otherwise it creates a proxy to that object. The proxy intercepts calls to the object and invokes the handlers in a pipeline, as shown in Figure 1.

Figure 1 Handlers Pipeline
Figure 1 Handlers Pipeline(Click the image for a smaller view)
Figure 1 Handlers Pipeline
Figure 1 Handlers Pipeline (Click the image for a larger view)

The PIAB is exactly what we need to keep the GetCustomer service method to a single line of code. We just need to define a policy to match the "GetCustomer" operation and include handlers for logging, validation, and exception handling.

Now that you understand PIAB policy, matching rules, and handlers, you need to tackle the problem of integrating this solution into WCF. Let's see how you could still leverage PIAB if the GetCustomer operation were defined in a WCF service.

WCF Architecture Overview

WCF is the Microsoft platform for building services that support a variety of transports including TCP/IP, HTTP, MSMQ, named pipes, and WS-* protocols. At a high level, the WCF architecture consists of a service model layer and a channel layer at both the sender and receiver, as shown in Figure 2. The service model layer provides the API for user code to interface with WCF; the channel layer handles the messaging and transport details.

Figure 2 WCF Architecture
Figure 2 WCF Architecture(Click the image for a smaller view)
Figure 2 WCF Architecture
Figure 2 WCF Architecture (Click the image for a larger view)

When you host a WCF service, you have to define the address, binding, and contract—the ABCs—that define the service's endpoint. These ABCs can be defined in either code or the configuration file, and they effectively control the channel layer. WCF takes care of building up the channels needed for the binding and protocol, listens for incoming messages, and sends them to the intended service instances through invocation of its methods.

As discussed previously, to enable PIAB for a type, its instance needs to be created via the PIAB factory. When hosting a WCF service, you only provide the type of the service class. This implies that the WCF runtime will be responsible for creating an instance of the service when it is needed. So how can we go about altering this behavior so that a service class instance can be created by the PIAB factory within the WCF runtime? In order to do so, we need to delve deeper into the WCF runtime architecture and find out how WCF instantiates the service.

Dispatcher

Take a look at the receiving application in Figure 2. The diagram shows a simplified view of the physical WCF dispatcher model, as there actually exist channel dispatchers and endpoint dispatchers. Ignoring that detail doesn't affect what we're trying to accomplish, and it makes our discussion a lot easier. Note that the dispatcher sits within WCF's service model layer. The dispatcher plays an essential role for WCF, as it is responsible for deserializing the WCF message, creating the service instance, and eventually dispatching the call. Upon creating the dispatcher object, the WCF runtime initializes an instance behavior object that contains an instance provider object. The instance provider is responsible for instantiating the service. This process is shown in Figure 3.

Figure 3 Service Instantiation
Figure 3 Service Instantiation(Click the image for a smaller view)
Figure 3 Service Instantiation
Figure 3 Service Instantiation (Click the image for a larger view)

Behaviors are the main extensibility point for WCF. By creating our own behaviors, we can alter the way WCF creates the instance of the target service, thereby putting the PIAB mechanism in place. Figure 4 shows how the PIAB is introduced when we create our own custom behavior and custom instance provider.

Figure 4 Introducing the PIAB
Figure 4 Introducing the PIAB(Click the image for a smaller view)
Figure 4 Introducing the PIAB
Figure 4 Introducing the PIAB (Click the image for a larger view)

Now, let's create a custom behavior and instance provider and plug them into the WCF runtime—we will demonstrate this with both configuration data and Microsoft® .NET Framework attributes. You will see that no code is necessary to make WCF instantiate services using the PIAB. Note that our custom behavior and instance provider are generically designed so that they are not tightly coupled with service types. This means you can download the code and use it for any services to which you would like to apply cross-cutting logic via the PIAB.

Extending IEndpointBehavior and IContractBehavior

Before we create our custom behavior, we must understand how WCF behaviors are discovered and applied, particularly in the context of instantiating services. When the WCF runtime is instructed to host a service, the service model layer will go through the following process:

  1. Find behaviors in configuration.
  2. Find behaviors defined as attributes on the service class or operation.
  3. Build channel listeners for handling messaging and transport.
  4. Apply dispatch behaviors for both the configuration and attributes described in Steps 1 and 2.

For our purposes, we need to define configuration or attributes that can be applied to our service code so that our custom behavior can be found by the WCF runtime. In Step 4 of the process, the WCF runtime will apply the dispatch behavior as defined in custom behaviors. This is our chance to substitute the default instance provider with our own instance provider.

Regarding behaviors, WCF exposes the following interfaces: IServiceBehavior, IEndpointBehavior, IContractBehavior, and IOperationBehavior. IServiceBehavior deals with behaviors at the host level, and IOperationBehavior is limited to the operation's scope, so they are not suitable for creating a service instance. Both IEndpointBehavior and IContractBehavior require the implementation of the ApplyDispatchBehavior method, and it is from within this method where we can make the switch and substitute our own custom instance provider.

So if both IEndpointBehavior and IContractBehavior offer the extensibility point for us to create a service instance via the PIAB factory, which one do we use? It turns out that we will use both. IEndpointBehavior can be used to extend the dispatch behavior in a configuration file, and IContractBehavior can be used to extend the dispatch behavior as a .NET attribute.

The Configuration Approach

The WCF configuration simply piggybacks on the .NET Framework configuration settings. In the app.config or web.config file of the .NET application that hosts the WCF runtime, there is a <system.serviceModel> section, which you will see later.

While seemingly complex, it is really well aligned with the WCF architecture, where service and client contain endpoints that consist of address, binding, and contract definitions.

Elements such as <behaviors>, <extensions>, and so on are simply details that are referenced by <service> and <endpoint> elements. In the configuration, behavior customization can be performed on a <service> or an <endpoint> element. The <behaviors> element contains <serviceBehaviors> and <endpointBehaviors>. Because the service level is not the appropriate place for us to customize behavior to instantiate a service type, we will provide a custom endpoint behavior.

As shown in Figure 5, our custom behavior will implement the IEndpointBehavior interface. Additionally, in order for it to be applied through configuration, it must derive from System.ServiceModel.Configuration.BehaviorExtensionElement and override the BehaviorType property and the CreateBehavior method.

Figure 5 Custom Behavior (Close Figure 5)
public class PolicyInjectionEndpointBehavior : BehaviorExtensionElement,
    IEndpointBehavior
{

    public override Type BehaviorType
    {
        get { return typeof(PolicyInjectionEndpointBehavior); }
    }

    protected override object CreateBehavior()
    {
        return new PolicyInjectionEndpointBehavior();
    }

    public void ApplyDispatchBehavior(ServiceEndpoint endpoint,
        EndpointDispatcher endpointDispatcher)
    {
        Type contractType = endpoint.Contract.ContractType;
        endpointDispatcher.DispatchRuntime.InstanceProvider = new
            PolicyInjectionInstanceProvider(contractType);
    }

    // ...
}

ApplyDispatchBehavior is the key method in IEndpointBehavior, as it allows for our custom endpoint behavior to substitute the WCF default instance provider with our custom instance provider—PolicyInjectionInstanceProvider. As shown in Figure 6, our provider implements the System.ServiceModel.Dispatcher.IInstanceProvider interface. The GetInstance method will then be called by the WCF runtime in order to get the instance of the service type. Our implementation of GetInstance is straightforward; it asks for the Enterprise Library's PIAB to create a policy injection proxy based on the service type and contract type (interface) of the received message. Because the default PIAB interception mechanism uses .NET remoting's transparent proxy, a type derived from MarshalByRefObject is required if no specific contract type (interface) is specified. Typically you don't have to worry about this requirement when using PIAB with WCF services since WCF contracts are .NET interfaces.

Figure 6 PolicyInjectionInstanceProvider (Close Figure 6)
public class PolicyInjectionInstanceProvider : IInstanceProvider
{
  private Type serviceContractType;
  private static PolicyInjectorFactory fac = new PolicyInjectorFactory();
  private static PolicyInjector injector;

  public PolicyInjectionInstanceProvider(Type t)
  {

    this.serviceContractType = t;
    if (injector == null)
        injector = fac.Create();
    }

  public object GetInstance(InstanceContext instanceContext,
    System.ServiceModel.Channels.Message message)
  {
    Type type = instanceContext.Host.Description.ServiceType;

    if (serviceContractType != null)
      return injector.Create(type, serviceContractType);
    else
    {
      return injector.Create(type);
    }
  }

  public object GetInstance(InstanceContext instanceContext)
  {
    return GetInstance(instanceContext, null);
  }

public  void  ReleaseInstance(InstanceContextinstanceContext,  object  instance)
  {
    IDisposable disposable = instance as IDisposable;
    if (disposable != null)
    {
      disposable.Dispose();
    }
  }

Now with these two custom types for IEndpointBehavior and IInstanceProvider ready, we can plug them into the configuration code, as shown in Figure 7. Note that in the <BehaviorExtensions> element, we add the type definition for PolicyInjectionBehavior and, we apply it to the endpoint for our service. Along with the PIAB configuration, we are now able to inject policies into the CustomerService service without writing any additional lines of code.

Figure 7 Configuration File (Close Figure 7)
<system.serviceModel>
  <services>
    <service name="CustomerService.CustomerService">
      <endpoint
        address="http://localhost:8000/CustomerService/
          CustomerService.svc"
        binding="wsHttpBinding"
        bindingConfiguration=""
        contract="CustomerService.ICustomerService"
        behaviorConfiguration="PolicyInjectionEndpointBehavior" />
    </service>
  </services>
  <behaviors>
    <endpointBehaviors>
      <behavior name="PolicyInjectionEndpointBehavior" >
        <PolicyEndpointBehavior />
      </behavior>
    </endpointBehaviors>
  </behaviors>
  <extensions>
    <behaviorExtensions>
      <add
        name="PolicyInjectionEndpointBehavior"
        type="PolicyInjectionBehaviors.PolicyInjectionEndpointBehavior,
          PIBehaviors, Version=1.0.0.0, Culture=neutral,
          PublicKeyToken=null" />
    </behaviorExtensions>
  </extensions>
</system.serviceModel>

The Attribute Approach

In .NET, another common method of declarative programming (in addition to the use of a configuration file) is to use attributes, which get compiled into the assembly as metadata and are visible to the CLR at run time. In Figure 8, we define a class, PolicyInjectionBehaviorAttribute, that implements System.ServiceModel.Description.IContractBehavior and System.ServiceModel.Description.IContractBehaviorAttribute interfaces. It will need to derive from the Attribute base class of .NET Framework in order to be applied as an attribute.

Figure 8 PolicyInjectionBehaviorAttribute Class (Close Figure 8)
[AttributeUsage(AttributeTargets.Class)]
public class PolicyInjectionBehaviorAttribute : Attribute,
  IContractBehavior, IContractBehaviorAttribute {
  public PolicyInjectionBehaviorAttribute() { }

  public Type TargetContract {
    get {
      return null; //return null so we apply to all contracts
    }
  }

  public void AddBindingParameters(ContractDescription description,
    ServiceEndpoint endpoint,
    BindingParameterCollection parameters) { }

  public void ApplyClientBehavior(ContractDescription description,
    ServiceEndpoint endpoint,
    ClientRuntime clientRuntime) { }

  public void ApplyDispatchBehavior(ContractDescription description,
    ServiceEndpoint endpoint,
    DispatchRuntime dispatch) {
    Type contractType = description.ContractType;
    dispatch.InstanceProvider = new
      PolicyInjectionInstanceProvider(contractType);
  }

  public void Validate(ContractDescription description,
    ServiceEndpoint endpoint) { }
}

The implementation for IContractBehaviorAttribute is very simple—we're returning null in the TargetContract property, as we're targeting any contract that has the PolicyInjectionBehaviorAttribute attached. The implementation of ApplyDispatchBehavior is similar to that of PolicyInjectionEndpointBehavior. We substitute the default instance provider with our own PolicyInjectionInstanceProvider. This is the same instance provider that is used for the endpoint behavior for the configuration approach. The code below shows how to inject policies through this attribute. Notice that no other effort is required from the developer besides applying this attribute:

[PolicyInjectionBehavior]
public class DecoratedCustomerService : ICustomerService {
  public Customer GetCustomer(string customerID) {
    return CustomerDAO.GetCustomer(customerID);
  }
}

The Sample Code

To demonstrate the use of the configuration-driven and attribute-based implementations of our custom WCF Behaviors, we created a sample that illustrates both. In the sample, we defined the service contract as an interface named ICustomerService. We then created two concrete implementations of that interface: CustomerService and DecoratedCustomerService. The only difference between these two classes is that DecoratedCustomerService has been decorated with the PolicyInjectionBehavior attribute.

Our sample code includes a console application that hosts both CustomerService and DecoratedCustomerService. We modified the configuration of the CustomerService class to introduce our PolicyInjectionEndPointBehavior. We also included a client application that calls each service once to perform GetCustomer functions. Here we show you callCustomerService, but callDecoratedCustomerService looks the same:

private static void callCustomerService() {
  CustomerService.CustomerServiceClient client =
    new CustomerService.CustomerServiceClient();
  Console.WriteLine("Retreiving Customer 1");
  CustomerService.Customer cust = client.GetCustomer("1");
  Console.WriteLine("Retreived Customer, Name: [" + cust.Name + "]");
}

For both the configuration and attribute approaches, we set up the PIAB configuration (see Figure 9) to include a logging handler for calls made to the services. You can add any other handlers, such as exception handling and caching, if you like. The logging handler in our sample writes audit trails to a flat file called audit.log. After the calls are made, audit.log is created in the same directory as the executing code unless the configuration is modified to create it elsewhere.

Figure 9 App.config in Visual Studio
Figure 9 App.config in Visual Studio(Click the image for a smaller view)
Figure 9 App.config in Visual Studio
Figure 9 App.config in Visual Studio (Click the image for a larger view)

As you can see, the PIAB is a good way to separate cross-cutting logic from domain-specific logic, and it can be used by WCF services. To give it a try, download the sample code from msdn.microsoft.com/msdnmag/code08.aspx. To apply the PIAB to your WCF Services, just add a reference to the PIBehaviors assembly and then either apply the PolicyInjectionEndpointBehavior via configuration or apply the PolicyInjectionBehaviorAttribute to your services directly.

Hugh Ang is a Senior Solution Architect at Avanade and is working as VP of Application Development for a client, overseeing its enterprise architecture as well as managing the application development department. You can read his blog at tigerang.blogspot.com.

David San Filippo is a Solutions Developer at Avanade and holds MCPD Enterprise Application Developer, MCSD.Net, and MCDBA credentials. You can read his blog at www.mtelligent.com.


WF How-To: Building State Machines with Windows Workflow Foundation

WF How-To

Building State Machines with Windows Workflow Foundation

Keith Pijanowski
This article discusses:
  • Sequential workflows vs. state machine workflows
  • When to use state machine workflows
  • Building and designing state machine workflows
  • Raising events to running workflows
  • Passing data to running workflows
This article uses the following technologies:
Windows Workflow Foundation, Visual Studio
Code download available at: WFStateMachines2008_02.exe (177KB)
Contents
State Machine Theory
State Machines and Windows WF
Communicating with State Machine Workflows
Events Required for Data Exchange and State Transitions
Defining an External Data Exchange Interface
Adding EventDriven Activities
Referencing the External Data Exchange Interface
Implementing an External Data Exchange Service
Add an External Data Exchange Service to the Workflow Runtime
Using the External Data Exchange
Useful State Machine APIs
Tying Up Loose Ends

T
he Microsoft® .NET Framework 3.0 introduced new capabilities for visually and declaratively building workflows that can be hosted by managed code. Out of the box, the .NET Framework 3.0 allows developers to build both sequential workflows and state machine workflows. Much has been written about sequential workflows because they closely resemble conventional programming techniques.

State machine workflows, on the other hand, represent a different way of thinking about program logic. When designed and implemented correctly, they are just as valuable as sequential workflows. Furthermore, this style of workflow is a good starting point for building human workflows. Consequently, state machine workflows should be a part of every architect's and developer's skill set.

State Machine Theory
Additional Resources

The code download for this article contains all the workflows, code figures, and code snippets shown in this article. It is designed as a sandbox or test arena for state machine workflows; consequently it will only run one instance of a workflow at a time.

This is a Visual Studio 2005 solution that contains the following projects:
StateMachineLibrary contains the two workflows presented here. These workflows are the SimpleStateMachine and the SupplyFulfillment workflows.
ConsoleHost is a console application used to host the simple state machine workflow presented at the beginning of this article.
WinFormHost is a Windows Forms application used to host the SupplyFulfillment workflow. It is in this project that the ExternalDataExchange is used to raise events to an instance of the SupplyFulfillment workflow.
ExternalDataExchange is a class library that contains the IEventService interface, the implementation of this interface—EventService, and the SupplyFulfillmentArgs class. It is this service that allows external events (and data) to be sent to the current state of a state machine workflow.

If you would like to know more, the following resources will be useful to you:

  • Using SqlTrackingService
  • Using SqlWorkflowPersistenceService
  • A Tour of Windows Workflow Activities
  • Simplify Development with the Declarative Model of Windows Workflow Foundation
  • Windows Workflow Foundation
  • Windows Workflow Foundation, Part 2

Business logic that is implemented as a state machine workflow follows a very different path than business logic implemented as a sequential workflow. Rather than flowing from activity to activity like a sequential workflow, state machines transition from state to state. Think of a state as a well-known step, stage, or even a status within a business process. States are useful because they indicate how much of a workflow has completed and how much must still occur. This lends itself very well to workflows that are long running.

A good indication that a business process is suited for a state machine workflow is that the requirements describe data or logic as moving in discreet steps (or states) from some starting point to some ending point. Take a look below at the description of a supply fulfillment business process that could be implemented as a state machine.

  1. Employees fulfilling customer orders may need additional supplies to fulfill an order. Requests for additional supplies should be submitted to a central system for processing.
  2. When a request is submitted, it will be assigned to a specific user's work queue for review.
  3. Once the request has been assigned, it will be either approved or rejected based on criteria external to the business process.
  4. A request that is rejected will be either reassigned or canceled based on criteria external to the business process. Reassigned orders will be processed the same as assigned orders. Canceled orders will be considered completed.
  5. If the employee request is approved, the resource that is required will be ordered.
  6. Once the order is received, the business process surrounding that order is considered complete.

Key steps of this business process that can be considered states are shown in Figure 1.

Figure 1 Logic Design of the Supply Fulfillment Workflow
Figure 1 Logic Design of the Supply Fulfillment Workflow

There are a few key aspects that are worth noting before building this workflow using Windows® Workflow Foundation (Windows WF). First, the workflow has exactly one initial state and exactly one completed state: submitted and completed, respectively. Every instance of a state machine workflow begins in the state that has been designated as the initial state and ends when the workflow reaches the state designated as the completed state.

Second, state transitions are predefined. In other words, part of the definition of each state is a list of allowable next states. Figure 2 lists the states and the allowed transitions.

Figure 2 Allowable State Transitions for the Supply Fulfillment (Close Figure 2)

State | Allowed Transitions
Submitted | Assigned
Assigned | Approved or Rejected
Approved | Ordered
Rejected | Assigned or Completed
Ordered | Completed
Completed | (None)

Third, backtracking is allowed. Notice that our requirements specify that rejected requests can be reassigned. A reassigned state could have been placed into the logical design, but this would not have been an efficient design. The requirements specify that reassigned requests are to be processed in the same manner as newly assigned requests, so there is no need to create a separate branch off of the rejected state for handling reassigned requests. In other words, transitioning a workflow back to a previous state allows for better reuse within the workflow.

Fourth, state machine workflows are suited for long-running processes. If a state machine workflow flows from its initial state to its completed state in a matter of milliseconds and never pauses for external input, then the workflow could be more efficiently implemented using a sequential approach.

State Machines and Windows WF

Before implementing the logical design discussed earlier, let's explore a simple Windows WF state machine to get a feel for all the tools WF provides. I'll build a simple state machine workflow as an introduction to the design-time experience of Visual Studio® .NET and the tools in the .NET Framework 3.0. Note that all the workflows and code snippets shown here come from a working Visual Studio solution that is packaged as a code download for this article. For more information, see the "Additional Resources" sidebar.

When you install the Visual Studio 2005 extensions for .NET Framework 3.0 (Windows WF) from go.microsoft.com/fwlink/?LinkId=105219, several new project templates are added to the Visual Studio .NET 2005 environment (see Figure 3). Note that if you are using Visual Studio 2008, all of these project templates have been included as a part of the core product. Once you select one of the state machine project templates, a Visual Studio .NET project will be set up with the correct references and a workflow file that can be used as a starting point for the development of a state machine workflow.

Figure 3 Workflow Foundation Project Templates
Figure 3 Workflow Foundation Project Templates(Click the image for a smaller view)
Figure 3 Workflow Foundation Project Templates
Figure 3 Workflow Foundation Project Templates (Click the image for a larger view)

The designer for a state machine workflow is much more restrictive than the designer for a sequential workflow. Most state machines will only require the State activity to be dragged onto the designer. (It is also possible to add an EventDriven activity to the designer for handling events that are not associated with a specific state.) The State activity represents a step, stage, or status within a state machine workflow. This activity is used to represent the initial state, the completed state, and all the potential states that may occur in between.

The first step in building a state machine is to place all the needed states on the design surface by dragging the state activity from the workflow toolbox and assigning each state a meaningful name. Workflow states are named using the properties dialog in the same way that Windows controls are named in a Windows Forms project. Figure 4 is an example of a simple state machine workflow containing three states: Submitted, Ordered, and Completed.

Figure 4 Simple State Machine Workflow
Figure 4 Simple State Machine Workflow(Click the image for a smaller view)
Figure 4 Simple State Machine Workflow
Figure 4 Simple State Machine Workflow (Click the image for a larger view)

Workflow Foundation needs to know the initial state and the completed state for the purposes of proper run-time management. Therefore, the next step is to specify them via the property dialog of the Workflow itself. To view this dialog make sure that the property dialog is visible within Visual Studio .NET and then click on the design surface of your workflow, or just right-click the workflow designer and select properties. The lower right-hand corner of Figure 4 shows the property dialog for a state machine workflow. The InitialStateName and CompletedStateName properties are used to specify the initial state and the completed state respectively. Once these properties are specified, the designer will modify the graphical depiction of the specified states. Look closely at Figure 4 and you'll see the green light icon on the initial state and the red light icon on the completed state.

The State activity is a composite activity, meaning that it can contain other activities. Once again there are restrictions to consider. A state activity may contain only one StateInitialization activity, only one StateFinalization activity, one or more EventDriven activities, and one or more other State activities for nested states. These are the only four out-of-the-box activities that can be placed inside a state activity. Nested states are beyond the scope of this article and the use of the EventDriven activity will be described in the next section. Our simple state machine example will only utilize the StateInitialization activity and the StateFinalization activity.

Experimenting with the StateInitialization and StateFinalization activities is a good way to get a feel for how to work with the state machine designer. Playing around with these two activities will also provide a deeper understanding of how to control the flow of a state machine workflow.

All of the states in the simple example in Figure 4 have been equipped with a StateInitialization activity and a StateFinalization activity—with the exception of the completed state. It turns out that once a state is specified as completed, no activities can be added to it. This is because the act of transitioning to the completed state signifies the end of a workflow's lifecycle.

When program control is passed to a state activity and it begins execution, the state activity will check to see whether a StateInitialization activity is present. If one is present, it will be executed. Similarly, when a state activity is about to pass control to another state, it will check to see if a StateFinalization activity is present. If so, it will be executed. It is important to note that both the StateInitialization activity and the StateFinalization activity are also composite activities. Unlike the state activity, there are very few restrictions on what can be placed inside these activities. In other words, all the activities that can be used in a sequential workflow can be placed inside these activities. Visual Studio will provide a different design view for placing logic into these composite activities. Figure 5 shows the design view that is displayed after double-clicking on the StateInitialization activity of the Submitted state shown in Figure 4. Here the Submitted state has been equipped with a StateInitialization activity named SubmittedInitialization. The purpose of this design view is to allow additional activities to be placed inside the StateInitialization activity. In the example in Figure 5, a Code activity and a SetState activity have been placed within the SubmittedInitialization activity. The following code has been placed into the execute event of the code activity:

Figure 5 Design View for a StateInitialization Activity
Figure 5 Design View for a StateInitialization Activity

private void codeSubmittedInitialization_ExecuteCode(object sender,
    EventArgs e)
{
    Console.WriteLine("The Submitted state has been initialized at " +
    DateTime.Now.ToString() + ".");
}

The SetState activity (setStateOrdered in Figure 5) is used to transition control to another state in the StateMachine workflow. No code is necessary to utilize this activity. Figure 6 shows the properties dialog for setStateToOrdered. The TargetStateName property must be set to a valid state in the StateMachine workflow. Immediately after this activity executes, the current state's StateFinalization activity will execute and control will pass to the Ordered state. No additional activities will execute within the SubmittedInitialization activity. Therefore, care must be taken to insure that it is the last activity. The StateFinalization activity for our Submitted State has been configured with a single code activity with the following code in its execute event:

Figure 6 Properties for a SetState Activity
Figure 6 Properties for a SetState Activity(Click the image for a smaller view)
Figure 6 Properties for a SetState Activity
Figure 6 Properties for a SetState Activity (Click the image for a larger view)

  private void
  codeSubmittedFinalization_  ExecuteCode(
  object sender, EventArgs e)
  {
    Console.WriteLine("The Submitted " +
    "state has been finalized at " +
    DateTime.Now.ToString() + ".");
  }

It is important to note that a SetState activity cannot be placed within a StateFinalization activity. This wouldn't make sense anyway since the only way a StateFinalization activity can execute is if a SetState has already executed. A SetState activity can only reside within a StateInitialization or an EventDriven activity.

To return the designer to the original design view, which shows the entire workflow, just click the workflow name that appears in the upper left corner of the designer shown in Figure 5. This area of the designer is used for navigation. Clicking on "SimpleStateMachine" returns the design view to the original view that shows all the states in the workflow. If your state machine is very complicated and contains many levels of nested states, this navigational area of the designer will show you exactly where you are in the workflow.

At this point, the state machine designer will detect any state transitions specified via the setState activities and draw the appropriate transition arrows on the design surface. The TargetStateName property for SetState activities can be modified in two ways. The first, which was just described, is to use the property dialog. The TargetStateName property can also be set directly from the workflow's design surface by dragging and dropping the arrow end of a transition arrow to the desired State activity.

The ordered state in our simple state machine workflow (Figure 5) was set up in a fashion similar to the submitted state. In other words, a Code activity was used in both the StateInitialization and the StateFinalization activities to write a message to the console. In addition, a SetState activity was used in the StateInitialization activity to transition control to the completed state in the state machine workflow.

We are now ready to use the workflow. The code to instantiate the workflow runtime, create an instance of this workflow, and start the workflow from a console application is shown here:

static void Main(string[] args)
{
    // Create the workflow runtime.
    using (WorkflowRuntime workflowRuntime = new WorkflowRuntime())
    {
        WorkflowInstance instance =
            workflowRuntime.CreateWorkflow(typeof(SimpleStateMachine));
        instance.Start();

        Console.ReadLine();
    }
}

After this code is executed, the following output will appear in the console output window:

The Submitted state has been initialized at 8/6/2007 7:40:36 PM.
The Submitted state has been finalized at 8/6/2007 7:40:36 PM.
The Ordered state has been initialized at 8/6/2007 7:40:36 PM.
The Ordered state has been finalized at 8/6/2007 7:40:36 PM.

Notice that the workflow described in this section implemented three of the needed states (Submitted, Ordered, and Completed) for the Supply Fulfillment workflow I described at the beginning of the article. I will now add the other states (Assigned, Approved, and Rejected) to this workflow to illustrate backtracking as well as states that can transition to more than one possible state. I will also add events.

Communicating with State Machine Workflows

Now that you've seen the basic moving parts of a WF state machine, I'd like to discuss some real-world considerations. State machines in the real world are often long-running, spending most of their lifetime paused within a state, waiting for input that is external to the workflow itself. An executing state can be paused to wait for external information using the EventDriven activity. As I stated previously, the EventDriven activity is one of the few activities that can be dropped into a State activity (the others are StateInitialization and StateFinalization). External information is passed to the EventDriven activity by way of events. Specifically, events are raised by an external data service and are received by the EventDriven activity. The EventDriven activity can be a little tricky to set up, but the following steps should help to get you started.

  1. Determine the events required for data exchange and state transitions.
  2. Define an External Data Exchange interface.
  3. Add EventDriven activities to the state machine workflow.
  4. Reference the external data exchange interface.
  5. Implement the External Data Exchange interface.
  6. Add the External Data Exchange to the workflow runtime.
  7. Use the External Data Exchange to exchange data and transition states.

Figure 7 shows exactly how the pieces mentioned in the previous steps fit together.

Figure 7 Configuring and Utilizing a State Machine Workflow
Figure 7 Configuring and Utilizing a State Machine Workflow(Click the image for a smaller view)
Figure 7 Configuring and Utilizing a State Machine Workflow
Figure 7 Configuring and Utilizing a State Machine Workflow (Click the image for a larger view)

Events Required for Data Exchange and State Transitions

The first step is to determine which events are needed by each state in the state machine workflow. State machine events typically cause the workflow to change states. Therefore, the best way to determine the needed events is to review requirements and look for key words that indicate state transitions. Recall that to build the logical design of Figure 1, we needed to review the requirements to determine the necessary states and state transitions. That information can also be used to determine the events that are needed. Figure 8 expands upon the data in Figure 2, which listed the state transitions for each state within the Supply Fulfillment workflow. Figure 8 shows each state in the workflow, the events that each state will listen for, and the state that will be transitioned to for each event listed. If data is needed by the workflow instance in order to move to a new state, then Workflow Foundation provides capabilities that allow information to be passed as event parameters.

Figure 8 Supply Fulfillment Workflow Events (Close Figure 8)

State | Event Name | Transition
Submitted | OnAssigned | Assigned
Assigned | OnApproved | Approved
Assigned | OnRejected | Rejected
Approved | OnOrdered | Ordered
Rejected | OnReassigned | Assigned
Rejected | OnCanceled | Completed
Ordered | OnOrderReceived | Completed

Notice that the Assigned state is listening for two events. The Assigned state will listen for both of these events at the same time. The event received determines the state transition. This is how external entities can determine the path that a state machine workflow will follow. (The Rejected state will also listen for two events.)

Defining an External Data Exchange Interface

The next step is to define these events within a standard .NET interface. Figure 9 shows the interface needed for the Supply Fulfillment workflow and Figure 10 shows the custom EventArgs class that is used to pass data to an instance of a workflow whenever one of these events is raised.

Figure 10 The Custom EventArgs Class (Close Figure 10)
namespace ExternalDataExchange
{
    [Serializable]
    public class SupplyFulfillmentArgs : ExternalDataEventArgs
    {
        public SupplyFulfillmentArgs(System.Guid InstanceId) :
            base(InstanceId) { }

        private string assignedTo;
        private string orderID;

        public string AssignedTo
        {
            get { return assignedTo; }
            set { assignedTo = value; }
        }

        public string OrderID
        {
            get { return orderID; }
            set { orderID = value; }
        }
    }
}
Figure 9 External Data Service Interface (Close Figure 9)
namespace ExternalDataExchange
{
    [ExternalDataExchange()]
    public interface IEventService
    {
        event EventHandler<SupplyFulfillmentArgs> Assigned;
        event EventHandler<SupplyFulfillmentArgs> Approved;
        event EventHandler<SupplyFulfillmentArgs> Rejected;
        event EventHandler<SupplyFulfillmentArgs> Reassigned;
        event EventHandler<SupplyFulfillmentArgs> Canceled;
        event EventHandler<SupplyFulfillmentArgs> Ordered;
        event EventHandler<SupplyFulfillmentArgs> OrderReceived;
    }
}

There are a few things to note about this interface. The first is the use of the ExternalDataExchange attribute. A class that implements an interface marked with the ExternalDataExchange attribute can be added to the workflow runtime engine just like any other service. (Persistence and tracking are examples of other services that can be added to the workflow runtime.)

Also notice that SupplyFulfillmentArgs, which is used as a parameter in the events declared in the interface, must inherit from ExternalDataEventArgs. Furthermore, ExternalDataEventArgs requires the workflow instance ID to be passed to its constructor so that the event will get raised to the correct workflow instance. Finally, SupplyFulfillmentArgs must be marked as serializable in case the workflow runtime needs to serialize it to persistent storage.

Adding EventDriven Activities

Once the interface for the External Data Exchange is set up, the next step is to add EventDriven activities to the state machine. Similar to the StateInitialization and the StateFinalization activities, the EventDriven activity is a composite activity. Its purpose, therefore, is to contain other activities. Once the EventDriven activity is added to a State activity, we will be able to drill into it in the same way we drilled into the StateInitialization and the StateFinalization activities for the purposes of defining additional logic. Unlike the StateInitialization and the StateFinalization activities, we have to follow a few rules when placing activities inside the EventDriven activity. It turns out that the EventDriven activity inherits from the Sequence activity and adds the restriction that the first child activity must be an activity that implements the public interface IEventActivity. All subsequent activities can be of any type. In the example presented here, the HandleExternalEvent activity is used. This activity is capable of receiving an event that is external to the workflow runtime. Two other activities that also implement the IEventActivity interface are the Delay activity and the WebServiceInput activity.

In the SupplyFulfillment workflow, there are seven logical events that will require an EventDriven activity. Figure 8 contains the information needed for placing the EventDriven activities into the correct state. It is also important to note that logic placed within these activities will ultimately result in the state changes. Figure 11 shows the designer view of the SupplyFulfillment state machine once all the EventDriven activities have been added. Notice that the transition arrows originate from the EventDriven activities and end on the State activities. Figure 12 shows the logic contained within the OnAssigned EventDriven activity.

Figure 11 EventDriven Activities in a State Machine Workflow
Figure 11 EventDriven Activities in a State Machine Workflow(Click the image for a smaller view)
Figure 11 EventDriven Activities in a State Machine Workflow
Figure 11 EventDriven Activities in a State Machine Workflow (Click the image for a larger view)
Figure 12 Logic within an EventDriven Activity
Figure 12 Logic within an EventDriven Activity

Technically, the StateInitialization and the StateFinalization activities which are shown in Figure 11 are no longer needed. They are included here and equipped with console messages for demonstration purposes only.

It is worth mentioning that the visual depiction of the Supply Fulfillment workflow in Figure 11 is just as easy to read as the logical diagram in Figure 1 that was created with Microsoft Visio®. Development artifacts that developers, architects, and end users can easily understand and verify are a core tenet of business process management.

Referencing the External Data Exchange Interface

Once the EventDriven activities are in place and the child activities for each EventDriven activity are set up in a manner similar to Figure 11, the next step is to configure the HandleExternalEvent activities. Remember that when setting up an EventDriven activity, one of the rules of construction is that the first child activity must be an activity like the HandleExternalEvent activity. This can be seen in the logic in Figure 12 where an EventDriven activity named OnAssigned has as its first child activity a HandleExternalEvent activity named handleAssigned, followed by a SetState activity named SetStateAssigned.

The HandleExternalEvent activity is what will cause a StateMachine workflow to block while waiting to handle an event that is raised by an external data exchange service. The specific event that a HandleExternalEvent activity will wait for is specified in the properties dialog (shown in Figure 13). The first property to set is the InterfaceType property, which must be set to an interface that is marked with the ExternalDataExchange Attribute. In the case of the Supply Fullfilment workflow, this interface is the IEventService interface shown in Figure 9. Once the InterfaceType property is set, the EventName property can be specified. This properties dialog will reflect over the interface specified via the InterfaceType property and provide a dropdown box for the EventName property. The dialog shown in Figure 13 is for the Supply Fulfillment workflow's OnAssigned event.

Figure 13 handleAssigned Properties
Figure 13 handleAssigned Properties(Click the image for a smaller view)
Figure 13 handleAssigned Properties
Figure 13 handleAssigned Properties (Click the image for a larger view)

Before leaving this dialog, there is one more consideration. Remember that all the events in the ExternalDataExchange.IEventService interface require a SupplyFulfillmentArgs argument. This argument can be used to pass data into an instance of a workflow when an external event is raised and the workflow receives the event. There are two ways to make this data type accessible to other activities within the workflow. One way is to use the "e" property shown in Figure 13. This property allows you to specify a property within the workflow that can be used to hold the argument passed by the event. Once this event parameter is set into a workflow property, it can be utilized by other activities in the workflow. Obviously the property specified will need to be of the same type as the event parameter. In the case of the SupplyFulfillment workflow, this would be the SupplyFulfillmentArgs type. Alternatively, the Invoked property can be used to specify an event handler within the workflow. This event handler will be called after the external event is received and it will be passed a parameter of type ExternalDataEventArgs, which can be cast into the correct type (SupplyFulfillmentArgs).

It is important to note at this point that the HandleExternalEvent activity deals only with the ExternalDataExchange interface and not with an actual implementation of the interface.

Implementing an External Data Exchange Service

In order to successfully implement an external data exchange service, you need to implement the interface referenced by the HandleExternalEvent activities in the workflow. Specifically, the IEventService interface must be implemented. The following code creates the EventService class and implements the interface:

[Serializable]
public class EventService : IEventService
{
    public event EventHandler<SupplyFulfillmentArgs> Assigned;
    public event EventHandler<SupplyFulfillmentArgs> Approved;
    public event EventHandler<SupplyFulfillmentArgs> Rejected;
    public event EventHandler<SupplyFulfillmentArgs> Reassigned;
    public event EventHandler<SupplyFulfillmentArgs> Canceled;
    public event EventHandler<SupplyFulfillmentArgs> Ordered;
    public event EventHandler<SupplyFulfillmentArgs> OrderReceived;
    ...
 }

Since events can only be raised from within the class in which they are defined, we need to add helper functions to facilitate raising events. This class will need one helper function per event. The following code shows the helper function for the Assigned event:

public void RaiseAssignedEvent(System.Guid instanceId, string
    assignedTo)
{
      // Check to see if event is defined
    if (this.Assigned != null)
    {
        // Create the EventArgs for this event
        LocalService.SupplyFulfillmentArgs args = new
            LocalService.SupplyFulfillmentArgs(instanceId);
        args.AssignedTo = assignedTo;

        // Raise the event
        this.Assigned(this, args);
    }
}

Note that when the event is raised there are two parameters that are passed to all registered delegates. The first parameter is the sender. The code above passes the current instance of the EventService class, which must be marked as serializable for an instance of it to be passed as the sender. If it is not, then you will receive an error stating that the event could not be delivered. Here is an example of this error message:

  Event "Assigned" on interface type
  "ExternalDataExchange.IEventService"
  for instance id "0f9e8545-39bf-48b6-b890-
  c12dec11cb9f" cannot be delivered.

The second parameter is the SupplyFulfillmentArgs parameter. When this class is instantiated, it requires the workflow instance identifier—the Guid that was generated when the workflow was first instantiated. It is this Guid that will be used by the workflow runtime to locate the workflow instance that should receive the event. For this to occur, however, an instance of the EventService class must be added to the workflow runtime as an external service.

Add an External Data Exchange Service to the Workflow Runtime

Since the workflow runtime manages all execution and interaction of instantiated workflows, the class that was just created previously must be added to the workflow runtime as an external data exchange service. The following code shows how to add an instance of the EventService class to the workflow runtime as an ExternalDataExchange service class:

// Create a new instance of the local service and host it in an
//ExternalDataExchangeService.
ExternalDataExchangeService externalDataSvc = new
    ExternalDataExchangeService();
wr.AddService(externalDataSvc);
eventService = new EventService();
externalDataSvc.AddService(eventService);

The workflow runtime is very particular about how this is done. The external data service must first be created and added to the workflow runtime before the EventService class can be added to the ExternalDataExchangeService class. If you try to use the ExternalDataExchangeService object before it is added to the workflow runtime, you will receive an error that states "WorkflowRuntime container does not contain ExternalDataExchangeService."

Once this is complete, events raised from the instance of the EventService class will find their way to the correct instance of a workflow and to the correct HandleExternalEvent activity within the workflow.

Using the External Data Exchange

Everything is now set up and the External Data Exchange can be used to exchange data and transition the states of the workflow. Using the External Data Exchange is as easy as calling into the helper functions that were set up within the EventService class shown earlier. The code snippet below shows the use of the helper function that will change the workflow state from Submitted to Assigned. Notice that the workflow instance ID and an "assignTo" parameter are passed into the function. Both of these parameters will be packaged into an instance of the SupplyFulfillmentArgs class so they can be passed to the workflow via the Assigned event. As I stated, the workflowInstanceID parameter is used by the workflow runtime so it can raise the Assigned event to the correct workflow instance. The assignTo parameter ("Joe") will be delivered directly to the workflow to be used during workflow execution:

eventService.RaiseAssignedEvent(workflowInstanceId, "Joe");

It is important to note that at any given point, a running state machine workflow is only listening for the events within the current State activity. If any other events are raised, the calling code will receive an exception. For example, once the SupplyFullfilment workflow is instantiated, it is in its initial state (the Submitted state), which means the workflow is waiting for the Assigned event to be raised. If any of the other helper functions are called that raise other events (Approved, Rejected, Reassigned, Canceled, Ordered, and OrderReceived), you will receive an error stating that the event could not be delivered. Below is an example of an error message that will occur if the Ordered event is called when the workflow is in any state other than Approved:

Event "Ordered" on interface type "ExternalDataExchange.IEventService" for instance id "59657ed0-1532-42c5-8abe-6523bf121fff" cannot be delivered.

As you can see, this very generic error message occurs whenever anything goes wrong with the delivery of an event and there is no information in this message that helps to determine exactly what went wrong.

Obviously, this raises a few questions as to how to write workflow code that is not fragile. Specifically, how can your code know the current state of a state machine instance so you can programmatically ensure that the correct event is sent to the workflow? It turns out that there are a handful of useful APIs that can be used by the code consuming a state machine workflow.

Useful State Machine APIs

WF comes with a utility class named StateMachineWorkflowInstance specifically designed for managing instances of state machine workflows. Figure 14 provides a brief description of the constructor, properties, and methods of the StateMachineWorkflowInstance class.

Figure 14 Constructor, Properties, and Methods of StateMachineWorkflowInstance (Close Figure 14)

Public Constructor
Name | Description
StateMachineWorkflowInstance | This is the constructor. It requires two parameters. The first is an instance of the workflow runtime. The second is a Guid representing the state machine instance that will be queried or managed by this class' properties and functions.
Public Properties
Name | Description
CurrentState | Read only. Returns a StateActivity object for the current state.
CurrentStateName | Read only. Returns a string that represents the name of the current state activity.
InstanceID | Read only. Returns a Guid that uniquely identifies the state machine workflow within the workflow runtime.
PossibleStateTransitions | Read only. Returns a ReadOnlyCollection<string> representing the possible state transitions from the current state.
StateHistory | Read only. Returns a ReadOnlyCollection<string> of the state transitions that the current workflow has already made. This property requires a tracking service to be added to the workflow runtime.
StateMachineWorkflow | Read only. Returns a StateMachineWorkflowActivity type. This is a copy of the workflow definition and not the live instance tree, so this property should never be used to retrieve runtime values.
States | Read only. Returns a ReadOnlyCollection<string> collection of all the states in the definition of the current workflow. Like the StateMachineWorkflow property, this property contains only definitions and not live data.
WorkflowInstance | Read only. Returns the current WorkflowInstance type which represents the current instance. This property provides access to the live instance tree.
Public Methods
Name | Description
EnqueueItem | Posts a message to the queue of a state machine workflow. Usually not needed. The HandleExternalEvent activity described in previous sections enqueues messages using a programming model that is less prone to error.
SetState | Provides a transition to a state specified via its only parameter. Can be used to transition to any state in the workflow—even if the state specified is not a transition that would occur under ordinary execution of the workflow.

If you do use this class, however, heed these warnings. First, state machines that are event-driven (like the SupplyFulfillment workflow) are inherently asynchronous. Therefore, if the public properties shown in Figure 14 are queried immediately after raising an event to a state machine instance, they will not accurately reflect the state of the workflow. Event-driven workflows will be idle once they have finished processing the current event and are waiting for the next event. Therefore, the OnWorkflowIdled event of the workflow runtime is a good place to query an instance of a state machine using these properties.

Second, the StateHistory property requires a tracking service to be added to the workflow runtime. If this property is used without a tracking service, you will receive the following error:

'StateHistory' property requires the 'System.Workflow.Runtime.Tracking.SqlTrackingService' service.

Finally, be careful when exposing the functionality of the SetState method to end users. This method can cause a transition to any state in the workflow and is best used by administrators or managers dealing with non-standard scenarios.

Tying Up Loose Ends

While I covered state machine theory, the design-time experience, and the basic activities needed to build a usable state machine, there are still a few important issues to consider. First, if a state machine is to run in a server environment, it must optimize system resources while executing and must be able to survive system restarts. Additionally, it may be desirable to track and save historic data so that business processes can be optimized based on tracking reports. For example, if the Supply Fulfillment workflow in this article were used to run a small business, it would be desirable to see how long it takes for each workflow to move from the Ordered state to the Completed state. If a specific supplier constantly violated service level agreements, the business process of acquiring new supplies could be optimized by simply omitting the offending supplier.

Another topic that deserves further investigation is human workflows. State machines are a good starting point for building human workflows. If a state machine is to be used to manage business processes with human input, care must be taken when building the user interface. The user should have a clear view of the desired path through the state machine and should only be presented with options that are relevant for the current state.

Keith Pijanowski is a Platform Strategy Advisor for the Microsoft Developer and Platform Evangelism team, helping customers apply new ideas and new technologies to business problems. Keith is also a frequent speaker at Microsoft events in the New Jersey and New York area. You can reach him at www.KeithPij.com.


Editor's Note: Overcoming Language Barriers with Elegant Design

Editor's Note
Overcoming Language Barriers with Elegant Design
Howard Dierking

Several weeks ago, I was in Barcelona where I was a speaker at the Tech•Ed Europe conference. To everyone who came to my session or stopped by the MSDN booth at the exhibition hall, many thanks—it was a fantastic experience that I hope to repeat.

On the last day of the conference, I decided to be somewhat adventurous, so I hopped on a subway and went into downtown Barcelona. I stopped into a small restaurant and attempted to order. I should emphasize the word "attempted" because nobody in the restaurant spoke English. And while I speak a whole bunch of Turing-complete languages, I don't speak Spanish. At any rate, the restaurant staff was very good-humored about my inability to effectively communicate what I wanted to eat (in fact, I imagine that my hostess is probably quite good at the game Charades). But that experience really got me thinking about the language tools we use—and how we use them.

As I'm writing this, Visual Studio 2008—and with it, C# 3.0 and Visual Basic 9.0—has been officially released to manufacturing and made available online to MSDN subscribers. As I'm sure you know, these new versions have even greater support for functional programming through core language features like lexical closures. Furthermore, the language extensions built in to support LINQ add a whole new level of expressiveness when it comes to a more declarative style of programming.

If you look around, particularly in Web development, you can see dynamic languages continuing to grow in popularity—so much so that Microsoft has built a runtime and set of language tools that allow popular dynamic languages like Ruby and Python to actually run on and take advantage of the Microsoft .NET Framework. And if you really want the best of all worlds (the safety of strong static typing with the feel of a dynamic language), you probably cheered Microsoft's recent announcement that F# would be released as an official, Visual Studio-supported language.

But it's not just about the choice of programming language anymore. Domain-specific languages are now starting to come into the mainstream, thanks in large part to the Ruby community. Even in more mainstream programming languages, debates around software design have shifted from object models and design patterns to the expressiveness of an API. As the complexity of software grows at an exponential rate, the requirements for programming languages grow in tandem. A language must provide more than just a means of generating instructions for the machine—it must allow the developer to express the actual intention behind code.

And it is with this notion of expressiveness where things get really interesting—because this is the characteristic that makes programming languages very personal. For example, let's say I want to find the number of words (we'll loosely define "word" to be a block of characters separated by a space) in this article with more than eight characters. Which of these is more expressive?

article.Split(new char[]{' '}).Where(w => w.Length > 8);

or

[\S\(\)]{9,}

The answer, of course, is that it depends not just on how you understand and think about the elements of software design but on how you understand and think about text parsing. Even within a single application, there are a variety of problems that, based on the individual, are most naturally expressed in different languages—and giving that individual the ability to solve those problems by using those natural expressions is the mark of elegant design.

Now, if we can just agree on a common language for ordering food ...

Thanks to the following Microsoft technical experts for their help with this issue: Cagri Aslan, Keith Boyd, Shawn Burke, Bill Chiles, James Clarke, Peter Donnelly, Charles Finkelstein, Elisa Flasko, Ted Hart, Pravin Indurkar, John Justice, Masahiko Kaneko, Lawrence Liu, John Robbins, and Chris Tavares.


Toolbox: Powerful Text Editing, Web Testing in .NET, Extended Unit Testing, and More

Toolbox
Powerful Text Editing, Web Testing in .NET, Extended Unit Testing, and More
James Avery

Contents
Edit Well with E Text Editor
Create Functional Web App Tests
A New Testing Framework
Pragmatic Unit Testing

Edit Well with E Text Editor

A good text editor is an essential tool for every programmer. A programmer's choice of text editor is also one of the more frequent debates you will hear in the halls of conferences and workplaces. TextMate, a Mac OS X text editor, has quickly risen to popularity on that platform based on its ease of use and powerful extensibility capabilities. E Text Editor is a new text editor that tries to bring that same simplicity and powerful extensibility to the Windows® platform.

E Text Editor lays a solid foundation with easy-to-use and simple text editor functionality. The program provides you with the ability to open multiple files at a time using tabs in a similar manner to the tabbed browsing format offered in many browsers. It also offers a project pane that provides you with a directory structure view of a given directory. In the example screenshot, I have an ASP.NET Web application folder open, and you can see each of the different directories and files (including the correct icons). The project pane makes it very easy to move around between directories and files in your project.

E Text Editor extensibility is centered on the concept of bundles, which provide a way to include specific functionality for various programming languages without increasing the bloat of the native application, similar to plug-ins or add-ins in other software. By default, E Text Editor includes bundles for the majority of languages any Microsoft developer is going to be looking for, including C, C#, JavaScript, HTML, CSS, XML, and more. The one language that appears to be missing is Visual Basic® .NET. However, E Text Editor includes the functionality to create your own bundles, so maybe someone will create and publish a Visual Basic .NET bundle soon.

Bundles can include a number of different features, the first of which is text completion. Unfortunately, this is not the text completion you'll find in IntelliSense® in Visual Studio®, but rather it is text completion based on a certain set of keystrokes (similar to code snippets in Visual Studio). For example, in the C# bundle there is a text completion setup that lets you quickly declare a new namespace by typing "ns" and then clicking the tab key. You can create a new property by typing prop and then tab and so forth. Each bundle contains text completions that are specific to the language the bundle is written for.

Bundles also contain formatting rules for their particular language. When you open a new file, you will see in the bottom middle of the status bar the format type being used for the document. For instance, opening a C# document will show C# in the toolbar and provide C# syntax highlighting and formatting. You can click on the format in the status bar to change how the document should be formatted. This is important, as certain files can be viewed in different formats. For example, you could have an .html file that could be ASP, ASP.NET, or another Web framework. Choosing the correct format will not only give you the best syntax highlighting and formatting but will also enable the text completions available in that bundle.

E Text Editor is a great new option in text editors, and I fully expect to hear it mentioned in the next text editor debate that I encounter.

Price: $34.95.

www.e-texteditor.com

E-Text Editor Offers Simple Text-Editing Functionality
E-Text Editor Offers Simple Text-Editing Functionality (Click the image for a smaller view)
E-Text Editor Offers Simple Text-Editing Functionality
E-Text Editor Offers Simple Text-Editing Functionality (Click the image for a larger view)

Create Functional Web App Tests

Unit testing can go a long way to helping avoid the testing and re-testing of an application, but unit tests usually test the layers and objects behind an application and not the application itself. All of your unit tests might pass, but there might be an error on your ASP.NET page or in your Windows Forms code that wasn't covered by your unit tests.

Watin, which stands for Web Application Testing in .NET, was inspired by Watir, a similar open source framework written in Ruby. Watin is an open source tool that can be used to write tests directly against your Web applications. Tests written in this way are called functional tests since they are testing the functionality of your application as opposed to a single unit of code, such as with unit tests.

Watin interfaces with the Internet Explorer® object to launch and interact with the browser in the same way that your user would. Watin can enter text into fields, select radio buttons, click checkboxes and buttons, and read values from your forms and pages. You can even sit and watch Watin do its work on your screen, which, with the help of a cleverly hidden book, can make it appear that you are working really hard.

Watin tests are written using your favorite unit-testing framework in conjunction with the Watin library. The Watin library provides the methods and objects needed in order to interface with the Internet Explorer object while your testing framework provides the tests and assertions.

Watin tests can be somewhat frustrating to write by hand. To identify textboxes and buttons, you have to use the client identifier, which you can find by viewing the source of your page or using one of the various popular browser toolbars.

An even better way to write Watin tests, though, is by using another open source project called the Watin Test Recorder. The Watin Test Recorder is a Windows Forms application that lets you create and record Watin tests by walking through the test using the browser.

To create a simple test, you would only need to navigate to the page, name your test, and then perform the actions that are needed to replicate your test. For example, you might navigate to your home page and enter a user name in the user name textbox, a password in the password textbox, and then click the login button.

The code for your test is displayed in the box at the bottom of the application. You can then copy this code out or save it directly from the recorder. After copying or saving the test, you will then need to add assertions. Using the previous example, you would write an assertion that confirms the user was logged in successfully. The assertion could either test that the browser redirected to the correct page or look for a certain section of text on that page.

Watin can be a great replacement for manually testing your Web application over and over again and helps you quickly find issues before your users do.

Price: Free; source code available.

watin.sourceforge.net

Write Tests against Your Web Apps with Watin
Write Tests against Your Web Apps with Watin (Click the image for a smaller view)
Write Tests against Your Web Apps with Watin
Write Tests against Your Web Apps with Watin (Click the image for a larger view)

A New Testing Framework

The Microsoft® .NET Framework has a number of established unit-testing frameworks including nUnit, csUnit, and mbUnit (which was featured last month in this column), but recently another new framework was launched called xUnit.net.

A couple of things make this framework stand out from the existing frameworks. Most significant is that it was built by James Newkirk and Brad Wilson. Newkirk, who helped build nUnit, is a product manager at Microsoft responsible for CodePlex, and he has written a number of books about unit testing. Brad Wilson is a long-time blogger (as thedotguy), a former member of the patterns and practices team, and a fellow Microsoft employee. The goal with this new framework was to take the best practices learned over the past half-decade about unit testing and build a new framework that embodies and encourages those practices.

There are quite a few changes in this framework compared with the existing .NET unit-testing frameworks. Most notable are the features that have been removed—in an effort to simplify the framework and the tests written using it, a number of attributes that are present in many testing frameworks have been removed. To make tests easier to read and understand, there are no setup or teardown attributes—all initialization and teardown code must be done in the individual tests.

The idea behind removing these attributes is that they only make it harder to understand what exactly a test is doing because the logic can be split over three different methods. By removing the setup and teardown attributes, Newkirk and Wilson are encouraging developers to put all of the logic for their tests into the single-test method.

The ExpectedException attribute is gone in favor of a new method on the Assert object called Throws. The Throws assertion allows the developer to be more explicit in specifying where the expected exception will come from, making false positives less frequent. TestFixtureSetup and TextFixtureTeardown have been replaced with an ITestFixture interface. The Ignore attribute has been removed and instead implemented as a property on the Test attribute.

xUnit.net is not just about removing features, though—a number of very useful features have been added to the framework as well. xUnit.net includes complete support for generics and anonymous delegates, which make tests more concise and accurate. Also, xUnit.net includes the ability to extend the functionality of the framework using custom comparers, custom test methods, and custom classes.

There is no GUI testing tool included with this release of xUnit.net, only a console runner and Visual Studio support using TestDriven.net.

I would not recommend dropping your current testing framework for xUnit.net just yet, but it is definitely something to keep your eye on, even if only to learn from the great ideas and best practices being put forth through this tool.

Price: Free; source code available.

codeplex.com/xunit

Pragmatic Unit Testing

It may be easy to get started with unit testing, but it can be very difficult to get your unit tests right. When I first started writing unit tests, I had only a couple of blog posts and articles to guide me—I sure wish I had a book like this to refer to then. Pragmatic Unit Testing in C# with NUnit (Pragmatic Bookshelf, 2006), by Andrew Hunt and David Thomas is the second volume in a three-volume series titled The Pragmatic Starter Kit. The first volume is about using source control, and the third volume covers project automation.

Pragmatic Unit Testing covers the basic ideas, principles, and benefits of unit testing, as well as the planning and writing of unit tests. The book uses NUnit for its examples, but the theories apply to just about any of the unit-testing frameworks available for .NET.

Offering a wealth of practical advice on testing, this book is both an excellent introduction to unit testing and a great guide on how to write better unit tests and solve common testing issues.

Price: $29.95.

pragprog.com/titles/utc2

(Click the image for a smaller view)

Send your questions and comments to toolsmm@microsoft.com.

James Avery runs his own .NET consulting practice,Infozerk Inc. He has written a number of books and articles, and his most recent book is Visual Studio Hacks (O'Reilly, 2005). You can e-mail James at javery@infozerk.com and read his weblog at www.dotavery.com/blog.


CLR Inside Out: Dynamic Languages and Silverlight

CLR Inside Out
Dynamic Languages and Silverlight
Jimmy Schementi

Contents
Getting DLRConsole
Using DLRConsole
Inside the Code
2 + 2. How Hard Can It Be?
But Wait, There's More
Start Playing Now!

C
hildren learn through creative playing. From LEGO to GI Joe and Barbie (they were my sister's, I swear), children tend to pick up toys and intuitively do something with them; they don't pick up the manual and read it, at least not for the purpose of understanding the toy! It's human nature to experiment and play with things to understand them, especially as a child when the "right way to do things" isn't burned into your brain. Such a hands-on approach is arguably one of the best ways to learn.

Applying that same idea to software, dynamic languages are traditionally associated with an interactive environment, giving you a run-evaluate-print loop (REPL), and allowing for an explorative trial-and-error approach to programming. In a sense, the REPL approach is a way of turning programming into playing. This is why dynamic language advocates often say, "You just have to play with it."

When my team was integrating dynamic languages into SilverlightTM, we immediately wanted to be able to play with Silverlight. Through some clever programming by Jim Hugunin, DLRConsole was born. This column will explore how DLRConsole works, and in turn discuss IronPython, DLR Hosting, and the DLR's integration with Silverlight.

Getting DLRConsole
Running CLRConsole Locally

You can download and run DLRConsole on your local machine, but getting set up is slightly more complicated because it requires a Web server. After setting up a Web server, you can extract DLRConsole.zip, place the folder in your Web server folder, and navigate to /DLRConsole/index.htm on your Web server. (Setting up a Web server on your machine is out of the scope of this column, but you can find some pointers for Windows Vista at iis.net/articles/view.aspx/IIS7/Deploy-an-IIS7-Server/Installing-IIS7/Install-IIS7-on-Vista, and for Mac OS 10.4 at docs.info.apple.com/article.html?path=Mac/10.4/en/mh1921.html.)

The only caveat when running on your own Web server is that you need to set up the server's configuration for supported DLR languages. You'll know this is an issue if you get this error:

Silverlight error message
ErrorCode: 2024
ErrorType: ParserError
Message: Invalid attribute value text/python.
XamlFile: DLRConsole.xaml
Line: 6
Position: 53

You will need to add the following MIME type mappings to your server configuration:

.py == text/python
.jsx == text/jscript

At this point, pat yourself on the back; you're running a DLR Silverlight app!

But before digging into DLRConsole, perhaps I should bring you up to speed on the Dynamic Language Runtime (DLR), a dynamic language support system that sits on top of the CLR and provides language services required by most dynamic language implementations. These services include a dynamic type system, dynamic method dispatching, dynamic code generation, and hosting APIs. For more information on the DLR, check out Jim Hugunin's blog post at blogs.msdn.com/hugunin/ archive/2007/04/30/a-dynamic-language-runtime-dlr.aspx.

DLRConsole is a Silverlight 1.1 sample that demonstrates a very lightweight code development and experimentation tool. (Please note that as of January 1, 2008 Silverlight 1.1 was renamed Silverlight 2.0.) It enables you to edit and run XAML and Python code and also to experiment with Python and JScript® in an interactive editor that supports code coloring and completion.

DLRConsole exercises the DLR's integration with Silverlight in two ways. First, it is a Silverlight application written in the DLR-compliant language, IronPython. Secondly, it utilizes the DLR hosting APIs to run the DLR code typed into the console and editor.

You can download the DLRConsole code to run locally from silverlight.net/Samples/1.1/DLR-Console/DLRConsole.zip. If you just want to use DLRConsole interactively, go to silverlight.net/Samples/1.1/DLR-Console/python for a hosted version. To run DLRConsole, you must have Silverlight installed on your machine. Running DLRConsole without Silverlight installed will prompt you with a link to download it, but if you'd like to install it ahead of time for Windows® or Apple platforms go to microsoft.com/silverlight/install.aspx. If you are running a Linux-based OS, check out Mono's Moonlight project, an implementation of Silverlight for Linux, at mono-project.com/Moonlight.

Using DLRConsole

The two ways to get DLRConsole running in your browser are either to just run it from silverlight.net, or run it locally on your own machine. For the purposes of this column, I'll demonstrate running from Silverlight.net, but if you want to download DLRConsole.zip, browse through the code, and run it on your development machine or a local server, see the sidebar "Running DLRConsole Locally" for some useful tips.

To get started, open silverlight.net/Samples/1.1/DLR-Console/python in your browser. Yes, that's it! You should see the DLRConsole open in your browser as shown in Figure 1.

Figure 1 Opening DLRConsole
Figure 1 Opening DLRConsole(Click the image for a smaller view)
Figure 1 Opening DLRConsole
Figure 1 Opening DLRConsole (Click the image for a larger view)

When DLRConsole loads, you are presented with a Python command prompt, letting you type Python code line-by-line and see its effects either right in the console or in the Silverlight canvas as shown in Figure 2. You can click on the different languages to change the console to that language and see a chunk of sample code in the language you select. Try mixing and matching languages as you type statements, and see how everything you did in one language is available in the other.

Figure 2 Writing Code in the Console
Figure 2 Writing Code in the Console(Click the image for a smaller view)
Figure 2 Writing Code in the Console
Figure 2 Writing Code in the Console (Click the image for a larger view)

The XAML tab shows you the XAML code behind the initial Silverlight Canvas pane (see Figure 3). You can edit it or add to it, and then update the canvas by pressing the Ctrl+Enter keys.

Figure 3 The XAML Pane
Figure 3 The XAML Pane(Click the image for a smaller view)
Figure 3 The XAML Pane
Figure 3 The XAML Pane (Click the image for a larger view)

The Python Code tab shows you a chunk of Python code that will produce a blue background with text that changes color when you hover over it. Again, you'll need to press the Ctrl+Enter keys to execute this code, for the first time and after you make any changes.

The About tab just links to the Dynamic Silverlight Web page on CodePlex (codeplex.com/dynamicsilverlight).

Inside the Code

Now that you've seen what DLRConsole can do, let's take a look at what files make up DLRConsole, and how it all works. Figure 4 lists the source files included in the DLRConsole download.

Figure 4 DLRConsole Source Files (Close Figure 4)

File | Description
Index.htm | Simple HTML page that includes Silverlight.js and CreateSilverlight.js and calls CreateSilverlight.
Silverlight.js | Contains logic to create the Silverlight application in your browser.
CreateSilverlight.js | Defines CreateSilverlight, which initializes Silverlight and loads DLRConsole.xaml.
DLRConsole.xaml | Small XAML file which points to DLRConsole.py and binds the Loaded event to the OnLoaded python method.
DLRConsole.py | Contains all the functionality of the app.
Wpf.py | Defines helper methods for interacting with WPF APIs.
README.txt | Documentation for DLRConsole.

The files Index.htm, Silverlight.js, and CreateSilverlight.js are used to initialize Silverlight. Though these files are important, I'm not going to discuss them here. You can find a deeper discussion about the basic Silverlight project setup at silverlight.net/QuickStarts/Start/CreateProject.aspx.

Wpf.py is a Python file that defines some helpers for interacting with WPF, such as a LoadXaml(url):

def LoadXaml(url):
    uri = MakeUri(url)
    request = System.Windows.Browser.Net.BrowserHttpWebRequest(uri)
    response = request.GetResponse()
    reader = System.IO.StreamReader(response.GetResponseStream())
    xaml = reader.ReadToEnd ()
    reader.Close()
    return XamlReader.Load(xaml)

This method actually downloads the provided URL and returns the result of XamlReader.Load.

DLRConsole.xaml loads DLRConsole.py and then calls the OnLoaded method defined in the Python code. It serves as a placeholder since Silverlight 1.1 Alpha apps are required to have one XAML file. (That is expected to change in future releases of Silverlight, which will allow for loading without requiring bootstrapping XAML files like DLRConsole.xaml.) The part to highlight is the x:Code tag—this is the hook into the DLR, causing Silverlight to load the file and execute it using the DLR. This allows any events defined in XAML to be bound to methods defined in a DLR language. The definition of Canvas is simply there to get the Python code running.

<Canvas x:Name="root"  Width="1000" Height="500"
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
>
    <x:Code Source="DLRConsole.py" Type="text/python" />
    <Canvas x:Name="loadingHelper" Loaded="OnLoaded" />
</Canvas

Coming in at a 2164 lines of Python, I'm not going to put the entire contents of DLRConsole.py here, but I will walk you through the most interesting parts.

From the time the XAML file is loaded and OnLoaded is called, Silverlight is being driven by Python and the DLR. OnLoaded initializes the Application class, and eventually you arrive at the Application.Initialize(self) method. This is where all the stuff on your screen comes from. It registers a KeyHandler, creates the ConsoleWindow, Editors, and the rest of the user interface, and starts the console up for your interactive enjoyment.

2 + 2. How Hard Can It Be?

So now you have an interactive console running in your browser. Wow! Let's take the canonical dynamic language test, 2 + 2, and see what happens.

First, type 2 + 2 into the console.

py> 2 + 2
4
py>

While typing you would expect to see what you're typing in. However, from a WPF point of view, this works by the first line in Application's Initialize method:

self.KeyHandler = TextInputHandler(self.root)

This constructs the TextInputHandler, which defines textual representations of keys, and binds the OnKeyDown method of TextInputHandler to the root Canvas's KeyDown event. On every KeyPress, it constructs a KeyPress object with a textual representation for the pressed key and tells its Target to handle the key. Target can be any class that has the HandleKey method. In this case, it is a ConsoleInput so it handles the key by inserting the textual representation through the TextBuffer object at the cursor and moves the cursor to the next column.

Though that takes a bit of work, there's nothing new here other than using WPF. However, when you press Enter, you expect to have the language compute the result of the line and display the answer. In this case, it's doing that computation in the chosen language on the user's browser! Now that's the interesting part; let's take a closer look at how it works.

To start, here's the method that handles the key in this case:

def HandleKey(self, key):
    if key.Text == '\n':
        self.DoInput(key.Ctrl)
    elif key.Text == '.':
        self.DoCompletions()
    elif key.Name == 'Up' and self.Caret.line == 0:
        self.DoHistory()
    else:
        Editor.HandleKey(self, key)

DoInput, in this case, calls DoSingleLine, which tries to parse the expression. If successful, it evaluates the expression, writes the output to the console, and sets the "_" convenience variable (see Figure 5). Just in case you're interested, you can look in the source code to see how TryExpression parses an expression and how GetEngine creates the LanguageEngine.

Figure 5 DoSingleLine Parsing (Close Figure 5)
def DoSingleLine(self, text, forceExecute):
    text1 = self.le.TryExpression(text)
    if text1:
        self.WriteInputToConsole(text)
        try:
            ret = self.Engine.Evaluate(text1, self.CurrentModule)
            if ret is not None:
                self.console.write(repr(ret) + '\n', 'output')
                self.CurrentModule.SetVariable('_', ret)
                except Exception, e:
                    self.HandleException(e)
        else:
            self.DoMultiLine(text, forceExecute)

So, that's what it takes to calculate 2+2 in DLRConsole. In fact, this covers all the one-liners you can do. With a little variation you can handle multiple liners, too. The most interesting parts are where the DLRConsole calls into the DLR to execute the code.

But Wait, There's More

2 + 2 is a pretty simple example. What about something more complicated? OK, how about .NET Framework integration? Take a look at this:

py> import System
py> from System.Collections import *
py> h = Hashtable()
py> h["a"] = "IronPython"
py> h{"b"] = "in Silverlight"
py> for e in h: print e.Key, ":", e.Value
a: IronPython
b: in Silverlight

As you can see, you can use System.Collections from IronPython, and it can act like Python hashes or .NET Framework hashes, whichever style you choose. I used this same example in the October 2006 issue of MSDN® Magazine (msdn.microsoft.com/msdnmag/issues/06/10/CLRInsideOut), but now it's running in a browser. Feel free to look through that article for a more detailed overview of IronPython and .NET integration, though keep in mind that the information is slightly outdated. You can find additional information at blogs.msdn.com/ironpython.

What about interacting with the canvas? Let's take a look at the Application class's StartConsole method:

def StartConsole(self, console, canvas, langInfo):
    ...
    console.CurrentModule.SetVariable("canvas", canvas)
    console.CurrentModule.SetVariable("wpf", wpf)
    ...

When DLRConsole initializes the console, it sets a couple of variables for you automatically, including a WPF helper module and a pointer to the canvas. Given these variables, you can write code like that shown in Figure 6, which implements the mouseover color-change effect. If you want to see this code in action, select the Python Code tab and press Ctrl+Enter.

Figure 6 Mouseover Events (Close Figure 6)
import wpf

canvas.Background = wpf.BlueBrush

r = wpf.TextBlock()
r.Text = "Touch Me"
canvas.Children.Add(r)
r.FontSize = 60
wpf.SetPosition(r, 40, 60)

def light(s, e):
  s.Foreground = wpf.SolidColorBrush(wpf.Colors.White)

def dark(s, e):
  s.Foreground = wpf.SolidColorBrush(wpf.Colors.Black)

r.MouseEnter += light
r.MouseLeave += dark

When typing in any of these examples, did you notice a little box pop up when you typed a class or object name followed by a period? DLRConsole supports something the team likes to call "code-sense." It's not quite as advanced as the Visual Studio® IntelliSense® feature—this is a sample, after all—but it still helps a great deal at finding out what methods or properties you have available to you on a given object (see Figure 7).

Figure 7 Code-sense Autocompletion
Figure 7 Code-sense Autocompletion

With this, you can use the up/down arrow keys to select which element you want, and then press Enter to select it and continue typing. To see how it works, let's look at the HandleKey method again:

def HandleKey(self, key):
    if key.Text == '\n':
        self.DoInput(key.Ctrl)
    elif key.Text == '.':
        self.DoCompletions()
    elif key.Name == 'Up' and self.Caret.line == 0:
        self.DoHistory()
    else:
        Editor.HandleKey(self, key)

That's a rather nice feature for just a little more than 100 lines of Python code.

From Python to JScript

As I mentioned before, DLRConsole supports multiple DLR languages, and the currently released bits show support for Python and JScript. Since all these languages are built on-top of the DLR, they are not only integrated with .NET but can they also share declarations across languages. This demonstrates sharing variables and methods between Python and JScript:

js> a = 2 + 2
4.0
py> b = a * 2
py> b
8.0
js> function times10(n) {
js|   return n * 10
js| }
py> times10(5)
50.0

To accomplish this, just click the language you want to program in and it changes the console prompt, syntax coloring, and, of course, the language. This works by telling the console to set its language, which delegates to the Editor:

def SetLanguage(self, name):
    self.le = GetEngine(name)
    self.Engine = self.CurrentEngine = self.le.engine
    self.TokenCategorizer =
        self.Engine.LanguageProvider.GetTokenCategorizer()

This may not seem very practical at first, but it enables code-reuse like never before by giving developers the freedom to code in their favorite language, but not limit the use of their code based on the implementation language. Arguably, developers are most creative when writing in their favorite language, so this may even help improve the quality of software.

Start Playing Now!

DLRConsole enables you to explore Silverlight, as well as dynamic languages, with minimal commitment—you don't even need to install any software beyond Silverlight. What better way to start experimenting than in someone else's hosted sandbox?

Keep your eyes open for a new release of dynamic language integration in Silverlight, as well as new and updated versions of our Silverlight dynamic language samples from codeplex.com/dynamicsilverlight. The next version should also include support for IronRuby, which you can download today from www.ironruby.net. And, as always, you can download IronPython from codeplex.com/ironpython. If you are interested in more Silverlight samples, you can find plenty of fun and interesting ones in the Silverlight Community Gallery: silverlight.net/community/communitygallery.aspx.

Send your questions and comments to clrinout@microsoft.com.

Jimmy Schementi is a Program Manager on the Dynamic Languages team at Microsoft. He's focused on making Silverlight application-authoring a fun and productive experience with dynamic languages, such as Ruby and Python, and building a community around those languages. You can contact Jimmy at http://jimmy.schementi.com.


Basic Instincts: XML Programming in Visual Basic 9.0

Basic Instincts
XML Programming in Visual Basic 9.0
Jonathan Aneja

B
y now you've probably heard of LINQ (or Language Integrated Query), the new query technology coming in Visual Studio® 2008. LINQ-enabled languages like Visual Basic® give you a rich set of query operators that can be applied to various data sources, such as in-memory collections, databases, datasets, and XML. That alone is pretty cool, but Visual Basic 9.0 actually goes beyond that and makes XML a first-class data type directly in the language.

Now you may be wondering why you would want an XML data type integrated directly into Visual Basic. Today many applications use XML for both storage and data transfer. XML has gained broad adoption across the industry due to its flexibility and simplicity, and it is used in many apps for both storage and data transfer. It works particularly well for transferring data among systems since it is self-describing (meaning that the structure of the data is included with the data). Plus, it's much easier to read data structured inside XML tags than it is to write parsing logic for various custom file formats.

The problem with XML, however, is that it has never been particularly easy for developers to work with. Awkward and inconsistent APIs, such as the Document Object Model (DOM), and languages such as XSLT and XQuery lead to writing a lot of tedious code that is often difficult to read and understand. But with the introduction of LINQ and Visual Basic 9.0, XML development becomes much easier. In this column I will explore the current XML programming experience, how LINQ improves the experience, and how Visual Basic provides even more support when working with XML.

Using the DOM

To get started, say I need to write out a list of customers in XML. My customer list has the following properties: FirstName, LastName, Address, City, State, and ZipCode. When I convert this to XML, it should look something like Figure 1.

Figure 1 Customer List Properties in XML (Close Figure 1)
<Customers>
  <Customer FirstName="Jane" LastName="Dow">
    <Address>123 Main St</Address>
    <City>Redmond</City>
    <State>WA</State>
    <ZipCode>10104</ZipCode>
  </Customer>
  <Customer FirstName="Matt" LastName="Berg">
    <Address>456 First St</Address>
    <City>Seattle</City>
    <State>WA</State>
    <ZipCode>10028</ZipCode>
  </Customer>
</Customers>

Using the DOM and Visual Studio 2005, I could write code that loops through my list and constructs the appropriate XML nodes, as shown in Figure 2. As you can see, that's a lot of code to write for something that should be pretty simple. It's also very hard to visualize the structure of the generated XML since it doesn't really match the structure of the code.

Figure 2 Using the DOM to Construct the XML Nodes (Close Figure 2)
'With Visual Basic 9, you no longer have to write code like this.
Public Function ConvertToXML(ByVal custList As List(Of Customer)) _
  As String

  Dim doc As New XmlDocument
  Dim root As XmlElement = doc.CreateElement("Customers")

  For Each cust As Customer In custList
    Dim custElement As XmlElement = doc.CreateElement("Customer")
    custElement.SetAttribute("FirstName", cust.FirstName)
    custElement.SetAttribute("LastName", cust.LastName)

    Dim address As XmlElement = MakeElement(doc, "Address", cust.Address)
    Dim city As XmlElement = MakeElement(doc, "City", cust.City)
    Dim state As XmlElement = MakeElement(doc, "State", cust.State)
    Dim zipcode As XmlElement = MakeElement(doc, "ZipCode", cust.Zip)

    With custElement
      .AppendChild(address)
      .AppendChild(city)
      .AppendChild(state)
      .AppendChild(zipcode)
    End With
    root.AppendChild(custElement)
  Next
  doc.AppendChild(root)
  Return doc.InnerXml
End Function

Private Function MakeElement(ByVal doc As XmlDocument, _
                             ByVal elementName As String, _
                             ByVal value As String) As XmlElement
  Dim element As XmlElement = doc.CreateElement(elementName)
  element.InnerText = value
  Return element
End Function

In the long term, maintaining this code will pose a huge inconvenience. If business requirements change and I find that I need to add some properties or attributes, I would have to wade through this code and figure out which nodes to update.

The LINQ to XML API

When coding against the DOM, I'm basically trying to construct the XML tree from the inside out. It is like I have to fight against the API to get my work done. Rather than having to construct a whole bunch of nodes and connect them together in the DOM, it'd be easier if I could just express the structure of the XML and then substitute the values in-line. This is exactly what the LINQ to XML API allows me to do—I can create the XML from the top down (otherwise known as functional construction).

Using LINQ, I can now write a much shorter and cleaner version of the function in Figure 2. The new, simpler solution is presented in Figure 3. This produces exactly the same output as the first solution, but, as you can see, the new snippet is shorter and much easier to read. I use the constructors for the XElement and XAttribute objects to construct the tree. And note this cool feature: these constructors have overloads that allow me to pass in additional LINQ to XML objects; for instance, I can pass an XElement into another XElement.

Figure 3 New Solution Using LINQ (Close Figure 3)
Public Function ConvertToXML(ByVal custList As List(Of Customer)) As _
  String

  Dim doc = New XElement("Customers", _
                From cust In custList _
                Select New XElement("Customer", _
                         New XAttribute("FirstName", cust.FirstName), _
                         New XAttribute("LastName", cust.LastName), _
                         New XElement("Address", cust.Address), _
                         New XElement("City", cust.City), _
                         New XElement("State", cust.State), _
                         New XElement("ZipCode", cust.Zip)))

  Return doc.ToString
End Function

The first line creates a new XElement and names it Customers; it also creates the related <Customer> and </Customer> tags. The second parameter passed into the constructor is a LINQ query that will become the contents inside these Customer tags. This query loops through all the customers in my list and transforms the results to XML. Everything after the Select statement is evaluated and at run time produces a collection of Customer elements.

Since I'm querying data, I can take advantage of LINQ to do such things as filtering, sorting, grouping, joining, and a variety of other operations. For example, I could modify the function to only return customers who live in Seattle by adding a Where clause to the query. If I were using the DOM to do this, I'd have to add an If statement to the For loop. Sorting the results in LINQ is now as easy as adding an Order By clause to my query. With the DOM, I would have to manually sort the results before constructing the XML.

The example I'm using is fairly simple, as I am really just scratching the surface of what can be done with LINQ. So far, I've used the XElement and XAttribute types. The API defines many other types, including XDocument, XNamespace, and XComment.

XML Literals and Embedded Expressions

While LINQ to XML provides a much simpler experience than you've had in the past, the approach still isn't quite as easy as it could be. This is where XML literals, a new concept introduced in Visual Basic 9.0, enters the picture.

To understand XML literals, you must first understand what a literal is. Consider the following variables:

Dim someString = "A string literal"
Dim someNumber = 256
Dim someDate = #2/27/2008#
Dim someBoolean = True

Each of the variables shown here is directly assigned a value, and in each case the actual value is called a literal. (Note that with the new Type Inference feature in Visual Basic 9.0, these declarations are still strongly typed, even though I haven't provided an "As <Type>" clause for each variable.)

Anything inside quotes is referred to as a string literal. And the value assigned to someDate, for example, would be a date literal. An XML literal is exactly the same concept. I can express XML exactly as I would express the data anywhere else (in its literal representation). That means no quotes around it; XML literals are not Strings. I can directly assign data to the customers variable, as shown in Figure 4, and the compiler will see that the value is literal XML. Thus, the variable's type will be inferred to be XElement.

Figure 4 Directly Assign Data to the Customers Variable (Close Figure 4)
Dim customers = <Customers>
                  <Customer FirstName="Jane" LastName="Dow">
                    <Address>123 Main St</Address>
                    <City>Redmond</City>
                    <State>WA</State>
                    <Zip>10104</Zip>
                  </Customer>
                  <Customer FirstName="Matt" LastName="Berg">
                    <Address>456 First St</Address>
                    <City>Seattle</City>
                    <State>WA</State>
                    <Zip>10028</Zip>
                  </Customer>
                </Customers>

When the compiler sees this XML, it will convert it into a series of calls to the LINQ to XML API. Not only is this easier to read, but the compiler actually optimizes this construction and produces faster code than I would get just using the API directly.

So far, all I've done is paste some XML into my project and store it in a variable—this isn't necessarily that useful. One common misconception about using XML support in Visual Basic 9.0 is that you should use it to store XML documents inside your code. This, however, is far from the truth. The point is to provide an easy syntax to create, query, and transform XML documents. I demonstrated how XML literals allow me to easily create documents. But in order for this to be useful, I need a way to insert values directly into the XML. This is where embedded expressions come into play.

An embedded expression essentially tells the compiler to "stop processing this XML literal for a second, evaluate the expression, and insert the result back into the XML." In order to do this, I use the <%= (expression) %> syntax. Figure 5 takes my ConvertToXml function and demonstrates how I can improve it using XML literals and embedded expressions.

Figure 5 Using XML Literals and Embedded Expressions (Close Figure 5)
Public Function ConvertToXML(ByVal custList As List(Of Customer)) _
  As String

  Dim doc = <Customers>
              <%= From cust In custList _
                Select <Customer FirstName=<%= cust.FirstName %>
                                 LastName=<%= cust.LastName %>>
                         <Address><%= cust.Address %></Address>
                         <City><%= cust.City %></City>
                         <State><%= cust.State %></State>
                         <ZipCode><%= cust.Zip %></ZipCode>
                       </Customer> %>
            </Customers>

  Return doc.ToString
End Function

This example uses a LINQ query to generate XML, but the data source that I am querying over is actually an in-memory collection of type List(Of T). This is where you can really see the beauty of LINQ—I can just as easily use the same query operators that I used for collections against databases and other XML documents without having to learn a new API for each. And you can see how I could take data from a SQL Server® database and easily transform it to XML using just a few lines of code.

XML Properties

Applications often need to examine XML (perhaps provided through a file or an RSS feed) and make decisions based on the data. This is frequently done through XSL transformations, but LINQ to XML makes this unnecessary.

In the example shown in Figure 6, I load in an XML document, filter by customers who live in Seattle, and then select the first and last name. After that, I loop through my query and display all the matching results on screen.

Figure 6 Sample Query Using XML Axis Properties (Close Figure 6)
Public Sub DisplaySeattleCustomers()

  Dim filePath = My.Application.Info.DirectoryPath & "\customers.xml"
  Dim doc = XDocument.Load(filePath)

  Dim query = From cust In doc...<Customer> _
              Where cust.<City>.Value = "Seattle" _
              Select Name = cust.@FirstName & " " & cust.@LastName

  For Each name In query
    MsgBox(name)
  Next
End Sub

At first glance, you might think something seems wrong here. If the compiler infers doc to be of type XDocument (and cust to be of type XElement), how am I able to type things like cust.<City> and cust.@FirstName? XElement obviously doesn't have these properties, as they're specific to the XML that I loaded. What's actually happening here is the Visual Basic compiler has saved me a fair amount of work through a feature called XML properties.

XML properties, sometimes referred to as XML axis properties, provide an easy way to retrieve values stored in XML. In the query shown in Figure 6, I actually use all three of the available axes: descendants, elements, and attributes.

In XML, a descendant is an element that is nested one or more levels below the current element. The expression "doc...<Customer>" uses the descendants axis, and it basically says "find all descendants named Customer." Remember that the XML I have been using throughout has a root node called Customers, and it contains Customer elements. The triple-dot syntax is essentially translated to doc.Descendants("Customer").

Now I am going to take a look at the cust.<City>.Value expression. In this case, I am using the elements axis, so the line is translated into cust.Elements("City").Value. You may be wondering why we need to explicitly call .Value before doing the comparison here (which we don't need to do for attributes). The reason is that elements such as City can contain other elements, so the expression cust.<City> actually returns an IEnumerable(Of XElement). The .Value extension property concatenates all values stored in that element, which in this case is just a simple text value.

The last expression, cust.@FirstName, uses the attributes axis. This is translated to cust.Attributes("FirstName") and returns the value stored in the attribute. Since attributes can't contain elements, there's no need to call .Value. I then use the Select method and combine the two fields into one, called Name, in order to make things simpler for display purposes.

XML Schema IntelliSense

It would have been great if IntelliSense® would have told me the name of the fields in my XML document while I was coding up that last example. The problem, however, is how can the compiler know the structure of the XML document? The answer is through XSD schemas. If you have a schema for your XML, you can use the Imports statement in Visual Basic to bring your schema into scope. Once you do that, you get real IntelliSense based on your schema.

The XML to Schema Tool for Visual Basic 2008 (which you can download at go.microsoft.com/fwlink/?LinkId=102501) allows you to quickly create XSD schemas based on existing XML files. You just point it at an XML file or Web URL, and it then does the grunt work of generating an XSD file.

Wrapping Up

Visual Basic is all about productivity. And with business applications increasingly depending on XML for things like storage, data transfer, and even user interfaces, it's important that Visual Basic incorporate features that make it easier for developers to work with XML. Visual Basic 9.0 steps up in a big way, offering rich XML support and powerful new query capabilities.

The simple yet powerful query syntax offered by LINQ enables you to write queries against XML just as easily as you would against a database or an in-memory collection. And once you start to play with Visual Basic 9.0, you will discover many new XML features that I haven't had a chance to touch upon here, including built-in support for XML namespaces, comments, and fragments.

Send your questions and comments to instinct@microsoft.com.

Jonathan Aneja is a Program Manager on the Visual Basic team at Microsoft. He works mainly on compiler features, such as LINQ, as well as on other projects, such as the Interop Forms Toolkit. Jonathan has been at Microsoft for about two years, where he started out in ISV Advisory Services.


Data Points: Designing an Entity Data Model

Data Points
Designing an Entity Data Model
John Papa

Code download available at: DataPoints2008_02.exe (174KB)
Contents
Understanding the EDM
Building an EDM with the Wizard
Stored Procedures in the Entity Model
Windows on EDM
Derived Entities
Wrapping Up

T
he Entity Framework is an exciting new technology being developed for ADO.NET. It allows developers to view data using a logical model instead of a physical model, offering more flexibility. I gave a detailed overview of the Entity Framework (which is due in the first half of 2008) in the July 2007 installment of Data Points. If you're not yet familiar with the Entity Framework, you can read my overview at msdn.microsoft.com/msdnmag/issues/07/07/DataPoints.

At the heart of the Entity Framework is the Entity Data Model (EDM). The EDM defines the entity types, relationships and containers that developers interact with via code. The Entity Framework maps these elements onto the storage schema exposed by a relational database. The EDM is exposed to the Entity Framework through XML that defines the conceptual application model. The conceptual model may be defined separately, or together with XML that defines the actual storage schema along with XML that defines the mapping between the two. While it is possible (and sometimes necessary) to manually edit the XML, it is much easier to create and modify an entity model and mapping using the new visual Entity Data Model designer tool.

In this month's column, I will discuss how you can design an entity model using the new visual EDM designer tool and how you can modify the underlying XML that defines both the model and the mapping. I'll start by describing all of the pieces that interact within the Entity Framework (including LINQ) and will then discuss where the EDM fits. Furthermore, I will demonstrate how to create an entity model and associated mappings using the visual designer tool. Finally, I will introduce you to several windows that can help you modify and explore the model and mappings.

Throughout the column, I will explain the roles of the different components of the EDM, such as the EntityType and Association. The examples in this column demonstrate how to create basic entities. Note that all the examples in this column use Visual Studio® 2008 and the corresponding Entity Framework Beta 3 (these require separate installations). The database I am using is the revised Northwind database that comes with the samples for Beta 3.

Understanding the EDM

Before I dive into how to create and manage an entity model, let me first explain what the EDM is and how it interacts with the other elements of the Entity Framework. The Entity Framework consists of many pieces, including the EDM specification and associated mappings, APIs that interact with the EDM, and tools that help define and manage the entity model and mappings. Once you've designed an entity model, you can write code against it using the different APIs, such as the EntityClient provider, or Object Services (including LINQ to Entities).

The EntityClient data provider has a similar model to traditional ADO.NET objects, as it uses EntityConnection and EntityCommand objects to return a DbDataReader. The commands for the EntityClient provider are written using Entity SQL, which is similar to T-SQL, but, instead of operating on database objects, it operates on the entities defined in the entity model and on the objects materialized through Object Services. You can use Object Services to interact with the EDM, either by using Entity SQL or by using LINQ to Entities. Object Services allows you to take advantage of the conceptual model's generated classes, which offer features such as strongly typed objects and persistence (see Figure 1).

Figure 1 Entity Framework Overview
Figure 1 Entity Framework Overview

These data access techniques allow you to interact with the conceptual entities defined within the entity model rather than the objects of a physical store such as a relational database. The data model and associated mappings are created by using either the visual designer tool or by manually editing the XML that defines them. The Entity Framework, shown in Figure 2, provides the glue between an application and its database. The EDM is used to describe the business entities through the conceptual entity model and the Entity Framework, through the mapping specification, then translates that to the physical storage of the database's tables, view, functions, and procedures.

Figure 2 Entity Framework Connects an App to its Database
Figure 2 Entity Framework Connects an App to its Database(Click the image for a smaller view)
Figure 2 Entity Framework Connects an App to its Database
Figure 2 Entity Framework Connects an App to its Database (Click the image for a larger view)

The application's entity model is described using the Conceptual Schema Definition Language (CSDL). CSDL is an XML Format that defines the entities and the associations between entities that developers will interact with via an API, such as LINQ to Entities. The Entity Framework also uses the Storage Schema Definition Language (SSDL), an XML format that defines the storage schema of the relational database and the Mapping Schema Language (MSL) to translate how the entities of the CSDL map to the storage schema described by the SSDL.

The CSDL is where the developer has the most influence since this is where the entities that she will most often interact with are defined. Figure 2 illustrates that some entities can be mapped directly to a single table in a database while others can be mapped to multiple tables. This entity determination is made by the development team based on the business model. The business model often operates on a single entity that exists in multiple physical tables in the database. You can also see in Figure 2 that an entity can be mapped to a database's view or an entity can obtain a method that will invoke a stored procedure. Entities can also be derived from other entities using inheritance in the conceptual model. These are just some of the ways in which the entity model can be designed using the EDM designer tools.

Building an EDM with the Wizard

To create an entity model, you start by adding a new ADO. NET Entity Data Model file to your project (see Figure 3). Once you do this, the Entity Data Model Wizard will prompt you to either generate a model from a database or begin with an empty model. Generating the model from existing database tables is a good way to start, as long as you have access to the database. Some development methodologies, such as domain driven design, advocate designing the entity domain model before you design the database. If this is your plan, you may want to create an empty model and then create your entities through the EDM visual designer. For my samples, I have built an entity model using the Northwind database as a starting point.

Figure 3 Add an EDM File to Your Project
Figure 3 Add an EDM File to Your Project(Click the image for a smaller view)
Figure 3 Add an EDM File to Your Project
Figure 3 Add an EDM File to Your Project (Click the image for a larger view)

The next screen of the wizard prompts you to enter the database connection information. Then, the wizard asks you to choose the database objects to include in the model. You can see in Figure 4 that I have selected all of the tables and stored procedures in the Northwind database, except for the ones that help with diagramming. The tables are initially mapped directly to entities and the stored procedures can be mapped to methods on the generated container.

Figure 4 Choose Database Objects
Figure 4 Choose Database Objects

After you've specified the database objects that you want to include in the model, the EDM wizard generates the .edmx file that defines the model and mappings and adds the appropriate references to the project that are required by the Entity Framework. The .edmx file is an XML file that contains four major sections: information about the visual layout of the conceptual model in the designer, the conceptual model's CSDL, the mapping layer's MSL, and the physical model's SSDL. All of this information is contained in a single file.

The .edmx file's designer information is used to assist Visual Studio in laying out the entity model in the designer; this is only used at design time. The CSDL, MSL, and SSDL are used at compile time to generate classes that will represent the EDM.

Stored Procedures in the Entity Model

I've chosen to add the stored procedures to the entity model that I created, but the stored procedures have only been added to the SSDL definition. Since a stored procedure may interact with many different tables or other objects in a database, the Entity Framework does not automatically map the stored procedure to any one specific entity in the CSDL. If you want to create a method that maps to a stored procedure, you can do so by editing the XML in the .edmx file.

To create a method that returns entities of a given type, I first choose a stored procedure. For this example, I will add a method called GetTenMostExpensiveProducts, which will execute the Ten Most Expensive Products stored procedure, returning results as Product objects. Since the SSDL already describes the stored procedure as a Function element, the next step is to add the method in the CSDL. This can be done by adding the FunctionImport element as a child of the EntityContainer element:

<FunctionImport Name="GetTenMostExpensiveProducts"
  EntitySet="Products"
  ReturnType="Collection(Self.Products)">
</FunctionImport>

The Name attribute indicates the name of the method on the entity container. The EntitySet attribute indicates the EntitySet, and the ReturnType attribute refers to the EntityType that will be returned (or the collection of EntityTypes in this case).

The reference to Self in this example is an alias that refers to the current namespace, which is NWModel. Note that either NWModel or Self would work here.

If parameters are needed on the method, you can add them by including a <Parameter> tag. For example, if there was a CategoryId parameter on the stored procedure that I wanted to also include on the method, I could add it to the method by including the following XML element inside of the FunctionImport element:

<Parameter Name="CategoryId" Type="Int32" Mode="in"/>

But there is no parameter in my example, so I have skipped this step.

At this point, the CSDL defines the method, the entity type returned by the method, and the EntitySet to which the returned entities belong, while the SSDL defines the stored procedure. Now the CSDL has to be mapped to the SSDL so that the conceptual method knows what stored procedure to execute. This mapping is done by inserting the following FunctionImportMapping into the EntityContainerMapping section of the MSL:

<FunctionImportMapping
  FunctionImportName="GetTenMostExpensiveProducts"
  FunctionName="NWModel.Store.Ten_Most_Expensive_Products"/>

Here, the FunctionImportMapping element uses the FunctionName attribute to refer to the fully qualified name of the Function element in the SSDL. The FunctionImportName attribute refers to the name of the FunctionImport element in the CSDL.

Windows on EDM

Once the EDM has been designed and built, you can use various windows to examine the status of the entity model you have created. The Class View window (see Figure 5), which is not new, is helpful when you need to examine the objects that are available to be developed against. It will show, for instance, my new GetTenMostExpensiveProducts method on the NWEntities EntityContainer class, as well as all of the classes that the NWModel namespace contains. There will be a class representing each EntityType in the CSDL, plus a class for the main NWEntities model.

Figure 5 EDM in the Class View
Figure 5 EDM in the Class View

There are also some new windows that let you look into the entity model and the associated mappings. These include the Entity Data Model Designer, the Entity mapping Details window, and the Entity Model Browser window.

The Entity Model Browser shows all of the CSDL and SSDL components. These include the CSDL components EntityTypes, Associations, EntitySets, AssociationsSets, and Function Imports, as well as all of the SSDL elements.

The Entity Model Designer (see Figure 6) shows a layout containing the model that you've generated. The entity model can be viewed and manipulated from this designer, which represents all of the conceptual model's elements shown in a visual layout. Note that in Figure 6 there is a Product entity—this represents the Product EntityType in the CSDL. Each EntityType contains a list of properties, both scalar and navigational.

Figure 6 Entity Data Model Designer
Figure 6 Entity Data Model Designer(Click the image for a smaller view)
Figure 6 Entity Data Model Designer
Figure 6 Entity Data Model Designer (Click the image for a larger view)

Navigation properties are used to navigate across associations in the CSDL. The navigation properties become public properties on their EntityType class and are used to refer to another entity, or set of entities, associated with the original entity. For example, the Product EntityType has a navigation property named Categories, which will reference a Category entity for a particular Product entity instance. The Category EntityType also has a navigation property named Products. This exists so that with a Category entity instance, you can reference its associated Product entities.

The designer can be used to add, edit, or remove entity types, associations, scalar properties, and navigation properties. You can, for example, change the names of all of the EntityTypes to a singular form that is popular among developers when naming entities. Simply click the EntityType's name and edit the name in place. Or you can select the EntityType and change its name in the Properties window. For these examples, I renamed all EntityTypes to a singular format, which is reflected in Figure 6. After making this change, I also had to fix the XML I modified to add the Stored Procedure call as a method of the NWEntities EntityContainer. This is pretty simple, as I just had to update the code to refer to the Product EntityType (as opposed to Products), as shown here:

<FunctionImport Name="GetTenMostExpensiveProducts"
  EntitySet="Products"
  ReturnType="Collection(Self.Product)">
</FunctionImport>

This brings up an important point. It is a good idea to name your main elements, such as EntityType, EntityContainer, and Association, before you start modifying the data model. This will minimize the number of manual changes you'll need to make, as well as minimize any code changes you might need to make for code you may have already written against the data model. In case you run into problems, the Error List window is your best friend as it will very likely show you which elements have invalid references in the XML.

Finally, while the designer shows the conceptual model, the Entity Mapping Details window provides a means to view and edit the mappings (the MSL) between the EDM and the data store.

Derived Entities

The tools that help you design your entity model can also help you modify the model. One of the keys to object-oriented programming is the concept of inheritance. The EDM supports creating and modifying inherited entities both in the XML and visually in the EDM designer.

To demonstrate the task of designing inherited entities, I will create a DiscontinuedProduct EntityType based on the Product EntityType. The Product EntityType has a Boolean scalar property named Discontinued, which I will use as the Condition to be evaluated in determining a product's specific instance type. Using the Entity Model Designer, I right-click in the designer and choose Add | Entity from the popup menu. I then enter DiscontinuedProduct as the new entity name and select Product as the base entity (see Figure 7).

Figure 7 Creating an Inherited Entity
Figure 7 Creating an Inherited Entity

In the next step, I want to define the differentiating factor between these entities using a Condition, so I select the Product EntityType in the designer and go to the Entity Mapping Details window. I then select the Discontinued property in the Column Mappings and remove the mapping. This removes the Discontinued property from both the Product and DiscontinuedProduct EntityTypes (When you create a Condition, the available expressions for the Condition must not already be used as a property). Next, I go to the Maps to Products section of the Entity Mapping Details window and add a Condition of Discontinued = 0. I then select the DiscontinuedProduct EntityType and add a Condition of Discontinued = 1.

Creating derived entities is really quite easy when you use this technique. If you want, you can also add additional properties to the derived entity beyond its base class's properties. When you create an instance of a DiscontinuedProduct in your .NET code and save it, the Entity Framework realizes that it must set the Discontinued column to a value of 0 in the database because of the Conditions that exist on it.

The Conditions are created in the EntityTypeMapping section of the MSL and simply enforce a filter on the retrieval of rows. When data is saved, the Conditions are used to determine the value to write to the underlying database column based on the derived type.

Wrapping Up

Using the design tools in Visual Studio 2008 and the XML files, you can design an Entity Data Model that uses inheritance, invokes stored procedures, and models your business instead of writing code directly against your relational database schema. Once a solid entity model has been designed, interacting with it from APIs, such as Object Services, is quite simple—and modifications to the data model will not require changes in the database since the mappings can insulate the conceptual model from the storage model. If some of the concepts I've discussed here regarding the Entity Framework are foreign to you, I highly recommend you check out my overview of the Entity Framework, which is available at msdn.microsoft.com/msdnmag/issues/07/07/DataPoints.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server. He can often be found speaking at industry conferences, such as VSLive, or blogging at codebetter.com/blogs/john.papa.


Cutting Edge: Customize Controls with AJAX Extenders, Part 2

Cutting Edge
Customize Controls with AJAX Extenders, Part 2
Dino Esposito

Code download available at: CuttingEdge2008_02.exe (434KB)
Contents
The Need for Masked Editing
The MaskedEdit Extender
Validating the Masked Input
Textbox Autocompletion
Building an Autocomplete Web Service
Styling the AutoComplete Extender
Incremental Search on Lists
Flying Context Menus
Wrapping Up

L
ast month I discussed how ASP.NET input controls such as textboxes and buttons could be enhanced by using AJAX control extenders. This month I'll add more advanced features, such as masked editing and autocompletion, using the Microsoft® .NET Framework 3.5 and the newest version of ASP.NET, which was in Beta 2 at the time of this writing. In addition, I'll be using the most recently available build of the ASP.NET AJAX Control Toolkit. For information on how to get the required software toolkits, refer to last month's installment of Cutting Edge (see msdn.microsoft.com/msdnmag/issues/08/01/CuttingEdge).

The Need for Masked Editing

In HTML, the only way to accept input data is through the <input> tag. In ASP.NET, the input tag is simply wrapped by the TextBox control. One problem with this control is that it doesn't restrict what a user can type in it. But with a little bit of JavaScript code, you can filter out unwanted text. That's what I covered last month. This month I'm adding masked editing, which allows characters to be filtered as they are typed and also to appear in a culture-specific format. Masked editing can be used for filtering, validation, automatic formatting, and localization. It can also be applied to a number of real-world data types, including dates, currency, time, ZIP codes, phone numbers, Social Security or VAT IDs, and so on. In the AJAX Control Toolkit, the MaskedEdit extender is a free component that when attached to a TextBox control allows you to control input for a number of common scenarios.

The MaskedEdit Extender

In the AJAX Control Toolkit, the MaskedEdit extender supports a few data formats, which are specified by the MaskEditType enum type:

public enum MaskedEditType
{
    None,
    Date,
    Number,
    Time,
    DateTime
}

You can use the extender to enter numbers, dates, times, and date/times. The extender decides its output based on given culture settings. The following code snippet shows the typical way to use the MaskedEdit extender with a textbox that accepts a date:

<asp:TextBox runat="server" ID="TextBox1" />
<act:MaskedEditExtender ID="MaskedEditExtender1" runat="server"
    TargetControlID="TextBox1"
    Mask="99/99/9999"
    MaskType="Date" />

The extender features a number of properties, which are listed in Figure A. You define an input mask mainly through two properties: Mask and MaskType. Mask has a default value of "" and specifies the mask of characters that is acceptable to the extender. MaskType has a default value of ""and indicates the mask type using any of the values defined by the MaskedEditType enumeration.

Figure A MaskedEdit Extender Properties (Close Figure A)

Property | Default Value | Description
AcceptAMPM | False | Boolean property that indicates whether an AM/PM symbol should be used.
AcceptNegative | None | Indicates whether a negative sign (-) is allowed. Feasible values come from the MaskedEditShowSymbol enumeration: None, Left, Right.
AutoComplete | True | Boolean property that indicates whether empty mask characters not specified by the user must be automatically filled in.
AutoCompleteValue | "" | Indicates the default character to use when Autocomplete is enabled.
Century | 1900 | Indicates the century to use when a date mask only has two digits for the year.
ClearMaskOnLostFocus | True | Boolean property that indicates whether to remove the mask when the textbox loses the input focus.
ClearTextOnInvalid | False | Boolean property that indicates whether to clear the textbox when the user has entered invalid text.
ClipboardEnabled | True | Boolean property that indicates whether to allow copy/paste with the clipboard.
ClipboardText | Your browser security settings don't permit the automatic execution of paste operations | Indicates the prompt text to use when a clipboard paste is performed.
CultureName | "" | Gets and sets the name of the culture to use.
DisplayMoney | None | Indicates whether the currency symbol is displayed. Feasible values come from the MaskedEditShowSymbol enumeration: None, Left, Right.
ErrorTooltipCssClass | "" | Gets and sets the CSS class for the tooltip message.
ErrorTooltipEnabled | False | Boolean property that indicates whether to show a tooltip message when the mouse hovers over a textbox with invalid content.
Filtered | "" | Gets and sets the list of valid characters for the mask type when the "C" placeholder is specified.
InputDirection | LeftToRight | Indicates the text input direction. Feasible values come from the MaskedEditInputDirection enumeration: LeftToRight, RightToLeft.
Mask | "" | Specifies the mask of characters that is acceptable to the extender.
MaskType | "" | Indicates the mask type using any of the values defined by the MaskedEditType enumeration.
MessageValidatorTip | True | Boolean property that indicates whether a help message will be displayed as the user types in the textbox.
OnBlurCssNegative | "" | Gets and sets the CSS class used when the textbox loses the input focus and contains a negative value.
OnFocusCssClass | "" | Gets and sets the CSS class used when the textbox receives the input focus.
OnFocusCssNegative | "" | Gets and sets the CSS class used when the textbox gets the input focus and contains a negative value.
OnInvalidCssClass | "" | Gets and sets the CSS class used when the text is not valid.
PromptCharacter | _ | Gets and sets the prompt character being used for unspecified mask characters.
UserDateFormat | None | Indicates a particular date format. Feasible values defined by the MaskedEditUserDateFormat enumeration.
UserTimeFormat | None | Indicates a particular time format. Feasible values defined by the MaskedEditUserTimeFormat enumeration.

The MaskType property informs the extender that the target control is going to accept a specific data type. The Mask property (of type string) indicates the sequence of characters that represents valid input for the textbox. For example, "12/6/07" and "12-09-2007" are both valid dates, but they use different input masks.

To build a mask, you use predefined symbols as placeholders. The list of supported symbols is in Figure 1. For example, the "999,999.99" mask causes your code to accept a number with a decimal separator and at most one thousand separator. Figure 2 shows the final user interface presented by a textbox extended with a masked editor. The appearance of the currency symbol is controlled by the DisplayMoney property, and each character the user must type is represented by a prompt. The default prompt is the underscore, but you can change it via the PromptCharacter property.

Figure 1 Predefined Placeholders for Input Masks (Close Figure 1)

Symbol | Description
9 | A numeric character
L | A letter
$ | A letter or a blank
C | A custom, case-sensitive character as defined by the Filtered property
A | A letter or a custom character as defined by the Filtered property
N | A numeric or custom character as defined by the Filtered property
? | Any character
/ | A date separator according to the current culture
: | A time separator according to the current culture
. | A decimal separator according to the current culture
, | A thousand separator according to the current culture
\ | An escape character
{ | The initial delimiter for repetition of masks
} | The final delimiter for repetition of masks
Figure 2 MaskedEdit Extender in Action
Figure 2 MaskedEdit Extender in Action(Click the image for a smaller view)
Figure 2 MaskedEdit Extender in Action
Figure 2 MaskedEdit Extender in Action (Click the image for a larger view)

For dates, you can also use extra properties such as AcceptAMPM, Century, and even a custom user format in addition to the predefined formats found in the MaskedEditUserDateFormat enumeration, shown here:

public enum MaskedEditUserDateFormat
{
    None,
    DayMonthYear,
    DayYearMonth,
    MonthDayYear,
    MonthYearDay,
    YearDayMonth,
    YearMonthDay
}

Many of the settings that influence the formatting applied by the MaskedEdit extender descend from the current culture. The CultureName property indicates the culture to apply. Note that this setting overrides the culture setting defined for the page through the UICulture attribute in the @Page directive.

Validating the Masked Input

While the masked extender provides dynamic formatting capabilities, an additional component—the masked validator—ensures that any text entered can be parsed back to the expected type:

  <act:MaskedEditValidator
      ID="MaskedEditValidator1" runat="server"
      ControlExtender="MaskedEditExtender1"
      ControlToValidate="TextBox2"
      IsValidEmpty="False"
      EmptyValueMessage="Number is required "
      InvalidValueMessage="Number is invalid" />

The MaskedEditValidator control is a custom validator that you optionally attach to the MaskedEdit extender so that the contents of the edited textbox are carefully verified. The validator ensures that the text matches the mask. It performs server and client validation and can be associated with a validation group, just like any other ASP.NET validator control. The properties exposed by the validator are listed in Figure B. The Text property of a masked textbox returns formatted text. For a date, the property returns something like "02/04/2007"; for a number input field, the property returns text like "1,200.00". The currency symbol is not included in the Text property, even though it is shown to the user in the page.

Figure B MaskedEditValidator Properties (Close Figure B)

Property | Description
AcceptAMPM | Indicates whether or not AM/PM is accepted on a time value.
ConTRolToValidate | Indicates the ID of the textbox to validate.
ConTRolExtender | Indicates the ID of the MaskedEditExtender conTRol attached to the textbox.
ClientValidationFunction | Gets and sets the name of the client JavaScript function used for custom validation.
EmptyValueBlurredText | Gets and sets the message displayed when the textbox doesn't have the input focus and is empty.
EmptyValueMessage | Gets and sets the message displayed when the textbox has the input focus but is empty.
InitialValue | Gets and sets the initial value of the textbox.
InvalidValueMessage | Gets and sets the message displayed when the textbox has the input focus but invalid content.
InvalidValueBlurredMessage | Gets and sets the message displayed when the textbox doesn't have the input focus but has invalid content.
IsValidEmpty | Indicates whether the textbox can be left empty.
MaximumValue | Gets and sets the maximum value of the input.
MaximumValueBlurredMessage | Gets and sets the message displayed when maximum value is exceeded and textbox doesn't have focus.
MaximumValueMessage | Gets and sets the message displayed when maximum value is exceeded and the textbox has focus.
MinimumValue | Gets and sets the minimum value of the input.
MinimumValueBlurredText | Gets and sets the message displayed when minimum value is exceeded and textbox does not have focus.
MinimumValueMessage | Gets and sets the message displayed when minimum value is exceeded and textbox has focus.
ValidationExpression | Gets and sets the regular expression used to validate the input.
TooltipMessage | Gets and sets the message displayed when the textbox has the input focus.

So how can you parse the value returned by Text into the logical data type—be it a date or a decimal? You can use the static Parse method on the DateTime and Decimal types. But you must pay attention to the culture you use. For example, "02/04/2007" can be either the fourth of February (U.S. culture) or the second of April (European culture). The fact is that there's no guaranteed matching between the culture used by the input page and the server page. There's a risk that the user will type the date according to the European culture and have it processed as U.S. culture data. Worse yet, the 1200 value entered using, say, Italian decimal and thousand separators in a numeric textbox may cause an exception to be thrown because the parser of the Decimal type defaults to the U.S. culture. Let's see how to work around these issues.

The key fact to remember is that extenders default to the en-US culture unless the CultureName property is explicitly set. On the server, the system defaults to the value of the UICulture property on the Page class.

In your codebehind class, you first obtain a CultureInfo object that reflects the culture used for the user interface. You can proceed as shown here:

string culture = "en-us";
if (!String.IsNullOrEmpty(MaskedEditExtender1.CultureName))
    culture = MaskedEditExtender1.CultureName;
CultureInfo info = new CultureInfo(culture);

Next, you call the Parse method specifying a format provider based on the selected culture:

NumberFormatInfo numberInfo = info.NumberFormat;
DateTimeFormatInfo dateInfo = info.DateTimeFormat;
DateTime when = DateTime.Parse(TextBox1.Text, dateInfo);
decimal amount = Decimal.Parse(TextBox2.Text, numberInfo);

Figure 3 shows the behavior of the same page when using different cultures for input.

Figure 3 Parsing Data Back to .NET Types Using Different Cultures
Figure 3 Parsing Data Back to .NET Types Using Different Cultures(Click the image for a smaller view)
Figure 3 Parsing Data Back to .NET Types Using Different Cultures
Figure 3 Parsing Data Back to .NET Types Using Different Cultures (Click the image for a larger view)

Textbox Autocompletion

Autocompletion is a feature you're undoubtedly familiar with. It predicts the word the user is typing based on the first few characters that are entered. Internet Explorer keeps track of all that's been typed into the address bar and form fields to populate autocomplete.

Of course, this feature is entirely browser-based and can be turned on and off for <input> and <form> tags by setting the autocomplete attribute to off. The autocomplete attribute is not a standard HTML attribute, but today nearly all browsers support it.

The AutoComplete extender in the AJAX Control Toolbox provides the same behavior for textbox controls, but it makes the developer responsible for all the logic that provides suggestions to the user. The extender creates a dropdown list-style panel and positions it right at the bottom of the textbox. You can style and animate the panel to your liking. Here's the code to associate an autocomplete extender with a textbox:

<act:AutoCompleteExtender runat="server" ID="AutoComplete1"
    TargetControlID="TextBox1"
    EnableCaching="true"
    MinimumPrefixLength="1"
    ServicePath="Suggestions.asmx"
    ServiceMethod="GetSuggestions" />

The extender is bound to a Web service that provides the words to populate the list. The MinimumPrefixLength property instructs the control about when to place a call to the Web service. The text already typed in will be used as input for the specified Web service method. The response is used to populate the dropdown list. Caching may be turned on, too. In this way, typing the same prefix more than once results in a single call to the Web service. Furthermore, depending on the way the Web service retrieves its data, you can also enable caching on the server and save some extra round-trips to a database or another remote data store.

The full list of properties supported by the AutoComplete extender can be found in Figure C. It should be noted that other extenders have properties in addition to those you'll find listed there. There is TargetControlID, which gets and sets the ID of the extended control, and Enabled, which allows the extender's capabilities to be turned on and off programmatically and is set to true by default. There's also the BehaviorID property, which sets the name of the client-side JavaScript object that provides the extended behavior. Finally, there are two more properties of interest—ClientState and EnableClientState. ClientState is a string property that holds the client state of the extender. The state is persisted using a hidden field whose name can be set through the ClientStateFieldID property. EnableClientState is a Boolean property that controls whether client state is enabled.

Figure C AutoComplete Extender Properties (Close Figure C)

Property | Description
Animations | Sets animations to be played when the flyout is shown and hidden.
CompletionInterval | Gets and sets the number of milliseconds after which the extender gets suggestions using the bound Web service.
CompletionListCssClass | Indicates the CSS class used to style the completion list flyout.
CompletionListHighlightedItemCssClass | Indicates the CSS class used to style the highlighted item in the completion list flyout.
CompletionListItemCssClass | Indicates the CSS class used to style the item in the completion list flyout.
CompletionSetCount | Gets and sets the number of suggestions to get from the bound Web service. Default is 10.
ContextKey | STRing property that indicates any page or user-specific information to pass to the bound Web service.
DelimiterCharacters | Indicates one or more characters that the extender will use to tokenize the text box content. The Web service will then use the last of these token to provide suggestions. Not set by default.
EnableCaching | Boolean property that indicates whether client-side caching is enabled. True by default.
FirstRowSelected | Boolean property that indicates whether the first option in the list will be automatically selected. False by default.
MinimumPrefixLength | Gets and sets the minimum number of characters in the text box buffer that TRigger the bound Web service. Default is 3.
ServiceMethod | Gets and sets the name of the method to invoke on the bound Web service.
ServicePath | Gets and sets the URL of the bound Web service.
UseContextKey | Boolean property that indicates whether the value of the ContextKey property should be used, if specified. False by default.

Building an Autocomplete Web Service

A Web service that works with the AutoComplete extender is an ASP.NET AJAX script service. It looks nearly the same as a regular ASP.NET Web service except that its class must be decorated with the ScriptService attribute. If you employ a Web service that lacks the attribute, each request to the associated ASMX endpoint causes an HTTP 500 error. While an exceptions interface is not shown to the end user (the extender manages to always degrade gracefully), no flyout with suggestions is ever displayed either:

[ScriptService]
public class SuggestionService : WebService
{
   ...
}

Note that while the script service doesn't require the WebService attribute, it can still use it to provide namespace information.

The name of any public method on the script service class that is flagged with the WebMethod attribute can be successfully assigned to the ServiceMethod property on the extender. A method that provides suggestions must have the following signature:

[WebMethod]
public string[] GetSuggestions(string prefixText, int count)

The first argument is the prefix text to generate suggestions. It matches the current content of the textbox, and its length is not smaller than the value of the MinimumPrefixLength property. The count parameter indicates how many suggestions are to be provided. The value of the count parameter comes from the value of the CompletionSetCount property.

If you plan to take advantage of the context key, which is a String property that indicates any page or user-specific information to pass to the bound Web service, then you should provide an override for any Web service methods you intend to use. Here's the signature:

public string[]GetSuggestions(
    string prefixText, int count, string contextKey)
{
   ...
}

The return value is always packed as an array of strings. Because of the ScriptService attribute, any communication between the server and client occurs through JavaScript Object Notation (JSON) strings.

You can leverage any supported attributes on Web service methods to improve the service's performance. For example, the CacheDuration attribute on the WebMethod attribute can force the service to cache the response of the method call for a specified duration. Likewise, you can enable session state if strictly required by the method's logic.

Figure 4 shows a sample service that offers suggestions on the customer name, which is coming from the Northwind database. A call made to a Web service every few characters typed may sound like a bad idea from a performance perspective. There are some factors, though, that make autocompletion more affordable than you might think. Client-side caching for the AutoComplete extender is based on local memory and is never persisted through a hidden field. The completion list associated with a given prefix is stored into an internal array using the prefix as the key. Then it is retrieved whenever the same prefix is isolated in the textbox buffer. As a result, if the user types the same prefix repeatedly, no new request is placed over the wire to the back-end service.

Figure 4 Sample Suggestion Service (Close Figure 4)
using System;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Data;
using System.Data.SqlClient;
using System.Web.Script.Services;

[WebService(Namespace = "MsdnMag.Articles")]
[ScriptService]
public class SuggestionService : System.Web.Services.WebService
{
    [WebMethod]
    public string[] GetSuggestions(string prefixText, int count)
    {
        DataView data = GetData();
        data = FilterData(data, prefixText);

        int totalCount = data.Count;
        if (data.Count > count)
            totalCount = count;
        string[] suggestions = new string[totalCount];

        int i = 0;
        foreach (DataRowView row in data)
        {
            suggestions[i++] = (string) row["companyname"];
            if (i >= count)
                break;
        }

        return suggestions;
    }

    private DataView GetData()
    {
        DataView view =
            (DataView)HttpContext.Current.Cache["Suggestions"];
        if (view == null)
        {
            SqlDataAdapter adapter = new SqlDataAdapter(
                "SELECT * FROM customers", "...");
            DataTable table = new DataTable();
            adapter.Fill(table);
            view = table.DefaultView;

            HttpContext.Current.Cache["Suggestions"] = view;
        }

        return view;
    }

    private DataView FilterData(DataView view, string prefix)
    {
        view.RowFilter = String.Format("companyname LIKE '{0}%'",
            prefix);
        return view;
    }
}

Client-side caching is clearly more effective than server-side caching, which would save you a few round-trips to the database but not round-trips to the Web server. In the code shown in Figure 4, the service loads all customers in the database and stores the list in the ASP.NET cache. It then retrieves the list of customers from there and filters the names that match the prefix.

As mentioned, the Web service you use to provide suggestions has to be an ASP.NET AJAX script service and will talk to the client via JSON. You'll need to decide if the service will be publicly available to SOAP clients as well. ASP.NET AJAX services have dual functionalities by default—both JSON and SOAP. You can turn off SOAP clients by adding the following to the web.config file of the host application:

<system.web>
  <webServices>
    <protocols>
      <clear />
    </protocols>
  </webServices>
</system.web>

With ASP.NET 3.5, you can also bind the AutoComplete extender to a Windows® Communication Foundation (WCF) service using a service (SVC) endpoint. The extender uses the following JavaScript code to place the call:

Sys.Net.WebServiceProxy.invoke(
   this.get_servicePath(),
   this.get_serviceMethod(),
   false,
   params,
   Function.createDelegate(this, this._onMethodComplete),
   Function.createDelegate(this, this._onMethodFailed),
   text);

For the WebServiceProxy class in the Microsoft AJAX client library, all that matters is that the specified endpoint can successfully handle requests with the content-type request header set to "application/json; charset=utf-8". To create a WCF service for providing suggestions, you need a class similar to the following:

[ServiceContract(Namespace = "MsdnMag.Services")]
[AspNetCompatibilityRequirements(
     RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class MySuggestionService
{
    [OperationContract]
    public string[] GetSuggestions(string prefixText, int count)
    {
      ...
    }
}

In addition, you need an SVC endpoint, as shown here:

<%@ ServiceHost Language="C#"
                Service="MySuggestionService"
                CodeBehind="~/App_Code/WcfSuggestions.cs" %>

To register the service, you can add an explicit endpoint to the serviceModel section of the web.config file using the webHttpBinding binding model. Or, you can specify the Factory attribute in the @ServiceHost directive and make it point to the following class:

System.ServiceModel.Activation.WebScriptServiceHostFactory

Styling the AutoComplete Extender

The AutoComplete extender supports properties through which you indicate the CSS classes to be used to style parts of the control. In particular, you can style the overall flyout region including borders and background color. You can also use different settings for all items and the selected item. No predefined style is supplied for the alternating item. By default, the first element in the dropdown completion list is not selected. The FirstRowSelected property, though, can be used to force the automatic selection of the first item.

The completion list shows up after the specified number of milliseconds has elapsed. This is controlled via the CompletionInterval property and set to one second by default. After that time has elapsed, the extender places the call to the service to get the list of suggestions.

You can use the Animations property to add an action to the procedure that brings up the list. You can define animations for two events—OnShow and OnHide. Animations are performed through the Animation extender and its related framework (see the example in Figure 5).

Figure 5 Using the Animations Property (Close Figure 5)
<Animations>
    <OnShow>
        <Sequence>
            <OpacityAction Opacity="0" />
            <HideAction Visible="true" />
            <Parallel Duration=".4">
                <FadeIn />
            </Parallel>
        </Sequence>
    </OnShow>
    <OnHide>
        <Parallel Duration=".4">
            <FadeOut />
        </Parallel>
    </OnHide>
</Animations>

The animation script fades the completion list in and out. The OpacityAction node sets the transparency, and HideAction displays the completion list along with any other visual effects. Finally, the FadeIn and FadeOut actions perform fading. Figure 6 shows an animated completion list.

Figure 6 The AutoComplete Extender in Action
Figure 6 The AutoComplete Extender in Action(Click the image for a smaller view)
Figure 6 The AutoComplete Extender in Action
Figure 6 The AutoComplete Extender in Action (Click the image for a larger view)

Incremental Search on Lists

Have you ever tried to find a particular item in a long list—for example, a country name in the list of all countries in the world? If you're fast enough to type a few consecutive letters, you may get the country you want, but even a few milliseconds of hesitating means that the next letter you type becomes the first letter of the search term. The ListSearch extender fixes this limitation. Note that the extender only applies to ASP.NET Web and HTML controls and can't be used with regular HTML elements such as SELECT or OPTION.

So, basically, the primary benefit of the ListSearch extender is that it lets you search for items in a list control by typing and prevents your search string from being lost after a few seconds; instead, it will remain there until you press Esc or post back to the server (both full and partial postbacks included). Any other character you type is appended to the search string and used to further narrow the list of items. Here's how you use a ListSearch extender:

<act:ListSearchExtender ID="ListSearchExtender1" runat="server"
     TargetControlID="DropDownList1"
     PromptCssClass="Prompt" />

The extender features three main properties: PromptText, PromptCssClass, and PromptPosition. All of them refer to the prompt text—the string that appears around the target control to provide visual feedback about the text being searched. The default value for PromptText is "Type to search." This text is usually displayed just above the target list control, as soon as the control gets focus. The PromptCssClass property gets and sets the name of the CSS class to apply to the prompt message. The prompt text is replaced by the search text as typed by the user. The Escape and Delete buttons play a key role here. The Esc button clears the current search and resets the list to the default selected item. The Del button removes the last typed character from the search text and updates the selection in the list. In addition, the Animations property works as I discussed for the AutoComplete extender a moment ago.

Flying Context Menus

It's good to offer users options for what they can do just as they are about to do it. Years ago, tooltips were a huge improvement because they provided additional information on the fly about the role of a certain control. The advent of Dynamic HTML made HTML tooltips possible.

Today, the HoverMenu extender makes it possible to display a flyout panel next to any ASP.NET server control. One of the most common uses of this extender is to display context menus that are activated by mouse movement events rather than a right-click.

The HoverMenu extender is activated when the user moves the mouse over the control. As a result, the specified popup panel is displayed near the control; the actual position can be controlled programmatically. In addition, a CSS style can be applied to the control to mark it with a hot state.

Figure 7 shows a radio button menu that provides suggestions about the text to insert in the textbox. Here's the declaration for the related extender:

Figure 7 HoverMenu Extender in Action
Figure 7 HoverMenu Extender in Action(Click the image for a smaller view)
Figure 7 HoverMenu Extender in Action
Figure 7 HoverMenu Extender in Action (Click the image for a larger view)

<act:HoverMenuExtender ID="HoverMenu1" runat="server"
    TargetControlID="TextBox1"
    PopupControlID="Panel1" />

The Panel1 control contains the list of radio buttons. To make sure that any selection makes its way into the textbox target control, you define a SelectedIndexChanged event in the radio button list and wrap everything in an UpdatePanel control. (See the companion source code in the download for this column for more details.)

The HoverMenu extender features a few properties in addition to PopupControlID and TargetControlID. The HoverCssClass property indicates the CSS class to apply to the target control when the hover menu is visible. PopupPosition indicates the position of the popup with respect to the target control—center, top, right, left, and bottom are possible values. OffsetX and OffsetY properties set horizontal and vertical offsets from the determined position. Finally, PopDelay indicates the delay between the DOM event and the display of the popup. 100 milliseconds is the default.

The DOM event that triggers the HoverMenu extender is the mouseover. The HoverMenu extender is not very different from the Popup extender. The key differences are in the activation mechanism and trigger. The Popup extender is activated when the user gives focus to a given target control. For the HoverMenu extender to activate, the user only has to move the mouse over the control.

Wrapping Up

The richer and more interactive you want your Web site to be, the more JavaScript you'll need. However, while hand-coded JavaScript is fine for simple features, it is often insufficient to achieve greater levels of interactivity. For these more complex features, you really need extenders and the Microsoft AJAX Client Library (see Cutting Edge, December 2007, at msdn.microsoft.com/msdnmag/issues/07/12/cuttingedge).

This installment of Cutting Edge completes my review of extenders in the Microsoft AJAX Control Toolkit. However, the toolkit contains much more than what I discussed here. To see what I mean, pay a visit to asp.net/ajax.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Introducing ASP.NET AJAX (Microsoft Press, 2007). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can reach him at cutting@microsoft.com or join his blog at weblogs.asp.net/despos.


Office Space: Security Programming in SharePoint 2007

Office Space
Security Programming in SharePoint 2007
Ted Pattison

Code download available at: OfficeSpace2008_02.exe (209KB)
Contents
External Security Principals and the SPUser Object
Adding Authenticated and External Users
Working with Permission Levels
WSS Groups
Identity, Elevation, and Impersonation
Securable Objects
Wrapping Up

Y
ou may already know the fundamentals of security programming with Windows® and ASP.NET security, but how well do you know the security layer that Windows SharePoint® Services 3.0 (WSS) adds on top? In this month's installment of Office Space, I will highlight new security terms and concepts that are introduced with WSS, and I will give you a jumpstart into the world of security programming using the WSS object model.

I recommend that you download the sample project that accompanies this column and follow along with the code I will present throughout the rest of this column. The project has been configured to run a batch file after the build process, which will compile all of the project components into a WSS solution package and install the package on the local WSS farm. Once you have built the project and installed the solution, you can navigate to any site collection and enable the feature named Security Demo, which is scoped to the site collection. The Site Actions menu will then allow you to navigate to custom application pages, which have code behind them demonstrating WSS security programming techniques.

External Security Principals and the SPUser Object

Most security models are based on security principals. Each security principal represents either a user or a group. Users have accounts against which they authenticate. Once authenticated, each user takes on an identity. In the scenario where a user has authenticated against a Windows account, you can use the Microsoft® .NET Framework security classes in the System.Security namespace to retrieve an identity that points back to the specific Windows account and allows you to discover the user's login name:

WindowsIdentity identity = WindowsIdentity.GetCurrent();
string WindowsLogin = identity.Name;

Using a WindowsIdentity, you can dynamically create a WindowsPrincipal, which will let you perform a test to see whether the current user is a member of an Active Directory® group or a local Windows group, like so:

WindowsIdentity identity = WindowsIdentity.GetCurrent();
WindowsPrincipal principal = new WindowsPrincipal(identity);
if( principal.IsInRole(@"LITWAREINC\AllFTE") ){
  // perform operation allowed for fulltime employees
}

ASP.NET supports both Windows authentication and Forms-based authentication (FBA). The User object in ASP.NET abstracts away the dependency on Windows accounts by modeling the User object on the IPrincipal interface instead of the WindowsPrincipal class. The ASP.NET runtime dynamically creates different types of IPrincipal objects depending on whether the current user has authenticated using a Windows account or an FBA account:

IPrincipal AspUser = HttpContext.Current.User;
string AspUserName = AspUser.Identity.Name;

The ASP.NET User object also provides a way to check to see if a user belongs to a particular role using the IsInRole method. For Windows users, the IsInRole method lets you see whether the current user is a member of an Active Directory group. If you are using FBA accounts along with an ASP.NET Role Provider, you can also use the IsInRole method to check whether the FBA user has been added to a specific ASP.NET role:

IPrincipal AspUser = HttpContext.Current.User;
if(AspUser.IsInRole("Site Administrators") {
  // perform privileged operation
}

The act of authentication produces some form of a receipt that is used by the system at run time to represent a user identity and to track membership in groups or roles. When a user authenticates against a Windows account, the receipt of authentication is a Windows security token. When the user authenticates against an FBA account, the receipt of authentication is an HTTP cookie created by the ASP.NET runtime and a specific authentication provider.

It's important to understand that WSS does not provide support for authenticating users. Instead, WSS makes use of underlying authentication components supplied by the various ASP.NET authentication providers. The value that WSS brings to securing a site has to do with configuring authorization and access control. WSS makes it possible to track external security principals‑such as Windows users, FBA users, Windows groups, and ASP.NET roles‑within the scope of a site collection. With WSS, you can also configure the permissions that are assigned to these external principals, which, in effect, grant users permissions to access securable WSS objects such as sites, lists, items, and documents.

Note that the site collection plays a significant role when you're configuring authorization and access control. WSS considers each site collection to be its own separate island when it comes to tracking external principals and configuring permissions. The high-level design of WSS intentionally isolates each site collection so that a user's security settings in one site collection do not impact permissions or access control policy in another site collection.

The WSS object model represents external security principals using SPUser objects. You can retrieve the SPUser object for the current user through the current SPWeb object:

SPWeb site = SPContext.Current.Web;
SPUser user = site.CurrentUser;
string DisplayName = user.Name;
string Login = user.LoginName;
string EMail = user.Email;
string User Notes = user.Notes;

The SPUser object exposes properties of an external security principal, such as login name, display name, and e-mail address. These properties are usually retrieved from an underlying user repository, such as an Active Directory domain, when the external principal is added to a site. The SPUser object also exposes properties that track WSS-specific metadata, such as the Notes field.

WSS maintains the profile data for external users, groups, and roles in a hidden list, which is known as the User Information List. Every time WSS provisions a new site collection, it automatically creates the User Information List as a hidden list in the top-level site. WSS then adds a new profile for each external principal the first time that principal is assigned permission or the first time it passes a security check to access a securable object. Note that the user profile stored in the User Information List does not extend across site collections—when users update their profile settings in one site collection, there are no changes to that user's profile settings in other site collections.

Another potential source of confusion is that SPUser objects do not always represent actual users. SPUser objects can also represent Active Directory groups and ASP.NET roles. WSS tracks a profile for each of these external principal types in the User Information List along with the profile data for external users.

Many of the programmatic aspects of the SharePoint security model are exposed at the site level through SPWeb objects. This is the case if you want to discover which users are members of the current site. An SPWeb object exposes three different collections of users, as shown in this code fragment:

SPWeb site = SPContext.Current.Web;
SPUserCollection c1 = site.Users;
SPUserCollection c2 = site.AllUsers;
SPUserCollection c3 = site.SiteUsers;

The Users collection has the smallest membership of these three collections. This collection includes all the external principals that have been explicitly assigned permissions within the current site.

The AllUsers collection includes all members of the Users collection, plus external users that have accessed objects within the site using implicit permissions through group or role membership. For example, imagine a user named Brian with the login of LITWAREINC\BrianC that has never been given explicit permissions to access a site and view a particular list. However, he might still be able to view the list because of his membership within an Active Directory group that has been configured with list view permissions. When Brian first accesses the site or one of its objects (say, a list using implicit permissions), he is added as a member of the AllUsers collection, but he is not added as a member of the Users collection.

The SiteUsers collection is an aggregation that combines membership for each AllUsers collection within the current site collection. The membership of this collection includes all external principals that have been assigned permissions to any object within the site collection as well as all external users that have been granted access to any of the site collection's objects using implicit permissions.

Adding Authenticated and External Users

So how do you create a new WSS user profile for a user that has authenticated using an Active Directory account? If you need to create a custom user interface component that allows standard users or the site collection owner to select a user or group from an Active Directory domain, you really should learn how to use the PeoplePicker control (see Figure 1). This is a handy, reusable control type that ships with WSS. You can add this control to a custom application page or a User control using a control tag that looks like this:

Figure 1 PeoplePicker Control
Figure 1 PeoplePicker Control(Click the image for a smaller view)
Figure 1 PeoplePicker Control
Figure 1 PeoplePicker Control (Click the image for a larger view)

<SharePoint:PeopleEditor
       ID="pickerPrincipal"
       AllowEmpty="false"
       ValidatorEnabled="true"
       MultiSelect="false"
       SelectionSet="User, SecGroup, DL"
       Width="280px"
       runat="server"   />

In this sample, I have configured the PeoplePicker control by assigning the SelectSet property with values of User, SecGroup, and DL. These SelectSet settings configure the control to allow the user to select and resolve a user, group, or distribution list against Active Directory.

You can programmatically access the PeoplePicker control properties to retrieve the associated login account names for the underlying accounts after the user has selected one or more security principals with the control. Then you can supply the code that actually adds these principals as site members and configures their access rights.

Now I should show you how to add an external user or group as a site member. After a brief look at the WSS object model, you might think that you should simply add external security principals directly into one of the SPUser collections, such as SiteUsers:

SPWeb site = SPContext.Current.Web;

site.SiteUsers.Add(@"LITWAREINC\BrianC",
                    "brianc@litwareinc.com",
                    "Brian Cox",
                    "Notes about Brian Cox");

site.SiteUsers.Add(@"LITWAREINC\AllFTE",
                    "allFTE@litwareinc.com",
                    "All Full-time Employees",
                    "Notes about FTE DL");

While this approach does create a profile in the User Information List for an external principal, it has little effect on security since it doesn't assign a permission. A better way to add a new external security principal involves assigning permissions so that users have access within the current site. However, you first need to learn how to create and assign permission levels.

Working with Permission Levels

A permission level is a named set of permissions that is defined within the scope of a site. WSS includes four built-in permission levels: Read, Contribute, Design, and Full Access. If you need greater granularity than this, you can create your own custom permission levels using the WSS object model or through the standard WSS administrative pages that are accessible to the site collection owner.

Permission levels are sometimes called roles, and they are represented in the WSS object model using SPRoleDefinition objects. You can assign a permission level to an external user or groups using a SPRoleAssignment object. For example, here I assign the built-in Contribute permission level to the Windows user with a login name of LITWAREINC\BrianC:

SPWeb site = SPContext.Current.Web;
SPRoleDefinition role = site.RoleDefinitions["Contribute"];
SPRoleAssignment roleAssignment;
roleAssignment = new SPRoleAssignment(@"LITWAREINC\BrianC",
                                       "brianc@litwareinc.com",
                                       "Brian Cox",
                                       "Notes about Brian Cox");

roleAssignment.RoleDefinitionBindings.Add(role);
site.RoleAssignments.Add(roleAssignment);

This technique makes it unnecessary to add the user to one of the SPUser collections since that is done automatically by WSS when an external user or group is assigned a permission for the first time within a site. The code you've just seen will create a user profile in the User Information List if one does not exist, and it will also add the user as a member of the current site's Users collection.

WSS Groups

While the WSS security model represents external security principals as SPUser objects, it also provides WSS groups as a means to ease the configuration of permissions within the scope of a site collection. For example, you can design a set of WSS groups within a site collection for specific user roles, such as Site Members, Content Manager, and Site Administrators. Once you have done this, you can configure the site's security settings by simply assigning permission levels to WSS groups instead of assigning permission levels directly to SPUser objects.

The obvious benefit to creating WSS groups is that they help to eliminate the ongoing need to reconfigure permissions as new external users and groups are added and removed. Instead, you can configure permissions once and for all when the site is created. Then, moving forward, you simply add and remove external users and groups to and from WSS groups. WSS groups really follow the exact same design principles as Active Directory groups, with the main difference being that WSS groups are defined and exist only within the scope of a single site collection.

WSS groups are represented in the WSS object model as SPGroup objects. The SPWeb object exposes two collections of SPGroup objects, named Groups and SiteGroups. The Groups collection includes all the WSS groups that have been directly assigned permission in the current site, while the SiteGroups collection is a super set of the Groups collection and includes all of the WSS groups that have been created inside the current site collection.

When you want to create a new WSS group, you should call the Add method exposed by the SiteGroups collection and then assign the new WSS group with one or more permission levels within a target site. Figure 2 shows an example of creating a new WSS group named Site Members and assigning it the built-in Contribute permission level within the current site.

Figure 2 Creating a New WSS Group (Close Figure 2)
SPWeb site = SPContext.Current.Web;
SPUser currentUser = site.CurrentUser;

// create new group
site.SiteGroups.Add("Site Members", currentUser, currentUser,
                     "Site Group created at " + DateTime.Now.ToString());

// assign permission level to new group
SPGroup NewGroup = site.SiteGroups["Site Members"];
SPRoleAssignment roleAssignment = new SPRoleAssignment(NewGroup);
SPRoleDefinition permLevel = site.RoleDefinitions["Contribute"];
roleAssignment.RoleDefinitionBindings.Add(permLevel);
site.RoleAssignments.Add(roleAssignment);

Once you have created a new WSS group, it is pretty easy to add external users and groups as members. An SPGroup object exposes an AddUser method, which accepts an SPUser object, which in turn allows you to add external users and groups:

SPWeb site = SPContext.Current.Web;
SPUser currentUser = site.CurrentUser;
SPGroup group = site.SiteGroups["Site Members"];
SPUser user1 = site.SiteUsers[@"LITWAREINC\BrianC"];
SPUser user2 = site.SiteUsers[@"LITWAREINC\AllFTE"];
group.AddUser(user1);
group.AddUser(user2);

Identity, Elevation, and Impersonation

The worker process for a WSS Web application is controlled through IIS Application Pools. The worker process identity for a Web application warrants your attention and is configurable through the SharePoint Central Administration application. You should configure the worker process identity for a Web application in terms of domain accounts (such as LITWAREINC\SP_WorkerProcess) instead of relying on local accounts (such a NETWORK SERVICE).

Note that the worker process identity for a Web application must be a privileged Windows account that has been configured with SQL Server permissions to read and write to one or more content databases. The worker process identity for the Web application running the SharePoint Central Administration site must be even more privileged since it requires read/write permissions to the farm's configuration database.

When the code behind a Web Part or a custom application page executes in response to a user request, the code does not execute under the worker process identity of the hosting Web application. Instead, WSS uses impersonation to switch the Windows security context over to another Windows account. In fact, if you look inside the web.config file for a WSS Web application, you will see the following entry:

<configuration>
  <system.web>
    <identity impersonate="true" />
  </system.web>
</configuration>

When a request is executed for a user that has been authenticated against a Windows account, the request impersonates the current user's Windows identity. However, the same approach is not possible for FBA users because FBA authentication does not create a Windows security token and has no Windows identity. Therefore, requests for users running under FBA authentication impersonate the identity of the Windows account that has been configured for anonymous access. The default assignment for this account in IIS is the IUSER_MACHINENAME account, but you can—and typically should—reconfigure this to point to a domain account.

Now it's time to step back and view WSS security programming at a higher level. The WSS security model often forces developers to differentiate between Windows identity and WSS user identity. This might not be so obvious within a request where the current Windows identity and the current WSS user identity both point to the same Windows login. However, in scenarios that utilize FBA, things get a bit more complicated. For example, the WSS user identity could point to an FBA user named Andrew while the underlying Windows identity is based on the IUSER_MACHINENAME account. As your code attempts to access WSS objects, WSS runs access checks using the user's WSS identity. And as your code attempts to access external objects outside of WSS, such as files maintained by the Windows operating system, the OS runs access checks using the Windows identity under which the code is currently executing.

There are times when you need your code to execute with greater permissions than is possessed by the current user. For example, imagine a scenario in which your code must write data to a list when processing a request for a user that has nothing more than read-only permissions. By default your code runs with the same permissions as the current user. However, you can make a call to the RunWithElevatedPrivileges method of the SPSecurity class to elevate the security context of your code. Note that a call to RunWithElevatedPrivileges elevates both the WSS user identity and the Windows identity.

Now imagine a scenario where a user has authenticated against a Windows account with a login of LITWAREINC\BrianC. A call to RunWithElevatedPrivileges will elevate the WSS user identity to the SHAREPOINT\System account. The SHAREPOINT\System account is built into the WSS runtime and is all powerful within the WSS authorization model. A call to RunWithElevatedPrivileges will also switch the Windows identity of the executing code so it runs under the worker process identity of the current Web Application:

// BEFORE ELEVATION
// WSS User identity = LITWAREINC\BrianC
// Windows identity = LITWAREINC\BrianC

SPSecurity.RunWithElevatedPrivileges(delegate() {
  // AFTER ELEVATION
  // WSS User identity = SHAREPOINT\System
  // Windows identity = LITWAREINC\SP_WorkerProcess
});

In some scenarios, you might choose to call the RunWithElevatedPrivileges method to change the Windows identity of the current call before you attempt to access a file inside the Windows file system or a SQL Server database. Also note that switching the Windows identity over to a process identity such as LITWAREINC\SP_WorkerProcess can eliminate the need to configure delegation within an Active Directory environment. This can be very valuable if you have custom Web Parts that access data in a remote SQL Server database using Windows integrated authentication.

And then there are other scenarios where you might want to call the RunWithElevatedPrivileges method to elevate the WSS user identity to SHAREPOINT\System so that your code can perform operations not allowed by the current user. Once your code is running as SHAREPOINT\System, you can do pretty much anything you want within the WSS authorization subsystem.

There is one tricky aspect of calling RunWithElevatedPrivileges to elevate to the SHAREPOINT\System account. For example, imagine you call RunWithElevatedPrivileges and then you attempt to access objects in the current site collection or site through the SPContext.Current property. You probably wouldn't expect your code to fail, but it might:

SPSecurity.RunWithElevatedPrivileges(delegate() {
  SPSite siteCollection = SPContext.Current.Site;
  // next line fails if current user is Contributor
  string siteCollectionOwner = siteCollection.Owner;
});

Why does this sample code fail after you have elevated the WSS user identity to SHAREPOINT\System? It has to do with when the SPSite object was created. The permissions in effect on an SPSite object and its child SPWeb objects do not depend on the current WSS user identity. Instead, the permissions depend on the WSS user identity at the time the SPSite object is created. Here, the SPSite object accessible through SPContext.Current was created earlier in the request, before your code had a chance to switch its WSS user identity.

Thus, the technique you should use requires that you create a new SPSite object after you have called RunWithElevatedPrivileges and elevated WSS user identity to SHAREPOINT\System:

SPSecurity.RunWithElevatedPrivileges(delegate() {
  using (SPSite elevatedSiteCollection = new SPSite(this.Site.ID)) {
    using (SPWeb elevatedSite =
      elevatedSiteCollection.OpenWeb(this.Web.ID)) {
      // access elevatedSiteCollection and
      //elevatedSite as SHAREPOINT\System
    }
  }
});

This makes it possible to open a site collection and the sites within so that your code can access objects as SHAREPOINT\System.

You might also find it necessary to impersonate a specific WSS user identity. This is common when writing the code for an event handler or a custom workflow template where the code is running as SHAREPOINT\System by default. For example, you might want to impersonate a specific WSS user identity before creating a new object so that WSS user is recognized as the owner of the new object.

In order to impersonate a WSS user identity, you must first create an SPUserToken object. You can do this by accessing the UserToken property of an SPUser object. Once you have the SPUserToken object, you can use it to create a new SPSite object using an overloaded version of the SPSite class constructor. This technique is shown in Figure 3.

Figure 3 Impersonating a WSS User Identity (Close Figure 3)
SPWeb siteCollection = SPContext.Current.Site;
SPWeb site = SPContext.Current.Web;

// get SPUser object and acquire token
SPUser targetUser = site.SiteUsers[@"LITWAREINC\BrianC"];
SPUserToken token = targetUser.UserToken;

// create new SPSite and SPWeb object to impersonate user
using (SPSite impersonatedSiteCollection =
              new SPSite(siteCollection.ID, token)) {
  using (SPWeb impersonatedSite =
               impersonatedSiteCollection.OpenWeb(site.ID)) {
    // WSS identity switched to impersonate BrianC
    // Windows identity does not change
  }
}

There are a few important things I should point out about using WSS user impersonation. First, impersonating a WSS user is different than a call to RunWithElevatedPrivileges because it does not change the current Windows identity. If, for example, a request is running under the Windows identity of LITWAREINC\SP_WorkerProcess before you impersonate a WSS user, the code continues to run under the same Windows identity. WSS user impersonation does not change the current Windows identity over to the identity of the impersonated user.

It is also important to point out that code must be running in a privileged state in order to impersonate another user. This isn't something you have to worry about in an event handler or a custom workflow template since the code is running as SHAREPOINT\System by default. However, code inside a Web Part or behind a custom application page might need to call RunWithElevatedPrivileges before it even has the ability to impersonate another WSS user identity.

Securable Objects

The real power in configuring WSS security involves the flexibility afforded by securable objects, such as sites, lists, and list items. Each securable object can contain an Access Control List (ACL), which is a binary data structure that WSS uses at run time to determine whether a security principal has been granted access. By default, the only securable object that has an ACL is the top-level site. Child objects (such as list, list items, and child site) all inherit the ACL of their parent unless they break the inheritance and thereby provide a unique ACL of their own.

The WSS object model contains an interface named ISecurableObject that models a securable object within a WSS site collection (see Figure 4). The ISecurableObject interface, which is implemented by SPWeb objects, SPList objects, and SPItem objects, provides a basis for conducting access checks at run time as well as for configuring permissions.

Figure 4 The ISecurableObject Interface
Figure 4 The ISecurableObject Interface(Click the image for a smaller view)
Figure 4 The ISecurableObject Interface
Figure 4 The ISecurableObject Interface (Click the image for a larger view)

As you begin to configure permissions within a site collection, it's important to understand that all of the sites, lists, and list items make up a single hierarchy of securable objects. By default, just the top-level site contains a unique ACL and defines permission-level assignments that dictate what permissions users have to access objects. All child objects inherit their permissions from the top-level site. However, you can be more granular about configuring access control by giving a securable object its own unique set of permission-level assignments. For example, the code shown in Figure 5 allows you to create a new document library and to configure it with a unique set of permissions.

Figure 5 Configuring a Unique Set of Permissions (Close Figure 5)
SPWeb site = SPContext.Current.Web;
Guid listID = site.Lists.Add("Proposals",
                            "Library desc",
                            SPListTemplateType.DocumentLibrary);

SPDocumentLibrary doclib = (SPDocumentLibrary)site.Lists[ListID];
doclib.OnQuickLaunch = true;
doclib.BreakRoleInheritance(false);
SPUser AllFteGroup = Web.SiteUsers[@"LITWAREINC\AllFTE"];
SPRoleAssignment assignAllFteGroup = new SPRoleAssignment(AllFteGroup);
SPRoleDefinition roleDesign = this.Web.RoleDefinitions["Read"];
assignAllFteGroup.RoleDefinitionBindings.Add(roleDesign);
doclib.RoleAssignments.Add(assignAllFteGroup);
doclib.Update();

This sample code breaks the default permission inheritance from the parent using a call to BreakRoleInheritance. If you call BreakRoleInheritance and pass a parameter value of true, the securable object is initially configured with an ACL that is a copy of the parent object's ACL. If you call to BreakRoleInheritance and pass a parameter value of false, the securable object is initially configured with an empty ACL. That means this document library provides no access to users who are not either owners or site administrators.

Windows SharePoint Services 3.0 has added a welcome security enhancement that allows you to configure permissions down to the level of the item or document. This is made possible through the WSS object model, since SPListItem objects also implement the ISecurableObject interface.

The code in Figure 6 creates a new document within a document library and then configures it with a unique set of permissions that differ from its parent document library. Note that this code uses a utility method, called WriteDocument, that accepts an SPDocumentLibrary reference and a file name. The method implementation uses the Office Open XML File Formats to create a Word document and to write it back to the document library. The WriteDocument method returns an SPFile reference, which can then be used to access the SPListItem associated with the document, allowing you to break permission inheritance and assign a unique set of permissions.

Figure 6 Set of Permissions that Differs from Its Parent (Close Figure 6)
SPWeb site = SPContext.Current.Web;
Guid listID = site.Lists.Add("Proposals",
                            "Library desc",
                            SPListTemplateType.DocumentLibrary);

SPDocumentLibrary doclib = (SPDocumentLibrary)Web.Lists[ListID];
doclib.OnQuickLaunch = true;
doclib.Update();
SPFile doc1 = WriteDocument(doclib, "Adventure Works Merger.docx");
doc1.Item.BreakRoleInheritance(false);
SPGroup group = Web.Groups["Litware Contact Managers"];
SPRoleAssignment assignContribute = new SPRoleAssignment(group);
SPRoleDefinition roleContibute = this.Web.RoleDefinitions["Contribute"];
assignContribute.RoleDefinitionBindings.Add(roleContibute);
doc1.Item.RoleAssignments.Add(assignContribute);
doc1.Item.Update();

Wrapping Up

I know that this discussion has been somewhat of a whirlwind tour through the security model of WSS. I've shown you how WSS tracks external security principals at the level of the site collection with a profile within the User Information List and explained how WSS represents these external security principals in the WSS object model using SPUser objects. I've also demonstrated how WSS provides support for WSS groups and presented a few programming techniques for elevating privileges and impersonating WSS users. These techniques provide the power and flexibility you need when creating a real-world application.

While WSS relies on an underlying system of components to perform authentication, it does take on the responsibility of authorization and access control. The WSS authorization model is largely based on a named set of permissions known as permission levels or roles. A permission level can be assigned to an SPUser object, but in practice you should generally choose to assign permission levels to WSS groups.

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group (www.TedPattison.net). Ted has just completed his book titled Inside Windows SharePoint Services 3.0 for Microsoft Press.


Test Run: The Microsoft UI Automation Library

Test Run
The Microsoft UI Automation Library
Dr. James McCaffrey

Code download available at: TestRun2008_02.exe (178KB)
Contents
Inside StatCalc
The UI Test Automation Code
Extending the Test Harness

T
here are several techniques you can use to test the user interface of a Windows® application. For example, you can use System.Reflection classes to test Microsoft® .NET Framework applications or you can test both .NET and native applications by calling Win32® API functions such as FindWindow—either with unmanaged C++ or with C# or Visual Basic® by using the P/Invoke mechanism.

In this month's column, I will show you how to get started with UI test automation using the new Microsoft UI Automation library, which is included in the .NET Framework 3.0 as part of Windows Presentation Foundation (WPF). You can think of it as a successor to the Microsoft Active Accessibility (MSAA) library, which was originally designed only for accessibility features but was somewhat coincidentally found to be useful for and adapted to UI automation. On the other hand, the UI Automation library was specifically designed from the beginning for both accessibility and UI test automation tasks. You can use the UI Automation library to test Win32 applications, .NET Windows Forms applications, and WPF applications on host machines running operating systems that support the .NET Framework 3.0 (such as Windows XP, Windows Vista®, Windows Server® 2003, and Windows Server 2008).

Let me cut to the chase and state that I believe the development of the UI Automation library is one of the most important advances in test automation to date, and it will quickly become the most commonly used technique for UI test automation for Windows applications. Compared with alternative approaches to UI automation, the UI Automation library is more powerful, often easier to use, and more consistent. In much the same way that the .NET Framework transformed software application development, I think it's very likely that the UI Automation library will revolutionize UI test automation.

A good way to show you where I'm headed is with a screenshot. Figure 1 shows a simple Windows application I'm going to test. The application is called StatCalc, and it computes an arithmetic, geometric, or harmonic mean of a set of integers. An arithmetic mean is just a simple average. For example, the arithmetic mean of 30 inches and 60 inches is just 45 inches. A geometric mean is used for ratios. For example, the mean of 30:1 and 60:1 is 42.4264:1. A harmonic mean is used for rates. For example, the average of 30 miles per hour and 60 miles per hour over a fixed distance is 40 miles per hour.

Figure 1 Sample App under UI Test Automation
Figure 1 Sample App under UI Test Automation(Click the image for a smaller view)
Figure 1 Sample App under UI Test Automation
Figure 1 Sample App under UI Test Automation (Click the image for a larger view)

The UI test automation shown in Figure 1 is a console application that launches the application under test, uses the UI Automation library to obtain references to the application and user controls on the app, simulates a user entering 30 and 60, and simulates clicking on the Calculate button control. The test automation then checks the resulting state of the application by examining the result TextBox control for an expected value, and it then prints a pass or fail result. I captured the screen shot just before the test automation closed the application under test.

In the rest of this column, I will briefly describe the StatCalc Windows application I am testing and will explain how to launch the application under test; how to use the UI Automation library to get references to the application and user controls; and how to simulate user actions and check application state. I'll also describe how you can extend and modify the test system presented here to meet your own needs. I think you'll find the ability to use the new UI Automation library a great addition to your testing tool set.

Inside StatCalc

Let's briefly look at the application under test so you'll understand what the goal of the test automation is. You'll also see why it is useful, but not absolutely necessary, to have access to the source code of the application when writing UI test automation.

The StatCalc application is a simple Windows-based form. I used C# to code the application, but the UI Automation library works with Win32 applications and WPF applications, too. For simplicity, I accepted the Visual Studio® default control names of Form1, label1, textBox1, groupBox1, radioButton1, radioButton2, radioButton3, button1, and textBox2. I added top-level menu items, File and Help, using a MenuStrip control (available starting with the .NET Framework 2.0) rather than the older MainMenu control. The functionality of the StatCalc application is contained in the button1_Click method and is shown in Figure 2.

Figure 2 StatCalc Application Code (Close Figure 2)
private void button1_Click(object sender, EventArgs e) {
  string[] sVals = textBox1.Text.Split(' ');
  int[] iVals = new int[sVals.Length];
  for (int i = 0; i < iVals.Length; ++i)
    iVals[i] = int.Parse(sVals[i]);

  if (radioButton1.Checked) {
    double sum = 0.0;
    foreach (int v in iVals)
      sum += v;
    double result = (double)(sum / iVals.Length);
    textBox2.Text = result.ToString("F4");
  }
  else if (radioButton2.Checked) {
    double product = 1.0;
    foreach (int v in iVals)
    product *= (double)v;
    double result = NthRoot(product, iVals.Length);
    textBox2.Text = result.ToString("F4");
  }
  else if (radioButton3.Checked) {
    double sum = 0.0;
    foreach (int v in iVals)
      sum += (1/ (double)v);
    double result = (double)(iVals.Length / sum);
    textBox2.Text = result.ToString("F4");
  }
}

When a user clicks on the Calculate button, control is transferred to the button1_Click method, which fetches the value in textBox1 as a single string. It then uses the String.Split method to parse each value into a string array. Next, the click handler converts the string array into an int array. The button1_Click logic branches based on which radioButton control has been selected, computes the appropriate mean as type double, and then displays the result in the textbox2 control as a string formatted to four decimal places.

When computing a geometric mean, the application calls a local helper method named NthRoot, which is defined as:

private static double NthRoot(double x, int n) {
  return Math.Exp( Math.Log(x) / (double)n );
}

I am not doing any normal error checking here so I can keep the size of my application code small and easy to understand.

As a general rule of thumb, when writing test automation using the UI Automation library, you do not need access to the source code of the system under test. In most cases, you access controls by their captions (such as Calculate) rather than by their internal Name property (such as button1). However, some controls, such as textbox controls, do not have captions.

To access non-captioned controls with UI automation it is necessary to know their implied index order. This is the order in which controls are loaded onto the main Form control (or parent control) when the Form is initialized. The implied index order is not necessarily the same as the order in which you add controls to the Form object at design time, and it is also not the same as the tab index order.

You can most easily determine the implied index order of non-captioned controls by viewing the application source code. For example, if I examine the Form1.Designer.cs file for the StatCalc application and expand the Windows Form Designer generated code region, I see this code:

this.Controls.Add(this.button1);
this.Controls.Add(this.groupBox1);
this.Controls.Add(this.label1);
this.Controls.Add(this.textBox2);
this.Controls.Add(this.textBox1);
this.Controls.Add(this.menuStrip1);

This means that button1 has implied index 0, groupBox1 has index 1, and so on. Notice that the textBox2 control implied index is less than the textBox1 control (even though when I designed the StatCalc application, I placed textBox1 onto the Form object before I placed textBox2 onto the Form). Also, controls radioButton1, radioButton2, and radioButton3 are child controls of the groupBox1 control rather than Form1. As you'll see shortly, the level of depth in the control tree is important when writing test automation using the UI Automation library.

So, if you have access to the source for the application under test, you can easily determine the implied index order for controls. But what if you do not have access to the source code? One way to determine control implied index order is to use the Spy++ tool that ships with Visual Studio. After launching Spy++ you can press Ctrl+F5 and then drag the GUI Finder Tool onto any control/window to get detailed information about the control/window. Figure 3 shows the result of targeting the textBox1 control with Spy++. In the background you can see that textBox1 has handle 00080820. In the foreground the next control after textBox1 has caption menuStrip1, and the control previous to textBox1 does not have a caption—this is the textBox2 control.

Figure 3 Finding Implied Index Order with Spy++
Figure 3 Finding Implied Index Order with Spy++(Click the image for a smaller view)
Figure 3 Finding Implied Index Order with Spy++
Figure 3 Finding Implied Index Order with Spy++ (Click the image for a larger view)

So even if you do not have source code access for the application under test, you will have the ability to eventually infer the implied index order for all the application's controls by examining each control using Spy++.

Now, manually testing even this tiny StatCalc application through its user interface would be boring, error-prone, time-consuming, and inefficient. You would have to enter some inputs, click the Calculate button control, visually verify the result value, and manually record the pass/fail result into a spreadsheet or database. A much better approach is to use the UI Automation library to simulate a user exercising the application and then determine if the application has responded correctly. By automating tedious test cases, you can free up time for more interesting and useful manual test cases where your experience and intuition play a big role.

The UI Test Automation Code

The overall structure of my test harness is shown in Figure 4. After launching Visual Studio, I create a new Console Application program. I decided to use C#, but you should be able to easily convert my test automation code to Visual Basic or any other .NET-compliant language if you wish. Next I add project references to the UIAutomationClient.dll and UIAutomationTypes.dll libraries. These libraries are part of the .NET Framework 3.0 and are typically located in the %PROGRAMFILES%\Reference Assemblies\Microsoft\Framework\v3.0 directory.

Figure 4 UI Test Automation Code Structure (Close Figure 4)
using System;
using System.Windows.Automation;
using System.Diagnostics;
using System.Threading;

namespace TestScenario {
  class Program {
    static void Main(string[] args) {
      try {
        Console.WriteLine("\nBegin WPF UIAutomation test run\n");
        // launch StatCalc application
        // get refernce to main Form control
        // get references to user controls
        // manipulate application
        // check resulting state and determine pass/fail
        Console.WriteLine("\nEnd test run\n");
      }
      catch(Exception ex) {
        Console.WriteLine("Fatal error: " + ex.Message);
      }
    }
  }
}

The UI Automation library architecture uses a client-server point of view and naming conventions. From a UI test automation standpoint this means that the application under test is called the server and the test harness is considered the client—the test harness client requests UI information from the application under test (the server). The UIAutomationClient.dll library is essentially the test automation library used by UI Automation clients. Additionally, the UIAutomationTypes.dll library contains various type definitions used by both UIAutomationClient.dll and other UI Automation server libraries.

In addition to the UIAutomationClient.dll and UIAutomationTypes.dll libraries, you will also see the UIAutomationClientSideProvider.dll and UIAutomationProvider.dll libraries. The UIAutomationClientSideProvider.dll library contains a set of code for working with controls that were not built with support for automation—this could include legacy controls and custom .NET controls. Because my application is using all standard controls (which were built to support UI automation), I don't need this library. The UIAutomationProvider.dll library is a set of interface definitions that can be used by developers who are creating custom UI controls and who want the controls to be accessible by the UI Automation library.

After I have added my Project References, the System.Windows.Automation namespace is now visible to my test harness program, and I can add a using statement to the namespace. For the sake of convenience, I add using statements that point to the System.Diagnostics namespace (so I can easily use the Process class) and to the System.Threading namespace (so I can easily use the Thread.Sleep method). As usual with any test automation, I wrap my harness with a top-level try/catch block to handle any fatal errors. Test automation is inherently tricky, and Exceptions are the rule rather than the exception so to speak.

My test automation code begins by launching the application under test:

Console.WriteLine("\nBegin WPF UIAutomation test run\n");
Console.WriteLine("Launching StatCalc application");
Process p =
    Process.Start("..\\..\\..\\StatCalc\\bin\\Debug\\StatCalc.exe");

Notice that I capture the process information of the application under test. Next I fetch a reference to the host machine's Desktop window as an AutomationElement:

AutomationElement aeDesktop = AutomationElement.RootElement;

Most objects of interest for test automation using the UI Automation library are of type AutomationElement. Here I am able to get a reference to the top-level visual element (the Desktop) by using the AutomationElement's static RootElement property. When performing UI test automation, it is useful to think of every visual entity (controls, windows, and so on) on the host machine as part of a hierarchical tree structure with the desktop window as the root element. This is a common principle for most kinds of UI automation including automation with the UI Automation library.

Next I need to get a reference to the main Form object, which is essentially the StatCalc application. The following illustrates a simple way to do this:

// naive approach
Thread.Sleep(5000);
AutomationElement aeForm =
    AutomationElement.FromHandle(p.MainWindowHandle);
if (aeForm == null)
    Console.WriteLine(
        "Could not find the StatCalc Form");

I pause for a few seconds in order to give my application under test time to launch, and then I use the FromHandle method to get a reference to the application. Recall that I saved the process information of the application under test when I launched the application.

Although this technique works, there are two minor problems with the approach. First, there is no good way to guess how long to wait before attempting to access the Form object. So you must be conservative and put in a very long delay time, which slows your test automation significantly, especially if you are running hundreds or thousands of test cases. Second, this approach assumes that the test harness launches the application under test (allowing you to save the process information about the application). But, in some testing situations, you want the application to already be running, so you will not have immediate access to the application under test process information (although you can programmatically get this information).

In most situations, a better approach to getting a reference to Form object of the application under test as an AutomationElement is code like that shown in Figure 5. Instead of pausing an arbitrary amount of time before blindly requesting a reference to the application, I put an attempt to get the reference inside a loop with a slight delay (100 milliseconds in this case). I track the number of times my harness enters the waiting loop, and I exit if I get a non-null reference (meaning I found the Form object) or if I exceed a maximum number of attempts (50 in this case).

Figure 5 Getting the Test App AutomationElement (Close Figure 5)
// a better approach
AutomationElement aeForm = null;
int numWaits = 0;
do
{
    Console.WriteLine("Looking for StatCalc . . . ");
    aeForm = aeDesktop.FindFirst(TreeScope.Children,
        new PropertyCondition(
            AutomationElement.NameProperty,
            "StatCalc"));

    ++numWaits;
    Thread.Sleep(100);
}

while (aeForm == null && numWaits < 50);

if (aeForm == null)
    throw new Exception("Failed to find StatCalc");
else
    Console.WriteLine("Found it!");

Additionally, instead of using FromHandle, I use the FindFirst method, which proceeds to search for the first available reference to an AutomationElement that meets the specified condition passed in as a pair of arguments. My first argument, TreeScope.Children, instructs FindFirst to look only at immediate child controls of its context (aeDesktop in this case) rather than look at all descendants of its context.

The second argument to FindFirst is a condition:

new PropertyCondition(AutomationElement.NameProperty, "StatCalc")

You can interpret this code to mean, "The automation element with caption (Name property) equal to StatCalc." Instead of using a traditional string approach to conditions to identify controls, such as Property == StatCalc, the designers of the UI Automation library decided to create a PropertyCondition class. This approach may seem a bit awkward and wordy at first, but it becomes familiar quite quickly.

Basically, the use of a fully object-oriented PropertyCondition class to identify controls, instead of using simple strings, does require more code when you write test automation, but it also has several advantages. A PropertyCondition tends to makes the intent of your test automation code very clear (at least after you've become familiar with the paradigm). Because a PropertyCondition object is more strongly typed than a string, using a PropertyCondition allows an integrated development environment such as Visual Studio to provide design-time IntelliSense® auto-completion help. It also enables the compiler to perform better error checking.

After getting a reference to the main Form object of the application under test, I fetch references to all the user controls. For example, to get a reference to the Calculate button control, I can write this:

Console.WriteLine("Finding all user controls");
AutomationElement aeButton = aeForm.FindFirst(TreeScope.Children,
  new PropertyCondition(AutomationElement.NameProperty, "Calculate"));

The pattern to get a user control is exactly the same as that for getting the Form control. Consistency like this is an advantage of the UI Automation library compared to many other automation libraries. As is the case with most types of coding, with UI automation there are usually several different ways to accomplish a task. For example, instead of the previous code to get a reference to the Calculate button control, I can also write this:

AutomationElement aeButton = aeForm.FindFirst(TreeScope.Children,
  new PropertyCondition(AutomationElement.ControlTypeProperty,
    ControlType.Button));

Instead of using the NameProperty, I can use the ControlTypeProperty with a value of ControlType.Button. You can interpret this to mean, "Fetch the first button control found on the Form." This approach works here only because I have only one button control on my Form object.

The ControlType class is a key Microsoft UI Automation idea. The class is very simple and can be used to identify the type of any control. Getting references to my textbox controls, which do not have captions or names, requires a different technique:

AutomationElementCollection aeAllTextBoxes =
  aeForm.FindAll(TreeScope.Children,
    new PropertyCondition(AutomationElement.ControlTypeProperty,
      ControlType.Edit));

AutomationElement aeTextBox1 = aeAllTextBoxes[1];
AutomationElement aeTextBox2 = aeAllTextBoxes[0];

Here I use the FindAll method to retrieve all textbox controls that are direct children of my Form control. Notice I use ControlType.Edit to specify textbox controls rather than something like ControlType.TextBox, as you might have guessed. I store all my textbox controls in an AutomationElementCollection, which I can then access by index.

Recall from the previous section that the textBox2 control has a lower implied index than the textBox1 control, so textBox2 is at location [0] in the collection and textBox1 is at location [1] in the collection. There are several other approaches you can use to get references to non-captioned controls, but the FindAll method is often a good choice.

Here is how I obtain a reference to the radioButton1 control:

AutomationElement aeRadioButton1 =
  aeForm.FindFirst(TreeScope.Descendants,
    new PropertyCondition(AutomationElement.NameProperty,
      "Arithmetic Mean"));

Notice that I use TreeScope.Descendants rather than TreeScope.Children because my current context is the Form object. The radioButton controls are children of the groupBox1 control, which is a child of the Form1 control, so the radioButton controls are descendants of Form1, but not children. Alternatively, I could first obtain a reference to the groupBox1 control (as a child of Form1) and then a reference to radioButton1 (as a child of the groupBox1 control).

After obtaining references to the radioButton2 (Geometric Mean) and radioButton3 (Harmonic Mean) in exactly the same way as with radioButton1, I can begin to exercise my application under test. Here is how I simulate a user typing some input into the textBox1 control:

Console.WriteLine("\nSetting input to '30 60'");
ValuePattern vpTextBox1 =
  (ValuePattern)aeTextBox1.GetCurrentPattern(ValuePattern.Pattern);
vpTextBox1.SetValue("30 60");

The SetValue method probably does not come as a surprise, but notice that I do not access SetVaue directly through the aeTextBox1 object. Instead I use an intermediary ValuePattern object. The concept of AutomationPattern objects such as ValuePattern is probably the biggest conceptual stumbling block for engineers new to the UI Automation library. You can think of pattern objects as an abstract way to expose a control's functionality that is independent from the control's type or appearance. Put another way, you can use specific AutomationPattern instances such as ValuePattern to enable specific control functionality.

I simplify things further by thinking that a control's ControlType exposes what kind of control the control is, and a control's Pattern exposes what the control can do. It's important to note here that a control can support multiple patterns. Here is how I use an AutomationPattern to select the radioButton2 control:

Console.WriteLine("Selecting 'Geometric Mean' ");
SelectionItemPattern ipSelectRadioButton2 =
  (SelectionItemPattern)aeRadioButton2.GetCurrentPattern(
    SelectionItemPattern.Pattern );
ipSelectRadioButton2.Select();

This time I use the SelectionItemPattern to enable a selection. The name of the GetCurrentPattern method often confuses UI Automation library beginners. From a test automation point of view, the method is setting, not getting, a specified AutomationPattern. But from a client-server perspective, the automation client code is fetching a particular property from the application under test server code. The code I use to simulate a click on the Calculate button control should help clarify:

  Console.WriteLine("Clicking on Calculate button");
  InvokePattern ipClickButton1 =
    (InvokePattern)aeButton.GetCurrentPattern(
    InvokePattern.Pattern);
  ipClickButton1.Invoke();
  Thread.Sleep(1500);

Here, I use the InvokePattern to enable a button click and then execute the click using the Invoke method. Notice I pause 1.5 seconds to give my application time to respond. I could also go into a delay loop, checking periodically to see if the result textBox2 field is empty or not, using the technique I will demonstrate shortly. There are 18 AutomationPattern types, and I have listed them in Figure 6. Understanding AutomationPattern classes is the key to knowing how to exercise your application under test using the Microsoft UI Automation library.

Figure 6 Patterns (Close Figure 6)

DockPattern
ExpandCollapsePattern
GridPattern
GridItemPattern
InvokePattern
MultipleViewPattern
RangeValuePattern
ScrollPattern
ScrollItemPattern
SelectionPattern
SelectionItemPattern
TablePattern
TableItemPattern
TextPattern
TogglePattern
TransformPattern
ValuePattern
WindowPattern

At this point in my test automation code, I have launched the application under test, entered "30 60" into the input (textBox1) control, selected the Geometric Mean (radioButton2) control, and clicked on the Calculate (button1) control. Now I examine the textBox2 control to see if I have an expected value:

Console.WriteLine("\nChecking textBox2 for '42.4264'");
TextPattern tpTextBox2 =
  (TextPattern)aeTextBox2.GetCurrentPattern(TextPattern.Pattern);
string result = tpTextBox2.DocumentRange.GetText(-1);

Again I use an AutomationPattern, this time TextPattern, to prepare a call to the GetText method. Notice that I call GetText indirectly through a DocumentRange property, which returns a text range that encloses the main text of a "document"—which is a textbox in this case. The argument to GetText is meant to indicate that I am not placing a maximum limit on the length of the return string.

An alternative way to read the contents of the textBox2 control is to use the GetCurrentPropertyValue method, like this:

string result =
  (string)aeTextBox2.GetCurrentPropertyValue(
    ValuePattern.ValueProperty);

The GetCurrentPropertyValue method returns the current property for an AutomationElement control if a property has been set, or returns the value of the default property for the control if the property of the context control has not been set. In this case, I haven't set a property for the textBox2 control, so GetCurrentPropertyValue with an argument of ValueProperty returns the value of the default property for the tetxtBox2 control, which is the Text property. Using GetCurrentProperty saves time when coding but can be risky because current property of a control can change at run time.

With the result of the calculation in hand, I check against an expected value to determine my test scenario pass/fail result:

if (result == "42.4264") {
  Console.WriteLine("Found it");
  Console.WriteLine("\nTest scenario: Pass");
}
else {
  Console.WriteLine("Did not find it");
  Console.WriteLine("\nTest scenario: *FAIL*");
}

And now I can close the application under test:

Console.WriteLine("\nClosing application in 5 seconds");
Thread.Sleep(5000);
WindowPattern wpCloseForm =
  (WindowPattern)aeForm.GetCurrentPattern(WindowPattern.Pattern);
wpCloseForm.Close();

I use the Close method of the WindowPattern class, which is a thematic approach for the UI Automation library.

As an alternative to close the application under test, you can manipulate the File | Exit menu item starting with:

AutomationElement aeMenu =
  aeForm.FindFirst(TreeScope.Children,
    new PropertyCondition(AutomationElement.ControlTypeProperty,
      ControlType.Menu));

However the ControlType.Menu type is the.NET Framework 1.1 MainMenu control and does not work with the newer MenuStrip control. Because my application under test uses MenuStrip, I cannot use the ControlType.Menu type. This is a rare exception; the UI Automation library supports the vast majority of user controls and also provides automation support for custom controls through functionality contained the UIAutomationProvider.dll library. At this point my test automation is complete and I can record my test case result and launch another test scenario.

Extending the Test Harness

The code presented in this month's column gives you a solid foundation for getting started with creating your own test automation using the UI Automation library. As you have seen, the library is very extensive, and its pure object-oriented programming model is likely a bit different than other libraries you have used. However, I'm sure you will quickly learn how to use it to write powerful UI test automation. Three key concepts worth focusing on are the use of the ControlType class to identify the type of a UI control, the AutomationPattern class to identify the functionality of a UI control, and the PropertyCondition class to specify a particular UI control.

There are many, many ways you can extend the ideas in this introductory test automation harness that I've presented. One obvious extension is to parameterize the harness. Instead of hardcoding inputs and expected values, you can pass this information to your test harness on the command line. Better yet, create a test case data store as a text file, database table, or XML file and iterate through each test case, parsing out test case data and passing the data into the test harness. I discuss such design patterns in detail in my August 2005 column at msdn.microsoft.com/msdnmag/issues/05/08/TestRun.

The code featured in this column does not perform result logging—and in most cases you will want to record test case results to a data store instead of merely printing a pass/fail message to the command shell.

I strongly suspect that the UI Automation library will quickly become the most widely used framework for UI test automation. More than anything else, there is now a single, standard, comprehensive, consistent, and well-tested UI automation library that is freely available to software engineers. This is already giving rise to a community with public forums, blogs, sample code, and so on.

This is not to say I think that the UI Automation library will completely replace alternatives such as commercial UI test automation frameworks and custom UI automation frameworks. However, the UI Automation library has significant advantages over these alternatives. Compared to commercial UI test automation frameworks, the UI Automation library is less expensive, and, because it ships with the .NET Framework 3.0, it is a de facto standard.

Compared to custom UI test automation libraries written by one or two software engineers, the UI Automation library contains much more functionality, and you only have to learn one API set instead of a new API for every custom automation library. However, commercial UI test automation frameworks have an advantage in that they are often tightly integrated with tools such as test case managers and bug-reporting software tools. And lightweight custom UI test automation libraries have an advantage in that they are generally easier to use for quick-and-dirty test automation, and they can be adapted for special testing situations. Yet, as the presence of the .NET Framework 3.0 becomes more common through the adoption of Windows Vista, you can expect to see a rapid increase in the use of the UI Automation library.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft's Redmond, Washington, campus. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006) and can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Foundations: What's New for WCF in Visual Studio 2008

Foundations
What's New for WCF in Visual Studio 2008
Juval Lowy

Code download available at: Foundations2008_02.exe (329KB)
Contents
.NET Framework Cross-Targeting
WCF-Provided Host
WCF-Provided Test Client
WCF Service Libraries
Adding Service References

V
isual Studio® 2008 and the underlying Microsoft® .NET Framework 3.5 offer new tools and support for Windows® Communication Foundation (WCF). They don't change the basic capabilities of WCF 1.0 (released with the .NET Framework 3.0); rather, they extend and complete it.

Visual Studio 2008 automates manual WCF tasks, which includes updating proxy references, and eliminates repetitive tasks such as creating simple host projects. Visual Studio also addresses some tough problems such as cross-targeting and data contract type sharing. In this column I'll walk through the new features, point out their advantages, and explain any pitfalls and workarounds. Although I'll use C# project settings here, everything applies to Visual Basic® as well unless I state otherwise.

.NET Framework Cross-Targeting

Previous Visual Studio releases always targeted the version of the .NET Framework they shipped with. For example, Visual Studio 2005 can only generate assemblies that target the .NET Framework 2.0. This practice did not reflect the reality most developers face. Typically, developers need to maintain old versions of their applications written for previous versions of .NET while at the same time using the new version of Visual Studio on the application's next version.

Moreover, this practice meant that developers who were maintaining applications written for previous .NET Framework versions could not benefit from productivity enhancements, such as the code refactoring support introduced in Visual Studio 2005.

The problem was that there was no cross-targeting of .NET Framework versions. You either had to have multiple versions of Visual Studio installed or compensate with separate testing and deployment builds. Visual Studio 2008 tries to address this by providing adequate (albeit imperfect) support for targeting multiple versions of the .NET Framework. Since the .NET Framework 3.0 and the .NET Framework 3.5 actually use the same CLR version as the .NET Framework 2.0, and the only differences are in new referenced assemblies, Visual Studio can still target the same runtime yet provide cross-targeting for the .NET Framework versions 2.0, 3.0, and 3.5 (where the .NET Framework version number corresponds to release numbers, not runtime versions—that is still the CLR 2.0).

In Visual Studio 2008, the Application pane of the project Properties contains a new combobox called Target Framework that allows you to target the .NET Framework versions 2.0, 3.0, and 3.5 (see Figure 1).

Figure 1 Target Framework Property in Visual Studio 2008
Figure 1 Target Framework Property in Visual Studio 2008(Click the image for a smaller view)
Figure 1 Target Framework Property in Visual Studio 2008
Figure 1 Target Framework Property in Visual Studio 2008 (Click the image for a larger view)

The Target Framework value is only in effect at development time; it has no effect at run time (your assembly still targets the .NET 2.0 CLR). The value you choose represents the oldest version of the .NET Framework your assembly can be built against. New projects are configured by default to target .NET Framework 3.5. It gets a bit more complicated when adding references; if you downgrade the Target Framework version while referencing higher-version assemblies, Visual Studio 2008 will prompt you, fault the reference, and fail the build. Visual Studio 2008 will not allow you to add a reference to a .NET Framework assembly that requires a higher version of the Framework than your project. If you add a reference to another project in the same solution that is of a higher version, Visual Studio 2008 will alert you as to the possible conflict. If you add a reference by browsing to an assembly, Visual Studio 2008 will not intervene to stop you.

With respect to languages and cross-targeting, note that in C# (but not Visual Basic), you can restrict use of features such as anonymous types and extension methods in a .NET Framework 2.0 or 3.0 project by restricting the compiler version. You can do this by going to the Build pane, clicking the Advanced button, and selecting ISO-2 (C# 2.0) for the language version (instead of the default, which has not been standardized as of yet).

When opening a Visual Studio 2005 WCF project in Visual Studio 2008, the upgrade process will keep the framework version at 2.0. While this will actually work (remember, the underlying runtime version is unchanged), I recommend manually setting it to version 3.0 or 3.5, as required.

The Target Framework version matters the most when taking advantage of new project templates. WCF workflow and syndication projects must be built targeting the .NET Framework 3.5; the Service Library project requires targeting the .NET Framework 3.0 or 3.5. The Add Service Reference feature (described later in this column) is only available when selecting Framework version 3.0 or 3.5 for the project.

WCF-Provided Host

Visual Studio 2008 ships with a ready-made, general-purpose service host called WcfSvcHost.exe. It's found under C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE. For ease of use, I recommend adding that location to the system's Path variable. WcfSvcHost is a simple command-line utility, accepting two parameters: the file path to a .NET assembly containing the service class or classes, and a file path to the host .config file. For example:

WcfSvcHost.exe /service:MyService.dll  /config:App.config

The specified service assembly can be a class library assembly (DLL) or an application assembly (EXE). WcfSvcHost will launch a new process that will automatically host all the service classes listed in the services section of the specified .config file. Note that these service classes, along with their service contracts and data contracts, need not be public types—they can be internal. In addition, auto-hosted services need not provide any metadata, but they can publish metadata if they choose to.

WcfSvcHost is a Windows Forms application that resides as a desktop tray icon. To close the host, simply select Exit from the tray icon context menu. Terminating the host this way is an ungraceful exit—WcfSvcHost will abort all calls currently in progress, and clients are likely to receive an exception. If you click on the WcfSvcHost tray icon, it will bring up a dialog listing all the hosted services (see Figure 2).

Figure 2 WcfSvcHost Services List
Figure 2 WcfSvcHost Services List (Click the image for a smaller view)
Figure 2 WcfSvcHost Services List
Figure 2 WcfSvcHost Services List (Click the image for a larger view)

The dialog also shows the status of the service and its metadata address, which you can copy to the clipboard, perhaps for use later when adding a reference to the service. Closing the WcfSvcHost UI merely collapses it back to the tray.

WcfSvcHost is designed to eliminate the need during development for a separate host assembly to accompany your service library. Developing such host projects is a repetitive task; these hosts typically contain the same lines of code over and over again, and they tend to bloat the solution when you have multiple service libraries. For development and testing purposes, you can integrate WcfSvcHost directly into your Visual Studio 2008 service library projects. In the Debug pane of the project properties, specify WcfSvcHost.exe as the external program to start, and then specify your class library name and its .config file (the one auto-generated and auto-copied to the bin folder) as arguments.

With that done, when you run the class library (something you could not do before), it will be automatically hosted by WcfSvcHost with the debugger attached to that process. When you stop debugging, Visual Studio 2008 will abort the host ungracefully.

You can even use WcfSvcHost in your .NET Framework 3.0 applications and with Visual Studio 2005 projects, since all WcfSvcHost requires is the .NET Framework 3.0. Simply copy WcfSvcHost from a machine where Visual Studio 2008 is installed. To make things easier, I recommend adding WcfSvcHost to the Global Assembly Cache (GAC) on your .NET Framework 3.0 machine.

The last feature of WcfSvcHost is its ability to automatically launch a client application and even provide the client with optional parameters specific for that application:

WcfSvcHost.exe /service:MyService.dll  /config:App.config
               /client:MyClient.exe    /clientArgs:123,ABC

This is useful in automated testing and even simple deployment scenarios to launch both the host and the client.

The biggest drawback of WcfSvcHost is that it is only suitable for simple scenarios where you do not require programmatic access to the host instance before opening it or programmatic access to its event model once opened. Unlike hosting with IIS or the Windows Activation Service (WAS), there is no equivalent service host factory support. Consequently, there is no ability to dynamically add base addresses, configure endpoints, throttle calls, configure custom behaviors at the host level, and so on. My experience with WCF is that in all but the simplest cases, eventually you will need programmatic access to the host instance, so I do not view WcfSvcHost as a full-fledged production-worthy host, as I do the WAS or a dedicated self host.

WCF-Provided Test Client

In addition to the service host, Visual Studio 2008 ships with a simple, general-purpose test client for rudimentary testing that you can use to invoke operations on most services. The test client, WcfTestClient.exe, is found after normal installation at C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE. You must provide WcfTestClient with a single command-line argument containing the metadata address of the service to test:

WcfTestClient.exe http://localhost:9000/

You can specify any metadata address, such as HTTP-GET, or metadata endpoint over HTTP, TCP, or IPC (named pipes). You can also specify multiple metadata addresses:

WcfTestClient.exe http://localhost:8000/ net.tcp://localhost:9000/MEX

WcfTestClient is a Windows Forms 3.5 application (see Figure 3). In this figure, the tree control on the left contains the tested services and their endpoints. You can drill into an endpoint's contract and select an operation. The information specific to that invocation will be displayed in a tab on the right pane. An example is the simple contract and implementation in Figure 4.

Figure 4 A Sample Service (Close Figure 4)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   string MyMethod(int someNumber,string someText);
}
class MyService : IMyContract
{
   public string MyMethod(int someNumber,string someText)
   {
      return "Hello";
   }
}
Figure 3 Using WcfTestClient
Figure 3 Using WcfTestClient(Click the image for a smaller view)
Figure 3 Using WcfTestClient
Figure 3 Using WcfTestClient (Click the image for a larger view)

The method tab will let you provide an integer and a string as operation parameters in the Request section, as in Figure 3. When you click the Invoke button, it will dispatch the call to the service and display the returned value or out-parameters in the Response. If the operation is one-way, WcfTestClient will notify you in a message box upon the operation's successful dispatch. In case of an exception, WcfTestClient will display the exception information in a message box and let you issue additional calls.

WcfTestClient does not maintain a transport-level session (or any other session) with the tested service. All calls are made on new proxy instances. In addition, all calls are made asynchronously so that the UI is kept responsive. However, while the calls are asynchronous, WcfTestClient will only let you dispatch one call at a time.

WcfTestClient functions by silently creating an assembly from a proxy file, complete with a .config file, and then loading it from a temporary location. If you click on the Config File item in the tree, you can actually grab that .config file (the same one generated when adding a service reference), and you can display it in its own tab.

Unlike the old Visual Studio ASMX Web services test page, WcfTestClient allows you to invoke operations with enumerations, composite parameters such as classes or structures (each being a composite of other classes or structures), and even collections and arrays of parameters. Simply expand the items in the Request section, set their values from dropdown lists (such as enum values), and invoke the call. If the operation accepts a collection or an array, you will also need to set the length. For example, for the following operation, Figure 5 shows the resulting request and response:

Figure 5 Specifying an Array Length and Values
Figure 5 Specifying an Array Length and Values(Click the image for a smaller view)
Figure 5 Specifying an Array Length and Values
Figure 5 Specifying an Array Length and Values (Click the image for a larger view)

[OperationContract]
bool ProcessNumbers(int[] numbers]

Much the same way, the Response pane will contain any returned composite value or out parameters. This brings us to a drawback of WcfTestClient, which is that in order to specify different services to test, you have to shut it down, change the command-line arguments, and relaunch it. It would have been better had it been possible to provide the service addresses in the GUI as well.

You can integrate WcfTestClient directly into your Visual Studio 2008 solution. First, add a class library project to the solution and delete all references, folders, and source files, as you have no need for them. Next, set WcfTestClient.exe as the external start program and provide the metadata address (or addresses) of the tested service (or services), such as the .svc address of an IIS or WAS hosted project, or, for that matter, any other metadata address of a host project, inside or outside your solution.

Note that you cannot use WcfTestClient on machines with only .NET Framework 3.0, since it makes use of an internal .NET Framework 3.5 tree grid control (the one used to represent composite parameters).

Of course, you can combine both WcfTestClient and WcfSvcHost in a single step to automatically host a service in a service library and test it:

WcfSvcHost.exe /service:MyService.dll    /config:App.config
               /client:WcfTestClient.exe
               /clientArgs:http://localhost:9000/

However, with WcfSvcHost, specifying the metadata arguments is optional. By default WcfSvcHost will pipe in to the specified client application the metadata addresses it found in the service .config file. You should specify the metadata address explicitly only if the service or services do not provide their own metadata or if you would like the test client to use different addresses. Also, if the service .config file contains multiple metadata endpoints for a given service, they will be provided in this order of precedence: HTTP, TCP, IPC, HTTP-GET. You can incorporate these steps in Visual Studio 2008 for a seamless hosting and testing experience. Specify WcfSvcHost.exe as the start-up program along with the .config file and WcfTestClient.exe as the client.

WCF Service Libraries

As a function of the Target Framework, Visual Studio 2008 offers several new WCF project templates. The New Project dialog offers a combobox that lets you specify the Target Framework version (2.0, 3.0, or 3.5), as shown in Figure 6.

Figure 6 WCF Project Templates
Figure 6 WCF Project Templates(Click the image for a smaller view)
Figure 6 WCF Project Templates
Figure 6 WCF Project Templates (Click the image for a larger view)

If you select Framework 2.0, then no new templates are available. With Framework 3.0 there is a new project template called the WCF Service Library. This project type is merely the pre-built usage of WcfSvcHost and WcfTestClient, and it is very similar to the technique that I mentioned previously (combining the two). Note that with the WCF Service Library template there is no need to specify WcfSvcHost.exe as the start-up program or the .config file because the project file contains a new ProjectTypeGuids element for a WCF service library.

An unfortunate side effect of the template is that stopping the debugger does not terminate the test client; over time, your desktop becomes littered with orphaned clients. To fix this, simply revert to the manual explicit steps described earlier.

The WCF Service Library also provides a simple template for a service contract, its implementation, and the matching .config file.

The Syndication Service Library allows you to implement an RSS feed over a WCF endpoint, and it starts you off with a simple service contract that returns a feed, its implementation, and a matching .config file. You can host and expose your feed like any other service. The syndicated endpoints make use of the new WebHttpBinding binding. This new binding is designed to receive Web requests and cannot be used for normal service invocations.

The Sequential Workflow Service Library template allows you to implement an endpoint's contract operations as workflow activities or, for that matter, to expose a workflow as a service. The project will contain a single sequential activity that implements a simple contract and a matching .config file. The client still interacts with what looks like a traditional endpoint, and yet the implementation is purely workflow-driven.

The State Machine Workflow Service Library template uses a state machine instead of a sequential workflow to implement its operations (trigger state transitions). The workflow project templates make use of WcfSvcHost and WcfTestClient, just as the plain WCF Service Library does. The workflow templates also make use of the new context bindings to manage passing the workflow instance ID to support durable workflows. I will address these binding in detail in a future column.

Adding Service References

The Visual Studio 2005 extensions for .NET Framework 3.0 provided a rudimentary ability to add a reference to a WCF service, without many of the advanced features of SvcUtil. Visual Studio 2008 features a new service reference dialog, as shown in Figure 7.

Figure 7 Add Service Reference Dialog
Figure 7 Add Service Reference Dialog(Click the image for a smaller view)
Figure 7 Add Service Reference Dialog
Figure 7 Add Service Reference Dialog (Click the image for a larger view)

You can bring up the new dialog in any project by right-clicking anywhere inside the project in the Solution Explorer and selecting Add Service Reference from the context menu. Note that the project must be configured to target .NET Framework 3.0 and above to enable this option.

In the Add Service Reference dialog, you first need to specify the service metadata address (not the service URL, as the dialog states) and click Go to view the available service endpoints (not Services, as labeled). You must specify a namespace (such as MyService) to contain the generated proxy and click OK to generate the proxy and update the .config file. Note that, in most cases, Visual Studio 2008 is not smart enough to infer the cleanest binding values, and it will therefore butcher the .config file by declaring all the default values for the bindings. This issue will be addressed in a future release of Visual Studio. If you care about maintaining the .config file, before adding the reference, open the .config file, add the reference, then perform a single Undo (Ctrl+Z), and manually add the .config file entries in the client section.

The Discover button lets you discover WCF services in your own solution as long as they are either hosted in a Web site project or in one of the new WCF service libraries. In the case of a Web site project, Visual Studio 2008 will either retrieve the metadata from IIS or will launch the ASP.NET file-system-based development server. In the case of a WCF service library, WCF will auto-launch its host (WcfSvcHost) to get the metadata.

The Advanced button brings up the settings dialog that lets you tweak the proxy generation almost as if you were using SvcUtil (see Figure 8). The more intuitive options let you configure the visibility of the generated proxy and contracts (public or internal); you can generate message contracts for your data types for advanced interoperability scenarios in which you have to comply with an existing, typically custom, message format, and you can click the Add Web Reference button to convert the reference to an old ASMX Web service reference.

Figure 8 Service Reference Advanced Options
Figure 8 Service Reference Advanced Options(Click the image for a smaller view)
Figure 8 Service Reference Advanced Options
Figure 8 Service Reference Advanced Options (Click the image for a larger view)

The Generate asynchronous operations checkbox adds for each operation in the imported contract a matching pair of Begin<operation> and End<operation> elements that allows the client to issue the call asynchronously on a worker thread and later sync up with the completion of the operation by either providing a completion callback method or by blocking for completion. For example, given this contract definition

[ServiceContract]
interface ICalculator
{
   [OperationContract]
   int Add(int number1,int number2);
}

the imported contract will look like Figure 9.

Figure 9 Imported Async Contract (Close Figure 9)
[ServiceContract]
interface ICalculator
{
   [OperationContract]
   int Add(int number1,int number2);

   [OperationContract(AsyncPattern = true)]
   IAsyncResult BeginAdd(int number1,int number2,
                         AsyncCallback callback,object asyncState);
   int EndAdd(IAsyncResult result);

   //Rest of the methods
}

The matching client code for asynchronous invocation will look like the following:

CalculatorClient proxy = new CalculatorClient();
int sum;
AsyncCallback completion = (result)=>
                           {
                              sum = proxy.EndAdd(result);
                              Debug.Assert(sum == 5);
                              proxy.Close();
                           };
proxy.BeginAdd(2,3,completion,null);

While you could use these methods as-is, the completion callback provided to Begin<operation> is called on a thread from the thread pool. This presents a serious problem if the callback is used to access some resources that have affinity to a particular thread or threads. The classic example is a Windows Forms (or WPF) application that dispatches a lengthy service call asynchronously (to avoid blocking the UI) and then wishes to update the UI with the result of the invocation. Using the raw Begin<operation> is disallowed since only the UI thread is allowed to update it. To better handle this situation, the ClientBase<T> base class has been extended with a protected InvokeAsync method that picks up the synchronization context of the client and will use it to invoke the completion callback, as shown in Figure 10.

Figure 10 Async Callback Management in ClientBase<T> (Close Figure 10)
public abstract class ClientBase<T> : ...
{
   protected delegate IAsyncResult BeginOperationDelegate(
             object[] inValues,AsyncCallback asyncCallback,object state);

   protected delegate object[] EndOperationDelegate(IAsyncResult result);

   //Picks up sync context used for completion callback
   protected void InvokeAsync(BeginOperationDelegate beginOpDelegate,
                              object[] inValues,
                              EndOperationDelegate endOpDelegate,
                              SendOrPostCallback opCompletedCallback,
                              object userState)
   {}
   //More members
}

ClientBase<T> also provides an event arguments helper class and two dedicated delegates used to begin and end the asynchronous call. The generated proxy class that derives from ClientBase<T> makes use of the base functionality. The proxy will have a public event called <operation>Completed that uses a strongly typed event argument class specific to the results of the asynchronous method, and two methods called <operation>Async used to dispatch the call asynchronously:

partial class AddCompletedEventArgs : AsyncCompletedEventArgs
{
   public int Result
   {get;}
}

class CalculatorClient : ClientBase<ICalculator>,ICalculator
{
   public event EventHandler<AddCompletedEventArgs> AddCompleted;

   public void AddAsync(int number1,int number2,object userState);
   public void AddAsync(int number1,int number2);

   //Rest of the proxy
}

The client can also subscribe an event handler to the <operation>Completed event, to have that handler called upon completion. The big difference using <operation>Async as opposed to Begin<operation> is that the <operation>Async methods will pick up the synchronization context of the client and will fire the <operation>Completed event on that synchronization context, as shown in Figure 11.

Figure 11 UI-Friendly Asynchronous Call Invocation (Close Figure 11)
partial class CalculatorForm : Form
{
   CalculatorClient m_Proxy;

   public MyClient()
   {
      InitializeComponent();

      m_Proxy = new CalculatorClient();
      m_Proxy.AddCompleted += OnAddCompleted;
   }
   void CallAsync(object sender,EventArgs args)
   {
      m_Proxy.AddAsync(2,3); //Sync context picked up here
   }
   //Called on the UI thread
   void OnAddCompleted(object sender,AddCompletedEventArgs args)
   {
      Text = "Sum = " + args.Result;
   }
}

The Collection type combobox lets you specify how to represent to the client certain kinds of collections and arrays found in the service metadata. For example, if the service operation returns one of the IEnumerable<T>, IList<T>, or ICollection<T> collections, then by default the proxy will present it as an array. For example, the following service-side operation

[OperationContract]
IEnumerable<int> GetNumbers();

will be expressed on the proxy as:

[OperationContract]
int[] GetNumbers();

However, you can request that Visual Studio 2008 use another collection such as BindingList for data binding, a List<T>, Collection, LinkedList<T>, and so on. If a conversion is possible, the proxy will use the requested collection type instead of an array, like so:

[OperationContract]
List<int> GetNumbers();

A similar feature is available for dictionaries. Normally, if the service operation returns a serializable dictionary such as this

[Serializable]
class MyDictionary<K,T> : IDictionary<K,T>
{...}

[OperationContract]
MyDictionary<int,string> GetDictionary();

then the proxy class will express that dictionary as a Dictionary<T,K>, which is the default value of the following Dictionary collection type combobox:

[OperationContract]
Dictionary<int,string> GetDictionary();

However, you can request other dictionary types, such as the SortedDictionary<T,K>, HashTable, or the ListDictionary, and the proxy will use that dictionary instead if possible:

[OperationContract]
SortedDictionary<int,string> GetDictionary();

By far, the most important feature of the new service reference feature is its ability to share data contract types across assemblies. With Visual Studio 2005, if a client added a service reference to two independent services that supported the same data contract, the client would get two distinct yet identical types representing the same data contract. With Visual Studio 2008, by default, if any of the assemblies referenced by the client already has a data contract type matching a data contract type exposed in the metadata of the referenced service, Visual Studio 2008 will not import that type again. It is worth emphasizing again that the existing data contract reference must be in another referenced assembly, not in the client project itself. This limitation may be addressed in a future release of Visual Studio. For now, the workaround and best practice is obvious: factor all of your shared data contracts to a designated class library and have all clients reference that assembly.

The advanced settings dialog of the service reference lets you configure data contract sharing. The "Reuse types in the referenced assemblies" checkbox is checked by default, but you can turn this feature off. Despite its name, it will only share data contracts, not service contracts. Using the radio buttons underneath it (see Figure 8), you can also instruct Visual Studio 2008 to reuse data contracts across all referenced assemblies, or restrict the sharing to specific assemblies by placing a check mark next to them in the list.

Once a reference is added, your project will have a new folder called Service References, inside of which is a service reference item for each referenced service (see Figure 12).

Figure 12 Service References Folder
Figure 12 Service References Folder

You can always right-click on the reference and select Update Service Reference to regenerate the proxy and update the client's .config file. This is possible because the service reference item also contains a file that records the original metadata address used.

You can also select Configure Service Reference to bring up a dialog similar to the advanced settings dialog used when adding the reference. The configure service reference dialog lets you change the service metadata address as well as the rest of the advanced proxy settings.

Send your questions and comments to mmnet30@microsoft.com.

Juval Lowy is a software architect with IDesign providing WCF training and architecture consulting. He is also the Microsoft Regional Director for the Silicon Valley. Juval's recent book is Programming WCF Services (O'Reilly, 2007), and he can be contacted at www.idesign.net.


.NET Matters: Stream Pipeline

.NET Matters
Stream Pipeline
Stephen Toub

Q In my application, I'm encrypting and compressing quite a bit of data. As these are computationally intensive operations, I was expecting to see 100 percent CPU utilization in Task Manager, but I noticed that on my dual-core machine, it's topping out at around 50 percent. I'm assuming this is because only one core is being used, which is a shame given that the process takes a non-trivial amount of time to run. Is there any way I can get this encryption and compression process to use both processors? I'm using CryptoStream and GZipStream from the Microsoft® .NET Framework.

A First, just to make sure, you're doing the compression operation before the encryption operation, correct? If not, change it if you can. Good encryption will generate relatively uncompressible data. If you switch the order of the operations so that you first compress and then encrypt, not only should you end up with a smaller file, but the encryption will most likely take less time as it'll be operating on less data. As an example of this, I downloaded the text to War and Peace from the Gutenberg Project (www.gutenberg.org) and ran it through the two orderings. Encrypting (with RijndaelManaged and the default key size) and then compressing resulted in a data stream 250 percent larger than the one generated by compressing and then encrypting, and it took 50 percent longer to execute.

Now, to answer your actual question: yes, there are several approaches you could take here. The first would be to parallelize the actual compression and encryption operations. You probably don't want to (and shouldn't) re-implement the functionality in GZipStream and CryptoStream, so, until the .NET Framework team parallelizes them for you, you'll want an alternate solution.

If you don't care about the actual output format (for example, you need compression, but you don't care that it actually adheres to the gzip standard), you could chunk your input and process each chunk in parallel. For example, on your dual-core machine, you could divide in half the input byte array being passed to your GZipStream currently and, instead, process one half with one GZipStream and the other half with another GZipStream. You can then save these out to your output file one after the other.

Note, though, that you'll probably need to include some header information around your output so that your decompression process will adequately be able to determine where one GZipStream ends and the next begins. GZipStream currently buffers data as it reads from the input stream, so it may end up consuming more data from the input stream than it actually needs, which means you'd need to reset the position in the input stream after the first GZipStream completed its decompression.

One of the nice things about the chunking approach is that you can get the process to scale relatively well to more than two cores, as you can simply create as many chunks as are necessary to saturate your processors. But with only two cores, and with two operations (compression and encryption), a more practical solution for you at this point would probably be to create a parallel stream pipeline. You can do one operation on one processor at the same time that the other operation is working on the other processor. Now, obviously they can't be working on the same data at the same time, as you'd end up with two outputs (one compressed and one encrypted) that would be relatively useless to you given the problem you're trying to solve (one output both compressed and encrypted). Instead, however, you can mimic what you're probably doing today, where using the decorator pattern you pass the output from one stream as the input to the next (shown in Figure 1):

Figure 1 Passing Data through Streams
Figure 1 Passing Data through Streams

using (CryptoStream encrypt = new CryptoStream(
        output, transform, CryptoStreamMode.Write))
using (GZipStream compress = new GZipStream(
        encrypt, CompressionMode.Compress, true))
    CopyStream(input, compress);
...
static void CopyStream(Stream input, Stream output){
    byte[] buffer = new byte[0x1000];
    int read;
    while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
        output.Write(buffer, 0, read);
}

Here, the CopyStream method is copying from the input stream into the compression stream. When the compression stream has compressed a buffer of data, it in turn writes that data out as the input into the encryption stream. And similarly, when the encryption stream has completed a buffer of data, it in turn writes that out to the output stream. This is known as a pipeline.

Parallelizing a pipeline is a very natural concept that manifests frequently in the "real world." Consider a group of people sending out invitation letters. One person is in charge of folding the letters to fit in an envelope, one person is in charge of putting the folded letters into an envelope, and one person is in charge of sealing and stamping the envelope. For the first letter, the only person doing anything is the person folding the letter, and the rest of the people are idle. But then the first person hands off the folded letter to the envelope stuffer. At that point, the first person is busy folding the next letter, and the second person is busy stuffing an envelope, but the third person is still idle. When the first and second people complete their tasks, the stuffed envelope is handed to the third person, who seals and stamps it, the folded letter is handed to the second person, who stuffs it, and the first person begins folding a new letter. From this point on, until there are only two more letters left, all three people will be fully utilized (fully utilized, that is, except for the time it takes to hand off the letters and envelopes between people). So, even though each person is only doing one task at a time, assuming there are enough letters to be sent, the majority of the time we could theoretically be getting three times as much work done by having three people involved in the task.

We'll follow the exact same process for our stream pipelining task. Instead of passing around folded letters or envelopes, of course, we'll be passing around buffers of data, since that is the mechanism streams use to communicate. However, unlike with our sequential version, where one stream wrote directly to another stream, direct synchronous communication is no longer possible due to each of those streams operating in parallel on separate threads. We need another mechanism to re-enable that communication. For that task, I've written a BlockingStream, shown in Figure 2.

Figure 2 BlockingStream Implementation (Close Figure 2)
public class BlockingStream : Stream
{
    private object _lockForRead;
    private object _lockForAll;
    private Queue<byte[]> _chunks;
    private byte[] _currentChunk;
    private int _currentChunkPosition;
    private ManualResetEvent _doneWriting;
    private ManualResetEvent _dataAvailable;
    private WaitHandle[] _events;
    private int _doneWritingHandleIndex;
    private volatile bool _illegalToWrite;

    public BlockingStream()
    {
        _chunks = new Queue<byte[]>();
        _doneWriting = new ManualResetEvent(false);
        _dataAvailable = new ManualResetEvent(false);
        _events = new WaitHandle[] { _dataAvailable, _doneWriting };
        _doneWritingHandleIndex = 1;
        _lockForRead = new object();
        _lockForAll = new object();
    }

    public override bool CanRead { get { return true; } }
    public override bool CanSeek { get { return false; } }
    public override bool CanWrite { get { return !_illegalToWrite; } }

    public override void Flush() { }
    public override long Length {
        get { throw new NotSupportedException(); } }
    public override long Position {
        get { throw new NotSupportedException(); }
        set { throw new NotSupportedException(); } }
    public override long Seek(long offset, SeekOrigin origin) {
        throw new NotSupportedException(); }
    public override void SetLength(long value) {
        throw new NotSupportedException(); }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (buffer == null) throw new ArgumentNullException("buffer");
        if (offset < 0 || offset >= buffer.Length)
            throw new ArgumentOutOfRangeException("offset");
        if (count < 0 || offset + count > buffer.Length)
            throw new ArgumentOutOfRangeException("count");
        if (_dataAvailable == null)
            throw new ObjectDisposedException(GetType().Name);

        if (count == 0) return 0;

        while (true)
        {
            int handleIndex = WaitHandle.WaitAny(_events);
            lock (_lockForRead)
            {
                lock (_lockForAll)
                {
                    if (_currentChunk == null)
                    {
                        if (_chunks.Count == 0)
                        {
                            if (handleIndex == _doneWritingHandleIndex)
                                return 0;
                            else continue;
                        }
                        _currentChunk = _chunks.Dequeue();
                        _currentChunkPosition = 0;
                    }
                }

                int bytesAvailable =
                    _currentChunk.Length - _currentChunkPosition;
                int bytesToCopy;
                if (bytesAvailable > count)
                {
                    bytesToCopy = count;
                    Buffer.BlockCopy(_currentChunk, _currentChunkPosition,
                        buffer, offset, count);
                    _currentChunkPosition += count;
                }
                else
                {
                    bytesToCopy = bytesAvailable;
                    Buffer.BlockCopy(_currentChunk, _currentChunkPosition,
                        buffer, offset, bytesToCopy);
                    _currentChunk = null;
                    _currentChunkPosition = 0;
                    lock (_lockForAll)
                    {
                        if (_chunks.Count == 0) _dataAvailable.Reset();
                    }
                }
                return bytesToCopy;
            }
        }
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        if (buffer == null) throw new ArgumentNullException("buffer");
        if (offset < 0 || offset >= buffer.Length)
            throw new ArgumentOutOfRangeException("offset");
        if (count < 0 || offset + count > buffer.Length)
            throw new ArgumentOutOfRangeException("count");
        if (_dataAvailable == null)
            throw new ObjectDisposedException(GetType().Name);

        if (count == 0) return;

        byte[] chunk = new byte[count];
        Buffer.BlockCopy(buffer, offset, chunk, 0, count);
        lock (_lockForAll)
        {
            if (_illegalToWrite)
                throw new InvalidOperationException(
                    "Writing has already been completed.");
            _chunks.Enqueue(chunk);
            _dataAvailable.Set();
        }
    }

    public void SetEndOfStream()
    {
        if (_dataAvailable == null)
            throw new ObjectDisposedException(GetType().Name);
        lock (_lockForAll)
        {
            _illegalToWrite = true;
            _doneWriting.Set();
        }
    }

    public override void Close()
    {
        base.Close();
        if (_dataAvailable != null)
        {
            _dataAvailable.Close();
            _dataAvailable = null;
        }
        if (_doneWriting != null)
        {
            _doneWriting.Close();
            _doneWriting = null;
        }
    }
}

The concept behind BlockingStream is a bit different than most other streams in the .NET Framework. As with other streams, it derives from System.IO.Stream and overrides all of its abstract methods and properties. However, when it comes to position and thread safety, it exhibits behavior that's a bit unorthodox. Most streams in the .NET Framework are not thread safe, meaning that multiple threads can't safely access an instance of the stream concurrently and most streams maintain a single position at which the next read or write will occur. BlockingStream, on the other hand, is thread safe, and, in a sense, it implicitly maintains two positions, though neither is exposed as a numerical value to the user of the type.

BlockingStream works by maintaining an internal queue of data buffers written to it. When data is written to the stream, the buffer written is enqueued. When data is read from the stream, a buffer is dequeued in a first-in-first-out (FIFO) order, and the data in it is handed back to the caller. In that sense, there is a position in the stream at which the next write will occur and a position at which the next read will occur.

Note, however, that every read request doesn't necessarily modify the queue. If a reader requests an amount of data equal to or greater than the size of the next buffer in the queue, then the read request will be satisfied with that one buffer (Read can successfully return even if it returns less data than was requested). But if the user requests less than the size of the next buffer, the remaining data from the buffer is stored separately (not in the queue) so that it can be used to satisfy the next Read request (see _currentChunk in Figure 2). If the next Read request can be satisfied entirely with the remainder form the previous Read request, the queue will not be modified on that request.

BlockingStream uses a ManualResetEvent to signal to consumers whether data is available. If a thread tries to read data from the BlockingStream and there is no data available to read, the thread will block on this unset ManualResetEvent until data has been written. When data is written, the writer sets the event to wake up a consumer and notify it that data is available for reading. In fact, if there are multiple readers waiting for data, they will all wake up, which requires some additional logic in the Read method.

The Read method is implemented as one big loop that will continue to loop until either there is data available to fulfill the read request or until a writer has called SetEndOfStream, which informs the stream that there will be no more data; in that case, it would be silly to continue to block readers waiting for data, as they'd be waiting for a very long time. Each of these conditions is represented by a ManualResetEvent, the one mentioned earlier and another one that represents whether writing is complete. The first thing Read does once it enters the loop is to wait for either of these events to be set (using WaitHandle.WaitAny).

BlockingStream is useful for our stream pipeline because it enables the same stream-to-stream pattern used implicitly in the sequential version, but now supporting cross-thread implementations. The idea is that I can write methods that represent each of the Stream-based operations I need performed, as shown in Figure 3.

Figure 3 Methods for Stream-Based Operations (Close Figure 3)
static void Compress(Stream input, Stream output){
    using (GZipStream compressor = new GZipStream(
           output, CompressionMode.Compress, true))
        CopyStream(input, compressor);
}

static void Encrypt(Stream input, Stream output) {
    RijndaelManaged rijndael = new RijndaelManaged();
    ... // setup crypto keys
    using (ICryptoTransform transform = rijndael.CreateEncryptor())
    using (CryptoStream encryptor = new CryptoStream(
            output, transform, CryptoStreamMode.Write))
        CopyStream(input, encryptor);
}

Each of these methods can be executed on separate threads, with the output from one piped as the input to the other as a BlockingStream. Initially, the second operation will be blocked waiting on its input BlockingStream. As soon as the first operation writes data to the stream, the second operation will wake up, retrieve that data, and process it. I've coded this functionality into the StreamPipeline class, shown in Figure 4.

Figure 4 StreamPipeline (Close Figure 4)
public class StreamPipeline : IDisposable
{
    private Action<Stream, Stream>[] _filters;
    private List<BlockingStream> _blockingStreams;

    public StreamPipeline(params Action<Stream, Stream>[] filters)
    {
        if (filters == null) throw new ArgumentNullException("filters");
        if (filters.Length == 0 || Array.IndexOf(filters, null) >= 0)
            throw new ArgumentException("filters");

        _filters = filters;

        _blockingStreams = new List<BlockingStream>(_filters.Length - 1);
        for (int i = 0; i < filters.Length-1; i++)
        {
            _blockingStreams.Add(new BlockingStream());
        }
    }

    public void Run(Stream input, Stream output)
    {
        if (_blockingStreams == null)
            throw new ObjectDisposedException(GetType().Name);
        if (input == null) throw new ArgumentNullException("input");
        if (!input.CanRead) throw new ArgumentException("input");
        if (output == null) throw new ArgumentNullException("output");
        if (!output.CanWrite) throw new ArgumentException("output");

        ThreadStart lastStage = null;
        for (int i = 0; i < _filters.Length; i++)
        {
            Stream stageInput = i == 0 ? input : _blockingStreams[i - 1];
            Stream stageOutput =
                i == _filters.Length - 1 ? output : _blockingStreams[i];
            Action<Stream, Stream> filter = _filters[i];
            ThreadStart stage = delegate
            {
                filter(stageInput, stageOutput);
                if (stageOutput is BlockingStream)
                    ((BlockingStream)stageOutput).SetEndOfStream();
            };
            if (i < _filters.Length - 1)
            {
                Thread t = new Thread(stage);
                t.IsBackground = true;
                t.Start();
            }
            else lastStage = stage;
        }
        lastStage();
    }

    public void Dispose()
    {
        if (_blockingStreams != null)
        {
            foreach (BlockingStream stream in _blockingStreams)
                stream.Dispose();
            _blockingStreams = null;
        }
    }
}

StreamPipeline is very simple to use. The constructor accepts a params array of delegates, each of which expects an input Stream and an output Stream; these are called filters. It then creates and stores one BlockingStream to use between each pair of filters. Notice that the Compress and Encrypt methods I showed previously conform to the necessary delegate signature, so they can be used as filters (shown in Figure 5).

Figure 5 StreamPipeline in Action
Figure 5 StreamPipeline in Action(Click the image for a smaller view)
Figure 5 StreamPipeline in Action
Figure 5 StreamPipeline in Action (Click the image for a larger view)

The Run method accepts the input and output Stream that should be at the beginning and at the end of the pipeline. In our example, this might be a FileStream representing the data on disk to be compressed and encrypted, and an output FileStream to store the resulting data back to disk:

using (FileStream input = File.OpenRead("inputData.bin"))
using (FileStream output = File.OpenWrite("outputData.bin"))
using (StreamPipeline pipeline = new StreamPipeline(Compress, Encrypt))
{
    pipeline.Run(input, output);
}

Internally, for each stage in the pipeline, the Run method generates a delegate that runs the appropriate filter provided with the proper input and output streams. It creates new threads to run each of the work items, except for the last one, which it runs on the thread invoking Run. The current thread would need to block for the work to complete anyway, so it makes sense to reuse that thread to complete some of the processing rather than waste its resources.

Note my decision not to use the ThreadPool in this case. Normally, my first instinct is to queue work items into the ThreadPool rather than starting new Threads. In this case, however, the work I'm queueing can block waiting on other queued work items to perform some operations. That has the potential to possibly deadlock (see msdn.microsoft.com/msdnmag/issues/04/12/NETMatters for an example of where developers with the .NET Framework 1.x frequently ran into this), and as such I avoid the situation by creating custom threads for each operation.

That decision, of course, has its own drawbacks. For example, threads are expensive to create and tear down, so if the filters are short lived, the overhead here could dominate other aspects of the computation. If you were going to run a bunch of different input and output streams through the same StreamPipeline, it might be advantageous to rewrite the relevant portions of the class to keep the Threads alive and to reuse them from run to run. Given the constraints of your own applications, using ThreadPool might also be more appropriate.

Now that we've built our StreamPipeline, the key question is, how does it perform? After all, the whole point of this exercise was to utilize both cores on the machine in order to improve computation time. In my tests on my dual-core machine, using the same War and Peace data mentioned previously, this sequential pipeline implementation ran 20-25 percent faster than the sequential version. Disappointed? A question I hope would come into your mind at this point is, why didn't it run 100 percent faster? Is there something wrong with this implementation? After all, it's executing on two processors now, so shouldn't it run twice as fast?

I mislead a bit with my stuffing envelopes example from the beginning of this column. I don't know about you, but it takes me longer to fold a letter in thirds than it takes me to stuff that folded letter into an envelope. Given that, in our human pipeline, it's likely that the second person doing all the envelope stuffing will sit idly after stuffing each envelope, waiting for the next folded letter to arrive from his predecessor in the pipeline. So while we did throw more people at the problem, the problem didn't scale linearly to the number of people involved.

The same is true for our compression and encryption example. I timed how much time in the sequential implementation was spent doing the compression and how much was spent doing the encryption. Any guesses? Only 20 percent of the time was spent doing the encryption. Thus, our pipeline implementation is achieving very close to the maximum speedup possible in this situation.

That's not to say this implementation is the best it could be. There are several possible performance issues with it that could become more apparent as the number of processors increased, the most obvious of which is that you'd need at least one filter per processor to even have the opportunity of leveraging all processing power in the machine. Other possible issues include the fact that we're taking multiple locks within the BlockingStream implementation (one of which isn't necessary if there will only ever be one reader using the instance at a time, which is the case with the StreamPipeline's usage of BlockingStream). But for the relatively small amount of code we had to write, even a 20-25 percent performance gain like the one demonstrated here with compression and encryption can be quite valuable.

Also note that pipelines can be very effective when stages can themselves process multiple pieces of data in parallel. For example, in our envelope stuffing example, if multiple people were folding envelopes (or if the second person helped the first person fold when he didn't have any stuffing to do), the throughput of the entire system would increase. For the compression/encryption example, this would be akin to the chunking approach discussed earlier, such that the chunks are compressed in parallel in that stage of the pipeline, then passed on to the encryption stage which also encrypts the compressed chunks in parallel, and so on. The more fine-grain concurrency introduced into the application, the better the chance that all processors in the machine will be able to lend a hand in getting to the solution faster, especially as technology moves forward and we find ourselves with more and more processors in the average personal computer.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.


{ End Bracket }: Finding the Best of the Best

{ End Bracket }
Finding the Best of the Best
Howard Dierking

MSDN Magazine recently had the opportunity to sit down with Scott Pitasky, General Manager of Talent Acquisition at Microsoft, and members of his team to learn how Microsoft approaches recruitment. We learned that as the world gets smaller, Microsoft is continually adjusting not only its strategies for recruiting but, in some cases, aspects of its very culture in order to find and hire the best of the best. Several enthusiastic staffing managers and technical leaders participated in this discussion, and we'd like to thank them for their insights: James Hunt and Randall Boseman located in Redmond, Amit Vasishta of Microsoft India, and Chris Gootherts of Microsoft China.

If you search the Web for "interviewing at Microsoft," you'll find countless stories—some funny, some terrifying. However, many of those stories address a very fundamental question: what qualities does Microsoft look for in a software developer? Addressing this question allowed Pitasky and his team to describe how some of the historical qualities have been reshaped or completely replaced. At the core, however, three primary qualities were discussed: passion, intelligence, and the ability to solve problems.

The word "passion" came up often, clearly a Microsoft priority. What's unique, however, is in how the interview process has changed to evaluate passion in a candidate. Historically at Microsoft, passion meant being passionate about what Microsoft was doing at the time—and, as a result, many talented candidates were overlooked. More recently, passion has come to include any thing or activity that an individual feels strongly about. Boseman described an interview where the candidate talked in great detail of purchasing a motorcycle in Turkey and riding it through the Middle East and Europe. The candidate's enthusiasm for that experience was a better predictor of his passion than any technical debate.

Intelligence seems like such an obvious quality for someone in the software engineering field. However, like passion, intelligence has come to be redefined over the past several years. You've no doubt heard the stories about "puzzle questions" in Microsoft interviews. The problem with such questions is that they only determine one kind of intelligence—and many other types, such as interpersonal intelligence, are required to successfully create great software. Realizing this, Microsoft has significantly broadened its notions of intelligence to capture the qualities and experiences of great candidates who can help Microsoft innovate and grow in areas that the company is just starting to explore. To quote Pitasky: "We've gotten smarter about how we define smart."

According to Pitasky, the notion of a good "fit" for Microsoft is an old concept that is no longer relevant. Instead, hiring managers look for candidates who demonstrate a marked ability to learn and problem-solve. This has required a change in Microsoft interviewing culture over the past several years, says the group.

So how does an interviewer adequately measure these abilities in today's culture? Boseman says it best: "I would know exactly who is a good fit for a team if I could just have 30 minutes to play basketball with them. You can learn a ton about a person by watching how they play on a team." And while there's no plan for adjusting the interviewing strategy to include a game of 5 on 5, the group did describe how the use of role-play scenarios (either real or hypothetical) enables the interviewer to get a glimpse into how a candidate might behave in a team setting.

So as Microsoft has grown larger and the world has gotten smaller, the new global recruiting philosophy is that the company "hires from anywhere for anywhere." However, this requires Microsoft to take into account many differences in region. For example, in the United States, the number of computer science majors has been steadily dropping by about 15 percent per year. As a result, Microsoft has had to look toward other types of degrees and search technical communities for local heroes. On the other end of the spectrum, in India, you find a large number of candidates with engineering degrees. There, the local challenge is finding a diverse group of individuals who can not only solve deep technical problems but who also have the potential to mentor and to manage. Meanwhile, Israel has a large percentage of senior-level developers, while China has a noticeable gap in its mid-level and senior ranks. In general, one of the group's top priorities is in further developing Microsoft into a more global company, starting with the very people it hires.

Our discussion concluded with a funny interview story. A recruiter had scheduled an interview in Redmond with a candidate from Michigan, but the candidate didn't mention he had a medical condition that prevented him from flying. On the day of the interview, the recruiter arrived at work to find the candidate's RV in the parking lot—he had driven all the way from Michigan. Now that's passion!

Howard Dierking is the Editor-in-Chief for MSDN Magazine.