 MSDN Magazine June 2007

MSDN Magazine June 2007

C# 3.0: The Evolution Of LINQ And Its Impact On The Design Of C#

LINQ gives allows developers to take advantage of the power of queries. Find out how it came to be.

Anson Horton VISUAL BASIC: Unleash The Power Of Query In Visual Studio "Orcas"

LINQ, coming in the next version Visual Studio, codename "Orcas", adds type-safe data querying to .NET languages.

Ting Liang and Kit George C++: An Inside Look At The Next Generation Of Visual C++

The next version of Visual Studio (code-named “Orcas”) will help C++ developers take full advantage of the new capabilities in Windows Vista.

Tarek Madkour OFFICE UI: New VSTO Features Help You Customize Word And Outlook

See how to use Microsoft Visual Studio Tools for the Microsoft Office System to build powerful custom applications against the 2007 Microsoft Office system.

Steve Fox and Paul Stubbs SILVERLIGHT: Get Started Building A Deeper Experience Across The Web

Build rich, compelling, cross-platform, interactive applications with Microsoft Silverlight.

Laurence Moroney CONCURRENCY: Synchronization Primitives New To Windows Vista

Windows Vista has lots of goodies for native C++ developers including condition variables, slim reader/writer locks, and one-time initialization.

Robert Saccone and Alexander Taskov Editor's Note: Visual Studio at 10

Ruminations on ten years of Visual Studio.

Joshua Trupin Toolbox: XML Comments, Online Payments, and More

This month, online payment processing services, XML comments the easy way, and more.

Scott Mitchell CLR Inside Out: Reflections on Reflection

This month's installment of CLR Inside Out takes on reflection and the System.Reflection namespace.

Mike Repass Data Points: SQL Server Management Objects

SQL Server Management Objects offer developers a robust toolset for backing up and restoring databases, and issuing DDL commands, as John Papa explains.

John Papa Advanced Basics: Using RibbonX from Visual Basic

Ken Getz prepares Visual Basic developers to use RibbonX.

Ken Getz Cutting Edge: Transactional Workflows

When modeling business logic, workflows inevitably represent transactional tasks, so you need to know how to code transactional semantics in the Windows Workflow Foundation.

Dino Esposito Service Station: WCF Addressing In Depth

This month Aaron Skonnard looks at addressing details surrounding endpoint communication, many of which enable more advanced messaging scenarios.

Aaron Skonnard Wicked Code: UpdatePanel Tips and Tricks

Jeff Prosise explains when it's better to use UpdatePanel and when it's better to use asynchronous calls to WebMethods or page methods instead.

Jeff Prosise Foundations: ActivityExecutionContext in Workflows

This month Matt Milner explains a critical component in the execution, persistence, and compensation of workflows.

Matt Milner Concurrent Affairs: Asynchronous Device Operations

Jeff Richter uses the AsyncResult<TResult> class to implement the CLR’s Asynchronous Programming Model to perform hardware device operations asynchronously.

Jeffrey Richter Netting C++: EEK! An Introduction

Stanley Lippman begins a look at using C++/CLI as a purely .NET-compliant language.

Stanley B. Lippman .NET Matters: Handling Messages in Console Apps

Many developers who use the Microsoft .NET Framework think that application type is tied to the libraries that can be used in that application. Stephen Toub clarifies.

Stephen Toub { End Bracket }: Proud to Be a Developer

A developer doesn't just code; he or she writes code that is fast, reliable, well-tested, secure, maintainable and globalizable.

Adam Barr


C# 3.0: The Evolution Of LINQ And Its Impact On The Design Of C#

C# 3.0

The Evolution Of LINQ And Its Impact On The Design Of C#

Anson Horton
This article is based on a prerelease version of Visual Studio code-named "Orcas." All information herein is subject to change.
This article discusses:
  • C# and LINQ
  • The evolution of LINQ
  • SQL querying from code
This article uses the following technologies:
LINQ, C#
Contents
Lambda Expressions
Extension Methods
Anonymous Types
Implicitly Typed Local Variables
Object Initializers
Query Expressions

I
was a huge fan of the Connections series, hosted by James Burke, when it aired on the Discovery Channel. Its basic premise: how seemingly unrelated discoveries influenced other discoveries, which ultimately led to some modern-day convenience. The moral, if you will, is that no advancement is made in isolation. Not surprisingly, the same is true for Language Integrated Query (LINQ).

In simple terms, LINQ is a series of language extensions that supports data querying in a type-safe way; it will be released with the next version Visual Studio, code-named "Orcas." The data to be queried can take the form of XML (LINQ to XML), databases (LINQ-enabled ADO.NET, which includes LINQ to SQL, LINQ to Dataset and LINQ to Entities), objects (LINQ to Objects), and so on. The LINQ architecture is shown in Figure 1.

Figure 1 LINQ Architecture
Figure 1 LINQ Architecture(Click the image for a smaller view)
Figure 1 LINQ Architecture
Figure 1 LINQ Architecture (Click the image for a larger view)

Let’s look at some code. A sample LINQ query in the upcoming "Orcas" version of C# might look like:

var overdrawnQuery = from account in db.Accounts
                     where account.Balance < 0
                     select new { account.Name, account.Address };

When the results of this query are iterated over using foreach, each element returned would consist of a name and address of an account that has a balance less than 0.

It’s immediately obvious from the sample above that the syntax is like SQL. Several years ago, Anders Hejlsberg (chief designer of C#) and Peter Golde thought of extending C# to better integrate data querying. Peter, who was the C# compiler development lead at the time, was investigating the possibility of making the C# compiler extensible, specifically to support add-ins that could verify the syntax of domain-specific languages like SQL. Anders, on the other hand, was conceiving a deeper, more specific level of integration. He was thinking about a set of "sequence operators" that would operate on any collection that implemented IEnumerable, as well as remote queries for types that implemented IQueryable. Ultimately, the sequence operator idea gained the most support, and in early 2004 Anders submitted a paper about the idea to Bill Gates’s Thinkweek. The feedback was overwhelmingly positive. In the early stages of the design, a simple query had the following syntax:

sequence<Customer> locals = customers.where(ZipCode == 98112);

Sequence, in this case, was an alias for IEnumerable<T>, and the word "where" was a special operator understood by the compiler. The implementation of the where operator was a normal C# static method that took in a predicate delegate (that is, a delegate of the form bool Pred<T>(T item)). The idea was for the compiler to have special knowledge about the operator. This would allow the compiler to correctly call the static method and create the code to hook up the delegate to the expression.

Let’s suppose that the example above would be the ideal syntax for a query in C#. What would this query look like in C# 2.0, without any language extensions?

IEnumerable<Customer> locals = EnumerableExtensions.Where(customers,
                                                    delegate(Customer c)
        {
            return c.ZipCode == 98112;
        });

This code is frightfully verbose, and worse, it requires significant digging to find the relevant filter (ZipCode == 98112). And this example is simple; imagine how much more unreadable this would be with several filters, projections, and so forth. The root of the verbosity is the syntax required for anonymous methods. In the ideal query, the expression would require nothing but the expression to be evaluated. The compiler would then attempt to infer the context; for example, that ZipCode was really referring to the ZipCode defined on Customer. How to fix this problem? Hardcoding the knowledge of specific operators into the language didn’t sit well with the language design team, so they started looking for an alternate syntax for anonymous methods. They wanted it to be extremely concise, and yet not necessarily require more knowledge than the compiler currently needed for anonymous methods. Ultimately they devised lambda expressions.

Lambda Expressions

Lambda expressions are a language feature that is similar in many ways to anonymous methods. In fact, if lambda expressions had been put into the language first, there would have been no need for anonymous methods. The basic idea is that you can treat code as data. In C# 1.0, it is common to pass strings, integers, reference types, and so on to methods so that the methods can act on those values. Anonymous methods and lambda expressions extend the range of the values to include code blocks. This concept is common in functional programming.

Let’s take the example above and replace the anonymous method with a lambda expression:

IEnumerable<Customer> locals =
    EnumerableExtensions.Where(customers, c => c.ZipCode == 91822);

There are several things to notice. For starters, the brevity of the lambda expression can be attributed to a number of factors. First, the delegate keyword isn’t used to introduce the construct. Instead, there is a new operator, =>, which tells the compiler that this isn’t a normal expression. Second, the Customer type is inferred from the usage. In this case, the signature of the Where method looks something like:

public static IEnumerable<T> Where<T>(
    IEnumerable<T> items, Func<T, bool> predicate)

The compiler is able to infer that "c" refers to a customer because the first parameter of the Where method is IEnumerable<Customer>, such that T must, in fact, be Customer. Using this knowledge, the compiler also verifies that Customer has a ZipCode member. Finally, there is no return keyword specified. In the syntactic form, the return member is omitted but this is merely syntactic convenience. The result of the expression is still considered to be the return value.

Lambda expressions, like anonymous methods, also support variable capture. For example, it’s possible to refer to the parameters or locals of the method that contains the lambda expression within the lambda expression’s body:

public IEnumerable<Customer> LocalCusts(
    IEnumerable<Customer> customers, int zipCode)
{
    return EnumerableExtensions.Where(customers,
        c => c.ZipCode == zipCode);
}

Finally, Lambda expressions support a more verbose syntax that allows you to specify the types explicitly, as well as execute multiple statements. For example:

return EnumerableExtensions.Where(customers,
    (Customer c) => { int zip = zipCode; return c.ZipCode == zip; });

The good news is that we’re much closer to the ideal syntax proposed in the original paper, and we were able to get there with a language feature that is generally useful outside of query operators. Let’s take a look at where we are again:

IEnumerable<Customer> locals =
    EnumerableExtensions.Where(customers, c => c.ZipCode == 91822);

There is an obvious problem here. Instead of thinking about the operations that can be performed on Customer, the consumer currently has to know about this EnumerableExtensions class. In addition, in the case of multiple operators, the consumer has to invert his thinking to write the correct syntax. For example:

IEnumerable<string> locals =
    EnumerableExtensions.Select(
        EnumerableExtensions.Where(customers, c => c.ZipCode == 91822),
        c => c.Name);

Notice that the Select is the outer method, even though it operates on the result of the Where method. The ideal syntax would look more like the following:

sequence<Customer> locals =
    customers.where(ZipCode == 98112).select(Name);

So, would it be possible to move closer to the ideal syntax with another language feature?

Extension Methods

Much better syntax, it turns out, was to come in the form of a language feature known as extension methods. Extension methods are basically static methods that are callable through an instance syntax. The root of the problem for the query above is that we want to add methods to IEnumerable<T>. However, if we were to add operators, such as Where, Select, and so on, every existing and future implementer would be required to implement those methods. The vast majority of those implementations would be the same, though. The only way to share "interface implementation" in C# is to use static methods, which is what we’ve done with the EnumerableExtensions class used previously.

Let’s suppose we were to write the Where method as an extension method instead. The query could then be rewritten as:

IEnumerable<Customer> locals =
    customers.Where(c => c.ZipCode == 91822);

For this simple query, this syntax is very close to the ideal. But what exactly does it mean to write the Where method as an extension method? It’s actually fairly straightforward. Basically the signature of the static method changes such that a "this" modifier is added to the first parameter:

public static IEnumerable<T> Where<T>(
    this IEnumerable<T> items, Func<T, bool> predicate)

In addition, the method must be declared within a static class. A static class is one that may contain only static members and that is denoted by the static modifier on the class declaration. That’s all there is to it. This declaration instructs the compiler to allow Where to be called with the same syntax as an instance method on any type that implements IEnumerable<T>. The Where method must, however, be accessible from the current scope. A method is in scope when the containing type is in scope. Therefore, it’s possible to bring extension methods into scope through the Using directive. (See the sidebar "Extension Methods" for more information.)

Extension Methods

It’s clear that extension methods help simplify our example query, but are they a generally useful language feature outside of that scenario? It turns out that there are many uses for extension methods. One of the most common will probably be to provide shared interface implementations. For example, suppose you have the following interface:

interface IDog
{
    // Barks for 2 seconds
    void Bark();
    void Bark(int seconds);
}

This interface requires that every implementer write an implementation for both overloads. With the "Orcas" version of C#, the interface could simply be:

interface IDog
{
    void Bark(int seconds);
}

An extension method could be added in another class:

static class DogExtensions
{
    // Barks for 2 seconds
    public static void Bark(this IDog dog)
    {
        dog.Bark(2);
    }
}

Now the implementer of the interface need only implement a single method, but the clients of the interface may freely call either overload.

We now have a syntax that is very close to the ideal for the filter clause, but is that all there is to the "Orcas" version of C#? Not quite; let’s extend the example a bit by projecting out only the customer’s name, as opposed to the entire customer object. As I mentioned earlier, the ideal syntax would take the following form:

sequence<string> locals =
    customers.where(ZipCode == 98112).select(Name);

With just the language extensions we’ve discussed, lambda expressions and extension methods, this could be rewritten as:

IEnumerable<string> locals =
    customers.Where(c => c.ZipCode == 91822).Select(c => c.Name);

Notice that the return type is different for this query—IEnumerable<string> instead of IEnumerable<Customer>. This happens because we are only returning the name of the customer from the select statement

That works really well when the projection is only a single field. However, suppose that instead of just the Name of the customer, we also want to return the customer’s address. The ideal syntax might look like this:

locals = customers.where(ZipCode == 98112).select(Name, Address);

Anonymous Types

If we were to continue using our existing syntax to return the name and address, we’d quickly run into the problem that there is no type that contains only a Name and Address. We could still write this query, however, by introducing that type:

class CustomerTuple
{
    public string Name;
    public string Address;

    public CustomerTuple(string name, string address)
    {
        this.Name = name;
        this.Address = address;
    }
}

We could then use that type, here CustomerTuple, to construct the result of our query:

IEnumerable<CustomerTuple> locals =
    customers.Where(c => c.ZipCode == 91822)
                 .Select(c => new CustomerTuple(c.Name, c.Address));

That sure seems like a lot of boilerplate code to project out a subset of the fields. It’s also often unclear what to name such a type. Is CustomerTuple really a good name? What if we had projected out Name and Age instead? That could also be a CustomerTuple. So, the problems are that we have boilerplate code and it doesn’t seem that there are any good names for the types that we create. Plus, there could also be many different types required, and managing them could quickly become a headache.

This is exactly what anonymous types are for. This feature basically allows the creation of structural types without specifying the name. If we rewrite the query above using anonymous types, here’s what it looks like:

locals = customers.Where(c => c.ZipCode == 91822)
                       .Select(c => new { c.Name, c.Address });

This code implicitly creates a type that has the fields Name and Address:

class
{
    public string Name;
    public string Address;
}

This type can’t be referenced by name, since it has none. The names of the fields can be explicitly declared in the anonymous type creation. For example, if the field being created is derived from a complicated expression, or the name simply isn’t desirable, it’s possible to change the name:

locals = customers.Where(c => c.ZipCode == 91822)
    .Select(c => new { FullName = c.FirstName + “ “ + c.LastName,
                       HomeAddress = c.Address });

In this case, the type that is generated has fields named FullName and HomeAddress.

This gets us closer to the ideal, but there is a problem. You’ll notice that I strategically omitted the type of locals in any place where I used an anonymous type. Obviously we can’t state the name of anonymous types, so how do we use them?

Implicitly Typed Local Variables

There’s another language feature known as implicitly typed local variables (or var for short) that instructs the compiler to infer the type of a local variable. For example:

var integer = 1;

In this case, integer has the type int. It’s important to understand that this is still strongly typed. In a dynamic language, integer’s type could change later. To illustrate this, the following code does not compile:

var integer = 1;
integer = “hello”;

The C# compiler will report an error on the second line, stating that it can’t implicitly convert a string to an int.

In the case of the query above, we can now write the full assignment as shown here:

var locals =
   customers
       .Where(c => c.ZipCode == 91822)
       .Select(c => new { FullName = c.FirstName + “ “ + c.LastName,
                          HomeAddress = c.Address });

The type of locals ends up being IEnumerable<?> where "?" is the name of a type that can’t be written (since it is anonymous).

Implicitly typed locals are just that: local within a method. It is not possible for them to escape the boundaries of a method, property, indexer, or other block because the type cannot be explicitly stated, and "var" is not legal for fields or parameter types.

Implicitly typed locals turn out to be convenient outside of the context of a query. For example, it helps simplify complicated generic instantiations:

var customerListLookup = new Dictionary<string, List<Customer>>();

We’re now in a good place with our query; we’re close to the ideal syntax and we’ve gotten there with general-purpose language features.

Interestingly, we found that as more people worked with this syntax, there was often a need to allow a projection to escape the boundaries of a method. As we saw earlier, this is possible by constructing an object by calling its constructor from within Select. However, what happens if there is no constructor that takes exactly the values you need to set?

Object Initializers

For this case, there is a C# language feature in the upcoming "Orcas" version known as object initializers. Object initializers basically allow the assignment of multiple properties or fields in a single expression. For example, a common pattern for object creation is:

Customer customer = new Customer();
customer.Name = “Roger”;
customer.Address = “1 Wilco Way”;

In this case, there is no constructor of Customer that takes a name and address; however, there are two properties, Name and Address, that can be set once an instance is created. Object initializers allow the same creation with the following syntax:

Customer customer = new Customer()
    { Name = “Roger”, Address = “1 Wilco Way” };

In our earlier CustomerTuple example, we created the CustomerTuple class by calling its constructor. We can achieve the same result via object initializers:

var locals =
    customers
        .Where(c => c.ZipCode == 91822)
        .Select(c =>
             new CustomerTuple { Name = c.Name, Address = c.Address });

Notice that object initializers allow the parentheses of the constructor to be omitted. In addition, both fields and settable properties can be assigned within the body of the object initializer.

We now have a succinct syntax for creating queries in C#. However, we also have an extensible way to add new operators (Distinct, OrderBy, Sum, and so on) through extension methods and a distinct set of language features useful in their own right.

The language design team now had several prototypes to get feedback on. So we organized a usability study with many participants who had experience with both C# and SQL. The feedback was almost universally positive, but it was clear there was something missing. In particular, it was difficult for the developers to apply their knowledge of SQL because the syntax we thought was ideal didn’t map very well to their domain expertise.

Query Expressions

The language design team then designed a syntax that is closer to SQL, known as query expressions. For example, a query expression for our example might look like this:

var locals = from c in customers
             where c.ZipCode == 91822
             select new { FullName = c.FirstName + “ “ +
                          c.LastName, HomeAddress = c.Address };

Query expressions are built on the language features described above. They are literally syntactically translated into the underlying syntax that we’ve already seen. For example, the query above is translated directly into:

var locals =
   customers
       .Where(c => c.ZipCode == 91822)
       .Select(c => new { FullName = c.FirstName + “ “ + c.LastName,
                          HomeAddress = c.Address });

Query expressions support a number of different "clauses," such as from, where, select, orderby, group by, let, and join. These clauses translate into the equivalent operator calls, which in turn, are implemented via extension methods. The tight relationship of the query clauses and the extension methods that implement the operators makes it easy to combine them if the query syntax doesn’t support a clause for a needed operator. For example:

var locals = (from c in customers
              where c.ZipCode == 91822
              select new { FullName = c.FirstName + “ “ +
                          c.LastName, HomeAddress = c.Address})
             .Count();

In this case the query now returns the number of customers who live in the 91822 ZIP Code area.

And with that, we’ve managed to end just about where we started (which I always find rather satisfying). The syntax of the next version of C# evolved over the past few years through several new language features to ultimately arrive very close to the original syntax proposed in the winter of 2004. The addition of query expressions builds on the foundations provided by the other language features in the upcoming version of C# and makes many query scenarios easier to read and understand for developers with a background in SQL.

Anson Horton has been a Program Manager at Microsoft for almost six years. He has been working on the C# team since its creation, and was on the C++ team prior to that. He has been involved in the design of the C# language and compiler, the C# project system, the C# IDE (IntelliSense), and the C# Expression evaluator and debugger. Anson maintains a blog at blogs.msdn.com/ansonh which he updates as infrequently as possible.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


VISUAL BASIC: Unleash The Power Of Query In Visual Studio &quot;Orcas&quot;

VISUAL BASIC

Unleash The Power Of Query In Visual Studio "Orcas"

Ting Liang and Kit George
This article is based on a prerelease version of Visual Studio code-named “Orcas.” All information herein is subject to change.
This article discusses:
  • Visual Basic and query integration
  • Supported clauses such as Where, OrderBy, and Select
  • Understanding and using query results
This article uses the following technologies:
Visual Basic, LINQ
Contents
LINQ Syntax
The From Clause
The Select Clause
Using the Results of a Query
Filtering with the Where Clause
Sorting with Order By
More Clauses
Summary

F
or a long time, developers have wanted to be able to use data querying techniques in their code, much as they might in their SQL queries. Now, with the upcoming release of Microsoft® Visual Studio,® code-named "Orcas," this kind of facility is becoming a reality. The next version of Visual Studio includes a new set of language features known collectively as Language Integrated Query (LINQ), which adds data querying capabilities directly to Visual Basic® and Visual C#®.

With LINQ, you can write your data access code directly in Visual Basic. You get compile-time syntax- and schema-checking, and better tools support (such as IntelliSense®) for authoring your queries. No longer must you write a query in a string and not even know until runtime if it’s well-formed.

With LINQ, the data-access code is abstracted from the underlying storage facility. This means you can use the same code constructs for querying relational databases, XML, and in-memory objects, and you can easily join information across different source domains. For example, you can create a union between a list of customer names stored in XML and one stored in a database table. In addition, LINQ allows deferred execution, which lets you break up a query into logical parts and still get the performance benefits of running the end result only once.

Finally, Visual Basic/XML integration virtually eliminates the conceptual barrier between the code you write and the document you’re trying to express or manipulate. Now let’s start learning how to write some queries.

LINQ Syntax

Using LINQ you can execute queries on a variety of sources, including Microsoft .NET Framework object collections, relational databases, or XML data sources. The general format of a LINQ query in Visual Basic looks like this:

From ... < Where ... Order By ... Select ... >

Of course, if you’re a SQL programmer, you’ll wonder why the query begins with From instead of Select. The reason is simple: IntelliSense. By indicating the data source first, IntelliSense can show appropriate lists of type members on the objects in your collection. This is important when you’re typing the Select clause.

In the following sections, we look at some of the standard LINQ clauses and see how they build on each other to allow the creation of robust, flexible queries. We’ll walk through each of the following in turn: the From clause, the Select clause, using the result of a query, the Where clause, and the Order By clause.

In the following examples, we sometimes refer to a Customer or an Order. For those examples, imagine they are defined as in Figure 1. Each Customer has a reference to its own collection of orders via the Orders property. This allows for a simple relationship between the two entities. The association is formed by the CustomerID foreign key’s presence in each Order. In addition to these type definitions, there’s a collection of Customers stored in the variable customers.

Figure 1 Customers and Orders (Close Figure 1)
Public customers As List(Of Customer)

Public Class Customer
    Public Property CustomerID As Integer
    Public Property FirstName As String
    Public Property LastName As String
    Public Property City As String
    Public Property Orders As List(Of Order)
End Class

Public Class Order
    Public Property OrderID As Integer
    Public Property CustomerID As Integer
    Public Property OrderDate As DateTime
End Class

The From Clause

The From clause is the heart of LINQ, since it’s required in every query. A query must always start with the From clause, and it’s the only clause that must be supplied explicitly. The From clause follows this basic pattern:

From <query variable> In <datasource>

Essentially, From represents a walk through a set of data. For that to work, it needs a variable that corresponds to each item of data in the source, much like an iteration variable used in a For Each statement. In clauses following the From statement, the query variable can be referenced to filter, sort, or otherwise manipulate data.

Here’s a really simple example of a From clause:

Dim numbers As Integer = {1, 7, 4, 2, 91, 12, 23, 37, 42}
Dim allNumbers = From number In numbers
For Each num In allNumbers
    Console.Write(num & “ “)
Next

In this example, the query returns a collection typed as IEnumerable(Of Integer). There are a couple of things to notice about this query as well. First, the Select is implied, which is why you don’t see it in the example. A Select statement is optional; if you don’t supply one, then the set of variables in scope is returned. The second thing is that this query is really not very useful. You could remove the query and simply iterate over the numbers variable using a For Each loop. The real power of LINQ is not evident just yet. But look at this query:

From <query variable> As Type In <datasource>

Dim checkboxes = From checkbox As Checkbox In groupJobType.Controls

This variant of the From statement allows you to specify a type statement following the query variable. This is useful if you want to retrieve types from a particular source, and retype them as you retrieve them, to a more (or perhaps less) concrete type. We happen to know that all of the controls in the groupJobType GroupBox are Checkbox controls, so we can safely type them to Checkbox as we retrieve them.

There is another variant of the From clause that uses the From keyword twice:

From <query variable> In <datasource> From <let variable> = <value>

From <query variable> In <datasource>, <let variable> = <value>

Dim evens = From Number In numbers From IsEven = (Number Mod 2 = 0)
Dim evens = From Number In numbers, IsEven = (Number Mod 2 = 0)

This query syntax is actually all one From clause, with the latter part known as the "Let" part of the From. Essentially we’re defining a second variable and assigning it to the specified value. In this case, we’re defining a second value on our resultset called "IsEven," which is a Boolean value—the result of whether or not our selected number is even. This of course means that what is returned is changed: it’s no longer just an IEnumerable of Integers, as each item in the returned enumerable contains both a number and a Boolean. We’ll see how to use the results of a query below. Notice in the general syntax expression provided that it’s possible to omit the second use of the From keyword and use a comma instead; both examples above are equivalent.

What distinguishes a Let variable from a standard query variable is that the Let variable is always assigned a value using =, while a query variable is followed by the In keyword. However, the Visual Basic team is actively exploring the way the Let variable is exposed, so there’s a good chance the supported syntax for Let will change between now and the time the product is released in final form:

From <queryVar1> In <datasource1>, <queryVar2> In <datasource2>, ...

Dim customerOrders = From cust In customers, ord In cust.Orders

This syntax allows you to specify multiple data sources. In the above example, we get a collection of customers and orders; essentially, this is an implicit Join on the two sources. Much of the time, the sources won’t be so easily related (your first query variable won’t have a collection of the second query variable on it), and therefore you’ll have to associate them with a Where clause. But in this case, our Orders collection is actually on each cust instance; therefore we’re only getting the orders that apply to the given customer.

The Select Clause

So far you’ve seen how to specify a data source. Note that in every example we’ve omitted the Select keyword. This simply means that we used the default Select behavior, which is to Select every query or Let variable specified (logically equivalent to Select *). But we also have the power to specify a Select clause, and we get considerably more flexibility when we do:

Select <varA>, <varB>, ...

Dim allNumbers = From number In numbers _
                 Select number

Dim evens = From Number In numbers From IsEven = (Number Mod 2 = 0) _
            Select Number, IsEven

Dim customerOrders = From cust In customers, ord In cust.Orders _
                     Select cust, ord

This is Select at its simplest. Compare the earlier queries to their equivalent queries in the From section here. Every query in these three examples with an explicit Select clause will have the same behavior as the earlier queries where Select was implicit.

Using Select gets more interesting, however, when you try variants on this theme:

Dim customerInfos1 = From cust In customers _
                     Select cust.FirstName, cust.LastName

Dim customerInfos2 = From cust In customers, ord In cust.Orders _
                     Select cust.LastName, ord.OrderID, ord.OrderDate

Notice first of all that a new general syntax is not specified (there’s no comment above the example). This is because if you look at these examples, the previous code is in the same style: FirstName and LastName are simply variables after all. But unlike the previous examples, we dereference the query variable and return only a couple of properties. This is a very powerful capability.

Another great feature of LINQ is that when you don’t need to rename fields, Visual Basic infers the names based on the names of the properties. Sometimes, however, you may want to rename fields, or merge data together, like so:

Select <aliasA> = <varA>, <aliasB> = <varB>, ...

Dim customerInfos1 = From cust In customers, ord In cust.Orders _
                     Select Date = ord.OrderDate, _
                            Name = cust.FirstName & cust.LastName

In some scenarios (like Date = ord.OrderDate), you may be doing it for personal preference. In another scenario (as in Name = cust.FirstName & cust.LastName) you actually want to create a new piece of data—a combination of fields. There is also a third scenario for using aliases. Imagine you’re selecting two fields from different sources that happen to have the same name. Name inference would give those variables the same property name on the resulting type, and this would fail to compile due to the conflict. You can use an alias for one (or both) conflicting field names to resolve the conflict.

One final point to note about the Select clause is that it rescopes your variables, so any clauses in a query can only see what Select exposes. We discuss this in more detail when we talk about the Where clause.

Using the Results of a Query

Now let’s see how to use the results of the query. To do so you’ll need to understand what’s formed when a query is executed:

Dim allNumbers = From number In numbers _
                 Select number

The result of any query will always be an IEnumerable(Of T). This is one of the most reassuring and powerful aspects of queries, since it means you can always iterate over the data. Sometimes you may know what the type of T is, and sometimes you may not, such as in the case of anonymous types.

Anonymous types are new with LINQ and are used to represent any result of a query with multiple returned field columns. An anonymous type is a type that you can’t directly name or reference in code but that the compiler treats just like other named types (the compiler gives the type a name, you just don’t know what it is, nor do you need to). The field name of an anonymous type can be specified directly in code or inferred by the compiler from the query expression automatically. For example, the use of anonymous types enables code like you saw earlier in which the pairings of {Num, IsEven} and {cust, ord} are effectively new types, each with two pieces of data (with Num, IsEven and Cust, Ord as respective, accessible field names):

Dim customerInfos1 = From cust In customers _
                     Select cust.FirstName, cust.LastName

Dim customerInfos2 = From cust In customers, ord In cust.Orders _
                     Select cust.LastName, ord.OrderID, ord.OrderDate

As you can see, anonymous types allow you to specify a varying number of fields in the Select clause without requiring you to specify an exact type beforehand that represents each particular combination of fields.

While we say that each query is an IEnumerable(Of T), you may have noticed that none of the examples actually specify a type for the variable representing the query. You may have become concerned that the variable is getting typed as System.Object and that you’re losing all strong typing support in the process.

Not so. A new compiler feature called type inference means that the compiler can infer the type of a variable declared with no type specifier, based on its initialization value (if you include the statement Option Infer On at the top of your code). For example, the following variables are statically typed based on the initialization value:

Dim x = 5             ‘ x is typed as an Integer
Dim y = DateTime.Now  ‘ x is typed as a DateTime
Dim z = GetCustomer() ‘ assuming GetCustomer returns a customer
                      ‘ type, z is typed as a Customer

x = “hello”           ‘ COMPILE-TIME ERROR: x can only store Integers

In the same way, a variable representing a query is strongly typed. This enables you to actually use members directly on that variable, without fear of those members being late-bound. Plus, you retain other benefits of strong typing such as IntelliSense support. The same kind of type inference happens in a For Each statement, when a new variable is declared with no type specifier:

 ‘ assuming allNumbers is a collection of Integers,
 ‘ the type of num is inferred to be an Integer
For Each num In allNumbers

In general, what you can’t easily know when using queries is what specific kind of object is being returned. This is where you have to put your trust in queries. Under the covers, different kinds of objects may be used to store your query results. All that’s important to know is that they will always be an IEnumerable(Of T), therefore you can iterate through them with a For Each or pass them to an API that accepts an IEnumerable object.

Having said that, there is a rule of thumb that will help you know what is returned, and this rule is important (you’ll see why in a second). When the results you’re returning consist of a single named type, the query returns an IEnumerable(Of <your type>). When the results you’re returning are of more than one type, the query returns an IEnumerable(Of <anonymous type>). So in the example above, which just returns an integer from an array of Integers, you’ll get an IEnumerable(Of Integer).

When the query result is an IEnumerable of a single type, using that result is easy. You simply use a For Each clause:

For Each num In allNumbers
    Console.WriteLine(num)
Next

There is nothing here that should surprise you since we have an IEnumerable of a single type. But now let’s look at a scenario that involves anonymous types:

Dim evens = From Number In numbers From IsEven = (Number Mod 2 = 0) _
            Select Number, IsEven

For Each numInfo In evens
    Console.WriteLine(“Is “ & numInfo.Number & “ even? “ & _
                      numInfo.IsEven)
Next

Because the query returns an object that has two pieces of data in it, the variable "evens" is an IEnumerable(Of <anonymous type>). Each entity we get out has fields named Num and IsEven; therefore, when we iterate over our evens collection, we need to dereference each of these pieces of data to retrieve them (numInfo.Num for example). Notice a couple of things here. First, we named the iteration variable numInfo. When iterating over collections of anonymous types, it’s a good idea to use a general name to indicate that it has multiple pieces of data. Second, we capitalized the query variables in the example, so when we dereference and use them inside the For Each loop, they look like standard properties. We didn’t need to do this for "number" in the previous example, since an anonymous type was not introduced.

Just to show how easy this is, let’s see one more example:

Dim customerInfos = From cust In customers, ord In cust.Orders _
                    Select Name = cust.FirstName & cust.LastName, _
                           Date = ord.OrderDate

For Each custInfo In customerInfos
    Console.WriteLine(“Customer Name: “ & custInfo.Name & _
                      “ Order Date: “ & custInfo.OrderDate)
Next

It really doesn’t get much easier. However, sometimes you may want to retrieve your first name and last name fields separately. Here’s a variant on the above that retains all of the data in our Customers and Orders (we want to use the other fields later) but that achieves the same for this specific iteration.

Dim customerInfos = From cust In customers, ord In cust.Orders _
                    Select Customer = cust, Order = ord

For Each custInfo In customerInfos
    Console.WriteLine(“Customer Name: “ & _
                       custInfo.Customer.LastName & “, “ & _
                       custInfo.Customer.FirstName)
    Console.WriteLine(“Order Date: “ & custInfo.Order.OrderDate)
Next

Again, nothing unexpected. By including each entire object in our results, we have to dereference the objects to get to the specific fields we want to consume. Note that we concatenate the last and first names inside the For Each loop.

We’ll leave you with one further tip. You can do some pretty cool things if you look at the members available on your query. For example, consider this code:

Dim customerInfos = From cust In customers, ord In cust.Orders _
                    Select Customer = cust, Order = ord
Console.WriteLine(“There are “ & customerInfos.Count & _
                  “ customer orders”)

This example uses Count directly on the result of the query to see how many items it has. We won’t get into all of the members you can find on the collection, but don’t forget to check out what kinds of things you can do with your resultset. Also note that these members are available on all collections resulting from a query. If you surmised that this means these members must be on IEnumerable(Of T), you’re right; even their implementations must be. This is a new capability called extension methods.

One concept you must understand about how queries execute is deferred execution. In deferred execution, an attempt is not made to retrieve the data from a query until it is actually used. This is important because it allows you to write the query in different parts, without multiple calls to the underlying data source, and there are significant performance benefits, especially when accessing information against a database (multiple calls in that situation should be especially avoided).

What this means is that the declaration of a query does not actually retrieve the data; only taking an action that needs the data causes it to be retrieved. The action you’ll probably take most often is to iterate over the query. But other possible actions include displaying the data or trying to use a method such as Count to find the number of entries in the query. Figure 2 shows where data retrieval is performed.

Figure 2 Data Retrieved in the For Each Clause (Close Figure 2)
Dim seattleCustomers = From cust In customers _
                       Where cust.City = “Seattle”

Dim recentOrders = From ord In orders _
                   Where ord.OrderDate.Year = 2007

Dim custInfos = From cust In seattleCustomers, ord in recentOrders _
                Where cust.CustomerID = ord.OrderID _
                Select Name = cust.LastName & “, “ & cust.FirstName, _
                              cust.CustomerID, ord.OrderDate _
                Order By OrderDate Descending, Name

‘ None of the above queries are run until this point.
‘ This means that the information is only retrieved once.
For Each custInfo In custInfos
    ‘ take action on the data here
Next

Filtering with the Where Clause

The Where clause makes your query more powerful by letting you filter on specific conditions. The Where clause operates just like an If statement. That is, whatever is possible in an If statement is possible in a Where clause:

Where <condition>

Dim evens = From num In numbers _
            Where num Mod 2 = 0 _
            Select num

Notice that this collection is called "evens," as in the previous examples. Unlike previous examples, however, it consists only of the even numbers. When you iterate over the results, you’ll only get the even numbered results:

Dim mids = From num In numbers _
           Where num > 10 And num < 50 _
           Select num

The beauty of the Where clause is its simplicity. Your <condition> is any condition you choose. Let’s see how it works on a more involved query:

Dim seattleCustomers = From cust In customers, ord In cust.Orders _
                       Where cust.City = “Seattle” _
                       Select Customer = cust, Order = ord

Dim seattle2003Orders = From cust In customers, ord In cust.Orders _
                        Where cust.City = “Seattle” _
                        And ord.OrderDate.Year = 2003 _
                        Select Customer = cust, Order = ord

Sometimes you might want to filter by one field, and Select a different field (or fields). This is very straightforward:

Dim dallas2005Orders = From cust In customers, ord In cust.Orders _
                        Where cust.City = “Dallas” _
                        And ord.OrderDate.Year = 2005 _
                        Select cust.CustomerID, ord.OrderID

Once you’ve declared your From clause, the order of the other clauses are optional—you can use them in any order you prefer:

Dim seattle2003Orders = From cust In customers, ord In cust.Orders _
                        Select Customer = cust, Order = ord _
                        Where Customer.City = “Seattle” _
                              And Order.OrderDate.Year = 2003

Compare this example to the example of the same name above. Notice that we’ve moved the Select clause to before the Where clause, which is perfectly acceptable. But notice also that the Where clause must change slightly and now refers to Customer and Order rather than cust and ord. Be very aware that the Select clause is a rescoping clause, thus any clauses after the Select clause can only see what the Select exposes, which means that this code will be invalid:

‘ This won’t compile
Dim dallas2005Orders = From cust In customers, ord In cust.Orders _
                       Select cust.CustomerID, ord.OrderID _
                       Where cust.City = “Dallas” _
                              And ord.OrderDate.Year = 2005

We refer to ord and cust inside the Where clause, but Where comes after Select and, therefore, those two variables have disappeared. After Select, all we’re left with is an anonymous type with two properties, CustomerID and OrderID. The code can be changed like so:

Dim dallas2005Orders = From cust In customers, ord In cust.Orders _
                        Select cust.CustomerID, ord.OrderID, _
                               cust.City, ord.OrderDate _
                        Where City = “Dallas” _
                              And OrderDate.Year = 2005

Note that many clauses, including Where and Order By, do not rescope the available variables.

Sorting with Order By

Order By is one of the fundamental clauses of any query. Order By sorts the results of your query by the specified field or fields:

Order By <query variable>

This is Order By at its simplest. Without any order direction specified, it’s assumed that you want the results in ascending order. Here’s the clause in action:

Dim nums = From num In numbers _
           Order By num _
           Select num

The next variant of Order By is almost identical to the first. It just assumes that you want to Order By a field inside a variable, rather than the variable itself:

Order By <field1>, <field2>, ...

Dim customerInfos1 = From cust In customers, ord In cust.Orders _
                     Order By cust.City _
                     Select Customer = cust, Order = ord

Dim customerInfos2 = From cust In customers, ord In cust.Orders _
                     Order By cust.City, ord.OrderDate _
                     Select Customer = cust, Order = ord

It’s fine to select either one or more than one field. Also, because Order By does not rescope the available variables, the following Select clause can use any variables available from the previous scope:

Order By <var1> [Ascending/Descending], _
         <var2> [Ascending/Descending], ...

Dim customerInfos = From cust In customers, ord In cust.Orders _
                    Order By cust.City Descending, ord.OrderDate _
                    Select Customer = cust, Order = ord

This is a powerful use of Order By. We have omitted the direction from the second column but included it on the first, which means we will sort first by city in descending order and then by order date in ascending order.

Of course, you can combine Order By with Where, so let’s finish with a couple of examples:

Dim orderedEvens = From num In numbers _
                   Where num Mod 2 = 0 _
                   Order By num _
                   Select num

Dim seattleCustomers = From cust In customers, ord In cust.Orders _
                       Order By cust.City Descending, ord.OrderDate _
                       Where City = “Dallas” _
                                     And OrderDate.Year = 2005 _
                       Select Name = cust.LastName & “, “ _
                                     cust.FirstName, ord.OrderDate

More Clauses

Discussing the other available clauses is a topic for another article, but many of the standard querying capabilities are supported, usually as clauses like the ones covered in this article. For example, joins, aggregations, and grouping are supported. You’ve seen implicit joins in examples above, but explicit joins can be executed using the Join keyword. Aggregating information is supported with the Aggregate clause, and grouping is supported with a combination of the Group By clause and pre-supplied aggregation methods such as Sum, or Average (though you can always create and consume your own as well). And standard query capabilities such as inserting, updating, and deleting are supported, just not directly from the query syntax.

Summary

LINQ provides basic query capabilities, including specifying a source (From), identifying data to return (Select), filtering (Where) and sorting (Order By). Advanced query capabilities to group, join, or aggregate your sets of data are also available. With these capabilities, supported by other features such as anonymous types or type inference, you have everything you need to write SQL-like queries directly in your Visual Basic code.

Ting Liang is a Software Design Engineer with the Microsoft Visual Basic Compiler Team. He has been involved extensively with the prototyping and implementation of the Visual Basic LINQ project.

Kit George is a Program Manager with the Microsoft Visual Basic Team. Kit helped design and deliver the Visual Basic LINQ features.


C++: An Inside Look At The Next Generation Of Visual C++

C++

An Inside Look At The Next Generation Of Visual C++

Tarek Madkour
This article is based on a prerelease version of Visual Studio code-named “Orcas.” All information herein is subject to change.
This article discusses:
  • Targeting apps to Windows Vista
  • Enhancements to MFC
  • Preparing applications for UAC
  • Marshaling native and CLR types
This article uses the following technologies:
Visual C++, Visual Studio
Contents
Prepare Your App for Windows Vista
Add Support for New Common Controls
Other MFC Enhancements
Prepare for User Account Control
Interoperate with Managed Code
Keep It Simple Using the Marshaling Library
Use STL/CLR on CLR Types
Build Code Faster
Conclusion

W
indows Vista™ exposes thousands of new native APIs that extend the Windows® platform. These APIs allow developers to create snazzier user interfaces, produce more responsive I/O operations, manage threads more effectively, and much more. The next version of Visual Studio,® code-named "Orcas," will help C++ developers take full advantage of the new capabilities offered by Windows Vista. It will do this by providing built-in support for all the new native APIs as well as additional support for the new controls and common dialogs in MFC and the dialog editors. In addition to all the native development improvements, Visual Studio "Orcas" extends the technology available to C++ developers to interoperate their native components with managed Microsoft® .NET Framework components by providing new libraries designed for bridging the gap between native and managed code. This article describes the details of these new technologies that will be available in the upcoming release of Visual Studio "Orcas."

Prepare Your App for Windows Vista

Visual Studio "Orcas" includes all the necessary headers and libraries needed to target all the versions of Windows from Windows 2000 through Windows Vista. This gives developers access to the low-level APIs exposed directly by the operating system and its components for use in C/C++ applications. In addition to providing access to the C-level APIs, Visual Studio "Orcas" extends MFC to use the new Windows Vista APIs where appropriate, and it encapsulates and exposes a subset of the APIs within the context of the MFC library.

Most of the enhancements made to MFC focus on UI. When an MFC application is recompiled with Visual Studio "Orcas," it gets an automatic face-lift for Windows Vista—that is, common dialogs will be enhanced, toolbars will follow Windows Vista themes, and window frames and title bars will sport Aero™ transparency. This face-lift happens without modifying any code—all you need to do is recompile the application. Of course, by porting the MFC application to the new version of Visual Studio, the developer gains access to the new controls and styles exposed in Windows Vista.

The updated MFC also does all the necessary work to make it easy to use common dialogs in Windows Vista. Consider the following MFC code, compiled with Visual Studio "Orcas":

    CFileDialog dlgFile(TRUE);
    dlgFile.DoModal();

Figure 1 MFC Common File Dialog on Windows Vista
Figure 1 MFC Common File Dialog on Windows Vista(Click the image for a smaller view)
Figure 1 MFC Common File Dialog on Windows Vista
Figure 1 MFC Common File Dialog on Windows Vista (Click the image for a larger view)

When executed on Windows Vista, the new fancy common file dialog will be displayed as shown in Figure 1. But on downlevel platforms, such as Windows 2000, MFC uses a fallback mechanism to show the old common file dialog, as shown in Figure 2.

Figure 2 MFC Common File Dialog on Downlevel Platforms
Figure 2 MFC Common File Dialog on Downlevel Platforms(Click the image for a smaller view)
Figure 2 MFC Common File Dialog on Downlevel Platforms
Figure 2 MFC Common File Dialog on Downlevel Platforms (Click the image for a larger view)

Note that the same code compiled with prior releases of Visual Studio and MFC would always result in the traditional common file dialog shown in Figure 2—even on Windows Vista. If you wanted to, you could override the new MFC behavior to default to always displaying the old, non-Windows Vista common dialogs. This is done by passing an extra parameter to the constructor of the CFileDialog class to set the bVistaStyle flag to FALSE.

In addition, the updated MFC classes allow developers to fall back to the underlying COM implementation of the common file dialogs in Windows Vista in order to perform more complex tasks, such as customizing the common dialogs. For example, the updated CFileDialog class has a getter that provides access to the IFileDialogCustomize interface, which allows developers to add controls to the dialog object.

Add Support for New Common Controls

Support for the new controls and styles available in Windows Vista enables developers to create great-looking, true Windows Vista applications. But Visual Studio "Orcas" also added support for some Windows XP controls and styles that were not implemented in previous releases of Visual Studio. Here’s a rundown of controls that were added to Visual Studio "Orcas":

Command Link Control A Command Link, shown in Figure 3, is a new style of button introduced in Windows Vista. It is essentially a button control with the BS_COMMANDLINK style flag set. The Command Link enables better user interfaces with buttons that actually indicate what action the button will execute along with a brief description of the action. This control is mostly used in Windows Vista Task Dialogs.

Figure 3 Command Link
Figure 3 Command Link

Split Button Control A Split Button, shown in Figure 4, is a Windows Vista button that has a default action as well as other possible actions accessible through a subtle UI gesture that displays a dropdown menu with more actions. A Split Button control is essentially a button control with the BS_SPLITBUTTON style flag set.

Figure 4 Split Button
Figure 4 Split Button

Network Address Control This control shown in Figure 5 is a Windows Vista simple edit box that validates input to restrict the input to an IPv4 address, an IPv6 address, or friendly DNS names.

Figure 5 Network Address
Figure 5 Network Address

Sys Link Control This is a Windows XP control that was not supported by prior releases of Visual Studio. It provides a convenient way to embed hypertext links in a window, as shown in Figure 6.

Figure 6 Sys Link
Figure 6 Sys Link

For each of the new supported controls, MFC also includes classes and notifications. Moreover, the Resource Editor in the IDE was updated to use the new MFC classes and controls. The Toolbox window shows all the new controls as native controls available to the MFC dialog editor. When the controls are dropped on an MFC dialog, they are rendered correctly on Windows Vista hosts; on downlevel platforms, placeholders are shown with an appropriate message indicating that the control cannot be rendered on non-Windows Vista platforms. (Figure 7 shows a pre-Windows Vista dialog while Figure 8 shows a fancier Windows Vista dialog box.) The Property Pages were also updated to show the styles and events available for the new controls. Finally, support was added to the MFC wizards to add control classes and members.

Figure 7 Dialog on Windows XP
Figure 7 Dialog on Windows XP

MFC in Visual Studio "Orcas" also extends the support for existing controls that were enhanced in Windows Vista. The following controls have extended support: TreeView, MonthCalendar, DateTimePicker, Static, Slider (TrackBar), and Progress controls. Furthermore, Windows Vista enhances many other controls using extended styles to provided smoother UI finish and richer user functionality. The styles for these controls can be set directly by sending a message to the respective control. For example, to enable rich tooltips on TreeView controls providing the ability to custom draw the contents with icon and text, you send a TVM_SETEXTENDEDSTYLE message to set the TVS_EX_RICHTOOLTIP extended style on the control.

Figure 8 Dialog Updated with Windows Vista Controls
Figure 8 Dialog Updated with Windows Vista Controls(Click the image for a smaller view)
Figure 8 Dialog Updated with Windows Vista Controls
Figure 8 Dialog Updated with Windows Vista Controls (Click the image for a larger view)

Applications wishing to conform to Windows Vista UI guidelines should leverage the new controls available in Windows Vista to create a more attractive and functional user interface.

Other MFC Enhancements

New Windows Vista UI idioms, such as optionally defaulting to hiding the application’s menu bar, have also been accounted for in Visual Studio "Orcas." This particular example makes the menu bar visible only when activated using the Alt key; otherwise the menu bar is hidden. The same technique is used by various Microsoft applications, such as Internet Explorer® 7, running on Windows Vista. To enable this behavior, the developer needs to set the appropriate flag on CFrameWnd by calling CFrameWnd::SetMenuBarVisibility(AFX_MBV_DISPLAYONFOCUS). Aero Wizards are also exposed by MFC. Now MFC wizards can be updated to use the Aero Wizard framework by setting m_psh.dwFlags |= PSH_AEROWIZARD in the CPropertySheet constructor.

Likewise, the Resource Editor has been updated to support the new Windows Vista UI guidelines. By default, new dialogs being added will conform to the UI guidelines in terms of control placement, control size, and fonts used. The Resource Editor also provides enhanced capabilities for viewing icons used in Windows Vista-based applications. It now supports viewing 32-bit color images, PNG images, large 256×56 icons, and PNG icons (as shown in Figure 9). In addition, it supports a better preview mode for icon files that contain multiple icon images.

Figure 9 Icon Support in the Resource Editor
Figure 9 Icon Support in the Resource Editor(Click the image for a smaller view)
Figure 9 Icon Support in the Resource Editor
Figure 9 Icon Support in the Resource Editor (Click the image for a larger view)

Finally, the Spy++ application was updated to monitor new messages introduced in Windows Vista. And Spy++ boasts added support for 64-bit applications in Visual Studio "Orcas."

Prepare for User Account Control

One of the most notable new security features in Windows Vista is User Account Control (UAC). With UAC, users run with least privilege, and administrative rights must be explicitly granted. This feature has very important implications. It’s now critical that developers are conscious of how their application will perform when run under UAC, which is the default on Windows Vista. In addition, it’s recommended that developers take an active role in enhancing desktop security by specifying the security level required by their application. Visual Studio "Orcas" helps developers prepare applications for UAC by removing the burden of manually generating the necessary XML manifests that UAC requires. Applications that do not have UAC manifests and that are not explicitly run with administrator privileges will run in a compatibility virtualization mode where all writes with a global impact, such as to the registry or to non-user disk locations, are redirected to per-user locations. Note, though, that the compatibility mode should only be considered a stop-gap measure since it can introduce subtle flaws in the application. We recommend that all applications designed for Windows Vista should be tested against UAC and should have an appropriate UAC manifest.

The UAC manifest can contain one of three invocation modes: As-Invoker, Highest-Available, and Require-Administrator.

As-Invoker This privilege level lets the application run with the same token as its parent process. This is the default and recommended UAC setting for all Windows Vista applications.

Require-Administrator This privilege level requires the application be run with full administrator privileges. Only members of the Administrators group on the local machine can run an application with this setting. Invoking an application with this privilege level will result in a UAC dialog requesting execution at this permission level.

Highest-Available This privilege level requires that the application be run with the highest privileges available to the current user. This often translates to a Require-Administrator privilege since so many users run as administrator.

So how do you generate a UAC manifest? Visual Studio "Orcas" exposes a new switch for LINK.EXE that enables the automatic generation of UAC XML manifest fragments. The format for the new switch, /MANIFESTUAC, is:

/MANIFESTUAC[:{NO|UAC fragment}]

The default for the switch is to generate a manifest with the As-Invoker privilege. Specifying No to the switch will generate no manifest and will cause the application to run in the compatibility mode. Specifying either requireAdministrator or highestAvailable will set the respective privileges on the generated UAC manifest.

Visual Studio "Orcas" also exposes the same options on a project’s property pages, as shown in Figure 10.

Figure 10 User Account Control Options for Visual C++ Projects
Figure 10 User Account Control Options for Visual C++ Projects(Click the image for a smaller view)
Figure 10 User Account Control Options for Visual C++ Projects
Figure 10 User Account Control Options for Visual C++ Projects (Click the image for a larger view)

By default, an As-Invoker UAC manifest is generated for all new applications created with Visual Studio "Orcas." All applications upgraded from previous versions of Visual Studio will also have UAC automatically enabled.

I should also note that Internet Explorer 7 uses the UAC features of Windows Vista to provide a more secure Protected Mode. This mode, which is the default, essentially runs components inside Internet Explorer with less than standard user privileges. This is achieved by launching components in a newly spawned instance of Internet Explorer that has lower rights. Consequently, components that need to run within the context of Internet Explorer 7, such as ActiveX® controls, need to be tested against Internet Explorer 7 Protected Mode.

Visual Studio "Orcas" adds a new option for C++ projects to enable debugging for applications under Web browsers. And the list of available debuggers now contains a Web browser debugger, which is also aware of Internet Explorer 7 Protected Mode and does all the necessary work to attach to the newly created lower privilege process.

Interoperate with Managed Code

In addition to the strong focus on native tools for C++ developers, Visual Studio "Orcas" extends its support for technologies that bridge the gap between native C++ components and .NET components that were developed using managed code. Since the introduction of the .NET Framework, increasing numbers of industry-level applications have been incorporating more managed components into their native C++ applications in areas where managed code is valuable and appropriate.

Visual Studio "Orcas" makes it easier to integrate those .NET components into existing and new native C++ code. Visual Studio now provides libraries that bring native idioms to managed code and help you to effectively write code that crosses the native-managed boundary.

Keep It Simple Using the Marshaling Library

The most fundamental requirement for mixing native and common language runtime (CLR) types is being able to marshal the data across both type systems. Fortunately, Visual Studio "Orcas" comes with a solution. It includes a new Marshaling Library that reduces the complexity of writing code that marshals types between the native world and the CLR. The Marshaling Library operations are based on existing .NET Framework routines, but reduce the complexity of working with these classes by exposing marshaling routines as members of an extensible template library.

The new Marshaling Library has built-in support for all of the different common string types used by C++ applications—such as ATL CString, BSTR, CComBSTR, char *, wchar *, std::string, std::wstring, and System::String^. And developers can add more conversions than those supported out-of-the-box. Here’s a simple example of using the Marshaling Library to convert a System::String to a char *:

#include <msclr/marshal.h>

void myfunc (String^ s)
{
    msclr::interop::marshal_context ctx;
    char *s2 = ctx.marshal_as<char*> (s);
    //...
}

Use STL/CLR on CLR Types

The container-iterator-algorithm structure of the Standard Template Library (STL) has simplified C++ development and it is now a pattern that many C++ developers are accustomed to. Visual Studio "Orcas" brings the same idioms that STL makes possible for native types to the world of CLR types using STL/CLR. This new capability has the potential to greatly improve the productivity of C++ developers by allowing the use of the same idioms in both the native and managed worlds. Additionally, this feature makes it easier to exchange data between STL containers and .NET collections.

STL/CLR provides the same containers for CLR types that STL provides for native types. The new containers are part of the cliext namespace. STL/CLR also maintains the same algorithmic complexity as STL. STL/CLR containers can hold garbage-collected CLR reference types and value types as well as both CLR and native built-in types; they cannot hold pure native types or pointers to native types.

STL/CLR also exposes iterators that work with its containers. STL/CLR iterators maintain the same hierarchy as that of STL iterators, and the STL/CLR iterators are accessed in exactly the same way. STL/CLR also exposes the same set of algorithms as STL. The algorithms can operate on STL/CLR and STL containers. The following sample shows STL/CLR containers and iterators in action:

#include <cliext/hash_map>

void myfunc ()
{
    cliext::hash_map<Int32, String^> m;
    m.insert (cliext::make_pair(5, L”five”));

    cliext::hash_map<Int32, String^>::iterator i = m.find(5);
    Console::WriteLine(“map[{0}] == {1}”, i->first, i->second);
}

Build Code Faster

Another improvement for developing C++ code that targets the CLR is that incremental builds are significantly faster. One sore point in the development of large CLR applications that have C++ components has traditionally been the need to rebuild dependencies, even when the core components have had only implementation changes and no interface changes. This was due to CLR assemblies packaging both the metadata and the implementation of a component in one binary. As a result, large C++ projects that depend on CLR assemblies would need to be rebuilt if any part of that CLR assembly changed.

The upcoming release of Visual Studio adds a new tool that creates a metadata-only assembly that captures the interface and only rebuilds dependent C++ projects when the interface assembly has changed. Changes to the implementation of an assembly do not trigger a rebuild of dependent assemblies. This can significantly reduce build times in the incremental build scenario, subsequently improving developer productivity.

While I’m on the topic of build throughput, one of my favorite features of Visual Studio "Orcas" is its ability to compile source files in parallel. This works for both native and managed code. CL.EXE provides a new switch, /MP, to enable this behavior. The syntax for the new switch is as follows:

/MP[n] use up to ‘n’ processes for compilation

Here, n essentially specifies the number of processors to be used. When the number of processors is not specified, the compiler uses the number of logical processors available on the machine. When a number is specified, exactly n processes are spawned for compilation.

The parallel compilation feature works by compiling different translation units in different processes; a single translation unit cannot be compiled across multiple processes. For example, consider the results of the following compiler invocation when run on a quad-processor machine:

CL /c /MP a.cpp b.cpp c.cpp d.cpp e.cpp f.cpp

The compiler will end up compiling four translation units in parallel with new translation units being compiled as soon as one of the four processors is freed up.

Now consider the results of the following compiler invocation run on a dual-processor machine:

CL /c /MP a.cpp

The compiler will perform a serial compilation of the single translation unit that it received since it does not perform parallel compilation within a translation unit.

Conclusion

The upcoming release of Visual Studio "Orcas" provides significantly enhanced tools for C++ developers looking to take advantage of the new features and capabilities offered by Windows Vista. In this release, thousands of new Windows Vista APIs are being made available to Win32® developers. MFC was also enhanced to make many Windows Vista features readily consumable for MFC developers. The IDE extends the Resource Editors to expose the new Windows Vista controls. And support has been added for creating UAC-aware applications.

Visual Studio "Orcas" also offers improvements for developers who need to bridge the gap between the native C++ code and the CLR. This is done with new libraries and tools that help improve developer productivity. Meanwhile, the new version of Visual Studio includes handy new features like parallel compilation to further improve the developer experience.

Tarek Madkour is a Lead Program Manager on the Visual C++ team at Microsoft. He has been working on different areas of the Visual C++ IDE and compiler for the past six years.


OFFICE UI: New VSTO Features Help You Customize Word And Outlook

OFFICE UI

New VSTO Features Help You Customize Word And Outlook

Steve Fox and Paul Stubbs
This article is based on a prerelease version of Microsoft Visual Studio Tools for the Microsoft Office System. All information herein is subject to change.
This article discusses:
  • New features in VSTO “Orcas” Beta 1
  • Using the Ribbon designer for Office documents
  • Customizing the Ribbon with XML
  • Data binding with Word content controls
  • Using VSTO to build custom Outlook form regions
This article uses the following technologies:
VSTO, Office, Visual Studio
Contents
Overview of VSTO
Customizing the Office UI with the Ribbon Designer
Customizing Advanced Ribbons Using XML
VSTO Word Content Controls
Outlook Form Regions
Wrap Up

T
he 2007 Microsoft® Office system has evolved into a robust business application platform that you can use to build and deploy a range of Office Business Applications (OBAs). OBAs are an emerging class of applications that address real-world and critical business problems. You can build them by seamlessly integrating existing Office software and services (Microsoft Office Word, Excel® Services and SharePoint® Server) with line-of-business (LOB) systems such as SAP. Couple the power and extensibility of the Office system with the upcoming release of Visual Studio® later this year (code-named "Orcas") and you have an extremely strong marriage of technology. In this article, we focus on how professional developers can use Microsoft Visual Studio Tools for the Microsoft Office System (VSTO) to build powerful custom applications against the 2007 Microsoft Office system.

Overview of VSTO

VSTO is a powerful set of tools and features that let developers extend and customize Microsoft Office applications using Visual Basic® and C#. The "Orcas" release of VSTO will bring even more power to the developer. It includes integration of managed code development with Microsoft Office and Microsoft Office SharePoint in unique and exciting ways. It also provides a Ribbon visual designer that lets you customize the Office Ribbon and program against it using a simple Windows® Forms-like programming model.

Furthermore, VSTO provides the ability to easily extend Microsoft Office Outlook® forms with a visual designer that supports the use of Windows Forms controls and Windows Presentation Foundation controls, and also supports the ability to import forms composed of Outlook built-in controls. In addition, SharePoint workflow in C# or Visual Basic is now a rapid application development (RAD) experience in Visual Studio. VSTO includes visual workflow designers and new support for quickly deploying and debugging a custom workflow on a SharePoint server, which can reduce the labor-intensive process from 15 steps and several days to a few clicks in a wizard.

VSTO also has visual designers for Word 2007 and Excel 2007 that appear inside the Visual Studio IDE. This allows the document to act like a visual design surface, which makes the creation of powerful document-based solutions easy. The ability to construct Word 2007 documents using the new Word content controls, with full support for ADO.NET-style data binding, drag and drop from the data sources window, and property grid support are all available in VSTO as well.

In this article, we focus on a subset of the above features. Specifically, we illustrate how you can customize the Office Ribbon, add Word content controls to Microsoft Office Word 2007, integrate a custom Ribbon with the content controls, and build Outlook form regions. Obviously, these are just a few of the features in Visual Studio "Orcas," but we discuss them here because they will be a part of the Beta 1 release, which should be available by the time you read this.

Customizing the Office UI with the Ribbon Designer

In VSTO "Orcas," there are two ways in which you can customize the Office Ribbon. The first is similar to Microsoft Visual Studio 2005 Tools for Office Second Edition (VSTO 2005 SE); that is, through the use of an XML class and definition you can define the elements of the Ribbon and then hook the Ribbon elements to event handlers (through callback code). Since you’re probably familiar with this method, we are going to focus more on the second method of customizing the Office Ribbon in VSTO "Orcas": the Ribbon Designer. (If you’re interested in customizing the Ribbon with VSTO 2005 SE, see Ken Getz’s Advanced Basics column in this issue of MSDN® Magazine.)

The Ribbon Designer, which includes a visual designer and an extensibility object model, makes it straightforward for developers to create, configure, and debug the Ribbon UI. As with other visual designers, you can drag and drop controls onto the design surface, set properties through the Properties window, and easily create event handlers for your new Ribbon items by double-clicking the control. Further, Office callbacks are mapped to the events on the VSTO Ribbon objects so developers write event handlers instead of callback methods. Let’s go ahead and customize a sample Ribbon for Word 2007 for illustration.

To begin, create a new Word 2007 project in either the C# or Visual Basic Office node and call it WordRibbonExample. After you create the project, Visual Studio automatically creates a number of project resource files. On the root project file (WordExample.cs), right-click and select Add New Item | Ribbon (Visual Designer). You can either provide a name for the Ribbon or accept the default name. This will create a default custom Ribbon on which you can add Ribbon elements.

The Ribbon Designer facilitates drag and drop functionality, so you can drag controls from the Office Ribbon Controls tab of the toolbox. Tabs, groups, and buttons are the Ribbon controls that are supported in the Beta 1 release of VSTO "Orcas." Supported Office applications are Word 2007, Excel 2007, PowerPoint® 2007, and Outlook 2007.

In the following example, we have added three groups (you must have groups as containers for the Ribbon controls) to the Ribbon tab—called My Custom Tab. Within each of the groups, we have added buttons and images. Figure 1 shows how to edit a control’s properties.

Figure 1 Ribbon Designer
Figure 1 Ribbon Designer(Click the image for a smaller view)
Figure 1 Ribbon Designer
Figure 1 Ribbon Designer (Click the image for a larger view)

A typical VSTO Ribbon customization consists of one or more Ribbon objects, each of which contains one or more Tab objects that contain one or more Group objects, and so on. Thus, a VSTO customization is constructed within a specific hierarchical structure of Ribbon objects. For example, the conceptual diagram in Figure 2 illustrates that you must have a tab as the highest level control. You can then add one or more group controls to the tab and one or more buttons, toggle buttons, and menus to the group. Within any given menu, though, you can only go five levels deep.

Figure 2 Control Hierarchy
Figure 2 Control Hierarchy

With a custom Ribbon now built, you can add event handlers to the Ribbon elements. You add an event handler to an individual Ribbon control basically the same way you do it with other controls: double-click the control and Visual Studio automatically adds a click event and opens the source-code view for you to enter your event handler code. For example, if you have a button named testButtonOne, you can double-click the button in the designer and the code view will open to the testButtonOne_click event where you can enter some event-handling code, like so:

private void testButtonOne_click(
    object sender, RibbonControlEventArgs e)
{
    // Simple message box to illustrate click event.
    testButtonOne.label = “New Label”;
    MessageBox.Show(testButtonOne.label);
}

Customizing Advanced Ribbons Using XML

In VSTO 2005 SE, you customized the Office Ribbon through an XML file, by selecting Ribbon Support when adding a new item to your project. This automatically created a Ribbon class, which is used to load the custom UI, and a Ribbon XML file, which defines the custom UI. Figure 3 illustrates the default XML code in the Ribbon Support item.

Figure 3 Default XML Ribbon Definition (Close Figure 3)
<customUI xmlns=”http://schemas.microsoft.com/office/2006/01/customui” onLoad=”OnLoad”>
  <ribbon>
    <tabs>
      <tab idMso=”TabAddIns”>
        <group id=”MyGroup”
               label=”My Group”>
          <toggleButton id=”toggleButton1”
                        size=”large”
                        label=”My Button”
                        screentip=”My Button Screentip”
                        onAction=”OnToggleButton1”
                        imageMso=”HappyFace” />
        </group>
      </tab>
    </tabs>
  </ribbon>
</customUI>

There are also default elements that are added to your project that help you customize the UI, such as a toggle button that calls a message box when the user clicks it, callback code that provides you with a sense for how you can build event handlers for the custom UI, and helper code.

While the new Ribbon Designer enables common Ribbon customization tasks, it does not provide support for the more advanced scenarios (such as repurposing built-in commands or working with some of the less commonly used controls). In the event that you want to customize the default Office UI or use control types not available in the designer, you can fall back to using Ribbon XML for your customization. You can either choose to start with the Ribbon (XML) item or you can use the designer as a starting point, then export to Ribbon XML and continue to define the custom UI using XML—but note that this is a one-time export. You will then need to manually define the callback signatures and port any event handler code you’ve written against the designer.

VSTO Word Content Controls

While the customization of the Ribbon enables you to create and wrap custom UI at the application level, VSTO Word content controls enable you to customize a document or template very simply to create a more structured and smarter document. Using drag and drop functionality, you can add controls to a Word 2007 template, build business logic into those controls, protect the content controls from user editing and deleting, and add simple data binding to local or enterprise databases. Because VSTO helps you to access powerful customization features (such as task pane customization), you can imagine a scenario where the document surface becomes an interface into enterprise data and the task pane (or a custom Ribbon) becomes the way in which users can populate the content controls within the Word 2007 document.

Using content controls, let’s create a simple Word template that can be built automatically using data from a customer database, and add a custom Ribbon that enables the end user to browse the data within the database. The resulting document will look like Figure 4.

Figure 4 Example Invitation Letter
Figure 4 Example Invitation Letter

To start, you must create a solution so your end users can customize the Word template by selecting the content of predefined areas—based on the data in the customer database. In Visual Studio, you create a new project and, in either Visual Basic or C#, select Office and then Word 2007 Document template. This opens a blank Word document within the Visual Studio IDE, along with the content controls in the Toolbox shown in Figure 5.

Figure 5 Word Controls
Figure 5 Word Controls

With the shell of the project, you can now add controls to the surface of the document. While adding controls is great, we also need to supply some structure to the document. To build this document template, you can either create the template within the Visual Studio environment or you can load a pre-defined template. For this example, we’ve first added some default text that will act as the invitation letter template and then added five content controls (by dragging and dropping the controls from the Toolbox to the surface of the document). The controls we added were a RichTextContentControl for the company name, company address, and addressee; and a DatePickerContentControl for the date and response date. After the controls are on the document surface, you can adjust the properties of the controls (similar to other controls within Visual Studio) through the Properties window. If you look back at Figure 4 you’ll see where we’ve added controls that, when hooked to the data browse event handlers, will allow a user to browse through the customer database.

If you’ve followed along, press F5 to build and run your solution. Because we have not hooked the controls to any existing database, there is no data to populate these fields. So close the Word document without saving, and return to the Visual Studio environment.

With the controls now placed in the document, you can reference a specific data source and bind the controls to it. For this sample, we created a very simple Microsoft Office Access™ customer database called CustomerDatabase. The database comprises one table with six columns: ID, CompanyName, CompanyAddress, ContactName, CurrentDate, and ResponseDate. Creating the reference is quite straightforward: you simply use the Data Source Configuration Wizard to bind the controls to specific records within the customer database.

To begin, click the Data menu and select Add a New Data Source. The Data Source Configuration Wizard will prompt you for a choice. Select Database and click Next. In the next dialog, click New Connection, and then in the Add Connection dialog, accept the default Data Source (which is Access) and then browse to your Access database. The wizard will then prompt you to select the tables and views that you want to associate with the project. If you created the same database that we did, select the CustomerInfo table and leave the Views option unchecked. Figure 6 illustrates this dialog.

Figure 6 Choose Your Database Objects Dialog
Figure 6 Choose Your Database Objects Dialog(Click the image for a smaller view)
Figure 6 Choose Your Database Objects Dialog
Figure 6 Choose Your Database Objects Dialog (Click the image for a larger view)

After you’ve completed this step, click Finish. Based on the information that you’ve provided, Visual Studio automatically creates a data set, table adapter, and binding source—the elements you will need to bind your controls to the data and programmatically manipulate the records in your database. For this example, Visual Studio has automatically assigned names for these as follows: customerDatabaseDataSet, customerInfoTableAdapter, and customerInfoBindingSource.

Now that you have the database connected to the project, you must bind the data to the specific controls. To do this, you use the Data binding property in the Properties window (see Figure 7). Select Text, and then select the specific database record you want to associate with each of the content controls. (Note that you can also drag the specific data elements from the Toolbox and drop them onto the control with which you want to associate the data.)

Figure 7 Data Binding Property
Figure 7 Data Binding Property(Click the image for a smaller view)
Figure 7 Data Binding Property
Figure 7 Data Binding Property (Click the image for a larger view)

With the specific records of the customer database now connected to each of the content controls, let’s customize the Ribbon and add event handlers to the new custom buttons to enable the end user to browse through data in the customer database. You can use the Ribbon Designer to create two buttons (a Next and a Previous button) and the Properties window to edit the properties of the tab, button group, and buttons, and also to map bitmap-format images to the buttons. The Figure 8 shows what this custom Ribbon might look like.

Figure 8 Data Browse Buttons
Figure 8 Data Browse Buttons

The last step is to add event handlers to each of the buttons so the end user can browse the data in the customer database. To do this, double-click each of the buttons and add code that will allow the user to move either forward or backward across the data. When you double-click a button, the code view for the Ribbon class opens and enables you to add the event handler code. The code below uses the Globals object to access the document, the binding source for the customer database, and eventually the MoveNext and MovePrevious methods.

private void button1_OnAction(
  object sender, ControlEventArgs e)
{
  Globals.ThisDocument.customerInfoBindingSource.MoveNext();
}

private void button2_OnAction(
  object sender, ControlEventArgs e)
{
  Globals.ThisDocument.customerInfoBindingSource.MovePrevious();
}

With the custom buttons now hooked into the content controls, you can click F5 to build and run the solution and test the integration of the custom Ribbon and the content controls. When you run the solution, click the Customer Data tab to view the custom Ribbon buttons you just added. When you click the buttons, as you see in Figure 9, in the upper-right-hand corner of the document, they allow you to browse through the data—loading the specific records into the content controls in the document.

Figure 9 Browsing the Customer Data in the Letter
Figure 9 Browsing the Customer Data in the Letter

This scenario is simple yet practical. It illustrates that with the help of the new Ribbon Designer and the Word content controls, you can create an automated application that can load and create personalized invitation letters for specific customers. As you learn more about how to develop against the 2007 Office system using VSTO, you’ll find ways to extend this scenario and build more sophistication into your customized applications. Now, let’s turn to Outlook form regions.

Outlook Form Regions

Outlook 2007 enables you to create form regions that you can use to display information to the user about a mail message, contact, or any other Outlook item. For example, when e-mail comes in from a customer, you can look up that customer in your system and display his order history in the form region. When you open a contact item, you can display a map of the contact’s address in the form region. In this section, you will create a form region that enables you to add notes about mail messages. The notes are entered and displayed in a form region below the message and persisted as a hidden attachment to the message (see Figure 10).

Figure 10 Outlook 2007 Form Region
Figure 10 Outlook 2007 Form Region(Click the image for a smaller view)
Figure 10 Outlook 2007 Form Region
Figure 10 Outlook 2007 Form Region (Click the image for a larger view)

To create an Outlook 2007 form region, you can again start with either a Visual Basic or a C# project. Adding a form region is as simple as adding a new Outlook Form Region item. Right-click the project, and choose Add New Item from the context menu. Choose Outlook Form Region from the Add New Item dialog and set the name as NotesFormRegion.cs. Click Add to launch the Outlook Form Region Wizard, which will guide you through the process of adding the form region. The first thing you need to choose is the type of form region to create. There are four types to choose from: Separate, Adjoining, Replacement, and Replace All.

Choose Adjoining, which will make the form region show at the bottom of the inspector. Click Next to continue. Set the name to VSTO Notes. Note that the title and description are not available for the Adjoining form region types. You can also configure how the form region is displayed in compose mode, reading mode, or the reading pane. Leave all of the defaults checked and then click Next to continue.

Choose which Outlook inspectors your form region is attached to. This example uses the mail message to show the form region only when a mail item is shown. You have now supplied all of the information needed for the wizard to create the form region.

After you click Finish, you will see a new NotesFormRegion.cs item added to your project and the Form Region Designer is opened. You can now design your form region as you would design a Windows Form, by dragging controls from the Toolbox onto the design surface. For this example, all you need to add is a single rich-text box, which is used to display the mail notes. Drag a RichTextBox control from the Toolbox onto the form region design surface. Set it to dock in the parent container so that it fills the entire form region. Set the background color to yellow so that it looks like a note. At this point you have a fully functional Outlook form region. You can press F5 to build and launch Outlook. Open a mail message to see your yellow textbox at the bottom of the message.

Now let’s write some code to make the note-taking form region functional. Right-click on the form region designer and choose View Code. In the code view, you can see that the form region derives from the Microsoft.Office.Tools.Outlook.FormRegionControl class. The FormRegionControl base class contains all of the plumbing to embed your control onto an Outlook form region. You will also notice that there are three event handlers. The FormRegionInitializing event is called when the form region is first created. The event allows you to cancel the creation of the form region, and through it you are also passed a reference to the Outlook item to which the form region is attached. The other two events, FormRegionShowing and FormRegionClosed, are called when the form region is displayed and closed. These events are where you will add your code. When the form region is closed, you want to save the text from the rich textbox as a hidden attachment in the message. In order to attach the text you must first save it to a temporary file. When you display the form region, you want to read the text from the hidden attachment and show it in the rich textbox. Again you will need to write the attachment out to a temporary file first. Figure 11 illustrates how this might be done.

Figure 11 Enabling Text Editing in Outlook 2007 Form Region (Close Figure 11)
Outlook.MailItem mailItem;
string NotesPath = “C:\\VSTO Note.txt”;
string NoteDisplayName = “VSTO Note”;

private void NotesFormRegion_FormRegionShowing(
    object sender, System.EventArgs e)
{
    mailItem = (Outlook.MailItem)this.OutlookItem;
    foreach (Outlook.Attachment attachment in mailItem.Attachments)
    {
        if (attachment.DisplayName == NoteDisplayName)
        {
            attachment.SaveAsFile(NotesPath);
            richTextBox1.Text = System.IO.File.ReadAllText(NotesPath);
        }
    }
}

private void NotesFormRegion_FormRegionClosed(object sender,
    System.EventArgs e)
{
    if (richTextBox1.Modified)
    {
        System.IO.File.WriteAllText(NotesPath, richTextBox1.Text);
        foreach (Outlook.Attachment attachment in mailItem.Attachments)
        {
            if (attachment.DisplayName == NoteDisplayName)
            {
                attachment.Delete();
            }
        }
        if (richTextBox1.TextLength > 0)
        {
            long position = 0;
            mailItem.Attachments.Add(NotesPath,
                Outlook.OlAttachmentType.olEmbeddeditem, position,
                NoteDisplayName);
        }
        mailItem.Save();
    }
}

After you’ve completed all of the steps above, press F5 and run your project. This will build your project and invoke Outlook 2007. When Outlook starts, open a message from your Inbox and type in some text to test it out!

Wrap Up

This article showed you how to customize the Office Ribbon, add content controls to a Word document, integrate a custom Ribbon with the Word document, and create an Outlook form region. These are just some of the Office development features available to you from VSTO in the Beta 1 release of Visual Studio "Orcas."

Steve Fox is a Program Manager with the Visual Studio Tools for Office (VSTO) team. He divides his time working externally with VSTO customers, providing VSTO training, delivering VSTO sessions at conferences and working internally with the development team to help evolve the VSTO product.

Paul Stubbs is a Program Manager with the Visual Studio Tools for Office and Visual Tools for Applications team. He coauthored VSTO for Mere Mortals with Kathleen McGrath (Addison-Wesley, 2006). He has spoken at TechEd and TechReady and participates in the developer community on the Microsoft forums. Read Paul’s blog at blogs.msdn.com/pstubbs.


SILVERLIGHT: Get Started Building A Deeper Experience Across The Web

SILVERLIGHT

Get Started Building A Deeper Experience Across The Web

Laurence Moroney
This article is based on a prerelease version of Silverlight. All information herein is subject to change.
This article discusses:
  • An introduction to Silverlight and XAML
  • Building a simple Silverlight app
  • Generating Silverlight apps dynamically on the server
  • Creating a XAML service
This article uses the following technologies:
Silverlight, XAML, JavaScript
Code download available at: Silverlight2007_06.exe (300KB)
Contents
Introducing Silverlight
XAML Overview
Inside the XAML
Transformations, Media, and Animations
A Simple Silverlight Application
Server Applications with Silverlight
Building a Weather App
Defining the Fade-In Animations
Creating a XAML Service
Delivering XAML to the Silverlight Front End
Summary

A
key component in the Microsoft strategy for next-generation Web development is a new technology named Microsoft® Silverlight™, formerly code-named "WPF/E." Designed to bring the Windows® Presentation Foundation experience to the Web, Silverlight provides rich, immersive content that fits seamlessly with the rest of the Web development environment, including ASP.NET AJAX.

To reach the Web community at large, Silverlight needs to run successfully on a number of popular operating systems and in the most popular browsers. As a result, the first release supports the Firefox and Safari browsers running on Mac OS X as well as Firefox and Internet Explorer® running on Windows. More operating systems and browsers should be supported as the product evolves. In addition to these capabilities, Silverlight is also completely self-contained and has no dependencies on other products such as Windows Media® Player for video playback or the Microsoft .NET Framework 3.0 for XAML parsing.

In this article, you get a high-level overview of the architecture of Silverlight, and get hands-on experience building several Silverlight applications, starting with a basic Hello World application. I then walk you through building a simple media player. You also see how, while Silverlight is a client-side technology, it fits into a larger server-oriented picture, including the ability to be used on servers running PHP or Java.

Introducing Silverlight

At its core, Silverlight is a browser plug-in that renders XAML, exposing its internal Document Object Model (DOM) and event model to the browser in a scriptable manner. Thus, a designer can put together a XAML document containing graphics, animations, and timelines and a developer can hook these up to code in a page to implement functionality. As XAML is based on XML, the document defining the UI that gets downloaded to the client is text-based and thus friendly to search engines and firewalls. In addition, the XAML may be assembled and emitted at run time by a server application, providing not only a rich graphical experience, but also a highly customizable and dynamic one.

The anatomy of a simple Silverlight application, using a static XAML file defining its UI and JavaScript for event handling, is shown in Figure 1. The browser instantiates the plug-in and, as part of this process, loads the XAML file. Events within this file, such as clicking on a button, are captured by the browser and processed by JavaScript. As the DOM of the Silverlight content is exposed, the JavaScript code can also dynamically update the Silverlight contents, changing the state of the rendered content.

Figure 1 Sample App
Figure 1 Sample App

The architecture that supports the Silverlight application is shown in Figure 2. The main programming interface is the JavaScript DOM API. This allows you to respond to events raised within the Silverlight XAML (such as when content has finished loading or when an animation is complete). You can also call methods to manipulate the presentation (such as starting an animation or pausing video playback). Underneath this is the XAML parsing engine. The parser creates the in-memory XAML DOM for use by the presentation core, which handles the rendering of the graphics and animations defined by the XAML. In addition, the runtime contains the codecs necessary for playback of WMV, WMA, and MP3 multimedia content.

Figure 2 Silverlight Architecture
Figure 2 Silverlight Architecture

Finally, the runtime contains the presentation core, which manages the rendering process. This presentation runtime is built into a browser plug-in that supports several flavors of Windows as well as Mac OS X, using any of several browsers as discussed previously. The end result is a self-contained graphics and media rendering engine that may be plugged into the browser and scripted via JavaScript.

XAML Overview

XAML is an XML-based language that may be used to define graphical assets, user interfaces, behaviors, animations, and more. It was introduced by Microsoft as the markup language used in Windows Presentation Foundation, a desktop-oriented technology that is part of the .NET Framework 3.0. It was designed to help bridge the work between designers and developers in creating applications.

Traditionally, designers used one suite of tools and resources for creating an application, and developers used their own separate tools. The mismatch between toolsets had the potential to adversely affect the resulting application. Microsoft introduced the new Expression suite of tools, particularly Microsoft Expression® Design and Microsoft Expression Blend, to allow design professionals to put together graphical artifacts and user interfaces respectively, expressing the end result as XAML, which a developer can then take and build an application from.

The XAML used by the first release of Silverlight differs from that used by Windows Presentation Foundation in that the former is a Web-oriented subset of the full XAML available for the desktop. As such, if you are familiar with Windows Presentation Foundation XAML, you’ll notice some things missing such as the <Window> tag, page resources, data binding, and the rich control model.

In XAML you define elements using XML tags. At the root level of every Silverlight document is a Canvas tag, which defines the space on which UI elements will be drawn. This Canvas should contain the XML namespace declarations that Silverlight needs.

<Canvas
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”>
</Canvas>

A Canvas can have one or more children, including child Canvases that can create their own children. Children of a Canvas have relative positions to their parent Canvas, not to the root Canvas. Here’s an example of a Canvas containing a Rectangle, and the rectangle is placed 25 pixels from the top-left corner of its parent.

<Canvas Width=”250” Height=”200”>
  <Rectangle Canvas.Top=”25” Canvas.Left=”25”
       Width=”200” Height=”150” Fill=”Black” />
</Canvas>

Inside the XAML

Silverlight XAML supports a number of shapes that can be orchestrated into complex objects. The basic supported shapes are Rectangle, Ellipse, Line, Polygon, PolyLine, and Path. Most of these are self-explanatory. PolyLine allows you to define a series of joined line segments. Path allows you to define a nonlinear path (like a scribble) across the Canvas.

Brushes determine how objects are painted on the screen. Their contents are painted using a Fill and their outlines are painted using a Stroke. There are solid-color brushes, gradient brushes, and image brushes. Solid color brushes are implemented using either a fixed color on the fill attribute (such as Fill="Black" used in the previous example) or using SolidColorBrush as an attached property like this:

   <Rectangle Canvas.Top=”25” Canvas.Left=”25”
          Width=”200” Height=”150”>
      <Rectangle.Fill>
         <SolidColorBrush Color=”Black” />
      </Rectangle.Fill>
   <Rectangle>

Colors can be defined by name (141 named colors are supported) or by hex RGB definition.

Gradient brushes are implemented by defining a gradient range and a number of gradient stops across a normalized space. For example, say you wanted a linear gradient to move from right to left, changing from black to white through the shades of gray as you go. You would specify a gradient stop at 0 (the beginning of a normalized line) that is black, and a gradient stop at 1 (the end of a normalized line) that is white. Silverlight will then paint the gradient for you. Gradients may also be painted in two dimensional space, with a normalized rectangle defining the space (0,0 is top-left and 1,1 is bottom-right). So to define a rectangle filled with a 2D gradient with red in the top-left, black in the bottom-right, and a smooth gradient between them, you would use XAML like this:

<Rectangle Width=”200” Height=”150” >
  <Rectangle.Fill>
    <LinearGradientBrush StartPoint=”0,0” EndPoint=”1,1”>
      <LinearGradientBrush.GradientStops>
        <GradientStop Color=”Red” Offset=”0” />
        <GradientStop Color=”Black” Offset=”1” />
      </LinearGradientBrush.GradientStops>
    </LinearGradientBrush>
  </Rectangle.Fill>
</Rectangle>

Objects may also be painted using ImageBrushes, and the image will be clipped or stretched as appropriate. So, for example, you can specify to fill an Ellipse using an ImageBrush with XAML like this:

<Ellipse Width=”200” Height=”75” >
  <Ellipse.Fill>
    <ImageBrush ImageSource=”http://.../logo.jpg” />
  </Ellipse.Fill>
</Ellipse>

Text can be rendered in XAML using the TextBlock tag. This gives you control over aspects of the text such as content, font, size, wrapping and more. Here are some examples:

<TextBlock>Hello</TextBlock>

<TextBlock FontSize=”18”>Hello</TextBlock>

<TextBlock FontFamily=”Courier New”>Hello</TextBlock>

<TextBlock TextWrapping=”Wrap” Width=”100”>
  Hello there, how are you?
</TextBlock>

In addition, Silverlight supports keyboard events that can be used to implement text input. You can define a keyboard event (KeyDown or KeyUp) on the root element, and use the event arguments to derive which key was pressed.

Transformations, Media, and Animations

XAML allows you define a number of transformations on objects. RotationTransform rotates the element through a defined number of degrees, ScaleTransform can be used to stretch or shrink an object, SkewTransform skews it in a defined direction by a defined amount, TranslateTransform moves an object according to a defined vector, and MatrixTransform can combine all of the above.

Transformations may be grouped so that an object has a number of transformations placed on it by defining those transformations as part of the group. Figure 3 shows an example. In this case, there are three Ellipses in the Canvas. As the transform is defined at the Canvas level, each one of these Ellipses will be rotated and scaled.

Figure 3 Multiple Transformations (Close Figure 3)
<Canvas xmlns=”...” xmlns:x=”...”>
  <Canvas.RenderTransform>
    <TransformGroup>
      <RotateTransform Angle=”-45” CenterX=”50” CenterY=”50”/>
      <ScaleTransform ScaleX=”1.5” ScaleY=”2” />
    </TransformGroup>
  </Canvas.RenderTransform>
  <Ellipse Width=”100” Height=”100” Fill=”Yellow” />
  <Ellipse Canvas.Top=”25” Canvas.Left=”25” Width=”10” Height=”10”
           Fill=”Black” />
  <Ellipse Canvas.Top=”25” Canvas.Left=”65” Width=”10” Height=”10”
           Fill=”Black” />
</Canvas>

Audio and video content is controlled using the MediaElement tag. This tag takes a source attribute that points to the media to be played. An object defined using this tag provides many methods and events that control media playback. Later in this article, you’ll see an example of a simple media player that is built using XAML, with JavaScript enabling the user interaction (play, pause, stop, and so on). Defining a MediaElement is straightforward:

<Canvas xmlns=”...” xmlns:x=”...”>
   <MediaElement Source=”xbox.wmv” />
</Canvas>

Animations in XAML are implemented by defining how properties should be changed over time via a timeline. Animation definitions are contained within a Storyboard. There are a number of different types of animation, including DoubleAnimation, which changes numeric properties; ColorAnimation, which changes colors and brushes; and PointAnimation, which changes two-dimensional values. These animations can either be linear or keyframe based. In the case of a linear animation, the animation changes smoothly along the defined timeline. With a keyframe-based animation, the animation can move between discrete values along the way.

Figure 4 shows an example of a simple animation that changes the width of a circle over time. The AutoReverse property is set to True, instructing the engine to continue the animation, reversing along the way so the circle goes from wide to narrow to wide instead of repeating a wide to narrow transformation.

Figure 4 A Simple Animation (Close Figure 4)
<Canvas xmlns=”...” xmlns:x=”...”>
  <Canvas.Triggers>
    <EventTrigger RoutedEvent=”Canvas.Loaded”>
      <BeginStoryboard>
        <Storyboard>
          <DoubleAnimation
            Storyboard.TargetName=”theCircle”
            Storyboard.TargetProperty=”Width”
            From=”200” To=”1” Duration=”0:0:2”
            AutoReverse=”True”/>
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Canvas.Triggers>
  <Ellipse x:Name=”theCircle” Width=”100” Height=”100” Fill=”Yellow” />
</Canvas>

A Simple Silverlight Application

With Visual Studio 2005, you can use an application template for Silverlight (part of the Silverlight SDK download) that allows you to easily build applications for Silverlight. For this article I used the Visual Web Developer™ Express application, which is available for download from MSDN®.

This example is a very simple media player that plays WMV video content. It uses a Channel 9 video (channel9.msdn.com), but you can change this to play whatever video you prefer. The XAML is pretty straightforward and can be seen in its entirety in Figure 5. For the sake of simplicity, TextBlock elements are used for the Play, Stop, and Pause buttons.

Figure 5 A Simple Media Player (Close Figure 5)
<Canvas
    xmlns=”http://schemas.microsoft.com/client/2007”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    Loaded=”javascript:root_Loaded”>

    <MediaElement
        x:Name=”mPlayer” Width=”640” Height=”400”
        Source=”http://download.microsoft.com/download/6/5/2/6527a090-aef4-4ed2-8d7e-8946e94257a0/WM_IN_Sumit_Chauhan.wmv” />

    <TextBlock Canvas.Top =”0” Text=”Play”
        MouseLeftButtonDown=”javascript:PlayVideo”>
    </TextBlock>
    <TextBlock Canvas.Top =”16” Text=”Stop”
        MouseLeftButtonDown=”javascript:StopVideo”>
    </TextBlock>
    <TextBlock Canvas.Top =”32” Text=”Pause”
        MouseLeftButtonDown=”javascript:PauseVideo”>
    </TextBlock>
</Canvas>

The MediaElement tag is used to define the video. It is given a name, mPlayer, and has its dimensions set to 640×400. The Source tag is pointed at a WMV video that’s hosted as a download on the Microsoft Web site.

Next are the three TextBlock elements that are used to define the Play, Stop, and Pause buttons. These buttons are spaced vertically in the top left-hand side of the canvas. Note that Canvas.Left isn’t being set, and it defaults to 0. These TextBlock elements define a JavaScript event handler to fire when they are clicked (MouseLeftButtonDown). When the mouse button is pressed on them, Silverlight passes the event up to the browser, where it is captured and handled by JavaScript.

Let’s now put together an application that uses this XAML and implements the JavaScript event handlers using Visual Web Developer Express. For this to work, you’ll need the silverlight.js file, which you can get from the SDK download for Silverlight or from any of the Silverlight samples in the sample download site on MSDN.

Using Visual Web Developer Express, create a new Web site. Within this site, create a folder called \js and add the silverlight.js file to it. Right-click the \js folder and select Add New Item. Select JScript File in the new file dialog, and call it eventhandlers.js.

Next, right-click on the project in Solution Explorer, click Add New Item, and create a new XML file called videoplayer.xaml. Populate it with the XAML listed in Figure 5.

Finally, add a new HTML page to the project and call it Default.htm. Edit the HTML page so that it loads the JavaScript libraries and uses them to create an instance of the Silverlight player that will render the XAML content. Figure 6 shows the full source code for Default.htm.

Figure 6 Default.htm Loads the Media Player (Close Figure 6)
<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN”
    “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>

<html xmlns=”http://www.w3.org/1999/xhtml” >
<head>
    <title>Untitled Page</title>
    <script type=”text/javascript” src=”js/silverlight.js”></script>
    <script type=”text/javascript” src=”js/eventhandlers.js”></script>
</head>
<body>
    <form>
         <div id=”AgControl1Host” style=”background:#FFFFFF”>
            <script type=”text/javascript”>
                Sys.Silverlight.createObjectEx(
                    {source: ‘videoplayer.xml’,
                     parentElement:’AgControl1Host’,
                     Id:’AgControl1’,
                     properties:{
                         width:’1024’,
                         height:’530’,
                         background:’white’,
                         isWindowless:’false’,
                         framerate:’24’,
                         version:’0.8.5’},
                     events:{onError:null, onLoad:null,
                         onNotInstalled:null},
                     context:null});
            </script>
         </div>
    </form>
</body>
</html>

The script tags at the top of the page import your JavaScript libraries, silverlight.js and eventhandlers.js. The silverlight.js library manages the downloading and instantiation of the Silverlight plug-in. It has the browser and OS abstractions so that you don’t need to worry about them; you can just implement a new Silverlight control with the Sys.Silverlight.createObjectEx function. The Silverlight plug-in itself should be within a DIV block that is given an ID. In this case, the DIV containing the plug-in is called AgControl1Host. You’ll use this ID when instantiating the Silverlight content.

The Silverlight plug-in is instantiated by creating a new instance of the Silverlight control, passing it a bunch of parameters to construct it. See the Silverlight SDK for more information on these parameters.

The final step is to implement the JavaScript functions that handle the events raised by clicking on the Play, Stop, and Pause buttons. This code should be put in eventhandlers.js.

function PlayVideo(sender,args)
{
   var mPlayer = sender.findName(“mPlayer”);
   mPlayer.Play();
}
function PauseVideo(sender,args)
{
   var mPlayer = sender.findName(“mPlayer”);
   mPlayer.Pause();
}
function StopVideo(sender,args)
{
   var mPlayer = sender.findName(“mPlayer”);
   mPlayer.Stop();
}

In the XAML, the Stop TextBlock defined the MouseLeftButtonDown attribute to point to JavaScript:StopVideo. You implement the function by creating a JavaScript function of the same name that takes sender and args parameters. You can use the findName function in Silverlight to find an element within its DOM that matches the named parameter. Therefore, to stop the video from playing, you use the findName API to find the MediaElement. In the XAML, this element was called mPlayer. The findName API will return a reference to this object that you can load into a JavaScript var. Now that you have a reference, you can simply call its Stop method to stop the video playing. The Play and Pause functionality is similar.

And that’s it! A simple video player put together with Silverlight in no time flat. Run your application and you’ll see something like Figure 7.

Figure 7 The Simple Video Player
Figure 7 The Simple Video Player(Click the image for a smaller view)
Figure 7 The Simple Video Player
Figure 7 The Simple Video Player (Click the image for a larger view)

This is all very nice if you want to hardcode the URI of the video into your XAML, but that’s not really the most feasible use case. Let’s take a look at changing this from a static HTML page into an active Web form that can accept the video as a parameter.

The first step is to remove the reference to the media file from the XAML. Change the MediaElement in your XAML to remove the Source attribute. It should look like this when you’re finished:

<MediaElement x:Name=”mPlayer” Width=”640” Height=”400”/>

Next, create a new Web Form (ASPX) named VideoPlayer.aspx in your application. This form should instantiate a Silverlight control sourced with videoplayer.xaml, as was done in Figure 6. In addition, it should contain the following JavaScript code:

<script type=”text/javascript”>
  function root_Loaded(sender,args) {
    var mPlayer = sender.findName(“mPlayer”);
    mPlayer.Source =
      “<% Response.Write(Request.Params[“VideoURI”]); %>”;
    mPlayer.Play();
}
</script>

The XAML contains an event declaration for an event to fire when the Canvas loads. This is captured by the root_Loaded event handler in JavaScript. This function uses ASP.NET to pull the VideoURI off the HTTP request on the server, and uses the VideoURI to fill in the JavaScript function, thus setting the video source. When the page executes, the video (at the VideoURI parameter) will be rendered by Silverlight. Be aware that this very simple example doesn’t have any error handling or filtering, which you should add if you want this type of functionality in your apps.

Note that when you call this page with a URI like this:

http://localhost/MSDN1/
VideoPlayer.aspx?videouri=xbox.wmv

ASP.NET will emit a page containing a JavaScript block that looks like this:

   <script type=”text/javascript”>
     function root_Loaded(sender,args)
     {
       var mPlayer =
         sender.findName(“mPlayer”);
       mPlayer.Source = “xbox.wmv”;
       mPlayer.Play();
     }
   </script>

This JavaScript then loads the xbox.wmv file into the MediaElement and Silverlight plays the video. This shows that using server-side active pages (ASPX, PHP, or Java) can greatly enhance your Silverlight end user experiences by making them much more flexible.

Server Applications with Silverlight

It’s hardly fair to think of Silverlight solely as a client technology meant to embed rich content in the browser. Such rich content is already possible through use of closed plug-ins built as Java Applets, ActiveX® controls, or Flash applications. By contrast, Silverlight is an open technology in the sense that the UI is defined in text-based XAML and the programmability is achieved using JavaScript. This allows developers to easily build applications that interact with back-end servers.

For example, consider a weather application. If you wanted to produce a weather application on the client-side, you could build a Java applet, ActiveX control, or Flash application that consumes a Web service and then deploy this applet to the client. However, this increases the communication needs between the client and the server. What if this data source is a paid-subscription service? The person deploying the application has to take on the job of authenticating licenses for any clients accessing the data service, and that takes time away from building domain-specific business logic.

However, if the application can be assembled on the server side and all data is passed back to the client, then this work is alleviated. By using a tool such as Expression Blend, a template for the application’s user interface can be assembled and expressed as XAML. The developer, at runtime, then takes the relevant data and inserts it into the template, returning the finished XAML to the client that renders it. No logic or connectivity is required on the client side (other than the initial connectivity to the server that serves it up, of course), and the deployment and management of the client is thus kept relatively easy.

Building a Weather App

In this section, you’ll see how to build a simple XAML file that implements a weather application (see Figure 8). The finished weather application will render three days worth of temperature forecasts for a specific ZIP Code that is passed to the application as a parameter on its address within the browser. It will also render a graphical representation of the weather, the correct dates, the name of the location specified by the ZIP Code, and animations of the weather data. The complete listing of the XAML for the weather application is available from the MSDN Magazine Web site.

Figure 8 Silverlight Weather App
Figure 8 Silverlight Weather App(Click the image for a smaller view)
Figure 8 Silverlight Weather App
Figure 8 Silverlight Weather App (Click the image for a larger view)

At the top of the document is the opening Canvas tag. This is a basic container on which you will draw your objects. You aren’t limited to one Canvas, and it makes sense for you to group your related elements together using a Canvas. So, for example, each day contains a graphic, a high temperature, a low temperature, and various labels, and all of this is collected into a XAML template. Figure 9 is an example of one particular day. In this case, the Canvas contains a number of TextBlocks and an Image control. This canvas is given a name cnv2 and is placed on screen with an opacity of 0 (meaning that it is initially invisible). Its height, width, left, and top are set to put it in a specific location on the screen.

Figure 9 Elements Collected in a Day (Close Figure 9)
<Canvas x:Name=”cnv2” Width=”624” Height=”101” Opacity=”0”
        Canvas.Left=”8” Canvas.Top=”165”>

<TextBlock x:Name=”lblDate2” Width=”513” Height=”26” Canvas.Left=”97” FontFamily=”Tahoma” FontSize=”20” FontWeight=”Bold” Foreground=”#FF24AB46” Text=”Wednesday, December 18, 2006” TextWrapping=”Wrap”/>

<TextBlock x:Name=”lblHigh2” Width=”70” Height=”59” Canvas.Left=”112” Canvas.Top=”30” FontFamily=”Tahoma” FontSize=”48” FontWeight=”Bold” Foreground=”#FFF64F12” Text=”27” TextWrapping=”Wrap”/>

<TextBlock x:Name=”lblDegHigh2” Width=”62” Height=”59” FontFamily=”Tahoma” FontSize=”48” FontWeight=”Bold” Foreground=”#FFF64F12” Text=”°F” TextWrapping=”Wrap” Canvas.Left=”172” Canvas.Top=”30”/>

<TextBlock x:Name=”lblLow2” Width=”71” Height=”59” FontFamily=”Tahoma” FontSize=”48” FontWeight=”Bold” Foreground=”#FF444DB5” Text=”27” TextWrapping=”Wrap” Canvas.Left=”265” Canvas.Top=”30”/>

<TextBlock x:Name=”lblDegLow2” Width=”67” Height=”59” FontFamily=”Tahoma” FontSize=”48” FontWeight=”Bold” Foreground=”#FF444DB5” Text=”°F” TextWrapping=”Wrap” Canvas.Left=”327” Canvas.Top=”30”/>

<Image x:Name=”imgWeather2” Width=”97” Height=”87” Canvas.Top=”4” Source=”http://www.nws.noaa.gov/weather/images/fcicons/ra80.jpg” Stretch=”Uniform”/>

</Canvas>

The TextBlocks are used to render text in Silverlight. There are a number of them here, some used as hardcoded labels such as the "°F" labels that are stored in the TextBlocks called lblDegHigh2 and lblDegLow2. See the x:Name attribute to find the name.

The TextBlocks called lblDate2, lblHigh2, and lblLow2 contain placeholder information—right now they are hardcoded for specific data and temperature, but your ASP.NET code will override these with the live date and temperatures for the specified ZIP Code.

Defining the Fade-In Animations

A Canvas node supports triggers that fire off actions in response to an event. In this case, the Canvas.Triggers are specified to occur when the page is loaded as shown here:

<Canvas.Triggers>
  <EventTrigger RoutedEvent=”FrameworkElement.Loaded”>
    <BeginStoryboard>
...

Thus, when the page gets loaded, the triggers will fire and an animation Storyboard will begin. The animation Storyboard defines the animation behaviors. Here is the animation that is used on the Canvas (containing the text placeholders and graphic for a specific day’s weather) called cnv1:

<DoubleAnimationUsingKeyFrames BeginTime=”00:00:00”
    Storyboard.TargetName=”cnv1”
    Storyboard.TargetProperty=”(UIElement.Opacity)”>
  <SplineDoubleKeyFrame KeyTime=”00:00:00” Value=”0”/>
  <SplineDoubleKeyFrame KeyTime=”00:00:01” Value=”1”/>
</DoubleAnimationUsingKeyFrames>

The animation type is a DoubleAnimationUsingKeyFrames. You will be amending the opacity (defined by UIElement.Opacity) of the Canvas and its children from 0 to 1, causing it to fade in over a period of time. The time is defined by the keyframes associated with the animation.

When the XAML renders, the Storyboard will begin, causing the first element to fade in from opacity 0 to opacity 1 between the beginning of the rendering (time of 0 seconds) until 1 second later. Look at the XAML and you’ll see that the other Canvasses fade in between 1 and 2 seconds, and between 2 and 3 seconds respectively, giving an effect of each one appearing in turn.

XAML is a form of valid XML, so elements can be named with attributes, and from that naming they can be addressed. So, for example, the placeholder element for the high temperature for Day One looks like this:

<TextBlock x:Name=”lblHigh1” Width=”67” Height=”59”
        Canvas.Left=”112” Canvas.Top=”30” FontFamily=”Tahoma”
        FontSize=”48” FontWeight=”Bold” Foreground=”#FFF64F12”
        Text=”27” TextWrapping=”Wrap”/>

This TextBlock is given the name lblHigh1. The service (which you will see in the next step) uses an XMLDocument to load the template XAML, find the node with this name, and change its text property with C# code that looks like this:

string xpath = “//d:TextBlock[@x:Name=’lblHigh1’]”;
xNode = xmlDoc.SelectSingleNode(xpath, mng);
string high = myResults[n - 1].MaxTemperatureF;
xNode.Attributes[“Text”].Value = high;

Creating a XAML Service

When you use Silverlight in a Web page, one of the parameters that the control takes is the location of the XAML to render. In this case, the XAML is dynamic, changing based on the ZIP Code for which the weather is being requested. As such, you’ll create a Web application that generates and returns the XAML, and the page that renders it will refer to this page. This is the first step in building this Silverlight application.

Using Visual Studio®, create a new Web Project. Add a \js folder containing silverlight.js, just like you did with the earlier video player. Add a new Web reference to the project, referencing the Web service defined at:

http://www.webservicex.net/WeatherForecast.asmx?WSDL

Give the service the name weatherservice. Add a new Web Form to the project and call the Web Form XAMLSource.aspx. Select the Add Code as a Separate Page option. Make sure that the top of the code page has the following set of using directives:

using System;
using System.Data;
using System.Configuration;
using System.Collections;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
using System.Xml;
using System.Xml.XPath;

For the rest of this section, you’ll be adding code to the Page_Load event handler on XAMLSource.aspx. First, load the template (called Scene.xaml) into an XMLDocument object:

XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(Server.MapPath(“Scene.xaml”));

Next, set up the read parameter. You’ll be calling the page using a query string parameter specifying the required ZIP Code:

http://server/XamlSource.aspx?ZIP=<something>

To retrieve the parameter, use this code:

string strZip = “98052”;
if (Request.Params[“ZIP”] != null)
    strZip = Request.Params[“ZIP”];

This initiates the string that will hold the parameterized ZIP locally, and sets it to a default if it isn’t passed in as part of the HTTP Request. Next, create the call to the Web service that serves the weather information:

weatherservice.WeatherForecast myWeather = new
    weatherservice.WeatherForecast();

weatherservice.WeatherForecasts myForecast =
    myWeather.GetWeatherByZipCode(strZip);

weatherservice.WeatherData[] myResults = myForecast.Details;

This code creates an instance of the Web service proxy (created for you by Visual Studio when adding a reference to the service’s WSDL) that it calls myWeather. The Web service returns data in the WeatherForecasts data structure, so it creates an instance of this data structure (called myForecast) by calling the GetWeatherByZipCode method on the service, passing it a string. The Details member of the forecast object is an array of WeatherData types, so the variable myResults is set up as an instance of this.

The next step is to initialize the XMLDocument for searching using XPath. As the XML uses the XAML schema as a namespace for some of the elements (prefixed with an x, such as x:Name), you’ll need to define a namespace for all of the elements in the page, even those that aren’t prefixed. For example, you cannot search for nodes matching the XPath //TextBlock, but you can specify the default namespace to have a new prefix (in this case d for dummy), and search for the default nodes using this prefix, so //d:TextBlock will work. Here’s the code:

NameTable myn = new NameTable();
XmlNamespaceManager mng = new XmlNamespaceManager(new NameTable());
mng.AddNamespace(“d”,
    “http://schemas.microsoft.com/winfx/2006/xaml/presentation”);
mng.AddNamespace(“x”, “http://schemas.microsoft.com/winfx/2006/xaml”);

Now that you can use XPath, you can find the node that contains the town name and change its Text property to be the name of the town that the service identified at that ZIP Code.

XmlNode xNode = xmlDoc.SelectSingleNode(
    “//d:TextBlock[@x:Name=’lblTownName’]”, mng);

xNode.Attributes[“Text”].Value =
    myForecast.PlaceName + “,” + myForecast.StateCode;

Similarly, you can loop through the next three days, find the placeholder nodes that contain the desired locations for their values, and prune the correct values into the XML with the code shown in Figure 10.

Figure 10 Updating Weather Information (Close Figure 10)
string strXPath = “”;
string strDate = “”;
string strHigh = “”;
string strLow = “”;
string strWeatherURL = “”;

for (int n = 1; n <= 3; n++)
{
  strXPath = “//d:TextBlock[@x:Name=’lblDate” + n.ToString() + “’]”;
  xNode = xmlDoc.SelectSingleNode(strXPath, mng);
  strDate = myResults[n - 1].Day;
  xNode.Attributes[“Text”].Value = strDate;

  strXPath = “//d:TextBlock[@x:Name=’lblHigh” + n.ToString() + “’]”;
  xNode = xmlDoc.SelectSingleNode(strXPath, mng);
  strHigh = myResults[n - 1].MaxTemperatureF;
  xNode.Attributes[“Text”].Value = strHigh;

  strXPath = “//d:TextBlock[@x:Name=’lblLow” + n.ToString() + “’]”;
  xNode = xmlDoc.SelectSingleNode(strXPath, mng);
  strLow = myResults[n - 1].MinTemperatureF;
  xNode.Attributes[“Text”].Value = strLow;

  strXPath = “//d:Image[@x:Name=’imgWeather” + n.ToString() + “’]”;
  xNode = xmlDoc.SelectSingleNode(strXPath, mng);
  strWeatherURL = myResults[n - 1].WeatherImage;
  xNode.Attributes[“Source”].Value = strWeatherURL;
}

Now that you’ve fully populated your XML document with the template and the correct data, the last step is to return it to the caller. You do this by setting the MIME type and writing the contents of the XMLDocument object directly to the response stream, like so:

Response.ContentType = “text/xml”;
Response.Write(xmlDoc.InnerXml);

Next, remove the placeholder HTML that Visual Studio gave you in your ASPX page. Remove everything except the first line from the ASPX file. It should look something like the following code when you’re finished:

<%@ Page Language=”C#” AutoEventWireup=”true”
         CodeFile=”XamlSource.aspx.cs” Inherits=”_Default” %>

The final step is to add a new XML file to your solution and call it Scene.xaml. You will have seen this file being referred to in the code earlier. Get the full weather XAML and paste it into this file. You can now execute the page, and you should see the XAML returned by the XAMLSource.aspx page in your browser.

Delivering XAML to the Silverlight Front End

In the previous steps, you built the template XAML, and then built a Web form that takes this XAML, consumes a Web service, and puts the results of the Web service call into the XAML at the correct place. The next step is to serve up a page that contains the Silverlight control and points it at this XAML service.

Open up the Web site that you used in the previous sections and add a new Web form called Default.aspx. Modify the HTML Code on Default.aspx to look like Figure 11.

Figure 11 More Changes to Default.aspx (Close Figure 11)
<%@ Page Language=”C#” AutoEventWireup=”true”
    CodeFile=”Default.aspx.cs” Inherits=”_Default” %>

<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN”
 “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>

<html xmlns=”http://www.w3.org/1999/xhtml”>
<head>
    <title>Weather</title>
    <meta http-equiv=’Content-Type’
          content=”text/html; charset=windows-1252”/>
    <script type=”text/javascript” src=”js/silverlight.js”></script>
</head>
<body>
<div id=”AgControl1Host” style=”background:#FFFFFF”>
    <script type=”text/javascript”>
        Sys.Silverlight.createObjectEx(
            {source: ‘WeatherSite/XamlSource.aspx?ZIP=<%
                        if (Request.Params[“ZIP”] == null)
                            Response.Write(“98052”);
                        else
                           Response.Write(Request.Params[“ZIP”]);
              %>’, parentElement:’AgControl1Host’, Id:’AgControl1’,
              properties:{width:’1024’, height:’530’,
                  background:’white’, isWindowless:’false’,
                  framerate:’24’, version:’0.8.5’},
              events:{onError:null, onLoad:null, onNotInstalled:null},
                  context:null});
    </script>
</div>
</body>
</html>

The important part of this is in the call to Silverlight.createObjectEx, where the static videoplayer.xaml has been replaced with a call to the XAML source application, passing it the parameter that this Default.aspx page received for the ZIP Code:

“WeatherSite/XamlSource.aspx?ZIP=<%
    if (Request.Params[“ZIP”] == null)
        Response.Write(“98052”);
    else
        Response.Write(Request.Params[“ZIP”]);
%>”

This takes the request parameter to this page and appends it to a URL, which will call the XamlSource page that you created in the last step. This will then make a call to the XamlSource, passing the ZIP Code, and getting a XAML document back, which is then rendered by Silverlight. The results of running Default.aspx?ZIP=90210 can be seen in Figure 8.

Summary

In this article, you got a deep introduction to Silverlight. You also looked at some applications that go beyond just parroting "Hello, World," such as how XAML is used to construct user interfaces, how it can be tied to JavaScript for interactivity, and how you can move beyond a pure client paradigm into a server-based one to make your applications more flexible. Using this methodology, you saw how to create a simple video player that accepts the URI of a video stream and plays it back, making it ideal for use in scenarios such as blogs.

Additionally, you learned how to build a data-aware application, in the form of a weather monitor that has a very simple client distribution model. You simply had the Silverlight component call the server for the correct XAML and render it, forming a rich, thin client.

While ASP.NET and the Microsoft stack represent one very productive platform for building Web applications such as this, the download for this article also contains a Java version of this application. This version offers the same functionality as the ASP.NET version—consuming the Web service to get the data and then loading the XAML into an XML Document DOM. It then fishes out the appropriate nodes for the template data and replaces their values with those from the Web service before writing the finished XAML out as a URI. The JSP (not listed) is then constructed to reference the URL of this servlet to render the XAML using Silverlight.

The road is just beginning for Silverlight on the Web. With the richness of a XAML rendering engine, and the flexibility of using standard Web development technologies, Silverlight is something well worth adding to your developer toolbox. For more information, take a look at the Silverlight developer center on MSDN.

Laurence Moroney is a Silverlight technology evangelist at Microsoft. He is author of nine computing books and over 100 technology articles, including the upcoming Programming Web.NEXT from Microsoft Press. When not writing articles or code, he is usually driving the product team crazy with questions about feature support in Silverlight.


CONCURRENCY: Synchronization Primitives New To Windows Vista

CONCURRENCY

Synchronization Primitives New To Windows Vista

Robert Saccone and Alexander Taskov
Code download available at: VistaSynchronization2007_06.exe (174KB)
This article discusses:
  • Condition variables
  • Slim reader/writer lock
  • One-time initialization
  • Double-checked locking
This article uses the following technologies:
Windows Vista, C++
Contents
Condition Variables
Slim Reader/Writer Lock
One-Time Initialization
Conclusion

W
indows Vista ships with a raft of new and exciting technologies for developers, including Windows® Presentation Foundation, Windows Communication Foundation, and Windows Workflow Foundation. In fact, Windows Vista™ has introduced so many new .NET-friendly technologies that it is easy to overlook all the new features and functionalities that native C/C++ developers can put to work in their apps.

In this article, we discuss some of the new capabilities in Windows Vista that impact native C/C++ developers. Specifically, we focus on several of the new thread synchronization features introduced with the new operating system: condition variables, slim reader/writer locks, and one-time initialization.

Condition Variables

The condition variable has existed in other threading libraries for some time and has been sorely missed from the Windows SDK. Basically, a condition variable is used to synchronize a group of threads based on the result of some conditional test. While this can be done using a combination of existing synchronization constructs, the condition variable has the ability to release an acquired lock and go into a sleep state in a single atomic operation. It also provides a much clearer and less error-prone way to implement the desired behavior.

The Windows SDK for Windows Vista (available for download) exposes condition variables with the CONDITION_VARIABLE structure. You can use the InitializeConditionVariable function to set up the structure. There is no function to clean up or destroy the CONDITION_VARIABLE structure as it’s not needed by the underlying implementation.

Using the functions SleepConditionVariableCS (when a critical section is used) or SleepConditionVariableSRW (when a slim reader/writer lock is used) functions, you can have threads wait on a condition variable. These sleeping threads will be released when another thread calls WakeConditionVariable or WakeAllConditionVariable, depending on whether the calling thread wants one thread to be released or all threads waiting on the condition variable to be released.

The common producer/consumer problem represents a scenario where condition variables would come in handy. This classic example refers to a scenario where a producer generates data, placing it in a buffer, while a consumer grabs pieces of data from the buffer to be processed. The problem refers to the need to make sure the producer doesn’t try to add data to a full buffer and the consumer doesn’t try to grab data from an empty buffer. We will work through this scenario to show you how the condition variable can help.

For this example, we will create a single producer thread that posts numerical data to a shared queue. We will then create five consumer threads. Each consumer thread will remove an item from the queue and process it. When finished with the current piece of data, the consumer thread will loop around, repeating the process infinitely.

In earlier versions of Windows, the producer/consumer problem could be solved with a combination of Win32 events and a critical section. The critical section protects the shared resource from concurrent access and an event signals when the resource is available for the consumer.

In our first attempt at solving this problem, we use a Standard Template Library (STL) list of integers as the shared resource. Since the list expands dynamically, we don’t need an event to signal when the list is not full; we only need to know when it’s not empty so the consumer knows that it has something to consume. (If you were to use a fixed-size array for the shared queue, then you would need a not-full event to make sure you don’t overwrite the bounds of the buffer.) We then declare and initialize a CRITICAL_SECTION object and an auto-reset event, which is used to indicate when the list is not empty.

The producer thread shown in Figure 1 will first try to acquire the critical section and, if it succeeds, will then insert an integer value at the end of the shared list. The thread then releases the critical section and sets the not-empty event. Only one of the threads waiting on this event will be released since we are using an auto-reset event. The consumer thread shown in Figure 1 will check to see if the queue is empty. If the queue is not empty, the thread will remove an item and release the critical section. If the queue is empty, the consumer thread will go back to sleep, waiting on the not-empty event. While the first consumer thread is busy processing the item it removed from the queue, the producer can wake up another consumer thread to pick up the next piece of work to keep the queue moving.

Figure 1 Solution That Doesn’t Use Condition Variable (Close Figure 1)

Producer
unsigned _stdcall ProducerFunc(void *pParam)
{
    for (unsigned int i = 0;i < g_uiIterations;i++)
    {
        EnterCriticalSection(&g_csLock);
        // Produce work
        g_listWork.push_back(i++);
        LeaveCriticalSection(&g_csLock);
        SetEvent(g_hNotEmpty);
        Sleep(g_uiProducerDelay); // Simulating work
    }
    return 0;
}
Consumer
while (true)
{
    EnterCriticalSection(&g_csLock);

    if (g_listWork.empty())
    {
        LeaveCriticalSection(&g_csLock);
        WaitForSingleObject(g_hNotEmpty,INFINITE);
    }
    else
    {
        i = g_listWork.front();
        g_listWork.pop_front();

        LeaveCriticalSection(&g_csLock);

        wcout << L"Thread " << iThread << L" Consumed: " << i << endl;
        Sleep(g_uiConsumerDelay); // Simulating work
    }
}

Although this solution will generally work, there are some limitations to this implementation. For one, getting the synchronization objects properly initialized can be tricky. For example, we had to decide whether the producer should wake only one consumer thread or all of them when data is pushed onto the list. This is controlled by how we initialized the event. If we use an auto-reset event, only one thread will be released. However, if we want to wake all the threads, we would use a manual-reset event and have to remember to call ResetEvent at the correct time.

It’s essential to make sure that when the queue is empty, the consumer thread releases the critical section before waiting on the not-empty event. We also have to make sure we don’t mistakenly use PulseEvent in the producer thread to signal the not-empty event because this would introduce a race condition. The problem occurs if the consumer thread is preempted immediately after it releases the critical section but before it calls WaitForSingleObject and then the producer thread calls PulseEvent. PulseEvent is not sticky and will only release those threads currently waiting on the event. When the preempted thread resumes, the event will not be signaled and the wakeup will be lost.

Condition variables make it much easier to get the solution right. This approach still uses the critical section, but it replaces the not-empty event with a condition variable. We initialize the condition variable in the main function by calling InitializeConditionVariable with a pointer to our CONDITION_VARIABLE structure.

The consumer thread shown in Figure 2 enters the critical section and checks to see if the queue is empty. If so, SleepConditionVariableCS is called. This function releases the critical section and puts the thread to sleep in one atomic operation, eliminating the race condition that can occur if the consumer is preempted in between these two tasks. The SleepConditionVariableCS function also takes a timeout parameter, allowing us to do something else if we don’t want to wait too long.

Figure 2 Solution That Uses Condition Variable (Close Figure 2)

Producer

for (unsigned int i = 0;i < g_uiIterations;i++)
{
    EnterCriticalSection(&g_csLock);

    // Produce work
    g_listWork.push_back(i);

    LeaveCriticalSection(&g_csLock);

    WakeConditionVariable(&g_condNotEmpty);

    Sleep(g_uiProducerDelay);
}
Consumer
while (true)
{
    EnterCriticalSection(&g_csLock);

    while (g_listWork.empty())
    {
        SleepConditionVariableCS(&g_condNotEmpty,&g_csLock,INFINITE);
    }

    i = g_listWork.front();
    g_listWork.pop_front();

    LeaveCriticalSection(&g_csLock);

    wcout << L"Thread " << iThread << L" Consumed: " << i << endl;
    Sleep(g_uiConsumerDelay); // Simulating work
}

When the condition variable is signaled, the thread wakes up and automatically locks the critical section again before the SleepConditionVariableCS function returns. The consumer thread then loops back around and checks again to see if the queue is empty. It’s very important to test the condition again when the thread is released because its state could have been changed by another thread right before the consumer was released; in addition, the condition variable is subject to spurious wakeups so it may run before the condition has changed. We then remove the work item from the queue and release the critical section.

The producer thread shown in Figure 2 starts out by entering the critical section. Once it has the lock, the producer thread pushes a new work item onto the queue and then releases the critical section. Now it can release a consumer thread by calling WakeConditionVariable. This function releases only one thread, just like the solution that doesn’t use a condition variable. If we want to release all of the consumer threads, we would call the WakeAllConditionVariable function. The condition variable syntax makes it clear how the thread will be dispatched since the function names indicate what they will do. Using events makes this much more error-prone as the desired behavior is not specified with event-signaling functions but when the event is initialized.

If the producer thread is preempted right after it releases the critical section but before it calls the Wake function, there is an opening for another thread to modify the state of the queue before the consumer thread is released. This is why we said that the condition must be checked again by the consumer thread when it is released.

Condition variables can also be more efficient in some scenarios. The SleepConditionVariableCS and SleepConditionVariableSRW APIs try to avoid trips to kernel mode when possible. However, the example that doesn’t use condition variables will always incur at least one kernel-mode roundtrip when it calls WaitForSingleObject.

As a result, the native condition variable implementation provides many benefits over homegrown implementations without sacrificing flexibility. The resulting code is simpler and easier to read, and performance may be improved. This greatly reduces the probability of introducing many subtle bugs that can be very difficult to discover.

Slim Reader/Writer Lock

A reader/writer lock is used to protect a piece of data to which you want to allow multiple readers concurrent access, but in the case of an update, only exclusive access to the writer. Typically, these locks are best used in scenarios where the data needs to be read frequently and updated infrequently. Using reader/writer locks when appropriate should lead to increased scalability.

Reader/writer locks have been around for some time. Before the release of Windows Vista, however, if you were writing native user mode code on Windows and needed a reader/writer lock, the only choices you had were to write your own or to adapt an implementation from a textbook. Windows Vista includes a type of reader/writer lock called the slim reader/writer (SRW) lock. Let’s take a look at the feature set of this new synchronization primitive, examine the APIs that are available to use it, and compare its performance to an alternative implementation constructed using standard Win32® critical sections, semaphores, and events.

The SRW lock is designed to be fast and (as its name implies) small, while maintaining its resource efficiency as it is used. It is built on top of the Windows kernel keyed events mechanism, which was introduced to solve resource exhaustion problems that can occur when applications use large numbers of CRITICAL_SECTION objects. Some reader/writer locks are designed to favor readers over writers, and vice versa. But the SRW lock is designed to favor neither. This means that if your application requires that data updates take priority over data reads, you might want to consider a different reader/writer lock that favors writers. However, before you write your own lock, you should try out the SRW lock to see how it behaves in the context of your application.

Two other features that reader/writer locks sometimes support are the ability to acquire the lock recursively and the ability to upgrade (or downgrade) the access a thread has already been granted to lock. First, regarding recursive acquires: if the locking policy you’ve designed for your application requires that synchronization objects be acquired recursively, this is very possibly a red flag telling you to re-examine your locking policy to eliminate the recursion. This is our opinion and results from the additional overhead of executing the lock acquisition and release code multiple times and, perhaps more importantly, because ensuring that the balance of lock releases with the lock acquisitions is often difficult to prove correct.

The SRW lock doesn’t support recursive acquisition. Such support would impose additional overhead as per-thread counts would need to be tracked in order to maintain correctness. SRW locks also don’t support upgrading from shared to exclusive access, nor do they support downgrading (which is less common) from exclusive to shared access. Supporting the ability to upgrade would probably have caused unacceptable complexity and extra overhead that would impact even the common cases of shared and exclusive acquisition code in the lock. It would also require that policy be defined with respect to how to choose from waiting readers, waiting writers, and readers waiting to upgrade, which would violate the basic design goal of no favoritism.

The first step in using a slim reader/writer lock is to declare an SRWLOCK structure and initialize it using InitializeSRWLock:

VOID WINAPI InitializeSRWLock(PSRWLOCK SRWLock);

The SRW lock breaks away from the customary Win32 pattern where each object has an initialize and cleanup function. When you’re finished with an SRW lock, there is no cleanup function that needs to be called.

Once initialized, the basic APIs that are now available are:

VOID WINAPI AcquireSRWLockExclusive(PSRWLOCK SRWLock);

VOID WINAPI ReleaseSRWLockExclusive(PSRWLOCK SRWLock);

VOID WINAPI AcquireSRWLockShared(PSRWLOCK SRWLock);

VOID WINAPI ReleaseSRWLockShared(PSRWLOCK SRWLock);

The AcquireSRWLockExclusive function, as the name implies, acquires the lock exclusively for the caller. Once exclusive access is granted, all other threads requesting either type of access will be blocked until the lock is released using the complementary ReleaseSRWLockExclusive function. AcquireSRWLockShared, on the other hand, acquires the lock with shared access. In this case, other threads that also request shared access do not have to wait if the lock is un-owned or has already been acquired by another thread with shared access.

Slim reader/writer locks can be used with condition variables using the SleepConditionVariableSRW function:

BOOL WINAPI SleepConditionVariableSRW(
  PCONDITION_VARIABLE ConditionVariable,
  PSRWLOCK SRWLock,
  DWORD dwMilliseconds,
  ULONG Flags
);

SleepConditionVariableSRW releases an SRW lock and waits on the specified condition variable as one atomic operation. The function doesn’t return until the condition variable has been signaled or the timeout specified in dwMilliseconds has elapsed. If dwMilliseconds is INFINITE, the function will never timeout. If the Flags parameter specifies CONDITION_VARIABLE_LOCKMODE_SHARED, the function assumes that the SRW lock is held with shared access; otherwise exclusive access is assumed and upon successful return the lock will have been re-acquired with the specified access.

While experimenting with these APIs, we discovered some issues that warrant extra care when coding. Notice that none of the lock acquisition or release APIs are typed as returning a result. If the SRW lock is not currently owned by a thread and a call to either ReleaseSRWLockShared or ReleaseSRWLockExclusive is made, a STATUS_RESOURCE_NOT_OWNED structured exception will be thrown. This is nice because the error is surfaced in a manner that is obvious to the developer.

Earlier we mentioned that recursive acquisition of the lock is not supported. The result of an attempt to recursively acquire exclusive access again is that the second call to AcquireSRWExclusive will never return as it will be self-deadlocked with the thread remaining blocked inside, waiting for the lock to be released. You would probably need to attach a debugger to the process to see the condition and figure out what’s going on. Calling AcquireSRWShared from a thread that has already acquired shared access will also lead to a deadlock if another thread has attempted to acquire the lock exclusively in between AcquireSRWShared calls.

Extra care should also be taken in making sure that the correct acquire and release function pairs are used—mismatching AcquireSRWLockExclusive with ReleaseSRWLockShared (or vice versa) doesn’t cause any exceptions to be raised. It would have been helpful if exceptions would have been raised for these error scenarios, but detecting the errors may have imposed unwanted resource and performance overhead.

The sample code for this article includes the source for a program called ReaderWriterExample, which allows for experimentation with different reader/writer lock implementations. The lock types that the program supports are the SRW lock, two variants of a reader/writer lock that we implemented, and a critical section. The custom reader/writer locks were both built using a Win32 critical section to protect the lock’s data, an event for signaling waiting readers, and a semaphore for waiting writers. The difference between the two custom locks is that one favors writers and one favors neither readers nor writers.

All of our anecdotal testing was done using a 64-bit version of Windows Vista on a system with two dual-core Intel Xeon 3.20 GHz processors. For these tests, the system was configured with hyperthreading disabled.

Test 1 consisted of 2,000,000 iterations of each thread where the work done while the locks are held is minimal. Therefore, the results reflect the actual cost of lock acquisition and release. The results are summarized in Figure 3.

Figure 3 Testing 2,000,000 Iterations (results measured in seconds) (Close Figure 3)

Lock Type | 1 Reader | 1 Writer | 2 Readers/ 2 Writers | 3 Readers/ 1 Writer | 4 Readers | 4 Writers
SRW Lock | 0.126 | 0.119 | 0.589 | 0.667 | 0.871 | 0.517
Custom R/W Lock - Favors Neither | 1.238 | 0.257 | 27.095 | 4.076 | 2.466 | 53.567
Custom R/W Lock - Favors Writers | 1.228 | 0.260 | 31.306 | 6.083 | 2.307 | 53.567
Critical Section | 0.134 | 0.133 | 1.084 | 1.021 | 1.036 | 1.009

There are some interesting findings here. First, when using just one reader thread or one writer thread (the first two columns of results), there is no lock contention and the performance of the SRW lock is very close to that of the critical section. In the case of four writer threads, all competing for exclusive access to the lock, the SRW lock takes only about half the time the critical section approach takes. The SRW lock used as an exclusive mode lock appears to perform better than the critical section and may be worth considering as a substitute. Notice that in the columns labeled 2 Readers/2 Writers and 3 Readers/1 Writer, the SRW lock performs significantly faster than when using a critical section. This demonstrates the benefit of a reader/writer lock allowing data readers to work in parallel.

What can we say about our homegrown reader/writer locks? Their performance looks terrible compared to the two built-in Windows locks, especially when operating under a heavy load of mixed contention from readers and writers. However, take a look at the results for 2 Readers/2 Writers and 3 Readers/1 Writer. Notice that the policy of the lock does have an effect on the overall behavior and performance. The lock that favored writers performed slower than the lock that favored neither readers nor writers. This is because parallelism was reduced to give writers preference when they were waiting to update. Of course, the goal is often to make sure the readers see the most current data and so using a lock that favors writers is the correct choice in that case.

Another issue with the custom-built locks is exhibited when four writers are competing for the lock. Why is the performance so terrible when there’s a lot of contention for exclusive access? The problem is that each thread is competing for access to the critical section that protects the internal data. Contention on the critical section causes a trip to kernel mode, where the thread is put to sleep on the event. Once the critical section has been entered, the state of the lock is examined to see if any readers are holding shared access or if another writer already has exclusive access. In either of these cases, the thread that wants exclusive access has to leave the critical section and then wait on the semaphore, which will be signaled when the thread has exclusive access and can continue to run. As you can see, frequently having to wait on multiple synchronization objects in order to get exclusive access really hurts the performance.

The ReaderWriterExample application includes a number of command-line switches so that its behavior can be tailored to allow for experimentation with different scenarios using the different types of reader/writer locks. The data that is protected by the lock is a simple LONG type. However, the application allows for the specification of an additional amount of work to be done while holding the lock in order to simulate accessing or updating a more complex data structure. The additional work parameter can be specified individually for readers and writers, allowing for the simulation of data structures that incur more overhead from updates than from reads. There are also parameters to specify how much work a reader/writer should do in between each access to the lock. Work is simulated by spinning in a loop performing a calculation.

We already know that the homegrown locks don’t perform well when they are under heavy load with lots of contention from both readers and writers. Furthermore, the SRW lock performed well in all of the cases in the first example. Does this mean you’ll never want to build your own reader/writer locks now that SRW locks are available? Not necessarily. Let’s look at what happens when the locks are used in scenarios that are more typical of how we expect the locks to be used in an application.

In these scenarios, the number of shared accesses by a reader is 1,000,000 while the number of exclusive accesses by the writers is 150,000. This keeps with what we said earlier—that reader/writer locks make sense when the ratio of shared accesses to exclusive updates is high. Additionally, readers hold the lock for 2000 units of work to simulate a read request, while writers hold the lock for 3000 work units to simulate the added expense of an update to the data. A reader thread performs 100 units of work between each access to the lock so that the spacing between accesses is short while the writer performs 10,000 work units before attempting exclusive access to cause a longer amount of time between updates. This reduces the overall contention on the locks.

We ran this test using 2 readers/2 writers and 3 readers/1 writer. The results are summarized in Figures 4 and 5. The numbers in the tables are the elapsed time, measured in seconds, needed for each thread to complete its work. For reader threads, the second number is the number of updates that the reader thread observed to the data.

Figure 5 Testing Higher Reads to Updates in a 3 Readers/1 Writer Scenario (results measured in seconds) (Close Figure 5)

3 Readers / 1 Writer Scenario | Slim R/W | Critical Section | Custom R/W Lock Favors Neither | Custom R/W Lock Favors Writers
Reader Thread 1 | 2.345 (761 updates) | 7.650 (21,266 updates) | 6.313 (97,997 updates) | 8.365 (132,290 updates)
Reader Thread 2 | 2.282 (631 updates) | 7.486 (11,466 updates) | 6.267 (102,102 updates) | 8.170 (140,144 updates)
Reader Thread 3 | 2.324 (633 updates) | 7.581 (20,013 updates) | 6.321 (98,100 updates) | 8.134 (126,568 updates)
Writer Thread 1 | 7.970 | 11.990 | 8.010 | 8.446
Total Execution Time | 7.970 | 11.990 | 8.010 | 8.446
Figure 4 Testing Higher Reads to Updates in a 2 Readers/2 Writers Scenario (results measured in seconds) (Close Figure 4)

2 Reader / 2 Writers Scenario | Slim R/W | Critical Section | Custom R/W Lock Favors Neither | Custom R/W Lock Favors Writers
Reader Thread 1 | 1.892 (1,185 updates) | 5.044 (19,689 updates) | 1.868 (7,728 updates) | 5.664 (133,215 updates)
Reader Thread 2 | 1.920 (1,177 updates) | 3.906 (16,223 updates) | 1.861 (7,402 updates) | 5.559 (139,283 updates)
Writer Thread 1 | 7.575 | 9.996 | 7.372 | 7.321
Writer Thread 2 | 7.574 | 10.250 | 7.378 | 7.317
Total Execution Time | 7.575 | 10.250 | 7.378 | 7.321

These results show more parity between our homegrown locks and the SRW lock. Notice how the different policies affect the results. The lock that favors writers causes the readers to wait while writers are allowed access, which means that updates were given higher priority and in turn allowed the readers to see more of the updates. Prioritizing updates can be vital in an application so understanding the policy of the lock and its performance characteristics under the expected load is an important consideration.

SRW locks are a great new synchronization primitive on the Windows platform. For the first time, Windows offers a built-in reader/writer lock for native systems programmers. It performs well under many different circumstances and should be the first reader/writer lock you use. There will be times when your application will benefit from a different lock policy but, as we demonstrated, building a custom reader/writer lock that performs well in a wide range of scenarios isn’t straightforward.

One-Time Initialization

The problem of how to ensure correct initialization of an object or resource that is to be shared by multiple threads is one that arises time and time again in building multithreaded systems. C and C++ offer no help in solving this problem because the language standard makes no mention of multithreading support. Consider an example where there’s to be one instance of a logger object used for logging messages and where one of the requirements is that the object be instantiated on demand instead of created at the start of program execution. The single-threaded initialization code shown in Figure 6 will not execute correctly in the presence of multiple threads simultaneously executing inside the GetLogger function to access the logger object in our system.

Figure 6 Initialization (Close Figure 6)

Single-Threaded Initialization
Logger* GetLogger()
{
    // C++ will execute the Logger’s constructor
    // only the first time this method is called.
    static Logger logger;

    return &logger;
}
Thread-Safe Initialization
Logger* GetLogger()
{
    static Logger* pLogger = 0;

    EnterCriticalSection(&cs);

    if (pLogger == 0)
    {
        try
        {
            pLogger = new Logger();
        }
        catch (...)
        {
            // Something went wrong.
        }
    }

    LeaveCriticalSection(&cs);

    return pLogger;
}

The initialization of the static Logger object may actually occur more than once because the compiler doesn’t place any synchronization around its construction. This may result in a corrupted logger object that in a best-case scenario will generate an exception when used—but there’s no guarantee the failure will be obvious and the system may run for a long time before anyone notices there’s a problem. One way to fix this problem is to rewrite the function and introduce synchronization so that it works correctly in the presence of multiple threads, as shown by the thread-safe initialization code in Figure 6.

Now each thread that enters the GetLogger function will attempt to enter the critical section, which means that only one thread at a time will be allowed inside the protected code block. A thread that’s executing in the critical section checks the value of pLogger and only if it is NULL will an instance of the Logger object be created. This will occur only once when the first thread enters the critical section. All other threads that subsequently enter will find that pLogger is not NULL and exit the critical section without doing any further work. By the time the return statement is reached, the value of pLogger will be non-NULL and can be returned to the caller.

On the surface, the thread safe initialization code seems like a reasonable solution. However, there is a penalty to be paid here, particularly if many threads end up calling the GetLogger function at the same time. Once the first thread has made it through allocating, constructing, and setting the pLogger pointer, there really isn’t any need for subsequent threads to enter the critical section object anymore. The pointer is always going to be valid. This realization has led to a design pattern in C++ programming called the double-checked locking pattern.

Figure 7 shows an implementation of GetLogger that uses the double-checked locking pattern. The pattern dictates that the pLogger variable is tested up to two times. The first time it checks the variable, it is done outside the critical section. If it finds that pLogger is NULL, the thread will then enter the critical section. Once inside, it will once again test pLogger against NULL before instantiating and setting the pLogger variable since the thread may have been blocked when waiting on the critical section due to a race with another thread.

Figure 7 Double-Checked Locking (Close Figure 7)
Logger* GetLogger()
{
    volatile static Logger* pLogger = 0;

    if (pLogger == NULL)
    {
        EnterCriticalSection(&cs);

        if (pLogger == NULL)
        {
            try
            {
                pLogger = new Logger();
            }
            catch (...)
            {
                // Something went wrong.
            }
        }

        LeaveCriticalSection(&cs);
    }

    return pLogger;
}

It appears that the double-checked locking version offers the best of all worlds. Only the few threads that enter GetLogger before the object is instantiated will be forced to be synchronized; those that come after won’t have to enter the critical section at all. What more could we want?

The double-checked locking pattern, while conceptually simple, has up until now proven to be very difficult to code correctly. This is because the C++ standard doesn’t define a threading model. It assumes only one thread of execution, and it doesn’t define a way for a developer to express constraints on relative instruction ordering, which gives the compiler latitude to reorder the reads and writes to memory. In a multithreaded environment, the reordering can cause a thread to observe a write to memory before all the statements that occur before it in the source code have actually executed. In the case of the double-checked locking code, it is possible that the pLogger variable will be updated with the address of the memory location allocated for the Logger object before the Logger constructor has executed. A second thread that observed the non-NULL value would never enter the critical section and return the address of an object that isn’t fully constructed. (You can read about similar issues in managed code in Vance Morrison’s article "Understand the Impact of Low-Lock Techniques in Multithreaded Apps".) In past releases of the Visual Studio® C++ compiler, even using the volatile qualifier on the variable pLogger wasn’t enough to ensure correct behavior in a multithreaded scenario. However, with the release of Visual Studio 2005, the double-checked locking pattern can be made to reliably execute on the Windows platform as long as the variable pLogger is qualified with the keyword volatile.

Even though Visual Studio 2005 makes it possible to correctly implement the double-checked locking pattern, it’s still very easy to get the implementation wrong simply by omitting the volatile qualifier on the variable holding the instance pointer or omitting the check inside the critical region; the program will still compile and appear to work. Having a mechanism that just works would be a valuable addition to C++. But rather than waiting for a new revision from the standards body, Windows Vista provides just such a facility called one-time initialization. No matter what hardware platform you’re using Windows on, this is guaranteed to work. One-time initialization allows for both synchronous and asynchronous initialization. First we’ll take a look at synchronous initialization.

Conceptually, synchronous initialization works the same way the double-checked locking pattern works. Of the first n threads that attempt to simultaneously execute initialization, only one will actually instantiate the resource—the others remain blocked until initialization is completed. Once initialization has completed, subsequent threads that attempt to access the resource will not be blocked; the stored resource will simply be returned.

The first thing that needs to be done is to declare and initialize an instance of the INIT_ONCE structure. You use the InitOnceInitialize function to perform the initialization. This must be done for both synchronous and asynchronous one-time initialization, and it must be done before the structure is used with any other one-time initialization function. Its definition is:

VOID WINAPI InitOnceInitialize(
  PINIT_ONCE InitOnce
);

The following function, meanwhile, is used to perform synchronous one-time initialization:

BOOL WINAPI InitOnceExecuteOnce(
  PINIT_ONCE InitOnce,
  PINIT_ONCE_FN InitFn,
  PVOID Parameter,
  LPVOID* Context
);

The first parameter, InitOnce, is a pointer to the instance of the INIT_ONCE structure. All threads that pass the same address of an INIT_STRUCTURE will be synchronized with respect to each other while initialization is occurring. The second parameter, InitFn, is a pointer to a function that the developer writes; this function performs the actual initialization. The third parameter, handily named Parameter, is an optional value that will be passed back to the InitFn when it is called. Here you can specify any information needed to perform the initialization. The final parameter, Context, is a pointer to a void pointer. This is where the address of the initialized object will be stored if the function succeeds. Note that I’m using the word "object" to describe the result of the one-time initialization. It doesn’t necessarily have to be a pointer to a C++ object. This can be anything that fits into a pointer-size value on the platform upon which execution is taking place. InitOnceExecuteOnce returns TRUE if it succeeds, FALSE if it does not.

Now we’ll look at the function pointed to by InitFn. The definition of the function has the following signature:

BOOL CALLBACK InitOnceCallback(
  PINIT_ONCE InitOnce,
  PVOID Parameter,
  PVOID* Context
);

The parameters are the same as those in the InitOnceExecuteOnce function. The InitOnceCallback function is expected to store the address of the initialized object in Context. Figure 8 contains the GetLogger function rewritten to use synchronous one-time initialization. Notice that the callback function indicates failure by returning FALSE. By returning FALSE from the callback, the system allows another thread that attempts initialization to execute the callback function in hopes that it will be successful. This continues until either there are no more initialization attempts or the callback returns TRUE.

Figure 8 Synchronous One-Time Initialization (Close Figure 8)
BOOL WINAPI InitLoggerFunction(PINIT_ONCE intOncePtr,
                               PVOID Parameter,
                               PVOID* contextPtr)
{
    try
    {
        Logger* pLogger = new Logger();
        *contextPtr = pLogger;
        return TRUE;
    }
    catch (...)
    {
        // Something went wrong.
        return FALSE;
    }
}

Logger* GetLogger()
{
    static INIT_ONCE initOnce;

    PVOID contextPtr;
    BOOL status;

    status = InitOnceExecuteOnce(&initOnce,
                                 InitLoggerFunction,
                                 NULL,
                                 &contextPtr);

    if (status)
    {
        return (Logger*)contextPtr;
    }

    return NULL;
}

Asynchronous one-time initialization is somewhat more complicated than the synchronous version, but it allows for one-time initialization without causing any of the threads performing the initialization to be blocked while waiting for initialization to complete. Thus, until initialization has completed, all threads that attempt initialization are allowed to concurrently execute and race through the initialization function where each initializes its own private copy of the object followed by an attempt to register the object as the one and only initialized object. Only one thread will be the so-called winner of the object registration and the remaining threads (the losers) must destroy their private object instances and then make a query to the system for the winning object.

The two APIs that are used for asynchronous one-time initialization are:

BOOL WINAPI InitOnceBeginInitialize(
  LPINIT_ONCE InitOnce,
  DWORD dwFlags,
  PBOOL fPending,
  LPVOID* Context
);

BOOL WINAPI InitOnceComplete(
  LPINIT_ONCE lpInitOnce,
  DWORD dwFlags,
  LPVOID lpContext
);

Figure 9 shows the same GetLogger function that was used above, but now it has been rewritten to use asynchronous one-time initialization. We’ll go through this routine step by step. The first thing a thread does is call InitOnceBeginInitialize. The thread must supply the pointer to the INIT_ONCE structure to be used, and the dwFlags parameter should be set to INIT_ONCE_ASYNC to specify that this thread is attempting to begin asynchronous one-time initialization. The state of initialization is communicated back to the caller using the remaining two parameters, fPending and Context, which are both specified as pointers so the function can update them. InitOnceBeginInitialize returns TRUE if it is successful; otherwise it returns FALSE, indicating that something went wrong and initialization cannot proceed. The fPending parameter must be examined to determine whether another thread has already completed the initialization. If fPending is FALSE, initialization has already completed and the initialized object has been stored in the Context parameter. In this case, the only work that remains is for GetLogger to convert Context to a Logger* and return it to the caller.

Figure 9 Asynchronous One-Time Initialization (Close Figure 9)
Logger* GetLogger()
{
    static INIT_ONCE initOnce;

    PVOID contextPtr;
    BOOL    fStatus;
    BOOL    fPending;

    fStatus = InitOnceBeginInitialize(&initOnce,INIT_ONCE_ASYNC,
        &fPending, &contextPtr);

    // Failed?
    if (!fStatus)
    {
        return NULL;
    }

    // Initialization already completed?
    if (!fPending)
    {
        // Pointer to the logger is contained context pointer.
        return (Logger*)contextPtr;
    }

    // Reaching this point means that the logger needs to be created.

    try
    {
        Logger* pLogger = new Logger();

        fStatus = InitOnceComplete(
            &initOnce,INIT_ONCE_ASYNC,(PVOID)pLogger);

        if (fStatus)
        {
            // The Logger that was created was successfully
            // set as the logger instance to be used.
            return pLogger;
        }

        // Reaching this point means fStatus is FALSE and the object this
        // thread created is not the sole instance.
        // Clean up the object this thread created.
        delete pLogger;
    }
    catch (...)
    {
        // Instantiating the logger failed.
        // Fall through and see if any of
        // the other threads were successful.
    }

    // Call again but this time ask only for
    // the result of one-time initialization.
    fStatus = InitOnceBeginInitialize(&initOnce,INIT_ONCE_CHECK_ONLY,
        &fPending,contextPtr);

    // Function succeeded and initialization is complete.
    if (fStatus)
    {
        // The pointer to the logger is in the contextPtr.
        return (Logger*) contextPtr;
    }

    // InitOnceBeginInitialize failed, return NULL.
    return NULL;
}

Things get more interesting when fPending comes back with a value of TRUE. This indicates that initialization should proceed and that there may be other threads also running the initialization code at the same time as the caller’s thread. In Figure 9, this results in the creation of a new Logger object followed by an attempt to set the instance as the one and only instance result from the asynchronous one-time initialization. This can be seen in the InitOnceComplete call that occurs after the new statement. The first parameter specifies the pointer to the same INIT_ONCE structure that was used previously, the INIT_ONCE_ASYNC flag is passed as the second parameter, and the pointer to the logger instance is supplied in the third parameter. If InitOnceComplete returns TRUE, then this instance is the one that will be returned to all callers of GetLogger.

Should InitOnceComplete return FALSE, a logger created by a different thread was stored and the caller should destroy the instance it created so that any resources it consumes are not stranded. After this, InitOnceBeginInitialize is called again, but this time instead of using the INIT_ONCE_ASYNC flag, the INIT_ONCE_CHECK_ONLY flag is used. The INIT_ONCE_CHECK_ONLY flag is used to check to see if initialization has completed. If it has, the initialization value that was stored is copied into the supplied Context pointer parameter. If a valid initialization value is returned, the function will return TRUE and the fPending parameter will be set to FALSE.

So how do you choose between synchronous and asynchronous one-time initialization? In a uniprocessor system, it makes sense to use synchronous since only one thread can be executing at a time. In a multiprocessor or multicore system, you’ll need to try to quantify the costs that may be incurred if multiple object instances are created. Costs can include time, memory (or other resource types that may be scarce), and the number of threads that will typically be concurrently executing the initialization code before an object has been successfully initialized. Synchronous initialization might be a better choice if the object takes more than a negligible amount of time to construct or if it uses significant amounts of memory either per object or aggregated across all objects that might be simultaneously created. Additionally, if the process of object creation can cause the threads performing the initialization to block, then synchronous initialization is the better choice since the opportunity for concurrency is then diminished and little would likely be gained from asynchronous initialization.

The source code for this article includes a sample, called OneTimeInit.exe, that demonstrates both synchronous and asynchronous one-time initialization of a shared event handle. The program takes the number of threads to run and a flag indicating whether to perform synchronous or asynchronous initialization. The specified number of threads are spawned and each thread attempts to retrieve a handle to a Win32 event object used to signal when the thread should terminate. The progress of each thread is sent to stdout, allowing you to see exactly how the initialization progresses. Each thread does some contrived work until the terminate event is signaled by the main thread. To get program usage information, simply run the program without any parameters.

There are a couple of loose ends that need to be mentioned with regard to one-time initialization. We noted that with synchronous initialization the callback function will continue to be called until it succeeds. Suppose your application scenario dictates that the initialization should only be attempted once and never again if it fails. One way to do this is to have the callback function still return TRUE but also set a special value in lpContext that other threads can check for as an indication that a failure occurred. Another mechanism is to use the InitiOnceBeginInitialize/InitOnceComplete APIs for synchronous initialization instead of the InitOnceExecuteOnce API. To do this, omit the INIT_ONCE_ASYNC flag from the dwFlags parameter and place the initialization code between the calls to InitOnceBeginInitialize and the InitOnceComplete API, rather than in a separate callback function. In the case of failure, the INIT_ONCE_INIT_FAILED flag is used with the InitOnceComplete API to indicate that no further initialization attempts should occur. An optional failure value can be set using lpContext, which other threads can check for.

The second loose end is that one-time initialization doesn’t help with cleanup of the object that was created. There are a number of ways to deal with this and no one solution fits every situation. If the object is something that the operating system cleans up after the process terminates, consider leaking it. But beware that this can show up as an issue if you use tools that monitor your programs for resource leaks. In cases where the object needs to have its cleanup function called to ensure proper behavior, you’re left with little choice but to coordinate the call of the function after all threads that are using the object have terminated. However, you may be able to use a technique similar to the one used in the sample. The sample uses a static object that the initialized HANDLE is registered with so that it can be automatically destroyed at the end of the program execution after all the threads have terminated.

Conclusion

In this article, we’ve shown you that there are a lot of important enhancements to the thread synchronization primitives in Windows Vista for native C/C++ developers. These enhancements allow developers to more easily address thread synchronization problems. However, this article has only scratched the surface. We encourage you to investigate these and other changes further. For starters, check out the thread pool API enhancements. Unfortunately, we don’t have space to discuss the thread pool API enhancements here, but they are a valuable addition worth careful consideration.

We’d like to thank Neill Clift and Arun Kishan for answering our questions and providing insightful feedback. In addition, we’d also like to thank Jerry Albro for reviewing the content and providing us with valuable feedback.

Robert Saccone is an Architect in the Forefront Server Security group. His areas of interest are large-scale software design, distributed systems, and operating systems implementations.

Alexander Taskov is a Senior Software Development Engineer on the Forefront Server Security team at Microsoft. He is responsible for the design and development of security features on the Forefront Server Security product.


Editor's Note: Visual Studio at 10

Editor's Note
Visual Studio at 10
Joshua Trupin

1997 was a happenin’ year. The first Web boom was taking hold. Everyone on the block had a Tamagotchi (a briefly popular virtual pet), the Spice Girls ruled our airwaves, Austin Powers wasn’t yet tiresome, and Microsoft released Visual Studio.

This month we’re observing the 10th anniversary of Visual Studio, the program that pulled together a set of powerful development tools in one place and made it easier for programmers to stay productive.

In the May 1997 issue of Microsoft Systems Journal (the forerunner of MSDN Magazine), we introduced Visual Studio for the first time:

Microsoft Visual Studio 97 goes a long way toward providing you with a well-stocked toolbox to help you complete your development projects. Visual Studio targets developers and teams that create dynamic Web-based applications or other distributed applications. Visual Studio takes all of Microsoft’s development tools, adds a rich set of new tools for creating Web content, and ties it all together with comprehensive documentation. Even if you’re not creating distributed applications but you use two or more Microsoft developer products, you’ll probably find Visual Studio suits you.

Since 1997, Visual Studio has evolved both vertically and horizontally. We watched and documented it as it became the tool of choice for .NET-based development. More recently, we’ve seen Visual Studio Team System gain a foothold in the market, with its comprehensive solutions for application lifecycle management.

What do we see as the major advances in Visual Studio over the years? Believe it or not, one involved a complete overhaul while the other was about granularization.

When Visual Studio .NET first appeared as an early prerelease in 2000, it was an entirely new paradigm—cross-language programming with the common language runtime (CLR)—packaged so that it was as accessible for Visual Basic users as for C++ users. In addition, a new language (C#) quickly became popular with developers—and noticeably with our authors and contributors. It wasn’t something we pushed, either. Our long-term writers saw what a good language it was right off the bat, and a lot of that was thanks to Visual Studio .NET making it so accessible.

Years later, Visual Studio 2005 hit the shelves. Now you could try out a different language with a no-cost Express edition. Being able to download and start coding with a free, lightweight yet feature-rich IDE is immensely appealing.

The availability of Express versions of the Visual Studio environment also means that our magazine content is even more relevant to readers. Gone are the days where you had to skip data articles because you didn’t have SQL Server installed. Now you can just go get it for the cost of a download (35 to 70MB).

At MSDN Magazine, we think everyone should solve life’s little problems in the best possible manner—by programming them away. There’s nothing like a clean sheet of programming space in your IDE, and Visual Studio now brings this experience to anyone who has an Internet connection. No, it won’t bring your virtual pet back to life, but an hour of coding every day will help keep your mind sharp for much longer than the next 10 years.

Note: due to a printing error, the last page of CLR Inside Out was omitted from the May issue. Since we love our readers, the piece is reprinted in its entirety after Page 24. —J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Luca Bolognese, Neill Clift, Nikola Dudar, Joe Duffy, Eric Faller, Matt Gibbs, Mike Harsh, Anders Hejlsberg, Luke Hoban, Arun Kishan, Karen Liu, Marian Luparu, Bogdan Ionut Mihalcea, George Mileka, Raj Pai, Doug Purdy, Mark Rideout, Rachel Falzone Schaw, Pete Sheill, Amanda Silver, Weitao Su, Deepu Thomas, Scott Wisniewski, and Michiel Wories.

Joshua Trupin


Toolbox: XML Comments, Online Payments, and More

Toolbox
XML Comments, Online Payments, and More
Scott Mitchell

Contents
Quickly Add XML Comments to Your Visual Studio Projects
Accept Online Credit Card Payments
Improve Your Programming Experience in Visual Studio
A Guide to Developing Data-Driven Applications

Quickly Add XML Comments to Your Visual Studio Projects

The Microsoft® .NET Framework simplifies the task of documenting classes, methods, properties, and events through XML comments. XML comments are special XML tags embedded within source code comments that provide metadata about classes and their members. During compilation, this metadata is converted into a stand-alone XML file by Visual Studio®, which can then be converted into help files using tools like the Microsoft Sandcastle project.

GhostDoc Generates XML Comments Automatically
GhostDoc Generates XML Comments Automatically (Click the image for a smaller view)
GhostDoc Generates XML Comments Automatically
GhostDoc Generates XML Comments Automatically (Click the image for a larger view)

While XML comments assist in automating the conversion of source code comments into a rich help document, the comment text must still be written, a job many developers find mind-numbing and laborious. Why not use automation to expedite the creation of these comments? That’s exactly what GhostDoc 1.9.5 attempts to accomplish.

GhostDoc is a free Visual Studio add-in, created by Roland Weigelt, that assists in authoring XML comments. Once GhostDoc has been installed, using it to auto-generate XML comments is as simple as point and click. For example, to add XML comments to a method, simply right-click within the method and choose the "Document this" option from the context menu. GhostDoc will then automatically generate the XML comment text for the method based on its type, parameters, name, and other contextual information.

If you’re generating documentation for a property or method that uses a type in the .NET Framework, GhostDoc will use the documentation that Microsoft has already written for that type. Assuming you use Pascal casing or camel casing, GhostDoc can break apart the name into words and analyze those individual words to generate the documentation. All of the documentation logic is handled by GhostDoc’s generation rules. The built-in rules can be customized and new ones can be added. These rules can be exported for use on other machines as well.

Except for the simplest members, you can’t rely on GhostDoc to write your XML comments for you completely. GhostDoc’s auto-generated comments are suggestions and should be reviewed by a developer. Regardless, GhostDoc serves as a good first stab at documentation and is a great time-saver.

Price: Free.

www.roland-weigelt.de/ghostdoc

Accept Online Credit Card Payments

During my career as a developer, I’ve often considered taking some application or component I’ve created and selling it to other developers. But figuring out how to process online payments always seemed complex.

Today, virtually all online transactions are paid for using a credit or debit card, and to accept these types of payments you need two things: a merchant account from a financial institution (such as your neighborhood bank or an online entity like PayPal), and a payment gateway. A merchant account allows payments to be collected from another bank account or credit card. A payment gateway is a system that manages the electronic processing of a payment.

Many companies provide payment gateway services. One of the easiest I’ve found to use is Authorize.NET. Once you’ve created an account with Authorize.NET, you can collect payment information from your customers either through the Authorize.NET Web site or from your own Web site. With the first approach, customers are sent to Authorize .NET’s Web site as the final step in placing an order. There they enter their payment information and are charged the specified amount. Authorize.NET then redirects them back to your Web site.

With the second approach, a customer’s payment information is entered directly from your Web site and then sent to Authorize.NET’s servers behind the scenes as an XML payload via a secure HTTP request. This XML data interchange can be done programmatically with a few lines of code in an ASP.NET page. Alternatively, there are a variety of third-party shopping cart products that provide support for Authorize.NET.

Regardless of how the customer’s payment information is collected, Authorize.NET interfaces with your merchant account and the customer’s credit or debit card account, billing the customer and crediting your account. Amazingly, this entire process typically takes just a few seconds.

Authorize.NET provides both address verification and card code verification to help deter credit card fraud. And Authorize.NET can be configured to handle recurring payments.

Keep in mind that Authorize.NET is just the payment gateway. You also need a merchant account, which carries its own setup, monthly, and transaction fees. Many financial institutions charge in the neighborhood of $100 for setup, $30 or so as a monthly fee, and another 2.0-3.0% plus $0.05-$0.50 fee per transaction.

Price: $149.00 setup, $20.00 monthly, and $0.10 per transaction.

www.authorize.net

Improve Your Programming Experience in Visual Studio

With each new version, Visual Studio adds more wizards, features, and tools that aid in developer productivity and efficacy. While the enhancements built in to each new version are helpful, there are a host of other tools that can be used to increase productivity, as evidenced by the robust Visual Studio add-in market. These add-ins provide even more features geared toward assisting developers write, refactor, and test code.

One such Visual Studio add-in is ReSharper 2.5 by JetBrains, which offers intelligent error highlighting with quick-fix options, extended refactoring support, customizable code formatting, unit testing, NAnt and MSBuild script editing, and a variety of tools for quickly searching and navigating through the types and files of a Visual Studio project. The features I found most handy were the quick-fixes and the navigation and search options. Quick-fixes appear as a lightbulb icon next to a line of code where ReSharper has detected a warning or error. For example, if you forget to perform a type cast, ReSharper notifies you via a lightbulb icon with a quick-fix option to automatically inject the appropriate cast.

ReSharper Notifies You about Quick-Fixes via a Lightbulb Icon
ReSharper Notifies You about Quick-Fixes via a Lightbulb Icon (Click the image for a smaller view)
ReSharper Notifies You about Quick-Fixes via a Lightbulb Icon
ReSharper Notifies You about Quick-Fixes via a Lightbulb Icon (Click the image for a larger view)

The navigation and search capabilities of ReSharper are ideal for large projects where there are many types and files. The Navigate from Here feature allows you to quickly jump to a variable or type’s declaration or base class. Moreover, it can list any inheritors as well as all places where the variable or type is used in the project. A Find Results window makes it easy to search the project for a particular type. The File Structure window shows the types and members of the current file. From the File Structure window you can reorder the types or members in the file and quickly jump from the window to a particular member in the file.

Note that the Visual Studio .NET 2003 version of ReSharper works only with C# code. However, the Visual Studio 2005 version does offer some Visual Basic® features, though not all of those that are available for C# developers. ReSharper also includes some features for editing ASP.NET Web pages.

Price: $249 for a single-user license.

www.jetbrains.com/resharper

A Guide to Developing Data-Driven Applications

When I first started programming data-driven Visual Basic applications back in the late 1990s, a coworker encouraged me to pick up a copy of William R. Vaughn’s Hitchhiker’s Guide to Visual Basic and SQL Server, a tome whose heft and thickness earned it the nickname "the doorstop." Throughout the 90s, Vaughn published six editions of his popular Guide. After a nine-year hiatus he has released the seventh edition, written with Peter Blackburn, and renamed Hitchhiker’s Guide to Visual Studio and SQL Server (Addison-Wesley, 2007).

Like the previous editions, the seventh provides detailed and thorough discussions on important topics, from installation of SQL Server™ 2005 to T-SQL basics to creating and working with data through a .NET application. The book starts with a look at application architectures, noting different data access technologies and comparing and contrasting the different versions of SQL Server 2005. Next, Vaughn steps the reader through installing SQL Server 2005 and addresses security and management issues. This is followed by a look at SQL essentials: SELECT, INSERT, UPDATE, and DELETE statements; views; UDFs; TRIGGERs; stored procedures; database design; normalization; data and relational integrity; and so on.

(Click the image for a smaller view)

Hitchhiker’s Guide to Visual Studio and SQL Server is split pretty evenly between discussions of SQL Server-specific topics and working with data using Visual Studio. The Visual Studio-related material include accessing databases through the IDE, using ADO.NET to programmatically work with data, building Typed DataSets, and techniques for modeling database entities as objects within an application architecture.

Vaughn’s knowledge and accessible writing style make this book a must-have for .NET developers creating data-driven applications. As with his past books, Vaughn’s code examples are all in Visual Basic, but that shouldn’t dissuade C# developers as the real value of this book lies in the concepts and architectural advice, not in cut and paste code samples.

Price: $59.99.

www.aw-bc.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at www.ScottOnWriting.NET.


CLR Inside Out: Reflections on Reflection

CLR Inside Out
Reflections on Reflection
Mike Repass

Code download available at: CLRInsideOut2007_06.exe (163KB)
Contents
Reflection in .NET
Inspecting Types and Members
Invoking Code Dynamically
Putting It Together
Efficient Type Processing for Serialization
Finishing Up

Are your goals of clean componentization frustrated by the need to share too much type information across libraries? Perhaps you want the efficiency of strongly typed data storage, but it’s costly to update your database schema whenever your object model evolves and so you’d like to infer type schema at run time? Do you need to ship components that accept arbitrary user objects and process them in some type-intelligent way? Do you wish callers of your library could describe their types to you programmatically?

If you find yourself struggling to maintain strongly typed data structures, yet still maximize runtime flexibility, then you’ll likely want to consider reflection and how it can improve your software. In this column, I explore the System.Reflection namespace in the Microsoft® .NET Framework and how it can benefit your development experience. I walk through some simple examples and finally tackle the real-world scenario of serialization, in which I show how reflection and CodeDom can work together for efficient processing of runtime data.

Before diving into System.Reflection, I’d like to step back a bit and discuss programmatic reflection in general. To start, reflection can be defined as any functionality offered by a programming system that allows the programmer to inspect and manipulate code entities without knowing their identification or formal structure ahead of time. That’s a mouthful, so I’ll dissect it piece by piece.

First, what does reflection offer? What can you use it for? I prefer to organize typical reflection-centric tasks into two categories: inspection and manipulation. Inspection entails analyzing objects and types to gather structured information about their definition and behavior. Typically this is done with little or no prior knowledge about them, apart from some basic provisions. (For instance, in the .NET Framework, everything inherits from System.Object and an object-typed reference is your typical starting point for reflection.)

Manipulation uses the information gained through inspection to invoke code dynamically, create new instances of discovered types, or even restructure types and objects on the fly. It’s important to note that, for most systems, manipulating types and objects at run time will incur a performance penalty when compared to the equivalent operations done statically in the source code. This is a necessary trade-off given the dynamic nature of reflection, but there are many tricks and best practices available for optimizing your Reflection performance (see msdn.microsoft.com/msdnmag/issues/05/07/Reflection for more in-depth information on optimizing usage of reflection).

Now, what’s the target of reflection? What does the programmer actually inspect or manipulate? In my definition of reflection, I used the novel term "code entities" to highlight the fact that, from a programmer’s perspective, reflection technology can sometimes blur the conventional distinction between objects and types. For instance, a typical reflection-centric task might be:

  1. Start with a handle to an object O and use reflection to acquire a handle to its associated definition, a type T.
  2. Inspect type T and acquire a handle to its method, M.
  3. Invoke method M on another object, O’ (also of type T).

Note I’m traversing from an instance to its underlying type, from that type to a method, and then using a handle to that method to invoke it on a different instance—clearly something that is not possible with conventional C# programming techniques in source code. I’ll revisit this scenario later with a concrete example, once I’ve explored the System.Reflection in the .NET Framework.

Some programming languages offer reflection natively via their syntax whereas other platforms and frameworks (such as the .NET Framework) make it available as a system library. Regardless of how it’s exposed, the potential for using reflection technology in a given scenario is rather complex. A programming system’s ability to offer reflection depends on many factors: did the programmer express his concepts well with the features of the programming language? Does the compiler embed enough structural information (metadata) in the output so as to allow later interpretation? Is there a runtime subsystem or host interpreter available to consume that metadata? Does a platform library expose the results of this interpretation in a manner that’s useful for the programmer?

If you maintain a complex, object-oriented type system in your head, but express your code in simple C-style functions with no formal data structures, then it’s clearly impossible for your program to dynamically infer that the pointer in some variable v1 points to an object instance of some type T. Because, after all, type T is a concept in your head that is never captured in your explicit programming statements. But if you use a more flexible object-oriented language (such as C#) to express the abstract structure of your program and directly introduce the concept of type T, then the compiler will transform your ideas into a form that suitable logic could later understand, as might be provided by the common language runtime (CLR) or a dynamic language interpreter.

Is reflection exclusively a dynamic, runtime technology? Simply put, no it isn’t. There are many times throughout the development and execution cycle when reflection might be available and useful to the developer. Some programming languages are implemented by standalone compilers that transform high-level code directly into machine-consumable instructions. The output file consists solely of the translated input and there’s no support logic available at run time to take an opaque object and dynamically analyze its definition. This is exactly the scenario of many traditional C compilers. Since there’s little support logic available in the target executable, you can’t do much dynamic reflection, but the compiler will frequently offer static reflection—for example, the commonly used typeof operator allows the programmer to check type identity at compile time.

At the opposite end of the spectrum, there are interpreted programming languages that are always executed by a host process (scripting languages often fall into this category). Since the program’s full definition is available (as the input source code) and joined together with the full language implementation (as the interpreter itself), all the necessary technology is present to support self-analysis. Such dynamic languages frequently offer exhaustive reflection functionality, allowing you a rich set of tools for dynamically analyzing and manipulating your program.

The .NET Framework CLR and its hosted languages like C# fall somewhere in the middle. A compiler is used to transform source code into IL and metadata which, though lower level and perhaps less "logical" than the original source, still contain much of the abstract structural and type information. Once the program is launched and hosted by the CLR, the System.Reflection library of the base class library (BCL) can consume this information and return information about an object’s type, a type’s members, a member’s signature, and so on. Furthermore, it can support invocation as well, including late-bound invocation.

Reflection in .NET

To take advantage of reflection when programming with the .NET Framework, you use the System.Reflection namespace. This namespace provides classes which encapsulate many runtime concepts—such as assemblies, modules, types, methods, constructors, fields, and properties. The table in Figure 1 shows how classes in System.Reflection correspond to their conceptual runtime counterparts.

Figure 1 System.Reflection Classes (Close Figure 1)

Language Component | Corresponding .NET Class
Assembly | System.Reflection.Assembly
Module | System.Reflection.Module
Abstract Member | System.Reflection.MemberInfo (base class for everything below)
Type | System.Type
Property | System.Reflection.PropertyInfo
Field | System.Reflection.FieldInfo
Event | System.Reflection.EventInfo
Abstract Method | System.Reflection.MethodBase (base class for everything below)
Method | System.Reflection.MethodInfo
Constructor | System.Reflection.ConstructorInfo

Though quite important, System.Reflection.Assembly and System.Reflection.Module are primarily useful for locating and loading new code into the runtime. In this column, I leave them behind and assume that all relevant code is already loaded.

For inspecting and manipulating loaded code, the typical pattern focuses on System.Type. Normally, you start by obtaining a System.Type instance for the runtime type of interest (via Object.GetType). Then you use the various methods of System.Type to explore the type’s definition and acquire instances of the other classes inside System.Reflection. For instance, if you’re interested in a particular Method, you’ll want to acquire an instance of System.Reflection.MethodInfo for that Method (possibly via Type.GetMethod). Likewise, if you’re interested in a particular field, you’ll want to acquire an instance of System.Reflection.FieldInfo for that Field (possibly via Type.GetField).

Once all the necessary reflection instance objects are obtained, proceed along the path of inspection or manipulation, as appropriate. For inspection, you use the various descriptive properties on the reflection classes to gain the information you need (Is this a generic type? Is this an instance method?). For manipulation, you can dynamically invoke the execution of methods, create new objects by invoking constructors, and so on.

Inspecting Types and Members

Let’s jump into some code and explore the usage of basic reflection for inspection. I’ll focus on type analysis. Starting with an object, I’ll retrieve its type and then explore some interesting members (see Figure 2).

Figure 2 Retrieving Object Type and Members (Close Figure 2)

Code

using System;
using System.Reflection;

// use Reflection to enumerate some basic properties of a type...

namespace Example1
{
    class MyClass
    {
        private int MyField = 0;
        public void MyMethod1() { return; }
        public int MyMethod2(int i) { return i; }
        public int MyProperty { get { return MyField; } }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Reflection Demo Example 1");

            MyClass mc = new MyClass();
            Type t = mc.GetType();
            Console.WriteLine("Type Name: {0}", t.Name);

            foreach(MethodInfo m in t.GetMethods())
                Console.WriteLine("Method Name: {0}", m.Name);

            foreach (PropertyInfo p in t.GetProperties())
                Console.WriteLine("Property Name: {0}", p.Name);
        }
    }
}
Output

Reflection Demo Example 1
Type Name: MyClass
Method Name: MyMethod1
Method Name: MyMethod2
Method Name: get_MyProperty
Method Name: GetType
Method Name: ToString
Method Name: Equals
Method Name: GetHashCode
Property Name: MyProperty

The first thing to notice is that I got many more lines describing methods than you might expect upon first glance at the class definition. Where did these extra methods come from? Anyone well versed in the .NET Framework object hierarchy will recognize the methods inherited from the universal base class of Object itself. (In fact, I actually used Object.GetType to retrieve the type in the first place.) Additionally, you can see the getter function for the property. Now, what if you just wanted those functions that are explicitly defined on MyClass itself; in other words, how do you hide the inherited functions? Or perhaps you just want the explicitly defined instance functions?

A quick trip to MSDN® online reveals that you want to use the second overload of GetMethods, which accepts a BindingFlags parameter. By combining various values from the BindingFlags enumeration, you can instruct the function to only return the desired subset of methods. Replace the call to GetMethods with a call to:

GetMethods(BindingFlags.Instance | BindingFlags.DeclaredOnly |
           BindingFlags.Public)

As a result, you get the following output (notice the absence of the static helper function and the functions inherited from System.Object):

Reflection Demo Example 1
Type Name: MyClass
Method Name: MyMethod1
Method Name: MyMethod2
Method Name: get_MyProperty
Property Name: MyProperty

What if you know the names (fully qualified) of a type and member ahead of time? How do you shift gears from type enumeration to type retrieval? The example in Figure 3 shows how to use string literals that describe type information to retrieve their actual code counterparts via Object.GetType and Type.GetMethod. Taking the code in the first two examples, you have the basic components necessary to implement a primitive class browser. You can discover a runtime entity via name and then enumerate its various relevant properties.

Figure 3 Retrieve Type and MethodInfo via Strings (Close Figure 3)
using System;
using System.Reflection;

// use Reflection to retrieve references to a type and method via name

namespace Example2
{
    class MyClass
    {
        public void MyMethod1() { return; }
        public void MyMethod2() { return; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(“Reflection Demo Example 2”);

            // note that we must use the fully qualified name...
            Type t = Type.GetType(“Example2.MyClass”);
            MethodInfo m = t.GetMethod(“MyMethod1”);

            Console.WriteLine(“Type Name: {0}”, t.Name);
            Console.WriteLine(“Method Name: {0}”, m.Name);
        }
    }
}

Invoking Code Dynamically

So far, I’ve acquired handles to runtime objects (such as types and methods) purely for descriptive purposes, like outputting their names. But how can you do more? How can you actually invoke a method? Figure 4 shows how to acquire a MethodInfo for a member of a type and then use MethodInfo.Invoke to actually call the method dynamically.

Figure 4 Calling a Method Dynamically (Close Figure 4)
using System;
using System.Reflection;

// use Reflection to retrieve a MethodInfo for an
// instance method and invoke it upon many object instances

namespace Example3
{
    class MyClass
    {
        private int id = -1;

        public MyClass(int id) { this.id = id; }

        public void MyMethod2(object p)
        {
            Console.WriteLine(
                “MyMethod2 is being invoked on object with “ +
                “id {0} with parameter {1}...”,
                    id.ToString(), p.ToString());
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(“Reflection Demo Example 3”);

            MyClass mc1 = new MyClass(1);
            MyClass mc2 = new MyClass(2);

            Type t = mc1.GetType();
            MethodInfo method = t.GetMethod(“MyMethod2”);

            for(int i = 1; i <= 5; i++)
                method.Invoke(mc2, new object[]{i});
        }
    }
}

A few key points are in order for this example. First, you retrieve a System.Type instance from an instance of MyClass, mc1. Next, retrieve a MethodInfo instance from that Type. Finally, when you invoke MethodInfo, you bind it to a different instance of MyClass (mc2) by passing it in as the first parameter of Invoke.

As mentioned previously, this example blurs the boundaries between type and object usage that you might expect from typical source code. Logically speaking, you retrieve a handle to a method and then invoke that method as if it belonged to a different object. This might be second nature to the programmer experienced with a functional programming language, but to the coder only familiar with C#, it might seem unintuitive to decouple object implementation from object instantiation.

Putting It Together

Now that I’ve explored the basics of inspection and invocation, I’ll put things together with a concrete example. Imagine that you want to ship a library with a static helper function that must process objects—but at design time, you don’t know anything about the types of these objects! It’s up to the caller of the function to instruct it on how to extract meaningful information from these objects. The function will accept a collection of objects and a string descriptor of a method. It will then iterate through the collection, calling the method on each object and aggregating the return values with some function (see Figure 5).

Figure 5 Extracting Information from Objects (Close Figure 5)
using System;
using System.Collections.Generic;
using System.Reflection;

namespace Example4
{
    class Program
    {
        static void Main(string[] args)
        {
            // prepare some objects for our function to process
            object[] objs = new object[] {
                new IntReturner(1), new IntReturner(2),
                new IntReturner(3), new SonOfIntReturner(4),
                new SonOfIntReturner(5), new SonOfIntReturner(6),
            };

            Console.WriteLine(
                “Attempting to compute average, “ +
                “passing in array with {0} elements.”, objs.Length);

            int average = ComputeAverage(objs, “GetInteger”);

            Console.WriteLine(“Found an average of {0}!”, average);
        }

        public static int ComputeAverage(
            IEnumerable<object> objs, string methodname)
        {
            int sum = 0, count = 0;

            Type firstType = null;
            MethodInfo firstMethod = null;

            foreach (object o in objs)
            {
                if (firstMethod == null)
                {
                    firstType = o.GetType();
                    firstMethod = firstType.GetMethod(methodname);
                }

                sum += (int)firstMethod.Invoke(o, null);
                count++;
            }

            // note that we use integer division here (not floating point)
            if (count == 0) return 0;
            return sum / count;
        }
    }

    class IntReturner
    {
        protected int value = -1;
        public IntReturner(int i) { value = i; }
        public virtual int GetInteger()
        {
            Console.WriteLine(
                “GetInteger called on instance of IntReturner, “
                “I’m returning {0}!”, value);
            return value;
        }
    }

    class SonOfIntReturner : IntReturner
    {
        public SonOfIntReturner(int i) : base(i) { }
        public override int GetInteger()
        {
            Console.WriteLine(
                “GetInteger called on instance of SonOfIntReturner, “
                “I’m returning {0}!”, this.value);
            return value;
        }
    }

    class EnemyOfIntReturner
    {
        protected int value = -1;
        public EnemyOfIntReturner(int i) { value = i; }
        public virtual int GetInteger()
        {
            Console.WriteLine(
                “GetInteger called on instance of EnemyOfIntReturner, “
                “I’m returning {0}!”, value);
            return value;
        }
    }
}

For purposes of this example, I’ll document some constraints. First, the method described by the string parameter (and necessarily implemented by each object’s underlying type) will accept no parameters and will return an integer. The code will iterate through the collection of objects, calling the specified method and gradually computing the average of all the values. And finally, since this isn’t production code, I won’t worry about validating parameters or integer overflow when aggregating the sum.

As you look through the example code, observe that the protocol between the Main function and the static helper ComputeAverage does not depend on any type information apart from the universal base class of Object itself. In other words, you could completely alter the type and structure of the objects you’re passing around, but so long as you can always use a string to describe a method that returns an integer, your ComputeAverage function will work!

There’s a critical gotcha to be aware of that is related to MethodInfo (and reflection in general) hidden in this latest example. Note that, inside the foreach loop of ComputeAverage, the code only grabs a MethodInfo from the first object in the collection and then binds for invocation for every subsequent object. As coded, this works great—it’s a simple example of MethodInfo caching. But there’s a fundamental limitation here. A MethodInfo instance can only be invoked on instances of types from the same hierarchy as the object from which it was retrieved. You see this in action because you pass in instances of both IntReturner and SonOfIntReturner (which inherits from IntReturner).

In the sample code, I’ve included a class called EnemyOfIntReturner that implements the same basic protocol as the other two classes, but does not share any common shared type. In other words, the interface is logically equivalent, but there’s no overlap in the type hierarchy. To explore the usage of MethodInfo in this scenario, try adding another object to the collection, an instance acquired with "new EnemyOfIntReturner(10)" and then run the example again. You’ll hit an exception explaining that the MethodInfo cannot be used to Invoke on the specified Object because it’s completely unrelated to the original type from which the MethodInfo was obtained (even though the method name and basic protocol is equivalent). To make your code production-quality, you must be prepared for this scenario.

A possible solution might include analyzing the type of all incoming objects on your own and maintaining an interpretation of their shared type hierarchy (if any). When the next object is of a type divergent from any known type hierarchy, you need to acquire and store a new MethodInfo. Another solution might be to catch the TargetException and simply reacquire a MethodInfo instance. Both of the solutions I’ve mentioned here have pros and cons. Joel Pobar wrote a great article for this magazine in the July 2005 issue on MethodInfo caching and Reflection performance which I highly recommend.

Hopefully, this example has demonstrated that adding some reflection to your application or framework can add a lot of flexibility for later customization or extensibility. Admittedly, using reflection can be somewhat cumbersome when compared to the equivalent logic in your native programming language. If you feel that adding reflection-based late binding to your code is too much grief for you or your customers (because, after all, they must describe their types and code to your framework somehow), it might be possible to strike a balance with just the right amount of flexibility.

Efficient Type Processing for Serialization

Now that we’ve walked through the basics of .NET Reflection with a few examples, let’s tackle a real-world scenario. If your software interacts with other systems via Web services or another out-of-process remoting technology, then it’s likely you’ve encountered serialization. Serialization is essentially the translation of a living, in-memory object into a data format suitable for transmission over the wire or storage on disk.

The System.Xml.Serialization namespace in the .NET Framework provides a powerful serialization engine with XmlSerializer, which can consume arbitrary managed objects and convert them into XML—with the option of translating the XML data back into typed object instances at a later date, a process called deserialization. The XmlSerializer class is a powerful, enterprise-ready piece of software and should be your first choice if you’re faced with serialization in your project. But for educational purposes, let’s explore how serialization (or another, similar instance of runtime-type processing) might be accomplished.

The scenario: you’re shipping a framework that needs to consume object instances of arbitrary user types and convert them into some intelligent data format. For instance, suppose you have an in-memory object of type Address like this:

(pseudocode)
class Address
{
    AddressID id;
    String Street, City;
    StateType State;
    ZipCodeType ZipCode;
}

How do you generate an appropriate data representation for later consumption? Perhaps just a simple textual rendering will do:

Address: 123
    Street: 1 Microsoft Way
    City: Redmond
    State: WA
    Zip: 98052

If you have full knowledge of the formal data types you’ll need to translate ahead of time (such as when you’re writing the code), then it becomes very easy:

foreach(Address a in AddressList)
{
    Console.WriteLine(“Address:{0}”, a.ID);
    Console.WriteLine(“\tStreet:{0}”, a.Street);
    ... // and so on
}

However, things get interesting when you have no prior knowledge of the types you’ll need to interact with at run time. How do you author general framework code like this?

MyFramework.TranslateObject(object input, MyOutputWriter output)

First, you’ll need to decide which type members are of interest for serialization. Possibilities include only capturing members of certain types, such as primitive system types, or providing a mechanism by which the type author can describe which members need to be serialized, such as by using custom attributes as markup on type members). You could only capture members of certain types, such as primitive system types, or the type author can describe which members need to be serialized (possibly by using custom attributes as markup on type members).

Once you’ve documented which data structure members will be translated, you need to code logic that can enumerate and retrieve them from an incoming object. Reflection does the heavy lifting here by allowing you to query both data structure and data value.

For sake of simplicity, let’s design a lightweight translation engine that takes an object, grabs the values of all its public properties, converts them to strings by directly calling ToString, and then proceeds to serialize these values. The algorithm will look something like this, for a given object named input:

  1. Call input.GetType to retrieve a System.Type instance describing the input’s underlying structure.
  2. Use Type.GetProperties with an appropriate BindingFlags parameter to retrieve the public properties as PropertyInfo instances.
  3. Use PropertyInfo.Name and PropertyInfo.GetValue to retrieve the properties as key-value pairs.
  4. Call Object.ToString on each value to convert it (primitively) to string format.
  5. Package the object type’s name along with the collection of property names and string values into the correct serialization format.

This algorithm simplifies matters considerably, but captures the essence of what is necessary to take a runtime data structure and convert it into self-describing data. However, there’s a catch: performance. As mentioned earlier, reflection is costly for type processing and value retrieval. For this example, I perform the full type analysis per every instance of the type that’s supplied.

What if you could somehow capture or persist your understanding of the type’s structure, so that later on you could trivially retrieve it and efficiently process new instances of that type—in other words, skip ahead to Step #3 in the example algorithm? The good news is that this is entirely possible using functionality from the .NET Framework. Once you understand the type’s data structure, you can use CodeDom to generate code on the fly that is bound to that data structure. You’ll generate a helper assembly containing a helper class and method which references the incoming type and accesses its properties directly (like any other reference in managed code) and thus the performance hit for type inspection is paid just once.

Now I’ll amend the algorithm. For a new type:

  1. Acquire a System.Type instance corresponding to that type.
  2. Use the various accessors of System.Type to retrieve the schema (or at least the subset of the schema that is of interest for serialization) such as property names, field names, and so on.
  3. Use the schema information to generate a helper assembly (via CodeDom), which is linked against the new type and processes instances efficiently.
  4. Use code in the helper assembly to extract the instance’s data.
  5. Serialize the data as appropriate.

For all incoming instances of a given type, you can skip ahead to Step #4 for a tremendous performance boost when compared to explicitly inspecting for every instance.

I’ve developed a basic serialization library called SimpleSerialization that implements this algorithm with reflection and CodeDom (it’s available as part of the download for this column). The main component is a class called SimpleSerializer, which the user constructs with a System.Type instance. Within the constructor, the new instance of SimpleSerializer analyzes the provided Type and generates a temporary assembly with a helper class. This helper class is tightly bound to the provided data type and can process instances just as if you’d written the code yourself with full knowledge of the type ahead of time.

The SimpleSerializer class has the following layout:

class SimpleSerializer
{
    public class SimpleSerializer(Type dataType);

    public void Serialize(object input, SimpleDataWriter writer);
}

Amazingly simple! The constructor does most of the heavy lifting: it uses Reflection to analyze the type’s structure and then generate a helper assembly with CodeDom. The SimpleDataWriter class is just a data sink for illustrating the common serialization pattern.

For serializing instances of a simple Address class, the following pseudocode gets the job done:

SimpleSerializer mySerializer = new SimpleSerializer(typeof(Address));
SimpleDataWriter writer = new SimpleDataWriter();
mySerializer.Serialize(addressInstance, writer);

Finishing Up

I highly encourage you to play around with the sample code, especially the SimpleSerialization library. I’ve added comments throughout the interesting parts of SimpleSerializer and it’s my hope that it will prove informative. Of course, if you’re faced with the need to do serious serialization in production code, then definitely rely the technologies provided in the .NET Framework (such as XmlSerializer). But if you find yourself needing to consume arbitrary types at run time and process them efficiently, I hope you’ll be able to adopt my SimpleSerialization library for your own scenario.

I’d like to thank CLR developers Weitao Su (Reflection) and Pete Sheill (CodeDom) for their guidance and feedback.

Send your questions and comments to clrinout@microsoft.com.

Mike Repass is a Program Manager on the .NET Framework CLR team. He works on Reflection, CodeDom, and various parts of the execution engine. You can reach him on his blog at blogs.msdn.com/mrepass.


Data Points: SQL Server Management Objects

Data Points
SQL Server Management Objects
John Papa

Code download available at: DataPoints2007_06.exe (173KB)
Contents
SMO Essentials
Listing the Servers
Connecting to a Server
Listing Databases
Issuing a DDL or DML Command
Iterating Lists of Objects
Backing Up a Database
Restoring a Database
Verifying a Database Backup
Wrapping Up

Not only must database developers query and manipulate data, but they must also perform administrative tasks on a regular basis. SQL Server™ Management Objects (SMO) offer developers a robust toolset for operations such as backing up and restoring databases, and issuing Data Definition Language (DDL) commands. Using SQL SMO you can also connect to SQL Server, iterate through a collection of database objects and perform a variety of tasks against them.

In this column, I discuss how SMO can be used to examine a database’s objects and to perform a variety of administrative tasks. Specifically, I cover topics such as how to design your project to use SMO and how to connect to a server. I also walk through a sample application that issues DDL or Data Manipulation Language (DML) commands against the database. Finally, I demonstrate how to use SMO to perform database backups, restores, and backup verification.

SMO Essentials

SMO is a Microsoft® .NET Framework library of objects that let you manage one or more database servers. It is designed to work with the new features that SQL Server 2005 introduced, but it will also connect to SQL Server 2000 databases with support for their respective legacy features. However, some of the SMO classes, properties and methods only work with SQL Server 2005 to support the new features like SQL Service Broker and snapshot isolation.

While the COM-based SQL Distributed Management Objects (SQL-DMO) still ships with SQL Server 2005, it has not been updated to include direct access to the newest SQL Server 2005 features as has SMO. When designing a .NET application that will manage a SQL Server database, it is recommended that you choose SQL SMO over using COM interop with DMO. You should also consider upgrading applications that currently use DMO to instead use SMO.

You can install SMO during the installation of Microsoft® SQL Server 2005 by checking the option to install the Client Tools. Then, when creating a project that will use SMO, you first reference the appropriate assemblies. The two main DLLs you must reference are Microsoft.SqlServer.Smo.dll and Microsoft.SqlServer.ConnectionInfo.dll. The former contains the core SMO classes (such as Database, Server, Backup, and Restore) while the latter contains the library of classes that help manage the connections to the SQL Server objects. In many cases, you’ll also need to reference Microsoft.SqlServer.SqlEnum.dll, which mostly contains useful enumerations, and Microsoft.SqlServer.SmoEnum.dll, which provides the Urn class for programmatic access to Uniform Resource Name (URN) addresses that uniquely identify Microsoft SQL Server objects.

Listing the Servers

Using SMO you can retrieve a list of available SQL Server installations on a network. You can specify arguments that control whether to search the network for SQL Server or to just search locally. The SmoApplication class exposes a static method named EnumAvailableSqlServers. When passed a Boolean value of true, the method will only search for and return all local servers. Passing a value of false to this method will cause it to search for all available servers on the network. The following code will get a list of SQL Server instances available on the network and return them to an ADO.NET DataTable:

DataTable dtServers = SmoApplication.EnumAvailableSqlServers(false);
foreach (DataRow row in dtServers.Rows)
{
  string sqlServerName = row[“Server”].ToString();
  if (row[“Instance”] != null && row[“Instance”].ToString().Length > 0)
      sqlServerName += @”\” + row[“Instance”].ToString();
}

It then iterates through the rows of the DataTable and formats the name of the SQL Server and its instance (if instances exist). Unfortunately, this method can be unreliable as it relies on UDP broadcasts combined with a timeout, so you may have inconsistent or incomplete results.

Connecting to a Server

The next several examples are all part of the application that is available for download or for browsing on the MSDN® Magazine Web site. When building an application with SMO, one of the first steps is often to connect to a SQL Server instance. There is nothing out of the ordinary here; establishing a connection requires that you pass the name of the SQL Server (and its instance name if multiple instances exist) and the login credentials. If you want to connect using Windows® Authentication, you can specify this using SMO. However, if you want to connect using SQL Authentication, you must pass a valid login name and password.

Figure 1 shows the sample application where a user can enter the name of a SQL Server instance and his credentials. In this figure, I entered the name of an instanced SQL Server and told it to use Windows Authentication to connect to the server. When I clicked the connect button, the list of databases was populated.

Figure 1 Connecting to a Server via SMO
Figure 1 Connecting to a Server via SMO(Click the image for a smaller view)
Figure 1 Connecting to a Server via SMO
Figure 1 Connecting to a Server via SMO (Click the image for a larger view)

I created a class called SMOHelper in the sample application to assist with some of the common routines that my application will use. An instance of the SMOHelper class can be used to interact with a single SQL Server. It has a constructor that accepts the information required to connect to a server either through Windows or SQL Server Authentication. Figure 2 shows the SMOHelper constructor grabbing and setting the appropriate class properties, which will be used by the methods of this class to perform database operations.

Figure 2 Accepting the Server Connection Information (Close Figure 2)
public SMOHelper(string serverAndInstanceName,
      string userName, string password,
      bool useWindowsAuthentication)
{
  this.server = new Server();
  int slashPos = serverAndInstanceName.IndexOf(‘\\’);
  if (slashPos >= 0)
  {
    this.serverName = serverAndInstanceName.Substring(0, slashPos);
    this.instance = serverAndInstanceName.Substring(slashPos + 1);
  }
  else
  {
    this.serverName = serverAndInstanceName;
    this.instance = string.Empty;
  }
  this.userName = userName;
  this.password = password;
  this.useWindowsAuthentication = useWindowsAuthentication;
}

A connection to a server is made by creating an instance of the Server class and setting its ConnectionContext class’s properties. The code in Figure 3 sets the connection based upon the type of authentication that the user selected on the form. While you can explicitly call the ConnectionContext’s Connect method, this isn’t actually necessary, as SMO automatically takes and releases connections from a pool. See blogs.msdn.com/mwories/archive/2005/05/02/smoconnections.aspx for more information.

Figure 3 Connecting to a Server (Close Figure 3)
public void Connect()
{
    server.ConnectionContext.ServerInstance = ServerAndInstanceName;
    server.ConnectionContext.LoginSecure = useWindowsAuthentication;
    if (!this.useWindowsAuthentication)
    {
        server.ConnectionContext.Login = this.userName;
        server.ConnectionContext.Password = this.password;
    }
    // server.ConnectionContext.Connect();
}

Listing Databases

Once connected to the server, the list of databases on that server can be retrieved through the Server object’s Databases property. In my sample application, I used data binding to bind the list of databases on the server to a combobox control. I iterated through the DatabaseCollection and created a List<string> to contain the names of the databases. This list is then bound to the combobox control where the user can select a database and perform an action on it. The following code snippet loops through the databases and builds the list of name strings:

public List<string> GetDatabaseNameList()
{
    List<string> dbList = new List<string>();
    foreach (Database db in server.Databases)
        dbList.Add(db.Name);
    return dbList;
}

In the code snippet shown above, I only wanted a list of database names, so I simply created a List<string>. However, the combobox control could just as easily have been bound to a list of a custom classes (like List<MyDatabase>) that wrap the functionally rich objects that are typically returned from many SMO methods.

Issuing a DDL or DML Command

Like ADO.NET, SMO can be used to execute commands on a database to retrieve or modify data using standard DML. While SMO is not optimized to handle complex DML, it can be used to perform simple DML statements.

The form in Figure 1 shows a textbox where a user can enter a DML or DDL statement and then execute it. If the query should retrieve a rowset (that is, it is a SELECT statement), SMO will retrieve the rowset via its ExecuteWithResults method. Otherwise, if the command does not return a rowset (INSERT, UPDATE, or DELETE) SMO will execute the command using its ExecuteNonQuery method. The following code sample executes a query and returns the rowset back into an ADO.NET DataSet. The DataSet is then bound to a DataGridView on the form:

  DataSet ds = db.ExecuteWithResults(txtSQL.Text);
  dgvResults.DataSource = ds.Tables[0];

The next code snippet will execute a non-result-returning command against the database object:

  db.ExecuteNonQuery(sql);

The commands that you use do not have to be DML commands, they can also be DDL commands. For example, you can execute an ALTER TABLE, CREATE PROCEDURE, RESTORE or even DBCC CHECKDBB command using the ExecuteNonQuery method. For example, the following command could be used to create a Person table:

    string sql = “CREATE TABLE Person “ +
       “(PersonID INT NOT NULL PRIMARY KEY,
       FullName VARCHAR(100))”;
    db.ExecuteNonQuery(sql);

Iterating Lists of Objects

SMO can also be used to locate a specific database object and perform a context-sensitive operation on it. SMO’s Database class exposes a series of collections of each of the objects that it contains. Figure 4 shows a partial list of the different collections that the Database class exposes.

Figure 4 Collections (Close Figure 4)

Defaults
FileGroups
Roles
Rules
Schemas
StoredProcedures
Tables
Triggers
UserDefinedFunctions
Users
Views

Figure 5 shows the example application displaying all of the Tables, Stored Procedures and Users in the Northwind database on my development computer. Getting a list of the tables that exist in a specific database is easy enough. Once you have the Server object and you get the Database object from the Server object you want, you can access the TablesCollection through the Tables property. Figure 6 shows how to access the Database’s collections of objects, loop through them, and build a custom object of your own. In this code, I simply grab the name of the object and put it in a List<string>, which I then use to bind to the listbox controls on the SmoLists Form (shown in Figure 5).

Figure 6 Retrieving Lists (Close Figure 6)
public List<string> GetTableNameList(Database db)
{
    List<string> tableList = new List<string>();
    foreach (Table table in db.Tables) tableList.Add(table.Name);
    return tableList;
}

public List<string> GetStoredProcedureNameList(Database db)
{
    List<string> storedProcedureNameList = new List<string>();
    foreach (StoredProcedure storedProcedure in db.StoredProcedures)
        storedProcedureNameList.Add(storedProcedure.Name);
    return storedProcedureNameList;
}

public List<string> GetUserNameList(Database db)
{
    List<string> userNameList = new List<string>();
    foreach (User user in db.Users) userNameList.Add(user.Name);
    return userNameList;
}
Figure 5 Database Lists
Figure 5 Database Lists(Click the image for a smaller view)
Figure 5 Database Lists
Figure 5 Database Lists (Click the image for a larger view)

Of course, there are many more properties and methods contained within these objects besides the Name property. For example, you can also execute methods such as Revoke or Grant on a Table object that will set permissions for the Table.

Backing Up a Database

One of the more common tasks that developers use SMO for is to backup a database. SMO makes it relatively simple to perform a database backup using its Backup class. Figure 7 shows the sample Form that will allow you to back up a database and then restore it.

Figure 7 Backup and Restore
Figure 7 Backup and Restore(Click the image for a smaller view)
Figure 7 Backup and Restore
Figure 7 Backup and Restore (Click the image for a larger view)

When you run the application and click the Backup button, you may notice the progress bar. I added a ProgressBar control to the Form in order to give the user some feedback on the status of the backup or restore operation. The SMO Backup class exposes some events that can be used to help track the progress of a backup. (I will examine these momentarily.)

When a user clicks the Backup button, the btnBackup_Click event handler executes, as you see here:

private void btnBackup_Click(
   object sender, EventArgs e)
{
   pbDBAction.Maximum = 100;
   pbDBAction.Style = ProgressBarStyle.Blocks;
   pbDBAction.Value = 0;
   pbDBAction.Step = 10;
   string databaseName = cboDatabase.SelectedValue.ToString();
   smoHelper.BackupDatabase(databaseName);
}

First, I set the Progressbar control’s properties so that it will appear as a set of blocks and so that it will start out empty. Next, I grab the name of the database from the selection in the combobox control and pass that to the BackupDatabase method that I created in Figure 8 for the SMOHelper class.

Figure 8 Backing Up a Database (Close Figure 8)
public void BackupDatabase(string databaseName)
{
    Console.WriteLine(“*** Backing up ***”);

    Backup backup = new Backup();
    backup.Action = BackupActionType.Database;
    backup.Database = databaseName;
    backup.Incremental = false;
    backup.Initialize = true;
    backup.LogTruncation = BackupTruncateLogType.Truncate;

    string fileName = string.Format(“{0}\\{1}.bak”,
      testFolder, databaseName);
    BackupDeviceItem backupItemDevice =
      new BackupDeviceItem(fileName, DeviceType.File);
    backup.Devices.Add(backupItemDevice);
    backup.PercentCompleteNotification = 10;
    backup.PercentComplete += backup_PercentComplete;
    backup.Complete += backup_Complete;

    backup.SqlBackup(server);
}

In BackupDatabase, I first create an instance of the Backup class. Then I set the Action property to the BackupActionType.Database enumeration value (you can also backup Logs and Files). Of course, I then set the name of the database to backup and set a handful of other properties. Since I want a full database backup, I instruct the backup not to be an incremental backup by setting the Incremental property to false.

When backing up a database, you need to specify the file path location and filename of where you want to create the backup file. In my code, I put all backups in the same folder for simplicity (see the testFolder variable). Once I choose where to create the backup file, I create a BackupDeviceItem and add it to the Backup class’s Devices list. I chose to save the backup as a file, but you can save the backup to other destinations such as a LogicalDevice or a Tape as well.

I added an event handler to the PercentComplete event so that the backup process would notify the calling Form and its ProgressBar control. This event occurs at an interval defined by the PercentCompleteNotification property. In my sample application, this is what makes the ProgressBar control show progress. I also added an event handler to the Complete event so that the Form can be notified when the backup has completed. (See the sample application for details on how the events are hooked together.)

Finally, I issue the SqlBackup method, which is what actually performs the database backup task. Keep in mind that backing up a database may take longer than you expect depending on a number of factors, including the size of the database and its contents.

Restoring a Database

SMO exposes a Restore class that can be used to restore a database from a file or from another backup device. In the sample application, I restore the database that we just backed up, and the process is similar. Note that I don’t restore it to the database from where it was backed up but rather I restore the database to a new file name and a new logical name. In essence, this creates a copy of the database using backup methods. Of course, you can choose to restore a database over the existing database, but this process will come in handy.

The code that appears in Figure 9 displays the RestoreDB method that I use in my sample application to restore the database. My first step is to create an instance of the Restore class and then I set the name of the backup file that I will be restoring from.

Figure 9 Restoring a Database (Close Figure 9)
public void RestoreDB(string databaseName)
{
    Restore restore = new Restore();
    string fileName = string.Format(“{0}\\{1}.bak”,
        testFolder, databaseName);
    restore.Devices.Add(new BackupDeviceItem(fileName, DeviceType.File));

    // Just give it a new name
    string destinationDatabaseName =
        string.Format(“{0}_newly_restored”, databaseName);

    // Go grab the current database’s logical names for the data
    // and log files. For this example, we assume there are 1 for each.
    Database currentDatabase = server.Databases[databaseName];
    string currentLogicalData =
        currentDatabase.FileGroups[0].Files[0].Name;
    string currentLogicalLog = currentDatabase.LogFiles[0].Name;

    // Now relocate the data and log files
    RelocateFile reloData =
        new RelocateFile(currentLogicalData,
        string.Format(@”{0}\{1}.mdf”, testFolder,
            destinationDatabaseName));
    RelocateFile reloLog =
        new RelocateFile(currentLogicalLog,
        string.Format(@”{0}\{1}_Log.ldf”, testFolder,
            destinationDatabaseName));
    restore.RelocateFiles.Add(reloData);
    restore.RelocateFiles.Add(reloLog);

    restore.Database = destinationDatabaseName;
    restore.ReplaceDatabase = true;

    restore.PercentCompleteNotification = 10;
    restore.PercentComplete += restore_PercentComplete;
    restore.Complete += restore_Complete;
    restore.SqlRestore(server);
}

Next, I create a new name for the database that I will restore (Northwind_newly_restored) and I get the names of the logical data and log files from the FileGroups and the Files collections. I need these names so that I can relocate the physical files for the backup to a new name (since I am restoring the database to a new and separate database). I then add an instance of the RelocateFile class (one for each file that I am relocating) to the restore object’s RelocateFiles collection.

I set the name of the database to restore and tell SMO to replace the existing database, if one already exists, with the same name. Since I want the ProgressBar to show the progress of the restore operation, I set the PercentComplete and Complete event handlers accordingly.

Verifying a Database Backup

It won’t do you much good to have a database backup if it is corrupted. Verifying a backup is a good idea, and can easily be done by creating an instance of the Restore class, setting the DeviceType to its appropriate medium, adding the device and executing the SqlVerify method. This will return a Boolean value indicating whether or not the backup is verified. The code below demonstrates how to perform this verification:

Restore restore = new Restore();
restore.DeviceType = DeviceType.File;
restore.Devices.Add(@”c:\MySMOTest.bak”);
boolean verified = restore.SqlVerify(server,out msg);

Wrapping Up

SMO makes it easy to perform database administration operations in .NET code. While all of these tasks can be performed with T-SQL directly on a database or through the SQL Server Management Studio, being able to use SMO offers broad flexibility to include these types of operations in .NET applications. In fact, SQL Server Management Studio relies on SMO for most of its management tasks.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a professional consultant, trainer and mentor using Microsoft .NET technologies. A Microsoft C# MVP, John has authored several articles and books on architecture and data access technologies including ADO.NET, XML, and SQL Server. He is also a frequent speaker at industry conferences such as VSLive. Get in touch with John at www.johnpapa.net.


Advanced Basics: Using RibbonX from Visual Basic

Advanced Basics
Using RibbonX from Visual Basic
Ken Getz

Contents
Getting Ready
Creating a Sample Add-In
Create a Custom Color Gallery
Create GetEnabled
Create GetItemCount
Create GetItemHeight and GetItemWidth
Create the Images
Create the Screen Tip and Test the Add-In
Add Action Callbacks
Adding Ribbon Control Interaction

In the February 2007 issue of MSDN Magazine, Eric Faller demonstrated how to create extensions for the 2007 Microsoft® Office system using the RibbonX API and XML markup (see msdn.microsoft.com/msdnmag/issues/07/02/RibbonX). In the same issue, Paul Stubbs introduced managed add-ins for 2007 Office system applications (specifically, for PowerPoint® 2007) and showed how you can create them using Visual Studio® 2005 Tools for Office Second Edition. In this column, I’ll continue their excellent work, trying to anticipate some questions you’ll likely encounter as you work with RibbonX and create your own Ribbon customizations.

Keep in mind that because neither the 2007 Office system nor Visual Studio 2005 provides a real designer for the Ribbon, creating customizations for it requires a little extra effort. There’s no pretty user interface for creating your Ribbon customizations, and everything you do happens in XML, plus either .NET-compatible or Visual Basic® for Applications (VBA) code. (The next release of Visual Studio Tools for Office does provide a designer for the Ribbon. For more information, see the article by Steve Fox and Paul Stubbs in this issue.) For more on the 2007 Office system Ribbon in general, stop by msdn.microsoft.com/office/tool/ribbon. Here you’ll find the most up-to-date information available from Microsoft on working with and customizing, the 2007 Office system Ribbon.

Getting Ready

To work with the various controls on the Ribbon, you need to know the identifier (that is, the msoID attribute) corresponding to each Ribbon element. Download "2007 Office System Document: List of Control IDs". This download (available in either Excel® 2003 or Excel 2007 format) includes a series of spreadsheets containing full disclosure for each Ribbon element. Without this information, modifying the existing Ribbon is next to impossible. Note that it is also possible to get the ID of any control by hovering over it in the Customization pane in the Options dialog.

Make sure you have a copy of the XML schema that describes RibbonX on your development computer. With this schema (named customUI.xsd), the Visual Studio 2005 XML editor can provide IntelliSense® as you create the Ribbon customization markup. You can download the "2007 Office System: XML Schema Reference". The download includes both the customUI.xsd file and the reference materials for this and other schemas that the 2007 Office system uses. Once you’ve downloaded and extracted customUI.xsd, copy it into the %Program Files%\Microsoft Visual Studio 8\Xml\Schemas folder. (Replace %Program Files% in the previous path with the location of the Program Files folder on your own computer.) With the schema file installed in this folder, Visual Studio 2005 can make the schema available to you when you create a new Ribbon customization file.

If you want to store the Ribbon customization as part of a Word 2007, Excel 2007, or PowerPoint 2007 document, handling Ribbon callbacks in VBA code, you generally must crack open the document using a ZIP file-handling tool, create the customization part, and set up the necessary relations to the part. This is a lot of work, and you can avoid it by downloading the Custom UI Editor Tool from OpenXMLDeveloper.org. Although the tool won’t help you create Ribbon customizations, it will help you in several important ways: it handles inserting the customization into a Word, Excel, or PowerPoint document or template, setting up the necessary parts and relations; it can insert icons into the document for your customization; it validates the RibbonX markup against customUI.xsd; and it can generate VBA-style callback handler stubs so you don’t need to look up the exact format of the callback procedures. Even if you’re creating a Visual Studio Tools for Office (VSTO) add-in, or a shared Visual Studio COM add-in, the last two features can be useful.

Of course you’ll need documentation. MSDN® supplies a set of three articles online, coauthored by Frank Rice and me, that offer examples, references, and answers to frequently asked questions. These articles provide a good start until Microsoft supplies complete help files for RibbonX in one place. You’ll find the articles at Part 1, Part 2, and Part 3.

Finally, you need to consider where you’re going to store your Ribbon customization. As described in the first of the three articles, you have several options. You can store the RibbonX markup in a Word, Excel, or PowerPoint document directly, creating a custom UI part and setting up relations to the part; and you can use the CustomUI Editor Tool I mentioned earlier to help insert the custom UI into the document. For customizations to the Access™ 2007 Ribbon, you can store the XML markup in a table, and you can retrieve and activate the customization using code. You can also store the customization in the USysRibbons table, and Access can load it as your database loads. Each of these alternatives creates a document or template-based customization.

If you want to create an application-based customization, you’ll need to think about creating a COM add-in for 2007 Office system applications; to do so, you can either create an add-in using the Visual Studio 2005 Shared Add-in template, or with Visual Studio 2005 Tools for Office Second Edition (this is the simpler, more robust approach). Again, the first MSDN article demonstrates how to use both of these techniques.

Given all the tools and documentation, you’re ready to create your own RibbonX customizations. Though you may not use them all as you work through this column, they’ll prove invaluable when you create your own customizations.

Because the XML markup is the same no matter where you store the content, I’ve chosen the simplest solution that allows you to handle Ribbon callbacks using managed code—that is, I’ll create an add-in for Excel 2007 using VSTO 2005 SE. (If you don’t have VSTO 2005 SE installed, you can download it and get more information from the product’s portal at msdn2.microsoft.com/aa905543.aspx.)

Creating a Sample Add-In

To get started, you’ll need to create and test a simple add-in using VSTO 2005 SE. I’ll quickly summarize the instructions provided in the first of the three MSDN articles I mentioned, creating an Excel add-in rather than a Word add-in. For more detailed explanations, see the full article.

First, create a new project in Visual Studio 2005. In the New Project dialog box, in the Project Types pane, expand the Office node and select 2007 Add-ins. In the Templates pane, select Excel Add-in. Name the new add-in RibbonAddin, and then click OK to create it. (Note that the add-in template creates both the add-in project, and a setup project. For now, disregard the setup project. It makes it easier for you to deploy your add-in but won’t be necessary for this demonstration.)

On the Project menu, click Add New Item and in the dialog box, select Ribbon Support. Click Add to accept the default name, Ribbon1.vb. In addition to the Ribbon1.vb file, the template adds an XML file named Ribbon1.xml, which you’ll modify later. On the Project menu, click RibbonAddin Properties. Click the Resources tab and, from Solution Explorer, drag Ribbon1.xml onto the Resources design surface. Close the Resources window and save the resources.

In Solution Explorer, double-click Ribbon1.xml to view its contents. The default markup, shown in Figure 1, creates a new group labeled My Group on the AddIns tab. The group contains a toggle button labeled My Button. Pressing the button calls the OnToggleButton1 callback procedure.

Figure 1 Ribbon1.xml (Close Figure 1)
<customUI
  xmlns=http://schemas.microsoft.com/office/2006/01/customui
  onLoad=”OnLoad”>
  <ribbon>
    <tabs>
      <tab idMso=”TabAddIns”>
        <group id=”MyGroup”
               label=”My Group”>
          <toggleButton id=”toggleButton1”
                        size=”large”
                        label=”My Button”
                        screentip=”My Button Screentip”
                        onAction=”OnToggleButton1”
                        imageMso=”HappyFace” />
        </group>
      </tab>
    </tabs>
  </ribbon>
</customUI>

In Solution Explorer, within the Ribbon1.vb file, uncomment the ThisAddIn partial class. In the Ribbon1 class, modify the GetCustomUI procedure so it returns the value of the new resource like so:

Return My.Resources.Ribbon1

The provided GetResourceText helper procedure opens the XML content manually and retrieves the text using a reader. You can remove the procedure at this point, if you want.

In most cases, you’ll need to be able to refer to the host application from within the Ribbon customization, so in the Ribbon1 class, add the following declaration:

Public Application As Excel.Application

You also need to initialize this variable, so in the ThisAddIn class’s RequestService method, add the following line of code immediately after the procedure creates the instance of the Ribbon1 class:

ribbon.Application = Application

Expand the Ribbon Callbacks region in the code, and examine the OnLoad callback procedure:

Public Sub OnLoad(ByVal ribbonUI As Office.IRibbonUI)
    Me.ribbon = ribbonUI
End Sub

This procedure takes the Office.IRibbonUI value passed to the Ribbon1 instance and stores it away so that later callbacks can interact with the Ribbon as well. Without this step, you wouldn’t be able to perform important operations, such as refreshing the Ribbon after you change the contents of an item.

In addition, you’ll find the sample callback procedure that handles pressing and releasing the toggle button created by the XML markup. It simply displays an appropriate alert each time you press or release the new toggle button:

Public Sub OnToggleButton1( _
  ByVal control As Office.IRibbonControl, _
  ByVal isPressed As Boolean)
    If isPressed Then
        MessageBox.Show(“Pressed”)
    Else
        MessageBox.Show(“Released”)
    End If
End Sub

Although I’ve modified the instructions slightly, you should be able to follow along with the original article if you want a more complete explanation. For now, save and run your project, verifying that Visual Studio launches Excel 2007. Select the Add-Ins tab, and press and then release the My Button toggle button. You’ll see the alert displayed by the add-in, in each case.

For the rest of the demonstrations in this column, you’ll modify the content in Ribbon1.xml and add more complex callback procedures to the Ribbon1 class. Though there are probably an infinite number of things you can do with the Ribbon, we have space limitations so I’ll demonstrate just two fun uses of the API.

First, I’ll assume you want to display your own gallery, containing a subset of the available colors, so that users can fill the background of a cell. To do so you’ll create your own color gallery. Figure 2 shows how it will look. Second, I’ll show how you can control whether your color gallery is available using a checkbox, as in the upper-left hand corner of Figure 2.

Figure 2 Create Your Own Gallery Using RibbonX
Figure 2 Create Your Own Gallery Using RibbonX

Create a Custom Color Gallery

Although users can select fill colors from the Excel FillColor gallery, as shown in Figure 3, you might want to limit the color choices to a specific subset. To accomplish this, you can create your own color gallery. If you examine the Excel FillColor gallery, however, you might notice that getting to the gallery is simpler than the one in Figure 2; the difference is that the Excel gallery appears when you click the SplitButton control, but the gallery you create must hang off a menu. It would be nice if you could create a gallery directly as a child of a split button, but based on the customui.xsd schema, a SplitButton control can only contain a button or a toggle button, along with a menu. The sample you’ll create hangs the gallery control off a simple menu.

Figure 3 Excel FillColor Gallery
Figure 3 Excel FillColor Gallery

To get started using the add-in you created, replace the content of Ribbon1.xml with the markup in Figure 4. You may find it instructive to actually type this code from scratch, rather than copying it in from the online version of this column. It’s useful to become familiar with the way IntelliSense helps you as you create RibbonX markup. This code creates the customization, calling the onLoad procedure you already created as it starts. In addition, this markup creates a new tab named demoTab, displaying the text "Ribbon Customization". Within the tab, the markup creates a group labeled "gallery Control", and within the group, a splitButton control containing a button and a menu. The button uses the built-in CellFillColorPicker ID to provide the image. I got the actual ID for this control by perusing the workbook containing Excel Ribbon control IDs I downloaded from the Microsoft site mentioned earlier. At this point, save and run the add-in to verify the behavior.

Figure 4 New Contents of Ribbon1.xml (Close Figure 4)
<?xml version=”1.0” encoding=”utf-8” ?>
<customUI
   xmlns=http://schemas.microsoft.com/office/2006/01/customui
   onLoad=”onLoad”>
  <ribbon>
    <tabs>
      <tab id=”demoTab” label=”Ribbon Customization” >
        <group id=”galleryGroup” label=”gallery Control”>
            <splitButton id=”highlightSplitButton” >
              <button id=”highlightButton”
                imageMso=”CellFillColorPicker”/>
              <menu id=”highlightMenu” >
              </menu>
            </splitButton>
        </group>
      </tab>
    </tabs>
  </ribbon>
</customUI>

Continue the modification of the markup by inserting the element in Figure 5 into the body of the <menu> element. This markup creates the gallery control and makes use of almost all of the control’s available callback procedures. First, the markup specifies the label, and indicates that the control shouldn’t display item labels (you just want the colors to appear without labels for this example). In addition, the code specifies the number of rows and columns to be displayed within the gallery. The markup designates the remainder of the gallery’s properties by specifying callback procedures—as Excel 2007 renders the Ribbon customization, it calls each of the procedures listed here as it needs to retrieve data. (For example, it calls the GetItemWidth method when it needs to determine the width of an item.) In addition, the markup specifies a button to be displayed at the bottom of the gallery. You can have as many of these buttons as you like, but this example requires only one.

Figure 5 Gallery Control (Close Figure 5)
<gallery id=”highlightGallery”
  label=”Select Color”
  showItemLabel=”false”
  columns=”5” rows=”3”
  onAction=”InsertFillColor”
  getEnabled=”GetEnabled”
  getItemCount=”GetItemCount”
  getItemHeight=”GetItemHeight”
  getItemWidth=”GetItemWidth”
  getItemImage=”GetItemImage”
  getItemScreentip=”GetItemScreenTip”>
  <button id=”noColorButton”
    label=”No color”
    onAction=”RemoveFillColor”
    screentip=”Remove color”
    supertip=”Remove any applied color.”/>
</gallery>

In order to implement the gallery control’s behavior, you must now supply each of the callback procedures listed in the markup. To do this, open Ribbon1.vb in the code editor window and find the Ribbon Callbacks region. While you don’t have to place the code in this region, it helps organize your code. In the following paragraphs I’ll walk you through the creation of each of the required callback procedures.

Note that you don’t need to use any of these callback procedures. You can, instead, supply static values for each of the various gallery control properties. That is, rather than specifying the getItemHeight attribute in the markup, you can instead supply the itemHeight attribute along with a fixed value. This example uses code for each of these properties, so you can dynamically vary the layout of the gallery.

In order to display the gallery, you’ll need a list of colors, so insert the following code into the Ribbon1 class:

Private image As New Bitmap(25, 25)
Private colors As Color() = _
  {Color.Yellow, Color.GreenYellow, Color.Turquoise, _
   Color.Magenta, Color.Blue, Color.Red, Color.DarkBlue, _
   Color.Teal, Color.Green, Color.Violet, Color.DarkRed, _
   Color.YellowGreen, Color.Gray, Color.DarkGray, Color.Black}

This code creates a class-level Bitmap instance that you’ll use to create the various images and an array of Color values that represent the various fill colors. Make sure you insert this and the remainder of the code in the Ribbon1 class, not in the ThisAddIn partial class.

Create GetEnabled

In the Ribbon1.vb class, add the following callback procedure. This procedure matches the required signature for this particular callback. If your callback doesn’t match the right signature, 2007 Office system applications won’t be able to call the procedure:

Public Function GetEnabled( _
       ByVal control As Office.IRibbonControl) As _
           Boolean

     Dim returnValue As Boolean = False
     Select Case control.Id
       Case “highlightGallery”
         returnValue = True
     End Select
     Return returnValue
   End Function

To find the signature for this (and all the other) callback procedures, refer to the documentation in the third article of the three-part set mentioned earlier.

The 2007 Office system passes a reference to the control that triggered the callback as a parameter to the GetEnabled method, and the sample code uses that information to create a Select Case statement based on the caller’s Id property. Using a design pattern like this, you can create a single GetEnabled callback procedure that can work with any Ribbon control that shares the same callback procedure signature. In this case, the procedure simply returns True, indicating that the gallery should be enabled.

Create GetItemCount

The getItemCount callback allows you to calculate the number of items to display in the gallery, and you must specify this callback procedure for each gallery that you create, if you want to specify the number of items to be displayed. In your add-in, insert the following procedure in the Ribbon1 class:

Public Function GetItemCount( _
 ByVal control As Office.IRibbonControl) As Integer
  Dim returnValue As Integer = 0

  Select Case control.Id
    Case “highlightGallery”
      returnValue = 15
  End Select
  Return returnValue
End Function

As before, this callback receives a reference to the control that triggered it and returns a scalar value indicating the number of items to be displayed in the gallery.

If you don’t like the technique of including a Select Case block in each callback, using the Id property of the calling control to determine the behavior of the procedure, you can just as easily create a separate callback procedure for each control that shares a callback. I like consolidating behavior this way, but it’s certainly not required.

Create GetItemHeight and GetItemWidth

Your code specifies the height and width of each item in the gallery. Each item must be the same size, and the Bitmap instance you created earlier controls that size. Therefore, add the callback procedures in Figure 6 to the Ribbon1 class to return the correct dimensions to Excel.

Figure 6 GetItemHeight and GetItemWidth (Close Figure 6)
Public Function GetItemHeight( _
 ByVal control As Office.IRibbonControl) As Integer

  Dim returnValue As Integer = 0
  Select Case control.Id
    Case “highlightGallery”
      returnValue = image.Height
  End Select
  Return returnValue
End Function

Public Function GetItemWidth( _
 ByVal control As Office.IRibbonControl) As Integer

  Dim returnValue As Integer = 0
  Select Case control.Id
    Case “highlightGallery”
      returnValue = image.Width
  End Select
  Return returnValue
End Function

Create the Images

Although you could easily load images from disk or from a resource file for each of the gallery items, this example simply needs blocks of color. The GetItemImage callback procedure thus needs to draw a solid block of the appropriate color for each gallery item. The GetItemImage callback procedure receives not only a reference to the control that triggered the callback, but also an index, indicating which image it should return. Insert the callback procedure shown in Figure 7 into the Ribbon1 class.

Figure 7 GetItemImage (Close Figure 7)
Public Function GetItemImage( _
 ByVal control As Office.IRibbonControl, _
 ByVal index As Integer) As Object

  Select Case control.Id
    Case “highlightGallery”
      Using g As Graphics = Graphics.FromImage(image)
        Using brush As New SolidBrush(colors(index))
          g.DrawRectangle(Pens.Gray, _
            New Rectangle(0, 0, image.Width - 1, image.Height - 1))
          g.FillRectangle(brush, _
            New Rectangle(1, 1, image.Width - 2, image.Height - 2))
        End Using
      End Using
  End Select
  Return image
End Function

The GetItemImage procedure creates a Graphics object from the Bitmap you created earlier and creates a new solid brush using the supplied value to index into the array of colors. The code draws a gray rectangle a little smaller than the bitmap, and then fills the rectangle with the appropriate color.

Create the Screen Tip and Test the Add-In

As you hover the mouse over each of the images, Excel should display the name of the color. The GetItemScreenTip callback procedure requests the text to be displayed for each image, and your job is to simply return the name for each color. While for simplicity I’ve chosen to use GetItemScreenTip, GetItemLabel might actually be a better option as it would allow visually impaired users to get the labels of the gallery items, even though the labels are visually hidden with the showItemLabel attribute. Items will automatically use their labels as their screen tips if no screen tips are explicitly provided. Add the following procedure to the Ribbon1 class to add this behavior:

Public Function GetItemScreenTip( _
 ByVal control As Office.IRibbonControl, _
 ByVal index As Integer) As String

  Dim returnValue As String = String.Empty
  Select Case (control.Id)
    Case “highlightGallery”
      returnValue = colors(index).Name
  End Select
  Return returnValue
End Function

At this point, you’ve provided all the callback procedures you need in order to display the gallery. Save and run your project. Visual Studio loads Excel 2007, and when you select the Ribbon Customization tab on the Ribbon, you’ll find your splitButton control. Click the dropdown arrow, select the menu item, and your gallery appears. Click any button on the gallery, however, and you’ll receive an error—you haven’t yet supplied the callback procedure that handles button clicks in the gallery. You have to quit Excel and return to Visual Studio to complete this portion of the add-in.

Add Action Callbacks

The markup for your customization includes two "action" callbacks: InsertFillColor for the gallery itself, and RemoveFillColor for the button at the bottom of the gallery. In order to avoid the errors that occur when you select a button on the gallery, or the button at the bottom of the gallery, add the following callback procedures to the Ribbon1 class:

Public Sub InsertFillColor( _
 ByVal control As Office.IRibbonControl, _
 ByVal selectedId As String, ByVal index As Integer)

  CType(Application.Selection, Excel.Range).Interior.Color = _
        ColorTranslator.ToOle(colors(index))
End Sub

Public Sub RemoveFillColor( _
 ByVal control As Office.IRibbonControl)

  CType(Application.Selection, Excel.Range). _
       Interior.Color = Excel.Constants.xlNone
End Sub

Note that the two procedures use different signatures: the first procedure handles the gallery control’s OnAction callback, and the second handles the button control’s OnAction callback. The gallery’s OnAction callback supplies not only a reference to the control and the index of the item within the gallery that you selected, but also a string containing an ID. This selectedId parameter is the ID of the selected item, whereas the control’s Id property is the property of the gallery itself.

The InsertFillColor and RemoveFillColor callback procedures are specific to this gallery, so I haven’t bothered using a Select Case block within the procedures. In addition, the two procedures convert the Application.Selection value to an Excel.Range, and set the Interior.Color property to either the selected color, or to no color at all. The ColorTranslator.ToOle method handles converting colors from the System.Drawing.Color managed type to colors that VBA and Excel understand.

To test your add-in, once again save and run the project. When you select an item from the gallery, your add-in sets the fill color for the selected range to the color you’ve selected. Clicking the button at the bottom of the gallery removes all color from the selected range.

At this point, you’ve created a simple Ribbon customization that shows off a number of the different callbacks that are available. What if you want to enable and disable the gallery, based on user input? For example, what if you want a checkbox control on the Ribbon that allows you to control the availability of the gallery? I’ll add this slightly more complex feature next.

Adding Ribbon Control Interaction

If you’re creating a Windows application, it’s simple to have a checkbox change the enabled status of another control. From the appropriate event handler, set the Enabled property of the control, and the runtime environment takes care of the details. Even when building Web applications using ASP.NET, you don’t need to worry about the details of exactly how a control’s enabled status changes—you set a property, and the ASP.NET runtime manages the control’s rendering the next time you force the page to redraw.

Unlike many other environments, however, programming Ribbon controls doesn’t allow you to interact directly with properties of controls. Instead, the best you can do is set the status of a local variable, invalidate the control whose property you want to change, and have the control’s callback procedures render the control as you request.

To demonstrate this sort of behavior, add a checkbox control to your Ribbon customization and have the checked status of the checkbox determine the enabled status of the gallery control. In the Ribbon1.xml file, within the existing group element, beneath the existing splitButton element, add the following markup:

<checkBox id=”enableColorsCheckBox” label=”Enable Color Selection?”
  screentip=”Enable color selection”
  onAction=”HandlePressedAction”
  supertip=”Select to enable or disable color selection.”
  getPressed=”GetPressed”/>

The onAction attribute indicates which callback procedure you want to call when the user checks or unchecks the checkbox, and the getPressed attribute specifies the callback to be called as the Ribbon initializes the value of the checkbox.

In the Ribbon1 class, add a variable that can maintain the current checked state of the checkbox:

Private enableColors As Boolean = False

Modify the existing GetEnabled procedure, so that rather than simply returning True, the method returns the value of the enableColors variable:

returnValue = enableColors

In order to ensure that the state of the checkbox control reflects the value of the enableColors variable, add the following callback procedure. The 2007 Office system Ribbon calls this code as it initializes the state of the checkbox control:

Public Function GetPressed( _
 ByVal control As Office.IRibbonControl) As Boolean

  Dim returnValue As Boolean = True
  Select Case control.Id
    Case “enableColorsCheckBox”
      returnValue = enableColors
  End Select
  Return returnValue
End Function

Finally, you need some way to handle what happens when the user checks or unchecks the checkbox. The trick is to both set the value of the enableColors variable and force the Ribbon to invalidate the gallery control. By invalidating the control, you force it to reinitialize, setting the enabled status of the control appropriately:

Public Sub HandlePressedAction( _
 ByVal control As Office.IRibbonControl, _
 ByVal pressed As Boolean)

  Select Case control.Id
    Case “enableColorsCheckBox”
      enableColors = pressed
      ribbon.InvalidateControl(“highlightGallery”)
  End Select
End Sub

The line of code that invalidates the gallery control shows off two important aspects of automating the Ribbon: First, if the add-in hadn’t stored away the reference to the Ribbon that it was passed when it was initialized, you wouldn’t have any way to interact with the Ribbon in later procedures. That’s why it’s crucial that your Ribbon code handle the OnLoad callback, storing the reference to the Ribbon it has passed into a class-level variable. In addition, the sample code demonstrates the InvalidateControl method of the IRibbonUI interface. You can call the InvalidateControl method, passing a specific control to invalidate, or you can call the Invalidate method (and invalidate the entire Ribbon). Either way, you must call one of these procedures in order to tip off the Ribbon to the fact that it must redraw one or all controls.

As before, save and run the project. In Excel 2007, select the Ribbon Customization tab on the Ribbon, and you’ll find the new checkbox in addition to the existing splitbutton control. Attempt to display the gallery, and you’ll find the menu item disabled. Check the checkbox, try again to display the gallery, and this time it’s available.

As you can see, interactions between Ribbon controls require a little more work than you might expect, based on the architecture of the Ribbon itself. You can reduce the size of your code if you don’t intend to add any other controls to the Ribbon—the Select Case statement in each callback procedure makes it easy to extend the code later, but it’s not required.

You can use this same technique for many different Ribbon-based scenarios. In any situation in which you need to modify the layout, behavior, or content of any control based on an "event" of a different control, use class-level variables to maintain the state, and invalidate the control or the entire Ribbon.

As much as I thought I wouldn’t like the new user interface, I’ve found I was wrong—the Ribbon provides an excellent way to present choices to users, and the declarative model is far easier to program than the old Office CommandBar model.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies, and a courseware author for AppDev. He is coauthor of ASP .NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer’s Handbook (Sybex, 2002), and VBA Developer’s Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Cutting Edge: Transactional Workflows

Cutting Edge
Transactional Workflows
Dino Esposito

Code download available at: CuttingEdge2007_06.exe (182KB)
Contents
Categorizing Transactional Tasks
ACID Transactions in Workflows
Exploring the TransactionScope Activity
Configuring a Transactional Workflow for Execution
Modeling Business Processes with Windows Workflow Foundation
About Compensation
Performance and Workflow Transactions
Conclusion

Windows® Workflow Foundation, a pillar of the Microsoft® .NET Framework 3.0, provides the programming model, runtime engine, and tools for developing workflows that can be incorporated in .NET-based applications or exposed as services to any clients it may concern. So when do you really need a workflow in a custom solution?

A workflow certainly helps when the business logic and rules are subject to frequent changes. This may happen because you customize the same application to the needs of different customers or just because the customer specifically required this type of flexibility. A workflow may lie behind a business process management (BPM) server to orchestrate business services exposed to the modules of a distributed, composite application. Finally, a workflow lends itself to support the implementation of any form of long-running, cross-company business logic.

From a developer’s perspective, a workflow is a set of activities that, taken together, express the desired behavior. When called to model some real-world business logic, a workflow inevitably ends up dealing with transactional tasks. So how would you code transactional semantics in the Windows Workflow Foundation?

Nearly all enterprise systems have a workflow inside to take care of particularly complex problems, and nearly all complex problems contain a transaction. There are two main types of transactions to handle. One type includes classic, short-lived transactions that are often referred to as ACID transactions. The other type can be described as global, long-running, business-wide transactions. This sort of transaction is often a component of a business process and may consist of multiple ACID transactions. The success or failure of these ACID transactions will contribute to the overall outcome of the larger business process that is represented by the workflow.

The Windows Workflow Foundation provides ad hoc activities to make the implementation of both ACID and business-wide transactions not only effective, but relatively easy to code and update. In this column, I’ll cover the set of activities that provide for any sort of transactional tasks you might build into a Windows Workflow Foundation workflow.

Categorizing Transactional Tasks

A Windows Workflow Foundation workflow may contain one or more blocks of activities that should be seen as atomic tasks. Once you wrap such activities in an outermost transactional block, the Windows Workflow Foundation runtime ensures that all the contained activities either succeed or fail. Such transactions are referred to as ACID transactions, drawing from the description of the expected behavior—atomic, consistent, isolated, and durable (see Figure 1).

Figure 1 ACID Transaction Behavior (Close Figure 1)

Feature | Description
Atomic | Either all of the operations of the transaction are successfully completed or none of them are completed.
Consistent | Resources involved with the transaction must be in a legal state when the transaction begins and when it ends. The transaction cannot violate integrity constraints or business rules.
Isolated | Operations in the transaction appear isolated from all other operations. No operation outside the transaction sees the data in an intermediate state.
Durable | Once the transaction has been successfully completed, the effects are persistent and cannot be undone.

A whole Windows Workflow Foundation workflow may also be treated as a single long-running transaction (LRT). In this case, you use the workflow semantics to describe a business process, or a part of it, where multiple services, distinct software platforms, and different companies are involved. An LRT transaction may take minutes, days, or even weeks before the outcome is known. The process may implement a multistep operation and span over the information systems of multiple companies. Typically, an LRT transaction is made of one or more ACID transactions that succeed or fail independently. However, because the process is long and complex, it may happen that a further step in the process determines a condition that is incompatible with the result of a previous ACID transaction. In that situation, it is too late to roll back the results of an already committed transaction, but its effects must be compensated for in some way.

Although the word "transaction" is commonly used to label both ACID and long-running processes, there is a distinct difference between the two. The word transaction typically refers to a sequence of operations to be processed and treated as a single unit of work. The operation usually comes to an end in a matter of seconds and never involves a suspension or indefinitely long wait for user input or other human-led action. It doesn’t need persistence and is considered an all-or-nothing operation. As you’ll notice, the description of what constitutes a typical transaction is essentially the description of an ACID transaction.

So what exactly is a long-running transaction? Is it essentially an ACID transaction that lasts longer and can be suspended and resumed when required? A long-running transaction is a much looser use of the term transaction. An LRT is a cohesive set of actions that involve loosely coupled and independent systems.

The long-running nature of certain transactions can make managing them as an ACID transaction more complex or completely impractical. ACID transactions often require that some critical data be locked for the duration of the transaction—something that is not an issue for transactions that take only a matter of seconds. But if any of the resources involved with a particular business transaction can’t stay locked for the entire duration of the process, then the ACID pattern is not appropriate. The various operations must complete as independent operations with some surrounding logic to orchestrate actions and results, and it must be ready to compensate for errors and faults if required. The orchestrating logic can be built in a Windows Workflow Foundation workflow acting as a BPM server or constructed in a Windows Workflow Foundation composite activity that is then used in an outermost workflow.

ACID Transactions in Workflows

An ACID transaction is a short-lived transaction where the decision to commit or roll back is typically made within seconds. The rollback logic must be able to cancel the ongoing operation and all of the intermediate steps it might be made of, and it must compensate for the effect of previous operations. From the user’s perspective, it is as if nothing ever occurred. The typical database transaction coded in a stored procedure that atomically works over two or more tables is the perfect example of an ACID transaction.

An ACID transaction can be local or distributed. A local transaction involves a single resource, such as the database you’re connected to. A distributed transaction spans multiple heterogeneous resources and requires a transaction processing monitor. The Distributed Transaction Coordinator (DTC) is the transaction processing monitor for Microsoft Windows 2000 and for later releases of Windows.

To build a transactional segment in your workflow, you use the TransactionScope activity (see Figure 2). All activities composed in the scope of the transaction form a unit of work that fulfills the classic ACID schema. When all child activities have successfully completed, the transaction commits and the workflow proceeds. If an exception is thrown from any of the child tasks, the TransactionScope activity performs a rollback operation.

Figure 2 TransactionScope Activity in Visual Studio 2005
Figure 2 TransactionScope Activity in Visual Studio 2005(Click the image for a smaller view)
Figure 2 TransactionScope Activity in Visual Studio 2005
Figure 2 TransactionScope Activity in Visual Studio 2005 (Click the image for a larger view)

The sample workflow shown in Figure 2 accepts an order number and proceeds with payment. Money is first withdrawn from the customer’s account and then added to the vendor’s account. The operation must be atomic in that either both steps are successfully accomplished or both fail. In any case, the involved resources must be left in a consistent state where no constraint is violated and where data integrity is guaranteed.

The internal steps are transparent to both the developer and the end user. Ideally, no external software components should access the state of any transacted resources before the state is finalized (this constraint is sometimes relaxed for performance reasons at the possible expense of correctness). Isolation, therefore, requires a serializable underlying transaction, which is the most expensive type of transaction that databases and similar resource managers support.

Exploring the TransactionScope Activity

In Windows Workflow Foundation, you use the TransactionScope activity as a container of child activities that are run sequentially and with full respect of the ACID semantics. If you use the Parallel activity within a transacted scope, then activities run concurrently within the activity, but sequentially with respect to the rest of the scope.

The TransactionScope activity exposes two properties that can be used declaratively to configure the transaction: IsolationLevel and TimeoutDuration. The IsolationLevel property is of type IsolationLevel—an enumeration defined in the System.Transactions namespace. The isolation level of a transaction determines what level of access other transactions have to internal data before a transaction is completed. Figure 3 details what happens with the most commonly used levels.

Figure 3 Isolation Levels Supported by TransactionScope (Close Figure 3)

Level | Description
READ UNCOMMITTED | The least restrictive level, this ignores locks placed by other transactions. You can read modified data that has not yet been committed by other transactions (dirty reads).
READ COMMITTED | The default isolation level for SQL Server, this prevents dirty reads. However, it allows transactions to modify, insert, or delete data within the current transaction. This may produce non-repeatable reads and phantom rows.
REPEATABLE READ | This prevents dirty reads and stops other transactions from modifying or deleting data that has been read by the current transaction. New data can be inserted.
SERIALIZABLE | The most restrictive level, this holds the locks until the transaction is complete. No reads and no updates are allowed until the transaction completes.
SNAPSHOT | Only for SQL Server 2005, this specifies that data read within a transaction will never reflect changes made by other simultaneous transactions.
CHAOS | In this case, the pending changes from more highly isolated transactions cannot be overwritten.

Its default value for IsolationLevel is Serializable. This is the safest and most expensive option, which is not strictly required in all cases. A serializable transaction locks data for the duration of the transaction preventing other concurrently running transactions from accessing the same data. The TransactionScope activity, however, mitigates this option by setting a timeout of 30 seconds. If you are concerned about concurrency and need to run transactions on SQL Server™ 2005, you can opt for the new Snapshot isolation level. Unlike some of the other isolation levels, Snapshot isn’t limited to being set only programmatically. It can be applied to SQL Server 2005 databases, administratively configured using the following statement:

ALTER DATABASE <name> SET ALLOW_SNAPSHOT_ISOLATION ON

The TransactionScope activity is a wrapper built around an instance of the .NET Framework 2.0 TransactionScope class—one of the most programmer-friendly classes ever designed. Just wrap everything in a TransactionScope object, and you’re pretty much finished. The object takes care of everything else. It determines whether you need a local or a distributed transaction, enlists any necessary distributed resources, and proceeds with local processing otherwise. As the code reaches a point where it won’t be running locally, it escalates to the DTC as appropriate.

You can enlist any objects with a transaction that implements the ITransaction interface. The list includes any ADO.NET 2.0 data providers, as well as Microsoft Message Queue (MSMQ).

When some code invokes the Complete method of the TransactionScope object, it indicates that all operations in the scope of the transaction are completed successfully. Note that the method does not physically terminate a distributed transaction, as the commit operation will still happen on TransactionScope disposal. However, after calling the Complete method, you can no longer use the distributed transaction.

The TransactionScope activity makes transactional tasks as easy as grouping activities in the Visual Studio® 2005 designer. Consider the code behind the activities shown in Figure 2. The key code is shown in Figure 4.

Figure 4 Code Behind the Activities (Close Figure 4)
private void WithdrawFunds_ExecuteCode(object sender, EventArgs e)
{
    Console.WriteLine(“Funds withdrawn to process order: #” +
        orderNo.ToString());

    // Execute a SQL operation on ClientAccount
    SqlHelper.ExecuteNonQuery(conn,
        System.Data.CommandType.Text,
        “use bank update ClientAccount set Balance = Balance - 100”);
}

private void AddFunds_ToVendor_ExecuteCode(object sender, EventArgs e)
{
    Console.WriteLine(“Funds added to the vendor’s account for order #” +
        orderNo.ToString());

    // Execute a SQL operation on VendorAccount
    SqlHelper.ExecuteNonQuery(conn,
        System.Data.CommandType.Text,
        “use bank update VendorAccount set Balance = Balance + 100”);
}

The transactional code contains three Code activities. The first runs a database command and updates the balance of the client account by subtracting a given amount. Next, the same amount is added to the balance of the vendor account. Given the characteristics of the underlying TransactionScope object, databases can easily be distributed.

More importantly, the transfer of the funds must be an atomic operation. If the execution flow reaches the end of the TransactionScope activity, the transaction completes successfully and commits. Should any exception be raised at some point, the transaction automatically rolls back any work. All of this is completely transparent to the workflow developer. As a workflow developer, you don’t need to take care of compensation or even fault handling.

The sample code throws an exception if the order number is odd, as the code behind the last activity in Figure 2 shows:

void CheckConsistency_ExecuteCode(object sender, EventArgs e)
{
    if (orderNo % 2 > 0)
        throw new DiscontinuedProductException();
}

The exception thrown is a custom exception object that describes a particular situation in the application. That’s enough to trigger the rollback mechanism of the underlying .NET TransactionScope object. Figure 5 shows the application in action and the message you receive in case of a rollback. Figure 6 illustrates a successful transaction that’s been committed.

Figure 6 Money Transfer Being Committed
Figure 6 Money Transfer Being Committed(Click the image for a smaller view)
Figure 6 Money Transfer Being Committed
Figure 6 Money Transfer Being Committed (Click the image for a larger view)
Figure 5 Money Transfer Being Canceled
Figure 5 Money Transfer Being Canceled(Click the image for a smaller view)
Figure 5 Money Transfer Being Canceled
Figure 5 Money Transfer Being Canceled (Click the image for a larger view)

If you need to catch any exception that originates around the workflow, and in particular around the TransactionScope activity, switch to the fault handlers view and add a FaultHandler activity. You can then configure the activity to catch a particular exception type and add to its scope as many activities as needed to handle the exception. This doesn’t make much sense in an ACID scenario, but in general you could run another transaction while handling the exception. The goal of the fault handler is to undo the partial and unsuccessful work of an activity in which an exception has occurred. However, as far as a transaction scope is concerned, the rollback is automatic and there’s no need to write ad hoc SQL commands to cancel the effects of previous database operations.

Configuring a Transactional Workflow for Execution

There are a few restrictions on the use of the TransactionScope activity that you must be aware of. First, you can’t suspend the workflow from within a transaction using the Suspend activity. In addition, a TransactionScope activity can’t be nested inside another TransactionScope activity or in any activities that implement the ICompensatableActivity interface. Examples of such activities are the CompensatableTransactionScope and CompensatableSequence activities in the toolbox.

A transactional workflow requires a persistence service; an exception is thrown if no such service is available. You register a persistence service with the workflow runtime in the client application like so:

string conn = “...”;
workflowRuntime.AddService(
    new SqlWorkflowPersistenceService(conn));

The SqlWorkflowPersistenceService class is the default persistence service and is based on a SQL Server database. Persistence occurs when an ACID transaction completes or when the workflow instance becomes idle or is programmatically unloaded. The workflow runtime engine calls methods on the persistence service to save the state of the workflow instance. The workflow runtime engine determines when and how to perform persistence. The service takes care of actually saving and loading the workflow state to and from the data store of choice.

The database used by the SqlWorkflowPersistenceService class is not created upon installation but scripts are copied on the client machine for later use. The path of the script is:

%WINDOWS%\Microsoft.NET\Framework\v3.0\Windows Workflow Foundation\SQL

You can change the default name of the database. However, you can’t change the structure of the child tables. To use a different database layout, you need a custom persistence service. At its core, a custom persistence service is a class that inherits from WorkflowPersistenceService.

Modeling Business Processes with Windows Workflow Foundation

An ACID transaction is only a small part of a real-world workflow. Multiple ACID transactions are typically combined in a single workflow to express a given business process. So it may happen that a transactional activity successfully completes, but an exception is thrown in another activity later in the workflow. In this case, no automatic rollback is possible. The rollback is possible in the context of a transaction since there’s a resource manager behind to track operations and cancel them if need be. Workflows, though, don’t support fully transactional semantics; they just support transactional activities.

When you use Windows Workflow Foundation to model a business process, you might have various transactional blocks scattered through the workflow. Windows Workflow Foundation makes available two types of transactional activities: ACID and compensatable transactions. ACID transactions are fully represented by the TransactionScope activity. They either commit or roll back and have their results persisted. But what if you need to implement transactional tasks in the context of a larger process where you can’t lock data for too long and must be able to roll back past commits? For this, you need a special type of transaction activity—the CompensatableTransactionScope activity.

CompensatableTransactionScope supports a compensation mechanism. Compensation is any logic you run at some point to undo, mitigate, or compensate for the effects of previous operations. The point is that the compensatable transaction might contain child ACID transactions that, once committed, can’t be rolled back any longer. However, in case of a further failure, their effects must be compensated for in some way. Compensation is like rollback except that the developer is called to write any code used to compensate for the work done. Not all transactional tasks have to be compensatable—for instance, tasks that can be effectively represented with ACID transactions don’t need compensation. The TransactionScope activity, in fact, doesn’t support compensation.

Figure 7presents a sample workflow that uses a couple of compensatable transaction scopes. The business process models the lifecycle of an order. Once placed, the order causes the credit card of the customer to be charged (activity named Scope_ChargeCreditCard) and then moves money into the vendor’s bank account (activity named Scope_PayOrder). Both activities are transactional and compensatable. Say an exception occurs later in the workflow because, for instance, the ordered product is no longer available. At this point, a business exception occurs and all the work done so far must now be undone. The effect of any committed ACID transactions and nontransactional sequences of activities must be compensated. Each compensatable transaction or sequence activity incorporates some logic that just serves the purpose of undoing the work—all or in part.

Figure 7 Compensatable Transactions
Figure 7 Compensatable Transactions

In Windows Workflow Foundation, you define the undo code for each compensatable activity by switching the activity’s view to the compensation handler view (see Figure 8). In the compensation handler view, you list all activities that must be run to cancel the effect of the operation.

Figure 8 Adding Compensation Logic to Workflow Sequences
Figure 8 Adding Compensation Logic to Workflow Sequences(Click the image for a smaller view)
Figure 8 Adding Compensation Logic to Workflow Sequences
Figure 8 Adding Compensation Logic to Workflow Sequences (Click the image for a larger view)

The notion of compensation is similar to the notion of rollback. However, while rollback is a purely transactional operation, compensation applies to transactional and nontransactional sequences of operations. If you compare Figures 7 and 8, you’ll notice a slightly different user interface for the two compensatable transaction scopes. In Figure 8, instead of the direct code, you see the compensation code.

About Compensation

A good question is why bother with compensation? Isn’t one big ACID transaction with automatic rollback just as good? An ACID transaction is most appropriate when operations occur within the same database or within the same information system. It is also most appropriate when operations end quickly. When different companies and services are involved, defining the process in terms of the ACID semantics is often challenging. For it to be isolated and durable, you have to keep all resources of different companies locked for the duration of the task. This is frequently unreasonable, especially if the task is long. For it to be consistent and atomic, you need ad hoc compensation code.

Another good question is who triggers the compensation code.With fault handling, you handle one or more exceptions raised by the workflow. The handler of such exceptions is another special activity—the Compensate activity (see Figure 9). The Compensate activity triggers the code that fixes constraints and business rules put at risk by a business exception. You need to bind the Compensate activity to a compensatable activity in the workflow. You do that through the TargetActivityName property in the Visual Studio 2005 Workflow Extensions designer. If you set the TargetActivityName property to the name of a particular transaction or sequence, only that transaction or sequence will be compensated. You can add multiple Compensate activities and determine the order and granularity in which compensation should occur. If you just want to run compensation logic that mimics the rollback logic of a classic ACID transaction, then you bind the Compensate activity to the whole workflow. In this case, the compensation proceeds bottom-up from the exception point to the root of the workflow.

Figure 9 Compensate Activity
Figure 9 Compensate Activity

Compensation code may kick off additional ACID transactions in order to balance the effects of the workflow. Furthermore, compensation may fail. Therefore, to be on the safe side, you need to prepare appropriate exception handlers for the compensating code as well.

Performance and Workflow Transactions

One of the factors that has the deepest impact on the performance of a Windows Workflow Foundation workflow is persistence. A workflow can have a number of persistence points; some explicitly defined by programmers, others implicitly required by built-in and custom activities. The persistence service is automatically invoked when the workflow becomes idle, when the Unload method is invoked on the workflow instance, or when an activity decorated with the PersistOnClose attribute completes its work. Both built-in transactional activities defined in Windows Workflow Foundation—TransactionScope and CompensatableTransactionScope—require persistence on close. Therefore, an effective persistence service and, more importantly, an effective runtime environment for it, are critical to performance.

A transactional workflow has a couple requirements as far as its runtime machinery is concerned—persistence and transaction support. By default, the DefaultWorkflowCommitWorkBatchService service class is used to manage transactions via the .NET TransactionScope class. In this way, dynamic escalation to DTC is provided silently and only when required. However, there’s a particular scenario where you should not use the default service for the sake of performance.

When you use the out-of-the-box persistence service, you need to create a proper database—be it SQL Server 2005 or SQL Server 2000. The standard database also includes tables and stored procedures for the standard tracking service—the SqlTrackingService class. Consider a scenario where both services are enabled and share the same database, meaning you use exactly the same connection string. Persistence and tracking data are always written within the same transaction. If the database is SQL Server 2000, though, an escalation to DTC will occur unless you manage to have the services share the same connection object. Thus, when persistence and tracking data are to go in the same SQL Server 2000 database, you should use the SharedConnectionWorkflowCommitWorkBatchService instead of the default service. You add these services to the workflow runtime as shown here:

workflowRuntime.AddService(
    new SqlWorkflowPersistenceService(connString));
workflowRuntime.AddService(
    new SqlTrackingService(connString));
workflowRuntime.AddService(
    new SharedConnectionWorkflowCommitWorkBatchService(connString));

The SharedConnectionWorkflowCommitWorkBatchService service optimizes the workflow performance only in this particular scenario, as it avoids the overhead of extra connections to the database and DTC transactions. If your transacted workflow doesn’t need tracking or if it uses distinct databases for tracking and persistence, you are better off keeping with the default service for transaction support.

On a side note, I should point out that the tracking service and persistence service always operate within the same transaction. The tracking service has a Boolean property named IsTransactional. Setting the property to false doesn’t make the tracking service nontransactional. More simply, this means that tracking data is saved just in time when the corresponding method is called. When IsTransactional is true (the default setting), the TrackData method of the service adds data to the work batch. The work batch will then be flushed on the next persistence point in the workflow.

Conclusion

Modeling transactional tasks in Windows Workflow Foundation is actually quite easy. If you need an ACID transaction—meaning a transaction that is atomic, consistent, isolated, and durable—that will generate a response quickly, you should use the TransactionScope activity. Inside this activity, you compose other activities that access objects enlisted with the underlying ambient transaction. If, on the other hand, you need to orchestrate a number of loosely coupled services to model a task that may take a while to complete, then you’re better off using a CompensatableTransactionScope.

And remember that a transaction brings inherent costs due to required persistence and memory needs. If you only need compensation, then you should opt for the more lightweight CompensatableSequence activity.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Service Station: WCF Addressing In Depth

Service Station
WCF Addressing In Depth
Aaron Skonnard

Contents
Endpoints and Addresses
Addressing Fundamentals
IIS Addressing Considerations
Multiple Endpoints and Unique Addresses
Logical vs. Physical Addresses
Addressing Headers
Message Filters
Host Name Comparisons
Conclusion

Windows Communication Foundation provides a flexible model for exposing service endpoints and communicating with them through a variety of different communication protocols. This month, I dive into various addressing details that surround endpoint communication, many of which enable more advanced messaging scenarios.

Endpoints and Addresses

The Windows® Communication Foundation architecture cleanly separates physical communication details from the underlying service implementation. Developers spend most of their time working with familiar Microsoft® .NET Framework code to implement service contracts and to define service behavior. After the implementation is complete, they can decide how to host the service and expose it to consumers.

In order to host a particular service, you create a ServiceHost instance and define a collection of service endpoints. A service endpoint specifies an address, a binding, and a contract to use for communication. Windows Communication Foundation needs this information to build the necessary messaging runtime, also known as the channel stack, and to provide metadata in the form of Web Services Description Language (WSDL) to external consumers upon request.

You can supply service endpoints to a ServiceHost instance through code (see the ServiceEndpoint class) or through entries in the <system.serviceModel> configuration section (see the <endpoint> element). Anything you can accomplish in code, you can also accomplish in configuration and vice-versa. However, supplying endpoints via configuration usually provides greater flexibility for making post-development changes.

The ServiceHost must contain at least one service endpoint before you attempt to open it or the runtime will throw an exception; a service without at least one endpoint wouldn’t be very useful since you’d have no way to interact with it. A service can, however, expose more than one endpoint in order to accommodate multiple contracts or different types of consumers that have a wide range of capabilities or constraints.

When discussing endpoints, most folks tend to focus on the binding and contract while overlooking the endpoint address and the various issues surrounding it. This month, my goal is to buck that trend by uncovering the various addressing techniques available to you.

For a more thorough discussion of the Windows Communication Foundation programming model and how you work with endpoints in either code or the various configuration elements, see my article called "Learn the ABCs of Programming Windows Communication Foundation" in the February 2006 issue of MSDN® Magazine.

Addressing Fundamentals

When you define service endpoints, you have numerous options for specifying the address. You can specify an IP address or a host name, and you can specify a port number or just assume the default port for the transport. And, of course, you can include a specific path in the address (the part of the address that comes after the host/port).

There are two fundamental techniques you can use when specifying the address in Windows Communication Foundation. You can specify an absolute address for each endpoint or you can supply the ServiceHost with a base address and then specify relative paths for each endpoint. Specifying absolute addresses is a little easier to understand, but the base address technique typically makes things easier to manage.

You specify an absolute address by providing the fully qualified address in the endpoint definition as illustrated in Figure 1. In this example, I’ve specified three absolute addresses, but notice that the two HTTP addresses share the same base address of http://localhost:8080/calcservice. Given this situation, you have the option to provide the host with the base address and then use relative addresses when defining the individual endpoints.

Figure 1 Configuring Endpoints (Close Figure 1)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”>
        <endpoint address=”http://localhost:8080/calcservice”
                  binding=”basicHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”http://localhost:8080/calcservice/secure”
                  binding=”wsHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”net.tcp://localhost:8081/calcservice”
                  binding=”netTcpBinding”
                  contract=”ISimpleMath”/>
      </service>
    </services>
  </system.serviceModel>
</configuration>

You can supply one base address per transport when constructing a ServiceHost object, as shown here:

ServiceHost host = new ServiceHost(typeof(CalculatorService),
    // base HTTP address
    new Uri(“http://localhost:8080/calcservice”),
    // base TCP address
    new Uri(“net.tcp://localhost:8081/calcservice”));

It’s also possible to specify the base addresses in the configuration file along with the endpoints themselves. You do this by listing the base addresses within the <host> element for each <service>, like so:

<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”>
        <host>
          <baseAddresses>
            <add baseAddress=”http://localhost:8080/calcservice”/>
            <add baseAddress=”net.tcp://localhost:8081/calcservice”/>
          </baseAddresses>
        </host>
        <!-- endpoint definitions follow -->
        ...

Whether you do it in code or in configuration, both techniques accomplish the same thing—they configure the ServiceHost object with a base address for each transport in use. Once the ServiceHost has been configured this way, you can take advantage of relative URIs when defining endpoint addresses, as you see in Figure 2. When you open the ServiceHost, it determines the absolute address to use for each endpoint by resolving any relative addresses against the corresponding base address (the one that matches the binding’s transport).

Figure 2 Using Relative URIs (Close Figure 2)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”>
        <endpoint binding=”basicHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”secure”
                  binding=”wsHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint binding=”netTcpBinding”
                  contract=”ISimpleMath”/>
      </service>
    </services>
  </system.serviceModel>
</configuration>

Notice both the first and the last endpoints omit the "address" attribute, which is the same as leaving it empty. Hence, each of these endpoints will simply use the corresponding base address for its endpoint address; the first endpoint uses the base HTTP address while the last endpoint uses the base TCP address. The second endpoint uses a relative address of "secure", which will be resolved against the base HTTP address (it becomes http://localhost:8080/calcservice/secure).

Even when using base addresses, you can still specify absolute addresses for endpoints when desired. And the absolute addresses don’t have to match up in any way to the corresponding base address if one exists. Each endpoint can specify a wildly different location if you so choose.

The base address technique is mostly a convenience to reduce the number of places you’ll have to make changes when modifying the locations of your endpoints. Windows Communication Foundation also uses the base HTTP address by default to expose metadata when GET retrieval has been enabled (via the <serviceMetadata> behavior). You can, however, change the retrieval location using the behavior’s httpGetUrl property.

Clients have no awareness of the service’s base address and have no need to support something similar on their side of the wire. As a result, you won’t find anything related to base addresses in the client-side object model or the <client> configuration section. Clients simply choose a particular endpoint, which always comes configured with an absolute address, and that absolute address determines the address it will use during transmission.

These addressing fundamentals apply when self-hosting Windows Communication Foundation services (where you create and manage the ServiceHost instance yourself), but there are some special considerations to keep in mind when you host services within IIS.

IIS Addressing Considerations

When hosting in IIS, you don’t have to worry about creating or managing the ServiceHost instance. IIS takes care of this for you behind the scenes. You simply map a .svc endpoint to your service class, configure the service endpoints and behaviors in web.config, and let Windows Communication Foundation manage the process of creating and configuring the ServiceHost instance at runtime.

In this hosting scenario, the base HTTP address is determined by the IIS virtual directory housing the service along with the name of the .svc file. You, as the developer, really have nothing to say about the base address since it’s fully controlled by the IIS addressing scheme. As a result, Windows Communication Foundation happily ignores any base addresses you may specify in web.config in this scenario. If you want to change the base address for your endpoints, you’ll need to move the service to a different IIS virtual directory.

Not only does IIS control the base address, it forces all of your endpoints to actually use the same base address (unlike self-hosting). This means that if you do specify an absolute address for a particular endpoint, it must start with the base address corresponding to the virtual directory or you’ll get an exception. As a result, it really only makes sense to use relative addresses when hosting in IIS. You will need to use relative addresses when exposing more than one endpoint for a particular .svc service (we’ll discuss why you’d want to do this shortly).

Let’s take a look at an example. Suppose you have a file named calc.svc and you place it in a virtual directory that corresponds to http://localhost:8080/calcservice. The base address for this service will be http://localhost:8080/calcservice/calc.svc. Assuming you’ve enabled HTTP GET metadata retrieval, the help page will be exposed at that same address, which you can simply browse to in Internet Explorer.

Now, consider the endpoint configuration found in the virtual directory’s web.config file (in Figure 3). In this case, the address of the first endpoint becomes the same as the base address (http://localhost:8080/calcservice/calc.svc) since I left the endpoint address empty. The address of the second endpoint becomes the combination of the base address appended with "secure", like this: http://localhost:8080/calcservice/calc.svc/secure. And the address of the "mex" endpoint is http://localhost:8080/calcservice/calc.svc/mex. This can seem a bit strange to some people since the relative portion of the address is added to the right of the file name, but you have to remember that calc.svc is part of the base address so it has to work this way.

Figure 3 Sample Config for IIS-Hosted Service (Close Figure 3)
<configuration>
  <system.serviceModel>
    <services>
        <service name=”CalculatorService”>
          <!-- base address determined by IIS virtual directory -->
          <endpoint binding=”basicHttpBinding”
                    contract=”ISimpleMath”/>
          <endpoint address=”secure”
                    binding=”wsHttpBinding”
                    contract=”ISimpleMath”/>
          <endpoint address=”mex”
                    binding=”mexHttpBinding”
                    contract=”IMetadataExchange”/>
        </service>
        ...

Since I’m on the subject of IIS hosting, I’ll also point out that with IIS 5.0 and IIS 6.0, you can only use the various HTTP bindings in your endpoints. In other words, you cannot host a TCP endpoint in an IIS 5.0- or 6.0-hosted service—only HTTP endpoints are supported. This changes with the Windows Process Activation Services (WAS), which ships with Windows Server code-named "Longhorn." WAS makes it possible to host non-HTTP endpoints within IIS using the same .svc techniques discussed above.

Multiple Endpoints and Unique Addresses

There are a few reasons why you might wish to expose multiple endpoints on a particular service. One reason is to expose the same contract using a few different bindings. For example, you may have some consumers that can only deal with WS-I Basic Profile 1.1-compliant services (one binding) and others that can handle the full suite of standards (another binding). Or you may have some internal enterprise consumers that demand binary TCP transmissions for performance reasons (yet another binding). The ability to expose the same contract using different bindings allows you to accommodate all of these consumers at the same time.

When exposing multiple endpoints with different bindings, each endpoint address must be unique. This is because each endpoint requires a different transport listener and channel stack. Consider the service configuration in Figure 4. In this example, all of the endpoints expose the same contract (ISimpleMath) but each one uses a different binding, so each address must be unique. If you modify an endpoint to use the same address as one of the other endpoints, Windows Communication Foundation will throw an exception while opening the ServiceHost.

Figure 4 Sample Configuration Using Unique Addresses (Close Figure 4)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”>
        <endpoint address=”http://localhost:8080/calcservice”
                  binding=”basicHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”http://localhost:8080/calcservice/secure”
                  binding=”wsHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”net.tcp://localhost:8081/calcservice”
                  binding=”netTcpBinding”
                  contract=”ISimpleMath”/>
      </service>
 ...

If, however, multiple endpoints share the same binding, you can use the same address across those endpoints. This approach makes a lot of sense when you need to expose multiple contracts using the same binding, which is another major case in which you need multiple endpoints on a single service. In the following example, the two defined endpoints share the same address, but they expose different contracts:

<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”>
        <endpoint address=”http://localhost:8080/calcservice”
                  binding=”wsHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”http://localhost:8080/calcservice”
                  binding=”wsHttpBinding”
                  contract=”IScientific”/>
</service>
 ...

In this case, both endpoints end up sharing the same transport listener and channel stack (they simply dispatch to a different set of methods). Windows Communication Foundation is smart enough to know that it can create and use a single binding instance for both endpoints to share since both of them specified "wsHttpBinding" for the binding.

If you’re adding endpoints manually in code, and they’re going to share the same address, you’ll need to ensure that you use the same binding instance when calling AddServiceEndpoint. The following code example illustrates how not to do it:

ServiceHost host = new ServiceHost(typeof(CalculatorService));
host.AddServiceEndpoint(
    typeof(ISimpleMath), new WSHttpBinding(),
    “http://localhost:8080/calc”);
host.AddServiceEndpoint(
    typeof(IScientific), new WSHttpBinding(),
    “http://localhost:8080/calc”);
host.Open();
...

Even though both endpoints use the same type of binding, I’ve supplied two distinct instances when calling AddServiceEndpoint. If you attempt to run this code, Windows Communication Foundation throws an exception during the second call to AddServiceEndpoint indicating that a binding instance has already been associated with that particular endpoint address. Instead, you should create a single WSHttpBinding instance and supply it to both AddServiceEndpoint calls as shown here:

ServiceHost host = new ServiceHost(typeof(CalculatorService));
WSHttpBinding wsbinding = new WSHttpBinding();
host.AddServiceEndpoint(
    typeof(ISimpleMath), wsbinding, “http://localhost:8080/calc”);
host.AddServiceEndpoint(
    typeof(IScientific), wsbinding, “http://localhost:8080/calc”);
host.Open();
...

Clients don’t need to be concerned with the fact that both of these endpoints are sharing the same address. The client will still be presented with two distinct endpoints and will have to choose one. In this particular example, svcutil.exe actually generates two proxy classes, one for each endpoint, because they expose different contracts. It just so happens that both proxy classes target the same address.

Logical vs. Physical Addresses

In Windows Communication Foundation, each service endpoint actually has two addresses associated with it—a logical address and a physical address. The difference between these addresses is the same as the difference between "To" and "Via" in WS-Addressing. The logical address ("To") is the address that SOAP messages target. The physical address ("Via"), on the other hand, is the actual transport-specific network address that Windows Communication Foundation listens to for messages to arrive. Throughout the object model, Windows Communication Foundation refers to the logical address as "Address" or "Endpoint Address" and the physical address as "ListenUri".

The Windows Communication Foundation channel infrastructure revolves around the physical address since it’s responsible for receiving incoming messages using a particular transport protocol at a specific location. The Windows Communication Foundation dispatcher, on the other hand, is shielded from such networking details and instead focuses on mapping the incoming message to an endpoint, and ultimately to a method call.

When you specify endpoint addresses like I’ve done thus far, you’re actually supplying the logical address. However, the logical address is also used as the physical address unless otherwise specified. The following example illustrates how to output various endpoint details including both the logical and physical addresses:

...
foreach (ServiceEndpoint se in host.Description.Endpoints)
{
    Console.WriteLine(“Endpoint details:”);
    Console.WriteLine(“Logical address: \t{0}”, se.Address);
    Console.WriteLine(“Physical address: \t{0}”, se.ListenUri);
    Console.WriteLine(“Binding: \t{0}”, se.Binding.Name);
    Console.WriteLine(“Contract: \t{0}”, se.Contract.Name);
    Console.WriteLine();
}
...

If you run it against the preceding examples, it will print the same value for both addresses. In most situations, there’s no reason for the logical and physical addresses to be different. Nevertheless, there are situations where you may need to specify a different physical address. You can specify the ListenUri when defining an endpoint in either code or configuration. The following example illustrates how to do it in code:

host.AddServiceEndpoint(
    typeof(ISimpleMath), wsbinding,
    “urn:calcservice:simplemath”, // logical
    new Uri(“http://localhost:8080/calc”)); // physical (listenUri)
host.AddServiceEndpoint(
    typeof(IScientific), wsbinding,
    “urn:calcservice:scientific”, // logical
    new Uri(“http://localhost:8080/calc”)); // physical (listenUri)

You can accomplish the same thing in configuration using the listenUri attribute, as Figure 5 illustrates. Now when you run the code to print the endpoint details, you should see different values for the logical and physical addresses. In this example, both endpoints share the same physical address (in this case http://localhost:8080/calcservice), but each one has a unique logical address in the form of a URN.

Figure 5 Specifying the ListenUri (Close Figure 5)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”>
        <endpoint address=”urn:calcservice:simplemath”
                  listenUri=”http://localhost:8080/calcservice”
                  binding=”wsHttpBinding”
                  contract=”ISimpleMath”/>
        <endpoint address=”urn:calcservice:scientific”
                  listenUri=”http://localhost:8080/calcservice”
                  binding=”wsHttpBinding”
                  contract=”IScientific”/>
      </service>
      ...

If you need to guarantee that a ListenUri is unique on the machine, you can use the ListenUriMode property and set its value to ListenUriMode.Unique. When you do this, the channel infrastructure will either choose a unique port number (in the case of TCP, without port sharing enabled) or it will append a GUID to the end of the ListenUri address before using it.

It’s worth noting that you can also use a relative address when specifying the ListenUri and Windows Communication Foundation will resolve it against the corresponding base address just like it does for the logical endpoint addresses.

At this point, you may be wondering how clients will interface with a service configured with a different ListenUri address. Clients have no notion of a ListenUri. As far as clients know, there’s a single address for each endpoint. Since the WSDL definition contains the logical address for each endpoint, svcutil.exe embeds the logical address in the client configuration and will use it for transmission by default.

When a service has been configured with a different physical address, you’ll need an out-of-band mechanism to inform clients of the physical address to use. Then clients can instruct Windows Communication Foundation to route outgoing messages via the physical address, as if it were a router or some other type of intermediary. The ClientViaBehavior makes this easy to accomplish, as illustrated here:

SimpleMathClient client = new SimpleMathClient(
    “WSHttpBinding_ISimpleMath”);
client.Endpoint.Behaviors.Add(new ClientViaBehavior(
        new Uri(“http://localhost:8080/calcservice”)));
double sum = client.Add(3, 4);

It’s also possible to apply the ClientViaBehavior within the client’s configuration file as shown in Figure 6. Now, instead of trying to transmit the message to "urn:calcservice:simplemath", the client transmits the message via the same physical address used by the service endpoint. Once the message arrives at the ListenUri, the dispatcher can determine which endpoint to use by inspecting the logical address (the URN).

Figure 6 Applying the ClientViaBehavior in Configuration (Close Figure 6)
<configuration>
  <system.serviceModel>
    <client>
      <endpoint address=”urn:calcservice:simplemath”
                binding=”wsHttpBinding”
                behaviorConfiguration=”Via”
                contract=”Client.localhost.ISimpleMath”
                name=”WSHttpBinding_ISimpleMath”/>
      ...
    </client>
    <behaviors>
      <endpointBehaviors>
        <behavior name=”Via”>
          <clientVia viaUri=”http://localhost:8080/calcservice”/>
        </behavior>
      </endpointBehaviors>
    </behaviors>
    ...

Why would someone want to use these techniques? The answer is simple: routing. Any time you need to transmit messages through an intermediary en route to the ultimate destination, you’ll need to understand the difference between Address and ListenUri and use them appropriately. This is common in certain enterprise scenarios where a central "gateway" is used to handle common tasks such as security, logging, and statistics for all deployed services.

TcpTrace is exactly this type of intermediary—the only difference is it’s used by developers for diagnostic purposes. Let’s say that you want to use TcpTrace to intercept all messages en route to the service but you don’t want to change the service code. You can easily do this by configuring the client with a ClientViaBehavior pointing to the TcpTrace listen address, and presto, you’re done.

There’s another way to accomplish the same thing without requiring the client to use the ClientViaBehavior. When defining the service endpoint, you can use the same address that TcpTrace is listening on for the logical address (since this address shows up in WSDL, it causes the client to send messages to the TcpTrace listen address). Then you specify a different physical address for the endpoint (the ListenUri) and configure TcpTrace to forward to that address.

In this example, TcpTrace is just like any other intermediary performing simple routing. It’s not hard to extrapolate to more sophisticated custom routing scenarios.

Addressing Headers

In order to accommodate even more sophisticated routing and dispatching logic, you may want to annotate SOAP messages with custom addressing headers. Custom addressing headers are carried in SOAP messages alongside the traditional WS-Addressing headers. For example, if an incoming message contains the <premium> membership header, you dispatch to a feature-rich endpoint whereas if it contains the <basic> header, you route to the endpoint with basic functionality.

You may be wondering why you wouldn’t just specify this information as part of the service contract so it shows up in the message body. The issue is that the consumer may not know or have the required information. Rather, the information will need to be supplied by an intermediary en route to the endpoint. For instance, an intermediary looks up the supplied member id, determines the membership level, and adds the appropriate header to the outgoing message.

You can associate custom addressing headers with endpoints to define additional dispatching criteria. When you do this, only messages containing the required addressing headers will be dispatched to the given endpoint.

When clients need to be aware of any required headers, they should also be included in the service’s WSDL definition; addressing headers are typically described within an <EndpointReference> element as reference parameters (the <EndpointReference> is placed within the WSDL <port> element).

Although this may sound a bit complicated, Windows Communication Foundation makes it easy to work with addressing headers. Consider the service configuration in Figure 7. In this example, I’ve defined two endpoints that share the same logical/physical addresses but each exposes a different contract. Each endpoint also requires a specific addressing header. The first one requires the <basic> header while the second requires the <premium> header. Only incoming messages that contain one of these addressing headers will be dispatched to one of these endpoints.

Figure 7 Endpoints with Custom Addressing Headers (Close Figure 7)
<configuration>
  <system.serviceModel>
    <services>
      <service name=”CalculatorService”
               behaviorConfiguration=”metadata”>
        <host>
          <baseAddresses>
            <add baseAddress=”http://localhost:8080/calcservice”/>
          </baseAddresses>
        </host>
        <endpoint binding=”wsHttpBinding” contract=”ISimpleMath”>
          <headers>
            <basic xmlns=”http://example.org/level”/>
          </headers>
        </endpoint>
        <endpoint binding=”wsHttpBinding” contract=”IScientific”>
          <headers>
            <premium xmlns=”http://example.org/level”/>
          </headers>
        </endpoint>
      </service>
 ...

It’s also possible to specify addressing headers in code by manually creating the AddressHeader object, and supplying it to an EndpointAddress object. Then you can create a new ServiceEndpoint (based on the EndpointAddress) and add it to the host’s Endpoints collection, as shown here:

ServiceHost host = new ServiceHost(typeof(CalculatorService));
AddressHeader header =
    AddressHeader.CreateAddressHeader(“basic”,
       “http://example.org/level”, null);
EndpointAddress ea = new EndpointAddress(
    new Uri(“http://localhost:8080/calcservice/foobar”), header);

host.Description.Endpoints.Add(
    new ServiceEndpoint(
        ContractDescription.GetContract(typeof(ISimpleMath)),
        new WSHttpBinding(), ea));
...

If you inspect the WSDL for this particular endpoint configuration, you’ll find the details in the <service> element (see Figure 8). The <port> element contains a <wsa:EndpointReference> element, which in turn contains the required headers as reference parameters. When you run the WSDL through svcutil.exe, you’ll end up with a similar client-side configuration that includes the same <headers> element. Then when you use that particular endpoint, Windows Communication Foundation will automatically include the required <basic> header in the outgoing message, as you see in Figure 9.

Figure 9 Custom Addressing Headers Sent in SOAP (Close Figure 9)
<s:Envelope xmlns:s=”http://www.w3.org/2003/05/soap-envelope”
            xmlns:a=”http://www.w3.org/2005/08/addressing”>
  <s:Header>
    <a:Action s:mustUnderstand=”1”
     >http://example.org/calc/Add</a:Action>
    <a:MessageID
     >urn:uuid:db1ba7a6-ca2e-494b-b390-9f621e8b90f4</a:MessageID>
    <a:ReplyTo>
      <a:Address
       >http://www.w3.org/2005/08/addressing/anonymous</a:Address>
    </a:ReplyTo>
    <a:To s:mustUnderstand=”1”>http://localhost:8080/calcservice</a:To>
    <basic a:IsReferenceParameter=”true”
      xmlns=”http://example.org/level”/>
  </s:Header>
  <s:Body>
  ...
Figure 8 Custom Addressing Headers in WSDL (Close Figure 8)
...
<wsdl:service name=”CalculatorService”>
  <wsdl:port name=”WSHttpBinding_ISimpleMath”
             binding=”tns:WSHttpBinding_ISimpleMath”>
    <soap12:address location=”http://localhost:8080/calcservice”/>
    <wsa10:EndpointReference>
      <wsa10:Address>http://localhost:8080/calcservice</wsa10:Address>
      <wsa10:ReferenceParameters>
        <basic xmlns=”http://example.org/level”/>
      </wsa10:ReferenceParameters>
      ...
    </wsa10:EndpointReference>
  </wsdl:port>
  ...
</wsdl:service>
...

You can confirm that the dispatcher is looking for the <basic> header by removing the <headers> element from the generated client configuration and invoking the service again, at which point dispatching will fail.

Message Filters

You can dig down even further to customize the dispatching process by leveraging what are known as message filters. When Windows Communication Foundation dispatches incoming messages, it uses message filters to determine the matching endpoint, if one exists. You can choose which message filter to use or you can provide your own. This flexibility allows you to break free from the traditional dispatching model when using Windows Communication Foundation to implement things other than traditional SOAP—for instance, the techniques described here enable you to implement REST/POX-style services on the Windows Communication Foundation messaging foundation.

Each endpoint actually has two filters associated with it—an address filter and a contract filter. The address filter determines if the incoming message matches the endpoint’s "To" address and any required address headers, while the contract filter determines whether it matches the endpoint’s contract. Both filters are used by the dispatcher to determine the destination endpoint.

A filter is a simply a class that derives from MessageFilter, which defines a few abstract Match methods. There are several built-in implementations of MessageFilter including EndpointAddressMessageFilter and ActionMessageFilter, which are used as the default address and contract filters respectively. The EndpointAddressMessageFilter simply compares the "To" address to the endpoint address and expects an exact match. It also compares the addressing headers found in the incoming message with the collection of addressing headers required by the endpoint. The ActionMessageFilter compares the incoming "Action" value against the actions on the contract, again looking for an exact match.

There are other addressing filters that provide different behavior. For example, the PrefixEndpointAddressMessageFilter causes an incoming "To" address to match the endpoint address as long as it starts with the same address prefix (a looser match). There’s also a MatchAllMessageFilter, which causes all messages to match the given endpoint.

An easy way to change the message filter used by a service is through the AddressFilterMode property of [ServiceBehavior]. AddressFilterMode comes with three values: Any, Exact, and Prefix. Each of these values maps to the corresponding MessageFilter implementation. The following example illustrates how to configure my service to use the PrefixEndpointAddressMessageFilter:

[ServiceBehavior(AddressFilterMode=AddressFilterMode.Prefix)]
public class CalculatorService : ISimpleMath, IScientific
{
   ...

It’s also possible to change each endpoint’s filters by writing some code. You can even write your own MessageFilter implementation to incorporate some customized matching logic. Windows Communication Foundation does come with an XPathMessageFilter, which allows you to evaluate arbitrary XPath expressions against incoming messages during the matching process, single-handedly covering content-driven scenarios.

Host Name Comparisons

One final note about addresses and dispatching. It’s common to hard-code a specific host name into the physical address when defining an endpoint. But what if you want to configure the endpoint to listen for all host names bound to a particular machine (localhost, contoso, www.contoso.com, and so on)? Or what if you want to restrict an endpoint to only match a particular host name?

You can further control this via the HostNameComparisonMode, which is made available on most bindings as a property. HostNameComparisonMode comes with three settings: StrongWildcard (the default), Exact, and WeakWildcard.

StrongWildcard and WeakWildcard mean any hostname plus port combination will match the endpoint address (including IP addresses). StrongWildcard matches simply have higher priority than WeakWildcard matches. Exact means the hostname plus port must exactly match the endpoint address (a case-insensitive match at least).

Since the default mode is StrongWildcard, your endpoints should automatically handle the differences between localhost and your machine name, so you’ll only need to use this feature when you wish to further constrain the matching process.

Note that many of the addressing techniques discussed in this article will only work when used in conjunction with bindings configured to use SOAP 1.2 and WS-Addressing. If you’re having issues getting something to work, make sure you’re using a modern binding like the WSHttpBinding. For more information on how to enable a binding to use WS-Addressing, see last month’s column titled "WCF Messaging Fundamentals".

Conclusion

The Windows Communication Foundation messaging framework provides developers with numerous addressing features that are designed to simplify common hosting and communication scenarios and to increase flexibility when dealing with sophisticated routing and dispatching logic. Adding these addressing techniques to your toolbelt will increase your future productivity, not to mention your skill set for bending Windows Communication Foundation to your will.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a cofounder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight’s Applied Web Services 2.0, Applied BizTalk Server 2006, and Applied Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


Wicked Code: UpdatePanel Tips and Tricks

Wicked Code
UpdatePanel Tips and Tricks
Jeff Prosise

Contents
Update Highlighting
Canceling UpdatePanel Updates
Multiple UpdatePanels
Updates without UpdatePanels
Web Service != SOAP and XML

For better or for worse, the UpdatePanel control is the darling of the ASP.NET AJAX community. I say "for better" because UpdatePanel makes partial-page rendering shockingly easy and "for worse" because its simplicity and ease-of-use come at the cost of efficiency and, ironically, bandwidth.

While UpdatePanel brings AJAX magic to ordinary Web pages, it doesn’t bring the efficiency we normally associate with AJAX. Did you know, for example, that when an UpdatePanel control performs an asynchronous AJAX callback to the server to update its content, the request contains everything a conventional ASP.NET postback contains, including view state? Most developers assume that AJAX eliminates view state. That’s not the case, though, with UpdatePanel’s brand of AJAX.

If you’re going to use an UpdatePanel control, you need to know what you’re getting in to. In many cases, from a performance point of view, an application is better off not using UpdatePanel and instead using asynchronous calls to WebMethods or page methods. Doing so can reduce the volume of data traveling over the wire by an order of magnitude or more. However, it’s also a fundamental shift where the UI updates need to be handled explicitly by the developer with JavaScript in the page.

In addition, ASP.NET AJAX discussion forums are already filled with questions about customizing UpdatePanel. Many of these questions are easily answered once you know about PageRequestManager, the JavaScript class in the Microsoft® AJAX Library that provides the client-side support for UpdatePanels.

Now that ASP.NET AJAX has shipped, I’d like to examine UpdatePanel more closely—taking a close look at how you can customize it, optimize it, and even live without it. And that’s exactly what this column is all about.

Update Highlighting

Sometimes you can’t help but feel sorry for the developers at Microsoft. If they don’t do their job well enough, they get pounded by the public. However, they sometimes also get slammed if they do their job too well. For example, I recently got e-mail from a customer complaining that the ASP.NET AJAX UpdatePanel works a little too well.

UpdatePanel makes it extremely easy to take all that flashing and flickering that occurs when an ASP.NET page posts back to the server and turn it into smooth, flicker-free updates. UpdatePanel works its magic by converting postbacks into asynchronous callbacks (XML-HTTP requests) and using JavaScript on the client to refresh the part of the page encapsulated by the UpdatePanel control. The flashing and flickering goes away because the browser doesn’t repaint the page as it does during a postback.

The customer’s complaint was that users were sometimes not noticing that part of the page had been updated with new content. His question was simple: can the folks on the ASP.NET AJAX team make UpdatePanel flicker just a little so important updates won’t be missed?

The bad news is that the ASP.NET AJAX team probably isn’t interested in making UpdatePanels flicker. After all, getting rid of the flicker is why UpdatePanel was invented in the first place. The good news is that you can work a little AJAX magic in the browser to draw attention to updated UpdatePanels. The secret is the Sys.WebForms.PageRequestManager class in the Microsoft AJAX Library—the library of JavaScript classes that comprise the client half of ASP.NET AJAX. PageRequestManager manages the asynchronous callbacks launched by UpdatePanel. It’s also responsible for updating the content inside an UpdatePanel when an asynchronous callback completes.

PageRequestManager fires events in the browser before and after an update occurs. You can hook these events in JavaScript and run code that draws a user’s attention to the updated content. The key event is named pageLoaded. This event fires each time the page loads in the browser (it’s analogous to Page_Load in ASP.NET). It also fires each time an asynchronous callback launched on behalf of an UpdatePanel control completes and the content inside that UpdatePanel is updated. You can register a JavaScript handler for the pageLoaded event with two lines of code (which can be combined into one):

var prm = Sys.WebForms.PageRequestManager.getInstance();
prm.add_pageLoaded(pageLoaded);

The first line acquires a reference to the page’s PageRequestManager object. The second registers a JavaScript function named pageLoaded as a handler for pageLoaded events.

When called, a pageLoaded event handler receives an argument of type Sys.WebForms.PageLoadedEventArgs, which is another class in the Microsoft AJAX Library. PageLoadedEventArgs contains a get_panelsUpdated method that you can call to enumerate all the UpdatePanels, if any, whose content was just updated. By default, an UpdatePanel is nothing more than a DIV on the client-side so you can use JavaScript to flash the DIV, highlight it, or do whatever you want to do to draw a user’s attention to it.

The code listed in Figure 1 shows one way to use pageLoaded events to perform update highlighting. Each time an update occurs, this JavaScript flashes the Document Object Model (DOM) elements representing updated UpdatePanels by making them appear and disappear three times in quick succession. The flashing is performed by a helper function named flashPanels, which takes a flash count as an input parameter.

Figure 1 Flashing Updates (Close Figure 1)
<script type=”text/javascript”>

var prm = Sys.WebForms.PageRequestManager.getInstance();
prm.add_pageLoaded(pageLoaded);

var _panels, _count;

function pageLoaded(sender, args)
{
    if (_panels != undefined && _panels.length > 0)
    {
        for (i=0; i < _panels.length; i++)
            _panels[i].dispose();
    }

    var panels = args.get_panelsUpdated();

    if (panels.length > 0)
    {
        _panels = new Array(panels.length);

        for (i=0; i < panels.length; i++)
            _panels[i] = new Sys.UI.Control(panels[i]);

        flashPanels(3);
    }
}

function flashPanels(count)
{
    _count = (count << 1) + 1;

    for (i=0; i < _panels.length; i++)
        _panels[i].set_visible(false);

    window.setTimeout(toggleVisibility, 50);
}

function toggleVisibility()
{
    for (i=0; i < _panels.length; i++)
        _panels[i].set_visible(!_panels[i].get_visible());

    if (--_count > 0)
        window.setTimeout(toggleVisibility, 50);
}
</script>

Note how an updated UpdatePanel’s visibility is toggled on and off to create a flashing effect. Rather than interacting with DOM elements directly, the code wraps the DOM elements representing the UpdatePanels with Sys.UI.Control objects. Then it uses Sys.UI.Control’s set_visible and get_visible methods to toggle the visibility:

_panels[i].set_visible(!_panels[i].get_visible());

Sys.UI.Control is a JavaScript class found in the Microsoft AJAX Library—specifically, in MicrosoftAjax.js. The benefit to toggling visibility in this manner is that this is browser-independent. This works equally well in every browser that supports ASP.NET AJAX (which is virtually all modern browsers). JavaScript code that interacts directly with the browser DOM, on the other hand, would have to be tweaked to work in different browser types.

Canceling UpdatePanel Updates

The pageLoaded event is one of several events that the PageRequestManager class fires when an UpdatePanel goes back to the server to update its content. Another important event that PageRequestManager fires is initializeRequest, which fires before an asynchronous callback occurs.

Someone recently asked me whether it’s possible to decide at runtime whether an AsyncPostBackTrigger should be allowed to trigger an UpdatePanel update. The answer is yes. This is done by processing initializeRequest events.

The second parameter passed to an initializeRequest handler is an object of type initializeRequestEventArgs. That object contains a get_postBackElement method that identifies the button or other element that triggered the update. It also has a set_cancel method that you can use to cancel a callback before it occurs. Here’s an example of the set_cancel method in action:

<script type=”text/javascript”>

var prm = Sys.WebForms.PageRequestManager.getInstance();
prm.add_initializeRequest(initializeRequest);

function initializeRequest(sender, args)
{
    args.set_cancel(!confirm(‘Are you sure?’));
}
</script>

In this sample, before a callback executes, an intializeRequest handler pops up a confirmation box asking the user whether the update should proceed. Clicking Cancel in the confirmation box passes true to set_cancel, which stops the callback in its tracks. In real life, you probably aren’t interested in prompting the user for confirmation before allowing an update to proceed, but it might be useful to have the ability to cancel an update based on conditions elsewhere in the application.

Incidentally, it’s also possible to cancel asynchronous callbacks after they’ve been executed but before they’ve completed. PageRequestManager provides an abortPostBack method for doing this; it also provides a get_isInAsyncPostBack method for determining whether an asynchronous callback is pending. These methods are often used with UpdateProgress controls to present a cancellation UI.

Multiple UpdatePanels

A page can host several UpdatePanels. By default, when one UpdatePanel on a page updates, the other UpdatePanels on the page also update. Sometimes that’s what you want, but more often than not, you don’t need every UpdatePanel updating in response to other UpdatePanels.

You can be selective about which UpdatePanel instances update (and when) by setting the UpdateMode property of each UpdatePanel control on the page to "Conditional." Then, when one UpdatePanel updates and calls a server-side event handler, call UpdatePanel.Update on the other panels you want to update. This reduces the load on the server by reducing the number of controls that render, and it reduces the volume of data in the response because UpdatePanels that don’t update don’t add anything to the response.

Updates without UpdatePanels

AJAX isn’t just about creating a better user experience—it’s also about delivering greater efficiency on the wire. When a traditional ASP.NET postback occurs, all the data in the Web form, including view state, travels up to the server in the postback. View state is one reason ASP.NET pages, especially ones that use DataGrid and GridView controls, can seem sluggish. Pages with too much view state are a bane to performance—and pages with too much view state are all too common in ASP.NET applications.

One of the benefits of replacing ASP.NET postbacks with AJAX callbacks is that, done properly, AJAX callbacks transmit only the data that needs to be transmitted. This means they do not have to include view state in that transmission.

When you use UpdatePanel to perform flicker-free updates on a page, you may think you’re building in efficiency. After all, UpdatePanels use AJAX, right? Unfortunately, if you check out the traffic on the wire when an UpdatePanel updates, you’ll find you’re not saving much at all—at least not in the send. The View state data (and other data) that’s normally transmitted to the server during a postback is also transmitted during UpdatePanel callbacks. In fact, what goes up in an asynchronous XML-HTTP request from an UpdatePanel is almost identical to what goes up in a standard ASP .NET postback. Here’s the dirty little secret about ASP.NET AJAX: UpdatePanel is about ease of use, not efficiency on the wire.

There’s little you can do to make UpdatePanels more efficient, but you can forego the use of UpdatePanels and use other features of ASP.NET AJAX to update content on pages just as smoothly and much more efficiently. It takes a little more work, but the results are often worthwhile since you can dramatically reduce the volume of data traveling between the client and the server.

You can also reduce the load on your server. When an UpdatePanel calls back to the server, the page targeted by the callback goes through an almost complete lifecycle—the page is instantiated, the controls in the page are instantiated, and controls inside the UpdatePanel undergo a normal rendering cycle. That’s a lot of overhead just to update a portion of the page.

As an example, consider the page fragment in Figure 2. It provides a simple UI that allows the user to type in a ZIP Code and click a button to initialize the city and state fields with a city and state (see Figure 3). All the controls are hosted in an UpdatePanel, so the Button control’s postback is converted into an asynchronous callback and the event handler (GetCityAndState) is called on the server inside the callback. GetCityAndState (the code isn’t shown) reads the ZIP Code from the ZIP Code TextBox, converts it into a city and state, and initializes the TextBox and DropDownList representing the city and state, accordingly. Since this all happens inside an UpdatePanel, the update is smooth and flicker-free.

Figure 2 Filling City and State Fields with an UpdatePanel (Close Figure 2)
<asp:UpdatePanel ID=”UpdatePanel1” runat=”server”>
  <ContentTemplate>
    City:<br />
    <asp:TextBox ID=”City” runat=”server” />
    <br /><br />
    State:<br />
    <asp:DropDownList ID=”Region” runat=”server”>
        <asp:ListItem Value=”AL”>Alabama</asp:ListItem>
        <asp:ListItem Value=”AK”>Alaska</asp:ListItem>
        <asp:ListItem Value=”AZ”>Arizona</asp:ListItem>
          ...
        <asp:ListItem Value=”WV”>West Virginia</asp:ListItem>
        <asp:ListItem Value=”WI”>Wisconsin</asp:ListItem>
        <asp:ListItem Value=”WY”>Wyoming</asp:ListItem>
    </asp:DropDownList>
    <br /><br />
    Zip Code:<br />
    <asp:TextBox ID=”ZipCode” runat=”server” />&nbsp;
    <asp:Button ID=”AutofillButton” Text=”Autofill”
      OnClick=”GetCityAndState” runat=”server” />
  </ContentTemplate>
</asp:UpdatePanel>
Figure 3 City, State, and ZIP Code UI
Figure 3 City, State, and ZIP Code UI(Click the image for a smaller view)
Figure 3 City, State, and ZIP Code UI
Figure 3 City, State, and ZIP Code UI (Click the image for a larger view)

Here’s the problem. An UpdatePanel used this way improves the user experience, but it does little to reduce the volume of data being passed over the wire. The UpdatePanel hardly reduces the load on the server, either—up to the point that the controls inside the UpdatePanel render, the processing performed on the server is almost identical to what happens during a full-blown postback. It has to be this way because one of the benefits of the UpdatePanel control is that server-side event handlers like GetCityAndState work no differently inside an asynchronous callback than they do in a traditional postback. That means controls on the page have to be instantiated, they have to be initialized, they have to have access to view state, and so on.

Figure 4 demonstrates how to implement the same functionality without an UpdatePanel control. This time, the Autofill button is wired to a bit of JavaScript that fires off an asynchronous XML-HTTP request to an ASMX Web method named GetCityAndState. The call is placed through the JavaScript proxy named ZipCodeService, which is generated by the service reference in the ScriptManager control. GetCityAndState takes a ZIP Code string as input and returns a string array containing two items: the corresponding city and state. The completion function onGetCityAndStateCompleted retrieves the items from the array and inserts them into the city and state fields. Same result on the outside, but a very different way of going about it on the inside.

Figure 4 Filling City and State Fields without an UpdatePanel (Close Figure 4)
<asp:ScriptManager ID=”ScriptManager1” runat=”server”>
    <Services>
        <asp:ServiceReference Path=”ZipCodeService.asmx” />
    </Services>
    <Scripts>
      <asp:ScriptReference Name=”PreviewScript.js”
Assembly=”Microsoft.Web.Preview” />
    </Scripts>
</asp:ScriptManager>

City:<br />
<asp:TextBox ID=”City” runat=”server” />
<br /><br />
State:<br />
<asp:DropDownList ID=”Region” runat=”server”>
    <asp:ListItem Value=”AL”>Alabama</asp:ListItem>
    <asp:ListItem Value=”AK”>Alaska</asp:ListItem>
    <asp:ListItem Value=”AZ”>Arizona</asp:ListItem>
      ...
    <asp:ListItem Value=”WV”>West Virginia</asp:ListItem>
    <asp:ListItem Value=”WI”>Wisconsin</asp:ListItem>
    <asp:ListItem Value=”WY”>Wyoming</asp:ListItem>
</asp:DropDownList>
<br /><br />
Zip Code:<br />
<asp:TextBox ID=”ZipCode” runat=”server” />&nbsp;
<asp:Button ID=”AutofillButton” Text=”Autofill”
  OnClientClick=”autoFill(); return false;” runat=”server” />

<script type=”text/javascript”>
function autoFill()
{
    var tb = new Sys.Preview.UI.TextBox ($get(‘ZipCode’));
    var zip = tb.get_text();

    if (zip.length == 5)
        ZipCodeService.GetCityAndState (zip,
            onGetCityAndStateCompleted);
}

function onGetCityAndStateCompleted(result)
{
    if (result != null)
    {
        var tb = new Sys.Preview.UI.TextBox ($get(‘City’));
        tb.set_text(result[0]);

        var select =
            new Sys.Preview.UI.Selector ($get(‘Region’));
        select.set_selectedValue(result[1]);
    }
}
</script>

Here’s how the ASMX Web method called through the JavaScript proxy is implemented:

[ScriptService]
public class ZipCodeService : System.Web.Services.WebService
{
    [WebMethod]
    public string[] GetCityAndState(string zip)
    {
      ...
    }
}

This is a standard Web method in every respect except for the fact that the class to which it belongs is attributed ScriptService instead of WebService. ScriptService is synonymous with WebService, but it carries the added meaning that the Web service’s WebMethods are callable from client-side script.

In addition to allowing conventional WebMethods to serve as targets for XML-HTTP requests, ASP.NET AJAX also supports a special type of Web method known as the page method. Page methods are WebMethods implemented in Web pages—that is, in ASPX files or codebehind files rather than in ASMX files. Page methods allow developers to provide endpoints for XML-HTTP callbacks without building dedicated Web services.

Page methods must be public static methods and, like WebMethods, must be decorated with the WebMethod attribute. (WebMethods and page methods can also be decorated with the ScriptMethod attribute, which provides added control over what goes out over the wire.) On the client, page methods are called from JavaScript through the special PageMethods proxy.

Unlike Web services, page methods do not require service references. However, you do have to enable page methods by setting a ScriptManager control’s EnablePageMethods property to true, like so:

<asp:ScriptManager ID=”ScriptManager1” runat=”server”
  EnablePageMethods=”true” />

Under the hood, page methods offer the same efficiency as WebMethods. View state and other input is not transmitted to the server when page methods are called. And since page methods are static, they can be called without instantiating a page object. A call to a page method does not invoke the page lifecycle that is triggered by conventional ASP.NET requests.

Web Service != SOAP and XML

One of ASP.NET AJAX’s most important features is the ability to invoke WebMethods and page methods on a server using asynchronous XML-HTTP requests from browser clients. Yet when I tell people about this feature, I can’t help but cringe just a little.

When most of us hear the term Web service, we think about SOAP and XML. Neither of these technologies is typically mentioned in the same sentence as the word efficient. Yes, you can call WebMethods from JavaScript using ASP.NET AJAX. But no, ASP.NET AJAX doesn’t use SOAP and XML.

Figure 5 shows what travels over the wire when the Web method call in Figure 4 is executed. Other than the HTTP headers, the only data transmitted in the request is the ZIP Code that the user typed, and the only data returned in the response is a pair of strings representing a city and state. You won’t see any SOAP or XML (or view state, for that matter). Instead, input and output is encoded using JavaScript Object Notation (JSON), which is much less verbose than XML and easier to process. Rather than SOAP, the request and response use a simple protocol that is basically just HTTP. The combination of HTTP and JSON makes ASP.NET AJAX calls to WebMethods and page methods much more efficient than traditional Web service calls.

Figure 5 JSON-Encoded Input and Output (Close Figure 5)

Request

POST /Ajax/ZipCodeService.asmx/GetCityAndState HTTP/1.1
Accept: */*
Accept-Language: en-us
Referer: http://localhost:1997/Ajax/ZipCodePage.aspx
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; ...)
Host: localhost:1997
Content-Length: 15
Connection: Keep-Alive
Cache-Control: no-cache

{"zip":"98052"}
Response

HTTP/1.1 200 OK
Server: ASP.NET Development Server/8.0.0.0
Date: Fri, 29 Dec 2006 21:06:17 GMT
X-AspNet-Version: 2.0.50727
Cache-Control: private, max-age=0
Content-Type: application/json; charset=utf-8
Content-Length: 16
Connection: Close

{"REDMOND", "WA"}

JSON is an up-and-coming industry-standard serialization format. It’s also the native format employed by ASP.NET AJAX. JSON serialization and deserialization support is provided on the client-side by the Microsoft AJAX Library’s Sys.Serialization.JavaScriptSerializer class. On the server, support is provided by the System.Web.Script.Serialization.JavaScriptSerializer class.

Not all types are JSON-compatible. JSON cannot, for example, handle objects with circular references. When you need to return complex data types that aren’t JSON-compatible, you can actually use ASP.NET AJAX’s ScriptMethod attribute to serialize return types into XML. This technique is also useful for methods that return XML data, as demonstrated here:

[ScriptMethod (ResponseFormat=ResponseFormat.Xml)]
public XmlDocument GetData()
{
  ...
}

Alternatively, you can build and register custom JSON converters that allow types that aren’t normally JSON-compatible to be serialized and deserialized. The ASP.NET AJAX January Futures CTP contains three such converters: one for DataSets, one for DataTables, and one for DataRows.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He’s also a cofounder of Wintellect (www.wintellect.com), a software consulting and education firm that specializes in Microsoft .NET.


Foundations: ActivityExecutionContext in Workflows

Foundations
ActivityExecutionContext in Workflows
Matt Milner

Code download available at: Foundations2007_06.exe (179KB)
Contents
The Need for ActivityExecutionContext
Workflow Development
Rules and Conditions
Custom Activities
Conclusion

This month I am going to discuss the ActivityExecutionContext (AEC) in Windows® Workflow Foundation. This particular class is a critical component in the execution, persistence, and compensation of workflows and their activities, yet very little attention is paid to it in the documentation and samples. Many of the common problems developers encounter when building workflows and activities are related to how the ActivityExecutionContext works. Whether you are using the While activity, building a state machine workflow, or writing custom activities, it is essential that you know how to work with the AEC. So in this column, I’ll explain the ActivityExecutionContext, show you how to utilize it in your workflows, and discuss how you can use it when writing custom activities.

The Need for ActivityExecutionContext

There are several features of Windows Workflow Foundation that drive the need for (as well as shape the design of) the AEC—these are state management (or persistence), activity re-execution, and compensation. The state management services in Windows Workflow Foundation provide the ability to serialize the entire workflow state and store that information in a durable store so a workflow can be removed from memory and later resumed. In order to support this, there needs to be a container for the state information related to the working set, which includes the activity tree, communication queues, and internal data structures related to scheduling and execution. The activity tree is often referred to as the application state since this is the state created by the application developer and contained in the activities used to define the workflow. The rest of the state is referred to as the runtime state because it is primarily used by the Windows Workflow Foundation runtime to manage workflow execution.

The other two features that require the AEC are closely related. A common design pattern in workflow involves repeating steps. Consider the While activity that executes its child activity repeatedly as long as some condition is met. When you think about the underlying implementation of this execution, there are two main options for handling this. The first involves reusing the activity instances for each iteration and resetting them to their initial state before each execution. Though this might work for something like the While activity, what about the Replicator activity, which can execute its child activity in a parallel? In this case, you might need to have three or four different instances of the activities to be executing at the same time. The reset approach clearly won’t work here.

The second option for executing activities multiple times involves cloning the original activities and then executing the cloned activities for each iteration. In this model, each iteration of the activities will start with the same template—the initial state of the activities—and each cloned instance can be managed separately. One of the benefits of this model is that each clone can have its own state managed by the runtime, and this won’t interfere with the other clones. Each of these clones requires its own AEC to manage the state for that instance of the activities.

Compensation allows for an activity to do some work after it has already completed its main execution path. This lets an activity handle cases where errors in the process occur after the activity has completed and allows activities to return to a state consistent with the starting state. This might mean updating systems or data that have already changed. Basically, compensation requires that a set of actions taken by an activity can be accounted for after the activity has already completed its work. In order to enable this, the state for that work must be able to be saved and restored. In order to compensate for the changes made by a particular instance, and not some other instance, the activity must be able to have state managed for each iteration. Thus, use of the AEC for each iteration of a set of activities helps to enable compensation by allowing an activity to gain access to the correct state that it used when executing.

The AEC acts as the container for the runtime and application state, and each workflow will have one default AEC for the root activity (the workflow itself) and may also have child AECs if activities create any. For example, if you have a workflow with a single While activity and that While activity has a single Code activity, as shown in Figure 1, then each iteration of the While activity will create a child AEC.

Figure 1 ActivityExecutionContexts with a Simple While Activity
Figure 1 ActivityExecutionContexts with a Simple While Activity(Click the image for a smaller view)
Figure 1 ActivityExecutionContexts with a Simple While Activity
Figure 1 ActivityExecutionContexts with a Simple While Activity (Click the image for a larger view)

Workflow Development

A common problem that developers encounter is trying to write code in an event handler when the activity appears in a repeating activity. For example, suppose you have a series of activities within a While activity and you want to reference one from an event handler of another activity, as shown in Figure 2. For purposes of demonstration, I will use the Code activity and a custom ReadLine activity that reads input from the console. The ReadLine activity has a single property that exposes the text that was entered at the console. In this sample, I want to access the Text property of the ReadLine in the ExecuteCode event handler of the Code activity.

Figure 2 Event Handlers
Figure 2 Event Handlers

Your first thought may be that I am going to write code that accesses the ReadLine activity directly:

Console.WriteLine(“Text from ReadLine2: {0}”, readLine2.Text);

However, if I do this and run the workflow, I get output that looks like that shown in Figure 3. The problem is that I am referencing the ReadLine activity from the template and not the instance that was cloned for this iteration of the While activity. One common approach to address this involves gaining a reference to the current cloned instance of the activities in order to reference the correct cloned instance of the target—in this case, the ReadLine activity.

Figure 3 Missing Output with a Direct Reference
Figure 3 Missing Output with a Direct Reference(Click the image for a smaller view)
Figure 3 Missing Output with a Direct Reference
Figure 3 Missing Output with a Direct Reference (Click the image for a larger view)

In order to get a reference to the correct tree of cloned activities in an event handler, you can use the sender of the event, which will be the activity the event is defined on. In my sample, this is the Code activity. From there, I can navigate the activity tree to find the activity I am interested in. This approach is shown in the following code (I can use this code in place of the previous code to get the outcome I desire):

CodeActivity thisActivityInstance = sender as
  CodeActivity;
CompositeActivity parent =
  thisActivityInstance.Parent as CompositeActivity;

MSDNMag.WFActivities.ReadLine target =
  (MSDNMag.WFActivities.ReadLine)
  parent.GetActivityByName(“readLine2”);
Console.WriteLine(
  “Text from ReadLine in this AEC: {0}”, target.Text);

In this example, I get a reference to the instance of the Code activity I am working with and navigate via the parent activity to the correct child activity. Then I can access that particular instance to get the value of the Text property from this instance.

Another option is to use shared data between activities. This would allow an activity that is executing in a child AEC to access data from the default AEC—that of the workflow itself. In my example, I would define a string variable on the workflow class and then bind the Text property of the ReadLine activity to this property. This would ensure that when the ReadLine is executed, this property would be set with the correct value. The Code activity could then reference the data on the workflow which would be up to date. Using this model, I avoid the issue of the cloned activities and referencing the template as I am referencing the single instance of the workflow.

This approach works well for the While activity, where only a single iteration is active at a given time, but this approach gets more complicated when you start executing activities in parallel using the Replicator, ConditionedActivityGroup, or a custom activity. In that case, the shared state would need to be managed in a keyed collection of some sort, which would then require managing a key as well.

The cleanest and, I would argue, best solution is to use a custom activity and activity binding. For my simple example, I could use the custom WriteLine activity and bind its Text property to the Text property of the ReadLine activity. When you use activity binding within an AEC, the target activity is correctly resolved to the cloned instance within the same AEC. This provides a simpler model for connecting your activities and getting the results you expect.

Each of these three models for working with AECs is shown in the sample code available for this column on the MSDN® Magazine Web site. The AECSamples workflow contains three While activities, each showing one of the described methods for getting the results you expect.

State Machine workflows often cause problems similar to those described in the previous example for similar, though not always apparent, reasons. State machine workflows are intended to model process execution based on a set of changing states. The power of this model is that the process, or the thing that it models, can enter a given state more than once, providing a more flexible model than sequential workflow. Because of this design, it is possible that a given State activity can be executed more than once and each iteration must therefore be executed in a separate AEC. This means each State activity acts as a template and is cloned and run in a new AEC each time the state is entered. In addition, the State activity also runs each of the EventDriven activities it contains in a separate AEC. The same solutions I described earlier can be used in the state machine: activity tree traversal, shared state, and activity binding. The AECStateMachine sample workflow in the code download provides a concrete example.

Rules and Conditions

Another area where developers struggle with the AEC is when writing rules or conditions. Activity conditions are used to make decisions in a workflow, such as when you define the condition for the IfElse activity. There are two types you can usually choose from: CodeCondition and DeclarativeRuleCondition. The CodeCondition allows you to write some code in an event handler that sets a Boolean flag indicating the outcome of the condition check. A DeclarativeRuleCondition references a rule condition statement, which also evaluates to a Boolean value.

When using a CodeCondition within an AEC (an IfElse activity within an EventDriven activity in a state machine, for example), you see the same issues discussed previously with referencing activities in the AEC. A CodeCondition that tries to directly reference the ReadLine activity that begins the steps in an EventDriven activity will be accessing the template instead of the currently executing activity. After all, a code condition uses delegates just like the event handlers on an activity. For example, you might try to write this code to evaluate your condition and return True if the user enters "quit" on the console.

if (readLine1.Text != null && readLine1.Text.StartsWith(“quit”))
{
    e.Result = true;
}

Unfortunately, the Text property will always be null as the readLine1 instance is the template and a cloned instance will be used for each execution of the State. I can solve this problem using shared state at the workflow level or by getting access to the current activity tree using the sender parameter and navigating up and back down the tree:

//get the grandparent activity (the eventdriven)
CompositeActivity grandparent = (sender as Activity).Parent.Parent;

//figure the result by finding the child ReadLine activity
e.Result = ((MSDNMag.WFActivities.ReadLine)
    grandparent.GetActivityByName(
        “readLine1”)).Text.StartsWith(“quit”);

Note that I accessed the grandparent and not the parent activity. In the case of the IfElse activity, the condition is actually invoked by an IfElseBranch activity, which is a child of the IfElse activity. If I had used the parent, I would have been incorrectly referencing the IfElse activity and not the EventDriven. In this case, I would never find the readLine1 activity with the GetActivityByName method. My point is that when you use this model of reference resolving, you need to know which activity is acting as the sender so that you can properly traverse the activity tree.

The more challenging problems usually occur when you attempt to use the DeclarativeRuleCondition within a child AEC or you try to use the Policy activity to execute a set of rules. Rule conditions and policies are authored against the workflow class itself so the rules must be written in a way to gain access to the current dynamic instance of the activities that you reference in your rules. Being able to write rules of this type often depends on the capabilities exposed by the activity that created the child AEC. You could easily make the mistake of writing a typical condition for a declarative rule as:

this.readLine2.Text.StartsWith(“quit”)

This is similar to the incorrect code in the previous example when using the code condition.

Now you might be thinking that I can get around this problem by using the GetActivityByName method to get the current instance I want:

((MSDNMag.WFActivities.ReadLine)this.GetActivityByName(
    “AECRulesInitialState”).GetActivityByName(
        “eventDrivenActivity1”).GetActivityByName(
            “readLine2”)).Text.StartsWith(“quit”)

Unfortunately, this provides identical results, as it gets the exact same templates instead of the current executing instance. In order to get at the currently executing clone of the activities, you must depend on the capabilities exposed by the activity that created the new AEC. For example, the State activity exposes a method called GetDynamicActivity, which allows you to query for the currently executing instance of a child activity. Using this method, I can now write a declarative rule that correctly identifies the current instance of the ReadLine so I can make my decision correctly:

((MSDNMag.WFActivities.ReadLine)
    ((System.Workflow.Activities.StateActivity)
        this.GetDynamicActivity(
            “AECRulesInitialState”)).GetDynamicActivity(
                 “eventDrivenActivity1”).GetActivityByName(
                     “readLine2”)).Text.StartsWith(“quit”)

In this declarative rule, I first use the GetDynamicActivity method to get the currently executing instance of the AECRulesInitialState state activity. I then cast the result to a State activity, which allows me to use the GetDynamicActivity method again to access the currently executing instance of the EventDriven activity. Once I have a reference to the EventDriven activity, I no longer have to worry about any child AEC so I can use the GetActivityByName method to get a reference to the ReadLine activity and its Text property.

This same approach can be used with other activities in both state machine and sequential workflows. For example, the While activity has a property named DynamicActivity, which can be used to access the current clone of the activity template in event handlers or declarative rule conditions. The code download includes the state machine sample as well as a sequential workflow showing the use of the While activity.

Custom Activities

The issues I’ve discussed so far are relevant to all workflow developers and are essential to your ability to create workflows. Activity authors, however, also have to understand when and how to create and manage AECs when building composite activities.

When determining whether your custom composite activity needs to create a new AEC, ask yourself, "will my activity execute its child activities more than once?" If the answer is yes, then you need to create a new AEC for each iteration of the child activities. Consider two activities included in the base activity library of Windows Workflow Foundation: Parallel and While. I have already shown you that the While activity iteratively executes its child activity and must therefore run each iteration in a new AEC. The Parallel activity allows you to define multiple branches of execution and schedules each branch to run in an interleaved fashion. Even though the Parallel activity is executing multiple sequences of child activities, it is only executing each child one time. Even if the definitions of two branches appear identical, they are separate instances of the activity types and they execute independently.

To examine the process of writing this type of activity, I will create a ForEach activity that will iterate over a collection of data objects and execute its child activity once for each item in the collection. The full sample code is available in the code download for this column, but here I will focus on the code specifically related to managing the AECs.

First, I need three dependency properties to help manage state and data: CurrentIteration (a private property to manage the index), Items (the collection of data to be iterated over), and DataItem (an attached property to pass the item from the collection to the child activity being executed). There is nothing special about these properties related to the AEC; they are simply the state you need to manage to execute multiple iterations of the child activity.

Next, you need to work on the Execute method of the ForEach activity, which is responsible for starting the execution of the child activity:

if (EnabledActivities.Count == 0 || Items == null || Items.Count == 0)
    return ActivityExecutionStatus.Closed;

CurrentIteration = 0;

ExecuteIteration(executionContext);

return ActivityExecutionStatus.Executing;

It is important to quickly check to make sure there is work to do in the form of a child activity and some collection of data to iterate over. If not, the activity can close and execution can move on to the next activity in the workflow. If there is data and a child activity, then the iteration index is initialized and I call the ExecuteIteration method, which is where the work starts to get interesting. Finally, because I am scheduling the child to execute, I return a status of Executing.

The ExecuteIteration method is responsible for managing the creation of a new AEC in order to run the child activity:

Activity act = EnabledActivities[0];

ActivityExecutionContext newContext =
    parentCtx.ExecutionContextManager.CreateExecutionContext(act);

newContext.Activity.RegisterForStatusChange(
    Activity.ClosedEvent, this);
newContext.Activity.SetValue(
    ForEach.DataItemProperty, Items[CurrentIteration]);

newContext.ExecuteActivity(newContext.Activity);

The first step is to create a new ActivityExecutionContext using the ActivityExecutionContextManager (AECManager). The AECManager is available from any AEC and is the object used to control the creation and completion of child AECs. When the new AEC is created, the AECManager clones the activity that was passed in and uses the new activity when executing. That is why, in the following steps, I use the Activity property on the new AEC to refer to the activity when registering for status changes and asking the context to execute the activity. The only other item of note here is that I set the DataItemProperty on the child activity to the current data from the collection so the data is available to the dynamically executing activity.

Notice that to create the AEC, I pass in an activity that will act as the root activity in the AEC. This is an important concept to understand as it guides the design of your activity when you know that you need to create a new AEC based on a single activity. Most activities that create a new AEC, like the While and Replicator activities, only allow a single child activity. At first glance, this can seem like a limitation, but you can easily add a Sequence activity as the single child and then add as many children to the Sequence as needed. There are two steps to ensuring that you have a single child activity. The first is to create a custom validator, which ensures that your activity only has one child. The second is to create a designer, which does not allow more than one activity to be added to the ForEach. Both of these techniques are included in the sample code for the ForEach activity. For more discussion on designers and validators, see my December 2006 article "Windows Workflow: Build Custom Activities to Extend the Reach of your Workflows".

The final step of managing the AEC is to handle the Closed event of the child activity in order to complete the context. The sender parameter to this delegate is the AEC for my composite activity and can be used to control the activity and to get access to the AECManager in order to manage child AECs. The ActivityExecutionStatusChangedEventArgs contains a reference to the activity that has closed, which can then be used to get the correct AEC for that activity using the GetExecutionContext method of the AECManager. Once I have a reference to the AEC for the activity, I close that AEC using the AECManager. It is important that, in addition to all of the child activities being closed, the child AECs all get completed. Once the AEC has been completed, I can check to see if I have more data items. If so, I start another iteration and if not, I close my activity:

ActivityExecutionContext ctx = sender as ActivityExecutionContext;

e.Activity.UnregisterForStatusChange(Activity.ClosedEvent, this);

ActivityExecutionContext childContext =
    ctx.ExecutionContextManager.GetExecutionContext(e.Activity);
ctx.ExecutionContextManager.CompleteExecutionContext(childContext);

if (++CurrentIteration >= Items.Count) ctx.CloseActivity();
else ExecuteIteration(ctx);

The CompleteExecutionContext method has an overload that takes a Boolean parameter to indicate if the workflow state should be persisted at this point. By default, this value is False. However, if any child activities that implement the ICompensatable interface have completed, the AEC will be persisted regardless of the value passed for this parameter. This overload allows you to provide input as to whether the AEC needs to be persisted based on the needs of your activity and whether the state will need to be available later.

The last thing to do in my activity is to support the ability of other code to get a reference to the current instance of the child activity being executed. To do that, I add a DynamicActivity property on the ForEach activity and use the GetDynamicActivities method from the CompositeActivity base class. This method takes in an activity and returns an array of the dynamic instances of that activity. In this case, there will only ever be one dynamic instance at a time, so I can always return the first item in the array:

public Activity DynamicActivity
{
    get
    {
        if (EnabledActivities.Count == 0) return null;
        Activity[] activities =
            GetDynamicActivities(EnabledActivities[0]);
        if (activities.Length > 0) return activities[0];
        return null;
    }
}

It is important to keep in mind that other aspects of composite activity development might also involve managing the AEC—cancellation and fault handling for example. In these cases, the pattern for accessing the correct AEC is the same as shown here and the steps you take will depend upon your activity.

Conclusion

It is critical that workflow developers understand how the AEC works and when it is necessary to consider using the AEC. Likewise, it is essential for developers to know how to code workflows, activities, and rules in order to deal with the AEC. Writing a simple custom composite activity is a good way to get an understanding of how other activities may be managing their AEC. I recommend this as a learning technique for most workflow developers.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. As an instructor for Pluralsight, Matt teaches courses on Workflow, BizTalk Server, and Windows Communication Foundation. Matt lives in Minnesota with his wife Kristen and his two sons. Contact Matt via his blog at pluralsight.com/blogs/matt.


Concurrent Affairs: Asynchronous Device Operations

Concurrent Affairs
Asynchronous Device Operations
Jeffrey Richter

Code download available at: ConcurrentAffairs2007_06.exe (162KB)
Contents
CreateFile and DeviceIoControl
Win32 Device Communication
Synchronous Device I/O in Managed Code
Asynchronous Device I/O in Managed Code
DeviceIoControl’s Overlapped Parameter
Opportunistic Locks
Conclusion

In my last column, I demonstrated how to implement two base classes, AsyncResultNoResult and AsyncResult<TResult>. Both of these classes implement the IAsyncResult interface, which is at the core of the common language runtime (CLR) Asynchronous Programming Model (APM). In this column, I’m going to use the AsyncResult<TResult> class to implement the APM in a way that allows you to perform hardware device operations asynchronously. Reading this column will also give you insight into how the Microsoft® .NET Framework itself performs asynchronous file and network I/O operations. All of this code resides in my Power Threading library, available from wintellect.com.

CreateFile and DeviceIoControl

In Win32®-based coding, when you want to work with a file, you open it by calling the Win32 CreateFile function and then perform operations on the file by calling functions such as WriteFile and ReadFile. Ultimately, when you’re finished working with the file, you close it by calling the CloseHandle function.

In the .NET Framework, constructing a FileStream object internally calls CreateFile. Calling FileStream’s Read and Write methods internally calls the Win32 WriteFile and ReadFile functions respectively. And Calling FileStream’s Close or Dispose methods internally calls the Win32 CloseHandle function. When working with files, most applications just need to write data to and read data from the file and so the FileStream class offers most of the operations you need.

However, Windows® actually offers many more operations that can be performed on a file. For the more common operations, Windows provides specific Win32 functions like WriteFile, ReadFile, FlushFileBuffers, and GetFileSize. However, for infrequently used operations, Win32 doesn’t supply specific functions; instead, it offers one function, DeviceIoControl, that lets an application communicate directly with the device driver (such as the NTFS disk driver) responsible for manipulating the file. Examples of infrequently used file operations include opportunistic locking (microsoft.com/msj/0100/win32/win320100.aspx), manipulating the change journal (microsoft.com/msj/0999/journal/journal.aspx), compressing disk volumes/files, creating junction points, formatting/repartitioning disks, and working with sparse files.

Furthermore, DeviceIoControl can be used by any application to communicate with any hardware device driver. Using DeviceIoControl, an application can query the computer’s battery status, query or change the LCD’s brightness, manipulate a CD or DVD changer, query and eject USB devices, and much more.

Win32 Device Communication

Let’s take a quick look at the Win32 way of having an application communicate with a device and then wrap this functionality so that it can be used from managed code.

In Win32, the way you open a device is by calling the CreateFile function. CreateFile’s first argument is a string identifying the device to open. Normally, a path name is specified for the string causing CreateFile to open a file. However, you can pass special strings to CreateFile, causing it to open devices. Figure 1 shows some potential strings and describes the kind of device the string causes CreateFile to open. Be aware that some of these devices are restricted to the system or members of the Administrators group and so attempting to open some of these devices may fail unless your application is running with the required access permissions.

Figure 1 Special CreateFile Strings (Close Figure 1)

String Passed to CreateFile | Description
@“\\.\PhysicalDrive1” | Opens hard drive 1, allowing access to all its sectors.
@“\\.\C:” | Opens the C: disk volume, allowing you to manipulate this volume’s change journal.
@“\\.\Changer0” | Opens disc changer 0, allowing you to move discs around and perform other operations.
@“\\.\Tape0” | Opens tape drive 0, allowing you to back up and restore files.
@“\\.\COM2” | Opens communication port 2, allowing you to send and receive bytes.
@“\\.\LCD” | Opens the LCD device, allowing you to query and adjust brightness.

Now that you know how to open a device, let’s look at the Win32 function DeviceIoControl:

BOOL DeviceIoControl(
   HANDLE hDevice,           // Handle returned from CreateFile
   DWORD  dwIoControlCode,   // Operation control code
   PVOID  pInBuffer,         // Address of input buffer
   DWORD  nInBufferSize,     // Size, in bytes, of input buffer
   PVOID  pOutBuffer,        // Address of output buffer
   DWORD  nOutBufferSize,    // Size, in bytes, of output buffer
   PDWORD pBytesReturned,    // Gets number of bytes written to
                             // pOutBuffer
   POVERLAPPED pOverlapped); // For asynchronous operation

When calling this function, the handle parameter refers to a file, directory, or device driver obtained by calling CreateFile. The dwIoControlCode parameter indicates the operation to perform on the device. Each operation is simply identified via this 32-bit integer code. If an operation requires arguments, then the arguments are placed in fields of a data structure and the address of the structure is passed for the pInBuffer parameter. The size of the structure is passed in the nInBufferSize parameter. Similarly, if the operation returns some data, then the output data is placed in a buffer, which must be allocated by the caller. The address of this buffer is passed in the pOutBuffer parameter. The size of this buffer is passed in the nOutBufferSize parameter. When DeviceIoControl returns, it writes the number of bytes actually written to the output buffer in the pBytesReturned parameter (which is passed by reference). Finally, when performing synchronous operations, NULL is passed for the pOverlapped parameter. But when performing an asynchronous operation, the address of an OVERLAPPED structure must be passed in. I’ll explain this is great detail later in this column.

Now that you have a sense of how to communicate with devices via Win32, let’s start writing a managed wrapper over these Win32 functions so that you can write managed code in a .NET language to communicate directly with hardware devices.

Synchronous Device I/O in Managed Code

First, let’s focus on how to perform synchronous device operations. Later, I’ll add the code necessary to perform asynchronous operations. In C#, I have defined a static DeviceIO class that is a friendly wrapper around the Win32 CreateFile and DeviceIoControl functions. Figure 2 shows the public object model for this class.

Figure 2 Wrapping CreateFile and DeviceIoControl (Close Figure 2)
public static class DeviceIO {
   public static SafeFileHandle OpenDevice(String deviceName,
      FileAccess access, FileShare share, Boolean useAsync);

   public static void Control(SafeFileHandle device,
      DeviceControlCode deviceControlCode);

   public static void Control(SafeFileHandle device,
      DeviceControlCode deviceControlCode, Object inBuffer);

   public static TResult GetObject<TResult>(SafeFileHandle device,
      DeviceControlCode deviceControlCode) where TResult: new();

   public static TResult GetObject<TResult>(SafeFileHandle device,
      DeviceControlCode deviceControlCode, Object inBuffer)
          where TResult : new();

   public static TElement[] GetArray<TElement>(SafeFileHandle device,
      DeviceControlCode deviceControlCode, Object inBuffer,
      Int32 maxElements) where TElement : struct;
}

In a managed application, you can call OpenDevice, passing in a string similar to those shown in Figure 1, plus the desired access and sharing. OpenDevice will internally call CreateFile returning the handle to the device. The handle is actually returned wrapped in a SafeFileHandle object to ensure that it will ultimately be closed and to get the other benefits offered by way of all SafeHandle-derived classes. This SafeFileHandle object can be passed as the first argument to any of DeviceIO’s other static methods and this is ultimately passed as the first argument when calling the Win32 DeviceIoControl function.

The second argument to all the other methods is a DeviceControlCode, a simple structure (value type) that contains just one private instance field, an Int32, that identifies a command code you want to send to a device. I found that wrapping the Int32 command code into its own type caused the code to read better, be more type-safe, and benefit from IntelliSense® in Visual Studio®. Internally, the Int32 value contained inside a DeviceControlCode instance is passed as the second argument to the Win32 DeviceIoControl function.

Now, when examining device control codes, you’ll detect that there are a few common patterns, and I decided to offer friendly methods to make working with these common patterns convenient. All of these methods execute the operation synchronously; that is, the calling thread will not return until the operation is complete. Also, when calling any of these methods, you must specify false for the useAsync argument when calling OpenDevice.

The Control method encapsulates a pattern where the control code you’re sending to a device causes the device to perform some operation and the device has no resulting data to return. There is a Control method that takes just a SafeFileHandle and a control code and an overload that takes an additional Object argument. The additional Object argument allows you to pass some additional data to the device that it will use for an operation. When you look up control codes in the Win32 SDK documentation, the documentation will indicate what additional information (if necessary) you must pass for each control code. If you need additional data, you must define a managed type that is the equivalent of the Win32 data type, construct an instance of it, initialize its fields, and pass a reference to the instance for the inBuffer parameter.

The GetObject<TResult> method encapsulates a pattern where the control code you’re sending to a device causes the device to return some data back to your application. To get this return data, you must define a managed type that is the equivalent of the Win32 data type, and specify this type as the generic type, TResult, when calling GetObject. Internally, GetObject will create an instance of this type (which is why the generic type has the new constraint applied to it), the device will initialize the fields inside the call to DeviceIoControl, and GetObject will return the initialized object back out to you. When calling GetObject, there are two overloads allowing you to optionally pass additional data to the device via the inBuffer parameter.

The GetArray<TElement> method encapsulates the last pattern where the control code you’re sending to a device causes the device to return an array of elements to your application. To get these elements, you must define a managed type that is the equivalent of the data type of the Win32 element, and specify this type as the generic type, TElement, when calling GetArray. The managed data type must be defined as a structure (value type) so that the array’s memory is laid out linearly as required by DeviceIoControl; this is why TElement is constrained to struct. Also, when calling GetArray, you must specify the maximum number of elements that you want returned from the device via the maxElements parameter.

Internally, GetObject will create an array of these elements and pass the address of the array to DeviceIoControl, which will initialize the individual array elements. When DeviceIoControl returns, it returns the number of bytes actually placed in the array. GetArray uses this value to shrink (if necessary) the array to the exact size so that the number of elements in the array is equal to the number of elements initialized. This is a big convenience as any code that uses this array (returned from GetArray) can simply query the array’s Length property to get the number of items and use this value in its loop to process the returned elements.

Figure 3 shows the managed equivalent of the Win32 DisplayBrightness structure and its helper DisplayPowerFlags enumerated type. Figure 4 shows an AdjustBrightness method that uses my static DeviceIO class to constantly adjust your LCD’s brightness up and down. I must say that this is not a very useful application in real life, but it is a simple example that shows the basic concepts.

Figure 4 Adjusting LCD Brightness (Close Figure 4)
public static void AdjustBrightness(6) {
   // The code for setting LCD brightness; obtained by looking up
   // IOCTL_VIDEO_SET_DISPLAY_BRIGHTNESS in Platform SDK documentation
   DeviceControlCode s_SetBrightness =
      new DeviceControlCode(DeviceType.Video, 0x127,
         DeviceMethod.Buffered, DeviceAccess.Any);

   // Open the device
   using (SafeFileHandle lcd = DeviceIO.OpenDevice(@”\\.\LCD”,
      FileAccess.ReadWrite, FileShare.ReadWrite, false)) {

      for (Int32 times = 0; times < 10; times++) {
         for (Int32 dim = 0; dim <= 100; dim += 20) {
            // Initialize the equivalent of a Win32 DISPLAY_BRIGHTNESS
            // structure
            DisplayBrightness db =
               new DisplayBrightness(DisplayPowerFlags.ACDC,
                  (Byte)((times % 2 == 0) ? dim : 100 - dim));

            // Tell the LCD device to adjust its brightness
            DeviceIO.Control(lcd, s_SetBrightness, db);

            // Sleep for a bit and then adjust it again
            Thread.Sleep(150);
         }
      }
   }
}
Figure 3 Managed DisplayBrightness Structure (Close Figure 3)
[Flags]
internal enum DisplayPowerFlags : byte {
   None = 0x00000000, AC = 0x00000001, DC = 0x00000002, ACDC = AC | DC
}

internal struct DisplayBrightness {
   public DisplayPowerFlags m_Power;
   public Byte m_ACBrightness; // 0-100
   public Byte m_DCBrightness; // 0-100

   public Win32DisplayBrightnessStructure(
         DisplayPowerFlags power, Byte level) {
      m_Power = power;
      m_ACBrightness = m_DCBrightness = level;
   }
}

Asynchronous Device I/O in Managed Code

Some device operations, like changing LCD brightness, are not really I/O operations and so it makes sense to call DeviceIoControl synchronously when performing these kinds of operations. However, most calls to DeviceIoControl result in I/O (hence the name of the function) and, therefore, it makes sense to execute these operations asynchronously. In this section, I’ll explain how to P/Invoke from managed code out to the Win32 DeviceIoControl function in order to perform asynchronous I/O operations to files, disks, and other hardware devices.

My static DeviceIO class offers asynchronous versions of the Control, GetObject, and GetArray methods by way of the CLR’s APM. Figure 5 shows the public methods (not shown earlier) of this class that support the APM.

Figure 5 DeviceIO Asynchronous Methods (Close Figure 5)
public static class DeviceIO {
   public static IAsyncResult BeginControl(SafeFileHandle device,
      DeviceControlCode deviceControlCode, Object inBuffer,
      AsyncCallback asyncCallback, Object state);

   public static void EndControl(IAsyncResult result);

   public static IAsyncResult BeginGetObject<TResult>(
      SafeFileHandle device, DeviceControlCode deviceControlCode,
      Object inBuffer, AsyncCallback asyncCallback, Object state)
      where TResult: new();

   public static TResult EndGetObject<TResult>(IAsyncResult result)
      where TResult: new();

   public static IAsyncResult BeginGetArray<TElement>(
      SafeFileHandle device, DeviceControlCode deviceControlCode,
      Object inBuffer, Int32 maxElements, AsyncCallback asyncCallback,
      Object state) where TElement: struct;

   public static TElement[] EndGetArray<TElement>(IAsyncResult result)
      where TElement: struct;
}

As you can see, all the BeginXxx methods follow the CLR’s APM pattern; that is, they all return an IAsyncResult and the last two parameters are an AsyncCallback and an Object. The additional parameters match those of each method’s synchronous counterpart. Similarly, all the EndXxx methods accept a single parameter, an IAsyncResult, and each returns the same data type as each method’s synchronous counterpart.

In order to call any of these asynchronous methods, two things must happen. First, you must tell Windows that you want to perform operations on the device asynchronously. This is accomplished by passing the FILE_FLAG_OVERLAPPED flag (which has a numeric value of 0x40000000) to the CreateFile function. The managed equivalent of this flag is FileOptions.Asynchronous (which, of course, also has a value of 0x40000000). Second, you have to tell the device driver to post operation completion entries into the CLR’s thread pool. This is accomplished by calling ThreadPool’s static BindHandle method, which takes one parameter, a reference to a SafeHandle-derived object.

DeviceIO’s OpenDevice method performs both of these actions when you pass true for its useAsync parameter. Here is how my OpenDevice method is implemented:

public static SafeFileHandle OpenDevice(String deviceName,
   FileAccess access, FileShare share, Boolean useAsync) {

   SafeFileHandle device = CreateFile(deviceName, access, share,
      IntPtr.Zero, FileMode.Open,
      useAsync ? FileOptions.Asynchronous : FileOptions.None,
      IntPtr.Zero);

   if (device.IsInvalid) throw new Win32Exception();
   if (useAsync) ThreadPool.BindHandle(device);

   return device;
}

Now, let’s take a look at how BeginGetObject and EndGetObject work internally. The other asynchronous methods work similarly and you will be able to understand how they work easily after walking through BeginGetObject and EndGetObject. All of the BeginXxx methods are very simple; they construct an object or array if the operation returns some return value or array of elements, and then they immediately call an internal helper method, AsyncControl, which is implemented as shown in Figure 6.

Figure 6 AsyncControl (Close Figure 6)
private static DeviceAsyncResult<T> AsyncControl<T>(
    SafeFileHandle device,
   DeviceControlCode deviceControlCode, Object inBuffer, T outBuffer,
   AsyncCallback asyncCallback, Object state) {

   SafePinnedObject inDeviceBuffer = new SafePinnedObject(inBuffer);
   SafePinnedObject outDeviceBuffer = new SafePinnedObject(outBuffer);
   DeviceAsyncResult<T> asyncResult = new DeviceAsyncResult<T>(
      inDeviceBuffer, outDeviceBuffer, asyncCallback, state);

   unsafe {
      Int32 bytesReturned;
      NativeControl(device, deviceControlCode, inDeviceBuffer,
          outDeviceBuffer, out bytesReturned,
          asyncResult.GetNativeOverlapped());
   }
   return asyncResult;
}

When performing an operation, you can pass the address of a buffer containing additional input data to DeviceIoControl. If DeviceIoControl returns data, the memory for that data must be allocated before calling DeviceIoControl so that DeviceIoControl can initialize the data; the data is then examined when the operation completes. The problem is that the operation could theoretically take hours to complete and during this time garbage collections could occur that would move the data around. As a result, the addresses passed to DeviceIoControl would no longer refer to the desired buffers and memory corruption would occur.

You can tell the GC not to move an object around by pinning the object. My SafePinnedObject class is a helper class that pins an object in memory, preventing the GC from moving it. However, since the SafePinnedObject class is derived from SafeHandle, it comes with all the benefits you normally get with SafeHandle-derived types, including the assurance that the object will be unpinned at some point in the future. Figure 7 gives you an idea of how the SafePinnedObject class is implemented (I’ve removed some validation code to save space). For more information about SafeHandle-derived types and about pinning objects in memory, see my book CLR via C# (Microsoft Press®, 2006).

Figure 7 Pinning Objects in Memory with SafePinnedObject (Close Figure 7)
public sealed class SafePinnedObject : SafeHandleZeroOrMinusOneIsInvalid {
   private GCHandle m_gcHandle;  // Handle of pinned object (or 0)

   public SafePinnedObject(Object obj) : base(true) {
      // If obj is null, we create this object but it pins nothing
      if (obj == null) return;

      // If specified, pin the buffer and save its memory address
      m_gcHandle = GCHandle.Alloc(obj, GCHandleType.Pinned);
      SetHandle(m_gcHandle.AddrOfPinnedObject());
   }

   protected override Boolean ReleaseHandle() {
      SetHandle(IntPtr.Zero); // Just for safety, set the address to null
      m_gcHandle.Free();      // Unpin the object
      return true;
   }

   // Returns the object of a pinned buffer or null if not specified
   public Object Target {
      get { return IsInvalid ? null : m_gcHandle.Target; }
   }

   // Returns the number of bytes in a pinned object or 0 if not specified
   public Int32 Size {
      get {
         Object target = Target;

         // If obj was null, return 0 bytes
         if (target == null) return 0;

         // If obj is not an array, return the size of it
         if (!target.GetType().IsArray) return Marshal.SizeOf(target);

         // obj is an array, return the total size of the array
         Array a = (Array)target;
         return a.Length * Marshal.SizeOf(a.GetType().GetElementType());
      }
   }
}

After pinning both the input and output buffers, AsyncControl constructs a DeviceAsyncResult<TResult> object passing into it the two SafePinnedObject objects, an AsyncCallback, and an Object. DeviceAsyncResult<TResult> is another type internal to my implementation. This type is derived from AsyncResult<TResult> (as discussed in my last column), which means this type implements the APM’s IAsyncResult interface. When constructed, the DeviceAsyncResult object simply saves all the arguments passed to it in private fields and then returns.

DeviceIoControl’s Overlapped Parameter

Now that all this prep work is complete, AsyncControl is ready to call DeviceIoControl. It does that by calling NativeControl passing it the device handle, the control code, the input buffer, the output buffer, and a reference to an Int32 bytesReturned variable (which will basically be ignored by the method since the method will return before the operation completes). The most important argument is the last one: when performing an asynchronous operation, DeviceIoControl must be passed the address of a NativeOverlapped structure (the equivalent of a Win32 OVERLAPPED structure). AsyncControl obtains this by calling a helper method, GetNativeOverlapped, defined in my DeviceAsyncResult class. This helper method is implemented as follows:

// Create and return a NativeOverlapped structure to be passed
// to native code
public unsafe NativeOverlapped* GetNativeOverlapped() {
   // Create a managed Overlapped structure that refers to our
   // IAsyncResult (this)
   Overlapped o = new Overlapped(0, 0, IntPtr.Zero, this);

   // Pack the managed Overlapped structure into a NativeOverlapped
   // structure
   return o.Pack(CompletionCallback,
      new Object[] { m_inBuffer.Target, m_outBuffer.Target });
}

This method constructs an instance of the System.Threading.Overlapped class and initializes it. This is a managed helper class that lets you set up and manipulate an overlapped structure. However, you cannot pass a reference to this object to native code. Instead you must first pack the Overlapped object into a NativeOverlapped object; a reference to the resulting NativeOverlapped object can then be passed to native code. When you call Pack, you pass it a delegate referring to a callback method that a thread pool thread will execute when the operation completes. In my code, this is the CompletionCallback method.

Calling the Overlapped class’s Pack method does several things. It allocates memory for a native OVERLAPPED structure from the managed heap and pins it, which ensures that the memory will not be moved should a GC occur. It then initializes the fields of the NativeOverlapped structure’s field from the fields set in the managed Overlapped object. This includes creating a normal GCHandle for the IAsyncResult object that the Overlapped object refers to ensuring that the IAsyncResult object stays alive for the duration of the operation.

Pack then pins the callback method (CompletionCallback) delegate so that the fixed address can be passed to native code, thus allowing native code to call back to managed code when this operation completes. Pack also pins any additional objects referenced by its second parameter, an Object array. In my case, the inBuffer and outBuffer objects are already pinned because I wrapped them using my SafePinnedObject class. I need to pin them myself so that I can get their address by calling GCHandle’s AddrOfPinnedObject method.

However, the Pack method pins them again, but in a special way. Normally, when an AppDomain is unloaded, the CLR automatically unpins any objects, allowing them to be collected and thus preventing a memory leak. But Pack pins the objects until the asynchronous operation completes. So if an asynchronous operation is started and then the AppDomain is unloaded, the Pack-pinned objects will not be unpinned. After the operation completes, the objects will be unpinned allowing them to be collected; this prevents memory corruption.

Pack also records which AppDomain called Pack to ensure that the thread pool thread that calls the CompletionCallback method executes in the same AppDomain. Finally, Pack captures the stack of code access security permissions so that when the callback method executes, it executes under the same security permissions that the code initiating the asynchronous operation was under. You can call the Overlapped class’s UnsafePack method if you prefer not to propagate the stack of security permissions.

Pack returns the address of the pinned NativeOverlapped structure; this address can be passed to any native function that expects the address to a Win32 OVERLAPPED structure. In my AsyncControl method, the NativeOverlapped structure’s address is passed to NativeControl (see Figure 8), which passes it to the Win32 DeviceIoControl function.

Figure 8 Native Control (Close Figure 8)
private static unsafe void NativeControl(SafeFileHandle device,
   DeviceControlCode deviceControlCode,
   SafePinnedObject inBuffer, SafePinnedObject outBuffer,
   out Int32 bytesReturned, NativeOverlapped* nativeOverlapped) {

   Boolean ok = DeviceIoControl(device, deviceControlCode.Code,
      inBuffer, inBuffer.Size, outBuffer, outBuffer.Size,
      out bytesReturned, nativeOverlapped);
   if (ok) return;

   Int32 error = Marshal.GetLastWin32Error();
   const Int32 c_ErrorIOPending = 997;
   if (error == c_ErrorIOPending) return;
   throw new InvalidOperationException(
      String.Format(“Control failed (code={0})”, error));
}

When performing an asynchronous operation, DeviceIoControl returns immediately and, ultimately, the DeviceAsyncResult object that implements the IAsyncResult interface is returned from DeviceIO’s BeginObject<TResult> method.

When the operation completes, the device driver queues an entry in the CLR’s thread pool. You’ll recall that the driver knows to do this because DeviceIO’s OpenDevice method internally calls ThreadPool’s BindHandle method when the device is opened for asynchronous access. Eventually, a thread pool thread will extract this queued entry, adopt any saved permission sets, jump into the right AppDomain, and call the CompletionCallback method (see Figure 9). Don’t forget that the CompletionCallback method is defined inside the DeviceAsyncResult<TResult> class, which is derived from the AsyncResult<TResult> class.

Figure 9 CompletionCallback (Close Figure 9)
// Called by a thread pool thread when native overlapped I/O completes
private unsafe void CompletionCallback(UInt32 errorCode, UInt32 numBytes,
   NativeOverlapped* nativeOverlapped) {
   // Release the native OVERLAPPED structure and
   // let the IAsyncResult object (this) be collectable.
   Overlapped.Free(nativeOverlapped);

   try {
      if (errorCode != 0) {
         // An error occurred, record the Win32 error code
         base.SetAsCompleted(new Win32Exception((Int32)errorCode), false);
      } else {
         // No error occurred, the output buffer contains the result
         TResult result = (TResult)m_outBuffer.Target;

         // If the result is an array of values, resize the array
         // to the exact size so that the Length property is accurate
         if ((result != null) && result.GetType().IsArray) {
            // Only resize if the number of elements initialized in the
            // array is less than the size of the array itself
            Type elementType = result.GetType().GetElementType();
            Int64 numElements = numBytes / Marshal.SizeOf(elementType);
            Array origArray = (Array)(Object)result;
            if (numElements < origArray.Length) {
               // Create new array (size equals number of initialized
               // elements)
               Array newArray = Array.CreateInstance(
                   elementType, numElements);

               // Copy initialized elements from original array to new
               // array
               Array.Copy(origArray, newArray, numElements);
               result = (TResult)(Object)newArray;
            }
         }
         // Record result and call AsyncCallback method passed to BeginXxx
         // method
         base.SetAsCompleted(result, false);
      }
   }
   finally {
      // Make sure that the input and output buffers are unpinned
      m_inBuffer.Dispose();
      m_outBuffer.Dispose();
      m_inBuffer = m_outBuffer = null;   // Allow early GC
   }
}

The first thing that the CompletionCallback method does is free the nativeOverlapped object. This is extremely important as it releases the GCHandle on the IAsyncResult object, unpins all the objects passed to the Overlapped class’s Pack method, and also unpins the NativeOverlapped object itself allowing it to be GC’d. Forgetting to free the NativeOverlapped object results in leaking several objects.

The rest of CompletionCallback’s code is pretty straightforward. It is just a matter of recording in the AsyncResult<T> base class’s state whether the operation failed or the result of the operation (adjusting the array size if DeviceIO’s BeginGetArray was called to initiate the operation). In CompletionCallback’s finally block, there is some code to dispose of the SafePinnedObject objects, ensuring that the input and output buffers are no longer pinned and allowing them to be compacted by the GC and ultimately have their memory reclaimed by the GC when the application no longer cares about these buffers.

At some point, the application code will call DeviceIO’s EndControl, EndGetObject, or EndGetArray method, passing in the DeviceAsyncResult<TResult> object returned from the corresponding Begin method. Internally, all these End methods just call AsyncResult<TResult>’s EndInvoke method, which either throws the exception set into it inside CompletionCallback or returns the value set into it.

Opportunistic Locks

An opportunistic lock gives you a simple example that demonstrates an asynchronous device operation. Figure 10 shows a static OpLockDemo class whose Main method creates a file (using the FileOptions.Asynchronous flag). The SafeFileHandle embedded inside the FileStream is then passed to BeginFilter, which internally calls the DeviceIO’s BeginControl method, passing the code that establishes a request filter opportunistic lock on the file. Now the file system device driver will watch to see if any other application attempts to open the file and, if so, the device driver will queue an entry to the CLR’s thread pool. This will in turn call the anonymous method that sets the s_earlyEnd field to 1, which indicates that another application wants to access the file. The first application will then close the file allowing the other application to continue running with access to the file.

Figure 10 Demonstrating an Opportunistic Lock (Close Figure 10)
internal static class OpLockDemo {
   private static Byte s_endEarly = 0;

   public static void Main() {
      String filename = Path.Combine(
         Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
         @”FilterOpLock.dat”);

      // Attempt to open/create the file for processing (must be
      // asynchronous)
      using (FileStream fs = new FileStream(
             filename, FileMode.OpenOrCreate,
         FileAccess.ReadWrite, FileShare.ReadWrite, 8096,
         FileOptions.Asynchronous)) {

         // Request the Filter Oplock on the file.
         // When another process attempts to access the file, the system
         // will
         //    1. Block the other process until we close the file
         //    2. Call us back notifying us that another process wants to
         // access the file
         BeginFilter(fs.SafeFileHandle,
            delegate(IAsyncResult result) {
               EndFilter(result);
               Console.WriteLine(“Another process wants to access “ +
                                “the file or the file closed”);
               Thread.VolatileWrite(ref s_endEarly, 1);  // Tell Main
                                                         // thread to end
                                                         // early
            }, null);

         // Pretend we’re accessing the file here
         for (Int32 count = 0; count < 100; count++) {
            Console.WriteLine(“Accessing the file ({0})...”, count);

            // If the user hits a key or if another application
            // wants to access the file, close the file.
            if (Console.KeyAvailable ||
               (Thread.VolatileRead(ref s_endEarly) == 1)) break;
            Thread.Sleep(150);
         }
      }  // Close the file here allows the other process to continue
         // running
   }

   // Establish a Request filter opportunistic lock
   private static IAsyncResult BeginFilter(SafeFileHandle file,
      AsyncCallback asyncCallback, Object state) {

      // See FSCTL_REQUEST_FILTER_OPLOCK in Platform SDK’s WinIoCtl.h file
      DeviceControlCode RequestFilterOpLock =
         new DeviceControlCode(DeviceType.FileSystem, 23,
            DeviceMethod.Buffered, DeviceAccess.Any);

      return DeviceIO.BeginControl(file, RequestFilterOpLock, null,
         asyncCallback, state);
   }

   private static void EndFilter(IAsyncResult result) {
      DeviceIO.EndControl(result);
   }
}

To test all of this, start the OpLockDemo application. Then go to your desktop and attempt to delete the FilterOpLock.dat file that the application creates. This will cause the anonymous method to get called, ending the sample application.

Conclusion

Windows and its device drivers offer many features that the .NET Framework class library does not wrap. Fortunately, though, the .NET Framework does provide the appropriate mechanisms allowing you to P/Invoke to gain access to these useful features. The fact that you can perform these operations asynchronously means you can build robust, reliable, and scalable applications that take advantage of these features.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect, an architecture review, consulting and training firm. He is the author of several books, including CLR via C# (Microsoft Press, 2006). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


Netting C++: EEK! An Introduction

Netting C++
EEK! An Introduction
Stanley B. Lippman

In my previous series of columns, we moved an existing ISO-C++ application to the Microsoft® .NET Framework incrementally, illustrating the interoperability features of C++/CLI, and highlighting differences in the C++ and .NET object models. In this next series of columns, we’ll explore using C++/CLI as a purely .NET-compliant language. The application, for reasons that will soon become apparent, is code-named EEK! Because I am currently spending most of my time working on the server side of MMOGs (Massive Multiplayer Online Games), EEK! draws from certain aspects of that domain, although simplified to accommodate the limited bandwidth of a column, and with considerably more modest ambitions.

My out-of-office hobby, in an equally modest fashion, is molecular biology and questions of how the things that happen at that level of detail makes possible the behaviors we observe around us. Mice have been extensively studied by science, perhaps more so than any other living creature with the possible exceptions of the fruit fly and E. coli (a bacteria), neither of which I find particularly compelling. But mice are strangely familiar if you observe them over time. They cooperate to retrieve a pup that has fallen out of its nest, they exhibit differences in personality, and they seem to remember and learn from experience, both individually and as a group across generations.

As a sort of domestic Dian Fossey on a shoestring budget, I have, in the past, maintained an aquarium populated with mice that I anthropologically observed (a bit randomly, admittedly) and photographed. EEK! is my attempt to model that environment and the behavior of a mouse. I’ve code-named the framework EEK! in homage to my computer graphics friends at Disney Animation. They assure me it is the sound a mouse makes when it discovers it has been vectorized.

So, how do I begin? First off, I need to uncover the set of classes with which to implement my design. Some of these fall out of the description of the application. I’ll need my tank, the mouse, some way to tick off the passage of time, and some method of displaying and possibly storing the changes that occur over time. I’ll try to make use of as many .NET technologies as possible in the course of this series.

The classes that fall out of a design are in large part determined by the intention of the application. Were I to make EEK! an online virtual environment in which users logged on as virtual mice and interacted both with one another and the environment, then the design I’m going to pursue in the following months would be less than optimal because it does not facilitate interactive gameplay. It would be, in other words, the wrong level of detail to model. That is, the design that emerges from the same domain space is largely a consequence of the intention of the application, not of the domain space itself.

While a mouse is a mammal, and therefore shares many human characteristics, its intentions seem far less complex and purposeful. So what causes a mouse to do any particular behavior at any particular time? Well, it turns out that much of its behavior is driven at a molecular level that it is totally unconscious of, often by smells in its environment—emitted by chemicals called pheromones. That is what I hope to capture and explore in this design, and what makes it challenging.

Both the tank and mouse are concrete instances of a more abstract set of classes that root the hierarchy of our simulation framework. Figure 1 gives a sense of that hierarchy.

Figure 1 Hierarchy of the Simulation Framework (Close Figure 1)
public ref class Environment abstract;
public ref class Entity abstract;
public ref class NonSentientEntity : Entity abstract;
public ref class SentientEntity : Entity abstract;

public ref class Mouse : SentientEntity;
public ref class Tank : Environment;

public ref class ExerciseWheel : NonSentientEntity;
public ref class WaterDispenser : NonSentientEntity;
Figure 1 Hierarchy of the Simulation Framework (Close Figure 1)
public ref class Environment abstract;
public ref class Entity abstract;
public ref class NonSentientEntity : Entity abstract;
public ref class SentientEntity : Entity abstract;

public ref class Mouse : SentientEntity;
public ref class Tank : Environment;

public ref class ExerciseWheel : NonSentientEntity;
public ref class WaterDispenser : NonSentientEntity;

A sentient entity is capable both of perceiving its environment and of sensing its internal state and responding with purposeful behavior. I’ll need, therefore, to represent a set of behaviors and be able to initiate, terminate, and evolve these behaviors. This means that behaviors need to capture both action and state, where state represents a form of memory and modifies subsequent behavior.

A non-sentient entity is anything in the environment that the sentient entity may interact with. It may or may not have a function, but it maintains a state that represents its presence in the environment. This presence somehow needs to be communicated.

How should the mouse become aware of its environment and the things within it? For one thing, not all entities are aware of all aspects of their environment. Similarly, not all entities become aware of aspects of their environment in the same way. For example, because a mouse is so small, its visual acuity is limited. It can distinguish light and shadow, but it does not depend on sight the way we do as humans. It compensates for that by having a far greater acuity of smell, and a number of behaviors, as I mentioned earlier, are triggered by that. Sight, smell, touch, hearing—these are spoken of as modalities, and these are what we will model.

The environment consists of a collection of modalities that change their composition over time. A modality connects an entity to its environment in an analogous fashion to the way an iterator connects a container to an algorithm in the design of the Standard Template Library (STL). That is, each entity maintains a set of streaming modalities by which it imparts information into all or a subset of the environmental modalities—it may reflect light, for example, it may impart a smell, and so on. It is this set of information that represents its presence within the environment. Each sentient entity has a set of perceptual modalities through which to sample the environmental modalities and thus to perceive the world it inhabits in EEK!

A sentient entity has bound to its perceptual modalities a set of programmed behaviors—that is, actions it makes in response to its perceptions. If the outcome of that response is memorable, either negatively or positively, the behavior is either modified or reinforced. For example, if the mouse eats what it perceives as a food item and subsequently becomes wretchedly ill, the next time the mouse perceives that item it will avoid eating it.

The mouse moves between a finite set of states. The mouse begins in a state of Rest. When stimulated, either internally or by some perception of its environment, it enters a state of Restlessness, and will execute a sequence of behaviors with the goal of returning to Rest. If it receives neither external nor internal stimulation for some period, it moves to a state of Sleep. Alternatively, if it is frustrated in its goal-seeking, it can move into a state of Distress.

I’ll model an internal stimulation as a Drive. The mouse has a Metabolic component that drives its animate life, for example. This system requires energy, and as its energy quotient falls, a metabolic drive pushes the mouse to a state of restlessness with the goal of seeking out food.

An external stimulation arises through one of the entity’s perceptual modalities, and triggers an Emotion, which may combine with a drive. An emotion is the primary motivation that pushes the entity from a state of Rest to a state of Restlessness. The system will support eight fundamental emotions that vary in intensity.

For example, the mouse can feel Apprehension, which represents a low intensity response to something that is perceived as alien and therefore potentially threatening within its environment, such as my hand, perhaps, rooting around in the tank. That feeling can heighten to Fear, if I should clumsily reach to grab it, and it might boil over into Terror if I should seize hold of it. A stereotypic behavior maps to each emotion. For example, only a terrified mouse will actually bite.

There is a Motor component that bounds and determines the forms of behavior the entity can make. A mouse, for example, cannot inquire as to my intention in rooting around its tank, nor can it break through the glass siding and escape. It is physically constrained in its range of behaviors.

And this leads us to the fourth and most interesting component (at least in terms of the simulation) of EEK! That is an entity’s Cognitive component, which consists of a Genetic disposition and a personal History. For example, it may be disposed to feel Apprehension in situations that another entity of its kind might find of Interest. Based on its history, its disposition may strengthen or weaken or be transformed. It is this Cognitive component that allows an individual entity to modify its stereotypic behaviors over time.

Each capitalized word in the above descriptions, such as Emotion, Apprehension, Fear, and so on, represents a candidate class abstraction in my application. I’ll need to put it all together, of course. But I’ll look at that in later columns when I begin to implement the mouse.

The next question is how do I drive the simulation? Unlike the rodent entities within EEK!, I can’t depend on emotion. Rather, I must depend on the main logic of the simulation as listed in Figure 2.

Figure 2 Main Logic of Simulation (Close Figure 2)
void EEK::run()
{
    Init(); // initialize our environment
    while ( Time < EndTime )
    {
        Tick();    // advance time, trigger events
        Update();  // update modalities
        Poll();    // SEs perceive() and behave()
        Display(); // communicate what has happened
    }
    Terminate(); // capture final state
}
Figure 2 Main Logic of Simulation (Close Figure 2)
void EEK::run()
{
    Init(); // initialize our environment
    while ( Time < EndTime )
    {
        Tick();    // advance time, trigger events
        Update();  // update modalities
        Poll();    // SEs perceive() and behave()
        Display(); // communicate what has happened
    }
    Terminate(); // capture final state
}

In order to support multiple simulation implementations, I’ll abstract the simulation as an interface, and instantiate EEK as an instance. This is listed in Figure 3.

Figure 3 EEK as an Instance (Close Figure 3)
public interface class ISimulation
{
    void Run();

    void Init();
    void Tick();
    void Update();
    void Poll();
    void Display();
    void Terminate();

    property int Time    { int get(); }
    property int EndTime { int get(); }
}

public ref class EEK : ISimulation { ... };
Figure 3 EEK as an Instance (Close Figure 3)
public interface class ISimulation
{
    void Run();

    void Init();
    void Tick();
    void Update();
    void Poll();
    void Display();
    void Terminate();

    property int Time    { int get(); }
    property int EndTime { int get(); }
}

public ref class EEK : ISimulation { ... };

Initialization will be supported through a set of XML configuration files, and I’ll parse that using the XML Document Object Model (DOM) support within the Base Class Library (BCL) of the .NET Framework. I’ll persist the state of the simulation within a database, either Access® or SQL Server™, using the BCL Data support. I’ll use Windows Presentation Foundation for the display component. Eventually, I’ll want to provide some sort of Web presence, but I’ll defer that for the time being. I’ll encapsulate each of these services in an associated Manager class.

Of course, in programming, talking about a design can only get you so far, no matter how much thought you’ve given it. It doesn’t become truly interesting until you begin its implementation. I’ll initiate that in the next column.

No vectorized mice were harmed during the writing of this column.

Figure mouse

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. Later, Stan worked in feature animation both at Disney and DreamWorks and served as a Software Technical Director on Fantasia 2000. He has since served as Distinguished Consultant with JPL, and an Architect with the Visual C++ team at Microsoft. He is currently a senior software engineer at Emergent Game Technologies.


.NET Matters: Handling Messages in Console Apps

.NET Matters
Handling Messages in Console Apps
Stephen Toub

Code download available at: NetMatters2007_06.exe (153KB)

Q I’m using a Win32® API through P/Invoke. The API supports asynchronous operations and provides a callback mechanism that notifies me when the operation has completed. However, unlike other APIs I’ve dealt with that provide similar functionality, this API doesn’t accept a function pointer or delegate to call back through. Instead, it accepts a window handle to which it sends a specific message when the operation has completed. If I were using this from an application with a GUI, this wouldn’t be a problem. But I’m using it in a console application, and therefore I don’t have a window or a message loop to receive or process the notification message. What can I do? Am I stuck?

A Many developers who use the Microsoft® .NET Framework are under the impression that application type is tied to the libraries that can be used in that application, such that, for example, a console application can’t make use of the Windows® Forms classes. Those concepts are separate, however. When you create a new console application in Visual Studio®, it sets the output type of the application to be "Console Application" (the equivalent of the /t:exe command-line parameter to csc.exe). When you create a new Windows Forms application in Visual Studio, it sets the output type of the application to be "Windows Application" (the equivalent of /t:winexe), and also adds a reference to the System.Windows.Forms.dll. What many folks don’t realize, though, is that you can add a reference to System.Windows.Forms.dll in a console application and, conversely, you can change a Windows Forms project to compile to a "Console Application." The application type really just dictates whether or not a default console window will be launched for the process.

All of this is to say that in your console application, you can use the classes from System.Windows.Forms to help achieve your goal. I’ll create a class that relies on Windows Forms classes to make this a piece of cake, and I’ll use the mciSendString function exported from winmm.dll as my test case for this class, as it provides functionality similar to the API you’re having difficulty using. This same solution, though, will enable your console application to handle any kind of Windows message. For example, to take advantage of some of the new power management functionality available in Windows Vista™, you might want to handle the WM_POWERBROADCAST message. The class I provide in this column will enable you to do that.

To begin, I’ll start with the design I’m working towards. The Microsoft.Win32.SystemEvents class exported from System.dll provides a slew of static events that are used to watch for a variety of system happenings. Some of these are raised due to broadcast messages. For example, the DisplaySettingsChanging and DisplaySettingsChanged events are raised in response to a WM_DISPLAYCHANGE message.

When SystemEvents is initialized in a user-interactive application, if the thread of the application causing the SystemEvents initialization is marked as STA (single-threaded apartment, the default in a Windows Forms application), SystemEvents expects that this thread will have a message loop on which SystemEvents can piggyback. (Thus, if you were to erroneously mark a console application as STA and didn’t manually pump messages, none of the message-based events on SystemEvents would be raised.) If, however, the initializing thread is MTA (multithreaded apartment, the default in a console application in C#), SystemEvents will spawn its own thread. That thread will execute a message loop. In either case, SystemEvents creates a hidden window tied to the thread running the message loop. This window’s window procedure is responsible for processing relevant messages and raising the associated events.

While my implementation won’t be as sophisticated as SystemEvents, a basic design similar to that of SystemEvents will get us off the ground and running. My MessageEvents class exposes three public and static members:

public static class MessageEvents
{
    public static void WatchMessage(int message);
    public static event EventHandler<MessageReceivedEventArgs>
        MessageReceived;
    public static IntPtr WindowHandle;
    ...
}

The WatchMessage method is used to inform the MessageEvents class of messages that I, as the consumer, actually care about. (In contrast, SystemEvents has these hardwired into its implementation.) When one of these messages is received, the MessageReceived event is raised; this event carries with it a MessageReceivedEventArgs instance, which in turn holds onto the relevant System.Windows.Forms.Message struct representing the received message. Finally, for scenarios where an API will be targeting one specific window to which it will send messages, the MesssageEvents class exposes the handle for the underlying window used by MessageEvents to receive and process events. (Typically those APIs that use windows messages for notification will also work with HWND_BROADCAST, or 0xFFFF, which if used would also cause the notification message to reach our underlying window.)

That underlying window is created by MessageEvents. Unlike SystemEvents which intelligently decides whether to create an additional window, saving resources if doing so is unnecessary, I’ve taken the simpler approach of always creating a new thread and window. If you discover through measurement that this hurts your application’s performance, feel free to augment the type appropriately with the necessary additional logic.

MessageWindow, shown in Figure 1, is a private nested type within MessageEvents. It derives from Form, thereby also inheriting the underlying NativeWindow functionality available to all Control instances (I could rewrite MessageWindow to be lighterweight and go directly against NativeWindow, but again, I’ll leave that as an exercise for you if you deem it appropriate). MessageWindow exposes one public member, RegisterEventForMessage, which simply stores the supplied messageID into a message set, represented by a Dictionary<int, bool>. The class also overrides the WndProc method, which is the method executed in response to the window receiving a message. The override checks to see whether the received message is in the message set, and thus whether it is one previously registered for an event to be raised. If it is, WndProc raises the MessageReceived event. Regardless, all messages are passed on to the base class’s WndProc implementation.

Figure 1 MessageWindow Class (Close Figure 1)
public static class MessageEvents
{
    private static SynchronizationContext _context;
    public static event
        EventHandler<MessageReceivedEventArgs> MessageReceived;

    ...

    private class MessageWindow : Form
    {
        private ReaderWriterLock _lock = new ReaderWriterLock();
        private Dictionary<int, bool> _messageSet =
            new Dictionary<int, bool>();

        public void RegisterEventForMessage(int messageID)
        {
            _lock.AcquireWriterLock(Timeout.Infinite);
            _messageSet[messageID] = true;
            _lock.ReleaseWriterLock();
        }

        protected override void WndProc(ref Message m)
        {
            _lock.AcquireReaderLock(Timeout.Infinite);
            bool handleMessage = _messageSet.ContainsKey(m.Msg);
            _lock.ReleaseReaderLock();

            if (handleMessage)
            {
                MessageEvents._context.Post(delegate(object state)
                {
                        EventHandler<MessageReceivedEventArgs>
                            handler = MessageEvents.MessageReceived;
                        if (handler != null) handler(null,
                            new MessageReceivedEventArgs(
                                      (Message)state));
                }, m);
            }
            base.WndProc(ref m);
        }
    }
}

There are several interesting things to note about how this event is raised. First, MessageReceived is an event on MessageEvents, not MessageWindow. Typically, one class isn’t able to raise an event on another class through direct access. This is because of a trick played by the C# compiler. When I declared the MessageReceived event, I was really doing two things: declaring a public event and declaring a private delegate. The public event provides add and remove accessors, which is why any class is able to register and unregister handlers with the event. The private delegate is what is actually invoked when I "raise the event," even though in the C# source code it looks like I’m directly invoking the event (for more information, see msdn.microsoft.com/msdnmag/issues/06/11/NETMatters). Since the delegate is private, another class shouldn’t have access to it. However, because MessageWindow is a nested class within MessageEvents, MessageWindow does have access to the private implementation details of MessageEvents. In this case, that allows MessageWindow access to the private MessageReceived delegate, allowing it to be invoked directly by MessageWindow.

Another thing to notice is that the event isn’t raised on the same thread that’s executing WndProc. Instead, WndProc uses an instance of SynchronizationContext that MessageEvents grabs when it’s initialized (more on this shortly). The SynchronizationContext is used to raise the event through an asynchronous Post (rather than a synchronous Send) in the synchronization context of the application, whatever that may be. In a console application, SynchronizationContext.Post simply executes the delegate on a ThreadPool thread. In a Windows Forms application, WindowsFormsSynchronizationContext.Post executes the delegate by marshaling its invocation back to the UI thread. For more information on SynchronizationContext, see msdn.microsoft.com/msdnmag/issues/06/06/NETMatters.

The MessageEvents class only contains static members, and in fact the class itself is declared as a static class (the compiler will complain if an attempt is made to add a non-static member). Internally, it stores a static reference to a single MessageWindow that is created when MessageEvents is initialized:

private static MessageWindow _window;
private static IntPtr _windowHandle;

The implementation of MessageEvents’ two public methods is very straightforward, as most of the work is factored out into a separate initialization method. The WatchMessage method calls EnsureInitialized and then passes the message ID along to the MessageWindow’s RegisterEventForMessage method. The WindowHandle property also calls EnsureInitialized and simply returns the window handle created during initialization:

public static void WatchMessage(int message) {
    EnsureInitialized();
    _window.RegisterEventForMessage(message);
}

public static IntPtr WindowHandle {
    get {
        EnsureInitialized();
        return _windowHandle;
    }
}

That leaves us with how to implement EnsureInitialized. EnsureInitialized is responsible for getting the MessageEvents class up and running so that it can watch for and respond to all relevant messages. Thus, it needs to spawn off a new thread, create the broadcast window that will receive messages, and start a message loop running.

As shown in Figure 2, EnsureInitialized only runs the initialization logic once; if it sees that the MessageWindow has already been created, it bails. If initialization needs to be performed, it starts by grabbing the SynchronizationContext associated with the current thread. It could do this using SynchronizationContext.Current; however, SynchronizationContext.Current will return null if no context has been previously configured. Instead, I use AsyncOperationManager.SynchronizationContext, a simple wrapper around SynchronizationContext.Current that first sets up a default SynchronizationContext if none currently exists; that way, when it proceeds to return the value of SynchronizationContext.Current, it’ll never be returning a null value.

Figure 2 Starting the Background Message Loop (Close Figure 2)
public static class MessageEvents
{
    private static object _lock = new object();
    private static MessageWindow _window;
    private static IntPtr _windowHandle;
    private static SynchronizationContext _context;

    ...

    private static void EnsureInitialized()
    {
        lock (_lock)
        {
            if (_window == null)
                   {
                   _context = AsyncOperationManager.
                         SynchronizationContext;
                   using (ManualResetEvent mre =
                         new ManualResetEvent(false))
                {
                    Thread t = new Thread((ThreadStart)delegate
                    {
                        _window = new MessageWindow();
                        _windowHandle = _window.Handle;
                        mre.Set();
                        Application.Run();
                    });
                    t.Name = “MessageEvents”;
                    t.IsBackground = true;
                    t.Start();

                    mre.WaitOne();
                }
            }
        }
    }
}

With the context in hand, EnsureInitialized creates a new thread. This thread creates the MessageWindow and stores it, also storing the window’s handle into a separate static member, which, as shown previously, is used as the return value from the WindowHandle property. At this point, the window is up and running and I need to start a message loop running. For that purpose, I could write my own message loop, but it’s much easier to take advantage of one that already exists in the .NET Framework. The Application.Run method begins running a standard application message loop on the current thread, so I need only call this method to get that process underway.

That’s it for my MessageEvents class. To see it in action, I’ll use the mciSendString function from winmm.dll mentioned earlier. The mciSendString function is part of the Media Control Interface (MCI) in Windows and can be used to interact with a wide variety of multimedia devices:

[DllImport(“winmm.dll”, EntryPoint = “mciSendStringA”,
           CharSet = CharSet.Ansi)]
private static extern int MciSendString(
    string lpszCommand, StringBuilder lpszReturnString,
    int cchReturn, IntPtr hwndCallback);

To play a WAV file using mciSendString, a typical sequence of commands might look as follows:

open “C:\Windows\Media\chimes.wav” type waveaudio alias 12345
play 12345 wait
close 12345

The first command opens the specified device and assigns it an alias (12345, in this example) by which it will be referred in the future in calls to other commands. The second command tells the system to start playing the device and to not return until the command has completed execution. And the final command tells the system to close the device. Notice, however, that the P/Invoke signature I created for MciSendString has an hwndCallback IntPtr parameter. This is used in concert with a notify flag that can be added to various commands, including the play command:

play 12345 notify

Since the wait flag has not been specified, this command will start playback and immediately return. However, as the notify flag is specified, upon asynchronous completion of the command, an MM_MCINOTIFY message is sent to the HWND specified through that hwndCallback procedure:

MciSendString(“play 12345 notify”, null,
              0, MessageEvents.WindowHandle);

Notice how I’ve provided the MessageEvents.WindowHandle as the target HWND. Before executing this command, I can configure the MessageEvents class to receive those callback messages:

MessageEvents.WatchMessage(MM_MCINOTIFY);
MessageEvents.MessageReceived += delegate(
    object sender, MessageReceivedEventArgs e)
{
    Console.WriteLine(“Message received: “ + e.Message.Msg);
};

With this code in place, when a notify command completes, information about the resulting message will be written to the console. And of course, while I don’t necessarily need the MessageEvents class to do this in a Windows Forms application, it should work equally well in one, just as it does in a console application that doesn’t have its own message loop.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


{ End Bracket }: Proud to Be a Developer

{ End Bracket }
Proud to Be a Developer
Adam Barr

When I was a senior in college, I remember a conversation with a fellow computer science major who was going to work for an investment bank. He explained that he would be involved in designing software, but he would not busy himself with the mundane task of actually writing it. That would be handed off to a coder—evidently a lower species. In other words, me. Coming from a university environment in which the ability to hack code was the ultimate status, I felt vaguely insulted.

However, he had a point. Back then, I was just a coder (not that he was any better). Today, as in the title of Mike Gunderloy’s book Coder to Developer, I have moved beyond being a mere coder and now consider myself a developer. I know more than just how to write code that compiles; I can produce software that is fast, reliable, well-tested, secure, maintainable, globalizable, and on down the list of attributes of high-quality code. The software industry in general is maturing from an army of coders to an army of developers.

Now, if you ask developers about their next career step, they may say they want to be an architect. The word conjures up visions of titanium and frosted glass, and relegates the mere developer, by comparison, to the role of construction worker. So do I feel like taking the theoretical next step and styling myself as an architect? My answer is no.

No insult to architects is intended: software development does need people to lay out the interaction between components of a system, and to keep the big picture in mind. Nonetheless, I am proud to be a developer, and I hope every other developer can feel the same pride.

What if you envy the freedom of design available to physical architects like Frank Gehry or Rem Koolhaas? My response would be that the discipline of software engineering is too immature to climb that conceptual ladder. Architects can design buildings like the Bilbao Guggenheim Museum and the Seattle Public Library because they are supported by centuries of knowledge about civil engineering. The software industry can’t afford the luxury of having everyone operate at this level; most of us are still trying to build a one-story house that doesn’t collapse when you slam the front door.

When Microsoft examines the cause of its bugs, it identifies a significant number that aren’t design bugs—the kind that would be found when an architect reviews a design document—but neither are they coding bugs—when the source code doesn’t do what the programmer intended. This intermediate class occurs when the source code does what the programmer intended, but there is a localized error in the intent: issues like passing the wrong flags to a method, or misunderstanding the meaning of a configuration parameter. This isn’t the province of architects. It’s something that we developers have to do correctly on our own.

Fred Brooks alluded to this in his famous "No Silver Bullet" essay where he wrote "The essence of a software entity is a construct of interlocking concepts: data sets, relationships among data items, algorithms, and invocations of functions" and then went on to say "I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation." He is not talking about what architects do; he is talking about what developers do. He is saying, in effect, that being a coder may not be too hard, but being a developer certainly is. We should recognize the merit of what we do.

I know this is not the glory work of programming. It can be a thrill to perfectly architect multiple components in a way that neatly encapsulates variation and allows for future extension. Programmers like precision, and there is beauty in having all of the pieces fit together just so. But there is a large part of producing software that involves working like a skilled craftsman: performing code reviews, writing unit tests, cleaning up comments. Customers may not directly recognize how our hard work is ensuring their privacy or guarding their system against attack, but we know the value we provide. And there are areas like performance optimization and power management that are true black arts: not high-level architecture but close-in work, just us applying our skill and experience to the problem at hand.

I’m proud to say I’m a developer, not an architect. Maybe someday we’ll solve all the engineering problems in software, and then we can all become architects. In the meantime, we have some well-crafted software we need to finish.

Adam Barr has worked at Microsoft for 13 years. He has been a developer, a program manager, and is currently a knowledge engineer in the Engineering Excellence team, working as an internal consultant and instructor to support the development engineering process.