 MSDN Magazine May 2007

MSDN Magazine May 2007

JavaScript: Create Advanced Web Applications With Object-Oriented Techniques

Ray Djajadinata Got Game?: Unleash Your Imagination With XNA Game Studio Express

Charles Cox and Michael Klucher WPF: Customizing Controls For Windows Presentation Foundation

Shawn Wildermuth Migration: Convert A Java Web Application To ASP.NET Using JLCA

Brian Jimerson BizTalk Server: 8 Tips And Tricks For Better BizTalk Programming

Marty Wasznicky and Scott Zimmerman Collaborate: Help Teams Work Together With Web Services And Groove 2007

John C. Hancock Editor's Note: Daylight Savings

Joshua Trupin Toolbox: Code Exploration, SQL Queries, and More

Scott Mitchell CLR Inside Out: 9 Reusable Parallel Data Structures and Algorithms

Joe Duffy Office Space: Features for SharePoint

Ted Pattison Cutting Edge: Subclassing and Overriding ASP.NET Pages—Part II

Dino Esposito Test Run: Lightweight Testing with Windows PowerShell

Dr. James McCaffrey Security Briefs: Events in Windows Vista

Keith Brown Foundations: WCF Transaction Propagation

Juval Lowy { End Bracket }: Exercising Agility

James Waletzky


JavaScript: Create Advanced Web Applications With Object-Oriented Techniques

JavaScript

Create Advanced Web Applications With Object-Oriented Techniques

Ray Djajadinata
This article discusses:
  • JavaScript as a prototype-based language
  • Object-oriented programming with JavaScript
  • Coding tricks with JavaScript
  • The future of JavaScript
This article uses the following technologies:
JavaScript
Contents
JavaScript Objects Are Dictionaries
JavaScript Functions Are First Class
Constructor Functions but No Classes
Prototypes
Static Properties and Methods
Closures
Simulating Private Properties
Inheriting from Classes
Simulating Namespaces
Should You Code JavaScript This Way?
Putting It into Perspective

R
ecently I interviewed a software developer with five years experience in developing Web applications. She’d been doing JavaScript for four and a half years, she rated her JavaScript skill as very good, and—as I found out soon after—she actually knew very little about JavaScript. I didn’t really blame her for that, though. JavaScript is funny that way. It’s the language a lot of people (including myself, until recently!) assume they’re good at, just because they know C/C++/C# or they have some prior programming experience.

In a way, that assumption is not entirely groundless. It is easy to do simple things with JavaScript. The barrier to entry is very low; the language is forgiving and doesn’t require you to know a lot of things before you can start coding in it. Even a non-programmer can probably pick it up and write some useful scripts for a homepage in a matter of hours.

Indeed, until recently, I’d always been able to get by with whatever little JavaScript I knew, armed only with the MSDN® DHTML reference and my C++/C# experience. It was only when I started working on real-world AJAX applications that I realized how inadequate my JavaScript actually was. The complexity and interactivity of this new generation of Web applications requires a totally different approach to writing JavaScript code. These are serious JavaScript applications! The way we’ve been writing our throwaway scripts simply doesn’t cut it anymore.

Object-oriented programming (OOP) is one popular approach that’s used in many JavaScript libraries to make a codebase more manageable and maintainable. JavaScript supports OOP, but it does so in a very different manner from the way popular Microsoft® .NET Framework compliant languages like C++, C#, or Visual Basic® do it, so developers who have been working extensively with those languages may find doing OOP in JavaScript strange and counter-intuitive at first. I wrote this article to discuss in depth how the JavaScript language really supports object-oriented programming and how you can use this support to do object-oriented development effectively in JavaScript. Let’s start by talking about (what else?) objects.

JavaScript Objects Are Dictionaries

In C++ or C#, when we’re talking about objects, we’re referring to instances of classes or structs. Objects have different properties and methods, depending on which templates (that is, classes) they are instantiated from. That’s not the case with JavaScript objects. In JavaScript, objects are just collections of name/value pairs—think of a JavaScript object as a dictionary with string keys. We can get and set the properties of an object using either the familiar "." (dot) operator, or the "[]" operator, which is typically used when dealing with a dictionary. The following snippet

var userObject = new Object();
userObject.lastLoginTime = new Date();
alert(userObject.lastLoginTime);
does exactly the same thing as this:

var userObject = {}; // equivalent to new Object()
userObject[“lastLoginTime”] = new Date();
alert(userObject[“lastLoginTime”]);
We can also define the lastLoginTime property directly within userObject’s definition like this:

var userObject = { “lastLoginTime”: new Date() };
alert(userObject.lastLoginTime);

Note how similar it is to the C# 3.0 object initializers. Also, those of you familiar with Python will recognize that the way we instantiate userObject in the second and third snippets is exactly how we’d specify a dictionary in Python. The only difference is that a JavaScript object/dictionary only accepts string keys, rather than hashable objects like a Python dictionary would.

These examples also show how much more malleable JavaScript objects are than C++ or C# objects. Property lastLoginTime doesn’t have to be declared beforehand—if userObject doesn’t have a property by that name, it will simply be added to userObject. This isn’t surprising if you remember that a JavaScript object is a dictionary—after all, we add new keys (and their respective values) to dictionaries all the time.

So, there we have object properties. How about object methods? Again, JavaScript is different from C++/C#. To understand object methods, I first need to take a closer look at JavaScript functions.

JavaScript Functions Are First Class

In many programming languages, functions and objects are usually considered two different things. In JavaScript, this distinction is blurred—a JavaScript function is really an object with executable code associated with it. Consider an ordinary function like this:

function func(x) {
    alert(x);
}
func(“blah”);
This is how we usually define a function in JavaScript. But you can also define the same function as follows, where you create an anonymous function object, and assign it to variable func

var func = function(x) {
    alert(x);
};
func(“blah2”);
or even like this, using the Function constructor:

var func = new Function(“x”, “alert(x);”);
func(“blah3”);

This shows that a function is really just an object that supports a function call operation. That last way of defining a function using the Function constructor is not commonly used, but it opens up interesting possibilities because, as you may notice, the body of the function is just a String parameter to the Function constructor. That means you can construct arbitrary functions at run time.

To demonstrate further that a function is an object, you can set or add properties to a function, just like you would to any other JavaScript objects:

function sayHi(x) {
    alert(“Hi, “ + x + “!”);
}
sayHi.text = “Hello World!”;
sayHi[“text2”] = “Hello World... again.”;

alert(sayHi[“text”]); // displays “Hello World!”
alert(sayHi.text2); // displays “Hello World... again.”

As objects, functions can also be assigned to variables, passed as arguments to other functions, returned as the values of other functions, stored as properties of objects or elements of arrays, and so on. Figure 1 provides an example of this.

Figure 1 Functions Are First-Class in JavaScript (Close Figure 1)
// assign an anonymous function to a variable
var greet = function(x) {
    alert(“Hello, “ + x);
};
greet(“MSDN readers”);

// passing a function as an argument to another
function square(x) {
    return x * x;
}
function operateOn(num, func) {
    return func(num);
}
// displays 256
alert(operateOn(16, square));

// functions as return values
function makeIncrementer() {
    return function(x) { return x + 1; };
}
var inc = makeIncrementer();
// displays 8
alert(inc(7));

// functions stored as array elements
var arr = [];
arr[0] = function(x) { return x * x; };
arr[1] = arr[0](2);
arr[2] = arr[0](arr[1]);
arr[3] = arr[0](arr[2]);
// displays 256
alert(arr[3]);

// functions as object properties
var obj = { “toString” : function() { return “This is an object.”; } };
// calls obj.toString()
alert(obj);

With that in mind, adding methods to an object is as easy as choosing a name and assigning a function to that name. So I define three methods in the object by assigning anonymous functions to the respective method names:

var myDog = {
    “name” : “Spot”,
    “bark” : function() { alert(“Woof!”); },
    “displayFullName” : function() {
        alert(this.name + “ The Alpha Dog”);
    },
    “chaseMrPostman” : function() {
        // implementation beyond the scope of this article
    }
};
myDog.displayFullName();
myDog.bark(); // Woof!

The use of the "this" keyword inside the function displayFullName should be familiar to the C++/C# developers among us—it refers to the object through which the method is called ( developers who use Visual Basic should find it familiar, too—it’s called "Me" in Visual Basic). So in the example above, the value of "this" in the displayFullName is the myDog object. The value of "this" is not static, though. Called through a different object, the value of "this" will also change to point to that object as Figure 2 demonstrates.

Figure 2 “this” Changes as the Object Changes (Close Figure 2)
function displayQuote() {
    // the value of “this” will change; depends on
    // which object it is called through
    alert(this.memorableQuote);
}

var williamShakespeare = {
    “memorableQuote”: “It is a wise father that knows his own child.”,
    “sayIt” : displayQuote
};

var markTwain = {
    “memorableQuote”: “Golf is a good walk spoiled.”,
    “sayIt” : displayQuote
};

var oscarWilde = {
    “memorableQuote”: “True friends stab you in the front.”
    // we can call the function displayQuote
    // as a method of oscarWilde without assigning it
    // as oscarWilde’s method.
    //”sayIt” : displayQuote
};

williamShakespeare.sayIt(); // true, true
markTwain.sayIt(); // he didn’t know where to play golf

// watch this, each function has a method call()
// that allows the function to be called as a
// method of the object passed to call() as an
// argument.
// this line below is equivalent to assigning
// displayQuote to sayIt, and calling oscarWilde.sayIt().
displayQuote.call(oscarWilde); // ouch!

The last line in Figure 2 shows an alternative way of calling a function as a method of an object. Remember, a function in JavaScript is an object. Every function object has a method named call, which calls the function as a method of the first argument. That is, whichever object we pass into call as its first argument will become the value of "this" in the function invocation. This will be a useful technique for calling the base class constructor, as we’ll see later.

One thing to remember is never to call functions that contain "this" without an owning object. If you do, you will be trampling over the global namespace, because in that call, "this" will refer to the Global object, and that can really wreak havoc in your application. For example, below is a script that changes the behavior of JavaScript’s global function isNaN. Definitely not recommended!

alert(“NaN is NaN: “ + isNaN(NaN));

function x() {
    this.isNaN = function() {
        return “not anymore!”;
    };
}
// alert!!! trampling the Global object!!!
x();

alert(“NaN is NaN: “ + isNaN(NaN));

So we’ve seen ways to create an object, complete with its properties and methods. But if you notice all the snippets above, the properties and methods are hardcoded within the object definition itself. What if you need more control over the object creation? For example, you may need to calculate the values of the object’s properties based on some parameters. Or you may need to initialize the object’s properties to the values that you’ll only have at run time. Or you may need to create more than one instance of the object, which is a very common requirement.

In C#, we use classes to instantiate object instances. But JavaScript is different since it doesn’t have classes. Instead, as you’ll see in the next section, you take advantage of the fact that functions act as constructors when used together with the "new" operator.

Constructor Functions but No Classes

The strangest thing about JavaScript OOP is that, as noted, JavaScript doesn’t have classes like C# or C++ does. In C#, when you do something like this:

Dog spot = new Dog();
you get back an object, which is an instance of the class Dog. But in JavaScript there’s no class to begin with. This closest you can get to a class is by defining a constructor function like this:

function DogConstructor(name) {
    this.name = name;
    this.respondTo = function(name) {
        if(this.name == name) {
            alert(“Woof”);
        }
    };
}

var spot = new DogConstructor(“Spot”);
spot.respondTo(“Rover”); // nope
spot.respondTo(“Spot”); // yeah!
OK, so what’s happening here? Ignore the DogConstructor function definition for a moment and examine this line:

var spot = new DogConstructor(“Spot”);

What the "new" operator does is simple. First, it creates a new empty object. Then, the function call that immediately follows is executed, with the new empty object set as the value of "this" within that function. In other words, the line above with the "new" operator can be thought of as similar to the two lines below:

// create an empty object
var spot = {};
// call the function as a method of the empty object
DogConstructor.call(spot, “Spot”);
As you can see in the body of DogConstructor, invoking this function initializes the object to which the keyword “this” refers during that invocation. This way, you have a way of creating a template for objects! Whenever you need to create a similar object, you call “new” together with the constructor function, and you get back a fully initialized object as a result. Sounds very similar to a class, doesn’t it? In fact, usually in JavaScript the name of the constructor function is the name of the class you’re simulating, so in the example above you can just name the constructor function Dog:

// Think of this as class Dog
function Dog(name) {
    // instance variable
    this.name = name;
    // instance method? Hmmm...
    this.respondTo = function(name) {
        if(this.name == name) {
            alert(“Woof”);
        }
    };
}

var spot = new Dog(“Spot”);

In the Dog definition above, I defined an instance variable called name. Every object that is created using Dog as its constructor function will have its own copy of the instance variable name (which, as noted earlier, is just an entry into the object’s dictionary). This is expected; after all, each object does need its own copies of instance variables to carry its state. But if you look at the next line, every instance of Dog also has its own copy of the respondTo method, which is a waste; you only need one instance of respondTo to be shared among Dog instances! You can work around the problem by taking the definition of respondTo outside Dog, like this:

function respondTo() {
    // respondTo definition
}

function Dog(name) {
    this.name = name;
    // attached this function as a method of the object
    this.respondTo = respondTo;
}

This way, all instances of Dog (that is, all instances created with the constructor function Dog) can share just one instance of the method respondTo. But as the number of methods grow, this becomes harder and harder to maintain. You end up with a lot of global functions in your codebase, and things only get worse as you have more and more "classes," especially if their methods have similar names. There’s a better way to achieve this using the prototype objects, which are the topic of the next section.

Prototypes

The prototype object is a central concept in object-oriented programming with JavaScript. The name comes from the idea that in JavaScript, an object is created as a copy of an existing example (that is, a prototype) object. Any properties and methods of this prototype object will appear as properties and methods of the objects created from that prototype’s constructor. You can say that these objects inherit their properties and methods from their prototype. When you create a new Dog object like this

var buddy = new Dog(“Buddy“);
the object referenced by buddy inherits properties and methods from its prototype, although it’s probably not obvious from just that one line where the prototype comes from. The prototype of the object buddy comes from a property of the constructor function (which, in this case, is the function Dog).

In JavaScript, every function has a property named "prototype" that refers to a prototype object. This prototype object in turn has a property named "constructor," which refers back to the function itself. It’s sort of a circular reference; Figure 3 illustrates this cyclic relationship better.

Figure 3 Every Function’s Prototype Has a Constructor Property
Figure 3 Every Function’s Prototype Has a Constructor Property

Now, when a function (in the example above, Dog) is used to create an object with the "new" operator, the resulting object will inherit the properties of Dog.prototype. In Figure 3, you can see that the Dog.prototype object has a constructor property that points back to the Dog function. Consequently, every Dog object (that inherits from Dog.prototype) will also appear to have a constructor property that points back to the Dog function. The code in Figure 4 confirms this. This relationship between constructor function, prototype object, and the object created with them is depicted in Figure 5.

Figure 4 Objects Appear to Have Their Prototype’s Properties (Close Figure 4)
var spot = new Dog(“Spot”);

// Dog.prototype is the prototype of spot
alert(Dog.prototype.isPrototypeOf(spot));

// spot inherits the constructor property
// from Dog.prototype
alert(spot.constructor == Dog.prototype.constructor);
alert(spot.constructor == Dog);

// But constructor property doesn’t belong
// to spot. The line below displays “false”
alert(spot.hasOwnProperty(“constructor”));

// The constructor property belongs to Dog.prototype
// The line below displays “true”
alert(Dog.prototype.hasOwnProperty(“constructor”));
Figure 5 Instances Inherit from Their Prototype
Figure 5 Instances Inherit from Their Prototype

Some of you may have noticed the calls to hasOwnProperty and isPrototypeOf method in Figure 4. Where do these methods come from? They don’t come from Dog.prototype. In fact, there are other methods like toString, toLocaleString, and valueOf that we can call on Dog.prototype and instances of Dog, but which don’t come from Dog.prototype at all. It turns out that just like the .NET Framework has System.Object, which serves as the ultimate base class for all classes, JavaScript has Object.prototype, which is the ultimate base prototype for all prototypes. (The prototype of Object.prototype is null.)

In this example, remember that Dog.prototype is an object. It is created with a call to the Object constructor function, although it is not visible:

Dog.prototype = new Object();

So just like instances of Dog inherit from Dog.prototype, Dog.prototype inherits from Object.prototype. This makes all instances of Dog inherit Object.prototype’s methods and properties as well.

Every JavaScript object inherits a chain of prototypes, all of which terminate with Object.prototype. Note that this inheritance you’ve seen so far is inheritance between live objects. It is different from your usual notion of inheritance, which happens between classes when they are declared. Consequently, JavaScript inheritance is much more dynamic. It is done using a simple algorithm, as follows: when you try to access a property/method of an object, JavaScript checks if that property/method is defined in that object. If not, then the object’s prototype will be checked. If not, then that object’s prototype’s prototype will be checked, and so on, all the way to Object.prototype. Figure 6 illustrates this resolution process.

Figure 6 Resolving toString() Method in the Prototype Chain
Figure 6 Resolving toString() Method in the Prototype Chain(Click the image for a smaller view)
Figure 6 Resolving toString() Method in the Prototype Chain
Figure 6 Resolving toString() Method in the Prototype Chain (Click the image for a larger view)

The way JavaScript resolves properties access and method calls dynamically has some consequences:

  • Changes made to a prototype object are immediately visible to the objects that inherit from it, even after these objects are created.
  • If you define a property/method X in an object, a property/method of the same name will be hidden in that object’s prototype. For instance, you can override Object.prototype’s toString method by defining a toString method in Dog.prototype.
  • Changes only go in one direction, from prototype to its derived objects, but not vice versa.

Figure 7 illustrates these consequences. Figure 7 also shows how to solve the problem of unnecessary method instances as encountered earlier. Instead of having a separate instance of a function object for every object, you can make the objects share the method by putting it inside the prototype. In this example, the getBreed method is shared by rover and spot—until you override the toString method in spot, anyway. After that, spot has its own version of the getBreed method, but the rover object and subsequent objects created with new GreatDane will still share that one instance of the getBreed method defined in the GreatDane.prototype object.

Figure 7 Inheriting from a Prototype (Close Figure 7)
function GreatDane() { }

var rover = new GreatDane();
var spot = new GreatDane();

GreatDane.prototype.getBreed = function() {
    return “Great Dane”;
};

// Works, even though at this point
// rover and spot are already created.
alert(rover.getBreed());

// this hides getBreed() in GreatDane.prototype
spot.getBreed = function() {
    return “Little Great Dane”;
};
alert(spot.getBreed());

// but of course, the change to getBreed
// doesn’t propagate back to GreatDane.prototype
// and other objects inheriting from it,
// it only happens in the spot object
alert(rover.getBreed());

Static Properties and Methods

Sometimes you need properties or methods that are tied to classes instead of instances—that is, static properties and methods. JavaScript makes this easy, since functions are objects whose properties and methods can be set as desired. Since a constructor function represents a class in JavaScript, you can add static methods and properties to a class simply by setting them in the constructor function like this:

    function DateTime() { }

    // set static method now()
    DateTime.now = function() {
        return new Date();
    };

    alert(DateTime.now());

The syntax for calling the static methods in JavaScript is virtually identical to how you’d do it in C#. This shouldn’t come as a surprise since the name of the constructor function is effectively the name of the class. So you have classes, and you have public properties/methods, and static properties/methods. What else do you need? Private members, of course. But JavaScript doesn’t have native support for private members (nor for protected, for that matter). All properties and methods of an object are accessible to anyone. There is a way to have private members in your class, but to do so you first need to understand closures.

Closures

I didn’t learn JavaScript of my own volition. I had to pick it up quickly because I realized that I was ill-prepared to work on a real-world AJAX application without it. At first, I felt like I had gone down a few levels in the programmer hierarchy. (JavaScript! What would my C++ friends say?) But once I got over my initial resistance, I realized that JavaScript was actually a powerful, expressive, and compact language. It even boasts features that other, more popular languages are only beginning to support.

One of JavaScript’s more advanced features is its support for closures, which C# 2.0 supports through its anonymous methods. A closure is a runtime phenomenon that comes about when an inner function (or in C#, an inner anonymous method) is bound to the local variables of its outer function. Obviously, it doesn’t make much sense unless this inner function is somehow made accessible outside the outer function. An example will make this clearer.

Let’s say you need to filter a sequence of numbers based on a simple criterion that only numbers bigger than 100 can pass, while the rest are filtered out. You can write a function like the one in Figure 8.

Figure 8 Filtering Elements Based on a Predicate (Close Figure 8)
function filter(pred, arr) {
    var len = arr.length;
    var filtered = []; // shorter version of new Array();
    // iterate through every element in the array...
    for(var i = 0; i < len; i++) {
        var val = arr[i];
        // if the element satisfies the predicate let it through
        if(pred(val)) {
            filtered.push(val);
        }
    }
    return filtered;
}

var someRandomNumbers = [12, 32, 1, 3, 2, 2, 234, 236, 632,7, 8];
var numbersGreaterThan100 = filter(
    function(x) { return (x > 100) ? true : false; },
    someRandomNumbers);

// displays 234, 236, 632
alert(numbersGreaterThan100);

But now you want to create a different filtering criterion, let’s say this time only numbers bigger than 300. You can do something like this:

var greaterThan300 = filter(
    function(x) { return (x > 300) ? true : false; },
    someRandomNumbers);

And then maybe you need to filter numbers that are bigger than 50, 25, 10, 600, and so on, but then, being the smart person you are, you realize that they’re all the same predicate, "greater than." Only the number is different. So you can factor the number out with a function like this

function makeGreaterThanPredicate(lowerBound) {
    return function(numberToCheck) {
        return (numberToCheck > lowerBound) ? true : false;
    };
}
which lets you do something like this:

var greaterThan10 = makeGreaterThanPredicate(10);
var greaterThan100 = makeGreaterThanPredicate(100);
alert(filter(greaterThan10, someRandomNumbers));
alert(filter(greaterThan100, someRandomNumbers));

Watch the inner anonymous function returned by the function makeGreaterThanPredicate. That anonymous inner function uses lowerBound, which is an argument passed to makeGreaterThanPredicate. By the usual rules of scoping, lowerBound goes out of scope when makeGreaterThanPredicate exits! But in this case, that inner anonymous function still carries lowerBound with it, even long after makeGreaterThanPredicate exits. This is what we call closure—because the inner function closes over the environment (that is, the arguments and local variables of the outer function) in which it is defined.

Closures may not seem like a big deal at first. But used properly, they open up interesting new possibilities in the way you can translate your ideas into code. One of the most interesting uses of closures in JavaScript is to simulate private variables of a class.

Simulating Private Properties

OK, so let’s see how closures can help in simulating private members. A local variable in a function is normally not accessible from outside the function. After the function exits, for all practical purposes that local variable is gone forever. However, when that local variable is captured by an inner function’s closure, it lives on. This fact is the key to simulating JavaScript private properties. Consider the following Person class:

function Person(name, age) {
    this.getName = function() { return name; };
    this.setName = function(newName) { name = newName; };
    this.getAge = function() { return age; };
    this.setAge = function(newAge) { age = newAge; };
}

The arguments name and age are local to the constructor function Person. The moment Person returns, name and age are supposed to be gone forever. However, they are captured by the four inner functions that are assigned as methods of a Person instance, in effect making name and age live on, but only accessible strictly through these four methods. So you can do this:

var ray = new Person(“Ray”, 31);
alert(ray.getName());
alert(ray.getAge());
ray.setName(“Younger Ray”);
// Instant rejuvenation!
ray.setAge(22);
alert(ray.getName() + “ is now “ + ray.getAge() +
      “ years old.”);

Private members that don’t get initialized in the constructor can be local variables of the constructor function, like this:

function Person(name, age) {
    var occupation;
    this.getOccupation = function() { return occupation; };
    this.setOccupation = function(newOcc) { occupation =
                         newOcc; };

    // accessors for name and age
}
Note that these private members are slightly different from what we’d expect from private members in C#. In C#, the public methods of a class can access its private members. But in JavaScript, the private members are accessible only through methods that have these private members within their closures (these methods are usually called privileged methods, since they are different from ordinary public methods). So within Person’s public methods, you still have to access a private member through its privileged accessors methods:

Person.prototype.somePublicMethod = function() {
    // doesn’t work!
    // alert(this.name);
    // this one below works
    alert(this.getName());
};

Douglas Crockford is widely known as the first person to discover (or perhaps publish) the technique of using closures to simulate private members. His Web site, javascript.crockford.com, contains a wealth of information on JavaScript—any developer interested in JavaScript should check it out.

Inheriting from Classes

OK, you’ve seen how constructor functions and prototype objects allow you to simulate classes in JavaScript. You’ve seen that the prototype chain ensures that all objects have the common methods of Object.prototype. You’ve seen how you can simulate private members of a class using closures. But something is missing here. You haven’t seen how you can derive from your class; that’s an everyday activity in C#. Unfortunately, inheriting from a class in JavaScript is not simply a matter of typing a colon like in C#; it takes more than that. On the other hand, JavaScript is so flexible that there are a lot of ways of inheriting from a class.

Let’s say, for example, you have a base class Pet, with one derived class Dog, as in Figure 9. How do you go about this in JavaScript? The Pet class is easy. You’ve seen how you can do this:

Figure 9 Classes
Figure 9 Classes

// class Pet
function Pet(name) {
    this.getName = function() { return name; };
    this.setName = function(newName) { name = newName; };
}

Pet.prototype.toString = function() {
    return “This pet’s name is: “ + this.getName();
};
// end of class Pet

var parrotty = new Pet(“Parrotty the Parrot”);
alert(parrotty);

Now what if you want to create a class Dog, which derives from Pet? As you can see in Figure 9, Dog has an extra property, breed, and it overrides Pet’s toString method (note that convention for JavaScript is to use camel casing for methods and properties names, instead of Pascal casing as is recommended with C#). Figure 10 shows how it is done.

Figure 10 Deriving from the Pet Class (Close Figure 10)
// class Dog : Pet
// public Dog(string name, string breed)
function Dog(name, breed) {
    // think Dog : base(name)
    Pet.call(this, name);
    this.getBreed = function() { return breed; };
    // Breed doesn’t change, obviously! It’s read only.
    // this.setBreed = function(newBreed) { name = newName; };
}

// this makes Dog.prototype inherits
// from Pet.prototype
Dog.prototype = new Pet();

// remember that Pet.prototype.constructor
// points to Pet. We want our Dog instances’
// constructor to point to Dog.
Dog.prototype.constructor = Dog;

// Now we override Pet.prototype.toString
Dog.prototype.toString = function() {
    return “This dog’s name is: “ + this.getName() +
        “, and its breed is: “ + this.getBreed();
};
// end of class Dog

var dog = new Dog(“Buddy”, “Great Dane”);
// test the new toString()
alert(dog);

// Testing instanceof (similar to the is operator)
// (dog is Dog)? yes
alert(dog instanceof Dog);
// (dog is Pet)? yes
alert(dog instanceof Pet);
// (dog is Object)? yes
alert(dog instanceof Object);

The prototype-replacement trick used sets the prototype chain properly, so instanceof tests work as expected if you were using C#. Also, the privileged methods still work as expected.

Simulating Namespaces

In C++ and C#, namespaces are used to minimize the probability of name collisions. In the .NET Framework, namespaces help differentiate Microsoft.Build.Task.Message class from System.Messaging.Message, for example. JavaScript doesn’t have any specific language features to support namespaces, but it’s easy to simulate a namespace using objects. Let’s say you want to create a JavaScript library. Instead of defining functions and classes globally, you can wrap them in a namespace like this:

var MSDNMagNS = {};

MSDNMagNS.Pet = function(name) { // code here };
MSDNMagNS.Pet.prototype.toString = function() { // code };

var pet = new MSDNMagNS.Pet(“Yammer”);

One level of namespace may not be unique, so you can create nested namespaces:

var MSDNMagNS = {};
// nested namespace “Examples”
MSDNMagNS.Examples = {};

MSDNMagNS.Examples.Pet = function(name) { // code };
MSDNMagNS.Examples.Pet.prototype.toString = function() { // code };

var pet = new MSDNMagNS.Examples.Pet(“Yammer”);
As you can imagine, typing those long nested namespaces can get tiresome pretty fast. Fortunately, it’s easy for the users of your library to alias your namespace into something shorter:

// MSDNMagNS.Examples and Pet definition...

// think “using Eg = MSDNMagNS.Examples;”
var Eg = MSDNMagNS.Examples;
var pet = new Eg.Pet(“Yammer”);
alert(pet);

If you take a look at the source code of the Microsoft AJAX Library, you’ll see that the library’s authors use a similar technique to implement namespaces (take a look at the implementation of the static method Type.registerNamespace). See the sidebar "OOP and ASP.NET AJAX" for more information.

Should You Code JavaScript This Way?

You’ve seen that JavaScript supports object-oriented programming just fine. Although it was designed as a prototype-based language, it is flexible and powerful enough to accommodate the class-based programming style that is typically found in other popular languages. But the question is: should you code JavaScript this way? Should you code in JavaScript the way you code in C# or C++, coming up with clever ways to simulate features that aren’t there? Each programming language is different, and the best practices for one language may not be the best practices for another.

In JavaScript, you’ve seen that objects inherit from objects (as opposed to classes inheriting from classes). So it is possible that making a lot of classes using a static inheritance hierarchy is not the JavaScript way. Maybe, as Douglas Crockford says in his article "Prototypal Inheritance in JavaScript", the JavaScript way of programming is to make prototype objects, and use the simple object function below to make new objects, which inherit from that original object:

    function object(o) {
        function F() {}
        F.prototype = o;
        return new F();
    }
Then, since objects in JavaScript are malleable, you can easily augment the object after its creation with new fields and new methods as necessary.

This is all good, but it is undeniable that the majority of developers worldwide are more familiar with class-based programming. Class-based programming is here to stay, in fact. According to the upcoming edition 4 of ECMA-262 specification (ECMA-262 is the official specification for JavaScript), JavaScript 2.0 will have true classes. So JavaScript is moving towards being a class-based language. However, it will probably take years for JavaScript 2.0 to reach widespread use. In the meantime, it’s important to know the current JavaScript well enough to read and write JavaScript code in both prototype-based style and class-based style.

Putting It into Perspective

With the proliferation of interactive, client-heavy AJAX applications, JavaScript is quickly becoming one of the most useful tools in a .NET developer’s arsenal. However, its prototypal nature may initially surprise developers who are more used to languages such as C++, C#, or Visual Basic. I have found my JavaScript journey a rewarding experience, although not entirely without frustration along the way. If this article can help make your experience smoother, then I’m happy, for that’s my goal.

OOP and ASP.NET AJAX

OOP as implemented in ASP.NET AJAX is a little different from the canonical implementation discussed in this article. There are two main reasons for that: the ASP.NET AJAX version offers more possibilities for reflection (which is necessary for declarative syntaxes such as xml-script and for parameter validation), and ASP.NET AJAX aims at translating to JavaScript some additional constructs that are familiar to developers using .NET, such as properties, events, enumerations, and interfaces.

In its current widely available version, JavaScript lacks a few key concepts of OOP that .NET developers are familiar with, and ASP.NET AJAX emulates most of those.

Classes can have property accessors based on a naming convention (example to follow), as well as multicast events following a pattern that closely mirrors that provided by .NET. Private variables are based on the convention that members starting with an underscore are private. Truly private variables are rarely necessary, and this policy enables those variables to be inspected from a debugger. Interfaces are also introduced to enable type checking scenarios beyond the usual duck-typing (a type scheme based on the concept that if something walks like a duck and quacks like a duck, it’s a duck, or at least it can be treated like one).

Classes and Reflection

In JavaScript, there is no way to know the name of a function. Even if this was possible, it wouldn’t help us in most situations as a class constructor is typically built by assigning an anonymous function to a namespaced variable. What really constitutes the type name is the fully qualified name of this variable, which is equally inaccessible and of which the constructor function itself knows nothing. To work around this limitation and have rich reflection over JavaScript classes, ASP.NET AJAX requires the names of types to be registered.

The reflection APIs in ASP.NET AJAX work over any type, whether it’s built-in, a class, an interface, a namespace, or even an enumeration, and they include .NET Framework-like functions like isInstanceOfType and inheritsFrom to inspect the class hierarchy at run time. ASP.NET AJAX also does some type checking in debug mode where it makes sense to help developers catch bugs earlier.

Registering Class Hierarchies and Calling Base

To define a class in ASP.NET AJAX, you need to assign its constructor to a variable (notice how the constructor calls base):

MyNamespace.MyClass = function() {
    MyNamespace.MyClass.initializeBase(this);
    this._myProperty = null;
}
Then, you need to define the class members itself in its prototype:

MyNamespace.MyClass.prototype = {
    get_myProperty: function() { return this._myProperty;},
    set_myProperty: function(value) { this._myProperty = value; },
    doSomething: function() {
        MyNamespace.MyClass.callBaseMethod(this, “doSomething”);
        /* do something more */
    }
}
And finally you register the class:

MyNamespace.MyClass.registerClass(
    “MyNamespace.MyClass “, MyNamespace.BaseClass);
There is no need here to manage the constructor and prototype hierarchies as this is done for you by the registerClass function.

Bertrand Le Roy is a Software Design Engineer II on the ASP.NET AJAX team.

Ray Djajadinata is having fun developing AJAX applications in Barclays Capital Singapore. You can reach Ray at ray.djajadinata@gmail.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Got Game?: Unleash Your Imagination With XNA Game Studio Express

Got Game?

Unleash Your Imagination With XNA Game Studio Express

Charles Cox and Michael Klucher
This article discusses:
  • Game development basics
  • The Spacewar Starter Kit
  • Adding features to Spacewar
  • Customizing game graphics
This article uses the following technologies:
XNA Game Studio Express, Xbox 360, C#
Code download available at: XNA2007_05.exe (155KB)
Contents
Game Development Basics
The Spacewar Starter Kit
Understanding the Game Loop
Shields Up!
Rendering Details
Energize Shields!
Bonus Credit: Change Space!
Blast Off!

W
hile playing a video game, have you ever thought to yourself, "I could put something together like this"? XNA™ Game Studio Express is a new game development solution for both Microsoft® Windows® and Xbox 360™ targeted primarily at students, hobbyists, and independent game developers—people like you!

Based on Visual C#® 2005 Express Edition, XNA Game Studio Express is built on the XNA Framework, a managed-code class library designed for game development. The framework streamlines the process of writing code for two very different platforms—Xbox 360 and Windows XP SP2. In fact, much of the code you write will work on either system with no changes at all.

You can immediately start writing games for Windows simply by downloading XNA Game Studio Express from MSDN® at msdn2.microsoft.com/xna. To develop games on your Xbox 360 console, you’ll need to obtain a membership in the XNA Creators Club (for more information, see msdn2.microsoft.com/bb219592.aspx). As a member, you’ll have access to game assets as well as exclusive XNA Creators Club versions of starter kits.

Before we get started walking you through XNA Game Studio Express, you’ll need to download and install it. If you’d like help getting set up on your Xbox 360 console, you can find a video walkthrough at msdn.com/xna/videos. Look for the video titled "Getting Started with the XNA Creators Club."

Game Development Basics

Game development presents several unique challenges. Games must strive to be compelling for users—not only from a design perspective, but technically as well—with eye-catching graphics, all while maintaining reasonable frame rates. Intuitive ways for the user to interact with the game via input as well as quality audio output are additional considerations.

Before the introduction of XNA Game Studio Express, game developers used technologies such as DirectX® and OpenGL. These APIs allow basic access to the hardware, but are not intuitive. As a result, developers often assembled core components such as graphics, audio, and input into a framework called an engine, then layered specific game logic over the top of the framework. This abstraction is often costly and time-consuming to create and use.

Furthermore, managing memory and logic in C++, the language of choice for game development, is technically demanding, as is managing the many pieces of art content present in a game. No wonder game development has been particularly daunting for amateur developers. Similarly forbidding are the complex technical and business requirements of developing for next-generation video game consoles such as the Xbox 360.

Enter XNA Game Studio Express, built atop Visual C# 2005 Express Edition. XNA Game Studio Express is designed for hobbyists, students, and other game development enthusiasts, enabling them to create their own games on both Windows and Xbox 360 using C# and the Visual C# 2005 Express Edition integrated development environment.

XNA Game Studio Express includes the XNA Framework, a cross-platform game development API for both Windows and Xbox 360. The XNA Framework drastically simplifies not only graphics, audio, input, and storage, but also the fundamental timing and drawing loops used in all games.

Also included in XNA Game Studio Express is the XNA Framework Content Pipeline, which is integrated into the build environment of Visual C# 2005 Express Edition. With the Content Pipeline, game developers can now add their art content—such as 3D models, graphical textures, shaders, and audio—into their solution in the IDE, and the content will be built into runtime objects that can be accessed in code. This reduces the overhead associated with art content.

Previous game development solutions typically started from scratch: manually writing timing and render loops that managed each tick of time and ensured the game action was handled properly. XNA Game Studio Express constructs the timing and render loops automatically. You are provided with a Game class that handles its own timing and rendering, exposing an Update method where you can add objects to update each frame and a Draw method where you can add objects to render to the screen.

Furthermore, the provided game framework, called the Application Model, automatically sets up the appropriate graphics device without complicated device enumeration code. With this code in place, an XNA Game Studio Express project can be compiled and run, with a fully functional timing and rendering loop running from the moment it is created.

The Spacewar Starter Kit

In addition to XNA Framework code templates, XNA Game Studio Express comes with the Spacewar Starter Kit—a fully functional game, complete with source code and art assets you can modify. The Spacewar Starter Kit is a top-down space shooter game for two players, based on the original implementation of Spacewar, developed for the DEC PDP-1 in 1962 by Stephen Russell, Peter Samson, and Dan Edwards. The starter kit features two versions of the game: a Retro version that retains the original’s gameplay and graphics, and an Evolved version that features improved 3D graphics and sound.

We’ll open up the starter kit, try out the Evolved version, explore how the game is constructed, and then add a new feature: a shield that each player can use to temporarily block enemy shots.

First, launch XNA Game Studio Express, click the File menu, and then click New Project. When the New Project window opens, click the Spacewar Windows Starter Kit, and then click OK (see Figure 1). If you’ve joined the Creators Club and have set up your Xbox 360 to receive from XNA Game Studio Express, click the Spacewar Xbox 360 Starter Kit instead, and then click OK.

Figure 1 Opening the Spacewar Starter Kit
Figure 1 Opening the Spacewar Starter Kit(Click the image for a smaller view)
Figure 1 Opening the Spacewar Starter Kit
Figure 1 Opening the Spacewar Starter Kit (Click the image for a larger view)

The Spacewar Starter Kit will take a moment to load. When it has fully loaded, you’ll see the code files displayed in the Solution Explorer pane. You’ll also see instructions on how to play the game. For the best experience, plug in two Xbox 360 controllers. You can also use the keyboard; the controls are shown in the instructions page when the Spacewar Starter Kit is loaded.

Compile and run the Spacewar application by pressing F5. Once the game has started, select Evolved mode and try out the gameplay for a few moments.

Understanding the Game Loop

One of the core features provided by XNA is the game loop. This loop continually updates the state of the game (based on user input, in-game conditions, and any other applicable condition) and renders it (which involves drawing to the screen, playing appropriate audio, rumbling a controller, and providing any other form of output to the user). Understanding how the game loop in Spacewar works will better help you to understand some of the modifications we’ll be making to the Starter Kit in this article.

Game execution begins as in any other .NET application, with a Main method. The Main method simply constructs an instance of the SpacewarGame class and calls its base class’s Run method:

static void Main(string[] args)
{
    using (SpacewarGame game = new SpacewarGame())
    {
        game.Run();
    }
}

Note that SpacewarGame is derived from the Game class in the Microsoft.Xna.Framework namespace; it exposes key methods, properties, and events for the game lifecycle. You then override methods on Game to provide specific functionality for your game. For example, SpacewarGame overrides the BeginRun method (which is called from Run) to play the game’s title music, change its internal state machine to the logo screen stage, and create the initial camera from which a player views everything.

SpacewarGame’s ChangeState method, as called by the BeginRun method, is responsible for creating many of the objects used in the game. When the BeginRun method calls ChangeState with GameState.LogoSplash as an argument, it reacts by creating a TitleScreen. TitleScreen derives from FullScreenSplash, which in turns derives from the abstract Screen class defined in the Spacewar project. The Screen class represents a unit of rendering for the game, such as a splash screen, selection screen, or an actual game level. It exposes several virtual methods, which its derived types override to provide screen-specific functionality. TitleScreen overrides the Screen.Update method to check whether the user has pressed A, B, or X to indicate where to go next in the game (Evolved, Retro, or Information). TitleScreen is just one of several Screen-derived types used in Spacewar; others include SelectionScreen (for ship selection), EvolvedScreen (for playing the "evolved" version of the game), VictoryScreen (for when a player has won), and so forth.

Each Screen encapsulates the state necessary for that screen’s functionality along with update and render behavior specific to that screen. Both EvolvedScreen and RetroScreen, which represent the main game play for their respective game types, derive from SpacewarScreen:

public class SpacewarScreen : Screen
{
    protected Projectiles bullets;
    protected Ship ship1;
    protected Ship ship2;
    protected SceneItem sun;
    protected Particles particles;
    protected SceneItem backdrop;

    protected bool paused = true;
    protected int player1Score;
    protected int player2Score;

    ...
}

SpacewarScreen contains items to be rendered on the screen, including SceneItems for both ships (implemented as Ship, which derives from SpacewarSceneItem, itself derived from the base SceneItem class), all of the miscellaneous particles and projectiles to be rendered, the backdrop for the game, the sun, and so forth. It also contains game state such as player scores and whether or not the game is paused.

When a Game is created (in our case, a SpacewarGame), it ensures that a host is created to house the game and to provide application-level services to the game, namely notifications for events such as activation, deactivation, exiting, and idling. The latter is most important. The Game.Tick method is called by this idle handler, and the Tick method in turn calls the Game.Update and Game.DrawFrame methods.The Game.Update method is overridden by SpacewarGame.Update, which calls Update on whatever has been set as the current scene (from the ChangeState method). Game.DrawFrame calls Game.Draw, which is overridden by SpacewarGame to call Render on whatever is the current scene.

Shields Up!

It’s easy to augment Spacewar with new features. Let’s add an energy shield. This shield will protect a ship from damage caused by other items in the scene. We’ll need to edit two code files: Ship.cs and SpacewarScreen.cs. You can download it or walk through it at msdn.microsoft.com/msdnmag/code07.aspx.

The file Ship.cs is where we’ll do most of our coding. In Spacewar, the Ship.cs file defines a Ship object. The Ship object contains methods and data that define a spaceship object controlled by a player. Look in Solution Explorer for Ship.cs and double-click it to open the file.

First, we need to define a few variables to control the logic of the shield. The shield will stay up for five seconds, and can be used only once per round. Go to line 40 in Ship.cs, and add the following code (you’ll be adding this among other private members that store information such as whether the ship is currently invulnerable, inHyperspace, or inRecovery):

private bool inShield;
private bool shieldUsed;
private double exitShieldTime;
private const double shieldTime = 5.0;
This establishes a few variables that will track whether the shield is active, whether the shield has already been used, and how much time is left on the shield’s timer.

Let’s use a property so we can easily access whether the ship’s shield is active. This will be accessed by our modifications in SpacewarScreen.cs shortly. Go to line 188 in Ship.cs (what will now be line 188 after adding the previous snippet), and add the following code:

public bool Shield { get { return inShield; } }

So far, we’ve put a set of variables and an accessor in place. Let’s tie these variables to some game action. We’ll be modifying the Update method inside Ship.cs. As described, the Ship’s Update method is called by the game loop’s Update call, which the XNA Framework application model calls automatically once per frame (or more often, depending on how fast the game is running). Update is where you will move your game objects around, have them interact with each other, and take user input, which is what we’ll be doing in this Update method.

Next, we need to modify the Ship’s Update method to respond to input from the user to enable our new shield. Go to line 248 in Ship.cs, which is inside the Update method just before the check for "if (inHyperspace)", and add the following code:

if (XInputHelper.GamePads[player].XPressed && !shieldUsed)
{
    inShield = true;
    shieldUsed = true;
    exitShieldTime = shieldTime;
    Sound.PlayCue( Sounds.PhaseActivate );
}
One line of input, shown in the conditional statement, is all it takes to check for a keystroke or controller button press. XInputHelper, provided in the Spacewar Starter Kit, provides a wrapper around both the Xbox 360 controller and keyboard inputs to simplify input. Using XInputHelper, this if statement checks whether the X button is pressed for the player that the ship is associated with. If you are using a keyboard, the equivalent key is F for player 1, and Page Up for player 2.

You can easily change these keyboard mappings by modifying the settings.xml file in the Spacewar project. Settings.xml is unique to Spacewar (it’s loaded in SpacewarGame’s override of the Game.Initialize method), but XML data files are a good choice in general for game data. Since XNA Game Studio Express is built on the .NET Framework, the Xml namespace is exposed to games.

The code inside the conditional raises the shield, marks the shield as having been used this round, and starts the shield timer. It also uses the XNA Framework audio system to play a cue, which in turn plays a sound for the user. Spacewar abstracts the audio layer of the XNA Framework using enumeration values such as Sounds.PhaseActivate to identify the audio cues to play. Generally, you call cues using a string value. If the number of cues in your game is limited, such as in Spacewar, an enumeration is a good solution, especially if cue names are changing during development.

You add sounds to a game by using the Microsoft Cross-Platform Audio Creation Tool (XACT), provided with XNA Game Studio Express. All of the audio files and their associated cues are kept in an XACT project, which is displayed in a graphical interface that allows you to drag and drop sounds into the project. When you build your game, the XNA Framework Content Pipeline loads the XACT project and builds the necessary audio files. If you want to add a sound to Spacewar, you’ll have to take the extra steps of adding the cue name to the array of cue names and providing an enumeration value to the Sounds index into the cue name array. Then you can call your sound by using the Sounds enumeration.

In the Update call, you’ll want to add the necessary logic to run the shield timer if the shield is active. Go to line 348 (at the end of the "if (!Paused)" block), and add the following code:

if (inShield)
{
    exitShieldTime -= elapsedTime.TotalSeconds;
    if (exitShieldTime <= 0)
    {
        inShield = false;
        Sound.PlayCue( Sounds.PhaseExpire );
    }
}
This code simply decrements the time left on the shield by the elapsed time. Elapsed time is a measure of how much time has passed between calls to Update. This amount, applied to game logic such as motion or timers, simulates the passing of time, and is passed into Update by the XNA Framework. Once the shield time runs out, the shield is deactivated and an appropriate sound is played.

Rendering Details

While we’ve handled input and sound handily, the question remains: what will the shield look like? We’ll need to put some code into the Render method. So using some math and drawing functionality provided in the XNA Framework, we’ll draw a sprite—a 2D texture with transparency—on top of the ship’s 3D position.

In Solution Explorer, find the Content folder in the solution tree, open it, and then open the Textures folder beneath it. You’ll see a variety of .tga files. We’ll be using one of these as our shield texture. A .tga file represents a 2D graphical image, a sprite.

Go to line 433 in Ship.cs and make a new line immediately following it—after the end of the block for "if (showThrust && evolved)". In that new line, add the code shown in Figure 2.

Figure 2 Rendering a Sprite (Close Figure 2)
if (inShield && evolved)
{
    Texture2D shieldtexture =
        SpacewarGame.ContentManager.Load<Texture2D>(
        SpacewarGame.Settings.MediaPath + @”textures\circle” );

    batch.Begin( SpriteBlendMode.AlphaBlend );

    // Move into screen space.
    Vector3 location = device.Viewport.Project(
        position, SpacewarGame.Camera.Projection,
        SpacewarGame.Camera.View, Matrix.Identity );

    batch.Draw(
        shieldtexture,
        new Vector2( location.X - shieldtexture.Width / 2.0f,
                     location.Y - shieldtexture.Height / 2.0f ),
        new Rectangle( 0, 0, shieldtexture.Width, shieldtexture.Height ),
        new Color( new Vector4( 0.0f, 1.0f, 0.0f, 0.5f ) ), 0.0f,
        new Vector2( shieldtexture.Width / 2.0f,
                     shieldtexture.Height / 2.0f ),
        (3.0f * (float)(Math.Abs( Math.Cos( exitShieldTime ) ))) + 2.0f,
        SpriteEffects.None, 0.0f);

    batch.End();
}

Let’s walk through what’s happening in this code. First, using the XNA Framework and the XNA Framework Content Pipeline, the ContentManager class loads the texture called circle from the Content\Textures folder. The ContentManager class is an XNA Framework class that loads content from the Content Pipeline at run time. It is part of the prepopulated Application Model code in each new XNA Game Studio Express project. The name "circle" is associated with the circle.tga file in the Content\Textures folder. The XNA Framework Content Pipeline assigns a friendly name to each content file, which is used to load the content at run time.

Next, the texture is loaded into a Texture2D class provided by the XNA Framework. The Texture2D object will be drawn on the screen using a SpriteBatch (named batch in this code block), which represents a batch of textures that will be drawn on the screen at once.

We call batch.Begin to signal that sprite drawing is about to begin, and note that we want to use alpha blending. This will allow the circle texture to be transparent around its edges and partially transparent throughout, because we want to still be able to see the ship below the shield.

Before we draw, we must know where on the screen to draw. Spacewar Evolved is a 3D game, so the objects move around in three dimensions and are projected onto a 2D plane only when they are drawn. We must find the 2D position of the 3D ship before we know where we should draw the shield.

The XNA Framework allows us to get this information from the graphics device. We call device.Viewport.Project and pass in the 3D coordinates of the ship, as well as view and projection matrices. The view matrix defines the linear algebra transformation applied to each 3D object to place the object correctly relative to the player’s viewpoint. The projection matrix transforms the 3D coordinates for each object to 2D screen coordinates that allow the objects to be drawn to the screen. These matrices are often precomputed and stored. In Spacewar, they are kept in a Camera class that also defines the player’s viewpoint. Calling device.Viewport.Project with these values simulates transforming the ship’s location to 2D coordinates without actually drawing it at those coordinates. This call returns a vector that holds the 2D draw coordinates we will use to draw our shield.

The next step is to call batch.Draw, passing in the following:

  • The Texture2D object.
  • Our projected coordinates minus half the width and height of the sprite; this centers the sprite right on the ship.
  • The source size of the texture.
  • A color to multiply with the sprite’s existing colors. (We construct it as a Vector4, with three components representing red, green, and blue, and a fourth for alpha, or opacity. These particular values define a partially transparent green; the transparency allows us to see the ship underneath the shield, rather than obscuring it.)
  • A scaling factor. (We’re being tricky here by taking the cosine of the remaining shield time, which makes the size of the shield fluctuate over time, just for graphical kicks.)
  • A sprite effect flag that we set to None, but which could be used to flip the sprite horizontally or vertically.
  • A screen depth. A value of zero makes the sprite draw above everything else, so it’s guaranteed to render on top of the ship instead of underneath it.

Energize Shields!

At this point a player can raise a shield, visualize it, and have the shield disappear after an appropriate period of time. Unfortunately, the shield doesn’t yet provide any protection. To get that functionality, let’s open SpacewarScreen.cs and add two simple blocks of code, one for each player. Go to line 94 in SpacewarScreen.cs, and add the following code to the beginning of HitPlayer2:

if (ship2.Shield)
{
    // Make a bouncy sound rather than an explosion.
    Sound.PlayCue( Sounds.WeaponPickup );
    return;
}
And add the following at line 122, to the beginning of HitPlayer1:

if (ship1.Shield)
{
    // Make a bouncy sound rather than an
    // explosion.
    Sound.PlayCue( Sounds.WeaponPickup );
    return;
}

As described earlier, SpacewarScreen.cs represents the update logic for the Spacewar game loop. Many games, including Spacewar, utilize separate game logic loops for the various states of a game, such as the main menu, weapon selection, and the actual game itself. As the players progress through the games, control is handed off from state to state. In Spacewar, these states are called screens and are represented by the Screen class. SpacewarScreen is derived from Screen and handles the actual gameplay state, in which the two players control their spaceships and do battle.

In the SpacewarScreen class, the Update loop manages the game objects, such as the spaceships, weapons, and asteroids that float around the screen. When these objects collide with the spaceships, the logic calls HitPlayer1 or HitPlayer2, which calculates the damage to be dealt to the spaceships.

This code modifies the game logic in HitPlayer1 and HitPlayer2 so that when a ship is dealt damage, the shield is checked before any damage is resolved. If the shield is up, a deflection sound is played, and the damage function exits before taking any more action.

That’s it. Compile and run with F5, and try it out! From the Main Menu, select the Evolved game mode. Once you start the game, press the X button or the keyboard equivalent (F for player 1, Page Up for player 2) to raise the shield, as shown in Figure 3. Both players can use the shield only once per round, but the shield is very strong: it will absorb bullets of every type, asteroid hits will cause no damage—even the sun can’t hurt you!

Figure 3 Shields Energized!
Figure 3 Shields Energized!(Click the image for a smaller view)
Figure 3 Shields Energized!
Figure 3 Shields Energized! (Click the image for a larger view)

Take a spin around the block with the shield, but be sure not to get yourself too close to the sun when the shield runs out—it only lasts five seconds!

Some ideas about where to go from here: using some vector math, you could modify bullets to bounce off the shield rather than being absorbed. It could mean a painful surprise for an opponent when his own shots return to him! Or modify the damage system to make a shielded ship deal double damage when it strikes an opponent! Ramming speed!

The fun of writing games is in experimentation. With XNA Game Studio Express, it’s easy to get down to the good stuff and truly make a game your own creation.

Bonus Credit: Change Space!

XNA Game Studio Express doesn’t only help people worried about how to best code new features into their games. It also incorporates a feature set, mentioned earlier, called the XNA Framework Content Pipeline. The XNA Framework Content Pipeline helps you create 2D, 3D, and even audio content for your XNA Framework–based game.

Part of what the XNA Framework Content Pipeline does is to integrate the content building and compiling process into Visual C# Express. The added benefit of this is that you can launch certain types of content directly from Visual C# Express: you edit the content and then recompile to see the changes in your game. By compiling your content at the same time you compile your code, you can unearth content errors before you run your game.

The XNA Framework Content Pipeline supports various 2D graphics file formats, including .bmp, .jpg, .png, .tga, and .dxt. Two 3D file formats are also supported: .x and .fbx. Both of these 3D formats have file format exporters for a large number of both free and professional 3D graphics creation tools. The XNA Framework Content Pipeline is also designed to be extremely extensible, which means you may be able to find support for more of your favorite file formats by searching online.

Let’s explore the XNA Framework Content Pipeline using Spacewar. Back in XNA Game Studio Express, go to the Solution Explorer. Open the Content folder, then the Textures folder. Here, you’ll see many of the textures that Spacewar uses. Click the file B1_nebula01.tga; this is one of the backgrounds used when playing the game. The property grid at the bottom of the Solution Explorer should then show some relevant properties (see Figure 4). If you cannot see the property grid, right-click the B1_nebula01.tga file and click Properties.

Figure 4 Textures Properties
Figure 4 Textures Properties

In the property grid, you’ll see several unique properties dedicated to the XNA Framework Content Pipeline. The Content Importer property is being used here. An importer is responsible for taking the source data—in this case a .tga file—and transforming it to the XNA Framework content DOM. The content DOM holds data that has been transformed and normalized by the importers, so that processors can work independently of file formats. The next property is the Content Processor property. The processor is responsible for converting data from the content DOM to an object suitable for use at run time—in this case, a sprite. The Asset Name property is used for loading the asset at run time; you can turn off processing of the asset by setting the XNA Framework Content property to false.

Let’s try editing the B1_nebula01.tga graphic file to see how it changes in Spacewar. You’ll need an image editor capable of working with a .tga file. If you don’t have one, you can download Paint.NET from getpaint.net.

Double-click the file to launch the associated editor for that file. Then go ahead and edit the file a bit (see Figure 5). One easy way in Paint.NET would be to change the hue and saturation by clicking the Adjustments menu, then clicking Hue/Saturation. Move the sliders to change the hue, saturation, and lightness of the graphic, then click OK to commit the changes. After you’ve changed the file, save it, and return to XNA Game Studio Express. Compile and run the Spacewar project by pressing F5. XNA Game Studio Express will detect that the file has been changed and will rebuild and redeploy the new asset (even to the Xbox 360 console). When you start an evolved game, you should see your modified background!

Figure 5 Editing a Texture in Paint.NET
Figure 5 Editing a Texture in Paint.NET(Click the image for a smaller view)
Figure 5 Editing a Texture in Paint.NET
Figure 5 Editing a Texture in Paint.NET (Click the image for a larger view)

Blast Off!

These simple examples demonstrate only a fraction of the power of XNA Game Studio Express. The cross-platform combination of the XNA Framework and XNA Framework Content pipeline means that you’ll be able to focus on what really makes game development fun: great content and inventive gameplay.

Furthermore, we recommend that you join the XNA Creators Club so that you’ll be able to take your game development skills to the next level. Becoming a member of the club not only opens the door to developing your own games on the Xbox 360 platform, but also gives you access to exclusive content, samples, starter kits, and a lot more resources that will help you take your game from your imagination into reality.

Charles Cox is a Developer Educator in the Microsoft Game Platform Documentation Group. Previously a Technical Writer for DirectX and Microsoft Game Studios, Charles now develops and delivers XNA Game Studio Express tutorials and educational workshops.

Michael Klucher is a Program Manager on the XNA Game Studio team at Microsoft, working on the XNA Framework Content Pipeline and XNA Game Launcher for the Xbox 360 console. Contact Michael via his blog at klucher.com.


WPF: Customizing Controls For Windows Presentation Foundation

WPF

Customizing Controls For Windows Presentation Foundation

Shawn Wildermuth
Code download available at: WpfControls2007_05.exe (173KB)
This article discusses:
  • Composite controls
  • Styles and templates
  • Custom properties
  • Custom controls
This article uses the following technologies:
Windows Presentation Foundation
Contents
Using Composition
Using Styles
Using Templates
Authoring Controls
Custom Properties
Where Are We?

T
he breadth of the control model in Windows Presentation Foundation is quite staggering, but it is impossible to supply every control that you could ever need. That is where control authoring comes to the rescue. In this article, I show you how to customize existing controls using Windows® Presentation Foundation and how to create entirely new controls (or elements) for use in your projects.

Before you develop a custom control, you need to ask yourself if you really need one. In Windows Presentation Foundation, composition, styling, and templating enable you to customize existing controls to an extent unprecedented with past technologies. Before you decide to create a new control, let’s quickly review these three methods for customizing controls.

Using Composition

A common requirement is to create a composite control—a control made up of more than one control. Suppose you have a Play button to start playback of a video. The XAML and control are shown in Figure 1.

Figure 1 A Simple Play Control (Close Figure 1)
<StackPanel>
  <Button Height=”50” Width=”50” Content=”Play” />
  <Polygon HorizontalAlignment=”Center”
           Points=”0,0 0,26 17,13”
           Fill=”Black” />
</StackPanel>

You need to be able to take the play icon and put it on the button. You can use composition to actually embed XAML elements inside other XAML elements. For example, you can change the XAML to create the label and icon as the contents of the button. Placing these elements inside a container (StackPanel in this case) inside the button assigns them to the Content property of the Button class, as seen in Figure 2. This results in a button that works like any other button but has your content inside it.

Figure 2 All of the Content Inside the Button (Close Figure 2)
<StackPanel>
  <Button Height=”50” Width=”50”>
    <StackPanel>
      <TextBlock>Play</TextBlock>
      <Polygon Points=”0,0 0,26 17,13”
               Fill=”Black” />
    </StackPanel>
  </Button>
</StackPanel>

Using composition to create controls like this is simple. Unlike in presentation technologies like Windows Forms, Visual Basic® 6.0, and MFC, most controls are containers for other controls. It’s unnecessary to write a custom control when all you really need is a composite control.

Using Styles

What if all you need is to change the appearance of the control? Styles are the answer. You can specify a style of button that has a red border around it by creating a Style like this one.

<StackPanel>
  <StackPanel.Resources>
    <Style TargetType=”Button” x:Key=”RedButton”>
      <Setter Property=”BorderBrush” Value=”Red” />
    </Style>
  </StackPanel.Resources>
  ...
</StackPanel>

Now you can change the border of specific buttons by assigning a style to them, as seen in Figure 3. The first button is the standard appearance, while the second one binds itself to a shared style.

Figure 3 Applying a Style to a Button (Close Figure 3)

Figure 3
<Button Height=”50” Width=”50”>
  <StackPanel>
    <TextBlock>Play</TextBlock>
    <Polygon Points=”0,0 0,26 17,13”
             Fill=”Black” />
  </StackPanel>
</Button>
<Button Height=”50” Width=”50” Style=”{StaticResource RedButton}”>
  <StackPanel>
    <TextBlock>Play</TextBlock>
    <Polygon Points=”0,0 0,26 17,13”
             Fill=”Black” />
  </StackPanel>
</Button>

You can even use styles to change the appearance of all instances of a certain type of XAML element across a container. For example, instead of creating a reusable style to change the button, you could create a style that specifies the look of all buttons, as seen in Figure 4. This example sets the background of all buttons to a gray/green/gray gradient. The Style in this example omits the Key of the style. This causes it to affect all the elements specified in the TargetType attribute.

Figure 4 Applying a Style Across a Container (Close Figure 4)

Figure 4
<StackPanel>
  <StackPanel.Resources>

    <Style TargetType=”Button”>
      <Setter Property=”Background”>
        <Setter.Value>
          <LinearGradientBrush>
            <GradientStop Color=”#DDDDDD” Offset=”0” />
            <GradientStop Color=”#88FF88” Offset=”.6” />
            <GradientStop Color=”#EEEEEE” Offset=”1” />
          </LinearGradientBrush>
        </Setter.Value>
      </Setter>
    </Style>

  </StackPanel.Resources>

  <Button Height=”50” Width=”50”>
    <StackPanel>
      <TextBlock>Play</TextBlock>
      <Polygon Points=”0,0 0,26 17,13”
               Fill=”Black” />
    </StackPanel>
  </Button>

  <Button Height=”50” Width=”50”>
    <StackPanel>
      <TextBlock>Play</TextBlock>
      <Polygon Points=”0,0 0,26 17,13”
               Fill=”Black” />
    </StackPanel>
  </Button>

</StackPanel>

Using Templates

Styles are limited to setting default properties on XAML elements. For example, when I set the BorderBrush in the earlier examples, I could specify the brush but not the width of the border. For complete freedom of a control’s appearance, you need to use templates. To do this, you create a style and specify the Template property (see Figure 5). The Value of the Template property becomes a ControlTemplate element that specifies how to compose the control itself. In this example, I specify a button that is a circle with the play icon in the center. I do this by layering the play icon over an Ellipse element. The new template button can be seen next to a normal button.

Figure 5 Using a Template (Close Figure 5)

Figure 5
<StackPanel>
  <StackPanel.Resources>

    <Style TargetType=”{x:Type Button}” x:Key=”PlayButton” >
      <Setter Property=”Template”>
        <Setter.Value>
          <ControlTemplate TargetType=”{x:Type Button}”>
            <Grid>
              <Ellipse Width=”{TemplateBinding Width}”
                       Height=”{TemplateBinding Height}”
                       Stroke=”DarkGray”
                       VerticalAlignment=”Top”
                       HorizontalAlignment=”Left”
                       Fill=”LightGray” />
              <Polygon Points=”18,12 18,38 35,25”
                       Fill=”Black” />
            </Grid>
          </ControlTemplate>
        </Setter.Value>
      </Setter>
    </Style>

  </StackPanel.Resources>

  <Button Height=”50” Width=”50”>Normal Button</Button>
  <Button Height=”50” Width=”50” Style=”{StaticResource PlayButton}” />

</StackPanel>

In the end, styles and templates still only allow you to change the appearance of a control. To add behavior and other features to the button, you’ll need to create a custom control.

Authoring Controls

The first step you should take before writing your own control is to decide which method you will use for creating the control. There are two main ways to create controls in Windows Presentation Foundation: user controls and custom controls. There are benefits to both approaches.

With user controls, you get a simple development model that is similar to Windows Presentation Foundation application development. User controls are preferred when you need to compose a control out of existing components and do not need complex customization (as with templates and styles). Custom controls are a better choice when you want full control over appearance, need special rendering support, or want your control to be a container for other controls.

If you can’t decide which type of control to choose, pick a user control. If you run into a functionality wall with a user control, you can switch later to custom with little pain.

The first thing to do when creating a user control is to add a new item to your project. If you right-click your project and click Add, you might be tempted to pick the User Control option from the context menu. Unfortunately, this attempts to create a new Windows Forms user control. Instead, pick the Add New Item option. In the Add New Item dialog, pick the User Control (WPF) item.

Creating the new user control creates a new XAML file and backing code file. The XAML file is similar to the main file that is created with new Windows Presentation Foundation projects; the difference is that the root element of the new XAML file is a UserControl element. Inside the UserControl element, you create the content that composes your control.

For this example, continue with the same XAML used earlier to create a template for a PlayButton control. This new control will tie itself to a MediaElement to control playing or pausing some digital media. Figure 6 shows the PlayButton’s XAML.

Figure 6 PlayButton XAML (Close Figure 6)
Figure 7 PlayButton in a WPF Window
Figure 7 PlayButton in a WPF Window
<!-- PlayButton.xaml -->
<UserControl x:Class=”CustomWPF.PlayButton”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”>

  <Grid>

    <Ellipse Width=”50” Height=”50”
             Stroke=”DarkGray”
             VerticalAlignment=”Top”
             HorizontalAlignment=”Left”
             Name=”ButtonBack”
             Fill=”LightGray” />
    <Path Name=”PlayIcon” Fill=”Black”
          Data=”M18,12 18,38 35,25”/>
    <Path Name=”PauseIcon” Fill=”Black” Opacity=”0”
          Data=”M15,12 15,38 23,38 23,12z M27,12 27,38 35,38 35,12” />

  </Grid>

</UserControl>

From the template example, I have added a new Path, PauseIcon. Because the icon for pausing of the media could not be represented as a Polygon, it was easier to simply change PlayIcon to a Path so you can deal with each icon as a Path object in the codebehind. I want to be able to control a MediaElement element by pausing or playing the media when the button is clicked as well as changing the icon to correctly represent the action (pause or play) when the button is clicked.

Before adding that logic, let’s make sure that my button is displayed correctly and that it can be shown in a window. In this case, I want to show the new control on a window. Before you can use any custom element (user control or custom control) in a XAML document, you must create a reference to it. If the custom element is in the same project as the rest of your XAML, you can just refer to it by adding an XML namespace declaration. In the following lines of code, I created a namespace declaration (xmlns:cust) that specifies a common language runtime (CLR) namespace that the control is in.

<!-- MainWindow.xaml -->
<Window x:Class=”Tester.MainWindow”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:cust=”clr-namespace:CustomWPF“
    Title=”Control Viewer”
    Height=”100” Width=”200”>

  <!-- ... -->

</Window>

The clr-namespace (CustomWPF) specified in the XML namespace declaration matches the actual CLR namespace of the control (CustomWPF). If the control you want to use is in another assembly, you must also note the assembly name in the namespace declaration. The XML namespace declaration does not import the assembly into your project automatically; you must also add a reference to the assembly to your project manually.

<!-- MainWindow.xaml -->
<Window x:Class=”Tester.MainWindow”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:cust=”clr-namespace:CustomWPF;assembly=CustomWPF”
    Title=”Control Viewer”
    Height=”100” Width=”200”>

  <!-- ... -->

</Window>

Once you have the reference to the XML namespace, you can use it to create instances of your new user control. You do this using the name of the namespace declaration. This means the XML namespace alias, not the CLR namespace. After the XML namespace alias, you would specify the actual name of the control. This ensures that the name used in the XAML file matches the name of the class. To add an instance of the PlayButton class to the XAML you would specify cust:PlayButton as the element name.

<StackPanel>
  <TextBlock HorizontalAlignment=”Center”>User Control:</TextBlock>
  <cust:PlayButton />
</StackPanel>

Now you can see the PlayButton control hosted in a typical Windows Presentation Foundation Window as in Figure 7.

Custom Properties

As you author controls, you will find it necessary to implement properties to manage both the appearance of the control as well as its runtime behavior. For example, the PlayButton control will need the ability to get and set the color of the icon. To do this you can create a simple CLR property as seen in Figure 8 (note that Visual Basic sample code is available in the download for this article).

Figure 8 Getting the Icon’s Color (Close Figure 8)
// PlayButton.xaml.cs
public partial class PlayButton : System.Windows.Controls.UserControl
{
  // ...

  Brush _iconColor = Brushes.Black;

  public Brush IconColor
  {
    get {return _iconColor; }
    set
    {
      _iconColor = value;
      PlayIcon.Fill = _iconColor;
      PauseIcon.Fill = _iconColor;
    }
  }
}

Simple CLR properties like the IconColor property work well enough. You can set them in the XAML just using the name of the property:

<cust:PlayButton IconColor=”Black” />

Simple properties are not sufficient for most controls, though, because they do not support advanced features like data binding or animation support. For example, if you want to specify the IconColor of your control by data binding to the fill color of a rectangle in the XAML, it does not work, as you can see in Figure 9.

Figure 9 This Data Binding Won’t Work (Close Figure 9)
<!-- MainWindow.xaml -->
<Window x:Class=”Tester.MainWindow”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:cust=”clr-namespace:CustomWPF;assembly=CustomWPF”
    Title=”Control Viewer”
    Height=”100” Width=”200”>

  <StackPanel>
    <Rectangle Name=”theRect” Fill=”Red” />
    <TextBlock>User Control:</TextBlock>

    <!-- Simple Assignment Works -->
    <cust:PlayButton IconColor=”Blue” />

    <!—Data Binding Does Not -->
    <cust:PlayButton
      IconColor=”{Binding ElementName=theRect, Path=Fill}” />

  </StackPanel>
</Window>

To exploit all the features available, you must use a Dependency Property instead of a simple CLR property. Dependency properties allow the value of an element to be set by a variety of means, including animations and data binding. To implement one, create a static (shared in Visual Basic) DependencyProperty field on the control by calling DependencyProperty.Register method. This method registers your property and returns an instance of the created DependencyProperty.

Once you have the DependencyProperty field, you can use it to set and get the property by using the GetValue/SetValue methods of your control. For example, you can change the IconColor property of the PlayButton to a DependencyProperty as seen in Figure 10.

Notice that I removed the Brush field from the class. The DependencyProperty stores the value for each instance as well as the metadata about the property. This means that each instance of the PlayButton does not need to have its own field to store the data about the property.

Now that you have a DependencyProperty, you can use it in data binding or animation. Currently there is no good way of determining that the property has changed, nor is there a default value. Because setting the value of the property is not routed through the public property (the CLR property is a wrapper for the DependencyProperty, not vice versa), you cannot just use the set accessor to change the icon color when that value is changed. You need to add an event to be called when the property changes. To do this, specify a static or shared method to be called back when the property changes when you register the DependencyProperty. You can amend the registration to include a FrameworkPropertyMetadata object that specifies both the default value and a change callback, as shown here:

public static readonly DependencyProperty IconColorProperty =
        DependencyProperty.Register(
            “IconColor”,
            typeof(Brush),
    typeof(PlayButton),
    new FrameworkPropertyMetadata(Brushes.Black,
    new PropertyChangedCallback(OnIconColorChanged
    )));

Finally, you need to implement the callback. This callback is a static (or shared) method of your control that accepts the object that was changed as well as arguments that specify the old and new values. Typically, you would call a method on the changed object to update it. For example, if the IconColor has changed, you will want to set the Fill of both icons. Both the callback method and the update method are shown here:

private static void OnIconColorChanged(DependencyObject obj,
    DependencyPropertyChangedEventArgs args)
{
  // When the color changes, set the icon color
  PlayButton control = (PlayButton)obj;
  control.PlayIcon.Fill = control.IconColor;
  control.PauseIcon.Fill = control.IconColor;
}
To complete the control, you will also want a DependencyProperty to allow assigning of a MediaElement to control (see Figure 11).

Now that you have the property, you can add a MediaElement to the XAML and data bind that element to the new MediaPlayer property. When you add the MediaElement to the XAML, you need to set the LoadedBehavior to Manual, which allows you to manually control the playback. The new XAML is shown in Figure 12.

Figure 12 Revised MediaElement XAML (Close Figure 12)
<!-- MainWindow.xaml -->
<Window x:Class=”Tester.MainWindow”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
    xmlns:cust=”clr-namespace:CustomWPF;assembly=CustomWPF”
    Title=”Control Viewer”
    Height=”100” Width=”200”>
  <StackPanel>

    <MediaElement Width=”150” Height=”100”
         Name=”theMedia”
         Source=”http://download.microsoft.com/.../ctorrec9billg.wmv”
         LoadedBehavior=”Manual” />

    <TextBlock>User Control:</TextBlock>

    <cust:PlayButton MediaPlayer=”{Binding ElementName=theMedia}” />

  </StackPanel>
</Window>

Next, implement a click event on the PlayButton user control. First, add a MouseLeftButtonUp event on the control’s main Grid.

<!-- PlayButton.xaml -->
<UserControl x:Class=”CustomWPF.PlayButton”
    xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
    xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”>
  <Grid MouseLeftButtonUp=”PlayButton_Clicked”>
    <!-- ... -->
  </Grid>
</UserControl>
This allows you to implement the behavior of your control as well as changing the icon. This event handler implementation is seen in Figure 13. The completed control and video can be seen in Figure 14.

Figure 14 Completed User Control
Figure 14 Completed User Control
Custom Controls

The PlayButton control you just created works pretty well, but it lacks full template and theme support. If your control needs this support, you will need to build it as a custom control. Custom controls derive from other classes in the control hierarchy. For example, you can refactor the PlayButton control into a MediaButton control that is more reusable.

To create a MediaButton control, first pick Custom Control (WPF) in the Add New Item dialog of Visual Studio. This adds a new custom control class (MediaButton.cs) file to the project and also adds a theme folder with a generic.xaml file. The generic.xaml file contains a template for the new control class. It uses this XAML file to allow different themes for the control. The generic.xaml file is used as a fallback; for most controls this is the only theme file you will create. If you want to write a control that changes its appearance depending on the current theme, you can create theme files in this directory. Figure 15 shows the standard theme files and when they are used.

By specifying multiple themes, you can change the appearance of your controls based on the theme selection of the user. To develop a generic theme for my sample control, I can take the user control’s XAML file and place it inside the ControlTemplate tag. Once the XAML is in the template, you will want to use template binding to set properties of the XAML based on properties of the control. Until now, the PlayButton’s width and height were always set to fifty logical units. For a simple control that makes sense, but if you really want the control to be reusable, you should make the template resizable. To do this, replace the height and width of the template with the control’s height and width by marking the height and width in the template to {TemplateBinding Width} and {TemplateBinding Height}.

In the PlayButton, simply change the Opacity of the two icons based on whether it needs to show play or pause. Reaching down into the template to change the Opacity would work, but it would be rather clumsy. A better solution is to have a single icon Path object and change the drawing data to draw the right icon. This way, the size of the visual tree directly influences the performance of the XAML document. To make this work, introduce a new DependencyProperty to store the current icon to use. Create an enumeration that specifies which icon to use, then expose it as a DependencyProperty. With the new Icon property, you can modify the template to include triggers for when the Icon property changes.

The icon is using a Path element to define the appearance of each icon used by the button. This works fine if the button is always a fixed size, but since it is not, you need to find a way to resize the icon with the button. One solution is to create an overlay ellipse over the background circle and use a VisualBrush to paint the icon. The VisualBrush allows the background to size with the control. The completed generic.xaml template is shown in Figure 16.

This MediaButton should control a MediaElement like the one the PlayButton did. Copy the MediaPlayer DependencyProperty from the PlayButton to the MediaButton. Make sure to change any references in the copied code from PlayButton to MediaButton (especially in the DependencyProperty registration).

Unlike PlayButton, you do not need to handle the mouse click event in the template. Instead, you can override the OnMouseLeftButtonUp event to react to clicks. Inside this method, you can change the icon and play or pause media. The final custom control code can be seen in Figure 17. Now that the new control supports resizing and setting the icon property, you can give users more flexibility in changing the size and icon of the control. Figure 18 shows the control with different sizes and icons.

Figure 18 All Shapes and Sizes
Figure 18 All Shapes and Sizes

In this control example, I have directly derived from the System.Windows.Controls.Control class, but the nature of custom controls allows deriving from anywhere in the class hierarchy. You can use custom controls to override and change the behavior of built-in controls or build your own controls completely from scratch. For example, deriving from FrameworkElement would allow you to create a control with very little built-in layout mechanics. Deriving from Panel allows you to create your own specialized containers for other objects. Determining the right class to derive from is not easy; it depends on the requirements for your control.

Where Are We?

When you need specialized control functionality, you have several options, including composition, styles, and templates. By using componentization, you can often create compound controls, obviating the need for authoring a new control. When the appearance of a control is all you need to change, use styling. Lastly, templates allow complete control over the composition of an existing control. For more information on using templates to customize controls, see the Foundations column by Charles Petzold.

When you do decide to author a new control, you’ll still get the simplified programming model that appears to be just like writing your own Window or Page. Being able to create templates for your controls that look and even act differently in different operating system themes is a distinct advantage of custom controls.

Migrating an existing user control to a custom control is not particularly difficult, but because you are working with a template, instead of direct access to XAML objects, you will need to change how you engineer your custom control.

With Windows Presentation Foundation, the necessity for writing custom controls is the exception rather than the rule. Only when you are really creating custom behavior should you ever need to delve into the control authoring. For more information on controls in Windows Presentation Foundation, please see the SDK at msdn2.microsoft.com/ms754130.aspx.

Shawn Wildermuth is a Microsoft MVP, MCSD.NET, MCT, and the founder of Wildermuth Consulting Services. Shawn is also the author of Pragmatic ADO.NET (Addison-Wesley, 2002), and the coauthor of four Microsoft Certification Training Kits as well as the upcoming Prescriptive Data Architectures. He can be contacted through his Web site at www.wildermuthconsulting.com.


Migration: Convert A Java Web Application To ASP.NET Using JLCA

Migration

Convert A Java Web Application To ASP.NET Using JLCA

Brian Jimerson
This article discusses:
  • Resource locating
  • I/O streams
  • Logging and collections
  • Refactoring
This article uses the following technologies:
ASP.NET, JLCA, and C#
Code download available at: JLCA2007_05.exe (157KB)
Contents
About the JLCA
Locating Resources
Handling Input/Output APIs
Logging
Collections
Filters and HTTP Handlers
Source Tree and Naming Conventions
When to Refactor for Conventions
Directory Layout and Namespaces
Properties
Pascal-Casing Method Names
Conclusion

T
he typical software development cycle follows a straightforward model: gather requirements, design the application, write the code, test the software, and deploy it. Once in a while, though, a new development project is incepted simply based on the platform to which the customer wants to deploy the app. In this case, an existing application’s code base can be converted, or ported, to the desired platform.

In this article, I walk you through converting a Java Web application to an ASP.NET application implemented in C#. This article is based on an actual project I worked on. In that project, we had an existing Java-based application and a customer who wanted an ASP.NET version of it. I begin by introducing you to the Microsoft® Java Language Conversion Assistant (JLCA) and I demonstrate common development paradigms that do not have direct counterparts in the two platforms, such as:

  • Input/output
  • Resource resolution
  • Source tree layout and naming conventions
  • Leveraging the executing environment

This application is implemented as a SOAP-compliant Web service, with a traditional relational database persistent store. I won’t discuss the actual Web service presentation layer and SOAP interfaces, but rather the application that powers it. Sample code for this article is available for download.

About the JLCA

The JLCA is a tool used to convert a Java application into a C# application. The tool has shipped with Visual Studio® since Visual Studio .NET 2003. Currently in version 3.0, the JLCA is included with Visual Studio 2005 and is also available as a free download from the JLCA home page.

Version 3.0 includes enhancements to convert Java artifacts, such as Servlets and Java Server Pages (JSPs), as well as rich-client applications that use Swing or the Abstract Windowing Toolkit (AWT). In practice, the JLCA provides a very good place to start a conversion, but it will not successfully complete the entire process. So don’t expect this to be completely hands-free—you will still need to do some manual resolution of converted items after using the tool.

To get started with the JLCA in Visual Studio 2005, launch the wizard by clicking File | Open | Convert. The wizard’s screens are quite self-explanatory. The key piece of information you’ll need to enter, as shown in Figure 1, is the root directory of your existing Java project.

Figure 1 Enter the Root Java Directory in the JLCA Wizard
Figure 1 Enter the Root Java Directory in the JLCA Wizard(Click the image for a smaller view)
Figure 1 Enter the Root Java Directory in the JLCA Wizard
Figure 1 Enter the Root Java Directory in the JLCA Wizard (Click the image for a larger view)

The JLCA then proceeds to convert the Java source code to C#. The process doesn’t take too long. Anecdotally, our code base with approximately 100 class files took less than 10 minutes for the conversion to complete—just enough time to get a cup of coffee. Of course, this number will vary for different projects and systems.

After the conversion process is finished, the JLCA creates an HTML report of errors and warnings. These items are also entered into the generated C# code as comments by the offending members. To help you, each item also contains a hyperlink for more information on resolving the problem.

Many of the warnings can be safely ignored. They simply note differences in behavior between Java and C#, such as the warning that states: "Type castings between primitive types may have different behavior." Still, you should look at each warning to ensure these different behaviors won’t affect your application.

When you first look at the conversion report, there may seem to be an overwhelming number of issues reported. In this case, there were 816 errors and 16 warnings. Most of the errors, however, could be categorized into one of three categories and addressed pretty easily. The three categories they fell into were:

  • Members for which there are simply no C# or Microsoft .NET Framework equivalents.
  • Popular third-party Java libraries that don’t have direct equivalents in the .NET Framework (such as Hibernate and log4j).
  • Items that deal with class loading or resource resolution.

It is also worth noting that the JLCA does not seem to attempt to resolve imported packages (or using namespace statements) that it can’t find. Instead, it just passes them through to the generated C# code. If you were to try to compile your new C# application, you would probably get quite a few more compiler errors than the conversion report indicates.

Not to worry, though. As stated previously, the bulk of these errors falls into the same recurring patterns and can be addressed en masse. I look at resolutions to these common errors in the following sections, and I cover the other necessary tasks to make your converted app a true C# application.

Locating Resources

The process of resource locating—specifically, finding and loading file resources—in Java differs significantly from resource locating in C#. Java uses a class loader to load and parse a class definition at run time. Part of the responsibility of a class loader is to manage the context of a class and provide environmental facilities for classes loaded by it. In Java, a special type of environment variable, called a classpath, is used by a class loader to locate resources. A classpath is similar to a path environment variable in that it defines where a class loader should look for other classes and resources. An application that wishes to load another class or resource can do so by telling the class loader the location of the file relative to the classpath.

A very common approach to resource resolution in Java is to use a special type of file called a properties file for configurable information, such as connection or host information, paths to other resources, localization strings, and credentials to use for authentication. A properties file contains name=value pairs, which are separated by a new line. This format is very similar to INI files, but without the sections.

Furthermore, resource locating is always performed using file system notations—whether they are actually located in the file system. This alleviates the developer’s burden of having to know how the application is being deployed. Other types of resources, such as images and binary files, are located and loaded in this same manner. Here’s an example of locating and using a properties file in Java:

InputStream is = this.getClass().getResourceAsStream(
    “/application.properties”);
Properties properties = new Properties();
properties.load(is);
String myValue = properties.getProperty(“myKey”);
On the contrary, resources in the .NET Framework can be deployed and loaded in two different ways: as an embedded, binary resource with an assembly or as a file in the local file system.

The appropriate technique for accessing a resource depends on its location. For example, if it is a file system resource, you might do something like the following:

string fileName = Path.Combine(Path.GetFullPath(
    @”..\config\”), “properties.xml”);
Stream fileStream = File.Open(fileName, FileMode.Open);
//Do something with the stream
If, however, the resource is embedded in an assembly, your approach will be more like this:

Assembly assembly = Assembly.GetExecutingAssembly();
Stream fileStream = assembly.GetManifestResourceStream(
    GetType(), “properties.xml”);
//Do something with the stream

My team’s application was written with the assumption that it would not have to know how the resource was deployed to be able to resolve. Since there are many resources being loaded throughout the application, it would have taken a lot of effort to analyze each resource being loaded, determine how the resource is deployed, and then modify the code to load it appropriately. So instead we created a utility class called ResourceLocator.

ResourceLocator was designed to approximate a Java class loader’s ability to resolve a resource based on its classpath. Since all of the calls to load resources were written with this approach, it seemed like the least intrusive method of conversion. All we would need to do after ResourceLocator was written would be to change calls from Class.getResourceAsStream to ResourceLocator.LocateResource. This could be done with a simple find and replace in Visual Studio.

Basically, ResourceLocator takes the name and relative path of the resource to find and then attempts to find it by walking through available assemblies and the local file system. There are also overloaded methods that provide more granular control, such as specifying the order in which areas should be searched and choosing to search only assemblies or the file system. The source for ResourceLocator is included in the code samples on the MSDN® Magazine Web site.

You might be thinking that it is very costly to look at all of the available locations to find a resource, and this is certainly true. But all of the resources located and loaded in the application were then cached. This means that each resource is only loaded once during the execution of the application, and thus this overhead is mitigated (though it does increase memory consumption, which could be an issue if there are a large number of resources being loaded). Therefore, we decided the trade-off was acceptable, considering how many code changes that it saved us. These types of changes can also be modified over time, starting with the simple solution to get the port up and running quickly, and then slowly but surely changing the implementation to better match design and implementation guidelines for .NET-based applications.

Handling Input/Output APIs

There are several differences between the I/O APIs in Java and those in .NET that need to be dealt with after conversion. One significant difference is that .NET I/O streams are bidirectional, whereas Java I/O streams are unidirectional. This means that in .NET programming, you can, in theory, read from and write to the same stream. In Java, however, you can only read from a stream or write to a stream, but not both to the same stream. This difference doesn’t pose much difficulty during the conversion since this is a widened difference and the .NET streams provide at least as much functionality as their Java counterparts.

If the safety of unidirectional streams needs to be maintained, then the .NET I/O readers and writers can be utilized. These will wrap an underlying stream to provide reading or writing functionality. The result will be I/O operations that are programmatically similar to Java I/O operations.

For our application, direct access to streams is sufficient. The JLCA properly converts I/O operations, so no changes are necessary for compilation. However, we did carefully review the converted code since there was a significant opportunity for logic errors in these low-level operations.

Logging

Logging concerns the ability to write messages to a destination at particular points of execution in the code, such as caught exceptions, areas in logic where debugging information might be useful, and configuration information being loaded. Java and the .NET Framework both provide powerful frameworks for logging information, but their designs and implementations are very different.

In Java, application logging is usually accomplished through the Apache Software Foundation (ASF) log4j framework or the Java Logging APIs distributed with recent versions of the Sun Microsystems Java Development Kit (JDK). The Java Logging API is very similar in execution to log4j and, therefore, the two frameworks can be applied interchangeably for the purposes of this discussion. For .NET-based applications, the Microsoft Enterprise Library provides a robust application block for logging, called the Logging Application Block (see the Microsoft Logging Application Block homepage).

The standard logging frameworks in both Java and .NET provide strong capabilities, with support for design-time configuration, a host of destinations for logs (such as databases, files, and e-mail recipients), and so on. Note, however, that the API designs are different and thus need some manual intervention on your part during conversion.

To better understand the differences in the APIs, consider the two code snippets shown in Figure 2. These demonstrate a common logging scenario performed in Java and in C#. The snippet using log4j creates a static instance of a Log object from the getLog factory method and assigns it to the MyClass category. The code then prints some information to the Log at a debug level. The C# snippet, which uses the Logging Application Block, creates a new instance of a LogEntry class. This represents an entry in the destination log. The code then assigns the log entry a priority of 2, sets the category as Debug, and provides a message. It is then written to the Logger class.

Figure 2 Logging in Java and in C# (Close Figure 2)

Java with log4j
private static final Log log = Logger.getLog(MyClass.class);
...
//Somewhere else in the class
log.debug(“Printing some debug information.”);
C# with the Logging Application Block
Logger.Write(”Printing some debug information.”, “Debug”);

It is important to note that logging in both platforms is controlled by external configuration. Information such as logging destinations, selective filtering of log messages, and formatting of log entries is contained in this configuration. I’ve intentionally left the configuration out of this example, as it isn’t pertinent to the discussion.

Looking at these two examples, you can see that they perform very similar functions, but they are achieved in different ways. The log4j example uses the category to determine if a message is logged (based on its level) and where it is logged to. The Logging Application Block, meanwhile, uses a combination of filters for priorities and categories to determine what is logged.

The Logger class provides several overloaded Write methods, each with varying levels of flexibility (including an overload that allows you to provide a LogEntry instance which has lots of knobs and controls for tweaking how the information is logged). However, given the simple overload that’s used in Figure 2, we were able to accomplish the bulk of the conversion process with searching and replacing, in combination with regular expressions.

Logging is a very resource-intensive process and care needs to be taken so that it doesn’t affect your application’s performance. In the end, it only took a few hours to search and replace the logging invocations in the application.

Collections

Both the .NET Framework and Java offer a strong collections API. Both are very extensible and cover most of the scenarios you might encounter when working with collections. Let’s look closely at two frequently used types of collections: lists and dictionaries.

Lists are collections that can be accessed by an index. They are ordered collections and can be thought of as a kind of one-dimensional array. Dictionaries, on the other hand, are collections of name and value pairs. The name is the key used to access values in the collection. Note that there is not necessarily guaranteed ordering in dictionaries.

Figure 3 lists the equivalent C# and Java implementations for list and dictionary. The JLCA does a very good job of converting these Java collection classes to their .NET equivalents and there isn’t a lot left to be done after the conversion. At that point, warnings are generated stating that behaviors for these common collections are different on the two platforms, so the conversion should be verified. However, the vast majority of the conversion process should be successful and valid.

Figure 3 Equivalent Collection Classes in .NET and Java (Close Figure 3)

 | .NET | Java
List interface | IList, IList<T> | List
Common list classes | ArrayList, List<T> | ArrayList, Vector
Dictionary interface | IDictionary, IDictionary<TKey,TValue> | Map
Common dictionary classes | Hashtable, Dictionary<TKey,TValue> | HashMap, HashTable

That said, we encountered a problem with converting collections where we had used specialized collections in the original Java application. An example of this issue is the use of the Java LinkedHashSet class. According to the Java API documentation, the LinkedHashSet class ensures consistent ordering of entries in the HashSet (which is a type of dictionary), without the overhead imposed by other ordered dictionaries. While the definition of a LinkedHashSet is straightforward, the intent of its usage in the application was unclear. (The code was written by someone no longer on the team, and there wasn’t documentation as to why it was used.) Furthermore, the context of its usage didn’t give us anything, so it was unclear whether this class was used to fix a problem or perhaps for some other reason.

In looking through the code, we found no justification for its usage, and thus we had three options: assume it wasn’t actually needed in the original application, write our own implementation for our .NET app, or pick the closest appropriate collection in the .NET Framework. We assumed the specialized implementation of a LinkedHashSet was unnecessary, since there was no indication that ordered name value pairs were used anywhere else. As such, we substituted the basic .NET Hashtable class, and we verified the correct behavior with our existing unit and integration tests. However, if we had discovered performance or functional issues, we could have substituted the SortedDictionary<TKey, TValue> class in the .NET Framework 2.0 which represents a collection of key/value pairs that are sorted on the key; internally its implementation uses a set based on a red-black tree data structure.

In our project, there were only four uses of specialized Java collections classes, and they all presented similar circumstances. The additional functionality they provided was unused, and using the more generic .NET counterparts accomplished the task at hand.

I should note that our Java application was written using Java version 1.4. Generic collections, which allow strong typing of collection members, were not introduced to Java until version 1.5. Therefore, we did not have to delve into converting generics. Presumably converting collections in Java version 1.5 will thus add another layer of complexity to the conversion process since not only do the collections need to be converted, but so do their typed entries.

Filters and HTTP Handlers

A filter is a common paradigm used in J2EE Web applications to selectively intercept requests and responses to perform pre- and post-processing. Some common usages for filters are logging, usage auditing, and security.

Java filters implement the Filter interface, which defines particular lifecycle events. Filters are invoked by the application server by using a URL mapping to map Filter classes to full or partial URLs. When a match is made, the application server implements the lifecycle events of the mapped filter, passing a handle to the request and the response.

ASP.NET provides similar functionality in the form of an interface called IHttpHandler. Both filters and HTTP handlers have very simple interfaces, but are very powerful in what they can do. In our application, we had two different types of filters—one that used GZip compression to compress responses and one that intercepted requests to see if the request was from a known user agent.

The compression filter was implemented in the Java application to improve performance. It took every response stream, checked to see if the client supported GZip compression, and if so, compressed the response. Typically, this is unnecessary, as most modern HTTP servers provide this functionality without custom code. However, our application was contained as a Servlet application, which doesn’t necessitate an HTTP server, so the compression functionality was a value-added module.

The second filter, which rejected requests based on a user agent header value, is of more interest. Many of our clients wished to implement a level of authentication that compared the HTTP user agent header against a list of allowed agents. If the incoming request’s user agent value wasn’t an allowed user agent, the request should be rejected (usually by returning an HTTP unauthorized return value).

This type of request/response filter can be accomplished many ways. Most solutions, though, require a lot of injection of code or attributes. Luckily, J2EE and .NET both provide very simple mechanisms for intercepting, modifying, and adjusting requests and responses at an application level. Additionally, HTTP interceptors like these are not code-pervasive, meaning they are not lines of code in every class. Rather, they are separate classes that are managed and injected via the application server, so it is much easier to modify the functionality rather than perform a global search and replace.

The JLCA 3.0 does provide helper classes that aid in the migration of filters from Java to ASP.NET applications. Figure 4 shows a sample filter in Java that times the server processing, and it also shows how the JLCA attempts to convert this filter for ASP.NET. While in an ideal world the way to port to a .NET implementation probably requires rewriting the filter from scratch as an HTTP handler, JLCA provides the support class SupportClass.ServetFilter which gets you most of the way through a reimplementation. ServletFilter emulates the life cycle Java provides. While it doesn’t solve all problems, it can ease a port implementation.

Figure 4 JLCA Conversion Using ServletFilter (Close Figure 4)

Java
import javax.servlet.*;
import java.io.*;

public final class TimerFilter implements Filter
{

    public void doFilter(ServletRequest request,
                         ServletResponse response,
                         FilterChain chain)
        throws IOException, ServletException
    {

        long startTime = System.currentTimeMillis();
        chain.doFilter(request, response);
        long stopTime = System.currentTimeMillis();
        System.out.println(“Time to execute request: “ +
            (stopTime - startTime) + “ milliseconds”);
    }

    public void destroy() {}

    public void init(FilterConfig fc) {}
}
C#
using System;
using System.Web;

// UPGRADE_TODO: Verify list of registered servlet filters.
public sealed class TimerFilter : SupportClass.ServletFilter
{
    public override void DoFilter(HttpRequest request,
        HttpResponse response, SupportClass.ServletFilterChain chain)
    {

        long startTime =
            (System.DateTime.Now.Ticks - 621355968000000000) / 10000;
        chain.doFilter(request, response);
        long stopTime =
            (System.DateTime.Now.Ticks - 621355968000000000) / 10000;
        Console.Out.WriteLine(“Time to execute request: “ +
            (stopTime - startTime) + “ milliseconds”);
      }

      public void  destroy() {}

      // UPGRADE_ISSUE: Interface ‘javax.servlet.FilterConfig’
      // was not converted.
      public void  init() {}
}

That said, our team took the path of manually converting the Java filter functionality to an HTTP handler. In order to walk you through this, I should compare the two interfaces. A J2EE filter has three methods: init, destroy, and doFilter. The init and destroy methods are lifecycle methods that have some significance, but are outside of the scope of this discussion. The doFilter method, however, does the lion’s share of the work in a filter.

In our scenario, the doFilter method gets the user agent header variable from the incoming request and inspects it against a list of configurable, known user agents. The most difficult thing in conversion is the difference in objects passed as arguments to the actionable method.

In the Java Filter.doFilter method, there are three arguments passed: a request object, a response object, and a filter chain object. Conversely, the IHttpHandler class’s ProcessRequest method has a single argument: an HttpContext variable. The HttpContext variable references an HttpRequest and an HttpResponse object, and most of the same functionality can be accomplished by accessing the HttpContext’s members.

The Java FilterChain object is interesting. This represents a chain of filters in an HTTP request. A filter may pass on the request to the next filter in the chain, allowing responsibilities to be delegated in a sequential fashion. The FilterChain is not used very often, however, but when it is, similar behaviors can be achieved with IHttpModule implementations.

The FilterChain aside, converting Java filters to .NET IHttpHandlers is straight-forward. First, an implementation of IHttpHandler needs to be created. Then the logic within the filter’s doFilter method needs to be reconstructed into the implementation’s ProcessRequest method, utilizing the HttpContext’s members instead of the passed request and response objects. Finally, the IsReusable method needs to be defined in the implementation; for simplicity, it can just return false (another case where later on you can write more code to determine whether the same handler instance can, in fact, be reused by later requests for performance gains).

Source Tree and Naming Conventions

While the Java and C# languages are similar, there are differences in lexicography, source tree layout, and naming conventions. Here, I’d like to detail some of these differences.

Java packages (which are the equivalent of namespaces in C#) follow a convention of a reversed domain name, followed by application or module name, followed by functionality. These packages are usually deeply nested (typically four or five levels deep). C# namespaces, on the contrary, are usually grouped by functionally descriptive names and are typically shallow (usually one to four levels). In addition, Java packages are usually lower or camel-cased, while C# namespaces are usually Pascal-cased. Likewise, Java methods are usually camel-cased, while C# methods and properties are usually Pascal-cased.

C# interfaces usually begin with a capital I, denoting an interface (this is purely a convention and is in no way required for correct functionality). Old Java convention said that interface names should end with "able", denoting the ability to do something. This convention is rarely followed any longer, and now there usually isn’t any distinction between an interface’s name and a class’s name.

C# uses properties to implement accessing and mutating private members. Properties are metadata wrappers for get and set accessor methods. But Java doesn’t have properties. Accessors and mutators for a private member are typically implemented as methods called getters or setters. In other words, an accessor for a private field called "name" would be getName.

Finally, Java classes must be located in a directory that matches its declared package, relative to the root of the source files (or classpath). C# does not impose this restriction.

When to Refactor for Conventions

Although a converted C# application will compile and run without addressing these differences, it is always best practice to follow conventions. Choosing when to refactor your converted code to conform to conventions is a difficult decision. However, there are two factors that suggest this should be done sooner rather than later.

Since refactoring code to conform to conventions is not absolutely critical for the application to function, not to mention the fact that it is tedious at times, it may be deemed as unnecessary later in the project. A variety of things can cause it to become a low-priority task and it will therefore run the risk of never getting done. However, if there is a team of developers working on the project, refactoring code to make it appear familiar can help to increase the team’s efficiency and productivity. Don’t underestimate the importance of these tasks. Refactoring of this sort is as important as the other tasks involved in the conversion process to ensure a successful result.

Directory Layout and Namespaces

Naming and directory conventions are a subjective process, but there are general guidelines. For our project, assume that there is a class for custom XML parsing, and it is located in the com.mycompany.myapplication.xml.util namespace (and directory, after conversion). C# convention suggests that this class should be located in the Xml.Util namespace. Before refactoring, our directory tree would look similar to Figure 5. Dragging and dropping files in Visual Studio will allow you to accomplish the physical movement of the file, so that your directory tree now looks like Figure 6.

Figure 5 Java Source Tree
Figure 5 Java Source Tree
Figure 6 C# Source Tree
Figure 6 C# Source Tree

However, C# does not dictate that the directory location of a file match its declared namespace. Therefore, the namespace of the class is not updated to match the file system location. There is no automatic way of moving multiple classes to different namespaces in Visual Studio—the best way that I’ve found to do this is to perform a Find and Replace for the whole solution, as shown in Figure 7. This, of course, assumes that you’re moving all of the classes in one namespace to the same destination.

Figure 7 Find and Replace Namespace Declarations
Figure 7 Find and Replace Namespace Declarations(Click the image for a smaller view)
Figure 7 Find and Replace Namespace Declarations
Figure 7 Find and Replace Namespace Declarations (Click the image for a larger view)

Properties

The construction of properties in C# is significantly different from Java. Properties can be thought of as public fields that store the state of a class or, in UML terms, attributes of class. But Java does not have a property construct—instead, properties are represented in methods called getters and setters.

Consider a class that has an instance variable called "name". You don’t want to make this variable public because all control would be lost to modification of this variable. In Java, the standard method of providing access to this variable is through getters and setters, named so because the convention is to prepend a "get" or "set" to the variable name as the method name. Thus, if the name variable is a string, the Java getter and setter might look like this:

public String getName() {
    return this.name;
}

protected void setName(String name) {
    this.name = name;
}

C# provides property constructs to accomplish the same thing. Although they can be used for application logic, their intent is to provide protected access to private implementation details, as described above. Therefore, a C# implementation of the same access would look like this:

public String Name
{
    get {return this.name;}
    protected set {this.name = value;}
}

While C# properties accomplish the same objective, I find them to be clearer—in Java, methods that don’t modify a class’s state may begin with a get or set, leading to ambiguity. Therefore, I recommend that you refactor Java getters and setters to be C# properties.

That said, there isn’t an easy way to convert Java getters and setters to C# properties. A regular expression search and replace would be extremely complex, and the JLCA simply migrates the Java getter and setter methods as-is since there is no way to tell if the method is modifying a class’s state or performing some other functionality. Our approach to this problem was to use a more practical solution. Visual Studio provides a wizard for encapsulating private members with properties. This generates the desired properties without deleting the converted Java getters and setters.

As our team worked on code for other reasons, they also generated the C# properties in Visual Studio and deleted the corresponding getter and setter methods. After this step, developers would use the Visual Studio 2005 Find References functionality which provides a list of all call sites for a particular method. This gave them the references to the old getter and setter methods, so they could easily change the references to the new properties. This wasn’t an elegant solution, but it worked surprisingly well.

I should stress that this was considered a very important step in the porting process since properties are ingrained core aspect of C# and .NET and our goal was to create a C# application. (Remember, this application would eventually need to be supported by another team, which was expecting a C# application.)

Pascal-Casing Method Names

As I’ve already noted, Java method names are conventionally camel-cased. In other words, these names start with a lower-case letter, and each subsequent word boundary is upper-cased. C# convention says that methods and other members are Pascal-cased. Pascal-casing is similar to camel-casing, except that the first letter in the name is also upper case.

What does this mean to you? You should probably rename all of your methods to start with an upper-case letter instead of a lower-case letter. For example, the Java method

public void getResponseCode()
should be refactored as the C# method:

public String GetResponseCode()

As with converting getters and setters into properties, there is no easy way to go through all of the converted code and update members to conform to C# naming convention. However, we considered this a very important task since the members that were converted did not resemble C# members and, again, our team and our customer wanted this to be a proper C# application.

In theory, a code converter could be written to perform this task, and we considered doing this ourselves. However, we decided to take the same approach as we did with properties, updating member names piecemeal. We had several reasons for taking the manual approach.

For starters, our developers were already analyzing all of the code to update other elements, such as the getters and setters. The process could be incremental, since not changing the names of members did not affect the ability to compile or function. And, if for some reason one or two members were missed, the application wouldn’t break. The process didn’t take as much time as you might think—the main problem was that it was a tedious task.

Visual Studio 2005 has built-in refactoring support. Part of the refactoring support is safely renaming members, whereby a member can be renamed, and all references to that member are updated. It takes a little bit of time to go through all members you want to update, but it is an effective solution.

Conclusion

I have described to you a real case of converting a Java Web application to ASP.NET. Much of the heavy lifting was done by the JLCA, but there were several items that needed manual intervention on our part. Nevertheless, the JLCA is a powerful tool and allowed our application to be quickly ported to a production-quality .NET application.

My intent here is to demonstrate the viability of converting a Java application to .NET and to point out some of the issues that typically need to addressed beyond the JLCA’s capabilities. Every port of an application will present unique problems and I couldn’t address all of them in this article. However, I have offered some techniques and approaches that should help with any problems you might encounter.

Brian Jimerson is a senior technical architect for Avantia, Inc. (www.avantia-inc.com), a custom solutions provider based in Cleveland, Ohio. Brian has recently been helping clients define and implement infrastructure and architecture solutions for both Java and the .NET Framework.


BizTalk Server: 8 Tips And Tricks For Better BizTalk Programming

BizTalk Server

8 Tips And Tricks For Better BizTalk Programming

Marty Wasznicky and Scott Zimmerman
This article discusses:
  • Multi-Part Messages
  • Direct-Bound Ports
  • Creating Web services
  • Debugging XSLT
This article uses the following technologies:
BizTalk Server 2006, Visual Studio
Contents
1. Always Use Multi-Part Message Types
2. Always Try to Design Orchestrations with Direct-Bound Ports
3. Always Use Separate Internal and External Schemas
4. Never Expose Your Internal Schemas Directly in WSDL
5. Always Optimize the BizTalk Registry for Web Services
6. Always Set the Assembly Key File with a Relative Path
7. Never Overlook Free Sample Code
8. Debug XSLT in Visual Studio
Conclusion

O
nce, just half a day before a proof-of-concept with a major client, Marty was handed a complex BizTalk® solution to fix. The main component was an orchestration that integrated several back-end systems, making multiple calls out to each. The Orchestration Designer screen he was staring at consisted of mostly black lines connecting dozens of Receive and Send shapes to over 40 inbound and outbound ports—a nearly impossible design to debug. His solution: start over using Multi-Part Message Types and Direct-Bound Ports (Tips #1 and #2 below), and perform unit testing along the way. And guess what? He made it!

Marty’s success can be at least partially attributed to the design of BizTalk Server, which is built to meet the special challenges inherent in programming connected systems without, in some cases, even having to write code. But don’t let the pretty drag-and-drop flowcharts or the apparent lack of code fool you. BizTalk is a broad and powerful product that takes years to master. Along life’s path to becoming a BizTalk guru, you’ll encounter many situations that call for writing code based on the Microsoft® .NET Framework to supplement the sophisticated built-in message processing in BizTalk.

Until recently, there have been few opportunities for BizTalk newbies to accelerate their learning with secrets from the masters. (See the sidebar "Learning BizTalk Server 2006 Programming" for a few resources.) I got my first intensive introduction to BizTalk while attending one of Marty’s week-long BizTalk bootcamp classes, during which he continuously exhorted us to design systems properly and to test them under realistic load conditions—with PerfMon—before calling it a day. From those talks, we extracted a bucket of tips that can help you become a more effective developer. The purpose of this article is to share those useful tips with you, so let’s get started.

1. Always Use Multi-Part Message Types

BizTalk is all about messages, which, by the way, are not only for e-mail. Documents, InfoPath forms, large binary files, SQL records, flat files, and anything XML can be processed as messages, and you get the benefit of asynchronous communication. Once you get used to it, message-oriented programming is as cool and useful a paradigm as object-oriented programming.

Messages in BizTalk are data, and each message must be of a selected message type. The most common message type in BizTalk is a schema, which is to say the message is based on an .XSD file that specifies the record and field structure in the message. Please see msdn2.microsoft.com/en-us/library/ms942182.aspx for a full discussion of schemas and maps in BizTalk.

Some day when programming in BizTalk, you’ll surely decide you want to change the schema on which a message is based. The problem is that if you have already selected that message for a Send or Receive shape and connected it to an orchestration port, you’ll get this error when you try to change the Message Type:

Property value is not valid: One or more Send or Receive actions are
 connected to Ports and are using this Message. Please disconnect the
 actions before changing the Message Type.

Before jumping in to do as the error message suggests, let’s think for a moment about what that entails. First, you’ll have to inspect every Receive and Send shape to determine if it uses a Message variable that’s associated (has its Message Type set) with the schema you want to change. Today, there is no facility in BizTalk to provide an audit of this or to generate a dependency map; it just wouldn’t be a good practice to create a single orchestration with so many Receive/Send shapes that you would need such a feature. In the horrific orchestration mentioned earlier, most of the Receive and Send shapes were using different Message variables, but all were pointing to the same schema definition (all the same Message Type)!

Second, once you find all the Receive/Send shapes, you’ll have to delete their Port connections. Third, you’ll have to change the Message variable so that the Message Type property is set to the new schema and then reassociate the Message variable with each Receive/Send shape.

Fourth (and you thought you were done), you’ll have to find all the Port Types that are associated with the Ports you disconnected from Receive/Send shapes and reset their Operation’s Message Type properties as well. You see, when you originally associate a Receive or Send shape with a Port, BizTalk automatically sets the Message Type property in the Port type to match the Message Type of the Message variable for that Receive/Send shape. When you delete the connection between the Port and Receive/Send shape, BizTalk loses the Message Type of the Port.

Fortunately, there’s a better way. An axiom in computer science says that many problems can be solved by adding a level of indirection. That’s true once again in this case; all we need to do is use a Multi-Part Message Type to wrap the underlying schema. This is slightly more work initially, but it’s much more flexible and will save you time in the long run. Here’s how to do it.

When you right-click Messages in the Orchestration View tab to create a new message, you have four choices for the Message Type property (see the lower-right corner in Figure 1). It’s common to choose a schema for the message type, but for this tip let’s try something radically different. Click the + sign to expand Multi-part Message Types and then choose Create New Multi-part Message Type. Name your multi-part message type then click the + sign to expand it so you can see its MessagePart_1 member (that’s the BizTalk suggested name for the first part of your new multi-part message type). Now change the Identifier property from "MessagePart_1" to something better, like "Body" (just don’t use lower-case "body" as it’s a reserved word). Don’t forget to set the Message Body Part property to True (so it can act just like a regular message).

Figure 1 Setting the Message Type Property in Orchestration View
Figure 1 Setting the Message Type Property in Orchestration View (Click the image for a smaller view)
Figure 1 Setting the Message Type Property in Orchestration View
Figure 1 Setting the Message Type Property in Orchestration View (Click the image for a larger view)

That’s the end of the extra work. Finally, set the Type property for your new message part (such as Body) to be the same schema you would have used for a regular schema-based message type. Now your logical port and Receive/Send shape are using your multi-part message type, and you can easily change the underlying schema in the future without having to disconnect the port from the receive shape.

2. Always Try to Design Orchestrations with Direct-Bound Ports

Take a look at Figure 2, which shows the choices you have when configuring a port in Orchestration Designer. As you work through the Port Configuration Wizard, you are moving from top to bottom in this diagram. Notice the three Direct options in the diagram. Though the BizTalk UI offers "Direct" as the port type, we’ll use the vernacular here and refer to them as "Direct-Bound."

Figure 2 Port Options in the Orchestration Designer Port Configuration Wizard
Figure 2 Port Options in the Orchestration Designer Port Configuration Wizard(Click the image for a smaller view)
Figure 2 Port Options in the Orchestration Designer Port Configuration Wizard
Figure 2 Port Options in the Orchestration Designer Port Configuration Wizard (Click the image for a larger view)

The key to understanding port terminology in BizTalk is to understand the notions of logical ports (also called orchestration ports) and physical ports. To oversimplify, it’s the difference between creating ports in Orchestration Designer (logical), and using BizTalk Explorer or BizTalk Administration Console (physical).

When a developer creates a Specify Later port in Orchestration Designer, he’s configuring a logical port, leaving the corresponding physical port properties to be configured later by the BizTalk administrator. Giving the administrator the flexibility to configure the physical port in the production environment is a key reason why Specify Later is the most frequently used option.

With the Specify Now option, the developer does it all, both logical and physical configuration, and the solution is hardwired. This is not recommended, except for quick experiments.

With Dynamic ports, the final configuration of the physical port properties is again done by the administrator. However, in this case the developer can also write code to set certain outbound message properties at run time. This is useful for SMTP or HTTP, where the orchestration logic includes an Expression shape with a statement like this:

DynamicSendPort(Microsoft.XLANGs.BaseTypes.Address)=
    “mailto:john@contoso.com”;

While there are many options in Figure 2, the figure only shows the options in Orchestration Designer. Where do Static ports fit in? That term comes up when you create a physical port in BizTalk Explorer or in BizTalk Administration Console. If the developer creates a Specify Later or Dynamic port in Orchestration Designer (a logical port), then the administrator will choose among the four options shown in Figure 3 when creating the corresponding physical port in BizTalk Explorer or BizTalk Administration Console.

Figure 3 Physical and Logical Port Types (Close Figure 3)

Create This Physical Port Type | For This Logical Port Type
Static One-Way | One-Way / Receive or Send / Specify Later
Static Solicit-Response | Request-Response / Send Port / Specify Later
Dynamic One-Way | One-Way / Send Port / Dynamic
Dynamic Solicit-Response | Request-Response / Send Port / Dynamic
Learning BizTalk Server 2006 Programming

If you’re just getting started learning BizTalk programming, you’ll find it very helpful if you check out the Service Station column by Aaron Skonnard in the November 2005 issue of MSDN Magazine as well as the December 2005 column. At the BizTalk Server Developer Center you’ll find dozens of free webcasts.

At the intermediate-level, read BizTalk 2006 Recipes: A Problem-Solution Approach by Mark Beckner, published by Apress; and in the December 2006 issue of MSDN Magazine, Alex Starykh dives into adapter programming topics with BizTalk Adapters.

For intermediate to advanced material, see Pro BizTalk 2006 by George Dunphy (coauthored by Marty) from Apress and Professional BizTalk Server 2006 by Darren Jefford from Wrox, which should be available by the time you read this. Don’t let the version number in the title disappoint; Microsoft BizTalk Server 2004 by Scott Woodgate, published by Sams, is still substantially relevant. These books include several examples of using C# code to take BizTalk to the next level.

To earn your degree in integration systems philosophy and design, you need to read Enterprise Integration Patterns by Gregor Hohpe, published by Addison-Wesley. At first glance, you might think this won’t help you learn BizTalk, but it’s interesting to learn how BizTalk Server 2006 implicitly implements several of these patterns. For more information on Tip #1, these helpful blogs by Kurt Guenther and Charles Young are good reading:

  • www.infusionblogs.com/blogs/894.aspx
  • geekswithblogs.net/cyoung/articles/4634.aspx

For Tip #4, check out these expert blogs by Brian Loesgen and Kevin Lam:

  • geekswithblogs.net/bloesgen/archive/2005/10/27/58364.aspx
  • blogs.msdn.com/578572.aspx

For more BizTalk tips, see Alan Smith’s “seven habits” article and Scott Colestock’s handy article on BizTalk naming conventions:

  • geekswithblogs.com/asmith/articles/17333.aspx
  • traceofthought.net/misc/BizTalk%20Naming%20Conventions.htm

Just remember, Dynamic physical ports are for Dynamic logical ports (easy enough), but Static physical ports are for Specify Later logical ports. Also, note the terminology mismatch between Solicit-Response and Request-Response. Oh well.

With those preliminaries out of the way, we can finally talk about the advantages of Direct-Bound ports. Direct-Bound ports are great for sending a message from BizTalk to BizTalk. By the way, this is why it wouldn’t make sense for BizTalk to have a Direct option for Web ports. Web ports are for BizTalk to the Web, not BizTalk to BizTalk.

Direct-Bound ports are self-configured, without sacrificing flexibility or performance. A handy benefit of Direct-Bound ports is that neither the developer nor the administrator needs to create corresponding physical ports in BizTalk Explorer or BizTalk Administration Console. So Direct-Bound ports yield orchestrations that are more self-contained, and therefore more reusable and easier to redeploy independently.

You’ve got three Direct-Bound port types: Message Box Routing, Self-Correlating, and Orchestration-to-Orchestration (also called Partner Ports). The actual text of the three radio buttons in Orchestration Designer is much wordier, but when you have a look you’ll easily be able to map our shortened names to the UI.

Of these three port types, the Message Box variety is the one that comes up most often. It’s the one that permits your orchestration to be completely independent of other orchestrations, even though you might intend for it to communicate with itself. Ah yes, that does imply the possibility of an infinite loop. We’ll get to that in just a second.

When a Receive shape with the Activate property set to True is configured for Message Box Routing, it simply means that a new instance of the orchestration will fire up when a message arrives in the Message Box that matches the given subscription filter, regardless of where the message came from or who sent it. This could help you build a reusable orchestration, like a generic function in a utility library. Marty built an elegant message-based exception handling system using this technique, but that’s another story.

Suppose you’d like to start your orchestration in two ways: first, if the operator drops a "kickoff" message in a folder on a Static One-way FILE (physical) port; or second, if the orchestration sends itself a kickoff message to run another instance. Your first thought might be to use a Listen shape (like an OR operator for orchestrations) with two Receive shapes underneath it (a Specify Later for the FILE port, and a Direct for the Message Box). But the neat thing about Direct-bound Message Box ports is that, again, they don’t care where the message comes from. So you only need one Receive shape to start this orchestration!

Self-Correlating Direct-Bound ports are usually used when you want to send messages from one orchestration to the next, asynchronously. Say you have two orchestrations, A and B. Just go into Orchestration B, and create a logical Send Port under the Parameters section and set it to use the Port Type created in Orchestration A. Now, any messages you send from that port in Orchestration B will be immediately routed to the logical Receive port you created in Orchestration A. We’re basically leveraging the publish/subscribe nature of BizTalk to not only forward messages, but also stamp them with hidden unique correlation tokens, hence "Self-Correlating." In this case, the orchestrations do have some knowledge of each other because you’re sharing a port type between them. You can reduce the dependency somewhat by setting the Message Type of the Operation on the Port Type to System.Xml.XmlDocument. This is a great way to have orchestrations run asynchronously, while having some kind of broker be able to receive Acks or Nacks from other orchestrations.

Orchestration-to-Orchestration Direct-Bound (Partner) ports are for sending messages asynchronously. You’re adding a broader restriction to the subscription than you have with Message Box Direct-Bound ports. In this case, you’re saying that you’ll accept any message from the partner orchestration. One orchestration specifies the Direct-Bound port, and the other orchestration (the partner) specifies itself. Either the sender or the receiver can be the one that refers to the Direct-Bound port. The key here is that you’ll be wiring up two known solutions to communicate, whereas with Message Box Direct-Bound ports the sender and receiver are more loosely coupled.

Now for the fun part. A common pitfall with Direct-Bound ports, particularly the Message Box variety, is creating an infinite loop. Imagine a simple orchestration consisting of just two shapes, an Activate=True Receive shape (Direct-Bound, of course) and a Send shape that merely forwards the message to a FILE port. When this orchestration sends the message out, where does it go? As always, first to the Message Box. Whenever a message arrives in the Message Box, BizTalk searches for any subscriptions that match. And how will this message differ from the message that activated the orchestration in the first place? It won’t, so BizTalk will gladly fire off another instance of your orchestration to process it, and so on, until you run out of memory. One way to fix this is to copy the inbound message to a newly constructed outbound message and change the value of at least one promoted property so the subscription filter on your Receive shape will fail to match the new message. Run Administration Console and go to Group Hub | New Query | Open Query. The next step is to open BTSSubscriptionViewer.btq in your BizTalk\SDK\Utilities folder. You’ll see that BizTalk automatically creates a predicate when you set the message type for the Receive shape. In Orchestration Designer, add a predicate to the Filter Expression property so the full subscription will look like this:

BTS.MessageType == “http://MyInternalSchemas.MyProject#MyRootNode”
AND inbound_message(status) != 1

Or if you prefer not to change the schema to add a custom field like that, you can also add a clause to the subscription filter on the Receive shape to test that the BTS.Operation name is not the same as the one in your orchestration.

Remember when we said BizTalk is all about messages? Well, scratch that. It’s really all about subscriptions. Nothing happens to a message in BizTalk without a subscription on a Send port or an Orchestration.

3. Always Use Separate Internal and External Schemas

You should always transform messages you receive into your own canonical schema—regardless of how simple the schema is or who the source is.

For a small amount of processing, this buys you a lot of flexibility in case (or more appropriately, when) the sender changes the schema. You don’t want your orchestration logic to depend on any properties or fields in a schema that is controlled by someone else. If the sending party changes its schema, you only need to change your map, not your orchestration. It’s the indirection axiom again.

And by following the best practice of creating separate Visual Studio projects for maps, orchestrations, internal schemas, and external schemas, it’s easier to redeploy your updated solution. This becomes really crucial for Web services generated by the BizTalk Web Services Publishing Wizard.

If you’re lazy, you might be able to get by without inventing your own schema with new field names. Just copy the inbound schema you’ve been given and map all the fields from the source schema as is, but at least save the copy into your internal schemas project and call it your own.

This can also reduce the number of maps you need. Suppose you need to map three types of inbound messages to three types of outbound messages (perhaps not today, but maybe in the future). Applying this technique, you create a map to your canonical schema for each inbound schema and then a map from your canonical schema to each outbound schema. That’s only six maps to build and maintain, not nine.

Don’t Do It

BizTalk Server 2006 is a large application and fortunately it often provides several ways to accomplish things. But some are traps that most old-hands have learned to steer clear of for various reasons:

  1. Avoid BizTalk Explorer in BizTalk Server 2006. This feature was vital in BizTalk Server 2004 so Microsoft decided not to pull it from BizTalk Server 2006 even though the new application packaging feature and the redesigned BizTalk Administration Console made it redundant. Unfortunately, there are certain cases where using it can confuse things.
  2. Never click Deploy at the Project level in Visual Studio® 2005 Solution Explorer. It’s OK to build projects independently within a solution, but you should only click Deploy at the Solution level. The reason: Visual Studio tries to automatically keep track of dependencies, but separately deploying assemblies to BizTalk can cause it to lose track.
  3. Don’t expect that you can copy the schema files (.XSD) from one project to another without carefully editing the namespace attributes to match up. If you don’t, you’re due for a headache.
  4. BizTalk pros never use Quick Promote. By the time you take the steps necessary to correct the types it gives you, assuming you don’t forget to, you could just have quickly explicitly created them yourself.
  5. Don’t put maps in Orchestrations, unless you need to map multiple incoming messages into one message, or generate a new message using the modified (mapped) contents of an existing message as the base. For simpler deployment, it’s better to put your maps on Receive and Send ports. If your business partner revises their schema, or if you add a new partner that requires a new map, you don’t want to be forced to update both your schema and your orchestration.

4. Never Expose Your Internal Schemas Directly in WSDL

This means you should never click the first radio button in the BizTalk Web Services Publishing Wizard; always click the second button instead (see Figure 4). The first radio button is designed for publishing orchestrations and the second is designed for pure messaging solutions, but it’s a good idea to use the second option for orchestrations too.

Figure 4 Publish Your External Schema, Not Your Orchestration
Figure 4 Publish Your External Schema, Not Your Orchestration(Click the image for a smaller view)
Figure 4 Publish Your External Schema, Not Your Orchestration
Figure 4 Publish Your External Schema, Not Your Orchestration (Click the image for a larger view)

Yes, this is more complex, so we know you’re expecting a good reason why you’d want to do this. In a word, interface. Or if you prefer something more verbose, loose coupling! This buys you more freedom to change your orchestration without breaking the caller. Think of this as a specialization of the concept in Tip #3.

Let’s talk about how you can do this. Here’s one thing to remember about the wizard: when you get to Step 2, the Web Service dialog (see Figure 5), you need to right-click on practically everything. That’s how you’ll name the Web service and its methods that you’re creating.

Figure 5 Edit Your WSDL in the Web Service Dialog
Figure 5 Edit Your WSDL in the Web Service Dialog

You need to compile your schemas project in BizTalk before you get to this step in the Web Services Publishing Wizard, so you’ll be able to browse to the assembly. Actually, as you’ll see below, it makes sense to go ahead and deploy your solution before you run the wizard. One more aside; When you publish an orchestration as a Web service, you must change the Type Modifier property on your orchestration port type from Internal to Public before you build and deploy. If you didn’t do it when you first created the port, it takes some real digging to find this (highlight the port type in the Types pane in Orchestration View—not by clicking on the port on the Port Surface, which displays a different set of properties). But another benefit of publishing schemas as Web services is that you won’t have to do that anymore!

OK, let’s get back to the wizard, specifically Figure 5. You’ll need to right-click the Request and Response nodes to choose the schema on which the web messages are based. Orchestrations you publish will typically have a logical Receive shape and a Send shape for communicating with the caller of the Web service. Those shapes will have associated messages, and those messages will have a schema type (a multi-part message type is not required, just recommended). So you need to tell the wizard to use your external-facing schema (remember Tip #3). Then you’ll use maps in the Receive port (inbound and outbound maps) to transform the Web service request and response schemas to the internal schemas expected by your orchestration. When the client adds a reference to your Web service, he’ll get a proxy class that includes a type for your external-facing schema. The client will use that type to create and populate a request message to send to your Web service. And he’ll snag the response from your Web service into an object that is based on the external response schema you specify in Figure 5.

You’ll also have to remember to choose the XML Pipeline in the Receive location, otherwise BizTalk won’t create the BTS Message Context properties, one of which is the Message Type of the incoming Web service message. BizTalk knows the Message Type because it parsed it via the XML Disassembler in the pipeline and promoted the property, and it uses that type to execute the map to transform the request into the type the orchestration is expecting.

What’s really important is that the operation name you select in the wizard must exactly match the name of the operation you specify in the logical port in the orchestration. Simply accepting the defaults for everything won’t work because the default operation name in Orchestration Designer is "Operation_1", while the default name in the wizard is "WebMethod1" (see Figure 5). It’s a good practice to set them to a meaningful name in verb-noun format, such as UpdateInventory.

Remember, messages, even incoming Web service request messages, get routed to an orchestration based on a subscription. When you bind a logical port to a physical port, which is a requirement with Request-Response ports published as Web services (even when publishing via schemas), BizTalk creates an automatic subscription for you. You can view the subscription from the BizTalk Administration Console by selecting "Subscriptions" as the value in the query window. The subscription is slightly different for SOAP than what’s usually created for other logical-to-physical ports that aren’t Direct-Bound. Regular subscriptions that BizTalk creates for orchestrations will look like the following:

http://schemas.microsoft.com/BizTalk/2003/
system-properties.ReceivePortID == {some Guid}
AND
http://schemas.microsoft.com/BizTalk/2003/system-properties.MessageType ==
http://schemas.someschema#Root
AND
http://schemas.microsoft.com/BizTalk/2003/
system-properties.InboundTransportType != SOAP
OR
http://schemas.microsoft.com/BizTalk/2003/
system-properties.ReceivePortID == {some Guid}
AND
http://schemas.microsoft.com/BizTalk/2003/soap-properties.MethodName ==
 {Inbound Operation Name on Logical Port}

You can see that the name you type into the wizard as the MethodName must match the Operation name of the logical port within the orchestration, or the mapped Web request message will never get routed to the orchestration.

When you get to the Web Service Project dialog in the wizard, enter a URL for your Web service, for example: http://localhost/MyBizTalkWebService. Also, you may as well let BizTalk create the receive locations in your application for you. Since you already deployed your application, you can choose it here from the dropdown list. For more details about all this, be sure to see the article titled "How to Use the BizTalk Web Services Publishing Wizard to Publish Schemas as a Web service" in the BizTalk help file.

I know we said BizTalk is all about subscriptions, but that’s not our final answer. As you can tell from the last two tips, it’s really all about schemas. Every message and every map depends on a schema (or two) so mastering schemas is vital to BizTalk proficiency.

5. Always Optimize the BizTalk Registry for Web Services

If your BizTalk orchestration is published as a Web service, you’ll definitely want to tweak some of the default ASP.NET parameters used by BizTalk. Actually, these parameters affect any BizTalk artifact that uses the CLR including XLANG. BizTalk automatically multiplies these recommended values by the number of CPUs you have. Here are the steps:

1. Before modifying the Registry, back it up.

2. Create a file in Notepad with the following code.

Windows Registry Editor version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\BTSSvc$BTSHOST\CLR Hosting]
“MaxIOThreads”=dword:00000064
“MaxWorkerThreads”=dword:00000064
“MinIOThreads”=dword:00000019
“MinWorkerThreads”=dword:00000019

By creating the file in this manner, you can easily reuse it on other machines running BizTalk. The DWORD values are in hexadecimal (Registry scripts do not use the standard 0x prefix for hexadecimal values).

3. Replace the string "BTSHOST" with the name of the BizTalk host for your machine. You can run RegEdit first to examine the actual key name on your box.

4. If you save the file with a suffix of .REG, you can double-click it to run it on your BizTalk installation.

5. Run RegEdit and navigate to the CLR Hosting key to verify that the values have been added correctly.

See msdn2.microsoft.com/en-us/library/aa561380.aspx for more details about this and other optimizations and msdn2.microsoft.com/en-us/library/aa475435.aspx for an excellent article about performance tuning for low latency messaging.

6. Always Set the Assembly Key File with a Relative Path

This tip will help you check out the solution from version control and build it on a different box. Some developers on your team may arrange their environment with different drive letters, or you may change yours in the future. This is really simple, but you may be wondering what’s going on with all those dots.

The recommendation is for the first developer on the solution to create a strong name key file in the same folder with the Visual Studio solution file (.sln). Then create subfolders of that solution folder to hold each Visual Studio project by artifact type, such as maps, pipelines, orchestrations, internal schemas, external schemas, and any .NET components you’re compiling. By the way, despite how many times you may have heard that advice, you don’t have to think of it as a strict rule. If you know certain artifacts are likely to be revised together in the future (such as a map and its schemas) then you should put them in the same project to reduce the number of components you’ll have to redeploy. During development we can share the same key file among the projects within the solution and among developers. (Creating secure key files for production is another matter.) Just check the key file into version control along with the other artifacts in your solution. Now indicate the path to the key file for each assembly in the solution. The way to do that for BizTalk projects is "..\..\..\Key.snk", as shown in Figure 6. This works because BizTalk builds your binaries in a subfolder called something like [YourSolution]\[YourProject]\bin\Development. So three parent directory hops gets BizTalk back to your solution folder with the key file.

Figure 6 Configuring the Assembly Key File for a BizTalk Project
Figure 6 Configuring the Assembly Key File for a BizTalk Project(Click the image for a smaller view)
Figure 6 Configuring the Assembly Key File for a BizTalk Project
Figure 6 Configuring the Assembly Key File for a BizTalk Project (Click the image for a larger view)

Note that if you add .NET Framework Class Library projects to your solution, they’re compiled with a slightly different directory structure than BizTalk projects, and the Visual Studio UI to set the key file path doesn’t really allow you to enter a relative path. One solution is to edit the .csproj (for C#) or .vbproj (for Visual Basic®) as a text file. Search for the name of your key file that you browsed to in the Visual Studio project properties UI, and replace it with a relative path that goes up only one parent, like so: ..\Key.snk. Don’t forget to add your .NET Framework assembly to the GAC after each build. You can do it automatically with a post-build step in Visual Studio. It just requires full paths and quotes around strings with spaces, like this monster (all on one line):

“C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin\gacutil.exe” /i “$(TargetPath)” /F

7. Never Overlook Free Sample Code

Here’s a quick tip worth an ounce of gold. The BizTalk help file documents over 50 sample applications and scripts (installed in the SDK\Samples folder) that you can learn from, adapt, and reuse. You’ll also find over 30 useful BizTalk applications on this page: msdn2.microsoft.com/en-us/biztalk/aa937647.aspx. Finally, the Bloggers Guide to BizTalk (available on GotDotNet.com) is a compendium of more than 400 technical articles, including many useful code snippets and samples. I keep a shortcut to the Bloggers Guide on my desktop and it’s one of the first places I look when venturing into new BizTalk territory.

OK, we admit it, if you already knew this, then you might think this is a lame tip. But considering that it is otherwise difficult to find professional BizTalk sample code, we thought it worth reminding that this cache is always right at your fingertips. The thing is, these samples aren’t just useful when you’re getting started with BizTalk, they’re useful references again and again.

By the way, don’t forget to check back on MSDN for the BizTalk help file that’s updated periodically. You can download it in PDF or zipped .chm format, or browse it online.

8. Debug XSLT in Visual Studio

This could be the least well-known feature in Visual Studio, and it doesn’t require BizTalk to be installed. The XSLT debugger allows you to see the results of a transform while it’s running, set breakpoints, and examine local variables and the call stack. You can even step into your XSLT script from C# or Visual Basic.

Figure 7 shows a running example of a transform. I just created a blank solution and loaded this tiny XML data file to be transformed into HTML:

Figure 7 The XSLT Debugger in Visual Studio
Figure 7 The XSLT Debugger in Visual Studio(Click the image for a smaller view)
Figure 7 The XSLT Debugger in Visual Studio
Figure 7 The XSLT Debugger in Visual Studio (Click the image for a larger view)

<?xml version=”1.0” encoding=”utf-8”?>
<node attr=”debug” />
Then I entered the XSLT file you see in the left pane in Figure 7. It will create a Hello World document, and if the node element has an attr attribute equal to “debug”, then it will spit that out too.

Now with your cursor in the XSLT file (that’s important), set the Input property to be the path of the XML data file you’re going to transform. Be aware that right-clicking the XSLT filename itself in the Solution Explorer window will display a different set of properties, the File properties, not the Document properties. I wish that were more intuitive, but once you learn it, you’ve got it. And if there’s one thing that separates pros from average blokes, it’s the desire to explore. You can set the Output property too, or enjoy the show as the transformation come to life in the middle pane.

When the active document in Visual Studio is an XSLT file, the dropdown XML menu (on the main menubar) includes a Debug XSLT command. Just click that and you can step through your transformation code, even stepping into subroutines. Figure 7 is showing the value of @attr in the Watch window as the program has just performed the xsl:if test. As you can see from the output in the middle window, the test evaluated true, and the <h1> node was output.

Conclusion

Given the number of applications organizations develop today, it’s critical to use good tools and techniques to produce working solutions as quickly as possible. BizTalk will help you efficiently move from concept to working prototype and we hope these tips will save you some time in your efforts.

Marty Wasznicky (MCSE, MCSD, MCDBA, MCTS) is the Regional Program Manager for BizTalk Server in the Microsoft Connected System Division. He helps customers and Microsoft Partners adopt and implement Microsoft solutions in this space. Currently, he is working on Software Oriented Architecture and Enterprise Service Bus technologies. He lives with his wife in Los Angeles.

Scott Zimmerman (MCSD, MCTS) is a senior application development consultant with Microsoft specializing in BizTalk and .NET. He has won two awards for Web service design and the Overall World Frisbee Championship eight times. He lives with his wife Vera in Washington, D.C.


Collaborate: Help Teams Work Together With Web Services And Groove 2007

Collaborate

Help Teams Work Together With Web Services And Groove 2007

John C. Hancock
This article discusses:
  • An introduction to Groove 2007
  • Using Groove forms for gathering data
  • Groove Web services API
This article uses the following technologies:
Microsoft Office, .NET Framework
Code download available at: Groove2007_05.exe (524KB)
Contents
Building Solutions with Groove
Groove 2007 Architecture
Using Groove Forms to Collect Data
Accessing Groove Data from an Application
Using the Groove Web Services API
Registry Settings for Groove Web Services
Accounts and Identities
Getting a List of Workspaces and Tools
Fetching Data from Groove Forms
Loading Forms Data into a Database
Conclusion

I
n the real world, users often need to collect information while they are out of the office and unable to access their company networks. For example, a consumer research organization might send employees out to the field to ask people to fill in a paper survey, and then manually enter this data into a database. If you wanted to write a custom application that could be used in the field to automate this process, you would need to handle issues like managing a local data store and synchronizing it with a server database.

In this article, I show how the unique collaboration capabilities in Microsoft® Office Groove™ 2007 let you solve these problems as I integrate them into a custom Microsoft .NET Framework application. Groove provides the rich client for collaborative data gathering and secure synchronization of data between team members, and my custom .NET Framework application will be used to connect the data to a corporate database.

Groove is an environment for team collaboration that allows members to share data across organizational boundaries, regardless of the network they are connected to. In other words, participants don’t need to be part of the same network, making Groove a great collaboration tool when you need to include employees, partners, and customers without requiring the cooperation of IT departments. Groove is also excellent for mobile scenarios where users have to be able to go offline and still access and update the shared data, and also need to automatically synchronize data when they are back in connected mode.

Groove 2007 is part of the Ultimate and Enterprise editions of the 2007 Microsoft Office system, and can be purchased as a separate product. You can also install a free trial of Groove and download the Groove SDK from microsoft.com/downloads.

Building Solutions with Groove

You can build custom collaboration solutions using the out-of-the-box Groove features like file sharing and discussions just by creating workspaces with the relevant tools and then saving the workspace as a reusable template. If you need more flexibility, you can also build custom applications that take advantage of the secure collaboration facilities in Groove through the Web services-based API.

For the consumer research application solution I discuss here, the researchers who will use the application use Tablet PCs to fill in surveys while they are out in shopping malls. Ideally, the solution should work while the users are offline and then allow them to easily consolidate the information from the different researchers. In practice, it would be great if a researcher could simply walk into a café with a public Internet connection and have the solution automatically and securely synchronize the information with all participants.

The first step in designing the solution is for the research coordinator to create a new workspace in Groove. Every workspace can include multiple tools for particular requirements, such as a Files tool that allows participants to share documents with each other, and a Discussion tool so that people can ask and respond to questions from other participants.

Once the workspace is created, the research coordinator sends an invitation to all of the researchers to join the workspace. This invitation shows up as a notification to each participant, allowing them to download the workspace to their local machines. Everyone is now ready to collaborate—any change that a participant makes to his local copy of the workspace, such as adding a file or discussion topic, will be automatically and securely propagated to all the other participants when they are connected to a network.

Groove 2007 Architecture

Groove has an innovative architecture that is the source of its unique capabilities. First, Groove stores all the data on participants’ PCs rather than on a central server. This means that there is no need to devise a secure way for all participants to access a shared resource; whether the researchers are employees of the company or independent users, the research coordinator can easily create a shared workspace and invite them to participate.

Second, all of the data is always encrypted—both the local workspaces on disk and all network communications. One of the benefits of this is that researchers can synchronize their changes simply by connecting to through a public Internet connection. There is no need for all participants to have virtual private network (VPN) access to corporate resources, because the automatic synchronization is already encrypted.

Although most of Groove is client-based, there are also a few server components. Groove uses a relay server that acts as a store-and-forward service so that clients can communicate across firewalls using standard protocols; the relay server also stores temporary encrypted copies of workspace updates for offline users. In addition, there is a management server so that administrators can manage licenses and policy settings for all the users.

Standalone Groove users can create their own accounts when they install the client, which will automatically use relay servers hosted by Microsoft. Large companies with many Groove users might choose to deploy Office Groove Server 2007 as part of their own infrastructure so that they have more control over license management and can do things like integrate with Active Directory®. Small to mid-size enterprises can purchase a subscription to Office Groove Enterprise Services, which gives them access to a hosted version of the management tools for centralized control without requiring them to deploy their own infrastructure.

Using Groove Forms to Collect Data

For my solution, the workspace must include a survey form that the researchers can fill out for every consumer that they interview. Groove includes a Forms tool for designing custom forms that allows me to specify fields including data types, captions and other properties. Then the participants can add data to the workspace as shown in Figure 1.

Figure 1 Groove Workspace with Forms Tool
Figure 1 Groove Workspace with Forms Tool(Click the image for a smaller view)
Figure 1 Groove Workspace with Forms Tool
Figure 1 Groove Workspace with Forms Tool (Click the image for a larger view)

Forms can also include JavaScript or VBScript to extend their capabilities for building custom solutions. Also, there is some flexibility when it comes to UI styles and formatting because the forms use DHTML and CSS. If you are already using InfoPath® forms in your organization, you can use the Groove InfoPath Forms tool to import XML-based InfoPath forms definitions instead of using built-in forms in Groove.

When the workspace with the custom form has been downloaded to the researchers’ tablet PCs, they can disconnect and still use the form to add records while they are in the field. When they connect back to the network, all of the records they have added will be securely synchronized with the other participants. The research coordinator will get a popup notification from Groove so that she knows that there is new unread data in the workspace.

Collecting the data from the researchers is only part of the story though. Once all the data from the individual surveys is synchronized, the research coordinator needs to load all the data into the company’s database so that it can be analyzed further. The company’s analysts will be using a SQL Server™ database, an Analysis Services cube, and Excel® business intelligence capabilities to understand the survey results and make some recommendations.

Accessing Groove Data from an Application

To get data out of the Groove workspace and into a corporate database, the research coordinator could manually export the data from the Groove forms tool in a format such as CSV (comma-separated values) and then import it into the database. However, this manual process would quickly become unmanageable if she needed to load the survey data into the database for analysis as it became available, rather than all at once.

A better solution would be to build a custom .NET Framework application that uses the Groove Web services-based API to read the data entered by the field researchers. Install this application on the research coordinator’s PC and she can access both the Groove workspace and the SQL Server database because she is connected to the corporate network as shown in Figure 2, unlike the researchers in the field who may only be connected to the public Internet.

Figure 2 Groove Workspace Users and SQL Server Database
Figure 2 Groove Workspace Users and SQL Server Database(Click the image for a smaller view)
Figure 2 Groove Workspace Users and SQL Server Database
Figure 2 Groove Workspace Users and SQL Server Database (Click the image for a larger view)

This approach is similar to the way Groove is integrated with Windows® SharePoint® Services (WSS) 3.0. Teams can use a Groove workspace for collaborating on documents, and then publish the completed documents to a SharePoint document library so everyone in the company can take advantage of SharePoint’s capabilities such as search and workflow. The way this works is that someone with access to both Groove and the SharePoint site adds a SharePoint Files tool to the workspace and connects it to a SharePoint document library. All workspace participants now have copies of all the documents and can submit changes, which the connected user can then synchronize with the SharePoint document library when necessary.

Using the Groove Web Services API

Even though I am developing a client application, the programming model for Groove uses Web services rather than a managed API. It feels a little strange programming this way because the client application is calling Web services that are actually running on the same machine. These services will still be available even when the machine is disconnected from the network—the Groove client itself is answering the requests. For larger enterprise solutions, a server-based product called Groove Server Data Bridge with a similar programming model is available, but I am going with the client-side approach for this solution.

To create the client application shown in Figure 3, I start by adding Web references to my project for each of the Groove services that I want to access. The Groove SDK includes a WSDL file for each service, so I add the Web reference by specifying the path to the relevant .wsdl file, renaming the Web reference from the default "Web Reference" to something more useful and clicking the Add Reference button. In this application, I will be using the GrooveAccounts, GrooveForms2, GrooveSpaces, and GrooveTools services.

Figure 3 Load Groove Data into a SQL Server Database
Figure 3 Load Groove Data into a SQL Server Database(Click the image for a smaller view)
Figure 3 Load Groove Data into a SQL Server Database
Figure 3 Load Groove Data into a SQL Server Database (Click the image for a larger view)

Registry Settings for Groove Web Services

I will need to know the port that Groove is using before I can call the Web service—this defaults to 9080 but I should retrieve it from the GrooveLocalHTTPPort registry setting for the current user. I will also need to supply a value called a request key for every call, which the Web services use to authenticate the caller. I can retrieve this key from the LocalRequestKey registry setting, as shown in Figure 4. The request key is randomly regenerated every time Groove is restarted, so I should retrieve it from the registry before every call in case it has been regenerated since the last time I checked.

Figure 4 Registry Settings for Groove Web Services (Close Figure 4)
using (RegistryKey grooveSettingsRegKey = Registry.CurrentUser.OpenSubKey(
    “Software\\Microsoft\\Office\\12.0\\Groove”))
{
    // Get the URL for web services including the port
    string grooveHost = string.Format(“http://localhost:{0}”,
       grooveSettingsRegKey.GetValue(“GrooveLocalHTTPPort”, 9080));

    // Get the LocalRequestKey for authentication
    string localRequestKey = (string)grooveSettingsRegKey.OpenSubKey(
        “WebServices”).GetValue(“LocalRequestKey”);
    ...
}

The other registry setting that is used for Web service authentication is the response key. Every time I make a call to a Groove Web service, the response includes a key that I can check against the LocalResponseKey registry setting to make sure they match.

Accounts and Identities

Every Groove user has an account that is configured the first time he starts up Groove. The account is stored as a file on their computer (or on multiple computers), and includes information such as cryptographic keys and contact lists. Every account also has at least one identity that contains the contact information that is shared with other Groove users, such as an e-mail address and phone numbers. An account can also contain multiple identities if required so that, for example, a person can use one identity for business purposes and another for personal use such as sharing files with friends.

I can use the GrooveAccounts service to get the list of accounts and identities on the computer. After setting up the Web service call using the request key and port as shown in Figure 5, I can use the Read2 method of the GrooveAccounts service to return an array of Account2 objects. Each account has a collection of identities, so I can load up a list of identities to allow the user to select which one has the consumer research workspace as shown in Figure 3. (Most users only have one account with a single identity.)

Figure 5 List All Groove Accounts (Close Figure 5)
// Set up the web service call for GrooveAccounts
GrooveAccounts.GrooveAccounts svc = new GrooveAccounts.GrooveAccounts();
svc.GrooveRequestHeaderValue = new GrooveAccounts.GrooveRequestHeader();
svc.GrooveRequestHeaderValue.GrooveRequestKey = localRequestKey;
svc.Url = grooveHost + “/GWS/Groove/2.0/Accounts/”;

// Get a list of all the accounts
GrooveAccounts.Account2[] accounts = svc.Read2();

// Display all identities
foreach (GrooveAccounts.Account2 account in accounts)
    foreach (GrooveAccounts.Identity2 identity in account.Identities)
        Console.WriteLine(account.Name + “, “ + identity.Name);

Getting a List of Workspaces and Tools

Once I know which identity I am working with, I have enough information to get a list of the user’s workspaces using the Read method of the GrooveSpaces service. The identity object has a URI property that is unique for this identity and can be used for most other Groove Web service calls as shown in Figure 6, and also a Spaces property that contains the URL for the user’s workspaces.

Figure 6 List All Groove Workspaces (Close Figure 6)
private void LoadSpaces(GrooveAccounts.Identity2 identity)
{
    // Set up the web service call for GrooveSpaces
    GrooveSpaces.GrooveSpaces svc = new GrooveSpaces.GrooveSpaces();
    svc.GrooveRequestHeaderValue =
        new GrooveSpaces.GrooveRequestHeader();
    svc.GrooveRequestHeaderValue.GrooveRequestKey =
        GrooveSettings.RequestKey;
    svc.GrooveRequestHeaderValue.GrooveIdentityURL = identity.URI;
    svc.Url = GrooveSettings.GrooveHost + identity.Spaces;

    // Get the list of spaces
    _spaces = svc.Read(“”);

    // Load the list into a combobox
    comboWorkspaces.DisplayMember = “Name”;
    comboWorkspaces.DataSource = _spaces;
}

As you saw when I created the Groove workspace earlier, each workspace can contain multiple tools such as files, discussions and forms. When the user has selected the workspace in the client application, I can find the forms tool that contains the market research data using the Read method of the GrooveTools service. The workspace only has one forms tool so I can just look for a tool named "Forms", but if I wanted to get more sophisticated, I could display a list of all the tools of the right type and allow the user to pick the right one.

Fetching Data from Groove Forms

Now that I have written code to retrieve the Groove settings from the registry, and to get a list of accounts, identities, workspaces and tools, I am ready to tackle the original requirement of getting the data out of the Groove tool and into my SQL Server database for analysis. Fetching the data is actually pretty easy; I can just set up a query that describes what records I would like to fetch using a RecordQuery object and then pass it to the QueryRecords method of the GrooveForms2 service, as shown here:

// Query ALL records in the ALL view:
GrooveForms2.RecordQuery query = new GrooveForms2.RecordQuery();
query.ViewURI = “”;
query.UnreadRecordsOnly = false;
query.QueryMetadataOnly = false;
query.IncludeFileAttachmentContent = false;

// Issue the query
GrooveForms2.Forms2RecordDataSet recordDataSet =
    svc.QueryRecords(query);

The fun starts when you take a look at the Forms2RecordDataSet object that is returned by QueryRecords. Forms2RecordDataSet contains both the schema and the data for the form in XML format, so I am going to have to do some further manipulation before I can use it effectively. The easiest way of working with forms data is to convert the Forms2RecordDataSet object into a more familiar .NET DataSet object.

The Groove SDK documentation has a whole section devoted to this topic, and the code is also included in the sample download for this article, so I will just give you a high-level overview here (especially since this is the type of code that you write once and then reuse all over the place). In principle, you create a new DataSet and use the ReadXmlSchema method to set up the tables and column definitions by creating a string reader for the schema and prepending the appropriate XML version declaration to the string. When the schema has been created, you can use the ReadXml method to load the actual data.

One minor issue that you have to take into account relates to date-time columns. Groove returns all date-time fields in the Forms2RecordDataSet as universal time, while the .NET DataSet has them as local time by default. I can avoid problems by looping through all the columns that are created in the DataSet from the ReadXmlSchema method, and setting their DateTimeMode to Utc before I load the data.

Loading Forms Data into a Database

I can now loop through the DataSet and add a new row to the SQL Server database for each new record from the Groove form. The user could run the process any number of times, so I need to have some mechanism to avoid loading duplicate records into the database.

Every record in a Groove form has a system-generated field called _RecordID that uniquely identifies that record. If I include this field when I load records into the database, I can use it to check whether a record has already been added to the database and if so, update the existing database row rather than inserting a duplicate row. This will ensure that any survey records that were modified since the last time the process was run are correctly reflected in the SQL Server database.

Conclusion

Groove is a great environment for building solutions for distributed teams that would be difficult and costly to develop from scratch. Whether you stick to the out-of-the-box features like file sharing and forms or build entirely new applications that build upon the API, Groove is a great addition to your arsenal.

John C. Hancock is a senior consultant for Microsoft, specializing in business intelligence and .NET Framework development. He is the coauthor of Practical Business Intelligence with SQL Server 2005 (Addison-Wesley Professional, 2006), and can be contacted at www.johnchancock.net.


Editor's Note: Daylight Savings

Editor's Note
Daylight Savings

Sometime after the New Year, we became aware of a problem that had evidently been percolating for some time. For 2007, Daylight Saving Time moved up by a few weeks.

How did this happen? DST is strange enough. For 90 years, we’ve harnessed the sun by making it rise and set an hour later for a few months at a time. The science behind this is beyond our understanding, but it seems to involve an elaborate series of levers placed in space.

Technology has advanced far enough now so that we can get an earlier start on Daylight Saving Time in the United States. The Energy Policy Act of 2005 changes DST so that it now starts in mid-March and goes through the first week of November. This has two major effects that we’re aware of. First, there was a brief period of teeth gnashing in late February as the new dates approached. Second, the sun stays out an hour later on Halloween.

Suddenly, the media picked up on the Daylight Saving Time story. Out of nowhere, it became a major item for a few days. "Will this be like the Y2K bug?" "Will your machines all seize up as the airplane you’re on falls out of the sky at 2:00 AM?" "Isn’t there anything more important we should be covering?"

Of course, there was one key difference between DST and Y2K. In 1999, almost every type of computing system was affected in some way. People were worried about financial systems, travel problems, power utilities, you name it. Lousy TV movies were made showing the worst-case millennial scenario. Camping supply stores nationwide sold Y2K survival kits, consisting of plenty of bottled water, thermal blankets, and just to be safe, anti-zombie charms.

The DST change didn’t have nearly the same downside. For three weeks, our appointment calendars would be an hour out of sync and we’d miss some meetings. The local office supply store wasn’t hawking special DST survival kits with, well, pens and printed calendars. Most systems were quietly patched through Windows Update, and that was that.

You can no doubt see the analogies with general programming we’re developing here. You can relate to the worldwide panic over Y2K seven years ago—there’s always some problem that someone coded long ago, only to have it dumped in your lap at the last moment. Much of the time, these bugs consist of more panic than actual negative effect, but they become top priority.

The changes in DST are more like the problems we face every day in programming. Daylight Saving Time isn’t a universal concept. Different countries (and even different parts of the United States) observe different DST rules. Most programs don’t include their own DST code in the first place, instead relying upon the base operating system. In fact, making coding changes to account for daylight in your program would be a classic case of planning on behavior that’s "not guaranteed in future versions."

One of the great effects of Daylight Saving Time is that you get an extra hour of daylight to read the MSDN Magazine blog each evening. If you drop by blogs.msdn.com/msdnmagazine, you can catch the latest news from the magazine, chat with an editor or two, and discuss this month’s Editor’s Note. We’ll be doing this each month from now on, so stop by and let us know how this new DST policy has affected your work. —J.T.

Thanks to the following Microsoft technical experts for their help with this issue: David Callahan, Dave Detlefs, Matt Gibbs, Mike Harder, Bertrand Le Roy, Lawrence Liu, Val Menn, Kevin Moore, Chris Norman, Hugh Pyle, Gaurav Seth, Matt Shannon, Jim Truher, and Molly Yen.


Toolbox: Code Exploration, SQL Queries, and More

Toolbox
Code Exploration, SQL Queries, and More
Scott Mitchell

Contents
Analyze and Graphically Explore Your App’s Architecture
Upgrade Your SQL Query Editor
Take Screenshots Quickly and Easily
Developing .NET Apps

Analyze and Graphically Explore Your App’s Architecture

Large Microsoft .NET Framework applications can contain hundreds or thousands of classes organized into different assemblies that constitute distinct layers in the application’s architecture. While most developers work exclusively with a particular section of the architecture, senior developers and system architects are typically responsible for code and design decisions that impact various pieces of the architecture and their interconnections. For these code changes and architectural decisions, it is vital that senior developers and system architects have a solid understanding of the architecture’s structure and interdependencies. Commonly, developers resort to picking through the source code, which is often error-prone and time-consuming.

NDepend Provides a Matrix that Shows the Dependencies Among Various Assemblies
NDepend Provides a Matrix that Shows the Dependencies Among Various Assemblies (Click the image for a smaller view)
NDepend Provides a Matrix that Shows the Dependencies Among Various Assemblies
NDepend Provides a Matrix that Shows the Dependencies Among Various Assemblies (Click the image for a larger view)

Fortunately, there’s an easier way to explore a .NET Framework app’s architecture at a high-level view. NDepend version 2.0, by SMACCHIA.COM, is a tool that helps examine and analyze the metrics, complexities, and dependencies between the various pieces of an app’s architecture. Once an app’s assemblies and source code have been selected and analyzed, NDepend provides a rich user interface for exploring a multitude of metrics and information.

At the top of the interface is a treemap of the architecture’s assemblies. (A treemap is a graph in which the size of each element is dependent on its "weight.") The treemap depicts various metrics such as the number of lines of code or number of methods, which provides a quick way to gauge the relative complexity of the components in the architecture.

NDepend provides a matrix that shows the dependencies among the various assemblies. You can drill down to the assemblies’ classes, and from there to the classes’ methods. Right-click to see what namespaces, classes, or methods either directly or indirectly call upon or use the selected assembly, class, or method. The matrix can also help pinpoint cyclic dependencies.

NDepend’s most impressive feature is its query language, CQL, which is similar in syntax to SQL. CQL statements can be used for ad hoc reporting. The following CQL query returns the 10 namespaces with the most types:

SELECT TOP 10 NAMESPACES ORDER BY NbTypes DESC

CQL queries are also useful for code quality audits. This query produces a warning if there are any methods that have over 30 lines of source code (which indicates a complex method that should likely be refactored):

WARN IF Count > 0 IN SELECT TOP 10 METHODS WHERE NbLinesOfCode > 30

Along with its UI, NDepend can be accessed through the command line, making it easy to integrate into an automated build process.

Price: $411 for a single user license.

www.NDepend.com

Upgrade Your SQL Query Editor

While Enterprise Manager and SQL Server™ Management Studio provide an easy-to-use graphical environment for performing database-related tasks, their native query editors come across to many as simplistic text editors that lack modern features. There’s no IntelliSense® or integrated source control support, for example. Visual Studio® provides Database Projects for grouping CREATE scripts and other commonly used T-SQL within the IDE, but these serve more as a repository for database-related scripts than an interface for writing, testing, and running queries.

However, there are a number of third-party SQL query editors that offer the features developers today take for granted. One such product is ApexSQL Edit version 2005.03, by ApexSQL. ApexSQL Edit boasts a number of time-saving features, the most impressive one being the Object MemberLists, which provides functionality much like IntelliSense. Type a query, like "SELECT * FROM...", and immediately a drop-down list appears listing the database’s tables. This feature also extends to columns, variables, parameters, procedures, and so forth. Once you start writing queries using MemberLists, having to go back to an editor without it feels like running underwater. This feature alone could save you countless keystrokes and typos and makes ApexSQL Edit an interesting tool for any developer’s toolbox.

Write, Test, or Execute Queries
Write, Test, or Execute Queries (Click the image for a smaller view)
Write, Test, or Execute Queries
Write, Test, or Execute Queries (Click the image for a larger view)

Another time-saving feature is the Object Explorer, which lists details about the selected table, stored procedure, view, UDF, or other database object. This information includes the object’s data, T-SQL statements to drop and recreate the object, and its permissions and dependencies. There’s also a Notes tab where you can add any comments regarding the object. (These comments are stored in a database table that must first be added to the database.) For database tables, the Object Explorer also lists the table’s columns and indexes. The Object Explorer makes it easy to quickly look up information about a table, view, or other object that you are currently writing a query against.

ApexSQL Edit also offers SQL Server Reporting Service integration, built-in Unit Testing support, powerful find and replace features, customized SQL formatting, and much more. If you spend any significant amount of time writing, testing, or executing queries through the query editors shipped with Enterprise Manager or Management Studio, it’s time to upgrade your query editor and give ApexSQL Edit a try.

Price: $349 for a single user license.

www.apexsql.com

Take Screenshots Quickly and Easily

There are many scenarios in which a developer may need to take a screenshot. They make for great visual aids in documentation and are useful in capturing anomalies or error messages when testing and debugging. With Windows®, the Print Screen key copies the current window to the clipboard. The screenshot can then be pasted directly into a document or pasted in an image editor and saved in one of any number of image file formats. However, when taking many screenshots and saving them to files, the workflow—hit Print Screen; paste the screenshot into an image editor; crop the image, if needed; save the image to disk—is slow and repetitive.

There are a variety of freeware and third-party products available to remove this monotony and improve efficiency. One such product is Cropper version 1.9, by Brian Scott, which is written in C# with the source code freely available. Cropper can capture an entire window or a defined region. To capture an entire window, simply hit the Print Screen key. Double-clicking the Cropper icon in the task tray brings up a floating frame whose position and dimensions specify the capture region.

Cropper can save the screen capture to the clipboard, have it sent directly to the printer, or automatically save it to disk as a BMP, PNG, or JPG file. If the thumbnail feature is activated, Cropper will save both the full-sized screenshot and a thumbnail of a specified size. When capturing non-rectangular windows, Cropper can be configured to fill those transparent regions—such as the background behind the curved corners—with a particular color.

Price: Free; includes source code.

blogs.geekdojo.net/brian/articles/Cropper.aspx

Developing .NET Apps

Most successful computer trade books either focus on a very specific subset of a technology or provide a broad overview of the entire technology. How to Code .NET, by Christian Gross (Apress, 2006), doesn’t fall into either of these categories. While its title implies that the book is a weighty tome on creating .NET applications, its mere 232 pages indicate otherwise. It neither focuses on a specific subset of .NET nor does it provide a sweeping overview. Instead, it provides 30-50 pages each on four disparate topics: Test Driven Development (TDD), low-level .NET runtime and framework information, text-processing tips, and C# coding tricks.

The book starts, appropriately, with the TDD section. There are no wasted words here, just a quick overview of TDD’s principles, a look at NUnit, and a discussion on writing tests. After this, the rest of the chapter explores solutions to common TDD challenges, such as applying the principles of TDD to a project you’ve inherited that was not developed using TDD. The second chapter, .NET Runtime- and Framework-Related Solutions, offers explanations, tips, and tricks on lower-level topics. For example, there’s a discussion on the difference between value and reference types, code examples illustrating the benefits of delegates, and a look at the Object class’s Equals and GetHashCode methods and when you should override them.

(Click the image for a smaller view)

The third chapter provides some handy text-related solutions, such as discussions and code snippets for parsing and searching strings. C# Coding Solutions, the book’s lengthiest chapter, looks at a number of C# features and patterns, including inheritance, nullable types, overloaded functions, interfaces, and the factory pattern.

At 232 pages, How to Code .NET does not cover any topic in great detail and therefore is not intended for the reader who wants to explore these topics in great depth. But the book provides a well-written, informative look at a smattering of topics. Most intermediate .NET developers would benefit from reading How to Code .NET as it provides a number of useful tips and serves as a good starting point for exploring some of the more advanced .NET concepts.

Price: $29.99.

www.apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.net.


CLR Inside Out: 9 Reusable Parallel Data Structures and Algorithms

CLR Inside Out
9 Reusable Parallel Data Structures and Algorithms
Joe Duffy

Code download available at: CLRInsideOut2007_05.exe (156KB)
Contents
Countdown Latch
Reusable Spin Wait
Barriers
Blocking Queue
Bounded Buffer
Thin Event
Lock-Free LIFO Stack
Loop Tiling
Parallel Reductions
Conclusion

This column is less about the mechanics of a common language runtime (CLR) feature and more about how to efficiently use what you’ve got at your disposal. Selecting the right data structures and algorithms is, of course, one of the most common yet important decisions a programmer must make. The wrong choice can make the difference between success and failure or, as is the case most of the time, good performance and, well, terrible performance. Given that parallel programming is often meant to improve performance and that it is generally more difficult than serial programming, the choices are even more fundamental to your success.

In this column, we’ll take a look at nine reusable data structures and algorithms that are common to many parallel programs and that you should be able to adapt with ease to your own .NET software. Each example is accompanied by fully working, though not completely hardened, tested, and tuned, code. The list is by no means exhaustive, but it represents some of the more common patterns. As you’ll notice, many of the examples build on each other.

There’s something I should mention at the outset. The Microsoft® .NET Framework has several existing concurrency primitives. While I’ll show you how to build your own primitives, the existing ones are completely suitable for most situations. I just want to demonstrate that alternatives are sometimes worth considering. Besides, seeing these techniques in action will deepen your understanding of parallel programming in general. I will assume a basic understanding of the existing primitives, though. See Vance Morrison’s "What Every Dev Must Know About Multithreaded Apps" in the August 2005 issue of MSDN® Magazine for a comprehensive overview.

Now, let’s have a look at the techniques.

Countdown Latch

Semaphores are one of the better known data structures in concurrent programming for many reasons, not the least of which is that semaphores have a long history in computer science dating back to operating system design in the 1960s. A semaphore is simply a data structure with a count field, and it supports two operations: put and take (often called P and V, respectively). A put operation increments the semaphore’s count by one, and a take decrements it by one. When the semaphore’s count becomes zero, any subsequent attempts to take from it will block (wait) until another a concurrent put makes the count non-zero. Both are atomic, concurrency-safe operations, ensuring that concurrent puts and takes are serialized with respect to one another. Windows® has first-class kernel and Win32® support for semaphore objects (see CreateSemaphore and related APIs), and they are surfaced in the .NET Framework through the System.Threading.Semaphore class. Critical regions, as supported by Mutex and Monitor, are often characterized as a special semaphore with a count that toggles between 0 and 1—in other words, a binary semaphore.

A kind of "reverse semaphore" is often useful, too. That is to say, sometimes you want a data structure that supports waiting for the data structure’s count to reach zero. Fork/join parallelism—where a single "master" thread controls the execution of n "subservient" threads and then waits for them to finish—is quite common in data parallel programming, and having a reverse semaphore on hand is very useful for such cases. Most of the time, you don’t actually want waking threads to modify the count, so in this case we’ll call the structure a countdown "latch" to indicate that counts decrease, and that once set to the signaled state, the latch remains signaled (a property often associated with latches). Sadly, neither Windows nor the .NET Framework supports such a data structure. Thankfully, however, it’s simple to build one.

To build a countdown latch, you just initialize its counter to n, and have each subservient task atomically decrement it by one when it finishes, for example by surrounding the decrement operation with a lock or with a call to Interlocked.Decrement. Then, instead of a take operation, a thread could decrement and wait for the counter to become zero; when awoken, it will know that n signals have been registered with the latch. Instead of spinning on this condition, as in while (count != 0), it’s usually a good idea to let the waiting thread block, in which case you then have to use an event. Figure 1 is an example of a simple CountdownLatch type.

Figure 1 CountdownLatch (Close Figure 1)
public class CountdownLatch {
    private int m_remain;
    private EventWaitHandle m_event;

    public CountdownLatch(int count) {
        m_remain = count;
        m_event = new ManualResetEvent(false);
    }

    public void Signal() {
        // The last thread to signal also sets the event.
        if (Interlocked.Decrement(ref m_remain) == 0)
            m_event.Set();
    }

    public void Wait() {
        m_event.WaitOne();
    }
}

This is surprisingly straightforward, but can be tricky to get right. We’ll see some examples of how you might use this data structure later. Note that the basic implementation shown here leaves room for various improvements, including: adding some degree of spin waiting before invoking WaitOne on the event, lazily allocating the event instead of doing it in the constructor (in case spinning is sufficient to avoid blocking altogether, as is demonstrated by the ThinEvent later in this column), adding reset capabilities, and providing a Dispose method so that the internal event object can be closed when no longer needed. These are all left as exercises for the reader.

Reusable Spin Wait

Though blocking is generally preferred to busy waiting, circumstances do arise in which you may want to spin for some time before falling back to a real wait. The reason that this can be useful is subtle, and most people initially avoid spin waiting because it seems like pure wasted work; if a context switch (which happens any time a thread waits on a kernel event) costs several thousands of cycles (we’ll call this c), which it does on Windows, and if the condition the thread is waiting for occurs in fewer than 2c cycles time (1c for the wait itself, 1c for waking up), then spinning could decrease the overhead and latency caused by waiting, boosting your algorithm’s overall throughput and scalability.

Once you decide to use a spin wait, you must tread with care. There are many issues that may come up: making sure you call Thread.SpinWait inside the spin loop to improve hardware availability to other hardware threads on Intel Hyper-threaded machines; calling Thread.Sleep with an argument of 1 instead of 0 once in a while to avoid priority inversion; using a slight back-off to introduce randomization, improving locality (assuming the caller is continuously rereading shared state), and possibly avoiding livelock; and, of course, always yielding on a single-CPU machine (since spinning in such environments is entirely wasteful).

The SpinWait class is defined as a value type so that it’s cheap to allocate (see Figure 2). We can now use this algorithm to avoid blocking in the CountdownLatch algorithm shown previously:

Figure 2 SpinWait (Close Figure 2)
public struct SpinWait {
    private int m_count;
    private static readonly bool s_isSingleProc =
        (Environment.ProcessorCount == 1);
    private const int s_yieldFrequency = 4000;
    private const int s_yieldOneFrequency = 3*s_yieldFrequency;

    public int Spin() {
        int oldCount = m_count;

        // On a single-CPU machine, we ensure our counter is always
        // a multiple of ‘s_yieldFrequency’, so we yield every time.
        // Else, we just increment by one.
        m_count += (s_isSingleProc ? s_yieldFrequency : 1);

        // If not a multiple of ‘s_yieldFrequency’ spin (w/ backoff).
        int countModFrequency = m_count % s_yieldFrequency;
        if (countModFrequency > 0)
            Thread.SpinWait((int)(1 + (countModFrequency * 0.05f)));
        else
            Thread.Sleep(m_count <= s_yieldOneFrequency ? 0 : 1);

        return oldCount;
    }

    private void Yield() {
        Thread.Sleep(m_count < s_yieldOneFrequency ? 0 : 1);
    }
}

private const int s_spinCount = 4000;
public void Wait() {
    SpinWait s = new SpinWait();
    while (m_remain > 0) {
        if (s.Spin() >= s_spinCount) m_event.WaitOne();
    }
}

The choice of yield frequency and spin count is admittedly arbitrary. Just like Win32 critical section spin counts, these numbers should be chosen based on testing and experimentation, and the right answer is apt to differ from one system to the next. The MSDN documentation, for instance, recommends a spin count of 4,000 for critical sections based on experience from the Microsoft Media Center and Windows kernel teams, but your mileage will vary. The perfect number depends on many factors, including the number of threads waiting for an event at any given time, the frequency with which events occur, and so on. In most cases, you’ll want to eliminate the explicit yields by waiting on an event instead, as shown with the latch example.

You can even choose a dynamically adjusting count: for example, start at a medium number of spins, and each time spinning is unsuccessful, increment the count. Once the count reaches a predetermined maximum, stop spinning altogether and just issue the WaitOne right away. The logic is as follows: you are willing to burn up to the predetermined maximum number of cycles, but no more. If you find that this maximum is not enough to prevent the context switch, then doing the context switch right away will be cheaper in total. Over time, you would hope the spin count reaches a stable value.

Barriers

A barrier, also known as a rendezvous point, is a concurrency primitive that permits threads to coordinate with each other without the need for another "master" thread orchestrating everything. Each of the threads signals and waits, atomically, once it reaches the barrier. All threads are allowed to proceed only when all n have reached it. This can be used for cooperative algorithms, as are common in scientific, mathematical, and graphics domains. Using barriers is suitable in many computations; in fact, the CLR’s garbage collector even uses them. Barriers simply break a bigger computation into smaller, cooperative phases, for example:

const int P = ...;
Barrier barrier = new Barrier(P);
Data[] partitions = new Data[P];

// Running on ‘P’ separate threads in parallel:
public void Body(int myIndex) {
    FillMyPartition(partitions[myIndex]);
    barrier.Await();
    ReadOtherPartition(partitions[P – myIndex - 1]);
    barrier.Await();
    // ...
}

You’ll quickly notice that a countdown latch could be used in this situation. Instead of calling Await, each thread could call Signal followed immediately by a call to Wait; all threads will be released once they all reach the barrier. But there is a problem: our earlier latch doesn’t support reusing the same object multiple times, a handy property all barriers should support. The above example, in fact, requires this. You can use separate barrier objects to achieve this, but that would be wasteful; there’s no need for more than one since all threads are only ever in one phase at a time.

To solve this problem, you can start with the same basic countdown latch algorithm to deal with decrementing the counter, signaling the event, waiting, and so forth, extending it to support reuse. To do this, you need to use a so-called sense reversing barrier, which requires alternating between "even" and "odd" phases. You use a separate event for the alternative phases. Figure 3 is a sample implementation of such a Barrier data structure.

Figure 3 Barrier (Close Figure 3)
using System;
using System.Threading;

public class Barrier {
    private volatile int m_count;
    private int m_originalCount;
    private EventWaitHandle m_oddEvent;
    private EventWaitHandle m_evenEvent;
    private volatile bool m_sense = false; // false==even, true==odd.

    public Barrier(int count) {
        m_count = count;
        m_originalCount = count;
        m_oddEvent = new ManualResetEvent(false);
        m_evenEvent = new ManualResetEvent(false);
    }

    public void Await() {
        bool sense = m_sense;

        // The last thread to signal also sets the event.
        if (m_count == 1 || Interlocked.Decrement(ref m_count) == 0) {
            m_count = m_originalCount;
            m_sense = !sense; // Reverse the sense.
            if (sense == true) { // odd
                m_evenEvent.Reset();
                m_oddEvent.Set();
             } else { // even
                m_oddEvent.Reset();
                m_evenEvent.Set();

            }
        } else {
            if (sense == true) m_oddEvent.WaitOne();
            else               m_evenEvent.WaitOne();
        }

    }
}

The reason you need two events is subtle. One approach would be to do a Set immediately followed by Reset in Await, but this is dangerous and will lead to deadlocks for two reasons. First, another thread may have decremented m_count, but not yet reached the call to WaitOne on the event when the thread calls Set and then Reset in quick succession. Second, though the waiting thread might have reached the call to WaitOne, alertable waits, like the ones the CLR always uses, can interrupt the wait, temporarily removing a waiting thread from the wait queue so it can run an asynchronous procedure call (APC). The waiting thread will never see the event in a set state. Both cases lead to missed events and probable deadlocks. Avoid this by using separate events for odd and even phases.

You might want to go ahead and add spinning to the Barrier, just as with the CountdownLatch. But you’ll encounter a problem if you try: normally the spinning thread would spin until it saw an m_count of 0. With the implementation above, however, m_count will never actually reach 0 before the last thread resets it to m_originalCount. The naïve approach to spinning would lead to one or more threads spinning (forever) and all other threads being blocked (forever) on the next phase. The solution is easy. You spin, waiting for the sense to change, as shown in Figure 4.

Figure 4 Waiting (Close Figure 4)
public void Await() {
    bool sense = m_sense;

    // The last thread to signal also sets the event.
    if (m_count == 1 || Interlocked.Decrement(ref m_count) == 0) {
        m_count = m_originalCount;
        m_sense = !sense; // Reverse the sense.
        if (sense == true) { // odd
            m_evenEvent.Set();
            m_oddEvent.Reset();
        } else { // even
            m_oddEvent.Set();
            m_evenEvent.Reset();
        }
    } else {
        SpinWait s = new SpinWait();
        while (sense == m_sense) {
            if (s.Spin() >= s_spinCount) {
                if (sense == true) m_oddEvent.WaitOne();
                else               m_evenEvent.WaitOne();
            }
        }
    }
}

Because all threads must leave the Await from a previous phase before a subsequent phase can complete, you can be assured that all threads will either observe that the sense changed or will end up waiting on our event and being awakened that way.

Blocking Queue

In shared-memory architectures, the sole point of synchronization between two or more tasks is often a central, shared-collection data structure. Frequently one or more tasks are responsible for generating "work" for one or more other tasks to consume, referred to as a producer/consumer relationship. Simple synchronization for such data structures is typically straightforward—using a Monitor or ReaderWriterLock will do the trick—but coordination among tasks when the buffer becomes empty is harder. This problem is normally solved with a blocking queue.

Actually, there are several slight variants of blocking queues, ranging from the simple—where the consumer blocks only when the queue is empty—to the complex—in which each producer is "paired" with exactly one consumer; that is, the producer is blocked until a consumer arrives to process the enqueued item and, similarly, the consumer is blocked until a producer delivers an item. FIFO (first in first out) ordering is common, but not always necessary. Buffers can also be bounded, as we’ll see later. We only look at the pairing variant here, since the bounded buffer shown later includes the simpler blocking-when-empty behavior.

To implement this, just wrap a simple Queue<T> with synchronization on top. What kind of synchronization? Whenever a thread enqueues an element, it waits for a consumer to dequeue the element before returning. When a thread dequeues an element, should it find the buffer to be empty, it must wait for a new element to arrive. And of course after dequeueing, the consumer must signal the producer that it has taken its item (see Figure 5).

Figure 5 Blocking Queue (Close Figure 5)
class Cell<T> {
    internal T m_obj;
    internal Cell(T obj) { m_obj = obj; }
}

public class BlockingQueue<T> {
    private Queue<Cell<T>> m_queue = new Queue<Cell<T>>();
    public void Enqueue(T obj) {
        Cell<T> c = new Cell<T>(obj);
        lock (m_queue) {
            m_queue.Enqueue(c);
            Monitor.Pulse(m_queue);
            Monitor.Wait(m_queue);
        }
    }
    public T Dequeue() {
        Cell<T> c;
        lock (m_queue) {
            while (m_queue.Count == 0)
                Monitor.Wait(m_queue);
            c = m_queue.Dequeue();
            Monitor.Pulse(m_queue);
        }
        return c.m_obj;
    }
}

Notice that we call Pulse and then Wait in the Enqueue method, and similarly Wait and then Pulse in Dequeue. Because of the way monitors are implemented (internal events are set before the monitor is released), this can cause some thread scheduling ping-pong. We might consider building a finer-grained notification mechanism instead, perhaps using Win32 events. However, using full-fledged Win32 events like this can introduce quite a bit of overhead—particularly the allocation cost and kernel transitions when using them—so it might be time to consider alternatives. You could pool them, just as the CLR’s ReaderWriterLock does, or lazily allocate them, as our ThinEvent type (shown later) does. This implementation also has the disadvantage of allocating an object for each new element; an alternative approach might be to pool these objects too, but that also comes with some additional complexities.

Bounded Buffer

A resource consumption problem can arise in some kinds of queues. If the producer task or tasks create items at a rate faster than the consumer can process them, the system can run into unbounded memory usage.

To illustrate this, imagine a system in which a single producer enqueues 50 items/second, and the consumer only consumes at a rate of 10 items/second. First, the system is imbalanced and will not scale well with a 1-to-1 producer-to-consumer configuration. After just one minute, 2,400 items will have piled up in the buffer. If these items consume, say, 10KB apiece, then this represents 24MB of memory just for the buffer itself. After an hour, this will have grown to over 1GB. Weighting the number of producer threads to consumer threads is one solution, which, in this case, means a ratio of one producer to five consumers. But arrival rates are often volatile, causing periodic imbalances and resulting in dramatic problems; a simple, fixed ratio won’t solve this.

On a server, where programs are often long-running and expected to exhibit good uptimes, the potential for unbounded memory usage can wreak absolute havoc, possibly leading to a situation in which the server process must be recycled regularly.

A bounded buffer allows you to place a limit on the size that the buffer may reach before the producer is forced to block. Blocking the producer gives the consumer a chance to "catch up" (by allowing its thread to receive a scheduling timeslice) while at the same time eliminating the memory build-up issue. Our approach, again, is to simply wrap a Queue<T>, adding two wait conditions and two event notification conditions: a producer waits when the queue is full (until it becomes non-full) and a consumer waits when the queue is empty (until it becomes non-empty); a producer signals waiting consumers when it has produced an item, and a consumer signals a producer when it has taken an item, as shown in Figure 6.

Figure 6 Bounded Buffer (Close Figure 6)
public class BoundedBuffer<T> {
    private Queue<T> m_queue = new Queue<T>();
    private int m_consumersWaiting;
    private int m_producersWaiting;
    private const int s_maxBufferSize = 128;

    public void Enqueue(T obj) {
        lock (m_queue) {
            while (m_queue.Count == (s_maxBufferSize - 1)) {
                m_producersWaiting++;
                Monitor.Wait(m_queue);
                m_producersWaiting--;
            }
            m_queue.Enqueue(obj);
            if (m_consumersWaiting > 0)
                Monitor.PulseAll(m_queue);
        }
    }

    public T Dequeue() {
        T e;
        lock (m_queue) {
            while (m_queue.Count == 0) {
                m_consumersWaiting++;
                Monitor.Wait(m_queue);
                m_consumersWaiting--;
            }
            e = m_queue.Dequeue();
            if (m_producersWaiting > 0)
                Monitor.PulseAll(m_queue);
        }
        return e;
    }
}

Again, a somewhat naïve approach has been taken. But we do optimize the calls to PulseAll—since they are anything but cheap—by maintaining two counters, m_consumersWaiting and m_producersWaiting, and only signaling if the respective value is non-zero. There are further opportunities for improvement. For example, sharing a single event like this might wake up too many threads: if a consumer reduces the queue’s size to 0, and there are both waiting producers and waiting consumers, clearly you want to wake only the producers (at least at first). This implementation will service all waiters in FIFO order, meaning you might have to wake the consumers before any producer runs, only for them to find out that the queue is empty and subsequently wait again. Thankfully, ending up with simultaneous waiting producers and consumers is quite rare, but it can happen with regularity with small bounds sizes.

Thin Event

Win32 events have one big advantage over Monitor.Wait, Pulse, and PulseAll: they are "sticky." That means once an event has been signaled, any subsequent waits will immediately unblock, even if the threads had not begun waiting before the signal occurred. Without this feature, you frequently need to either write inefficient code where all waiting and signaling happens strictly inside a critical region—this is inefficient because the Windows scheduler always boosts an awoken thread’s priority, always incurring a context switch only for the waking thread to immediately wait again for the critical region—or you must employ some tricky, race-condition-prone code.

Instead of either approach, you can instead use a "thin event," a reusable data structure that spins briefly before blocking, lazily allocates a Win32 event only when necessary, and otherwise gives you manual-reset-event-like behavior. In other words, it encapsulates that tricky race-condition-prone code so that you don’t have to scatter it throughout your entire code-base. This example relies on some memory model guarantees described in Vance Morrison’s article, and should be used with extreme care (see Figure 7).

Figure 7 Thin Event (Close Figure 7)
public struct ThinEvent {
    private int m_state; // 0 means unset, 1 means set.
    private EventWaitHandle m_eventObj;
    private const int s_spinCount = 4000;

    public void Set() {
        m_state = 1;
        Thread.MemoryBarrier(); // required.
        if (m_eventObj != null)
            m_eventObj.Set();
    }

    public void Reset() {
        m_state = 0;
        if (m_eventObj != null)
            m_eventObj.Reset();
    }

    public void Wait() {
        SpinWait s = new SpinWait();
        while (m_state == 0) {
            if (s.Spin() >= s_spinCount) {
                if (m_eventObj == null) {
                    ManualResetEvent newEvent =
                        new ManualResetEvent(m_state == 1);
                    if (Interlocked.CompareExchange<EventWaitHandle>(
                            ref m_eventObj, newEvent, null) == null) {
                       // If someone set the flag before seeing the new
                       // event obj, we must ensure it’s been set.
                       if (m_state == 1)
                           m_eventObj.Set();
                    } else {
                        // Lost the race w/ another thread. Just use
                        // its event.
                        newEvent.Close();
                    }
                }
                m_eventObj.WaitOne();
            }
        }
    }
}

This basically mirrors the event state in an m_state variable, where the value 0 means unset and 1 means set. Waiting on a set event is now really cheap; if m_state is 1 in the entrance to the Wait routine, we return immediately, no kernel transitions needed. The trickiness arises when a thread waits before the event has been set. The first thread to wait must allocate a new event object and compare-and-swap it into the m_eventObj field; if the CAS fails, this means another waiter initialized the event, so we can just reuse it; otherwise, we must recheck that m_state hasn’t changed since we saw it last. Otherwise, m_state could be 1 and m_eventObj could be unsignaled, leading to a deadlock when we call WaitOne. The thread calling Set must set m_state first and then, if it sees a non-null m_eventObj, call Set on it. Two memory barriers are required: the second read of m_state must not be moved earlier, which we guarantee with the use of Interlocked.CompareExchange to set m_eventObj; and the read of m_eventObj in Set mustn’t be moved before the write to m_eventObj (a somewhat surprising legal transformation on some Intel and AMD processors, and the CLR 2.0 memory model, without the explicit call to Thread.MemoryBarrier). Resetting the event is generally not safe to do in parallel, so additional synchronization is needed by the caller.

You can now easily use this elsewhere, such as in the CountdownLatch and the queue examples above, typically with a sizeable performance gain, particularly if you use spinning intelligently.

We showed a tricky implementation above. Note that you can implement both auto- and manual-reset types using a single flag and monitors, considerably simpler than this example (but not always as efficient).

Lock-Free LIFO Stack

Building a thread-safe collection using locks is pretty straightforward, even when bounding and blocking complicate things (as we saw above). When all coordination occurs via a simple last-in-first-out (LIFO) stack data structure, however, using a lock could be more expensive than is absolutely required. A thread’s critical region, the time during which a lock is held, has a beginning and an end, the duration of which may span many instructions’ worth of time. Holding the lock prevents other threads from reading and writing simultaneously. This achieves serialization, which is of course what we want, but is strictly stronger than we need: we’re just pushing and popping elements on and off a stack, both of which can be achieved with regular reads and a single compare-and-swap write. We can exploit this fact to build a more scalable, lock-free stack that doesn’t force threads to wait unnecessarily.

Our algorithm works as follows. You use a linked list to represent the stack, the head of which represents the top of the stack, stored in the m_head field. When pushing a new item onto the stack, you construct a new node with the value you’re going to push onto the stack, read the m_head field locally, store it into the new node’s m_next field, and then do an atomic Interlocked.CompareExchange to replace the stack’s current head. If the head has changed at any point in this sequence (since it was first read), the CompareExchange will fail, and the thread must loop back around and try the whole sequence again. Popping is similarly straightforward. You read the m_head and try to swap it with our local copy’s m_next reference; if it fails, you just keep trying, as you see in Figure 8. Win32 offers an analogous data structure, called an SList, built using a similar algorithm.

Figure 8 Lock-Free Stack (Close Figure 8)
public class LockFreeStack<T> {
    private volatile StackNode<T> m_head;

    public void Push(T item) {
        StackNode<T> node = new StackNode<T>(item);
        StackNode<T> head;
        do {
            head = m_head;
            node.m_next = head;
        } while (m_head != head || Interlocked.CompareExchange(
                ref m_head, node, head) != head);
    }

    public T Pop() {
        StackNode<T> head;
        SpinWait s = new SpinWait();

        while (true) {
            StackNode<T> next;
            do {
                head = m_head;
                if (head == null) goto emptySpin;
                next = head.m_next;
            } while (m_head != head || Interlocked.CompareExchange(
                    ref m_head, next, head) != head);
            break;

        emptySpin:
            s.Spin();
        }

        return head.m_value;
    }
}

class StackNode<T> {
    internal T m_value;
    internal StackNode<T> m_next;
    internal StackNode(T val) { m_value = val; }
}

Note that this is a form of optimistic concurrency control: instead of blocking other threads from accessing data, you just proceed with the hope that you’ll "win" the race. If this hope turns out to be wrong, you can encounter liveness issues such as livelock. This design choice also implies that you can’t reliably achieve FIFO scheduling. All threads in the system will probabilistically make forward progress. And in fact, our system as a whole has deterministic forward progress, since the failure of one thread always means at least one other thread has made progress (a requirement for calling this "lock-free"). Sometimes it is useful to use an exponential backoff when a CompareExchange fails to avoid massive memory contention on m_head.

Also, we’ve taken a rather naïve approach to the case in which the stack has become empty. We just spin forever, waiting for a new item to be pushed. It’s straightforward to rewrite Pop into a non-waiting TryPop method, and a bit more complex to make use of events for waiting. Both are important features and left as exercises for the motivated reader.

We incur an object allocation for each Push, saving us from having to worry about so-called ABA problems. ABA occurs due to the internal reuse of nodes that have been popped off the list. Developers sometimes will try to pool nodes to reduce the number of object allocations, but this is problematic: the result can be that an atomic operation erroneously succeeds even though there have been a number of intervening writes to m_head. (For example: node A is read by thread 1, then removed by thread 2 and placed into the pool; B is pushed as the new head by thread 2, then A returned from the pool to thread 2 and pushed; thread 1 then succeeds with its CompareExchange, even though the A now at the head is different from the one it read.) A similar problem occurs if you try to write this algorithm in native C/C++; because the memory allocator can reuse addresses as soon as they are freed, a node can be popped, freed, and then its address can be subsequently handed out to a new node allocation, causing the same problem. We’ll omit any further discussion of ABA here—it has been described at length elsewhere.

Lastly, it’s possible to write a FIFO queue using similar lock-free techniques. This is attractive because threads pushing and popping concurrently will not necessarily conflict with one another, in contrast with the LockFreeStack shown above in which pushers and poppers are always fighting for the same m_head field. The algorithm is rather complex, however, so if you’re curious, I encourage you to take a look at Maged M. Michael and Michael L. Scott’s 1996 paper, "Simple, Fast, and Practical Non-Blocking and Blocking Concurrent Queue Algorithms".

Loop Tiling

Loop tiling is the practice of partitioning the input range or data for a loop and assigning each partition to a separate thread in order to achieve concurrency. This is the primary technique for achieving parallelism with some programming models, like OpenMP (see Kang Su Gatlin’s MSDN Magazine article, and is often referred to as a parallel forall loop, inspired by high-performance FORTRAN’s terminology. No matter if the range is just a set of indices:

for (int i = 0; i < c; i++) { ... }
or a range of data:

foreach (T e in list) { ... }
we can devise partitioning techniques that give us a tiled loop.

There are many data structure-specific partitioning techniques you might want to apply, certainly too many to enumerate in this column. So we’ll just focus on a common technique, which is to assign disjoint ranges of elements from the array to each partition. We simply calculate a stride which is roughly the number of elements divided by the number of partitions, and use that to calculate contiguous ranges (see Figure 9). This yields good spatial locality when the input is an array of value types, though other approaches are certainly valid, useful, and sometimes necessary.

Figure 9 Loop Tiling (Close Figure 9)
public static void ForAll(int from, int to, Action<int> a, int p) {
    ForAll<int>(null, from, to, null, a, p);
}
public static void ForAll<T>(IList<T> data, Action<T> a, int p) {
    ForAll<T>(data, 0, data.Count, a, null, p);
}

private static void ForAll<T>(IList<T> data, int from, int to,
        Action<T> a0, Action<int> a1, int p) {
    int size = from - to;
    int stride = (size + p - 1) / p;
    CountdownLatch latch = new CountdownLatch(p);

    for (int i = 0; i < p; i++) {
        int idx = i;
        ThreadPool.QueueUserWorkItem(delegate {
            int end = Math.Min(size, stride * (idx + 1));
            for (int j = stride * idx; j < end; j++) {
                if (data != null) a0(data[j]);
                else              a1(j);
            }

            latch.Signal();
        });
    }

    latch.Wait();
}

We offer two public versions of ForAll here: one that accepts a range of numbers, and the other that accepts an IList<T>, just like a foreach loop in C#. Both forward to the same helper overload, which invokes either an action passing the element from the list at the given index or passes the index itself. You could use the first overload where you’d typically place a normal for loop. For example, this code

for (int i = 0; i < 10; i++) { S; }
becomes:

Parallel.ForAll(0, 10, delegate(int i){ S; },
    Environment.ProcessorCount);
And you can use the second where you’d typically place a C# foreach loop, so that

List<T> list = ...;
foreach (T e in list) { S; }
becomes:

Parallel.ForAll(list, delegate(T e) { S; },
    Environment.ProcessorCount);

You need to be careful that no statements in S write to shared memory; otherwise you will need to add proper synchronization to the parallel versions. Versions can of course be written to accommodate any IEnumerable<T>, partition the iteration space in a different way, and so forth (all of which has been omitted from this column to conserve space). And in this example, the calling thread is "wasted" for the duration of the n subtasks. A better approach would be to use the calling thread to run one of the tasks itself, and then join with the others upon its completion. Extending the ForAll method to do this is trivial.

Parallel Reductions

There is a category of operations that can be performed using a reduction (also known as a fold or aggregation) where many values are combined in some way to produce a single output. A general reduction works as follows. You take a binary operator—that is, a function with two arguments—and compute it over a vector or set of elements of size n, from left to right. For j = 0 to n - 1, you invoke the binary operator, passing as input to the jth iteration the output of invoking the operator on element j - 1 as the first argument and the jth element itself as the second argument. A special seed value is used for the first argument to the 0th element, since there is no previous value to use. A final (and optional) result selector is then used to convert an intermediate value into a final result.

Let’s see an example of this. If the binary operator is + and the input is a vector of 5 elements, {1, 2, 3, 4, 5}, then the expanded computation looks like ((((1 + 2) + 3) + 4) + 5). If you convert this expansion into function-calling form, it looks like this (assuming 0 for a seed): +(+(+(+(+(0, 1), 2), 3), 4), 5) In other words, you just compute the sum of all of the input numbers. This is called a sum reduction. A straightforward translation of this generalized algorithm into a serial algorithm might look like this:

delegate T Func<T>(T arg0, T arg1);

T Reduce<T>(T[] input, T seed, Func<T> r) {
    T result = seed;
    foreach (T e in input)
        result = r(result, e);
    return result;
}
When invoking it, you can now simply sum up a set of numbers like this (in C# 3.0):

int[] nums = ... some set of numbers ...;
int sum = Reduce(nums, 0, (x,y) => x + y;);
All this is pretty abstract but, in addition to sum, many operations can be expressed as a reduction, as you see in Figure 10.

Figure 10 Operations Expressed as Reductions (Close Figure 10)

 | Seed | Binary Operator | Result Selector
Count | 0 | (a, b) => a + 1 | N/A
Sum | 0 | (a, b) => a + b | N/A
Min | NaN | (a, b) => a < b ? a : b | N/A
Max | NaN | (a, b) => a > b ? a : b | N/A
Average | { 0, 0 } | (a, b) => new { a[0] + b, a[1] + 1 } | (a) => a[0] / a[1]

Given the nums array above, we can use our reduction routine to find the min and max of the array:

int min   = Reduce(nums, int.MaxValue, (x,y) => x < y ? x : y;);
int max   = Reduce(nums, int.MinValue, (x,y) => x > y ? x : y;);
(Count is omitted because the partial results must be summed, requiring two separate binary operators, and Average is omitted because it also requires some extra steps.)

You can use a similar technique to the one discussed for loop tiling to break apart the input data and perform the reduction in parallel. Each partition will compute its own intermediate value that we’ll then combine into a single final value, using the same operator used to calculate the intermediate values. Why is this possible? Because all of the operations mentioned above are associative; recall from your elementary mathematics that an associative binary operator + simply means that (a + b) + c = a + (b + c); that is, the order of evaluation doesn’t matter to the correctness of the calculation.

For example, consider the sum reduction. If you partition the input data { 1, 2, 3, 4 } into two partitions, {1, 2} and {3, 4}, then, because + is associative, when you add the results of the independent additions, the result is the same: (1 + 2) + (3 + 4) = ((((1 + 2) + 3) + 4). In fact, any disjoint partitioning of the input yields correct results. Figure 11 shows a generalized Reduce method which takes the seed and binary operator as arguments, using a stride partitioning approach as described earlier.

Figure 11 A Reduce Method (Close Figure 11)
public delegate T Func<T>(T arg0, T arg1);

public static T Reduce<T>(IList<T> data, int p, T seed, Func<T> r){
    T[] partial = new T[p];
    int stride = (data.Count + p - 1) / p;
      CountdownLatch latch = new CountdownLatch(p);

    for (int i = 0; i < p; i++) {
        int idx = i;
          ThreadPool.QueueUserWorkItem(delegate {
            // Do the ‘ith’ intermediate reduction in parallel.
            partial[idx] = seed;
            int end = Math.Min(data.Count, stride * (idx + 1));
            for (int j = stride * idx; j < end; j++)
                partial[idx] = r(partial[idx], data[j]);
            latch.Signal();
        });
    }

    latch.Wait();

    // Do the final reduction on the master thread.
     T final = seed;
    for (int i = 0; i < p; i++)
        final = r(final, partial[i]);

    return final;
}

The master thread forks off a set of p worker threads, each of which calculates an intermediate result by performing a reduction over its own partition of data and placing the value into its dedicated slot in the intermediates array. The master thread then waits for all children to finish (using the CountdownLatch from earlier), and performs a final reduction step using the partial results from each child as the input. In the literature, tree reductions are quite common, where each node in the tree does a partial reduction on a certain number of intermediate results. While this theoretically leads to a more scalable algorithm, with the number of threads you’re apt to deal with and the associated synchronization costs on Windows, the reality is that the serial reduction as shown performs better for common values of p and binary operators.

There are plenty of opportunities for optimization here—such as reusing the calling thread for one of the partitions—but nevertheless, the example should illustrate the point well enough. To support operations like average, where the intermediate reduction operation is different for the intermediary and final reduction stages, and where a final result selection routine is needed, we need a slightly different API. This is a fairly simple exercise.

Conclusion

In this column, I’ve shown you some low-level parallel data structures and algorithms that can help you write managed code that takes advantage of multi-processor and multi-core architectures. As with all programming, abstractions tend to clump together in layers, with the most performance-critical typically residing at the bottom. It’s safe to say that many of the techniques shown in this column are at the very bottom, serving as a foundation for higher-level abstractions and application-specific parallel code. Though selecting the right data structures and basic algorithms is only one step in a longer process, I hope you’ll come away with a deeper understanding of parallel programming techniques, which will be instrumental to your success in this new landscape of programming.

Send your questions and comments to clrinout@microsoft.com.

Joe Duffy works on parallel programming models and infrastructure at Microsoft, and blogs regularly at www.bluebytesoftware.com/blog. Some of the code samples in this column are based on his upcoming book, Concurrent Programming on Windows, to be released by Addison Wesley sometime in 2007.


Office Space: Features for SharePoint

Office Space
Features for SharePoint
Ted Pattison

Code download available at: OfficeSpaceFeature2007_05.exe (190KB)
Contents
What’s a “Feature”?
Structure of a Feature
Feature Deployment and Activation
Extending the Feature
Custom Application Pages
Summary

In my previous Office Space column, I discussed the Office Open XML file formats. In particular, I showed you how to write codebehind an ASP.NET page to generate a .docx file for Microsoft® Word using the packaging API provided by the Microsoft .NET Framework 3.0. This month, I am building upon what I’ve already shown you how to do with the Office Open XML file formats. I look at integrating this code into a business solution that can be deployed in either Windows® SharePoint® Services (WSS) 3.0 or Microsoft Office SharePoint Server (MOSS) 2007.

The focus of this month’s Office Space column is to introduce the fundamental building blocks you’ll use to create business solutions with WSS and MOSS. I explain what a WSS "feature" is and show you how to build one that automates the creation of new lists and document libraries within a site. Along the way I add a custom application page to the custom business solution. This contains the Visual Basic® code needed to generate Word documents from data within SharePoint lists and save these documents within a SharePoint document library.

What’s a “Feature”?

When developing custom business solutions for WSS and MOSS, the first thing you need to do is gain a firm understanding of features (and I’m not referring to the general feature set). Features are a new developer-focused innovation that has been added to WSS 3.0. Basically, this provides a mechanism for defining site elements, which can be automatically activated within the context of a target site or a target site collection. The element types that can be defined by a feature include list instances, list types, menu commands, page templates, page instances, event handlers, and workflows. The example I discuss in this column involves a feature, named OfficeSpaceFeature, that creates a list and a document library and adds a custom menu item to the standard WSS Site Actions menu, allowing the user to navigate to a custom application page.

A feature consists of a directory created within a special WSS system directory located within the file system of each front-end Web server. The directory for a feature contains one or more XML-based files that contain Collaborative Application Markup Language (CAML). By convention, each feature directory contains a manifest file named feature.xml. This file defines the high-level attributes of the feature, such as its ID and its user-friendly Title.

Besides the feature.xml file, a feature usually contains one or more additional XML files (elements.xml, for instance) that define the actual elements that make up the feature. The directory can also contain other types of files for things like list definitions and page templates as well as resources like images, cascading style sheets, and JavaScript. The feature directory in my example includes an extra file that serves as the document template for a document library.

One good technique to help you get up to speed with features is to examine the standard set of features included with the basic WSS installation. Figure 1 shows an example of what the FEATURES directory looks like after you’ve installed WSS. As you can see, each feature has its own directory. The FEATURES directory will look very different if you have MOSS installed, since this will include more than 100 features.

Figure 1 Standard FEATURES Directory
Figure 1 Standard FEATURES Directory(Click the image for a smaller view)
Figure 1 Standard FEATURES Directory
Figure 1 Standard FEATURES Directory (Click the image for a larger view)

After you’ve created the directory with all the files that make up your feature, you must then install it with WSS at a farm-level scope. Once a feature has been installed with WSS, it can then be activated within the context of a site or site collection using an administrative page accessible through the Site Settings page.

The download for this column contains a Visual Studio® project named OfficeSpaceFeature, which contains a feature by the same name. Figure 2 shows a view of the Solution Explorer to give you an idea of what types of files are involved when you create a Visual Studio project to develop a custom feature for WSS or MOSS. I have created this column’s example as a Visual Basic DLL project. You can, of course, use a C# DLL project instead if you prefer. The managed code inside the assembly DLL must be compiled with a strong name and installed in the Global Assembly Cache (GAC) so it can be used for the feature’s event handlers.

Figure 2 OfficeSpaceFeature
Figure 2 OfficeSpaceFeature

Structure of a Feature

Before I examine the feature.xml file, consider that the files for this feature must be deployed in their own special directory inside the WSS system directory named FEATURES. The FEATURES directory is located within another WSS system directory named TEMPLATE, as shown in Figure 1. Given the requirements of feature deployment, it makes sense to create a parallel hierarchy of folders within a Visual Studio project used to develop a WSS feature. This makes it easier to copy the feature files to the correct location and test them as you do your development work.

Start by adding a folder named TEMPLATE to the root directory of the current project. Then create another directory inside TEMPLATE named FEATURES. Finally, create another directory inside FEATURES using the name of the feature project. In this case, the name of the project and directory is OfficeSpaceFeature, as you can see in Figure 2.

Now onto the CAML content inside the feature.xml file. The feature.xml file serves as a feature manifest where you specify the information that defines the high-level attributes of the feature itself. The feature.xml file for my example feature looks like that shown in Figure 3.

Figure 3 Sample Feature.xml File (Close Figure 3)
<?xml version=”1.0” encoding=”utf-8” ?>

<Feature Id=”AAEC2E08-1CCF-4712-AE5E-A33BEA53A325”
  Title=”A Sample Office Space Feature”
  Description=
    “Demoware from Ted Pattison’s Office Space column in MSDN Magazine”
  Version=”1.0.0.0”
  Scope=”Web”
  ImageUrl=”OfficeSpace/PithHelmet.gif”
  ReceiverAssembly=”OfficeSpaceFeature, [full strong name]”
  ReceiverClass=”OfficeSpaceFeature.FeatureReceiver”
  xmlns=”http://schemas.microsoft.com/sharepoint/”>

    <ElementManifests>
      <ElementManifest Location=”elements.xml” />
    </ElementManifests>

</Feature>

You can see that a feature is defined using a Feature element that contains several attributes, such as Id, Title, Description, Version, Scope, Hidden, and ImageUrl. You must create a new GUID for the Id attribute so your feature can be uniquely identified. You create the feature’s Title and Description attributes using user-friendly text. These attributes are displayed directly to users on the WSS administrative pages that are used to activate and deactivate features.

The Scope defines the context in which the feature can be activated and deactivated. The feature I am creating has a scope equal to Web, which means it can be activated and deactivated within the context of a site. If, instead, you assign a Scope value of Site, your feature can then be activated and deactivated within the scope of a site collection. The two other possible scopes for defining a feature are WebApplication scope and Farm scope.

As you can see, the Hidden attribute has a value of FALSE. This means that, once installed within the farm, the feature can be seen by users who might want to activate it. If you create a feature with the Hidden attribute value set to TRUE, the feature will be hidden in the list of available features shown to users. Hidden features must be activated from the command line, through custom code or through an activation dependency with another feature.

You may have noticed that the ImageUrl attribute has a value that points to one of the graphic images that is part of the basic WSS installation. This image will be shown next to the feature in the UI.

The last part of the feature.xml file shown in Figure 3 is the ElementManifests element. This element contains inner ElementManifest elements that reference other XML files where you will define the elements that make up the feature. In this case, there is a single ElementManifest element that uses the location attribute to point to a file named element.xml.

Note that when you add and modify XML within CAML-based files, such as feature.xml and elements.xml, you want to add support for XML schema-driven IntelliSense®. Inside the TEMPLATE directory there is a directory named XML that contains several XML schemas including one named wss.xsd. If you associate this schema file with feature files, like feature.xml and elements.xml, Visual Studio will provide IntelliSense, making it much easier to author a custom feature.

Now it’s time to examine what’s inside the element.xml file. Say you want to instantiate a document library whenever the feature is activated. You can create an elements.xml file that looks like the one shown in Figure 4.

Figure 4 Sample Elements.xml File (Close Figure 4)
<Elements xmlns=”http://schemas.microsoft.com/sharepoint/”>

  <ListInstance
   FeatureId=”00BFEA71-E717-4E80-AA17-D0C71B360101”
   TemplateType=”101”
   Id=”CustomerLetters”
   Description=”Letters sent to customers”
   OnQuickLaunch=”True”
   Title=”Customer Letters”
   Url=”CustomerLetters”>
  </ListInstance>

  <Module Name=”LetterTemplate” List=”101” Url=”CustomerLetters/Forms”>
    <File Url=”LetterTemplate.docx” Type=”GhostableInLibrary” />
  </Module>

  <!-- more elements to come... -->

</Elements>

The first element shown here, a ListInstance element, is used to create an instance of a list or a document library. Note that the ListInstance contains a FeatureId and a TemplateType to point to a specific list type and the ID for the feature in which that list type is defined. In this case, I am using the standard WSS document library type, which has a TemplateType identifier of 101 to create a new document library with a title of Customer Letters. Under the ListInstance element, there is also a Module element that contains an inner File element. This File element is used to provision a document template for the document library. More precisely, this File element provides WSS with the instructions to copy a file named LetterTemplate.docx from the feature directory into the WSS site. This makes it accessible to users.

While the declarative logic in elements.xml goes a long way to provide what you need to create elements within a WSS site, it often needs to be accompanied by programmatic logic. For my example, I have created a new document library and provisioned a file to serve as the document template. However, I need to supply some extra code that programs against the WSS object model to associate the document template with the document library.

In my feature.xml file, you’ll notice that the Feature element contains two attributes named ReceiverAssembly and ReceiverClass. These attributes are configured to point to a managed class known as a feature receiver, which supplies event handlers. The event handlers within a feature receiver class fire when a feature is installed or activated as well as when a feature is uninstalled or deactivated. You create a feature receiver by creating a class that inherits from the SPFeatureReceiver class as shown in Figure 5.

Figure 5 Class that Inherits from SPFeatureReceiver (Close Figure 5)
Imports Microsoft.SharePoint

Public Class FeatureReceiver : Inherits SPFeatureReceiver

  Public Overrides Sub FeatureActivated( _
      ByVal properties As SPFeatureReceiverProperties)
    ‘*** fires when feature is activated
  End Sub

  Public Overrides Sub FeatureDeactivating( _
      ByVal properties As SPFeatureReceiverProperties)
    ‘*** fires when feature is deactivated
  End Sub

  Public Overrides Sub FeatureInstalled( _
      ByVal properties As SPFeatureReceiverProperties)
    ‘*** fires when feature is installed
  End Sub

  Public Overrides Sub FeatureUninstalling( _
      ByVal properties As SPFeatureReceiverProperties)
    ‘*** fires when feature is uninstalled
  End Sub

End Class

For my sample, I am only supplying code inside the FeatureActivated event handler. This code uses the WSS object model to associate the document template with the Customer Letters document library. Inside the event handler this is done by obtaining an SPWeb reference to the current site and then an SPDocumentLibrary reference to the target document library. This makes it possible to modify the DocumentTemplateUrl property and save this change using a call to the Update method, like so:

Public Overrides Sub FeatureActivated( _
    ByVal properties As SPFeatureReceiverProperties)

  Dim site As SPWeb = CType(properties.Feature.Parent, SPWeb)

  Dim libLetterTemplates As SPDocumentLibrary
  libLetterTemplates = CType(site.Lists(“Customer Letters”), _
    SPDocumentLibrary)
  Dim templateUrl As String = _
    “CustomerLetters/Forms/LetterTemplate.docx”
  libLetterTemplates.DocumentTemplateUrl = templateUrl
  libLetterTemplates.Update()

End Sub

Feature Deployment and Activation

Now that I have created both the feature.xml file and the elements.xml file, I want to install my feature for testing purposes. There are four steps involved in installing my feature. First, I must install the assembly file named OfficeSpaceFeature.dll into the GAC. Then I have to copy the OfficeSpaceFeature directory to the WSS system FEATURES directory. The next step is to run an Stsadm.exe operation to install the feature with WSS. Finally, I need to activate the feature inside the context of a WSS site.

I can automate the first three steps by creating a batch file named install.bat at the root directory of my OfficeSpaceFeature project and adding the command-line instructions shown in Figure 6. If I wanted, I could also automate the fourth step by running the ActivateFeature operation with the Stsadm.exe utility. However, I’ve opted not to do this in my example because I want to take you through the process of manually activating the feature, as users will do through the WSS user interface.

Figure 6 Commands for Automating Installation (Close Figure 6)
@SET TEMPLATEDIR=”c:\Program Files\Common Files\Microsoft Shared\
                  web server extensions\12\TEMPLATE”
@SET STSADM=”c:\Program Files\Common Files\Microsoft Shared\
             web server extensions\12\bin\stsadm.exe”
@SET GACUTIL=”c:\Program Files\Microsoft Visual Studio 8\
              SDK\v2.0\Bin\gacutil.exe”

Echo Installing OfficeSpaceFeature.dll in GAC
%GACUTIL% -if bin\debug\OfficeSpaceFeature.dll

Echo Copying source files to WSS \TEMPLATE directory
xcopy /e /y TEMPLATE\* %TEMPLATEDIR%

Echo Installing feature with WSS
%STSADM% -o installfeature -filename  OfficeSpaceFeature\feature.xml -force

Echo Restarting IIS worker process
IISRESET

After I have finished adding instructions to the install.bat file, I configure Visual Studio to run it each time I rebuild the OfficeSpaceFeature project. I do this by going to the Build Events tab within the Project Properties and adding the following Post-build event command-line instructions:

cd $(ProjectDir)
Install.bat

The first line is required to change the current directory to that of the project directory. The second line runs the batch file to copy the feature files to the correct location and then install the feature with the InstallFeature operation of the command-line Stsadm.exe utility.

Now that the feature has been properly installed, I can activate it in context of a site. You can do this within any site in a WSS or MOSS farm by navigating to the Site Settings page. Under the Site Administration section, just click the link with the title Site Features. This should take you to a page like the one shown in Figure 7. Note that if you are working within a farm that has MOSS installed, you will see many more features than if you are working within a farm with just WSS.

Figure 7 Activating and Deactivating Site Features
Figure 7 Activating and Deactivating Site Features(Click the image for a smaller view)
Figure 7 Activating and Deactivating Site Features
Figure 7 Activating and Deactivating Site Features (Click the image for a larger view)

The title and description of the OfficeSpaceFeature appear on the Site Features page, so I easily locate it and click the Activate button. At this point, all the elements defined with declarative logic in the elements.xml should be provisioned within the current site. WSS then loads the receiver assembly from the GAC, creates an instance of the receiver class, and executes the receiver class’s FeatureActivated method.

My sample is quite simple. However, as you might imagine, you can provide a far more complex mixture of declarative CAML elements and WSS object model code in the FeatureActivated.

Extending the Feature

If you examine the elements.xml file, you’ll see that there is a second ListInstance element. It is used to create a list, titled Customers, using the built-in WSS Contacts list type. This ListInstance also demonstrates how to add a few default items to the list to provide some sample customer data.

I’ve shown you how to use declarative CAML logic for creating lists and document libraries. Now I want to point out that you can create a list or document library using custom code in the feature activation event that programs against the WSS object model.

The sample feature includes code inside the FeatureActivated event handler to create another list from the standard WSS Custom List type with the title Letter Templates. When you create a new list from the standard WSS custom list type, the list automatically contains one field named Title. My code creates a second column named Body, and it uses the WSS object model to add a few list items that supply the data used as the templates for creating customer letters. The code to do this is shown in Figure 8.

Figure 8 Create a List and Add a New Column (Close Figure 8)
Dim template As SPListTemplate = site.ListTemplates(“Custom List”)
Dim listLetterTemplatesID As Guid
listLetterTemplatesID = site.Lists.Add(“Letter Templates”, “”, template)
Dim listLetterTemplates As SPList = site.Lists(listLetterTemplatesID)
listLetterTemplates.Fields.Add(“Body”, SPFieldType.Text, True)
listLetterTemplates.OnQuickLaunch = False
listLetterTemplates.Update()

Dim newLetterTemplate As SPListItem

newLetterTemplate = listLetterTemplates.Items.Add()
newLetterTemplate(“Title”) = “Initial Greeting”
newLetterTemplate(“Body”) = “Thanks for your recent interest in Litware.”
newLetterTemplate.Update()

newLetterTemplate = listLetterTemplates.Items.Add()
newLetterTemplate(“Title”) = “Follow Up”
newLetterTemplate(“Body”) = “Thanks for your recent purchase.”
newLetterTemplate.Update()

Custom Application Pages

The concept of features and how to work with them is important to understand, but there is a second fundamental building block for designing and developing solutions for WSS and MOSS. The WSS architecture supports a special type of page known as an application page. The standard Site Settings page (settings.aspx) is a good example of an application page. The settings.aspx page is deployed once per front end Web server, yet it can be accessed from any site within a WSS farm.

Application pages are deployed as physical files on the file system of the front end Web server in a directory at the following path:

%PROGRAMFILES%\common files\microsoft shared
   \web server extensions\12\TEMPLATE\LAYOUTS
WSS maps the physical LAYOUTS directory to the virtual directory named _layouts whenever it provisions a new Web application. Using this mapping scheme, along with some additional processing logic, the WSS runtime makes each application page accessible within the context of any site in the farm. For example, assume there are three different sites in a WSS farm accessible through these three URLs:

http://Litwareinc.com
http://Litwareinc.com/sites/Vendors
http://Litwareinc.com:1001/sites/Accounting

An application page can be accessed by adding its relative path within the _layouts directory to the end of a site’s URL. For example, you can access the Site Settings page using any of these URLs:

http://Litwareinc.com/_layouts/settings.aspx
http://Litwareinc.com/sites/Vendors/_layouts/settings.aspx
http://Litwareinc.com:1001/sites/Accounting/_layouts/settings.aspx
Since there is only one version of an application page scoped at the farm level, it can be compiled into a single DLL and loaded into memory once. You never have to worry about there being different versions of an application page for different sites. Furthermore, application pages are not subject to attack from users who have permissions to customize site pages. Therefore, WSS does not prohibit them from containing in-line code.

Application pages are used extensively by the WSS team to supply much of the standard functionality for provisioning and administrating sites and the elements inside them. WSS also supports custom application pages for custom business solutions. The sample in this month’s column contains a custom application page named LetterGenerator.aspx, which is deployed like any other application page inside the LAYOUTS directory.

Note that LetterGenerator.aspx is not deployed directly in the LAYOUTS directory but instead in a directory named OfficeSpace that resides inside the LAYOUTS directory. It is best to create your own directory inside the LAYOUTS directory to host custom application pages so you can avoid any potential naming conflicts.

An application page is a special type of ASP.NET page that is typically written to inherit from a base class named LayoutsPageBase and to link to the standard Master Page, named application.master, created by the WSS team. Figure 9 gives an example of the classic Hello World custom application page.

Figure 9 Hello World Custom Application Page (Close Figure 9)
<%@ Assembly Name=”[Full assembly name for Microsoft.SharePoint]” %>
<%@ Page Language=”VB” MasterPageFile=”~/_layouts/application.master”
         Inherits=”Microsoft.SharePoint.WebControls.LayoutsPageBase” %>

<%@ Import Namespace=”Microsoft.SharePoint” %>

<script runat=”server”>
  Protected Overrides Sub OnLoad(e As EventArgs)
      MyBase.OnLoad(e)
      ‘*** get current site and web objects through WSS object model
      Dim siteCollection As SPSite = SPContext.Current.Site
      Dim site As SPWeb = SPContext.Current.Web
      ‘*** now program against WSS objects within the current site
      lblDisplay.Text = “Current site: “ & site.Title
  End Sub
</script>

<asp:Content ID=”Main” runat=”server”
             contentplaceholderid=”PlaceHolderMain” >
    <!-- ADD HTML elements and controls here for page content -->
    <asp:Label ID=”lblDisplay” runat=”server” />
</asp:Content>

As you can see, you can create an application page that contains standard ASP.NET controls along with code in standard ASP.NET page overrides, such as OnLoad. You can also add assembly references. If you add a reference to Microsoft.SharePoint.dll, you can then program against the current site collection and the current site by obtaining a reference through the SPContext class, as I’ve done in Figure 9.

Once you have added a custom application page to a solution, you need to provide a way for your users to navigate to it. By adding a CustomAction element to a feature, you can supply a link or a menu item. Here’s an example of the CustomAction element used in the OfficeSpaceFeature sample to add a menu item to the Site Actions dropdown:

<CustomAction
  Id=”OfficeSpaceLetterGenerator”
  GroupId=”SiteActions”
  Location=”Microsoft.SharePoint.StandardMenu”
  Sequence=”1001”
  Title=”OfficeSpace Letter Generator”
  Description=”Create letter using Open XML”
  ImageUrl=”/_layouts/images/crtpage.gif” >

    <UrlAction Url=”~site/_layouts/OfficeSpace/LetterGenerator.aspx”/>

</CustomAction>

This CustomAction element adds the custom menu item to the standard Site Actions menu. The UrlAction element inside the CustomAction element provides a Url attribute with a relative path to my custom application page. The string value assigned to the Url attribute has a dynamic token in the form of ~site. This will be replaced by WSS at run time with the actual path to the current site.

When users select the custom menu item, they are redirected to the custom application page named LetterGenerator.aspx, which displays the user interface shown in Figure 10. There is code in the OnLoad method of this page that populates two listboxes with the desired customer and letter type; it uses the WSS object model to pull data from the two lists that were created during feature activation. There are also two command buttons that provide two different approaches to creating customer letters as .docx files using the code I discussed in my previous Office Space column.

Figure 10 Custom Application Page
Figure 10 Custom Application Page(Click the image for a smaller view)
Figure 10 Custom Application Page
Figure 10 Custom Application Page (Click the image for a larger view)

When the user clicks on either command button, the event handler runs code to generate a letter using the Office Open File Formats. This code is nearly identical to what I covered in my previous column. The only thing that is different is the code used to name the file and to save it inside the WSS document library with the title Customer Letters.

This example demonstrates how to automate the creation of customer letters using the Office Open XML File Formats, and the synergy that exists between WSS and these file formats.

Once you have created a new .docx file, you need somewhere to put it. WSS provides a great place to store documents where they can be versioned and accessed by a team working in a collaborative environment. Figure 11 shows the Customer Letters document library in a site that has activated my OfficeSpaceFeature.

Figure 11 Customer Letters Document Library
Figure 11 Customer Letters Document Library(Click the image for a smaller view)
Figure 11 Customer Letters Document Library
Figure 11 Customer Letters Document Library (Click the image for a larger view)

Summary

This month’s column introduces you to features and custom application pages—two essential building blocks used to design and implement solutions for WSS and MOSS. At this point, you should have a good idea about how to get started with development and testing. In the next installment of Office Space, I will discuss how you can package a solution, such as the one I just described, for deployment in a production environment.

Send your questions and comments for Ted to instinct@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. Ted has just completed his book Inside Windows SharePoint Services 3.0 for Microsoft Press. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group.


Cutting Edge: Subclassing and Overriding ASP.NET Pages—Part II

Cutting Edge
Subclassing and Overriding ASP.NET Pages—Part II
Dino Esposito

Code download available at: CuttingEdge2007_05.exe (163KB)
Contents
Background
The Original Solution
Simulating a Very Simple Attack
How Does This Happen?
An Improved TextBox Control
Replacing the TextBox Control
The Definitive Solution

I recently came across a relatively large Web site that was full of extremely simple Web user controls—ASCX files, to be precise. The developers considered this approach necessary after they discovered unexpected behavior in a server control they were using.

As a result of this discovery, the developers replaced all occurrences of this server control throughout the site with a user control that contained a modified version of the original control. (At the same time, the developers also replaced numerous other controls, since they just couldn’t be sure what they would run into next.) The developers felt this extra layer of abstraction placed in between the page and the control was more reliable. In addition, there was the benefit that, if need be, a user control could easily be replaced in an ASP.NET application without touching the binaries and restarting the application. (This is not always the case, but there are deployment options that make this possible.)

I was called in to review the application and the first question I was asked was, "is there a better way to replace a server control throughout a site without extensively reworking every page?"

In my April 2007 column, I discussed a few approaches to the problem of applying limited, sometimes temporary, changes to an ASP.NET Web site without touching the original source code. This month, I explore more techniques that can be used to replace server controls and URLs in a declarative manner without the source code.

I didn’t have an immediate answer to their question, but I knew where to look for a solution. I thought that if I had developed the ASP.NET infrastructure, I would have put some sort of setting in the configuration file to let developers map tags to controls declaratively. This is not a completely new idea in ASP.NET. In ASP.NET 1.x, you can declaratively change some code-related aspects, such as the base class of Web pages and user controls. (However, this is only for pages and user controls that don’t explicitly use the Inherits clause in the Page directive.) So why not server controls, I thought. Well, my reasoning was correct: ASP.NET 2.0 provides the <tagMapping> section just for this purpose.

Background

I should share a little background. This scenario began during an internal security review when the customer discovered a security hole in an application that could lead to a classic SQL injection attack. The company applied a quick patch, but it only led to another type of problem.

In the customer’s Web site, many of the pages accept a fixed five-character code on the query string. This code is then used to compose a SQL statement. The company was still running code that looked like this:

Dim code As String = Request.QueryString(“Code”).ToString();
Dim command As String = _
    “SELECT * FROM customers WHERE id=’” & code & “’”
I sincerely hope that your sites are not still running similar code! It fully and blindly trusts any information that is passed through the query string, and it appends this information to a string that composes a SQL command. This is a serious security hazard. A knowledgeable hacker can easily find seemingly innocuous text that transforms the original and made-to-measure SQL command into a dangerous attack. If you need more details on SQL injection, start with Paul Litwin’s article "Stop SQL Injection Attacks Before They Stop You".

From a performance standpoint, sending out dynamically constructed commands is a mistake. These commands are not going to benefit from query plan reuse, since the code itself will likely be different every time they are submitted. Using a parameterized query or a stored procedure has two advantages: it ensures at least an automatic check on the type of the data being sent, and it benefits from query plan caching in SQL Server™ and other databases.

As explained in the March 2007 installment of Cutting Edge, you should validate a query string statically as you would do with the command line of a console utility. In this way, you disambiguate numbers, dates, and Boolean values, but you can’t do much when a string is expected. Here, the real problem is what’s in the string. You need some business logic to carefully validate the passed string. Verifying the length of the string, though, is a simple check that can be accomplished either through an HTTP module or with limited modification to the code.

The Original Solution

After identifying a SQL injection hole, the customer determined that limiting the size of the acceptable arguments to five characters—the real size of the code—would be a quick and effective workaround. With only five characters available, there’s not much a hacker can do against your database—at least, that was the hope. So the customer installed the HTTP module as presented in the March 2007 installment of Cutting Edge and checked the query string size of affected pages. No more than five characters were effectively sent to pages.

However, the application—which was a mix of new ASP.NET pages and revamped classic ASP pages—had some pages where the user could type the same code in a textbox and submit it. On the server, during postback, the specified code would be appended to a SQL statement using the same method illustrated earlier. Thus, the length of the text being submitted through the textbox also had to be limited. That’s an easy problem, the developers thought, and proceeded to set the MaxLength attribute of textboxes to the desired value:

<asp:textbox runat=”server” id=”TextBox1” MaxLength=”5” />

Everything appeared to be fixed. No code with more than five characters could slip into the middle tier of the site. This didn’t necessarily mean the site was safe against injections, but it certainly limited the potential for attacks. Or so they thought.

Simulating a Very Simple Attack

Consider a sample ASP.NET page like the one in Figure 1. The source code of the page is shown in Figure 2. The page features a textbox with the MaxLength attribute set to five and a submit button. When clicked, the button performs a postback and processes the contents of the textbox. Under normal circumstances, where the page is displayed in the browser, there’s really no way to type in more than five characters. If you try to paste in a longer string of text, it is cut off accordingly to the specified size.

Figure 2 Source Code of the Sample ASP.NET Page (Close Figure 2)

ASPX

<%@ Page Language="VB" AutoEventWireup="false"
    CodeFile="Default.aspx.vb" Inherits="_Default" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Max Length</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <h3>Try specifying more than 5 characters</h3>
        <asp:TextBox ID="TextBox1" MaxLength="5"
          runat="server"></asp:TextBox><br />
        <asp:Button ID="Button1" runat="server" Text="Click" />
    </div>
    </form>
</body>
</html>
Codebehind
Partial Class _Default : Inherits System.Web.UI.Page

    Protected Sub Page_Load(ByVal sender As Object, _
          ByVal e As System.EventArgs) Handles Me.Load
        TextBox1.Focus()
    End Sub
    Protected Sub Button1_Click(ByVal sender As Object, _
          ByVal e As System.EventArgs) Handles Button1.Click
        Response.Write("<b>Processing text: </b>" & TextBox1.Text & "<b>
   </b><hr/>")
    End Sub
End Class
Figure 1 Sample ASP.NET Page
Figure 1 Sample ASP.NET Page(Click the image for a smaller view)
Figure 1 Sample ASP.NET Page
Figure 1 Sample ASP.NET Page (Click the image for a larger view)

Now look at this from the attacker’s perspective. A typical attack against a Web page involves creating a copy of the page as plain HTML, altering some values, and then posting the "poisoned" version of the page. To get the HTML of the page, all the malicious user has to do is display the page as any typical user would: select View Source, then save the contents to a local HTML file. This is only possible if the attacker can physically access the page. For example, if the page is protected, the attacker must present valid credentials to view the page. However, stolen authentication cookies, phishing tactics, and other social engineering techniques can put the right information into the wrong person’s hands.

Once the markup of the ASP.NET page has been saved to the local machine, a couple of changes are required. First, the attacker must change the action attribute of the form, making it point to the absolute URL of the same ASP.NET page. Here’s the typical markup for the server form of a default.aspx ASP.NET page:

<form name=”form1” method=”post” action=”Default.aspx” id=”form1”>
The attacker would change it to the following:

<form name=”form1” method=”post”
      action=”http://targetserver/Source/Default.aspx” id=”form1”>

There’s no action attribute on the ASP.NET HtmlForm control, but when you use plain HTML, you can still post the contents of a form to any URL you wish. The second change regards the poisoned data being posted. I’ll add a value attribute to the markup emitted by the ASP.NET TextBox server control set to a string far longer than the regular five characters:

<input name=”TextBox1” type=”text”
maxlength=”5” id=”TextBox1”
value=”This is a far looooonger text” />

What do you think will happen when this HTML page is displayed on the attacker’s machine and the button clicked? The result is shown in Figure 3. The address bar of the browser window on the left shows that the page being displayed is a local HTML page. And in the browser window on the right, you can see that the modified content of the form has been posted to the remote ASP.NET application. The limit of five characters has been circumvented. This shows that a malicious user has a way to send your pages text of any size regardless of the MaxLength settings on textboxes.

Figure 3 Local HTML Page that Posts Strings of Any Length
Figure 3 Local HTML Page that Posts Strings of Any Length(Click the image for a smaller view)
Figure 3 Local HTML Page that Posts Strings of Any Length
Figure 3 Local HTML Page that Posts Strings of Any Length (Click the image for a larger view)

How Does This Happen?

You’re probably wondering where the problem lies. Is it in the browser? The ASP.NET runtime? Or perhaps the TextBox control? Yes, the real problem is in the TextBox control.

The TextBox server control doesn’t check MaxLength when a page that contains it is recreated on the server after a postback. Obviously, to be on the safe side, the TextBox should check the value of MaxLength and compare it to the length of the posted text before assigning the Text property.

An Improved TextBox Control

The TextBox, which is the server-side counterpart of the <input type=text> HTML tag, receives text that the user types into the input buffer. The TextBox needs to process this text to fire the TextChanged server event and make data available to other controls in the page. An ASP.NET control that processes posted data implements the IPostBackDataHandler interface like so:

Public Interface IPostBackDataHandler
  Function LoadPostData(ByVal postDataKey As String, _
          ByVal postCollection As NameValueCollection) As Boolean
  Sub RaisePostDataChangedEvent()
End Interface
The LoadPostData method checks whether the postback data of the TextBox control is different from its previous value and, if so, loads the content and returns true. Otherwise, it returns false.

The LoadPostData method is invoked between the Init and Load events of the page, right after the content of the view state has been restored for each control. The postDataKey parameter indicates the name within the posted collection that references the content to be loaded. The postCollection parameter brings in the collection of all posted values—query string or form collection, depending on the selected HTTP verb.

The RaisePostDataChangedEvent method is invoked later in the lifecycle to raise an optional control-specific event that signals whether the state of the control changed after the postback. In practice, the method RaisePostDataChangedEvent is only invoked if LoadPostData returns true.

Figure 4 shows the pseudocode that represents the implementation of LoadPostData for the System.Web.UI.WebControls.TextBox control. Basically, the method compares the value of the Text property read from the view state with the posted value. If the two differ, the posted value replaces the current value and becomes the new value of the Text property in the control.

Figure 4 LoadPostData Method of the TextBox Control (Close Figure 4)
Protected Overridable Function LoadPostData(ByVal postDataKey As?String, _
      ByVal postCollection As NameValueCollection) As Boolean
   MyBase.ValidateEvent(postDataKey)

   Dim oldText As String = Me.Text
   Dim newText As String = postCollection.Item(postDataKey)

   If (Not Me.ReadOnly AndAlso Not oldText.Equals( _
         newText, StringComparison.Ordinal)) Then
      Me.Text = newText
      Return True
   End If

   Return False
End Function

As you can see, the posted value is blindly assigned to the Text property without taking into account the length of the string. Each control can update as many properties as required in the LoadPostData method and can cross-check any properties that make sense to test. As demonstrated in Figure 4, the TextBox implementation of LoadPostData limits the verification to only ensuring that the control is not read-only and then proceeds to comparing the old and new text.

Figure 5 presents a brand new TextBox control with a slightly different implementation of the LoadPostData method. The overridden method simply cuts the posted text to the maximum allowed length before proceeding with the text comparison. As you can see in Figure 6, any text that exceeds the maximum length is automatically cut off and subsequently never used to generate further results during the postback. This happens regardless of the capabilities of the client browser.

Figure 5 Modified LoadPostData Method for TextBox (Close Figure 5)
Public Class TextBox : Inherits System.Web.UI.WebControls.TextBox

    Protected Overrides Function LoadPostData( _
            ByVal postDataKey As String, _
            ByVal postCollection As NameValueCollection) As Boolean
        Page.ClientScript.ValidateEvent(Me.UniqueID, String.Empty)

        Dim oldText As String = Me.Text
        Dim newText As String = postCollection.Item(postDataKey)

        If (newText.Length > Me.MaxLength) Then
            newText = newText.Substring(0, Me.MaxLength)
        End If

        If (Not Me.ReadOnly AndAlso Not oldText.Equals( _
                newText, StringComparison.Ordinal)) Then
            Me.Text = newText
            Return True
        End If

        Return False
    End Function

End Class
Figure 6 Text Exceeding Maximum Length is Cut Off
Figure 6 Text Exceeding Maximum Length is Cut Off(Click the image for a smaller view)
Figure 6 Text Exceeding Maximum Length is Cut Off
Figure 6 Text Exceeding Maximum Length is Cut Off (Click the image for a larger view)

If you compare the source code of the LoadPostData method in Figure 4 and Figure 5, you’ll notice a minor difference. In Figure 4, the method calls into ValidateEvent on its base class—the System.Web.UI.Control class. In Figure 5, the same code is replaced with a call to ValidateEvent on the ClientScriptManager object:

Page.ClientScript.ValidateEvent(Me.UniqueID, String.Empty)

Because the ValidateEvent method on the base Control class is declared Friend (internal in C#), you cannot invoke it from any class defined outside the System.Web.dll assembly. The call stack of the ValidateEvent method on the base Control class ends up invoking the ValidateEvent method on the ClientScriptManager object; an instance of the ClientScriptManager object is exposed through the ClientScript property of the Page class.

ValidateEvent is one of the tools used to implement event validation in ASP.NET 2.0. Event validation is a built-in feature intended to help prevent page to process events (and event arguments) not specifically generated by the page and the registered control.

Replacing the TextBox Control

So now you have a brand new TextBox control. It ensures that any text exceeding the maximum length that is assigned to the Text property will be detected and removed. How do you go about using this control in your ASP.NET pages? Simply register the control with each page and replace all occurrences of the original textbox.

In ASP.NET 2.0, you can save a lot of work by adding the following configuration script to the web.config file under the <controls> section of the <pages> block:

<add tagPrefix=”x” namespace=”Samples” assembly=”TextBox” />
This ensures that all pages under the scope of the web.config file will automatically register the specified tag and controls.

But one problem still remains, and it is quite a big one. The issue is how to switch between old and new textboxes? Fortunately, with ASP.NET 2.0, you have the <tagMapping> section in the configuration file:

<pages>
   <tagMapping>
     <add tagType=”System.Web.UI.WebControls.TextBox”
          mappedTagType=”Dino.Samples.TextBox” />
   </tagMapping>
</pages>

The <tagMapping> section allows you to remap a control type onto another at compile time. This remapping causes the mapped type to be used in place of the original type for all pages and user controls in the scope of the configuration file. As per the preceding code, Dino.Samples.TextBox will now be used wherever a system TextBox is referenced. All that you have to do is write the new control and edit the web.config file. It’s surprisingly easy and quite effective.

Needless to say, the remapped type must be a class that inherits from the original type. I should also note that the ASP.NET team used this feature in pre-RTM builds of ASP.NET AJAX Extensions 1.0 to replace original validator controls with new ones that work great with the UpdatePanel control.

The Definitive Solution

The customer correctly diagnosed an issue with the ASP.NET TextBox control and its way of processing posted data. And it adequately fixed the problem by creating a new TextBox control. Since the developers didn’t know of a better, possibly declarative, way to replace controls throughout a site, they replaced all occurrences manually and wrapped them in a user control. This was to minimize the impact of any future changes.

With the tagMapping feature, the solution was a piece of cake. The technique that uses tagMapping is neat and can be applied to replace buggy controls or to add new capabilities to existing controls. Note, though, that if the remapped control has new properties or methods, you’ll need to modify the source code in order to use these new properties and methods.

(Speaking of declarative mapping, ASP.NET 2.0 also features a <urlMappings> section. It is a direct child of <configuration>. The <urlMappings> section is ASP.NET 2.0’s declarative counterpart to the RewritePath method on the HttpContext object.)

In summary, you should be aware that the original ASP.NET TextBox control doesn’t trim any posted value for the Text property when MaxLength is set. But this column should help solve the issue, providing a modified control that works around the limitation. And you can declaratively inject it in your application with a simple new line in the web.config file.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Test Run: Lightweight Testing with Windows PowerShell

Test Run
Lightweight Testing with Windows PowerShell
Dr. James McCaffrey

Code download available at: TestRun2007_05.exe (155KB)
Contents
The Module Under Test
Ad Hoc Interactive Module Testing
Lightweight Module Test Automation
Wrapping Up

You can think of Windows PowerShell as a dramatic upgrade to the old cmd.exe command shell and associated .bat files. Although designed with systems administration tasks in mind, Windows PowerShell™ has features that also make it ideally suited for lightweight testing tasks. This month, I test Microsoft® .NET Framework-based code modules from both the Windows PowerShell command line and lightweight Windows PowerShell scripts.

Take a look at the screenshots shown in Figure 1 and Figure 2. Figure 1 shows Windows PowerShell performing ad hoc, interactive testing of a .NET DLL module named MyPointLib. Windows PowerShell provides syntax that lets you create objects and dynamically load libraries into the running shell process. I load the MyPointLib library under test, instantiate a Point object, probe the object’s available methods, invoke a Distance method, and verify a correct result.

Figure 1 Ad Hoc Testing with Windows PowerShell on the Command Line
Figure 1 Ad Hoc Testing with Windows PowerShell on the Command Line(Click the image for a smaller view)
Figure 1 Ad Hoc Testing with Windows PowerShell on the Command Line
Figure 1 Ad Hoc Testing with Windows PowerShell on the Command Line (Click the image for a larger view)

Figure 2 shows the output of a Windows PowerShell script. Here, you’ll see that I perform classic module testing—I instantiate Point objects of the MyPointLib library, call instance and static versions of the Distance method, and compare actual results with expected results to determine test case pass/fail results.

Figure 2 Classic Module Testing with a Script
Figure 2 Classic Module Testing with a Script(Click the image for a smaller view)
Figure 2 Classic Module Testing with a Script
Figure 2 Classic Module Testing with a Script (Click the image for a larger view)

In this column, I first describe the class library under test, and then I go over the Windows PowerShell commands shown in Figure 1. I also point out Windows PowerShell techniques for interactively testing .NET-based modules. Next I explain the script that produced the output in Figure 2. The source code for the test harness and the library under test are in the download.

I then explore how you can adapt the ideas presented here to meet your own testing needs, and then discuss the scenarios in which Windows PowerShell is more appropriate than alternatives such as unit testing frameworks like NUnit. I am confident you’ll find that Windows PowerShell is a great addition to your software testing arsenal.

The Module Under Test

For demonstration purposes, I kept the MyPointLib library quite simple (see Figure 3). (If you are an experienced .NET developer, you may want to skip ahead to the next section.) I haven’t included code necessary to a production implementation, such as exception handling.

Figure 3 The MyPointLib.cs Test Library (Close Figure 3)
using System;
namespace MyPointLib
{
  public class Point
  {
    private int x;
    private int y;

    public enum DistanceKind { Euclidean, CityBlock };

    public Point() { }

    public Point(int x, int y)
    {
      this.x = x; this.y = y;
    }

    public int X { get { return this.x; } set { this.x = value; } }

    public int Y { get { return this.y; } set { this.y = value; } }

    public override string ToString()
    {
      return “(“ + this.x + “,” + this.y + “)”;
    }

    public double Distance(Point p)
    {
      return Distance(this, p);
    }

    public static double Distance(Point p1, Point p2)
    {
      return Distance(p1, p2, DistanceKind.Euclidean);
    }

    public static double Distance(Point p1, Point p2, DistanceKind dk)
    {
      switch(dk)
      {
        case DistanceKind.CityBlock:
          return Math.Abs(p1.x - p2.x) + Math.Abs(p1.y - p2.y);
        case DistanceKind.Euclidean:
          return Math.Sqrt((Math.Pow(((double)(p1.x - p2.x)), 2)) +
            (Math.Pow(((double)(p1.y - p2.y)), 2)));
        default:
          throw new ArgumentException(“dk”);
      }
    }
  }
}

Though I used C# for MyPointLib, Windows PowerShell can be used to examine and test .NET libraries regardless of the implementation language, and Windows PowerShell can also be used to test classic COM libraries.

The MyPointLib library implements a two-dimensional Point object in rectangular (x,y) form. Notice I also declare a publicly visible enumeration type DistanceKind:

private int x;
private int y;

public enum DistanceKind { Euclidean, CityBlock };
There are several ways to compute the distance between two points. The Euclidean distance (probably the formula you learned in high school) is the most common. In another approach, the city block technique, you can only travel left and right along integer-valued-axis lines. (The city block distance is sometimes called the taxi cab distance.)

Accessing a custom enumeration type in a .NET library using Windows PowerShell requires special syntax, which I will point out shortly. I implement a default constructor and an auxiliary constructor that accepts two arguments:

public Point() {}

public Point(int x, int y)
{
  this.x = x; this.y = y;
}
The default constructor instantiates a point with coordinates (0,0). If you are going to test library modules with Windows PowerShell, you must know how to call both kinds of constructors (no-arguments and arguments). Because my x and y fields are private, I implement X and Y properties to get and set their values:

public int X { get { return this.x; } set { this.x = value; } }
Since I am using C# here, which is case-sensitive, I can use lowercase x for the field name and uppercase X for the property name (similarly y and Y) without any ambiguity. The ability to access class properties is fundamental to module testing with Windows PowerShell.

The MyPointLib library also overrides the ToString method:

public override string ToString()
{
  return “(“ + this.x + “,” + this.y + “)”;
}
My ToString method is very typical of the no-parameter methods you will want to access when testing with Windows PowerShell.

The Point class contains three different Distance methods. The first is an instance method:

public double Distance(Point p)
{
  return Distance(this, p);
}
Instance methods are the most common form of methods you will encounter when examining and testing .NET library modules using Windows PowerShell. The Distance method returns the Euclidean distance between the implied this-context Point object and the Point object P passed in as a parameter, according to the formula distance = sqrt( (x1 - x2)2 + (y1 - y2)2 ). It doesn’t implement that formula explicitly; instead, it delegates to the second Distance method, which is static:

public static double Distance(Point p1, Point p2)
{
  return Distance(p1, p2, DistanceKind.Euclidean);
}
The static Distance returns the Euclidean distance just as the instance version does; the only difference is how the two methods are called. As I’ll demonstrate next, calling a static method in a .NET library module with Windows PowerShell requires different syntax than calling an instance method. This method relies on the third Distance method for the actual computation. The third Distance method accepts a parameter that tells the method which kind of distance formula (Euclidean or CityBlock) to use when computing the distance:

public static double Distance(Point p1, Point p2, DistanceKind dk)
{
  switch(dk)
  {
    case DistanceKind.CityBlock:
      return Math.Abs(p1.x - p2.x) + Math.Abs(p1.y - p2.y);
    case DistanceKind.Euclidean:
      return Math.Sqrt((Math.Pow(((double)(p1.x - p2.x)), 2)) +
        (Math.Pow(((double)(p1.y - p2.y)), 2)));
    default:
      throw new ArgumentException(“dk”);
  }
}

The diagram in Figure 4 illustrates the difference between Euclidean distance and CityBlock distance between the points (0,0) and (3,4). The real purpose of this version of Distance is to show how to use enumeration types with Windows PowerShell. As you can see, the Euclidean distance is a direct line between points, while the CityBlock distance is longer, since it follows gridlines.

Figure 4 Euclidean vs. CityBlock Distance
Figure 4 Euclidean vs. CityBlock Distance(Click the image for a smaller view)
Figure 4 Euclidean vs. CityBlock Distance
Figure 4 Euclidean vs. CityBlock Distance (Click the image for a larger view)

You can build the MyPointLib library from the Windows PowerShell command line if you want. If, for example, the source code for the MyPointLib class library is named MyPointLib.cs and is located in the C:\ModTestWithPS\AppLibs directory, you can build the library with the C# compiler by typing the following simple command:

PS C:\ModTestWithPS\AppLibs> csc.exe /target:library MyPointLib.cs
This will create library MyPointLib.dll in directory C:\ModTestWithPS\AppLibs. This example assumes the Windows PowerShell path environment variable includes the location of the csc.exe program. I will discuss how to set the path variable in the next section.

The Point class contains the types of fields you are most likely to encounter when performing module testing: default and auxiliary constructors, get and set properties, an overridden ToString method that takes no arguments, an enumeration type, and instance and static methods that both accept arguments.

Ad Hoc Interactive Module Testing

Let’s now look at some ways in which you can interactively examine and test .NET libraries using Windows PowerShell. The first part of the output shown in Figure 1 indicates that you are using Windows PowerShell:

Windows PowerShell
Copyright (C) 2006 Microsoft Corporation. All rights reserved.
PS C:\> _

When you launch a new instance of Windows PowerShell, it executes any commands stored in a special profile file. You can view your profile file by typing notepad.exe $profile at the Windows PowerShell prompt. For the example shown in Figure 1, my profile file contains the following four statements:

set-location \
# write-host “Adding location of csc.exe and ildasm.exe to path”
$env:path += ‘;C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727’
$env:path += ‘;C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin’
The first statement uses the set-location cmdlet to change the current working directory to the root C:\ drive. In case you’re new to Windows PowerShell, let’s take a closer look at cmdlets. Windows PowerShell contains about 130 cmdlets (pronounced “command-lets”), but you can actually accomplish the vast majority of day-to-day file system tasks using only about a dozen or so of these. The set-location cmdlet can be abbreviated as “sl”. In fact, most cmdlets have commonsense aliases that map to old cmd.exe shell commands—in this case, “cd”. You can get a list of all cmdlets by typing get-command, and you can get help on a particular cmdlet using the get-help cmdlet with the target cmdlet’s name (for example, get-help set-location). Windows PowerShell is not case-sensitive.

The second statement in the profile file is a print statement that has been commented out. If the # comment character were removed, the message in double-quotes would be printed to the shell at startup. The third line adds the path to the csc.exe program to the shell path environment variable. The fourth line adds the path to the ildasm.exe program to the shell path variable.

(For security reasons, the default Windows PowerShell policy is not to allow scripts to execute. You can change this with the set-executionpolicy cmdlet, but you’ll need to do so outside of the profile; if the execution policy disables the execution of scripts, the profile won’t run at startup, and thus attempting to change the execution policy in the profile would be futile.)

After Windows PowerShell initializes, I use the set-location cmdlet to change the current working directory ("cd" is a standard alias for set-location, so I could have used it instead):

PS C:\> set-location .\ModTestWithPS\AppLibs
PS C:\ModTestWithPS\AppLibs> _
Then I display all files with a .dll extension located in the current directory:

PS C:\ModTestWithPS\AppLibs> gci *.dll

  Directory: Microsoft.PowerShell.Core\FileSystem::C:\ModTestWithPS\AppLibs

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        12/12/2006  10:31 AM       3072 MyMathLib.dll
-a---        12/12/2006  10:02 AM       4096 MyPointLib.dll
I use gci, which is a shortcut for get-childitem. (I could also have typed dir.)

Up to this point, I haven’t demonstrated anything special about Windows PowerShell. The next command begins to show you some of the power behind it:

PS C:\ModTestWithPS\AppLibs> [Reflection.Assembly]::LoadFile(‘C:\ModTestWithPS\AppLibs\MyPointLib.dll’)

GAC    Version        Location
---    -------        --------
False  v2.0.50727     C:\ModTestWithPS\AppLibs\MyPointLib.dll
Here I call directly into the .NET Framework and use the LoadFile method on the Assembly type in the System.Reflection namespace to load my class library. The syntax [...] indicates that Windows PowerShell should look for an instance of the type between the brackets. The two colons “::” indicates that Windows PowerShell should look for a static property or method. Wow! Windows PowerShell is based on and gives you full access to the .NET Framework.

Now that my class library is loaded, I can instantiate an object from the library:

PS C:\ModTestWithPS\AppLibs> $p1 = new-object MyPointLib.Point
Windows PowerShell is object-oriented, giving it several advantages when compared to older shells, such as cmd.exe and the UNIX-based bash. The $p1 is a Windows PowerShell variable. All variables are preceded by the $ character, making them easy to recognize in scripts. I use the new-object cmdlet to invoke the default, no-arguments, Point constructor. Remember the default Point constructor sets the coordinates of the point to (0,0). I do not use parentheses in the call to new-object because I’m not really calling a method. Rather, I’m providing arguments that are used by the constructor. This is a subtle but important distinction—cmdlets are command-line tools, and they behave that way.

In this example, I know that the MyPointLib library contains a Point class because I have access to the library source code. But what if you are testing a DLL and do not have access to the source code? One easy way to determine which classes are housed in a .NET-based DLL is to use the ildasm.exe tool.

PS C:\ModTestWithPS\AppLibs> ildasm.exe MyPointLib.dll /text /classlist

//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
// Class Point                  (public) (auto) (ansi)
// Class DistanceKind           (auto) (ansi) (sealed) (nested public)
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
(etc.)
The /text switch parameter tells ildasm.exe to send output to the shell instead of to the default GUI display. And the /classlist parameter instructs ildasm.exe to include class information in the output.

Another way to determine what classes are available in an assembly is with Windows PowerShell. The following little script allows you to get the types from the MyPointLib assembly:

PS> $asms=[AppDomain]::CurrentDomain.GetAssemblies()
PS> $asm=$asms|where {$_.fullname -match “MyPointLib”}
PS> $assembly.gettypes()
IsPublic IsSerial Name         BaseType
-------- -------- ----         --------
True     False    Point        System.Object
False    True     DistanceKind System.Enum
The first line collects all of the loaded assemblies in the current AppDomain. The second line looks for the assembly where its fullname matches “MyPointLib”. Finally, it retrieves the types from the assembly.

Next I use the get-member cmdlet to determine exactly what instance methods belong to a Point object:

PS C:\ModTestWithPS\AppLibs> $p1 | get-member -membertype method

Name        MemberType Definition
----        ---------- ----------
Distance    Method     System.Double Distance(Point P)
Equals      Method     System.Boolean Equals(Object obj)
GetHashCode Method     System.Int32 GetHashCode()
GetType     Method     System.Type GetType()
(etc.)
The | character is the pipe operator and is critical to Windows PowerShell. You can interpret the command above to mean, “take the $p1 Point object and then send it to the get-member cmdlet.” In this case, I ask for all public instance methods. I could have specified a -static switch to display static methods. Next, I call the ToString method:

PS C:\ModTestWithPS\AppLibs> $p1.tostring()
(0,0)
This example shows how to call a method that accepts no arguments. Notice that because Windows PowerShell is not case-sensitive, I can type $p1.tostring(), as well as $p1.ToString(). I could also have typed the longer write-host $p1.tostring(), but Windows PowerShell tries to infer what you mean and invoke the appropriate behavior.

Next, I instantiate a second Point object:

PS C:\ModTestWithPS\AppLibs> $p2 = new-object MyPointLib.Point(1,1)
More generally, this demonstrates how to call an object constructor that accepts one or more arguments. Although I did not invoke either of the two properties of the Point class in the screenshot in Figure 1, I could have done so like this:

PS C:\ModTestWithPS\AppLibs> $p2.X
1
To call a property, you just use normal dot notation, as you might have guessed.

Now that I have created two Point objects, (0,0) and (1,1), I can call one of the Distance methods:

PS C:\ModTestWithPS\AppLibs> $d = $p1.Distance($p2) | out-host
1.4142135623731
In this case, I call the instance version of the Distance method and store the result into a variable named $d. I pipe the operation to the out-host cmdlet so that the result will be displayed in the Windows PowerShell shell. Without the pipe, the distance would have been silently stored into $d. (I will demonstrate how to call the static version of the Distance method in the next section.)

Next, I check the return value of the Distance method with an expected value:

PS C:\ModTestWithPS\AppLibs> $check = [System.Math]::Sqrt(2)
PS C:\ModTestWithPS\AppLibs> $check
1.4142135623731
Here, I call the Sqrt method of the System.Math type to compute the square root of 2, and then I store it into a variable named $check. The Windows PowerShell interpreter will automatically determine that variable $check should be type double because that is what the Sqrt method returns. This is called dynamic typing and is common to most scripting languages. I could have called Sqrt this way:

PS C:\ModTestWithPS\AppLibs> [double] $check = [System.Math]::Sqrt(2)

A unique feature of Windows PowerShell is that you can explicitly specify data types for variables. You can use any fundamental .NET data type, such as [Int32] (or the C# equivalent, such as [int]). Specifying data types gives you runtime error checking: when you use [double] in this way, it means that you’ll get an error if you try to store something other than a double in that variable.

To summarize, Windows PowerShell gives you the ability to perform interactive, ad hoc investigation and testing of .NET-based class libraries. Because Windows PowerShell allows you to directly call methods in the .NET Framework, you can use the LoadFile method on the Assembly type to load a .NET-based DLL. You can use the existing ildasm.exe tool with a /classlist switch inside a Windows PowerShell shell to determine what classes are available to you (or use Windows PowerShell to directly query the target Assembly for what types are exposed) and the Windows PowerShell new-object cmdlet to instantiate an object.

After instantiating an object, you can use the Windows PowerShell get-member cmdlet to determine which methods and properties are available. You can also call instance methods in the library under test using normal dot notation.

Lightweight Module Test Automation

Now I’d like to describe how to write lightweight Windows PowerShell scripts to test .NET-based class libraries. I will describe in detail the script that produced the output shown in Figure 2. The entire script is shown in Figure 5.

Figure 5 File Harness.ps1 (Close Figure 5)
# file: harness.ps1
# test MyPointLib.dll using data in testCases.txt

write-host “`nBegin MyMathLib.dll module test run using PowerShell`n”

$ep = get-executionpolicy
if ($ep -eq ‘unrestricted’) {
  write-warning “: harness running in unrestricted mode`n”
}

write-host “Loading library MyPointLib.dll `n”
[Reflection.Assembly]::LoadFile(‘C:\ModTestWithPS\AppLibs\MyPointLib.dll’) | out-null

$f = get-content testCases.txt
foreach ($line in $f)
{
  write-host “--------------”
  $tokens = $line.split(‘:’)

  $caseID = $tokens[0]
  $method = $tokens[1]
  $inp = $tokens[2]
  $expected = $tokens[3]
  # The above lines can be written as
  # $caseID,$method,$inp,$expected = $line.split(“:”)

  $coords = $inp.split(‘,’)
  [int] $p1x = $coords[0]; [int] $p1y = $coords[1]
  [int] $p2x = $coords[2]; [int] $p2y = $coords[3]
  # the above lines can be written as:
  # $p1x,$p1y,$p2x,$p2y = $inp.split(“,”)

  write-host “Case ID: $caseID  “ -nonewline
  write-host “Inputs: ($p1x,$p1y) ($p2x,$p2y)”
  write-host “Method: $method “ -nonewline
  write-host “Expected: $expected”

  $p1 = new-object MyPointLib.Point($p1x,$p1y)
  $p2 = new-object MyPointLib.Point($p2x,$p2y)

  if ($method -eq “Instance Distance “) {
    $actual = $p1.Distance($p2)
    $a = “{0:f2}” -f $actual
  }
  elseif ($method -eq “Static Distance   “) {
    $actual = [MyPointLib.Point]::Distance($p1,$p2)
    $a = “{0:f2}” -f $actual
  }
  else {
    write-host “Not yet implemented”
    continue
  }

  write-host “Actual: “$actual
  write-host “Result: “ -nonewline

  if ($a -eq $expected) {
    write-host “ Pass “ -backgroundcolor green -foregroundcolor darkblue
  }
  else {
    write-host “*FAIL*” -backgroundcolor red
  }

  trap {
    write-host “Fatal error trapped at case “ $caseID
    continue
  }

} # main loop

write-host “--------------”
write-host “`nEnd test run`n”

# end script

I begin my Windows PowerShell test harness script with a couple of comments:

# file: harness.ps1
# test MyPointLib.dll using data in testCases.txt
The # symbol is the comment character for Windows PowerShell scripts. Like Visual Basic®, Windows PowerShell is line-based, so comments run through the end of a line.

My test harness reads test case data from an external file named testCases.txt, which is:

001:Instance Distance :0,0,3,4:5.00
002:Instance Distance :1,1,4,4:4.24
003:Chebyshev Distance:0,0,3,4:5.00
004:Chebyshev Distance:1,1,4,4:4.24
005:Static Distance   :0,0,3,4:6.00:deliberate fail
006:Static Distance   :0,3,3,0:4.24
Each line represents a single test case, and each field is delimited by the colon character. The first field is a test case ID. The second is a string that tells the test harness which Distance method to test. (The trailing spaces are for readability.) The third field is a set of four comma-delimited integers that represent (x,y) coordinates. The fourth field is an expected distance value. And the fifth, which is an optional field, is a test case comment. Notice test case 005 should generate a deliberate failure—the distance between (0,0) and (3,4) is 5.00, not 6.00.

Next I display a message to the Windows PowerShell shell:

write-host “`nBegin MyMathLib.dll module test run using PowerShell`n”
I use the write-host cmdlet, which is the most flexible way to send output to the console. Alternatives include using the write-output and out-host cmdlets, or simply placing a message without any cmdlet.

String literals in Windows PowerShell can be delimited by either single quotes or double quotes. Single quoted strings appear exactly as typed, but double-quoted strings evaluate certain escape characters inside the string. In this example, the back-tick character followed by "n" is the escape sequence for an embedded newline.

Next, my test harness script examines the current shell execution policy:

$ep = get-executionpolicy
if ($ep -eq ‘unrestricted’) {
  write-warning “: harness running in unresticted mode`n”
}
As you can see, I call the get-executionpolicy cmdlet and store the result into a variable $ep. If Windows PowerShell is in unrestricted execution mode, I use the write-warning cmdlet to display the text “WARNING” followed by a message in yellow text against a black background, as shown in Figure 2.

Then I load my library under test:

write-host “Loading library MyPointLib.dll `n”
[Reflection.Assembly]::LoadFile(
    ‘C:\ModTestWithPS\AppLibs\MyPointLib.dll’) | out-null
I already explained the process of calling the LoadFile method from the Windows PowerShell command line. When called from inside a Windows PowerShell script, the mechanism is exactly the same. Here I pipe output to the out-null cmdlet to suppress progress messages.

Now I read the entire contents of my test case data file into memory and iterate through each line:

$f = get-content ‘testCases.txt’
foreach ($line in $f)
{
  # process each test case here
}
I use the get-content cmdlet to read the entire contents of file testCases.txt into an object named $f. Then I use the Windows PowerShell foreach loop structure to iterate through the file contents a line at a time, storing the result into the arbitrarily named variable $line.

Inside the main processing loop, I parse each test case field:

$tokens = $line.split(‘:’)
$caseID = $tokens[0]
$method = $tokens[1]
$inp = $tokens[2]
$expected = $tokens[3]
I use the Split method to break each line at colon characters and store each part of the line into an array named $tokens. For readability, I transfer the contents of each cell of $tokens into descriptively named variables: $caseID, $method, $inp, and $expected. (Note that I use $inp because $input is a special, reserved Windows PowerShell variable.)

Then I parse my inputs:

$coords = $inp.split(‘,’)
[int] $p1x = $coords[0]; [int] $p1y = $coords[1]
[int] $p2x = $coords[2]; [int] $p2y = $coords[3]
Notice I use the explicit-typing feature of Windows PowerShell to tell my script that my input is type int.

After sending some messages to the shell, I am ready to instantiate two Point objects:

$p1 = new-object MyPointLib.Point($p1x,$p1y)
$p2 = new-object MyPointLib.Point($p2x,$p2y)
I described the new-object cmdlet in the previous section. The new-object cmdlet can also instantiate classic COM objects if you add a -com switch.

Now I can call my Distance method under test:

if ($method -eq “Instance Distance “) {
  $actual = $p1.Distance($p2)
  $a = “{0:f2}” -f $actual
}
elseif ($method -eq “Static Distance   “) {
  $actual = [MyPointLib.Point]::Distance($p1,$p2)
  $a = “{0:f2}” -f $actual
}
else {
  write-host “Not yet implemented”
  continue
}

I use an if/elseif/else control structure to branch on the value of the $method variable, which was read from test case input. Windows PowerShell has a complete set of control structures, including a switch statement. A minor quirk of Windows PowerShell is the use of Boolean operators (-eq, -lt, -gt, and so on) rather than syntactical tokens (==, <, >, and so on). Although this is primarily a matter of style, I do really like the Windows PowerShell approach. The designers of Windows PowerShell chose this approach because of file redirection. For example, if you do something like the following:

get-childitem > output.txt
then you want the > operation to be a file redirection, not a Boolean comparison. As such, the operators that are available in bash and Perl were employed.

I described how to call the instance version of the Distance method under test in the previous section. To call a static method, you prepend the method name with the namespace in square brackets followed by a pair of colon characters (as you might in C++):

[MyPointLib.Point]::Distance($p1,$p2)
Since the result of the instance and static Distance methods are type double, I format the actual result to two decimal places before comparing the actual result to the expected result:

$a = “{0:f2}” -f $actual
Basically, this says, “store into variable $a, the value in variable $actual, formatted to two decimal places.”

Now I compare the actual result to the expected result and display the test result:

if ($a -eq $expected) {
  write-host “ Pass “ -backgroundcolor green -foregroundcolor darkblue
}
else {
  write-host “*FAIL*” -backgroundcolor red
}
Notice my use of the -backgroundcolor and -foregroundcolor switch parameters of the write-host cmdlet. These provide an easy way to produce output with colored text.

A very nice feature of Windows PowerShell is the language’s error-handling capabilities. I’ve used the Windows PowerShell trap keyword to handle any exceptions that are thrown in my script:

trap {
  write-host “Fatal error trapped at case “ $caseID
  continue
}
Here, with the trap block inside my main processing loop, I simply print a message and then use the continue keyword to tell my script to continue execution with the next loop iteration.

Notice that my test harness does not test the third Distance method of the Point class, which uses an enumeration type. Take a look at its signature:

public enum DistanceKind { Euclidean, CityBlock };

public static double Distance(Point p1, Point p2, DistanceKind dk)
Calling this version of the Distance function would look something like this:

$dk = [MyPointLib.Point+DistanceKind]::Euclidean
$dist  = [MyPointLib.Point]::Distance($p1,$p2,$dk)

Wrapping Up

Windows PowerShell has all the features you need to write effective, lightweight test automation scripts. These features include a complete set of control structures, the ability to explicitly type variables, error-handling capabilities, and the ability to call simple cmdlets that produce functionality that would otherwise require several lines of code.

Windows PowerShell is a free, separate-install available from microsoft.com/powershell. It can be installed on machines running Windows® XP, Windows Vista™, Windows Server® 2003, and the next version of Windows Server code-named "Longhorn."

"But James," you say. "I already can perform module testing in several other ways. What advantages does Windows PowerShell offer me?" These other ways are not obsolete, but sometimes Windows PowerShell is more appropriate. It is particularly useful for ad hoc, interactive .NET library testing from the command line.

Before Windows PowerShell came along, your most practical option for testing a .NET-based library was to fire up Visual Studio®. But sometimes your host machine does not have Visual Studio installed, and sometimes you just want a quick way to call a method in the .NET library under test. These two situations are ideally suited for Windows PowerShell.

My experience has shown that Windows PowerShell also makes sense for very lightweight test automation scripts. Visual Studio has the advantages of integrated .NET Framework help and a sophisticated text editor. Windows PowerShell has the advantages of very low overhead and better interactive capabilities in most cases.

"But James! How long will it take me to learn yet another language?" Windows PowerShell was designed to be intuitive and easy to learn. Early adopters have reported that Windows PowerShell is very easy to pick up.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products, including Internet Explorer and MSN Search. He is also the author of .NET Test Automation: A Problem-Solution Approach (Apress, 2006). James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Security Briefs: Events in Windows Vista

Security Briefs
Events in Windows Vista
Keith Brown

Contents
Events and XML
The Event Manifest
Templates and Events
Building and Registering Your Resource DLL
Generating Events at Run Time
Summary

In last month’s column, I took a closer look at the eventing system that’s been around since Windows NT®, showing how you should be using it via the Microsoft® .NET Framework 2.0. In this column, I’ll show you the new eventing system, Windows® Eventing 6.0. You’ll be happy to know that you can get started experimenting with it today, since Windows Vista™ has this feature!

Figure 1 shows the new Event Viewer in Windows Vista. One of the most obvious changes from the previous version is the number of logs. If you look under the Windows Logs folder, you’ll see the classic event logs: Application, Security, System, as well as some newer logs such as the Setup log, which is designed for application setup programs. You’ll also see a folder for logs created specifically for particular applications and services. It may not look like there are many logs here, but if you open up the Microsoft subfolder, then open up the Windows subfolder, you’ll find well over 50 logs that are specific to various components of the Windows Vista operating system. For example, the Backup service has its own log, the WinLogon service has its own log, and so on.

Figure 1 Windows Vista Event Viewer
Figure 1 Windows Vista Event Viewer(Click the image for a smaller view)
Figure 1 Windows Vista Event Viewer
Figure 1 Windows Vista Event Viewer (Click the image for a larger view)

With so many logs, I initially wondered how an administrator would be able to find anything. But after spending some time looking at the Overview and Summary page, I realized that this new Event Viewer allows you to gather events that aren’t necessarily all from the same log into a single view. By expanding the list of Error events, I see events that came from the Application, System, and other more specific logs, such as the one for Virtual Server.

The old Event Viewer allowed you to filter a log based on criteria such as event level (Information, Error, and so on) and event source (sometimes referred to as an event provider). The new Event Viewer allows for similar filtering, but also allows you to create saved views, each of which can aggregate events from as many logs as you want. Application developers can now ship their own custom views along with their applications to help administrators find relevant messages. For example, in the folder called Custom Views I’ve created a view of all Error events across all logs that have occurred in the last hour. I also created a second custom view that finds all administrative or operational events in any log that comes from the source in the sample code for this application.

Events and XML

When you create a filter for a custom view, you’re actually creating a series of XPath queries that the eventing system will use to select the events for the view. Below is the view definition for one of my custom views. I’ve added some white space and shortened some text so it’ll be readable here, but it should give you a flavor for what’s going on.

<QueryList>
  <Query Id=”0” Path=”Application”>
    <Select Path=”Application”>
      *[System[Provider[@Name=’...’]
      and (Level=1 or Level=2)]]
    </Select>
    <Select Path=”MyCompany-...”>
      *[System[Provider[@Name=’MyCompany-...’]
      and (Level=1 or Level=2)]]
    </Select>
  </Query>
</QueryList>

Notice that each Select element has a Path. This indicates which log the XPath query will run against. So right out of the gate, it’s pretty clear that a single view can host events from several logs. Next, look at the query itself. It selects all events that have a System element with a Provider child element that has a particular Name attribute and a Level attribute of 1 or 2. Level is the numeric representation of whether the event is Critical, Error, Information, or some other type.

Wait, you might be thinking, events are represented as XML now? I was pretty excited to see this, because as with many developers, I’ve made a pretty big investment in XML already, and if working with events is as easy as working with XML, I’m a happy camper. Of course one of the first concerns I had when I saw this was the size of the logs. XML isn’t the most compact representation for data. But as you’ll see when I drill a little deeper into the guts of Windows Eventing, the event data is actually stored in a binary format, and is only rendered as XML on demand. And if you read my last column, you learned about message files, which factor out even more redundant data: the strings that make events readable by humans. Those strings often make up the majority of the data for an event, and factoring them out into message files is still an important feature that is used in the new eventing system, both for conserving space in the logs and for localization.

Figure 2 shows what an event looks like in XML. This is a failed logon audit event; apparently I mistyped my password recently. Again, I’ve shortened things a bit for readability, but this will give you a feel for what a typical event looks like.

Figure 2 An Event (Close Figure 2)
<Event xmlns=”http://schemas.microsoft.com...”>
 <System>
   <Provider Name=”...-Auditing” Guid=”...” />
   <EventID>4625</EventID>
   <Version>0</Version>
   <Level>0</Level>
   <Task>12544</Task>
   <Keywords>0x8010000000000000</Keywords>
   <TimeCreated SystemTime=”2007-01-15...” />
   <EventRecordID>22</EventRecordID>
   <Correlation />
   <Execution ProcessID=”588” ThreadID=”5736” />
   <Computer>mario</Computer>
 </System>
 <EventData>
   <Data Name=”SubjectUserSid”>S-1-5-18</Data>
   <Data Name=”SubjectUserName”>MARIO$</Data>
   <Data Name=”SubjectDomainName”>WORKGROUP</Data>
   <Data Name=”SubjectLogonId”>0x3e7</Data>
   <Data Name=”TargetUserSid”>S-1-0-0</Data>
   <Data Name=”TargetUserName”>Keith</Data>
   <Data Name=”TargetDomainName”>MARIO</Data>
   ...
  </EventData>
</Event>

There are two main elements in this event. The first part, System, includes not only information about the event itself, but also the environment in which the event was generated. Note that the process and thread IDs are recorded, along with the computer name (in the Name attribute of the Provider element). The second part, EventData, is where the parameters for the event are located. This is actual data inserted by the process that generated the event—in this case, the Local Security Authority subsystem.

Remember how filters in the Event Viewer use XPath queries to determine which events to show in a view? The query I showed earlier only looked at the System element, but now you can see how a query could drill into the EventData element as well, allowing filters to be much more precise than they ever were before.

Once again, keep in mind that none of the friendly, localized text of the message that you normally associate with an event is actually stored in the event itself, as you can see by examining the event in Figure 2. That text is stored in the provider’s message file. And the events themselves aren’t stored in XML, but rather in binary records. So the XML you’re seeing and the message text for the event was all pulled together, or rendered, when I examined the event through the Event Viewer. You can do the same thing programmatically by calling the EvtRender API.

Now that you’ve gotten a glimpse of what the new eventing system looks like from an administrator’s perspective, let’s drill down and see what it takes to generate these newfangled events for Windows Vista and Windows Server® code-named "Longhorn."

The Event Manifest

A traditional message file adheres to a well-documented format and works well for localizing messages. However, the information that a message file contains is insufficient to describe events, especially with all the new attributes introduced in Windows Vista. Additionally, when reading a .mc file, it is impossible to know which strings correspond to events and which do not.

Fortunately, Windows Vista introduces a new format for defining messages and events. Not only does this new format make it possible to fully describe events and all of their attributes, but as you might expect, it’s based on XML, which means parsing one yourself will be a no-brainer. I can see using this to generate helper classes for eventing like the one I built by hand in my last column. That’s the kind of code you don’t want to be writing by hand anyway!

To define a set of Windows events, you need to create a manifest, which is an XML file that contains a bunch of metadata, including descriptions of the events you plan to generate. Figure 3 shows an example of a manifest that I’ve put together for the sample code that accompanies this column.

Figure 3 Sample Manifest File (Close Figure 3)
<instrumentationManifest
  xmlns=”http://schemas.microsoft.com/win/2004/08/events”>

  <instrumentation xmlns:xs=”http://www.w3.org/2001/XMLSchema”
    xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
    xmlns:win=”http://manifests.microsoft.com/win/2004/08/windows/events”>

    <events>

      <!-- Publisher Info -->
      <provider name=”MyCompany-MyApp-MyComponent”
        guid=”{559b3019-e149-441f-b59b-d5ac6b87c7c3}”
        symbol=”MY_PUBLISHER” resourceFileName=”...\events.dll”
        messageFileName=”...\events.dll”>

        <channels>

          <!--  import the Application log defined in winmeta.xml -->
          <importChannel chid=”admin” name=”Application” />

          <!--  define my own channels -->
          <channel chid=”op”
            name=”MyCompany-MyApp-MyComponent/Operational”
            type=”Operational” isolation=”Application” enabled=”true”/>
        </channels>

        <!-- Event Templates -->
        <templates>
          <template tid=”dbConnFailed”>
            <data name=”DatabaseName” inType=”win:UnicodeString” />
            <data name=”ConnectionString”
                  inType=”win:UnicodeString” />
            <UserData>
              <DatabaseConnectionFailure xmlns=”http://example.org”>
                <name>%1</name>
                <connectionString>%2</connectionString>
              </DatabaseConnectionFailure>
            </UserData>
          </template>
        </templates>

        <!-- All the Events that can be published by this Publisher -->
        <events>
          <event value=”1001” level=”win:Informational” channel=”op”
            symbol=”MYEVENT_APP_STARTED”
            message=”$(string.msgAppStarted)” />
          <event value=”1002” level=”win:Informational” channel=”op”
            symbol=”MYEVENT_APP_STOPPED”
            message=”$(string.msgAppStopped)” />
          <event value=”1003” level=”win:Critical” channel=”admin”
            symbol=”MYEVENT_DB_CONN_FAILED” template=”dbConnFailed”
            message=”$(string.msgDatabaseConnectionFailed)” />
          <event value=”1004” level=”win:Critical” channel=”op”
            symbol=”MYEVENT_DB_CONN_FAILED_DETAIL”
            template=”dbConnFailedDetail”
            message=”$(string.msgDatabaseConnectionFailed)” />
        </events>

      </provider>
    </events>
  </instrumentation>
  <localization>
    <resources culture=”en-US”>
      <stringTable>

      <!-- This is how event data can be used as part of
           Message String -->
        <string id=”msgAppStarted” value=”MyApp started.” />
        <string id=”msgAppStopped” value=”MyApp stopped.” />
        <string id=”msgDatabaseConnectionFailed” value=”Failed to connect
         to the %1 database using the following connection string: %2.
         Check to see that the database is available, or update web.config
         with a new connection string in the appSettings section.” />

      </stringTable>
    </resources>
  </localization>
</instrumentationManifest>

Let me walk you through the various sections of the manifest. The first section defines a provider, which must be assigned a unique GUID and a name that will be recognizable to an administrator. The standard format for publisher names is [CompanyName]-[ApplicationName]-[ComponentName]. As you can see, there are some attributes on the publisher where you need to supply the full path to the executable file that contains your message definitions. However, the Event Log API will expand environment variables, so you can use strings like "%ProgramFiles%\MyApp\events.dll". Because these paths might vary depending on where you the user chooses to install your program, you will need to modify the manifest in your setup program.

The next section defines a set of channels. Channels are where your application will send events, although at run time you won’t be thinking about channels at all since, in the manifest, each event predefines which channel it should be funneled through. You simply raise the event and the Windows Eventing plumbing will route the event to its designated channel. What’s on the other end of a channel? Most likely it’ll be an event log or a high-performance trace log, depending on the type of channel. As you learn more about Windows Eventing, you may read documentation that blurs the distinction between channels and logs, so don’t let that surprise you. The concept of an event channel is new to Windows, and it may become more architecturally significant in the future.

Note that I referred to the Application channel by importing it. This is the global Application event log that you can see in Figure 1. I then create a new channel, and this will ultimately create a new event log file. This is a lot like the custom event log that I created in my previous column, and it will be used to store operational events that would otherwise clutter up the global application log.

There are four types of channels. Administrative and Operational channels point to traditional event logs and may be serviced by subscriptions and event-forwarding services. The difference between an administrative and operational log is mainly how it’s used. Administrative events should be infrequent and actionable; they should fire when there is a problem that the administrator needs to attend to. Operational events are the more mundane My app started/My app stopped type of events.

The other two types of channels, Analytic and Debug, are high-performance channels that will not be dumped into an event log because it’s assumed that they will generate a boatload of events. These channels are used for a high-performance form of event tracing known as Event Tracing for Windows (ETW). You can start a trace by enabling a channel in the Event Viewer, which will also allow you to view the events in the viewer (you will need to select "Show Analytic and Debug channels" in the View menu). For more information, see "Improve Debugging And Performance Tuning With ETW" from the April 2007 issue of MSDN® Magazine.

Templates and Events

Next in the manifest is the template section. This is where you define the structure of the <EventData> section of your events. By factoring out event structure into a template, you can reuse that template with different events. Take a look at how I defined the dbConnFailed template (see Figure 3). The first thing you’ll see are two data elements. This is like the formal argument list for the event: it lists an ordered set of data that your application is expected to supply when generating the event.

After the formal argument list is an optional section called <UserData>. This is where the manifest ends and your creativity takes over, because whatever XML you put inside the UserData element is what is going to show up in the event’s EventData section. Because this particular template is for a database connection failure, I called my top-level event data tag DatabaseConnectionFail-ure. Note that I also put it into its own XML namespace, which is natural because it’s an extension, not a known part of the schema for manifest files. If you forget to do this in your own templates, the message compiler will stop with an error. Do you see the placeholders where the data goes? %1 and %2 indicate the DatabaseName and ConnectionString arguments, respectively.

Before leaving the templates section, I want to point out that the formal parameter list is strongly typed. I’ve provided a list in Figure 4 of types that are recognized out of the box. These types and a bunch of other metadata are defined in winmeta.xml, which is a file provided with the Windows SDK for Windows Vista and Windows Server "Longhorn." You can find this file in the Include directory of the SDK, and you’ll need either to copy it into the directory where you’re compiling your manifest or to use the –W flag to specify the full path to winmeta.xml, because the message compiler needs it to process a manifest. If you forget to do this, MC.EXE will complain and remind you.

Figure 4 Predefined Input Types (Close Figure 4)

Type | Description
win:UnicodeString | A NULL-terminated UNICODE string.
win:AnsiString | A NULL-terminated ANSI string.
win:Int8 | A signed 8-bit integer.
win:UInt8 | An unsigned 8-bit integer.
win:Int16 | A signed 16-bit integer.
win:UInt16 | An unsigned 16-bit integer.
win:Int32 | A signed 32-bit integer.
win:UInt32 | An unsigned 32-bit integer.
win:Int64 | A signed 64-bit integer.
win:UInt64 | An unsigned 64-bit integer.
win:Float | An IEEE 4-byte floating-point value.
win:Double | An IEEE 8-byte floating-point value.
win:Boolean | A 32-bit value where 0 is false, 1 is true.
win:Binary | Variable size binary data.
win:GUID | A DCE-compliant 128-bit UUID. XML spec: {12345678-1234-4667-1234-123456789abc}
win:Pointer | A pointer; sized to the current platform (32-bit or 64-bit).
win:FILETIME | A Windows FILETIME struct.
win:SYSTEMTIME | A Windows SYSTEMTIME struct.
win:SID | A self-relative Windows SID structure. XML spec: S-1-0-0.
win:HexInt32 | A hexadecimal representation of an unsigned 32-bit integer.
win:HexInt64 | A hexadecimal representation of an unsigned 64-bit integer.

Speaking of the message compiler, you’ll need to make sure you’re using the version of MC.EXE that comes with the Windows Vista and Windows Server "Longhorn" SDK. This is easy when you’re experimenting; for now just be sure to run the message compiler from a Windows SDK command prompt as opposed to a Visual Studio command prompt.

Let’s move on to the events section. Here you can see that I’ve defined four events, each of which has a unique numerical value. This is the event ID, and it should be different for each event you define. Then there’s a level that determines how critical the event is. There are a set of predefined levels in winmeta.xml that should be familiar to anyone who has spent time working with the classic Windows event log. Next comes the channel, which refers to one of the channels defined earlier. This defines the initial direction that the event will travel. Features such as event forwarding and subscriptions, which are beyond the scope of this column, may ultimately determine where the event lands, but in most cases, the event will simply be recorded in the log specified in the channel definition, and that will be that (the event is written into the log no matter what; forwarding will use a copy of the event).

Each event also includes a human-readable string that describes it. Note that instead of hardcoding the text into the message itself, I refer to a string table, which you’ll find in the last section of my manifest. This allows the strings to be localized for each culture that your application supports. Also note how my database connection failure message includes placeholders just like the template did: this is how you plug those same parameters into your messages at run time.

Building and Registering Your Resource DLL

If you read my April 2007 column on traditional eventing, you’ll find that building a resource DLL containing your messages is a similar process on Windows Vista. Here’s my batch file that builds the resource DLL for the sample code:

REM buildEventDll.cmd
mc MyManifest.xml
rc /fo events.res MyManifest.rc
link /DLL /NOENTRY /SUBSYSTEM:WINDOWS /MACHINE:X86 events.res

As a side effect of running the message compiler, a C header file will be created with all of the symbols you specified in your manifest. You’ll see some of these symbols being used in Figure 5.

Figure 5 Generating Events (Close Figure 5)
#include <windows.h>
#include <winevt.h>
#include <evntprov.h>
# pragma comment(lib, “wevtapi.lib”)

// this header file generated by MC.EXE
#include “MyManifest.h”

void wmain() {
    REGHANDLE pub;
    EventRegister(&MY_PUBLISHER, 0, 0, &pub);
    EventWrite(pub, &MYEVENT_APP_STARTED, 0, 0);
    EventWrite(pub, &MYEVENT_APP_STOPPED, 0, 0);
    EventUnregister(pub);
}

Once the DLL is built, make sure your manifest points to it properly (recall the publisher attributes that have full paths pointing to the resource DLL). Then you can run a new tool called wevtutil.exe to register your application with the new eventing system. Do this from a Windows SDK command prompt:

wevtutil im MyManifest.xml
You can also unregister your application:

wevtutil um MyManifest.xml

One of the things this tool will do when you install a manifest is write some registry keys that indicate the path to your resource DLL. As of this writing, if those paths are not correct, wevtutil.exe won’t complain, but you won’t get the behavior you desire. Side effects of incorrect registration can include error events in the log or correct events but no messages.

Once you’ve registered your manifest, all of that metadata becomes available to an administrator, even before you generate your first event. From what I’ve seen on Windows Vista, custom event logs that you create by defining new channels don’t show up until you actually generate an event that targets that particular channel.

Generating Events at Run Time

Currently there are no wrappers in the .NET Framework for the new Windows Eventing APIs, so I’ll be showing some C++ code here. Bear with me; the folks at Microsoft are busy building some sample wrappers that you can use, and the next release of the .NET Framework (code-named "Orcas") will expose this functionality directly!

There are only a few APIs you need to know to get started generating events, and the short code snippet in Figure 5 shows a simple example of them in action. The sample code available for this column on the MSDN Magazine site shows how to go one step further and pass arguments for your events, as well as check for errors returned by these APIs.

The first line of code wires you up to the eventing system and the metadata that you registered earlier using wevtutil.exe. Before your application shuts down, be sure to unregister it. This code is really easy to write; you’ll find the majority of your time will be spent crafting the manifest, which is as it should be!

Summary

What about applications that use the old techniques for eventing? Will they work on Windows Vista and Windows Server "Longhorn?" Yes, they will. They won’t have nearly as much control over the structure of their events, but all those events you were writing to the Application event log will show up just fine in the new global Application event log, and all the traditional eventing APIs and .NET classes will still work on those platforms.

If you’re writing new code that’s targeting Windows Server "Longhorn," you should seriously consider using the new APIs and getting the full benefit of Windows Eventing 6.0!

Send your questions or comments for Keith at briefs@microsoft.com.

Keith Brown is a cofounder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight’s Applied .NET Security course, as well as several books, including the .NET Developer’s Guide to Windows Security, which is available in print and on the Web. Learn more at www.pluralsight.com/keith.


Foundations: WCF Transaction Propagation

Foundations
WCF Transaction Propagation
Juval Lowy

Code download available at: Transactions2007_05.exe (211KB)
Contents
Transaction Flow Configuration
Transaction Flow and Operation Contract
The Ambient Transaction
Transactional Service Programming
Transaction Propagation Modes
Client and Service Transactions
Voting and Completion
Conclusion

Transactions are the key to building robust, high-quality service-oriented applications. Windows® Communication Foundation provides simple, declarative transaction support for service developers, enabling you to configure parameters such as transaction flow and voting, all outside the scope of your service. In addition, Windows Communication Foundation allows client applications to create transactions and to propagate them across service boundaries. In this column, I take a closer look at configuring transaction propagation and voting in Windows Communication Foundation and the resulting programming models.

Transaction Flow Configuration

Windows Communication Foundation can propagate transactions across service boundaries. This enables a service to participate in the client’s transaction and for the client to include operation calls on multiple services in the same transaction. The client itself may or may not be a Windows Communication Foundation service. Both the binding and the operation contract configuration control the decision of whether the client transaction is propagated to the service.

I call any binding that is capable of propagating the client’s transaction to the service (if configured to do so) a transaction-aware binding. Specifically, only NetTcpBinding, NetNamedPipeBinding, WSHttpBinding, WSDualHttpBinding, and WSFederationHttpBinding are transaction-aware. By default, transaction-aware bindings do not propagate transactions. The reason is that, like most everything else in Windows Communication Foundation, it is an opt-in setting.

The service host or administrator has to explicitly consent to accepting incoming transactions, potentially from across an organization or across business boundaries. To propagate a transaction, you must explicitly enable it at the binding on both the service host and the client sides. All transaction-aware bindings offer the Boolean property TransactionFlow, which defaults to false, as shown here:

public class NetTcpBinding : Binding,...
{
   public bool TransactionFlow {get;set;}
   //More members
}
To enable propagation, simply set this property to true, either programmatically or in the host config file. For example, when using a config file this code snippet will set the property:

<bindings>
   <netTcpBinding>
      <binding name = “TransactionalTCP” transactionFlow = “true” />
   </netTcpBinding>
</bindings>

Note that the value of the TransactionFlow property is not published in the service metadata. If you use Visual Studio® 2005 to generate the client config file, you will still need to enable transaction flow manually.

Transaction Flow and Operation Contract

Using a transaction-aware binding and even enabling transaction flow does not mean that the service wants to use the client’s transaction in every operation or that the client has a transaction to propagate in the first place. Such decisions should be part of the contractual agreement between the client and the service. To that end, Windows Communication Foundation provides the TransactionFlowAttribute method attribute that controls if and when the client’s transaction flows into service:

public enum TransactionFlowOption
{
   Allowed,
   NotAllowed,
   Mandatory
}

[AttributeUsage(AttributeTargets.Method)]
public sealed class TransactionFlowAttribute :
   Attribute,IOperationBehavior
{
   public TransactionFlowAttribute(TransactionFlowOption flowOption);
}
Note that the TransactionFlow attribute is a method-level attribute because Windows Communication Foundation insists that the decision on transaction flow be made on a per-operation level:

[ServiceContract]
interface IMyContract
{
   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Allowed)]
   void MyMethod(...);
}
This is deliberate to enable the granularity of having some methods that use the client’s transaction and some that do not.

The value of the TransactionFlow attribute is included in the published metadata of the service. When you import a contract definition, the imported definition will contain the configured value. These attributes are TransactionFlowOption.NotAllowed, TransactionFlowOption.Allowed, and TransactionFlowOption.Mandatory.

When the operation is configured to disallow transaction flow via the attribute TransactionFlowOption.NotAllowed, the client cannot propagate its transaction to the service. Even if transaction flow is enabled at the binding and the client has a transaction, it will be silently ignored and not propagate to the service. As a result, the service will never use the client’s transaction, and the service and the client can select any binding with any configuration. TransactionFlowOption.NotAllowed is the default TransactionFlowOption value of the TransactionFlow attribute.

When the operation is configured to allow transaction flow by using TransactionFlowOption.Allowed, if the client has a transaction, then the service will allow the client’s transaction to flow across the service boundary. However, the service may or may not use the client’s transaction even though it was propagated. When choosing TransactionFlowOption.Allowed, the service can be configured to use any binding, transaction-aware or not, but the client and the service must be compatible in their binding configuration. When the service operation allows transaction flow, but the binding disallows it, the client should also disallow it in the binding on its side. Trying to flow the client transaction will cause an error because the transaction information in the message will not be understood by the service. However, when the service-side binding configuration is set to allow transaction flow, the client may or may not want to enable propagation on its side, and so may elect to set TransactionFlow to false in the binding even if the service has it set to true.

When the operation is configured for TransactionFlowOption.Mandatory, the service and client must use a transaction-aware binding with transaction flow enabled. Windows Communication Foundation verifies this requirement at the service load time and throws an InvalidOperationException if the service has at least one incompatible endpoint. TransactionFlowOption.Mandatory means the client must have a transaction to propagate to the service. Trying to call a service without a transaction throws an exception on the client. With mandatory flow, the client’s transaction always propagates to the service. Once again, the service may or may not use the client’s transaction.

Propagating the client transaction to the service requires, by its very nature, allowing the service to abort the client transaction. This implies that you cannot flow the client transaction to a service over a one-way operation because that call does not have a reply message. Windows Communication Foundation validates this at service load time and throws an exception when a one-way operation is configured for anything but TransactionFlowOption.NotAllowed:

//Invalid definition:
[ServiceContract]
interface IMyContract
{
   [OperationContract(IsOneWay = true)]
   [TransactionFlow(TransactionFlowOption.Allowed)]
   void MyMethod(...);
}

The Ambient Transaction

Introduced in the Microsoft® .NET Framework 2.0, the Transaction class from the System.Transactions namespace is used to represent a Windows Communication Foundation transaction:

[Serializable]
public class Transaction : IDisposable,ISerializable
{
   public static Transaction Current {get;set;}
   public TransactionInformation TransactionInformation {get;}
   public void Dispose();

   //More members
}

Developers rarely need to interact with the Transaction class directly. Instead, the .NET Framework 2.0 defines a concept called an ambient transaction, which is the transaction that is used for code currently executing. To obtain a reference to the ambient transaction, call the static Current property of Transaction:

Transaction ambientTransaction = Transaction.Current;
If there is no ambient transaction, Current returns null. Every piece of code, be it client or service, can always reach out for its ambient transaction. The ambient transaction object is stored in thread-local storage. As a result, when the thread winds its way across multiple objects and methods on the same call chain, all objects and methods can access their ambient transaction.

In the context of Windows Communication Foundation, the ambient transaction is paramount. When present, any Windows Communication Foundation resource manager (such as a SQL Server™ or a volatile resource manager) will automatically enlist in the ambient transaction. When a client calls a Windows Communication Foundation service, if the client has an ambient transaction and the binding and the contract are configured to allow transaction flow, the ambient transaction will propagate to the service.

The Transaction class is used both for local and distributed transactions. If no distributed transaction support is required, Windows Communication Foundation will use a local transaction manager. If the client tries to flow its transaction to a service, or if multiple durable resources are involved, the transaction will be promoted to a distributed transaction and managed by the Distributed Transaction Control (DTC). Each transaction has two identifiers used to identify the local and the distributed transaction. You obtain the transaction identifiers by accessing the TransactionInformation property of the Transaction class. TransactionInformation is of the type TransactionInformation defined as follows:

public class TransactionInformation
{
   public Guid DistributedIdentifier {get;}
   public string LocalIdentifier {get;}
   //More members
}

LocalIdentifier is never null; it represents the local transaction. DistributedIdentifier may be Guid.Empty if the transaction has not been promoted yet to a distributed transaction. The main use of these identifiers is for logging, tracing, and analysis. In this column, I use them as a convenient way in code to demonstrate transaction flow as a result of configuration.

Transactional Service Programming

For services, Windows Communication Foundation offers a simple and elegant declarative programming model. By default, the service class and its operations have no ambient transaction. This is the case even when the client transaction is propagated to the service.

Consider the service shown in Figure 1. The ambient transaction of the service will be null, even though the mandatory transaction flow guarantees the client’s transaction propagation. In order to have an ambient transaction, for each contract method the service must indicate it wants Windows Communication Foundation to scope the body of the method with a transaction using the TransactionScopeRequired property of OperationBehavior attribute, as shown in the following code:

Figure 1 Simple Service Contract (Close Figure 1)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Mandatory)]
   void MyMethod(...);
}

class MyService : IMyContract
{
   public void MyMethod(...)
   {
      Transaction transaction = Transaction.Current;
      Debug.Assert(transaction == null);
   }
}

[AttributeUsage(AttributeTargets.Method)]
public sealed class OperationBehaviorAttribute : Attribute,...
{
   public bool TransactionScopeRequired {get;set;}
   //More members
}
The default value of TransactionScopeRequired is false, which is why by default the service has no ambient transaction. Setting TransactionScopeRequired to true provides the operation with an ambient transaction:

class MyService : IMyContract
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod()
   {
      Transaction transaction = Transaction.Current;
      Debug.Assert(transaction != null);
   }
}

If the client transaction is propagated to the service, Windows Communication Foundation will set the client transaction as the operation’s ambient transaction. Otherwise, Windows Communication Foundation will create a new transaction for that operation and set the new transaction as the ambient transaction.

The diagram in Figure 2 demonstrates which transaction a Windows Communication Foundation service uses as a product of the binding configuration, the contract operation and the local operation behavior attribute.

In Figure 2, a non-transactional client calls Service 1. The operation contract is configured with TransactionFlowOption.Allowed. Even though transaction flow is enabled in the binding, since the client has no transaction, no transaction is propagated. The operation behavior on Service 1 is configured to require a transaction scope. As a result, Windows Communication Foundation creates a new transaction for Service 1, Transaction A. Service 1 then calls three other services, each configured differently.

Figure 2 Transactions Propagation
Figure 2 Transactions Propagation(Click the image for a smaller view)
Figure 2 Transactions Propagation
Figure 2 Transactions Propagation (Click the image for a larger view)

The binding used for Service 2 has transaction flow enabled, and the operation contract mandates the flow of the client transaction. Since the operation behavior is configured to require transaction scope, Windows Communication Foundation sets Transaction A as the ambient transaction for Service 2. The call to Service 3 has the binding and the operation contract disallow transaction flow. However, since Service 3 has its operation behavior require a transaction scope, Windows Communication Foundation creates a new transaction for Service 3 (Transaction B) and sets it as the ambient transaction for Service 3. Similar to Service 3, the call to Service 4 has the binding and the operation contract disallow transaction flow. Since Service 4 does not require a transaction scope, it has no ambient transaction.

Transaction Propagation Modes

Which transaction the service uses is a product of the flow property of the binding (two values), the flow option in the operation contract (three values), and the value of the transaction scope property in the operation behavior (two values). There are therefore 12 possible configuration settings. Out of these 12, 4 are inconsistent and are precluded by Windows Communication Foundation (such as flow disabled in the binding yet mandatory flow in the operation contract) or are just plain impractical. Figure 3 lists the remaining 8 permutations.

Figure 3 Transaction Propagation Modes (Close Figure 3)

Binding Transaction Flow | Transaction- FlowOption | Transaction- ScopeRequired | Transaction Mode
False | Allowed | False | None
False | Allowed | True | Service
False | NotAllowed | False | None
False | NotAllowed | True | Service
True | Allowed | False | None
True | Allowed | True | Client/Service
True | Mandatory | False | None
True | Mandatory | True | Client

Those 8 permutations actually result in only four transaction propagation modes. I call these four modes Client/Service, Client, Service, and None. Figure 3 also shows in green font the recommended way to configure each mode. Each of these modes has its place in designing your application. Understanding how to select the correct mode greatly simplifies thinking about and configuring transaction support.

The Client/Service mode, as its name implies, ensures the service uses the client transaction if possible, or a service-side transaction when the client does not have a transaction. The Client/Service mode is the most decoupled configuration, because the service minimizes its assumptions about what the client is doing. The service will join the client transaction if the client has a transaction to flow. Joining the client transaction is always good for overall system consistency. Imagine the service has a transaction separate from that of the client: that opens the way for one of these two transactions to commit while the other one aborts and leave the system in an inconsistent state.

When the service joins the client transaction, all the work done by the client and the service (and potentially other services the client calls) will be committed or aborted as one atomic operation. If the client does not have a transaction, the service still requires the protection of the transaction, and so this mode provides a contingent transaction to the service, by making it the root of a new transaction. Being the root makes the service control the lifespan of the transaction. Figure 4 shows a service configured for Client/Service transaction. Note that the service asserts it always has a transaction. The service cannot assume or assert whether it is the client’s transaction or a locally created one.

Figure 4 Configuring Client/Service Mode (Close Figure 4)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Allowed)]
   void MyMethod(...);
}

class MyService : IMyContract
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod(...)
   {
      Transaction transaction = Transaction.Current;
      Debug.Assert(transaction != null);
   }
}

The Client/Service mode is applicable when the service can be used standalone or it can be used as part of a bigger transaction. When selecting this mode you should be mindful of potential deadlocks. Specifically, if the resulting transaction is a service-side transaction, it may deadlock with other transactions trying to access the same resources because the resources would isolate access per transaction, and the service-side transaction will be a new transaction. When using the Client/Service mode, the service may or may not be the root of the transaction, and the service must not behave differently when it is the root or when it is joining the client’s transaction.

Client and Service Transactions

The Client mode ensures the service only uses the client’s transaction. You select the Client transaction mode when, by design, the service must use its client’s transactions and can never be used standalone. The main motivation for this is to avoid deadlocks and maximize overall system consistency. By having the service share the client’s transaction you reduce the potential for a deadlock, because all resources accessed will enlist in the same transaction so there will not be another transaction that competes for access to the same resources and underlying locks. By having a single transaction you maximize consistency because that transaction will commit or abort as one atomic operation. Figure 5 shows a service configured for the Client transaction mode.

Figure 5 Configuring Client Transaction Mode (Close Figure 5)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Mandatory)]
   void MyMethod(...);
}

class MyService : IMyContract
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod(...)
   {
      Transaction transaction = Transaction.Current;
      Debug.Assert(transaction.TransactionInformation.
                   DistributedIdentifier != Guid.Empty);
   }
}

Note that the method asserts the fact that the ambient transaction is a distributed one, meaning the transaction originated with the client. The Service mode ensures the service always has a transaction, separate from any transaction its clients may or may not have. The service will always be the root of a new transaction.

You select the Service transaction mode when the service needs to perform transactional work outside the scope of the client’s transaction. For example, when you want to perform some logging or audit operations, or when you want to publish events to subscribers regardless of whether your client transaction commits or aborts. Consider a logbook service that performs error logging into a database. When an error occurs on the client side, the client would use the logbook service to log the error or some other entries. In case of an error, after logging, the error on the client side aborts the client’s transaction. If the service were to use the client transaction, once the client transaction aborts, the logged error would be discarded from the database, and you would have no trace of it, thus defeating the purpose of the logging in the first place.

By configuring the service to have its own transaction, logging the error would occur even when the client transaction aborts. The downside, of course, is the potential for jeopardizing the system consistency, because the service transaction could abort while the client’s commits. The heuristic made when selecting this mode is that the service transaction is much more likely to succeed and commit than the client’s transaction. In the example of the logging service, this is often the case, because once deterministic logging is in place it will usually work, as opposed to business transactions that may fail due to a variety of reasons. In general, you should be extremely careful when using the Service transaction mode, and verify that the two transactions (the client’s transaction and the service’s transaction) do not jeopardize consistency if one aborts and the other commits. Logging and auditing services are the classic candidates for this mode.

The code in Figure 6 shows a service configured for Service transaction mode. You should note that the service asserts that it actually has a local transaction.

Figure 6 Configuring Service Transaction Mode (Close Figure 6)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   void MyMethod(...);
}

class MyService : IMyContract
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod(...)
   {
      Transaction transaction = Transaction.Current;
      Debug.Assert(transaction.TransactionInformation.
                   DistributedIdentifier == Guid.Empty);
   }
}

The None transaction mode means the service never has a transaction. This mode is useful when the operations performed by the service are nice to have, though not essential, and should not abort the client’s transaction if the operations fail. For example, a service that prints a receipt for a money transfer should not be able to abort the client transaction if the printer is out of paper. Another reason for employing the None mode is when you want to provide some custom behavior, and you need to perform your own programmatic transaction support or manually enlist resources such as calling legacy transactional code.

Obviously, there is danger in using the None mode because it can jeopardize the system consistency. For instance, if the calling client has a transaction and it calls a service configured as None, and then the client aborted its transaction, the result is that changes made to the system state by the service will not rollback. Another pitfall of this mode is when a service configured for None calls another service configured for a Client transaction. Such a call will fail because the calling service has no transaction to propagate.

Figure 7 shows a service configured for the None Transaction mode. Note that the service asserts it has no ambient transaction.

Figure 7 Configuring the None Transaction Mode (Close Figure 7)
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   void MyMethod();
}

class MyService : IMyContract
{
   public void MyMethod()
   {
      Transaction transaction = Transaction.Current;
      Debug.Assert(transaction == null);
   }
}

The None mode allows you to have a non-transactional service called by a transactional client. As stated previously, configuring for the None mode is mostly for nice-to-have operations. The problem with that is that any exception thrown by the None service will abort the calling client’s transaction, something that should be avoided with a nice-to-have operations. The solution is having the client catch all relevant exceptions from the None service to avoid contaminating the client’s transaction.

Out of the four modes, the Service and the None modes are somewhat esoteric. They are useful in the context of the particular scenarios mentioned, but remember that they harbor the danger of jeopardizing the system consistency. You should use the Client/Service or the Client transaction modes, and choose the mode based on the ability of the service to be used standalone as a function of potential deadlocks and consistency.

Voting and Completion

While Windows Communication Foundation is responsible for every aspect of the transaction propagation and overall management of committing or aborting the transaction across resources, it does not know whether the transaction should commit or abort. Windows Communication Foundation simply has no way of knowing if the changes made to the system state are consistent—that is, if they make sense. Every participating service must vote on whether the transaction should commit or abort. In addition, Windows Communication Foundation does not know when the transaction ends and when all the services complete their work. That is something the root service needs to indicate.

Windows Communication Foundation can automatically vote on behalf of the service to commit or abort the transaction. Automatic voting is controlled via the Boolean TransactionAutoComplete property of the OperationBehavior attribute:

[AttributeUsage(AttributeTargets.Method)]
public sealed class OperationBehaviorAttribute : Attribute,...
{
   public bool TransactionAutoComplete {get;set;}
   //More members
}
The TransactionAutoComplete property defaults to true. When set to true, if there is no unhandled exception in the operation, Windows Communication Foundation will automatically vote to commit the transaction. If there is an unhandled exception, Windows Communication Foundation will vote to abort the transaction. Only a service with a session can set TransactionAutoComplete to false, and doing do has drastic implications on the programming model which are beyond the scope of this column.

When the transaction ends is a product of who starts it. Consider a client that either does not have a transaction or just does not propagate its transaction to the service, and that client calls a service operation configured with TransactionScopeRequired set to true. That service operation becomes the root of the transaction. The root service can call other services and propagate the transaction to them. The transaction will end once the root operation completes the transaction. This is partly why TransactionAutoComplete is named that way. It does more than mere voting; it completes and terminates the transaction for a root service. Be aware that any of the down-stream services called by the root operation can only vote on the transaction and not complete it. Only the root both votes and completes the transaction. Using a transaction scope is practically the only way a non-service client can group multiple service calls into single transaction, as shown in Figure 8.

Figure 8 Using Transaction Scope
Figure 8 Using Transaction Scope(Click the image for a smaller view)
Figure 8 Using Transaction Scope
Figure 8 Using Transaction Scope (Click the image for a larger view)

When a non-service client starts a transaction, the transaction ends when the client disposes of the transaction object it uses, typically when disposing of a TransactionScope object. Having the option to create a root transaction scope enables the client to flow its transaction to services and to manage and commit the transaction based on the aggregated result of the services:

using(MyContractClient proxy1 = new MyContractClient())
using(MyOtherContractClient proxy2 = new MyOtherContractClient())
using(TransactionScope scope = new TransactionScope())
{
   proxy1.MyMethod(...);
   proxy2.MyOtherMethod(...);
   scope.Complete();
}

Conclusion

Transactions are instrumental in many applications, ensuring the proper recovery of the system after an error. Being able to group multiple services from multiple vendors across technology and service boundaries into a single atomic transaction is a key feature of Windows Communication Foundation and the standards it supports. Yet all developers have to do is chose the appropriate logical transaction propagation mode by setting a few attributes and enabling the flow in the binding. This is a testimony to the simple and capable programming model of Windows Communication Foundation. There is much more to transactions support in Windows Communication Foundation, especially when it comes to synchronization and instance management, and I will address these aspects in a future column.

To learn more about transactions and Windows Communication Foundation, see "Introducing System.Transactions in the .NET Framework 2.0" at and "Volatile Resource Managers in .NET Bring Transactions to the Common Type" in the December 2005 issue of MSDN Magazine.

Send your questions and comments to mmnet30@microsoft.com.

Juval Lowy is a software architect with IDesign providing WCF training and WCF architecture consulting. His recent book is Programming WCF Services (O’Reilly, 2007). He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at www.idesign.net.


{ End Bracket }: Exercising Agility

{ End Bracket }
Exercising Agility
James Waletzky

Physical exercise is a fact of life. Love it or hate it, you have to do it to stay healthy. What you do to stay fit, though, is far less important than that you abide by the core principles of exercising, that is, by incorporating flexibility, aerobic, and anaerobic training into your practices. Software development processes are similar—it is not as important to follow a prescribed development process exactly as it is to stick to the principles of the process. This idea is particularly true with agile development.

There are many software development processes that are considered "agile." Examples include Scrum (for project management), Extreme Programming (XP), and Feature-Driven Development (FDD). Following the prescribed steps in each process is a great way to start learning, but in reality no two software projects and teams are identical, so customization is usually required. Changing a process to suit your needs is encouraged—embracing change is one of the guiding principles of agile development—providing you do not violate the infrastructure upon which agile development is based.

What are the fundamental principles, or the essential nature, of agile software development? Agile focuses on:

Simplicity Agile development processes are not only simple, they encourage simple software designs. Keeping software designs simple improves maintainability and discourages over-engineering. The practice of test-driven development (TDD) in XP encourages writing the minimum amount of code to pass a test—nothing more. Couple this practice with continuous refactoring, or improving the design of existing code without changing functionality, and software complexity is minimized.

Embracing Change Two certainties in software development are that humans make mistakes (bugs) and that requirements change. Agile developers do not fear change and are always prepared to deal with it. For example, users often request features late in the development cycle. An agile developer listens to the user and incorporates the changes with minimal pain using practices like TDD, continuous integration, and refactoring.

Incremental Iteration Delivering small, shippable pieces of functionality frequently and incorporating frequent customer reviews helps keep development on track. Furthermore, restricting development to small chunks with a defined meaning of "complete" puts a focus on quality. In Scrum, iterations are typically 30 days and are called "sprints." Each deliverable in a sprint must hold to a high level of quality. FDD typically delivers frequent, tangible, working results every two weeks.

Continuous Feedback and Improvement Agile development encourages frequent reflection on what you have just done with the intent of making improvements in your development techniques. Scrum has a retrospection meeting at the end of each sprint. XP promotes refactoring code continuously to advance and simplify the internal design. Feedback-based mechanisms such as updating estimates every day in a Daily Scrum meeting reinforce learning and provide a consistently accurate picture of the project.

Collaboration Agile development emphasizes open communication channels within teams, between teams, and with customers. Regular communication among project managers, developers, testers, and the customer keeps everyone moving in the same direction and quickly addresses blocking issues. The Daily Scrum meeting and Pair Programming in XP are good examples of ways to improve collaboration within and outside your team.

Eliminating Waste The idea of waste elimination is borrowed from lean software development (www.poppendieck.com), which is often paired with agile development. "Waste" comprises tasks or work (such as features) that do not significantly contribute to what the customer wants or needs. For example, doing a deeply detailed requirements analysis and big design up front is often wasteful as change later in the development cycle negates much of the effort. It is better to stay high level and discover the details as you go.

Regardless of whether you are practicing Scrum, XP, FDD or some other agile process, these core principles of agile development can guide you toward the right decisions with focus on the user and high quality. The same holds true with physical exercise—undertaking a variety of activities on a regular basis, including working on the yard or helping a friend move, is what yields successful results. Simply abide by the fundamental principles and just do it!

For more on these principles, see www.agilemanifesto.org/principles.html. Now get out there and stay agile. Keep your body and your software projects healthy by following core principles.

James Waletzky is a Senior Software Design Engineer in the Microsoft Engineering Excellence group. James focuses on improving engineering practices at Microsoft through training, consulting, and tools, while emphasizing continuous improvement.