June 2009
Test-Driven Design: Using Mocks And Tests To Design Role-Based Objects
Use Test-Driven Development with mock objects to design object oriented code in terms of roles and responsibilities, not categorization of objects into class hierarchies. Isaiah Perumalla
Entity Framework: Anti-Patterns To Avoid In N-Tier Applications
Danny Simmons explores some anti-patterns you should look out for when building n-tier applications with the Entity Framework. Daniel Simmons
Velocity: Build Better Data-Driven Apps With Distributed Caching
Microsoft Velocity exposes a unified, distributed memory cache for client application consumption. We show you how to add Velocity to your data-driven apps. Aaron Dunnington
Peer Fun: A Peer-To-Peer Work Processing App With WCF
We demonstrate creating a peer-to-peer processing platform where multiple players function together for a common purpose: getting your work done. Matt Neely
Inside Windows 7: Introducing Libraries
This is Part 1 of a multipart article series on Windows 7. This article is about the new user profile storage concept in Windows 7, called Libraries. Yochay Kiriaty
Columns
Editor's Note: Architecture and Aggregates: Toolbox: Managing in the Cloud, UX Design Patterns, Anders Hejlsberg’s The C# Programming Language, and More:
Now that you're even managing projects in the cloud, you'll need some tools to help. This month we illustrate one, discuss UX design patterns, a book by Anders Hejlsberg, and more.Scott Mitchell
 CLR Inside Out: Memory Usage Auditing For .NET Applications:
Memory usage can have a direct impact on how fast an application executes and thus is important to optimize. In this article we discuss the basics of memory optimization for .NET programs.Subramanian Ramaswamy and Vance Morrison
Data Points: Building An Out-of-Browser Client With Silverlight 3:
Silverlight 2 applications are restricted to running inside a browser. However, Silverlight 3 applications can run inside the browser or out. Here we build a social networking app as a standalone Silverlight 3 application.John Papa
 Cutting Edge: Go Beyond HTML Forms With AJAX:
This month we examine forms in the context of AJAX applications and look at various approaches to implementing features such as auto-saving, just-in-time validation, and submission throttling.Dino Esposito
 Patterns in Practice: The Unit Of Work Pattern And Persistence Ignorance:
Jeremy Miller continues his discussion of persistence patterns by reviewing the Unit of Work design pattern and examining the issues around persistence ignorance.Jeremy Miller
Test Run: .NET Module Testing with IronPython:
This month we demonstrate how easy it is to use IronPython to test .NET-based libraries.James McCaffrey
 The Polyglot Programmer: Reaping The Benefits Of Cobra:
Cobra, a descendant of Python, offers a combined dynamic and statically-typed programming model, built-in unit test facilities, scripting capabilities, and much more. Feel the power here.Ted Neward
 Foundations: Customizing the New WPF Calendar Controls:
Late last year Microsoft released Calendar and DatePicker controls for WPF in the WPF Toolkit. We’ll show you how they work, and how you can customize them.Charles Petzold
Concurrent Affairs: Solving The Dining Philosophers Problem With Asynchronous Agents:
We use the new Asynchronous Agents Library in Visual C++ 2010 to solve the classic Dining Philosophers concurrency problem.Rick Molloy
 Windows With C++: Introducing Direct2D:
If you want to develop high-performance and high-quality commercial applications, you’ll still look to C++ and native code. Direct2D will help you deliver the graphics power you need.Kenny Kerr


Test-Driven Design
Using Mocks And Tests To Design Role-Based Objects
Isaiah Perumalla
Code download available at: MSDN Code Gallery
Browse the Code Online

This article discusses:
  • Testing interaction, not implementation
  • Discovering roles and abstracting conversations
  • Abstracting conversations
  • Refactoring code to clarify intent
This article uses the following technologies:
Test-Driven Development, NMock Framework
Contents
Interaction, Not Implementation
Reading Barcodes
Discovering Roles
Finishing the Sale
Abstracting Conversations
Calculating Receipts
Getting the Product Description
Refactoring
Wrapping Up
Within the realm of Test-Driven Development (TDD), mock objects can help you discover the roles that objects should play within a system, emphasizing how the objects relate to each rather than their internal structure. This technique can be employed to support good object-oriented design. Using mock objects as an aid to design turns out to be much more interesting than the common practice of using them simply to isolate a system from external dependencies.
One of the most important benefits of TDD is that it improves the overall design of your code by forcing you to think about the design of an object's interface based on its intended use rather than its implementation. Mock objects complement the TDD process of object-oriented systems, by allowing you to write the test code for an object as though it already had everything 3 from its environment. You do this by filling in the place of an object's collaborators with mocks. This lets you design the interfaces of an object's collaborators in terms of the roles they play, before any concrete implementations of them even exist. This leads to a process of discovery where the interfaces of an object's collaborators are brought into existence based on immediate requirements, driven by need.
Following this process of test-first development using mock objects, you can not only design an object's interface from its intended use, but you can also discover and design the interfaces an object needs from its collaborators.
This article describes how to use TDD with mock objects to design object-oriented code in terms of roles and responsibilities, not categorization of objects into class hierarchies.

Interaction, Not Implementation
One of the fundamental principles of object-oriented programming is to localize all logic operating on state to the object holding the state, and to hide an object's internal structures and their state transitions. The emphasis should be on how objects communicate with other objects in the environment when triggered by an event. In practice, this can be hard to achieve. The result of designing objects this way is that each object exposes no visible state nor any of its internal structure. Since there is no visible state, you cannot query any internal structures or state to test the object's behavior by making assertions upon its state. The only thing visible is the way the object interacts with its environment. However, you can track these interactions to verify an object behavior.
Using mock objects lets you discover how an object interacts with its collaborators by making assertions that an object sends the right message to its collaborators in a given scenario. This shifts focus and design effort from how objects are classified and structured to how objects communicate with each other. This, in turn, drives the design of your system to a "Tell, don't ask" design style where each object knows very little about the structure or state of its surrounding objects. This makes the system much more flexible by allowing you to change the behavior of the system by composing different sets of objects.
To illustrate this technique, I will walk you through a short example that demonstrates TDD using mock objects.

Reading Barcodes
I am building a point-of-sale system for a large supermarket chain. The product catalog systems are in the head office and accessed by a RESTful service. The main focus for the first feature is for the system to use barcode information, either manually entered or read from a barcode scanner, to identify an item, retrieve its price, and calculate the total receipt for the sale.
The system is triggered when the cashier enters commands using a touch screen or a barcode scanner. Commands from input devices are sent as a string in the following format:
  • Command: NewSale;
  • Command: EndSale;
  • Input: Barcode=100008888559, Quantity =1;
All barcodes follow the UPC scheme; the first six characters of the barcode identify the manufacturer code, and the next five characters identify a product code. The product information system requires the manufacturer code and product code to be able to retrieve product description for an item.
The system first needs to receive and interpret commands from input devices (keyboard and scanners). When the sale is finished, the system should calculate and print a receipt using the product catalog from the head office to retrieve product description and price.
The first step is to decode the raw messages sent from the input devices into something that represents check-out events. I start with the simplest command. The initiating new sale command simply triggers a new sale event within the system. I need an object to decode the raw messages from the input devices and convert them into something that represents an event in terms of the application domain. The first test is shown below:
[TestFixture]
public class CommandParserTests {

  [Test]
  public void NotifiesListenerOfNewSaleEvent() {
    var commandParser = new CommandParser();
    var newSaleCommand= "Command:NewSale";
    commandParser.Parse(newSaleCommand);
  }
}
Note that CommandParser doesn't even exist at this point. I am using the tests to drive out the interface this object provides.
How would I know whether this object interpreted the command sent from input devices correctly? Following the "tell, don't ask" principle, the CommandParser object should tell the object it has a relationship with that a new sale was initiated. At this point, I don't know who or what it has a relationship with. This is yet to be discovered.

Discovering Roles
So far, all I know about the CommandParser's collaborator is that it needs to know when events related to a sale are detected in the system. I will choose a name that describes just this functionality—SaleEventListener—and note that it represents a role in the system. A role can be viewed as a named slot in a software system that can be filled by any object that can fulfill the responsibilities of the role. To discover roles, you need to examine the visible interactions between an object and its collaborator when performing an activity in the system, focusing only on the aspects of the objects required for the activity to be described.
In C#, a role can be specified by an interface. In this example, the CommandParser requires an object from its environment that can play the role of SaleEventListener. However, interfaces alone are inadequate to describe how a group of objects communicate to fulfill a task. Using mock objects, I can describe this in my test as shown in Figure 1.
Figure 1 Defining the SaleEventListener Role
[TestFixture]
public class CommandParserTests {
  private Mockery mockery;

  [SetUp]
  public void BeforeTest() {
    mockery = new Mockery();
  }

  [Test]
  public void NotifiesListenerOfNewSaleEvent() {
    var saleEventListener = mockery.NewMock<ISaleEventListener>();
    var commandParser = new CommandParser();
    var newSaleCommand = "Command:NewSale";
    commandParser.Parse(newSaleCommand);
  }
}
I explicitly define the role SaleEventListener using an interface, ISaleEventListener. Why am I using a listener interface instead of the built-in support C# provides for events through delegates?
There are a number of reasons why I chose to use a listener interface over the events and delegates. The listener interface explicitly identifies the role the command parser collaborates with. Note that I'm not coupling the CommandParser to any particular class, but by using a listener interface, I am stating explicitly the relationship between the CommandParser and the role it collaborates with. Events and delegates may allow you to hook in arbitrary code to the CommandParser, but it doesn't express the possible relationships between the objects in the domain. Using a listener interface in this case allows me to use the communication patterns between the objects to clearly express the domain model.
In my example, the CommandParser will parse commands and send different types of events in terms of the application domain. These will be always hooked up at the same time. In this case it is much more convenient to pass a reference to an instance of a listener interface that can process a set of events rather than attaching different delegates to each different event. I then need an instance of this interface so the CommandParser can interact with it. I use the NMock framework to dynamically create an instance of this interface.
I now need to specify what I expect CommandParser object to tell the object playing the role of ISaleEventListener when it has interpreted a command from the input device. I specify this by writing an expectation on the mock implementation of ISaleEventListener:
[Test]
public void NotifiesListenerOfNewSaleEvent() {
  var saleEventListener = mockery.NewMock<ISaleEventListener>();
  var commandParser = new CommandParser();
  var newSaleCommand = "Command:NewSale";

  Expect.Once.On(saleEventListener).Method("NewSaleInitiated");

  commandParser.Parse(newSaleCommand);
  mockery.VerifyAllExpectationsHaveBeenMet();
}
The act of writing this expectation drove out the interface the CommandParser requires from its collaborator. Using a mock object, you can discover and design the interfaces an object requires from its collaborators before any implementations of these collaborators even exist. This allows you to stay focused on the CommandParser without worrying about the implementations of its collaborators.
To get this test to compile, I need to create the CommandParser class and the ISaleEventListener interface:
public class CommandParser {
  public void Parse(string messageFromDevice) {
  }
}

public interface ISaleEventListener {
  void NewSaleInitiated();
}
The test compiles, I run the test, and I get the following failure:
TestCase 'Domain.Tests.CommandParserTests.NotifiesListenerOfNewSaleEvent'
failed: NMock2.Internal.ExpectationException : not all expected invocations were performed
Expected:
  1 time: saleEventListener.NewSaleInitiated(any arguments) [called 0 times]
    at NMock2.Mockery.FailUnmetExpectations()
    at NMock2.Mockery.VerifyAllExpectationsHaveBeenMet()
The NMock framework reports that the mock implementation of ISaleEventListener was expecting the method NewSaleInitiated to be invoked once, but this never happens. To get the test to pass, I need to pass the mock instance of the saleEventListener to the CommandParser object as a dependency.
[Test]
public void NotifiesListenerOfNewSaleEvent() {
  var saleEventListener = mockery.NewMock<ISaleEventListener>();
  var commandParser = new CommandParser(saleEventListener);
  var newSaleCommand = "Command:NewSale";

  Expect.Once.On(saleEventListener).Method("NewSaleInitiated");

  ommandParser.Parse(newSaleCommand);
  mockery.VerifyAllExpectationsHaveBeenMet();
}
The test now explicitly specifies the dependency the CommandParser has on its environment, which specifies what message (method call) the saleEventListener should receive.
Here's the simplest implementation that passes this test:
public class CommandParser {
  private readonly ISaleEventListener saleEventListener;

  public CommandParser(ISaleEventListener saleEventListener) {
    this.saleEventListener = saleEventListener;
  }

  public void Parse(string messageFromDevice) {
    saleEventListener.NewSaleInitiated();
  }
}

Finishing the Sale
Now that the test is passing, I can move on to the next test. The next simple success scenario will be to test that the CommandParser can decode the finish sale command and notify the system.
[Test]
public void NotifiesListenerOfSaleCompletedEvent() {
  var saleEventListener = mockery.NewMock<ISaleEventListener>();
  var commandParser = new CommandParser(saleEventListener);
  var endSaleCommand = "Command:EndSale";

  Expect.Once.On(saleEventListener).Method("SaleCompleted");

  commandParser.Parse(endSaleCommand);
  mockery.VerifyAllExpectationsHaveBeenMet();
}
The test drives out the need for another method the ISaleEventListener interface must support.
public interface ISaleEventListener {
  void NewSaleInitiated();
  void SaleCompleted();
}
Running the test fails, as you would expect. NMock displays the following error message:
TestCase 'Domain.Tests.CommandParserTests.NotifiesListenerOfSaleCompletedEvent'
failed: NMock2.Internal.ExpectationException : unexpected invocation of saleEventListener.NewSaleInitiated()
Expected:
  1 time: saleEventListener.SaleCompleted(any arguments) [called 0 times]
I need to interpret the raw command and call the appropriate method on instance of the saleEventListener object. The simple implementation in Figure 2 should get the test to pass.
Figure 2 Implementing saleEventListener
public class CommandParser {
  private const string END_SALE_COMMAND = "EndSale";
  private readonly ISaleEventListener saleEventListener;

  public CommandParser(ISaleEventListener saleEventListener) {
    this.saleEventListener = saleEventListener;
  }

  public void Parse(string messageFromDevice) {
    var commandName = messageFromDevice.Split(':')[1].Trim();
    if (END_SALE_COMMAND.Equals(commandName))
      saleEventListener.SaleCompleted();
    else
      saleEventListener.NewSaleInitiated();
  }
}
Before moving on to the next test, I remove duplication in the test code (see Figure 3).
Figure 3 Updating the Tests
[TestFixture]
public class CommandParserTests {
  private Mockery mockery;
  private CommandParser commandParser;
  private ISaleEventListener saleEventListener;

  [SetUp]
  public void BeforeTest() {
    mockery = new Mockery();
    saleEventListener = mockery.NewMock<ISaleEventListener>();
    commandParser = new CommandParser(saleEventListener);
    mockery = new Mockery();
  }

  [TearDown]
  public void AfterTest() {
    mockery.VerifyAllExpectationsHaveBeenMet();
  }

  [Test]
  public void NotifiesListenerOfNewSaleEvent() {
    var newSaleCommand = "Command:NewSale";

    Expect.Once.On(saleEventListener).Method("NewSaleInitiated");

    commandParser.Parse(newSaleCommand);
  }

  [Test]
  public void NotifiesListenerOfSaleCompletedEvent() {
    var endSaleCommand = "Command:EndSale";

    Expect.Once.On(saleEventListener).Method("SaleCompleted");

    commandParser.Parse(endSaleCommand);
  }
}
Next, I want to ensure the CommandParser can process an input command with barcode information. The application receives a raw message in the following format:
Input:Barcode=100008888559, Quantity =1
I want to tell the object that plays the role of a SaleEventListener that an item with a barcode and quantity was entered:
[Test]
public void NotifiesListenerOfItemAndQuantityEntered() {
  var message = "Input: Barcode=100008888559, Quantity =1";

  Expect.Once.On(saleEventListener).Method("ItemEntered")
    .With("100008888559", 1);

  commandParser.Parse(message);
}
The test drives out the need for yet another method the ISaleEventListener interface must provide:
public interface ISaleEventListener {
  void NewSaleInitiated();
  void SaleCompleted();
  void ItemEntered(string barcode, int quantity);
}
Running the test produces this failure:
TestCase 'Domain.Tests.CommandParserTests.NotifiesListenerOfItemAndQuantityEntered'
failed: NMock2.Internal.ExpectationException : unexpected invocation of saleEventListener.NewSaleInitiated()
Expected:
  1 time: saleEventListener.ItemEntered(equal to "100008888559", equal to <1>) [called 0 times]
The failure message tells me that the wrong method was called on the saleEventListener. This is expected as I haven't implemented any logic in CommandParser to handle input messages containing barcode and quantity. Figure 4 shows the updated CommandParser.
Figure 4 Handling Input Messages
public class CommandParser {
  private const string END_SALE_COMMAND = "EndSale";
  private readonly ISaleEventListener saleEventListener;
  private const string INPUT = "Input";
  private const string START_SALE_COMMAND = "NewSale";

  public CommandParser(ISaleEventListener saleEventListener) {
    this.saleEventListener = saleEventListener;
  }

  public void Parse(string messageFromDevice) {
    var command = messageFromDevice.Split(':');
    var commandType = command[0].Trim();
    var commandBody = command[1].Trim();

    if(INPUT.Equals(commandType)) {
      ProcessInputCommand(commandBody);
    }
    else {
      ProcessCommand(commandBody);
    }
  }

  private void ProcessCommand(string commandBody) {
    if (END_SALE_COMMAND.Equals(commandBody))
      saleEventListener.SaleCompleted();
    else if (START_SALE_COMMAND.Equals(commandBody))
      saleEventListener.NewSaleInitiated();
  }

  private void ProcessInputCommand(string commandBody) {
    var arguments = new Dictionary<string, string>();
    var commandArgs = commandBody.Split(',');

    foreach(var argument in commandArgs) {
      var argNameValues = argument.Split('=');
      arguments.Add(argNameValues[0].Trim(),
        argNameValues[1].Trim());
    }

    saleEventListener.ItemEntered(arguments["Barcode"],
      int.Parse(arguments["Quantity"]));
  }
}

Abstracting Conversations
Before moving on to the next test, I need to refactor and tidy up the interactions between CommandParser and saleEventListener. I want to specify interactions between an object and its collaborators in terms of the application domain. The ItemEntered message takes into two arguments a string representing a barcode and an integer representing quantity. What do these two arguments really represent in the domain of the application?
Rule of thumb: If you are passing around primitive data types among object collaborators, it may be an indication you are not communicating at the right level of abstraction. You need to see whether the primitive data types represent concepts in your domain that you may have missed.
In this case, the barcode is decomposed into manufactureCode and itemCode, which represents an item identifier. I can introduce the concept of an item identifier in the code. This should be an immutable type that can be constructed from a barcode, and I can give the ItemIdentifier the responsibility of decomposing the barcode into a manufacturer code and an item code. Similarly, quantity should be a value object as it represents a measurement—for example, the quantity of an item could be measured by weight.
For now, I don't have a need yet to decompose the barcode or handle different types of measurements for quantity. I will simply introduce these value objects to ensure that communication between objects remains in domain terms. I refactor the code to include the concept of item identifier and quantity in the test.
[Test]
public void NotifiesListenerOfItemAndQuantityEntered() {
  var message = "Input: Barcode=100008888559, Quantity=1";
  var expectedItemid = new ItemId("100008888559");
  var expectedQuantity = new Quantity(1);

  Expect.Once.On(saleEventListener).Method("ItemEntered").With(
    expectedItemid, expectedQuantity);

  commandParser.Parse(message);
}
Neither ItemId nor Quantity exist yet. To get the test to pass, I need to create these new classes and modify the code to reflect these new concepts. I implement these types as value objects. The identities of these objects are based on the values they hold (see Figure 5).
Figure 5 ItemID and Quantity
public interface ISaleEventListener {
  void SaleCompleted();
  void NewSaleInitiated();
  void ItemEntered(ItemId itemId, Quantity quantity);
}

public class ItemId {
  private readonly string barcode;

  public ItemId(string barcode) {
    this.barcode = barcode;
  }

  public override bool Equals(object obj) {
    var other = obj as ItemId;
    if(other == null) return false;
    return this.barcode == other.barcode;
  }

  public override int GetHashCode() {
    return barcode.GetHashCode();
  }

  public override string ToString() {
    return barcode;
  }
}

public class Quantity {
  private readonly int value;

  public Quantity(int qty) {
    this.value = qty;
  }

  public override string ToString() {
    return value.ToString();
  }

  public override bool Equals(object obj) {
    var otherQty = obj as Quantity;
    if(otherQty == null) return false;
    return value == otherQty.value;
  }

  public override int GetHashCode() {
    return value.GetHashCode();
  }
}
With interaction-based tests using mock objects, you can streamline the interactions between an object and its collaborators by using tests to explicitly describe communication protocols between objects at a high-level of abstraction in terms of the domain. Since mocks allow you do this without having any concrete implementations of the collaborators to exist, you can try out alternate collaboration patterns until you have designed the collaborations between objects in terms of the application domain. Also by examining and closely following the interactions between object and its collaborator, it also helps dig out any domain concepts you may have overlooked.

Calculating Receipts
Now that I have an object to decode commands from input devices as point-of-sale events, I need an object that can respond and process these events. The requirement is to print out receipts, so I need an object that can calculate receipts. To fill these responsibilities, I look for an object that can play the role of a SaleEventListener. The concept of a Register comes to mind and seems to fit the role of a SaleEventListener, so I create a new class Register. Since this class responds to sale events, I make it implement ISaleEventListener:
public class Register : ISaleEventListener {
  public void SaleCompleted() { }
  public void NewSaleInitiated() { }
  public void ItemEntered(ItemId itemId, Quantity quantity) { }
}
One of the main responsibilities of the Register is to calculate receipts and send them to a printer. I will rig some events on the object by invoking its methods. I start with the simple scenario. Calculating the receipt for a sale with no items should have a total of 0. I need to ask the question: Who would know if the register has calculated the total for the items correctly? My first guess is a receipt printer. I express this in code by writing a test:
[Test]
public void ReceiptTotalForASaleWithNoItemsShouldBeZero() {
  var receiptPrinter = mockery.NewMock<IReceiptPrinter>();
  var register = new Register();
  register.NewSaleInitiated();

  Expect.Once.On(receiptPrinter).Method("PrintTotalDue").With(0.00);

  register.SaleCompleted();
}
The test implies that the Register object tells the receipt printer to print the total due.
Before moving forward let's take a step back and examine the communication protocol between the Register object and the receipt printer. Is the PrintTotalDue method meaningful to the Register object? Looking at this interaction it's clearly not the Register's responsibility to be concerned with printing receipts. The Register object is concerned with calculating the receipt and sending it to an object that receives receipts. I will choose a name for the method that describes just that behavior: ReceiveTotalDue. This is much more meaningful to the Register object. In doing this I discovered that the collaborating role the Register requires is a ReceiptReceiver rather than a ReceiptPrinter. Finding the appropriate name for a role is an important part of the design activity, as it helps you design objects that have a cohesive set of responsibilities. I rewrite the test to reflect the new name for the role.
[Test]
public void ReceiptTotalForASaleWithNoItemsShouldBeZero() {
  var receiptReceiver = mockery.NewMock<IReceiptReceiver>();
  var register = new Register();
  register.NewSaleInitiated();

  Expect.Once.On(receiptReceiver).Method("ReceiveTotalDue").With(0.00);

  register.SaleCompleted();
}
To get this to compile, I create an IReceiptReceiver interface to represent the role ReceiptReceiver.
public interface IReceiptReceiver {
  void ReceiveTotalDue(decimal amount);
}
When I run the test I get a failure, as expected. The mock framework tells me the method call ReceiveTotalDue was never made. To make the test pass I need to pass a mock implementation of IReceiptReceiver to the Register object, so I change the test to reflect this dependency.
[Test]
public void ReceiptTotalForASaleWithNoItemsShouldBeZero() {
  var receiptReceiver = mockery.NewMock<IReceiptReceiver>();
  var register = new Register(receiptReceiver);
  register.NewSaleInitiated();

  Expect.Once.On(receiptReceiver).Method("ReceiveTotalDue").With(0.00m);

  register.SaleCompleted();
}
The simple implementation here should get the test to pass:
public class Register : ISaleEventListener {
  private readonly IReceiptReceiver receiptReceiver;

  public Register(IReceiptReceiver receiver) {
    this.receiptReceiver = receiver;
  }

  public void SaleCompleted() {
    receiptReceiver.ReceiveTotalDue(0.00m);
  }

  public void NewSaleInitiated() { }

  public void ItemEntered(ItemId itemId, Quantity quantity) { }
}
The primitive type decimal used to represent the total amount due is just a scalar, which has no meaning in the domain. What this really represents is monetary values, so I will create an immutable value object to represent money. At this point, there is no need for multi-currency or rounding of monetary values. I simply create a Money class that wraps the decimal value. When the need arises, I can add currency and rounding rules in this class. For now, I will stay focused on the current task and modify the code to reflect this. The implementation is shown in Figure 6.
Figure 6 Using Money
public interface IReceiptReceiver {
  void ReceiveTotalDue(Money amount);
}

public class Register : ISaleEventListener {
  private readonly IReceiptReceiver receiptReceiver;

  public Register(IReceiptReceiver receiver) {
    this.receiptReceiver = receiver;
  }

  public void SaleCompleted() {
    receiptReceiver.ReceiveTotalDue(new Money(0.00m));
  }

  public void NewSaleInitiated() { }

  public void ItemEntered(ItemId itemId, Quantity quantity) { }
}

[Test]
public void ReceiptTotalForASaleWithNoItemsShouldBeZero() {
  var receiptReceiver = mockery.NewMock<IReceiptReceiver>();
  var register = new Register(receiptReceiver);
  register.NewSaleInitiated();

  var totalDue = new Money(0m);
  Expect.Once.On(receiptReceiver).Method("ReceiveTotalDue")
    .With(totalDue);
  register.SaleCompleted();
}
The next test will flesh out some additional behavior on the Register object. The Register should not calculate receipts if new sales are not initiated, so I write a test to specify this behavior:
[SetUp]
public void BeforeTest() {
  mockery = new Mockery();
  receiptReceiver = mockery.NewMock<IReceiptReceiver>();
  register = new Register(this.receiptReceiver);
}

[Test]
public void ShouldNotCalculateRecieptWhenThereIsNoSale() {
  Expect.Never.On(receiptReceiver);
  register.SaleCompleted();
}
This test explicitly specifies that the receiptReceiver should never receive any method calls on it. The test fails as expected with the following error:
TestCase 'Domain.Tests.RegisterTests.  ShouldNotCalculateRecieptWhenThereIsNoSale'
failed: NMock2.Internal.ExpectationException : unexpected invocation of   receiptReceiver.ReceiveTotalDue(<0.00>)
To get this test to pass, the Register object has to keep track of some state—to track whether there is a sale in progress. I can make the test pass with the implementation shown in Figure 7.
Figure 7 Keeping Track of State with Register
public class Register : ISaleEventListener {
  private readonly IReceiptReceiver receiptReceiver;
  private bool hasASaleInprogress;

  public Register(IReceiptReceiver receiver) {
    this.receiptReceiver = receiver;
  }

  public void SaleCompleted() {
    if(hasASaleInprogress)
      receiptReceiver.ReceiveTotalDue(new Money(0.00m));
  }

  public void NewSaleInitiated() {
    hasASaleInprogress = true;
  }

  public void ItemEntered(ItemId itemId, Quantity quantity) { }
}

Getting the Product Description
The product information systems are located in the head office, which is exposed as a RESTful service. The Register object will need to retrieve product information from this system to work out the receipt for a sale. I don't want to be constrained by the implementation details of this external system, so I will define my own interface in domain terms for the services the point of sale system needs.
I will write a test to calculate the receipt for a sale with a couple of items. To work out the total for a sale, the register needs to collaborate with another object. To retrieve [a] product description for an item, I introduce the role of a product catalog. This could be a RESTful service, a database, or some other system. The implementation detail is not important nor of any concern to the Register object. I want to design an interface that is meaningful to the Register object. The test is shown in Figure 8.
Figure 8 Testing Register
[TestFixture]
public class RegisterTests {
  private Mockery mockery;
  private IReceiptReceiver receiptReceiver;
  private Register register;
  private readonly ItemId itemId_1 = new ItemId("000000001");
  private readonly ItemId itemId_2 = new ItemId("000000002");
  private readonly
    ProductDescription descriptionForItemWithId1 =
    new ProductDescription("description 1", new Money(3.00m));

  private readonly
    ProductDescription descriptionForItemWithId2 =
    new ProductDescription("description 2", new Money(7.00m));
  private readonly Quantity single_item = new Quantity(1);
  private IProductCatalog productCatalog;

  [SetUp]
  public void BeforeTest() {
    mockery = new Mockery();
    receiptReceiver = mockery.NewMock<IReceiptReceiver>();
    productCatalog = mockery.NewMock<IProductCatalog>();
    register = new Register(receiptReceiver, productCatalog);

    Stub.On(productCatalog).Method("ProductDescriptionFor")
      .With(itemId_1)
      .Will(Return.Value(descriptionForItemWithId1));

    Stub.On(productCatalog).Method("ProductDescriptionFor")
      .With(itemId_2)
      .Will(Return.Value(descriptionForItemWithId2));

  }

  [TearDown]
  public void AfterTest() {
    mockery.VerifyAllExpectationsHaveBeenMet();
  }
...
  [Test]
  public void
    ShouldCalculateRecieptForSaleWithMultipleItemsOfSingleQuantity() {

    register.NewSaleInitiated();
    register.ItemEntered(itemId_1, single_item);
    register.ItemEntered(itemId_2, single_item);

    Expect.Once.On(receiptReceiver).Method("ReceiveTotalDue")
      .With(new Money(10.00m));

   register.SaleCompleted();
  }
}
This test designed the interface for the productCatalog that is required by the Register object. I also discovered the need for a new type, productDescription, that represents the description of a product. I will model this as a value object (immutable type). I stub the productCatalog to give a productDescription when queried with an ItemIdentifier. I stub the invocation of a ProductDescriptionFor method on productCatalog because this is a query method that returns the productDescription. The Register acts on the result of the query returned. What is important here is that the Register produces the correct side effects when the ProductCatalog returns a specified ProductDescription. The rest of the test verifies that the correct total is calculated correctly and is sent to the receipt receiver.
I run the test to get the expected failure:
unexpected invocation of receiptReceiver.ReceiveTotalDue(<0.00>)
Expected:
  Stub: productCatalog.ProductDescriptionFor(equal to <000000001>), will
       return <Domain.Tests.ProductDescription> [called 0 times]
  Stub: productCatalog.ProductDescriptionFor(equal to <000000002>), will
      return <Domain.Tests.ProductDescription> [called 0 times]
  1 time: receiptReceiver.ReceiveTotalDue(equal to <10.00>) [called 0 times]
The mock framework tells me that receiptReceiver should have received a total of 10, but a 0 was received. This is expected since we haven't implemented anything that can calculate the total. Figure 9 shows a first attempt at the implementation to get the test to pass.
Figure 9 Calculating the Total
public class Register : ISaleEventListener {
  private readonly IReceiptReceiver receiptReceiver;
  private readonly IProductCatalog productCatalog;
  private bool hasASaleInprogress;
  private List<ProductDescription> purchasedProducts =
    new List<ProductDescription>();

  public Register(IReceiptReceiver receiver,
    IProductCatalog productCatalog) {

    this.receiptReceiver = receiver;
    this.productCatalog = productCatalog;
  }

  public void SaleCompleted() {
    if(hasASaleInprogress) {
      Money total = new Money(0m);
      purchasedProducts.ForEach(item => total += item.UnitPrice);
      receiptReceiver.ReceiveTotalDue(total);
    }
  }

  public void NewSaleInitiated() {
    hasASaleInprogress = true;
  }

  public void ItemEntered(ItemId itemId, Quantity quantity) {
    var productDescription = productCatalog.ProductDescriptionFor(itemId);
    purchasedProducts.Add(productDescription);
  }
}
This code fails to compile since the Money class doesn't define an operation to add money. I now have a need for Money objects to perform addition, so I write a quick test to for the Money class to handle this immediate requirement.
[TestFixture]
public class MoneyTest {

  [Test]
  public void ShouldBeAbleToCreateTheSumOfTwoAmounts() {
    var twoDollars = new Money(2.00m);
    var threeDollars = new Money(3m);
    var fiveDollars = new Money(5m);
    Assert.That(twoDollars + threeDollars, Is.EqualTo(fiveDollars));
  }
}
The implementation to get this test passing is shown in Figure 10.
Figure 10 Updates to Add Money
public class Money {
  private readonly decimal amount;

  public Money(decimal value) {
    this.amount = value;
  }

  public static Money operator +(Money money1, Money money2) {
    return new Money(money1.amount + money2.amount);
  }

  public override string ToString() {
    return amount.ToString();
  }

  public override bool Equals(object obj) {
    var otherAmount = obj as Money;
    if(otherAmount == null) return false;
    return amount == otherAmount.amount;
  }

  public override int GetHashCode() {
    return amount.GetHashCode();
  }
}

Refactoring
Before going any further, I need to clarify the intent of my code. My tests don't track any of the Register object's internal details. All of this is hidden inside the Register object, which allows me to refactor and clarify the intent of the code.
The Register object is managing state related to the current sale, but I don't have an object that represents the concept of a sale. I decided to extract a Sale class to manage all state and behavior related to a sale. The refactored code is shown in Figure 11.
Figure 11 Adding a Sale Class
public class Register : ISaleEventListener {
  private readonly IReceiptReceiver receiptReceiver;
  private readonly IProductCatalog productCatalog;
  private Sale sale;

  public Register(IReceiptReceiver receiver,
    IProductCatalog productCatalog) {

    this.receiptReceiver = receiver;
    this.productCatalog = productCatalog;
  }

  public void SaleCompleted() {
    if(sale != null) {
      sale.SendReceiptTo(receiptReceiver);
    }
  }

  public void NewSaleInitiated() {
    sale = new Sale();
  }

  public void ItemEntered(ItemId itemId, Quantity quantity) {
    var productDescription =
      productCatalog.ProductDescriptionFor(itemId);
    sale.PurchaseItemWith(productDescription);
  }
}

public class Sale {
  private readonly List<ProductDescription> itemPurchased =
    new List<ProductDescription>();

  public void SendReceiptTo(IReceiptReceiver receiptReceiver) {
    var total = new Money(0m);
    itemPurchased.ForEach(item => total += item.UnitPrice);
    receiptReceiver.ReceiveTotalDue(total);
  }

  public void PurchaseItemWith(ProductDescription description) {
    itemPurchased.Add(description);
  }
}

Value Objects
In .NET terminology, value types refer to the primitive types supported by the CLR such as int, bool, structs, enum, etc. This should not be confused with value objects; these are objects that describe things. The important thing to note is that value objects can be implemented by classes (reference types); these objects are immutable and have no conceptual identity. Since Value objects have no individual identity, two values objects are considered equal if they have exactly the same state.
Wrapping Up
This example showed you how mock objects and TDD can guide the design of object-oriented programs. There are a number of important side benefits of this iterative discovery process. Using mock objects not only helped me discover the collaborators and object needs, but I was also able to describe and make explicit in my tests the roles an object's environment must support and communication patterns between the object under test and its collaborators.
Note that the tests show only what the object does in terms of domain concepts. The tests don't specify how the Register interacts with a Sale. These are internal details regarding how the Register object is structured to accomplish its work, and these implementation details remain hidden. I chose mocks only to explicitly specify interactions that should be visible externally between an object and its collaborators in the system.
The tests using mock objects lay out a set of constraints specifying what messages objects can send and when those messages should be sent. The interactions between objects are clearly described in terms of domain concepts. The domain consists of narrow role interfaces that make the system pluggable, and the behavior of the system can be altered easily by changing the composition of its objects.
None of the objects expose their internal state or structure, and only immutable states (value objects like money and itemId) are passed around. This makes programs easier to maintain and modify. One thing to note is that you normally would have started the development with a failing acceptance, which would have driven the requirements and the need for the CommandParser in my example.

Isaiah Perumalla is a senior developer at ThoughtWorks, where he is involved in delivering enterprise-scale software, using object-oriented design and Test-Driven Development.


Entity Framework
Anti-Patterns To Avoid In N-Tier Applications
Daniel Simmons

This article discusses:
  • Understanding n-tier
  • Don't distribute your objects!
  • Custom service or RESTful service?
  • Some n-tier anti-patterns
This article uses the following technologies:
Entity Framework
Contents
Understanding N-tier
Anti-Pattern #1: Tight Coupling
Anti-Pattern #2: Assuming Static Requirements
Anti-Pattern #3: Mishandled Concurrency
Anti-Pattern #4: Stateful Services
Anti-Pattern #5: Two Tiers Pretending to be Three
Anti-Pattern #6: Undervaluing Simplicity
As a member of the Entity Framework team, I frequently talk to customers about building applications that use the Entity Framework. Probably the topic I get asked about more than anything else is designing n-tier applications. In this article, I will try to set a foundation on which you can build for success in this part of your applications. The majority of the article is devoted to design anti-patterns for n-tier, which are usually the most important issues that I find. This is a topic where there are a lot of options and many issues to consider, so it is important to understand the overall space before making decisions for your particular application. In future articles, I will examine n-tier patterns for success and some of the key APIs and issues specific to the Entity Framework, and provide a sneak peak at features coming in the Microsoft .NET Framework 4 that should make n-tier significantly easier.

Understanding N-tier
Before I dive into the anti-patterns, it is important to have a common understanding of n-tier.
The first point to be clear on is the difference between tiers and layers. A well-designed application will have multiple layers with carefully managed dependencies. Those layers could live in a single tier or be split across multiple tiers. A layer is just an organizational concept in an application, while a tier denotes physical separation or at least a design that will allow physical separation if needed.
Any application that talks to a database has more than one tier unless that database runs in-process, but the application is not called n-tier unless it involves more tiers than just the database and application. Similarly, every ASP.NET application that involves a database is technically n-tier because there is the database, the Web server, and the browser. Unless you introduce Windows Communication Foundation (WCF) or Web services, you would not call that application n-tier since for most purposes the Web server and browser can be thought of as a single client tier. N-tier applications are those that have at a minimum a database tier, a middle tier that exposes a service, and a client tier.
While it does not sound like that big a deal on the surface, it turns out that implementing applications split across multiple tiers is difficult. There are a lot more pitfalls than you think. These pitfalls led Martin Fowler, in his book Patterns of Enterprise Application Architecture (Addison-Wesley, 2002), to make a very strong statement on the subject:
Don't distribute your objects!
Martin calls it the First Law of Distributed Objects.
As with every design rule, though, there are times when the law must be set aside. Your application may have a scalability problem that requires multiple tiers so you can apply more computing resources. Maybe you need to exchange data with a business partner or customer application. It may just be that you have security or infrastructure constraints that divide your application onto multiple computers or prevent one part of your application from talking directly to another part. When you need multiple tiers, you really need them.
While the anti-patterns presented in this article can be applied to a wide range of applications and technologies, the main focus will be creating and consuming custom WCF services that persist data using the Entity Framework.
Not surprisingly, many n-tier anti-patterns are a result of losing focus on the goal of your application. If you do not keep in mind what motivated you to use an n-tier architecture in the first place, or if you neglect critical persistence concerns, then it is all too easy to get in trouble. The following sections will look at some common problems.

Custom Service or RESTful Service?
REST, or Representational State Transfer, is a type of Web service that is rapidly gaining in popularity. So you might ask yourself what the difference is between RESTful services and custom Web services, and why you might choose one type over the other. The key difference between the two types is that REST services are resource-centric while custom services are operation-centric. With REST, you divide your data into resources, give each resource a URL, and implement standard operations on those resources that allow creation, retrieval, update, and deletion (CRUD). With custom services, you can implement any arbitrary method, which means that the focus is on the operations rather than the resources, and those operations can be tailored to the specific needs of your application.
Some services fit very naturally into the REST model—usually when the resources are obvious and much of the service involves management of those resources. Exchange Server, for instance, has a REST API for organizing e-mail and calendar items. Similarly, there are photo-sharing Web sites on the Internet that expose REST APIs. In other cases, the services less clearly match REST operations, but can still be made to fit. Sending e-mail, for example, can be accomplished by adding a resource to an outbox folder. This is not the way you would most naturally think about sending e-mail, but it is not too much of a stretch.
In other cases, though, REST operations just do not fit well. Creating a resource in order to initiate a workflow that drives monthly payroll check printing, for example, would be much less natural than having a specific method for that purpose.
If you can fit your service into the constraints of REST, doing so will buy you a lot of advantages. ADO.NET Data Services in combination with the Entity Framework makes it easy to create both RESTful services and clients to work with them. The framework can provide more functionality to RESTful services automatically because the services are constrained to follow a specific pattern. In addition, RESTful services have a very broad reach because they are so simple and interoperable. They work especially well when you do not know in advance who the clients might be. Finally, REST can be made to scale to handle very large volumes of operations.
For many applications, the constraints of REST are just too much. Sometimes the domain does not divide clearly into a single pattern of resources or the operations involve multiple resources at once. Sometimes the user actions and business logic around them do not map well to RESTful operations, or more precise control is required than can fit into those operations. In these cases, custom services are the way to go.
You can always build an application that has a mix of REST and custom services. Often the ideal solution for an application is a mixture of both.
Anti-Pattern #1: Tight Coupling
Chances are you have heard about the evils of tight coupling. So you always strive to keep your components as loosely coupled as possible, right? Yeah, right.
Loose coupling is more difficult than tight coupling, and often the performance is not as good. You start off with the best of intentions, but end up asking if the benefit is worth the cost. Why introduce an interface and dependency injection when you could just create an instance of the class and call the method directly? Why build an abstraction with custom objects mapped to the database instead of filling a DataTable and passing it around?
To make matters worse, you do not usually feel the pain of tight coupling until much later. In the short term, you gain some efficiency and get the job done, but in the long run evolving the application can become almost impossible.
If you have been building software for any time at all, you probably understand coupling tradeoffs fairly well when it comes to modules within a tier. When you have modules that work together closely, sometimes tight coupling is the right choice, but in other cases, components need to be kept at arm's length from one another so that you can contain the ripple effect of changes to the application.
When it comes to splitting parts of your application into separate tiers, the significance of coupling becomes much greater. The reason for this is simple. Tiers do not always change at the same rate. If you have a service that is consumed by many clients and you cannot guarantee all the clients will upgrade on demand, then you better make sure you can change that service without having to change the clients. If not, you will encounter a problem that is sometimes called shearing rates of change. Imagine your application being pulled in two different directions until it is forcefully ripped apart.
The trick is to identify which parts of the application might have different rates of change and which parts are tightly coupled to each other. First, consider the boundary between the database and the mid-tier. As your application grows, there is a good chance you will need to adjust the database to improve performance, and if you ever share the database between multiple applications, there is a very good chance you will want to evolve one application's mid-tier without changing the other one. Fortunately, using the Entity Framework already helps here because its mapping system provides an abstraction between your mid-tier code and the database. The same questions should be considered between the mid-tier and the client.
A particularly common and painful example of this anti-pattern in action is an architecture that uses table adapters to retrieve data from the database and Web services that exchange DataSets with the client. The table adapter moves the data into a DataSet with the same schema (thus tightly coupling the database to the mid-tier) and then the Web service exchanges that same DataSet with the client (thus tightly coupling the mid-tier to the client). This kind of system is easy to create—there are Visual Studio tools that lead you through the process nicely. But if you build a system that way, changes to any part of the system are likely to ripple to all other parts.

Anti-Pattern #2: Assuming Static Requirements
Speaking of changes to the system, sometimes you design around an assumption that requirements will remain static, but there are two cases where changing requirements have an especially significant impact. One comes from treating the client as trusted, and the other occurs when the mid-tier service assumes the client will be implemented using a particular technology.
While it is unlikely that trust boundaries will change unexpectedly, when it comes to data integrity, security, and trust, the consequences of getting it wrong are just too great. If you perform validation only on the client, for instance, and on the mid-tier trust that the data you receive is OK to send directly to the database without revalidating, the chance that something will eventually go wrong is much larger than you might think. Even knowing the service only runs within your intranet is not enough to keep your information safe. Someone might create another client using the same service or modify the first client to call the service from a different code path that skips validation. Who knows what might happen.
Further, once you have a service, it is more likely that regular code to be used in ways that you did not anticipate than—so much so that the generally accepted wisdom is that you should always validate and enforce some degree of security on the mid-tier even though that may mean validating or performing access control more than once.
The second issue, locking the client into a particular technology, is even more likely to be a problem. Technologies always change. If an application survives long enough, something will happen that forces technology adjustments, and clients are especially susceptible. You may initially design your application as a rich client desktop application and then later find you need to move it to a mobile phone or Silverlight. If that were the case, and you designed your service to exchange DataSets, then major surgery would be needed for the service and all existing clients.

Anti-Pattern #3: Mishandled Concurrency
While there is a tight coupling downside to exchanging DataSets, concurrency is a complex-but-important area that the DataSet handles well. Unfortunately many developers do not understand the nuances of managing concurrency, and to make things worse, a mistake with concurrency is the kind of problem that often only shows up once the application is in production. If you are lucky, it will manifest as an obvious failure. If not, it may cause corruption to your data over a long period of time without being detected.
At its core, concurrency management is fairly simple: guarantee data integrity even if two clients try to modify the same data at roughly the same time. Particularly attentive readers will note that these problems also come up in cases that are unrelated to n-tier, but concurrency issues are particularly relevant to the Entity Framework n-tier designs, because the Entity Framework's handling of n-tier scenarios creates unique concurrency challenges.
For most applications, the concurrency management technique of choice is optimistic concurrency. Even though many clients may access the database simultaneously, the number of times when the exact same entity is modified in conflicting ways is quite small. So you assume everything will work out, but take steps to detect if something goes wrong.
Detection is driven by one or more properties, collectively called the concurrency token, that change whenever any part of the entity changes. When the application reads an entity, it saves the value of the concurrency token. Later, when it wants to write that entity back to the database, it first checks to make sure that the value of the concurrency token in the database is the same now as it was when the entity was originally read. If it is, the update proceeds. If not, the update halts and throws an exception.
The Entity Framework supports optimistic concurrency by transparently tracking the original value of concurrency tokens when entities are queried and checking for conflicts prior to database updates. The problem with n-tier applications is that this process works transparently only as long as a single ObjectContext instance is used to track the entity from the time it is queried until the time SaveChanges is called. If you serialize entities from one tier to another, the recommended pattern is to keep the context around on the mid-tier only long enough for a single service method call. Subsequent calls will spin up a new instance of the context to complete each task. (Creating a new context instance for every service operation is an important recommendation in its own right, by the way. For more information, see Anti-Pattern #4: Stateful Services.)
Once developers begin to learn how the Entity Framework APIs work for this kind of disconnected operation—disconnected in the sense that the entities are disconnected from the context after the query, sent to another tier, and then re-connected when it is time to save—there is a tendency to fall into a nasty pattern:
  1. Query the entity and serialize it to the client. At this point, the concurrency token's current value is the same as the original value, and that is the only value sent to the client.
  2. The client receives the entity, makes changes, and sends a modified version of the entity back to the mid-tier.
  3. Since neither the client nor the service explicitly kept track of the concurrency token or what properties have been modified, the service queries the database to get the current state of the entity into a newly created context, then compares values between the current entity from the database and the one sent back from the client.
  4. The service calls SaveChanges, which performs optimistic concurrency checks while persisting the changes.
Did you see the problem? Actually there are two problems.
First, every time an entity is updated, it has to be read from the database twice—once when it is first queried and a second time right before the update—which creates a significant extra load on the system.
Second, and more importantly, the "original value" used by the Entity Framework to check if the entity has been modified in the database comes from the second query instead of the first one. That is, it comes from the query that happens right before the update. So the result is that the optimistic concurrency check made by the Entity Framework will almost never fail. If someone else modifies the entity between the first query and the second one, the system will not detect the conflict because the value used for the concurrency check is from after the other client's modification instead of before it.
There is still a small window (between the second query and the update) when the optimistic concurrency check could detect a problem, so you still have to write your program to handle the exception, but you will not really have protected your data from corruption.
The correct pattern is either to make a copy of the entity on the client and send back both the original version unmodified and the modified version or to write the client in such a way that it does not modify the concurrency token. If the concurrency token is updated by a server trigger or automatically because it is a row version number (probably the best plan anyway), then there is no reason to modify it on the client. The current value of the property can be left untouched and used as storage for the original value.
This is a reasonably sound approach because if a bug in the client causes the value to accidentally be modified, it is highly unlikely that you will get a false success. That bug might cause you to get a false failure, but that is much more acceptable than false success.
To make this approach work, when the mid-tier receives the entity from client, you need to attach it to the context and then go over its properties, manually marking them as modified. In either case, though, you will fix both of the problems with the anti-pattern at once. You will no longer query the database twice, and the concurrency check will be based on the correct value of the token (from the initial query) rather than some later value.

Anti-Pattern #4: Stateful Services
Given the comparative ease of developing client-server solutions, the next anti-pattern comes up when developers try to simplify things by keeping the context around across multiple service calls. This seems nice at first because it sidesteps the concurrency problems. If you keep the context alive on the mid-tier, then it will contain the correct original entity values. When you receive an entity back from the client, you can compare the updated entity with the version of the entity in the context and apply changes as appropriate. When you save the entity, the correct concurrency check will be made and no extra database query is required.
While this approach seems easy on the surface, there are a number of problems lurking. Managing the context lifetime can get tricky quickly. When you have multiple clients calling the services, you have to maintain a separate context for each client or risk collisions between them. And even if you solve those issues, you will end up with major scalability problems.
These scalability problems are not only the result of tying up server resources for every client. In addition you will have to guard against the possibility that a client might start a unit of work, but never complete it, by creating an expiration scheme. Further, if you decide that you need to scale your solution out by introducing a farm with multiple mid-tier server, then you will have to maintain session affinity to keep a client associated with the same server where the unit of work began.
A lot of effort and specialized technology has been expended on addressing these issues when, in fact, the best solution is to avoid them altogether by keeping your mid-tier service implementations stateless. Each time a service call is made, the mid-tier should spin up the necessary resources, handle the call, and then release all resources specific to that call. If some information needs to be maintained for a unit of work that extends across multiple service calls, then that information should be maintained by the client rather than the mid-tier so there is no session affinity, no need to expire unfinished units of work, and no server resources in use for a particular client in between service calls.

Anti-Pattern #5: Two Tiers Pretending to be Three
Another anti-pattern I encounter fairly often is also an attempt to simplify this process. Usually it shows up as a request something like, "Why can't you make the Entity Framework serialize queries across tiers?" followed almost immediately by, "Oh, and while you are at it, can you support initiating updates from another tier as well?"
These are probably features Microsoft could add to the Entity Framework, but if you stop and think about them for a minute in light of the other issues I have discussed, you would have to question whether this is a good idea.
If you could create an Entity Framework ObjectContext on the client tier, execute any Entity Framework query to load entities into that context, modify those entities, and then have SaveChanges push an update from the client through the mid-tier to the database server—if you could do all that, then why have the mid-tier at all? Why not just expose the database directly?
Remember Fowler's First Law of Distributed Objects. Keep in mind that the only time this kind of architecture makes sense is when you really, really need it. If you really need it, then you need better security or the ability to scale out with multiple servers, or some other thing that I suggest will not really be solved by introducing a mid-tier that is simply a thin proxy for the database. You might use this technique to subvert a restriction placed on you by a corporate policy, but it is hardly capturing the spirit of an n-tier application. My suggestion is to either invest in building an n-tier application to meet a particular need or, if you can get away with it, avoid n-tier altogether.

Anti-Pattern #6: Undervaluing Simplicity
This brings me to the last n-tier anti-pattern. In the name of avoiding all the anti-patterns discussed previously, it is easy to decide that you need to create the most carefully architected, multi-tier, fully separated, re-validating, super design that you can come up with. Then you can spend all your time building infrastructure and none of your time actually delivering value to your users.
It is important to think over your goals and consider whether you are going to need the investment n-tier requires. Simple is good. Sometimes a two-tier app is just the thing. Sometimes you need more tiers than that, but everything is under your control and trusted or you have an AJAX, Silverlight, or click-once client that auto-deploys so that you do not have to worry about shearing rates of change.
If you can make the problem simpler, do so. Put in all the effort for the full solution if you must, but by the same token make sure you put in enough effort to do the job in a way that meets your goals.

Danny Simmons is dev manager for the Entity Framework team at Microsoft. You can read more of his thoughts on the Entity Framework and other subjects at blogs.msdn.com/dsimmons/.


Velocity
Build Better Data-Driven Apps With Distributed Caching
Aaron Dunnington
Code download available from the MSDN Code Gallery
Browse the Code Online

This article is based on a prerelease version of Microsoft Velocity. All information is subject to change.
This article discusses:
  • Getting started with Velocity
  • Classifying data for distributed caching
  • A basic Velocity client application
  • Integrating with ASP.NET session state
This article uses the following technologies:
Microsoft Velocity, ASP.NET
Contents
Current Data-Centric Applications
How Velocity Helps
Data Classification
Logical Hierarchy
Partitioned and Local Cache
Configuration Store
Getting Started with Velocity
Cluster Deployment
A Basic Velocity Client Application
Designing the Cache
Named Caches
Organizing the Cache
Organizing the Reports
Organizing the Session
Programming a Cache Layer
Integrating with ASP.NET Session State
Going Forward
Data-driven applications have become pervasive. Data flows from a myriad of sources including relational sources, service-oriented applications, syndicated feeds, and data-centric documents and messages. Application architectures continue to evolve to leverage this wealth of data accessibility. Additionally, the underlying hardware trends and technological advances across processing, storage, memory, and connectivity have enabled application architectures to harness inexpensive, commodity hardware to effectively scale out these applications.
A new Microsoft project, code-named Velocity, provides a distributed, in-memory cache. Velocity enables you to create scalable, available, high-performance applications by exposing a unified view of distributed memory for client application consumption. Through Velocity, client applications can enhance performance by bringing data closer to the logic that consumes it, thus reducing pressure on the underlying data store. The Velocity cluster offers high availability to further insulate applications from data loss as well as increased load on the data tier. By leveraging distributed memory, Velocity enables these high-performance, available applications to flexibly scale out as application demand increases.
This article focuses on how key features of Velocity can be leveraged to deliver the next level of scalability, availability, and performance in new and existing distributed .NET applications.

Current Data-Centric Applications
Your typical current data-powered application could be something along the lines of a simple storefront over a database. This application enables users to browse a product catalog and purchase products through a shopping cart experience. Additionally, aggregated order data is utilized to display top-selling products as well as products commonly purchased together. This application architecture takes on the familiar separation of user interface, business logic, and data access as shown in Figure 1. This architecture, unfortunately, faces several key challenges in the areas of scalability, performance, and availability.
Figure 1 Typical Data-based Web App Architecture
First, there is a major scalability challenge inherent in that the application performs database operations upon every page load in the form of retrieval operations for catalog and top-seller data. As application load increases, this heavy frequency in database operations can pose severe scalability limitations and performance bottlenecks due to an increased contention for resources and the physical limitations of retrieving data from disk. Further, this architecture can dramatically impact the latency of the application as each page load results in operations that travel completely through to the data tier.
If the application utilizes traditional caching approaches in the Web tier to reduce constant pressure on the database, a second key scalability challenge that arises is rooted in coupling of state to specific server resources. Without decoupling cache state from specific servers in order to scale, the application must employ techniques such as sticky routing to tie user requests to respective server nodes. This can potentially lead to an uneven distribution of processing if particular areas of user requests result in a disproportionally greater amount of activity.
Lastly, the Web tier nodes responsible for caching do not deliver high availability to the overarching application as they are not acting in concert through an availability substrate.

How Velocity Helps
As mentioned earlier, Velocity is a distributed, in-memory cache. More specifically, Velocity exposes a unified cache tier to client applications by fusing together memory across servers. The Velocity architecture consists of a ring of cache servers running the Velocity Windows service as well as client applications that utilize the Velocity client library to communicate with the unified cache view (see Figure 2).
Figure 2 Velocity Architecture
Client applications can access the Velocity cache tier to store and retrieve any serializable CLR object through simple put and get operations as shown here:
CacheFactory factory = new CacheFactory();
cache = factory.GetCache("catalog");

cache.Put("Product-43", new Product("Coffee"));

Product p = (Product)cache.Get("Product-43");
Velocity enables the example Web app architecture to leverage a new cache tier as shown in Figure 3. By leveraging Velocity in the mid-tier, the application can achieve a new level of scalability, performance, and availability. More specifically, a high degree of scalability can be realized by minimizing the contention for database resources. More flexibility is instilled in this scalability as the architecture is freed from coupling clients to specific server nodes, respective to state, and performance improvements can be enabled by bringing data closer to the logic that consumes it, thus improving response time and reducing latency. Through Velocity, high availability is achieved by insulating the cluster with redundancy, therefore mitigating data loss as well as spikes in load on the data tier in the event of a node failure.
Figure 3 Using the Velocity Cache Tier
By interweaving a caching tier as shown, the application can now explicitly leverage Velocity through various layers and patterns. For example, the business logic could consume Velocity directly in the cache-aside pattern. A read-through/write-behind layer could be introduced under the business layer, thus allowing the consuming business logic to transparently communicate with Velocity. The Web tier can transparently leverage Velocity for session state management through the Velocity ASP.NET session store provider integration.

Data Classification
To fully take advantage of the Velocity feature set in your applications, it is important to understand the types of data commonly cached. These data types can be classified as reference data, activity data, and resource data.
Reference data refers to data that is primarily read-only, such as product catalog or user data. This data is written infrequently, perhaps once per day or week. However, the scale requirements on reference data demand a large number of read requests on these smaller pieces of data.
For example, in a storefront, as increasing loads of users browse the catalog, a tremendous number of product data store retrieval and list building operations are generated. As the product catalog reference data changes fairly infrequently and is highly reusable across clients, it is conducive to caching in that considerable pressure can be offloaded from the data tier by caching this data closer to the application.
Top-selling and commonly purchased-together product data can be described as reference data as well. As real-time requirements are not a necessity for these reports, the application would benefit greatly from eliminating the cost of analyzing orders upon every request to determine the current products associated with these groupings.
Activity data refers to data that is part of a business activity or transaction. The data described as part of the shopping cart experience in the storefront represents activity data. Activity data is that it is exclusively read and written, and thus is usually easily partitioned. After the lifetime of the activity—in this case upon completion of checkout—the activity data is usually retired from the cache and persisted to the underlying store for subsequent processing.
Currently, to maintain shopping cart state, the example application needs to either employ sticky routing to tie user sessions to specific server resources or persist state to the underlying data store. By caching this activity data in Velocity, you could eliminate the need for sticky routing and alleviate the underlying database from the responsibility of storing this primarily transient shopping cart data until the activity ultimately produces an order.
Resource data refers to data that is concurrently read and written, such as product inventory. During the ordering process, inventory levels may need to be frequently monitored to enforce levels or back-fill policies. However, as orders are processed, this data needs to be concurrently updated to account for the change in levels. Frequently, when dealing with resource data, to deliver performance and scale, coherence of data is relaxed. For instance, the ordering process may oversell items wherein separate processes such as back ordering could be triggered to reconcile inventory conflicts.
Before I get started integrating Velocity into the example application, it might be helpful to quickly review some core Velocity concepts.

Logical Hierarchy
The Velocity logical hierarchy consists of machines, hosts, named caches, regions, and cache items (see Figure 4). Machines can run multiple Velocity services, each service considered a cache host. Each cache host can run multiple named caches. These named caches can span across machines and are defined in configuration.
Figure 4 Logical Hierarchy of Velocity Elements
Each named cache stores a logical group of data, such as a product catalog. Named caches also set policies around availability, eviction, and expiration. Explicitly created regions—physically co-located data containers—may also exist within each named cache. As items are retrieved in bulk, regions become useful if the application consistently needs to address a group of cache items together. Additionally, these regions enhance usability by providing tag-based searching and enumeration of region cache items. Explicitly created regions are, however, an optional concept. Velocity will implicitly create default regions behind the scenes if no region is explicitly specified.
Lastly, within the implicitly and explicitly created regions are the cache items themselves, which are responsible for maintaining keys, objects, tags, timestamps, versions, and expiry data.

Partitioned and Local Cache
Velocity currently supports two caching types: partitioned and local. Partitioned caches distribute cache items across the hosts for particular named caches. Each overarching named cache can enable high availability, effectively causing Velocity to maintain a configurable number of backups for items within a named cache across machines in the Velocity tier.
Local caching runs on the client to speed up access to data retrieved from Velocity. When local caching is used in concert with the Velocity partitioned caches, as data is retrieved from the Velocity server ring, cache items are stored in object form within the process running the Velocity client. Further, as with server-side expiration policy, local caches can also each configure an expiration time for items.

Configuration Store
The Velocity configuration store consists of cache policies as well as the global partition map, which details the distribution of cache items across the cluster. Velocity uses the policies and global partition map defined within the configuration store to enforce the policies defined by named caches and to run the global partition manager, respectively.
The global partition manager is a component of Velocity that is responsible for conveying the landscape of cache items to the rest of the cluster. Clients ultimately utilize this information when communicating with the cluster to create routing tables that define correlations between data and Velocity service hosts. The global partition manager runs within one Velocity service host. If the host running the global partition manager goes down, another host within the Velocity cluster will assume responsibility by starting the global partition manager from the global partition map stored in the configuration store. Velocity enables the configuration store to be placed on a network share or stored within SQL Server.

Getting Started with Velocity
To get started with Velocity, let's begin with an overview of the Velocity installation process, the initial cluster deployment, and a basic Velocity client application.
Upon proceeding through the Velocity installer on a cache server, you are presented with the cache host configuration screen shown in Figure 5.
Figure 5 Velocity Cache Host Confi guration
The cache host configuration screen captures the cluster and service settings required to run your Velocity host. First, this screen allows you to choose whether the configuration store is placed on a network share or stored within SQL Server. For this sample, I'll use the XML Based File option to store the Velocity configuration on a network share.
Next, the cluster is assigned a name as well as an approximate node size. The cluster size configuration is used to optimize performance within Velocity by tuning internal data structures according to your cluster size. It is possible to reset the cluster size at a later time as your cluster evolves.
Finally, the service configuration values section captures the service port that the current host runs on, the port the cluster itself runs on, as well as the maximum amount of memory this Velocity host is to consume.
Click the Test Connection button to validate the connectivity of the configuration store. Click Save & Close to finish the installation. If the machine is running a firewall, make sure to allow the Velocity service through by adding an exception for DistributedCache.exe, located in the Velocity installation directory.
To install additional cache hosts, repeat these installation steps on new Velocity machines. When installing additional hosts, the cluster related settings will be detected automatically from the configuration store on the network share used during the installation of the first cluster host. For this example, I installed a three-node cluster.

Cluster Deployment
Once Velocity is installed, it can be administered through Windows PowerShell. To start the Velocity command window, run the Velocity administration shortcut placed on the desktop after installation as administrator. To start the cache cluster, run the Start-CacheCluster command. The Get-CacheHost command will display the status of hosts across the cluster. Get-CacheStatistics will display cache item statistics for a particular named cache. For more details on Velocity administration through Windows PowerShell, please see the documentation included with the Velocity installation.
After joining cache hosts to the Velocity cluster, the XML policy configuration shown in Figure 6 is produced on the network share in the ClusterConfig.xml file.
Figure 6 Velocity XML Policy Configuration
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="dcache" type="System.Data.Caching.DCacheSection,
      CacheBaseLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />
  </configSections>
  <dcache cluster="velsample" size="Small">
    <caches>
      <cache type="partitioned" consistency="strong" name="default">
        <policy>
          <eviction type="lru" />
          <expiration defaultTTL="10" isExpirable="true" />
        </policy>
      </cache>
    </caches>
    <hosts>
      <host clusterPort="22234" hostId="1916351620" size="2048" quorumHost="true"
          name="VELSAMPLE01" cacheHostName="DistributedCacheService"
          cachePort="22233" />
      <host clusterPort="22234" hostId="1249604550" size="2048" quorumHost="false"
          name="VELSAMPLE02" cacheHostName="DistributedCacheService"
          cachePort="22233" />
      <host clusterPort="22234" hostId="1016778186" size="2048" quorumHost="false"
          name="VELSAMPLE03" cacheHostName="DistributedCacheService"
          cachePort="22233" />
    </hosts>
    <advancedProperties>
      <partitionStoreConnectionSettings providerName="System.Data.SqlServerCe.3.5"
          connectionString="\\velsample01\velocity\ConfigStore.sdf" />
    </advancedProperties>
  </dcache>
</configuration>
The dcache configuration element represents the velsample cluster created during installation. This configuration section holds the policy configuration information that I set during the host installations, namely named caches, hosts, and configuration store settings.
In the caches element, Velocity automatically created a default, partitioned, named cache. This named cache configuration also holds the policy settings Velocity uses to enforce expiration and eviction settings that control how long cache items within this named cache should remain in memory as well as how Velocity removes these items once they expire. In this default configuration, the defaultTTL value indicates that cache items will remain in memory for 10 minutes. Subsequently, Velocity will use the Least Recently Used (LRU) algorithm to evict objects from memory as they expire.
In certain circumstances, it may be desirable to turn these expiration and eviction settings off. For example, in cases where the data set size is known to be relatively fixed, such as the product catalog in the example application. In this case, the eviction type would be set to none while the isExpirable setting would be set to false.
Next, in the hosts configuration element, each of the Velocity hosts specifies naming, size, and address information. Additionally, each host can act as a quorum host, or lead host. Lead hosts in Velocity are responsible for maintaining the stability of the cluster. These lead hosts collaborate to prevent split brain syndrome, a circumstance where a loss of communication within the cluster causes independently operating sub-clusters. A majority of Velocity lead hosts must be running for the entire cluster to run, so it is critical to insulate your cluster with enough lead hosts to ensure a majority of these lead hosts do not fail simultaneously.
There is some additional overhead involved in lead host communication. It is a good practice to run your cluster with as few lead hosts as necessary to maintain a quorum of leads. For small clusters, ranging from one to three nodes in size, it is acceptable to run all nodes as lead nodes as the amount of additional overhead generated by a small grouping of lead hosts will be relatively low. For larger clusters, however, to minimize overhead involved in ensuring cluster stability, it is recommended to use a smaller percentage of lead hosts—for example, 5 lead hosts for a 10-node cluster.
For my example app, to enable all three hosts to run as lead hosts, I can run a Stop-CacheCluster command, update the remaining quorumHost values to true, and then run Start-CacheCluster.
Finally, the advancedProperties configuration element specifies the storage provider for and location of the partition map. Here, the configuration specifies the network location of the partition map database. This partition map database is used by the host currently running the global partition manager.

A Basic Velocity Client Application
To set up your development environment for creating Velocity applications, first create a new console project. Add references to the following assemblies in your Velocity installation directory:
  • CacheBaseLibrary.dll
  • CASBase.dll
  • CASMain.dll
  • ClientLibrary.dll
Next, add an app.config file to your console project with the following configuration:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="dcacheClient"
      type="System.Data.Caching.DCacheClientSection, CacheBaseLibrary"
      allowLocation="true" allowDefinition="Everywhere"/>
  </configSections>

  <dcacheClient deployment="routing">
    <localCache isEnabled="false"/>
    <hosts>
      <host name="VELSAMPLE01" cachePort="22233"
        cacheHostName="DistributedCacheService"/>
    </hosts>
  </dcacheClient>
</configuration>
The dcacheClient section provides the Velocity client configuration. These settings specify the client deployment type, local cache options, as well as cluster address information.
The deployment can be set to either routing or simple. A routing deployment allows Velocity clients to communicate with any Velocity host in the cluster. A simple deployment directs clients to communicate with specific Velocity hosts, requiring an additional network hop within the cluster to communicate with the target host. Routing is the recommended deployment as it minimizes network overhead. However, if there are networking constraints allowing clients to only communicate with a subset of the Velocity hosts in your cluster, a simple deployment would be suitable.
The localCache element in this example specifies that local caching is disabled for the client application. If isEnabled was set to true here, as this console client application retrieved cache items from the Velocity cluster, these items would be preserved on the client-side in object form within the application process. As subsequent trips to the cluster would no longer be required, local caching could further enhance performance.
With local caching enabled, a ttlValue can specify the number of seconds objects should remain valid within the local cache. Subsequent requests for invalidated objects will cause the client to retrieve updates from the cluster.
It is important to note that clients utilizing local caching will retrieve objects locally even if these items have been updated on the cluster. As such, it is best to use local caching for infrequently changing data. In subsequent releases, the Velocity team is targeting a notifications feature that will allow the Velocity cluster to emit notifications of updated cache items to clients, thus allowing clients to refresh the local cache accordingly.
The hosts configuration settings specify addresses for the various hosts in your cluster. Multiple hosts may be specified here; though, only one host is required for the client to discover the other hosts in the cluster. Each host in the client configuration specifies name, port, and cacheHostName settings which correspond to the host settings shown previously in the cluster configuration.
Figure 7 shows a simple program that puts an object into the Velocity unified cache and then retrieves that object from the cache.
Figure 7 Using the Velocity Cache
using System;
using System.Data.Caching;

namespace VelocityClientConsole {
  class Program {
    static void Main(string[] args) {
      // Create the cache factory
      CacheFactory factory = new CacheFactory();

      // Get the default named cache
      Cache cache = factory.GetCache("default");

      // Add a string to the cache
      cache.Put("MyCacheKey", "Hello World");

      // Get the data from the cache
      string s = (string)cache.Get("MyCacheKey");

      Console.WriteLine(s);

      Console.ReadLine();
    }
  }
}
First, in the using statement, I reference the types within the System.Data.Caching namespace. In the Main method, I create a CacheFactory and retrieve the named cache entitled default. Then I use the Cache client instance to put the string object into the default cache and retrieve it.
The CacheFactory also provides additional constructors to create Cache objects that communicate with Velocity clusters not referenced in the client configuration. The CacheFactory will create the appropriate concrete Cache client implementation respective of the deployment and local cache settings that are specified either through configuration or programmatically via the CacheFactory constructor.
After running the console application through Windows PowerShell, I can verify that the cache item has been created through the Get-Cache and Get-CacheStatistics commands (see Figure 8). Get-Cache shows that an implicit region has been created to hold the item on the VELSAMPLE03 host. Additionally, Get-CacheStatistics shows size, region, and access information for the default named cache.
Figure 8 Viewing Cache Statistics in Windows PowerShell

Designing the Cache
Now that Velocity is up and running, let's begin integrating Velocity into the storefront application by designing the cache storage. In this example, the storefront application utilizes the Northwind database for product catalog and report reference data as well as sales order activity data. This data model is illustrated in Figure 9.
Figure 9 Storefront Data Model
The application utilizes transfer object classes to represent the underlying data store. The current storefront data access layer utilizes System.Data.SqlClient to populate transfer objects. If your application leverages object-relational mapping (O/RM) technologies for more powerful type mappings, objects produced by these frameworks could be cached in Velocity just the same. As long as your data access layer produces serializable CLR objects, Velocity will be capable of caching these objects.

Named Caches
As the properties that govern these types of data vary among reference and activity classifications, an organization strategy that demarcates these logical units into distinct named caches would benefit from caching policies suitable to the needs of each data type. In particular, by creating separate named caches for the catalog, report, and shopping cart data, you can tune the policies around expiration, eviction, and availability accordingly.
Since the product catalog can be classified as reference data, or shared, read-only data, it is very conducive to caching. This data, moreover, represents a logical unit of reference data, so it is a good candidate for defining a separate named cache.
To create this new named cache, use the New-Cache command.
New-Cache -CacheName catalog -Eviction none -NotExpirable
For the catalog cache, as the data changes fairly infrequently and the catalog size is predictable, you can turn off eviction as well as disable expiration. However, in scenarios where the cache size may grow unpredictably, it is necessary to utilize eviction and expiration to prevent out of memory conditions.
As the product reference reports, such as top-selling products, may fluctuate more frequently than the catalog itself, aggregating identifier listings for reports into a separate named cache has benefit in that these product listings can be decoupled from the underlying catalog named cache policies. Specifically, for these product reference related reports, you could create a separate named cache called reports with a more aggressive expiration and eviction strategy.
New-Cache -CacheName reports -Eviction LRU -TTL 1440
The reports named cache is created such product identifier report listings will expire after one day (1,440 minutes), and Velocity will use LRU algorithm to evict objects from the cache. Using a separate named cache for aggregated product identifier listings will allow the report related data and the product catalog data to vary independently according to different expiration and eviction policies. As such, top-selling products can be retrieved from the catalog cache using product identifier listings from the reports named cache. The top-selling report listing object will expire daily, whereas the product catalog remains cached, perhaps until a background process updates at a regular or triggered interval.
As session-related data is traditionally read and written, it can be classified as activity data. Since shopping cart data is critical to the completion of orders, there is also a strong requirement for high availability across this data. In the original application, to mitigate the loss of shopping cart data, this state would have been persisted through to the underlying data store until the completion of respective orders. In a Velocity-enabled storefront, as users progress through the shopping cart experience, activities based on cart data can be exclusively read from and written to the Velocity cluster.
Through a newly created session named cache, you can tune policies around the properties of activity-related data. In particular, for applications that previously cached this data in default InProc ASP.NET session state, more flexible scaling is enabled through Velocity SessionStoreProvider integration. Sticky routing is no longer need to couple user requests to specific servers according to state. Further, by enabling high availability on the session named cache, data loss is mitigated without persisting data to the database. Velocity insulates the cluster by redundantly storing cart data.
This New-Cache command below creates the session named cache.
New-Cache -CacheName session -Secondaries 1 -TTL 1440
The –Secondaries switch specifies the number of backups for each item stored within this named cache. In this example, Velocity will insulate 1 additional backup copy across the cluster for each shopping cart stored in the session named cache.

Organizing the Cache
The application needs to be able to address categories and products both independently as well as in category and product lists. There are several approaches you could take to organize these objects in the catalog named cache, including region-based organization and list-based organization. Both approaches focus on a degree of object distribution across the cluster. However, depending on the needs of your application, another organization approach might be more appropriate. For example, you could cache object graphs by key or you could employ page caching.
The product catalog could be organized using Velocity Regions. Regions are advantageous in the example scenario because they allow the application to address particular products in a category all together or independently. For instance, the application may need to present an entire product category listing in some cases, while it may need to display only a product detail page in other cases. Organizing the catalog cache using regions for product category associations would yield the hierarchy shown in Figure 10.
Figure 10 Region-based Organization
In this cache storage model, the category objects themselves would be stored independently whereas product objects would be stored in regions, relative to category membership. Another potential benefit of a region-based approach for the catalog involves co-location: as all objects in a region are guaranteed to reside on the same Velocity host machine, the entire product category region will be retrieved in bulk. A list-based approach requires multiple cache requests. Regions also enable the retrieval of items according to tags, which are secondary indexes for cache items.
Although a region-based approach presents benefit in bulk retrieval optimization, individual access for products, and tag-based retrieval, the co-location of items could also present a bottleneck. If the client application consumes a particular category region more actively than others, this region hot spot would prevent the inherent benefits in throughput provided by scaling out the data through the distribution of products across the cluster.
Using a list-based approach for product category organization would promote even distribution of items across the cluster, thus fully realizing the scalability benefits possible through the distribution of data. Where a region-based approach could potentially develop a hot spot of activity in a physically co-located grouping of products, a list-based approach allows you to distribute the load of any particular product category listing across the cluster. This organizational approach requires the storage of category objects, product objects, and category list objects that identify the independently stored product objects. The cache organization for this list-based approach is shown in Figure 11.
Figure 11 List-based Organization
Although a list-based approach to organizing the catalog distributes the load of catalog activity evenly across the cluster, to address a product category all together, multiple cache requests are necessary. On this topic, bulk read/write is a feature that the Velocity team is targeting for future releases.

Organizing the Reports
You can employ a similar list-based organization for the reports named cache. Decoupling these reference reports from the catalog named cache allows the policies that govern report and catalog data to vary independently. Effectively, this allows you to tune the expiration and eviction policies according to the respective data types. The fairly static catalog reference data disables expiration and eviction, while the more dynamic reports data enforces a more aggressive expiration and eviction policy. Organizationally, product key list objects within the reports named cache will identify product objects within the catalog cache, similar to the method used for product category listings, yet employed across named caches.
To retrieve products based on these aggregated reference reports, the list-based method is used across the reports and catalog named caches.

Organizing the Session
For activity-based scenarios that are exclusively read/write, approaching cache organization through the storage of object hierarchies may be an optimal choice. For example, in the session named cache, you can organize shopping cart objects that contain information about each product a particular user has selected: product, price, and quantity.
Whereas the catalog organization promoted a high degree of scalability by distributing items across the cache, the shopping cart cache items are not as highly reusable across requests. Performance can be increased by caching each user's shopping cart under a single key rather than distributing shopping cart items across the named cache as top-level cache items. More specifically, as the storefront application generates requests to the session named cache, each shopping cart can be retrieved in bulk through a single request. Additionally, since the session named cache was created using high availability, Velocity will protect the session data by maintaining copies of each shopping cart across the cluster. The session organizational approach is illustrated in Figure 12.
Figure 12 Session Cache Organization
Applications may maintain activity-based items directly through Velocity APIs, and ASP.NET applications may leverage Velocity's integration with the session store provider framework to transparently realize Velocity as a cache tier through the ASP.NET session API.

Programming a Cache Layer
The current architecture tiers are demarcated as the user interface (Web), the business logic layer (BLL), and the data access layer (DAL). As the current application deals primarily with data transfer objects produced by the data tier, caching data in these respective shapes facilitates delivering the application tiers with representations they understand. The storefront example introduced a cache layer in between the business and data tiers. By interweaving caching logic under the business tier, the layers that depend on the business logic uniformly benefit from validation of business rules before generating calls to the caching logic, while the business logic itself transparently consumes the cache tier. Additionally, the Web tier utilizes the Velocity ASP.NET session store provider integration.
To see this layering in action, let's walk through the cache layer starting by using the product category listing view. To begin, the category listing view calls into the business logic layer that validates the request and then consumes the cache layer by calling the GetProductsByCategory method on VelocityCatalogCache (see Figure 13). This coordinates marshaling data among the Velocity cache and the data store in a manner transparent to the business logic.
Figure 13 GetProductsCatalog
public IList<Product> GetProductsByCategory(int id) {
  IList<Product> products = null;

  IList<int> productIDs = (IList<int>)_catalogCache.Get("ProductCategoryList-" + id);

  if (productIDs != null) {
    products = new List<Product>(productIDs.Count);

    foreach (int productID in productIDs) {
      products.Add((Product)_catalogCache.Get("Product-" + productID));
    }
  }
  else {
    products = DataLayer.CatalogProvider.GetProductsByCategory(id);

    if (products != null && products.Count > 0) {
      productIDs = new List<int>(products.Count);
      foreach (Product p in products) {
        productIDs.Add(p.ProductID);
        _catalogCache.Put("Product-" + p.ProductID, p);
      }

      _catalogCache.Put("ProductCategoryList-" + id, productIDs);
    }
  }

  return products;
}
In this product category listing, the cache layer will act as a read-through provider to simplify the business layer. This is only one layering approach to programming caching logic using Velocity. For example, for fine-grain control at the business level, leveraging Velocity in a cache-aside pattern directly in the business layer would present a viable approach as well. In future releases, the Velocity team is targeting read-through/write-behind support wherein callbacks may be registered to coordinate marshaling data among the cache and underlying store, thus allowing clients to utilize this behavior through the Velocity cache APIs rather than a wrapper layer.
In the GetProductsByCategory cache layer method implementation, it is important to touch on the implication to concurrency. In this method, no concurrency model is currently being utilized. As the catalog is primarily read-only reference data that changes infrequently, the impact to concurrency will not be too severe. However, it would be possible for multiple threads or clients to perform identical updates, one after another, if they arrive at cache misses simultaneously. Alternatively, for scenarios where concurrency is critical—for example when caching various types of resource data—Velocity supports both optimistic and pessimistic concurrency models.

Integrating with ASP.NET Session State
Velocity provides the ability to transparently leverage the cache cluster through the ASP.NET SessionStoreProvider framework. This capability allows Web applications to access items in a specified name cache through the HttpContext.Session property.
This integration presents great benefit in scalability by decoupling cache state from specific servers. By leveraging Velocity for session, or activity-related data, more flexible scaling is enabled as the architecture is freed from coupling clients to specific server nodes, respective to state.
For scenarios that require high availability, such as the shopping activity data, Velocity enables you to easily insulate the cluster with redundancy as shown during the creation of the session named cache, thus delivering availability closer to the consuming logic.
To enable Velocity SessionStoreProvider integration in ASP.NET, the following configuration is added to the web.config file in the storefront ASP.NET application. This configuration associates the storefront ASP.NET session state with the previously created session named cache.
<system.web>
  <sessionState mode="Custom" customProvider="SessionStoreProvider">
    <providers>
      <add name="SessionStoreProvider"
        type="System.Data.Caching.SessionStoreProvider, ClientLibrary"
        cacheName="session" />
    </providers>
  </sessionState>
</system.web>
Then, to access the session named cache through ASP.NET, the storefront maintains user shopping cart activity data as in the Product.aspx page shown in Figure 14.
Figure 14 Accessing Cache Through ASP.NET
protected void AddToCartButton_Click(object sender, EventArgs e) {
  ShoppingCart cart = (ShoppingCart)Session["ShoppingCart"];

  if (cart == null) {
    cart = new ShoppingCart();
    Session["ShoppingCart"] = cart;
  }

  int productID = Convert.ToInt32(Request.Params["id"]);

  ShoppingCartItem item = cart[productID];
  if (item != null) {
    item.Quantity++;
  }
  else {
    Product p = ServiceLayer.CatalogService.GetProduct(productID);

    cart[productID] = new ShoppingCartItem() {
      ProductID = productID,
      Quantity = 1,
      UnitPrice = p.UnitPrice
    };
  }

  Response.Redirect("~/Cart.aspx");
}
Upon order completion, activity-oriented shopping carts can be removed from the session named cache and propagated to the underlying data store as shown here from the Purchase.aspx page.
protected void CheckoutButton_Click(object sender, EventArgs e) {
  ShoppingCart cart = (ShoppingCart)Session["ShoppingCart"];

  ServiceLayer.OrderService.ProcessOrder(cart,
    NameTextBox.Text, AddressTextBox.Text, CityTextBox.Text,
    RegionTextBox.Text, PostalCodeTextBox.Text, CountryTextBox.Text);

  Session["ShoppingCart"] = null;

  Response.Redirect("~/Congrats.aspx");
}
Through the ASP.NET session API, the application now seamlessly leverages Velocity to realize activity-oriented data closer to the consuming logic, eliminate the need for techniques such as sticky routing, and protect shopping cart activity data through the high-availability support previously enabled across the session named cache.

Going Forward
Velocity harnesses the evolution in distributed architectures as well as the underlying hardware trends to realize this pervasiveness of data closer to consuming applications. This article has only provided a first look at the features of and rationales driving Velocity.
The example storefront applications now leverages Velocity to cache reference catalog and report data as well as shopping cart activity data using policies suitable to each data type. Whereas scale was formerly inhibited by duplication of catalog data operations upon every page load, now by both minimizing load on the underlying data store and realizing our catalog reference data closer to the logic that consumes it, a high degree of scalability and performance is enabled. Response time is also improved, and you get more flexible scaling by freeing the underlying architecture from coupling resources to requests based on state. Without going to the underlying data store as needed before, data critical to the completion of orders is protected through high-availability support in Velocity.
If you would like additional insight into the rationale for distributed, in-memory caching, an overview of the entire Velocity feature set, and information about downloads, see the whitepaper "Microsoft Project Code Named Velocity" and the Velocity Developer Center.

Aaron Dunnington is a Program Manager on the Data Programmability team at Microsoft. He can be reached at http://blogs.msdn.com/velocity.


Peer Fun
A Peer-To-Peer Work Processing App With WCF
Matt Neely
Code download available from the MSDN Code Gallery
Browse the Code Online

This article discusses:
  • Basic peer-to-peer communication
  • Creating the Worker peer
  • Load balancing and work sharing
  • Backing up and restoring work items
This article uses the following technologies:
WCF
Contents
Basic Peer Communication
The Worker Peer
Making the Worker Work
The User Peer
Spectators
Load Balancing and Work Sharing
WorkItem Backup
Peer Fun
The group I'm a part of at Microsoft lets us take a week off from our regular duties to research a topic. I recently elected to spend my week researching peer-to-peer (P2P) networking. That research ballooned into a permanent interest in this new networking paradigm.
One of the reasons P2P piqued my interest was due to an internal documentation build system my team owns. We have authors writing thousands of help documents that need to be parsed, transformed, patched, and compiled. The documents can be compiled into various output formats and in various languages every night. To accomplish this feat, we have a server farm running our build software.
The mechanism we currently use is your typical client-server architecture. Build jobs are queued on a central server and doled out as build servers become free to process them. We occasionally encounter a backlog of jobs on the central server or a misbehaving build server. Many team members have a vested interest in the status of the build system, but the system is opaque and difficult to monitor.
This article is the result of envisioning a P2P version of this build system. I will discuss the emerging design of a generic work processing and peer-enabled application based on the Windows Communication Foundation (WCF) PeerChannel. I'll lead you through the construction of this system while discussing some of the design decisions that must be made in the process.

Basic Peer Communication
For this project, I'll start by defining my main work-processing peers. I'll give them the generic name of Workers. The plan will be to have one Worker per physical machine (although there are no technical limitations requiring this).
Each Worker will need to communicate and coordinate with others to accomplish work. In order to do this, it needs to have some knowledge about other Workers. So how will my Worker get this information?
One approach is to have each Worker poll every other Worker for its state. This mechanism is complex as it requires two phases: Worker discovery and Worker querying. In addition, if each Worker were to ping every other Worker, you'd have many unneeded messages traveling your network. Coupled with the fact that the WCF PeerChannel is inherently multicast, pinging a single Worker will really send a message to every Worker. For n Worker peers, I'd have n3 messages flying around. I declined this approach for a broadcast model.
In a broadcast model, every Worker will periodically send out its status. I call this a heartbeat message. Other Workers can decide what to do with this information, even if they just ignore it. I prefer this approach because as the Worker mesh topology changes, you can infer nodes coming online and offline by the presence or absence of heartbeat messages. Worker peer discovery is passive instead of active. This model also minimizes the network traffic needed just for system upkeep. With n Worker peers I'd have n2 messages in this model, which is an improvement over the first approach.
To create a service in WCF, you usually start with defining your service contract. This is the interface that defines the functionality a service exposes to clients. In a P2P scenario, each peer acts as both server and client, and the service contract created here will define how peers communicate with each other. Given the decision to broadcast Worker status, I can define my service contract:
[ServiceContract]
public interface IWorkerPeer {
  [OperationContract(IsOneWay=true)]
  void Heartbeat(WorkerStatus status);
}
My IWorkerPeer service contract has a single, one-way operation called Heartbeat. When a Worker calls this method, it will send out the pre-populated WorkerStatus object instance to every other Worker peer participating in my mesh.
Here's the simple definition of the WorkerStatus class:
[DataContract]
public class WorkerStatus {
  [DataMember]
  public Guid WorkerId;

  [DataMember]
  public WorkerState State;
}
It will contain a unique identifier for the Worker and a simple state enumeration value. In a more functional system, this could be an elaborate set of data incorporating extra descriptors, performance counters, Worker statistics (such as percent of failed work items), and more. As defined, though, this will minimally allow each Worker to know the status of every other Worker in the mesh.
The states I've defined are Unknown, Ready, Busy, and Offline. A state of Unknown means that I'm not sure of the Worker's state (perhaps due to the absence of a heartbeat from that Worker within a specified period of time). A Ready state means the Worker is functioning correctly and is ready to take on some work. A Busy state means the Worker is functioning correctly, but busy doing work. An Offline state means that the Worker is not available to do work. This could potentially be the state sent out from a Worker's final heartbeat or perhaps a Worker that is taken offline by an administrator for maintenance or debugging.

The Worker Peer
To make a functional Worker peer application, I need to expose and consume my service. So I created a simple console application and gave it a class named Worker as seen in Figure 1. In my Worker class, I added Start and Stop methods to handle the setup and teardown of the exposed service (which receives messages) as well as the client channel (for sending messages). I also use a System.Threading.Timer instance to periodically call the method SendHeartbeat. SendHeartbeat instantiates a WorkerStatus object, populates it with the appropriate data, and then sends it to all peers. As a service, when the IWorker.Heartbeat method gets called, it will simply output the message content to standard output.
Figure 1 The Worker Class
namespace WorkerPeerApplication {
  [ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]

  public class Worker : IWorker {
    private ServiceHost _host;
    private ChannelFactory<IWorker> _factory;
    private IWorker _workerClientChannel;

    private Guid _id;
    private volatile WorkerState _state;

    private Timer _heartbeat;

    public Worker() {
      _id = Guid.NewGuid();
      _state = WorkerState.Offline;
    }

    public void Start()  {
      _host = new ServiceHost(this);
      _host.Open();

      _factory = new ChannelFactory<IWorker>("IWorkerClient");
      _workerClientChannel = _factory.CreateChannel();

      _state = WorkerState.Ready;
      _heartbeat = new Timer(
        this.SendHeartbeat,
        null,
        new TimeSpan(0),
        //Time should really be read in from config.
        new TimeSpan(0, 0, 1));
    }

    public void Stop() {
      _state = WorkerState.Offline;
      _heartbeat.Dispose();
      this.SendHeartbeat(null);
      _factory.Close();
      _host.Close();
    }

    private void SendHeartbeat(object context) {
      WorkerStatus ws = new WorkerStatus();
      ws.WorkerId = _id;
      ws.State = _state;

      _workerClientChannel.Heartbeat(ws);
    }

    void IWorker.Heartbeat(WorkerStatus status) {
      Console.WriteLine(
        "{0}: {1}",
        status.WorkerId,
        status.State);
    }
  }
}
For completion, I need a Main method and the appropriate WCF configuration, which I put in the application configuration file. The Main method can be seen in the code download, and the configuration I use is shown in Figure 2.
Figure 2 Worker Configuration
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>

    <services>
      <service name="WorkerPeerApplication.Worker">
        <host>
          <baseAddresses>
            <add baseAddress = "net.p2p://MyPeerApplication" />
          </baseAddresses>
        </host>

        <endpoint
          address =""
          binding="netPeerTcpBinding"
          bindingConfiguration="unsecureBinding"
          contract="PeerWcfComm.IWorker"/>

      </service>
    </services>

    <bindings>
      <netPeerTcpBinding>
        <binding name="unsecureBinding">
          <security mode="None"/>
          <resolver mode="Pnrp"/>
        </binding>
      </netPeerTcpBinding>
    </bindings>

    <client>
      <endpoint
        name="IWorkerClient"
        address="net.p2p://MyPeerApplication"
        binding="netPeerTcpBinding"
        bindingConfiguration="unsecureBinding"
        contract="PeerWcfComm.IWorker" />
    </client>

  </system.serviceModel>
</configuration>
You can see that I define both the service the peer exposes and the peer's client-side configuration. The service configuration specifies the type of the service as well as its base address and endpoint. I'm using an unsecure binding and the Peer Naming Resolution Protocol (PNRP) for resolving peer names. The client configuration is almost a mirror of the service. The only noticeable difference is that I gave the client endpoint a name so that the ChannelFactory constructor can easily access the desired configuration.
Now that I have the shell of a Worker peer, let's test drive it. For an experiment, run two instances of this program at the same time. You'll see that at first each peer will output its own heartbeat information. After a few seconds, peers will begin to find each other and each Worker will receive status from the others.
Frankly, I consider it annoying and distracting for my own messages to be sent back to me. Thankfully, the PeerChannel team blog provides the code I need. You'll see a very useful class called RemoteOnlyPropagationFilter. Borrowing this, I can then add the following two lines of code to my Worker.Start method to ensure that the messages I send out don't also end up being sent to me:
PeerNode pn = ((IClientChannel)_clientChannel).GetProperty<PeerNode>();
pn.MessagePropagationFilter = new RemoteOnlyMessagePropagationFilter();
These two lines of code were added before I start the heartbeat timer.

Making the Worker Work
So far, I have an application that merely communicates among Worker peers. Workers don't actually do any work yet. I'll remedy this by augmenting the IWorker interface, adding the following method:
[OperationContract(IsOneWay = true)]
void DoWork(Guid workerId, WorkItem work);
This method allows an external entity to tell a Worker peer to do an item of work. Since the WCF messaging is multicast, I added a parameter to specify the ID of the Worker that is to perform the work. I'll discuss the WorkItem class in a moment. But for now, the interface change requires me to update the Worker class as seen in Figure 3. When someone tells a Worker to do some work, this request is added to an internal queue and eventually executed in the order received. To actually perform the work, I added a method called MonitorWork to the Worker class. The Start method will simply call ThreadPool.QueueUserWorkItem, pointing it to the MonitorWork method.
Figure 3 Revised Worker
public class Worker : IWorker {
  // Omitted for clarity ...
  private Queue<WorkItem> _workQueue;

  public Worker() {
    // Omitted for clarity ...
    _workQueue = new Queue<WorkItem>();
  }

  public void Start() {
    // Omitted for clarity ...
    ThreadPool.QueueUserWorkItem(MonitorWork);
  }

  private void MonitorWork(object context) {
    if (_workQueue.Count > 0) {
      lock (_workQueue)
        _currentWorkItem = _workQueue.Dequeue();

      DoWorkInternal(_currentWorkItem);
      _currentWorkItem = null;
    }

    if (_state != WorkerState.Offline)
      ThreadPool.QueueUserWorkItem(MonitorWork);
  }

  private void DoWorkInternal(WorkItem wrk) { ...}

  void IWorker.DoWork(Guid workerId, WorkItem work) {
    if (workerId == _id)
      lock (_workQueue)
        _workQueue.Enqueue(work);
  }
}
At a high level, MonitorWork will do something and then reschedule itself via another call to ThreadPool.QueueUserWorkItem. It will do this until the state of the Worker is Offline, which happens when Stop is called on the Worker object. The something it does is to check whether the work queue has anything in it. If the queue contains work items, MonitorWork will dequeue the next WorkItem and begin processing it. The behavior I want is for my Worker to serially process its work, although it could easily be modified to process items in parallel.
As you see from the code in Figure 3, I have not addressed what work is yet. I've merely passed this to the yet-to-be-defined DoWorkInternal method. So let's first look at the WorkItem class definition. As the system that I'm building for this article is task-agnostic, I'm going to simply define my work item as follows:
[DataContract]
public class WorkItem {
  [DataMember]
  public Guid WorkItemId { get; set; }

  [DataMember]
  public int PretendToWorkSeconds { get; set; }
}
I must again use a DataContract or MessageContract attribute so that WCF can correctly serialize my custom class. The WorkItem class has an identifier field, WorkItemId, and a field specifying the number of seconds to feign doing actual work (named PretendToWorkSeconds).
Now that a WorkItem is defined, I can process it. Here is my DoWorkInternal method implementation:
private void DoWorkInternal(WorkItem wrk) {
  this._state = WorkerState.Busy;
  Thread.Sleep(new TimeSpan(0, 0, wrk.PretendToWorkSeconds));
  this._state = WorkerState.Ready;
}
This is where I would normally do my meaningful work (for example, building the SQL Server 2008 setup documentation in Brazilian Portuguese). But for the sake of this article, I'm just going to sleep the specified time period.
Finally, I update my WorkStatus class with two new data members that will be set before a Worker's heartbeat is sent. They are the identifier of the work item currently being executed and a count of the pending work items on that Worker peer:
[DataContract]
public class WorkerStatus {
  // Omitted for clarity...

  [DataMember]
  public Guid CurrentWorkItem;

  [DataMember]
  public int PendingWorkItemCount;
}
Now that my Worker peers have some data, I have to be acutely aware of scenarios that bring about data loss. One of the tradeoffs of a decentralized networking paradigm like P2P is that there is no central repository for data. Each peer is responsible for collectively preserving the system's data. So what happens if a Worker peer goes down? Specifically for my system, what happens to the queued WorkItems?
I group peer shutdown scenarios into two categories: expected and unexpected. For shutdowns of an expected nature, I need to deal with the queued WorkItems. The paradigm for handling these items can vary depending on the nature of the application and data. If the data isn't important, I might be able to simply discard all the queued WorkItems. If the application is designed to be paused (or to restart automatically like a Windows service), perhaps serializing the items to disk and reading them upon startup is the correct solution. Yet another option is to simply forward the surplus WorkItems to other Workers prior to shutdown. I'll discuss unexpected shutdown later in this article in the context of data backup.

The User Peer
Thus far, I have a peer-enabled application that can accept and process defined work items. But who creates these work items and sends them to the Worker peers? One of the powerful features of a P2P system is that anyone can hook into the same communication infrastructure. All that is needed to participate (for unsecure meshes) is knowledge of the mesh address and the service contract. I'll demonstrate this by adding a new peer type called a User peer.
The class for the User peer is similar to the Worker peer discussed earlier. The User peer will also implement the IWorker interface, but doesn't need to provide an implementation of the DoWork method. (Remember to think of ServiceContract interfaces as a communication contract and not as an identity declaration. A User is not a Worker, but it participates in the IWorker peer mesh and thus implements that interface.)
The User peer is interested in tracking Workers as well as creating new work items and giving them to Workers. Figure 4 shows a screenshot of the User application. This application lists out the information received from Worker status messages. However, the functionality I want to focus on is the creation of WorkItems and then assigning them to Workers.
Figure 4 The User Peer Application
A design decision needs to be made regarding how WorkItems are assigned to Workers. One method would be to rely on the multicast nature of PeerChannel and broadcast a message to all Workers saying that a new WorkItem was available. The first Worker to respond that it is ready and able to take it wins the right to have the new WorkItem.
I don't like this approach for several reasons. First, the communication protocol would be quite chatty. The User would send out a "Who Wants a Work Item" message. Amenable Workers would respond with an "I'll Take It" message. The User would then have to finish up with a "Do the Work" message to the winning Worker. Also (due to the asynchronous nature of multiple, interacting peers), careful attention must be given to all potential race conditions that would arise.
Second, I'd have to define what "ready and able" means for a Worker. If all Workers are busy, do they not respond? What does a User do if no response is received?
Instead of this work-advertisement method, I'm choosing to reuse the information already provided from Worker heartbeats. This information gives me sufficient knowledge to compute which Worker should receive a newly created WorkItem. Thus, the algorithm for choosing the winning Worker doesn't reside in the Worker's code and can be altered independently from the Worker. For simplicity, I'm going to choose the Worker to send a WorkItem based upon the WorkerState, CurrentWorkItem, and PendingWorkItemCount members of the WorkerStatus objects. Here's the code behind the Create Work button:
private void btnNewWork_Click(object sender, EventArgs e) {
  WorkItem wrkItem = new WorkItem();
  wrkItem.WorkItemId = Guid.NewGuid();
  wrkItem.PretendToWorkSeconds = _rnd.Next(15);

  string nextWorker = GetNextWorker();
  _clientChannel.DoWork(nextWorker, wrkItem);

  lock (_workers)
    _workers[nextWorker].PendingWorkItemCount++;
}
A new, uniquely identified WorkItem is generated with a randomly selected time between 0 and 15 seconds (_rnd is a class-level field of type Random). I decide the next worker to send the work to, then send the work. I also manually increment my cached version of the Worker's PendingWorkItemCount. This helps avoid situations where two or more Workers have the same stats. If they do, it is possible to generate work faster than Workers send out status heartbeats. So I'll adjust my cached Worker stats and not unfairly send lots of work items to the same Worker.
The decision for selecting the next Worker to receive work is the purpose of the GetNextWorker method shown here:
private WorkerStatus GetNextWorker() {
  List<WorkerStatus> workers = new List<WorkerStatus>();
  lock (_workers)
    workers.AddRange(_workers.Values);

  return workers
    .Where<WorkerStatus>(
      w => w.State != WorkerState.Offline
      && w.State != WorkerState.Unknown)
    .OrderBy<WorkerStatus, int>(w => w.PendingWorkItemCount)
    .FirstOrDefault<WorkerStatus>();
}
Thanks to new extension methods in the Microsoft .NET Framework 3.5 libraries, eliminating Workers in the offline and unknown states and ordering by their pending work item counts is easy. A single (though wordy) statement can do both. I also return the first item in the enumeration. Using the FirstOrDefault method ensures the call succeeds even if no workers are available.
Now let's run another experiment. Start the User peer application and then start two separate Worker apps. After a few seconds, you should see the two Workers appear in the User application list. Now click the Create Work button several times. If everything works as expected, you can see that the User peer gave work items to both Workers. It should have done this in a fairly round-robin fashion. The Workers will continue processing their work items until finished.

Spectators
A requirement of many applications is the need for logging and tracing. To help further showcase the flexibility of a P2P-based system, I'm going to create a new peer type called Logger. My Logger peer will receive events pertinent to the system. The only events that could be currently tracked are those that are exposed by the IWorker interface: Worker status information and the assignment of WorkItems to a particular Worker. This level of detail is insufficient. There are many additional events that could be logged and tracked.
Although I could tack on new functionality to the IWorker interface, the messages sent for tracking are inconsequential to both Worker and User applications. Therefore, I'm going to create a new service contract called ISpectator. (I'm using the generic term Spectator instead of Logger or Reporter to emphasize two things. First, this peer type only receives communications and shouldn't generate any. Second, I am not dictating what a peer using this service does with the information received; it might only show the information in a UI.) Here's the service contract definition:
[ServiceContract]
public interface ISpectator {
  [OperationContract(IsOneWay=true)]
  void WorkItemStateChanged(WorkItemStatus status);
}
As a demonstration, I'm declaring just a single method called WorkItemStateChanged. A full-featured system will likely have many events to notify spectators of. The WorkItemStateChanged method takes a WorkItemStatus class defined here:
[DataContract]
public class WorkItemStatus {
  [DataMember]
  public Guid WorkerId;

  [DataMember]
  public Guid WorkItemId;

  [DataMember]
  public WorkItemState NewState;
}
This class allows the sender to specify the Worker sending the message, the WorkItem this message is for, and the new state of the WorkItem. Valid states are Queued, Executing, Finished, and Errored. As I'm sure you noticed, I don't have an equivalent WorkerStateChanged event defined. That's because Worker status is sent via the heartbeat message and I expect spectators to listen for these. With WCF, this is terribly easy to do. My service class will simply implement both IWorker and ISpectator interfaces.
The Logger service class can be seen in Figure 5. There isn't much structural difference between this class and the Worker. In the IWorker.Heartbeat method, the Logger has the typical caching of Worker status data. Additionally, if this is the first status seen for a Worker or if the state has changed for a Worker, then a message is written to standard output. Likewise, the ISpectator.WorkItemStateChanged method writes a message to standard output. A production system will likely do something much more interesting, like writing to a logging database for custom reports.
Figure 5 Logger
[ServiceBehavior(InstanceContextMode=InstanceContextMode.Single)]
public class Logger : IWorker, ISpectator {
  private ServiceHost _host;
  private Dictionary<string, WorkerStatus> _workers;

  public Logger() {
    _workers = new Dictionary<string, WorkerStatus>();
  }

  public void Start() {
    _host = new ServiceHost(this);
    _host.Open();
  }

  public void Stop() {
    _host.Close();
  }

  void IWorker.Heartbeat(WorkerStatus status) {
    bool changed = false;

    lock (_workers) {
      if (_workers.ContainsKey(status.WorkerId)) {
        WorkerStatus s = _workers[status.WorkerId];

        if (s.State != status.State)
          changed = true;
      }
      else
        changed = true;

      _workers[status.WorkerId] = status;
    }

    if (changed)
      LogWorkerStatusChange(status);
  }

  void IWorker.DoWork(string workerId, WorkItem work) {
    //Ignored. This can be assumed by the
    //WorkItemState.Queued status event.
  }

  void ISpectator.WorkItemStateChanged(WorkItemStatus status) {
    Console.WriteLine(
      "Worker: {0}| WorkItem: {1}| State: {2}",
      status.WorkerId,
      status.WorkItemId,
      status.NewState);
  }

  private void LogWorkerStatusChange(WorkerStatus status) {
    Console.WriteLine(
      "Worker {0}| State: {1}",
      status.WorkerId,
      status.State);
  }
}
The last action item in implementing the Logger is to set up configuration. Figure 6 shows the configuration for this WCF service. Noticeable differences are the lack of a client section and the addition of a service endpoint. Since a spectator is logically only a consumer of information, there is no reason to act like a client. Adding an endpoint to the service allows us to use the same service object instance for IWorker and ISpectator message sinks.
Figure 6 Configuration for Logger
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>

    <services>
      <service name="LoggerPeerApplication.Logger">
        <host>
          <baseAddresses>
            <add baseAddress = "net.p2p://MyPeerApplication" />
          </baseAddresses>
        </host>

        <endpoint
          address =""
          binding="netPeerTcpBinding"
          bindingConfiguration="unsecureBinding"
          contract="PeerWcfComm.IWorker"/>

        <endpoint
          address =""
          binding="netPeerTcpBinding"
          bindingConfiguration="unsecureBinding"
          contract="PeerWcfComm.ISpectator"/>

      </service>
    </services>

    <bindings>
      <netPeerTcpBinding>
        <binding name="unsecureBinding">
          <security mode="None"/>
          <resolver mode="Pnrp"/>
        </binding>
      </netPeerTcpBinding>
    </bindings>

  </system.serviceModel>
</configuration>
To complete the ISpectator functionality, I need to alter the Worker peer to make the appropriate calls. This includes a new client endpoint to the app.config:
<endpoint
  name="ISpectatorClient"
  address="net.p2p://MyPeerApplication"
  binding="netPeerTcpBinding"
  bindingConfiguration="unsecureBinding"
  contract="PeerWcfComm.ISpectator" />
This endpoint has a new name (so that my ChannelFactory constructor can find the correct settings) and the type of the new ISpectator service contract. For the code changes, I simply need to create a new ChannelFactory<ISpectator> and a new ISpectator client channel. I then changed the DoWorkInternal method on the Worker class to the slightly less aesthetically pleasing version shown in Figure 7. Essentially, before and after work item processing (sleeping the current thread) I'm going to call the WorkItemStateChanged method on the ISpectator channel, giving the appropriate state.
Figure 7 Work Status Tracking
this._state = WorkerState.Busy;
_spectatorClientChannel.WorkItemStateChanged(
  new WorkItemStatus() {
    WorkerId = _id,
    WorkItemId = wrk.WorkItemId,
    NewState = WorkItemState.Executing
  });

Thread.Sleep(new TimeSpan(0, 0, wrk.PretendToWorkSeconds));

this._state = WorkerState.Ready;
_spectatorClientChannel.WorkItemStateChanged(
  new WorkItemStatus() {
    WorkerId = _id,
    WorkItemId = wrk.WorkItemId,
    NewState = WorkItemState.Finished
  });

Load Balancing and Work Sharing
It's time to try another experiment. Again, create two Workers and a single User. Have the User generate a lot of work items for the Workers. Now, while the two Workers are working away, launch a third Worker instance. From this, you'll see an undesired behavior of my system: one Worker is idle while others have much work to do.
In a P2P application, new peers can come online at any time to participate in the mesh. In a server farm scenario, I want new peers to automatically join in and immediately start processing the collective workload.
The first change to make is in the Worker's IWorker.Heartbeat method implementation. Currently, the code simply outputs a Worker's state to the console. This needs to be replaced with code that will cache fellow Workers' states. I did this in the User and Spectator peers and can mostly cut and paste that code into the Worker class.
Now that the Worker peer can keep track of other Workers, I can add code to check for conditions where a Worker should share (forward) work items to idle Workers. I've altered the end of the Worker's MonitorWork method as shown in Figure 8.
Figure 8 Updated MonitorWork
if (_state != WorkerState.Offline) {
  if (_workQueue.Count > 1) {
    WorkerStatus firstIdleWorker = GetNextIdleWorker();

    if (firstIdleWorker != null) {
      WorkItem workToForward;
      lock (_workQueue)
        workToForward = _workQueue.Dequeue();

       _workerClientChannel.DoWork(
        firstIdleWorker,
        workToForward);
    }
  }

  ThreadPool.QueueUserWorkItem(MonitorWork);
}
Basically, after processing a work item and before rescheduling itself, the method checks for an idle Worker node. (I added the condition to do this only if I have more than one item in my own work queue. This is because it makes no sense to forward my only remaining work item to another Worker.) If an idle Worker is found, it dequeues a WorkItem and sends it to that idle Worker. The GetNextIdleWorker method contains the selection logic:
private WorkerStatus GetNextIdleWorker() {
  List<WorkerStatus> workers = new List<WorkerStatus>();
  lock (_workers)
    workers.AddRange(_workers.Values);
  return workers
    .Where<WorkerStatus>(
      w =>
        (w.State == WorkerState.Ready
        && w.PendingWorkItemCount == 0))
    .FirstOrDefault<WorkerStatus>();
}
The first thing I do is lock the _workers data cache and get a snapshot of the WorkerStatus values in it. I then look for Workers in the Ready state with no work pending, returning the first one found.

WorkItem Backup
If a WorkItem is moved to a machine and queued, what becomes of it if that Worker peer goes down accidentally? A process crash doesn't permit you to save data upon error. A good mitigation is to back up copies of WorkItems to adjacent peers. A complete backup solution consists of two actions: backup and restore. This isn't a trivial task.
The simplest backup solution would be to reuse the DoWork method. The first parameter specifies the ID of the Worker that is supposed to do the work. However, that message actually goes out to the entire mesh. I could take advantage of this fact and have all peers that do not match the given worker ID save the WorkItem as a backup. The actual targeted Worker of that message would eventually tell the others to delete that backup.
The tradeoff here is that every Worker is using up resources caching backups of work items. If there aren't many work items and Workers can process them pretty quickly, this might be a viable solution. However, if there are many work items in the system, it might get prohibitively expensive.
A better solution would minimize the total number of backups, but at the expense of some extra mesh messages.
Taking a page from Kevin Hoffman's article "How to Design State Sharing in a Peer Network", I can back up WorkItems to neighboring peers by sending a message limited to a single hop count. The backup request and subsequent communications can be as chatty as desired. Since these backup messages are meaningless to Users or Spectators, I won't use the IWorker service contract. I've created a new one called IWorkBackup:
[ServiceContract]
public interface IWorkBackup {
  [OperationContract(IsOneWay = true)]
  void BackupItem(WorkItemBackupRequest request);

  [OperationContract(IsOneWay = true)]
  void DeleteBackup(Guid workItemId);

  [OperationContract(IsOneWay = true)]
  void RestoringBackup(Guid workerId, Guid workItemId);
}
The first method is called by a Worker wishing to backup a WorkItem. The second is called when a WorkItem has been completed and other Workers can delete their copies of it. The last is called when a Worker determines that a backed up WorkItem should be restored and it intends to do so.
The message sent by BackupItem is defined by the WorkItemBackupRequest class:
[MessageContract]
public class WorkItemBackupRequest {
  [PeerHopCount]
  public int HopCount = 1;

  [MessageBodyMember]
  public WorkItem Item;

  [MessageBodyMember]
  public Guid OriginalWorkerId;
  public DateTime BackupRequestTimeStamp;
  public DateTime RestoreAt;
}
The HopCount member has the PeerHopCount attribute, which limits the message traversal through the mesh to the specified number of hops. I've hardcoded it to the value 1 so that this message will be sent only to adjacent (one hop) peers. There are also two members marked with the MessageBodyMember attribute: Item (the work item to be backed up) and OriginalWorkerId (the worker requesting the backup). These will be sent in the underlying SOAP message. I've also added two additional members for internal tracking.
When a Worker receives the RequestBackup message, it needs to cache that item. This is done easily enough by adding a new private field to store the backups:
private Dictionary<string, WorkItem> _workItemBackups;
Here is the implementation of the RequestBackup method for a Worker peer:
void IWorkBackup.BackupItem(WorkItemBackupRequest request) {
  request.BackupRequestTimeStamp = DateTime.Now;
  request.RestoreAt = DateTime.MaxValue;

  lock (_workItemBackups)
    _workItemBackups.Add(request.Item.WorkItemId, request);
}
This method sets the BackupRequestTimeStamp to the current time, and RestoreAt is defaulted to DateTime.MaxValue. These fields prove to be useful later.
Once work has been completed on the local Worker peer, it needs to send out a message to all peers that have potentially cached a backup of the work item. It does this by calling the DeleteBackup method and providing the ID of the WorkItem that has been completed.
Notice that I don't limit this message to adjacent peers like I did with the BackupItem message. The reason is that the peer mesh topology can change. In between the time that a backup was requested and the WorkItem is completed, Worker peers that were once adjacent may not be so now. Thus, I send out a general message to all Workers to delete their backup of this item, if they have one. For completeness, here's the code in the Worker peer that implements DeleteBackup:
void IWorkBackup.DeleteBackup(Guid workItemId) {
  lock (_workItemBackups)
    if (_workItemBackups.ContainsKey(workItemId))
      _workItemBackups.Remove(workItemId);
}
Backing up and removing work items is a breeze! However, the true difficulty arises in determining when a backed up work item needs to be restored. I need to determine both when a work item needs to be restored and who will restore it.
To determine when a work item should be restored, each Worker must periodically monitor its backup cache. For this, I've added another System.Threading.Timer called _backupMonitor, which gets set in the Start method for the Worker:
_backupMonitor = new Timer(
  this.MonitorBackups,
  null,
  new TimeSpan(0),
  new TimeSpan(0, 1, 0));
This can execute at a much slower period than the heartbeat, so I set the timer to fire every minute.
The method MonitorBackups handles the logic for restoring backups and can be seen in Figure 9. This method will iterate through the backed-up item and check each one for the need to be restored. In order to do this, some assumptions need to be made. First, I have a local variable named timeOutThreshold. This is the amount of time from when I was told to back up an item to when I should expect it to be processed. I hardcoded this to one minute (though it should ideally be configurable) because I know that my work items can last no longer than 15 seconds (this was defined by me in my User application). Thus, except in peak usage, this should be plenty of time. (A better solution might be for the backup request originator to determine this timeout and specify it in the request message. It has knowledge of its workload and could give a much better estimate.) If the current time is greater than the WorkItemBackupRequest.BackupRequestTimeStamp plus the timeOutThreshold, I'm going to investigate further. I won't automatically consider this work item to have failed because the original worker might simply be really busy. However, if the time has expired, I move on to checking the status of the original worker. If it is either Busy or Ready with items in its work queue I'll assume it is merely busy and will still get to this work item.
Figure 9 MonitorBackups
private void MonitorBackups(object context) {
  List<WorkItemBackupRequest> backups =
    new List<WorkItemBackupRequest>();
  TimeSpan timeOutThreshold =
    new TimeSpan(0, 1, 0); //This should be read in from config.

  lock (_workItemBackups) {
    backups.AddRange(_workItemBackups.Values);

    foreach (WorkItemBackupRequest backup in backups) {
      //If we've passed the RestoreAt time, we need to
      //take on this work item.
      if (backup.RestoreAt < DateTime.Now) {
        lock (_workQueue)
          _workQueue.Enqueue(backup.Item);

        _workItemBackups.Remove(backup.Item.WorkItemId);
      }

      //This work item has not been deleted yet.
      //This is suspicious so investigate further.
      else if (backup.BackupRequestTimeStamp.Add(timeOutThreshold) >
         DateTime.Now) {

        //Check to see if the original worker is still operating.
        //Regardless, we won't do anything and the backup
        //remains in our cache.
        if (!WorkerIsStillActive(backup.OriginalWorkerId)) {
          //The machine doesn't seem to be actively working.
          //Set a deadline for restoration of the work item.
          backup.RestoreAt = DateTime.Now.Add(timeOutThreshold);

          //Tell the world our intentions to restore this.
          _backupClientChannel.RestoringBackup(
            this._id, backup.Item.WorkItemId);

          Console.WriteLine("Planning to restore '{0}'",
            backup.Item.WorkItemId);
        }
      }
    }
  }
}

private bool WorkerIsStillActive(Guid workerId) {
  bool active = false;

  lock (_workers) {
    if (_workers.ContainsKey(workerId)) {
      WorkerStatus sts = _workers[workerId];
      if (sts.State == WorkerState.Busy)
        active = true;
      else if (sts.State == WorkerState.Ready &&
        sts.PendingWorkItemCount > 0)
        active = true;
    }
  }

  return active;
}
If I get to this point and see that both my assumed timeout has expired and that the original worker is unresponsive, I'll start the process of restoring that work item. This is done by calling the IWorkBackup.RestoringBackup method to announce my intention to all of the other peers. I also set the RestoreAt field of the WorkItemBackupRequest object to a future time. Although I am reusing the same TimeSpan value, this future time is logically different (and should ideally have a unique configuration point). RestoreAt is the time at which I will assume that all other peers have received my message and have had time to respond, if desired. More on this logic later. But for now, know that I set RestoreAt to the time when I will actually move the WorkItem from my backup cache to my work queue. The next time the MonitorBackups timer fires it will check this value and act appropriately.
This logic determines when a restoration occurs. It is the easiest part to determine. The trickiest is who should restore a work item. PeerChannel tries to maintain an optimal mesh and the topology can change as peers join and leave. As such, a single peer can have from two to seven adjacent nodes at any time. If a one-hop backup request was sent to all of them, I'd have several Workers all vying to restore the same work since they all have the same logic determining when a WorkItem should be restored. So what happens when they all simultaneously try and restore? In order to restore, the peers must first announce their intention by calling the IWorkBackup.RestoringBackup method (see Figure 10).
Figure 10 IWorkBackup.RestoringBackup
void IWorkBackup.RestoringBackup(Guid workerId, Guid workItemId) {
  lock (_workItemBackups) {
    //Only proceed if we have this item backed up.
    if (_workItemBackups.ContainsKey(workItemId)) {
      //Get the backup request for this work item
      WorkItemBackupRequest req = _workItemBackups[workItemId];

      //Check to see if we've decided to restore this.
      if (req.RestoreAt != DateTime.MaxValue) {
        //The restorer with the greatest id wins!
        if (this._id.CompareTo(workItemId) < 0)
          _workItemBackups.Remove(workItemId);
      }
      else {
        //We haven't decided to restore it and
        //someone else wants to. So let them.
        _workItemBackups.Remove(workItemId);
      }
    }
  }
}
At this point, there are three scenarios as seen from the viewpoint of a Worker that just received a RestoringBackup message from another Worker:
  1. The Worker is restoring an item I don't have cached.
  2. The Worker is restoring an item I have cached but don't yet intend to restore.
  3. The Worker is restoring an item I am also intending to restore.
Scenario one is simple as no action is needed. Scenario two is also simple in that I can merely remove the backed-up item from our cache. I haven't yet determined that it needed restoring, but someone else has so they can have it.
It is scenario three that requires tie-breaking. Each peer is acting autonomously and could simultaneously announce that they intend to restore an item. The messages could pass each other on the wire and thus more than one Worker could find itself in scenario three.
The magic question is: Who wins? Who restores and who ignores? The decision must be made on something that isn't relative between peers (like a system's time) and doesn't change. I decided that a tie goes to the highest valued Worker ID. An impartial call to Guid.CompareTo determines the winner. With this in place, the winner does the restore and the loser ignores.

Peer Fun
I admit it . . . P2P networking is fun. But this doesn't negate the fact that there are real-world scenarios where its decentralized nature makes it a good fit for applications. Furthermore, these applications aren't limited to chat or media pirating! Peer-enabling applications that require bulk processing is a natural fit, whether a custom application farm or a High-Performance Computing (HPC) scenario. It makes work coordination and processing on a large scale relatively easy.
The design of WCF makes it trivial to implement a service and evolve it as needed. The shared data and communication of a P2P mesh allows different clients to observe and participate in what's going on. This article gives a demonstration of creating a peer-enabled processing platform where multiple players (Workers, Users, and Spectators) can all function together for a common purpose: getting your work done.

Matt Neely is a Software Design Engineer at Microsoft on the SQL Server User Education Tools team. He spends the majority of his time convincing his friend Caleb he'll be mentioned in an upcoming article and shaving his dog (a goldendoodle) by himself with astonishingly horrific results. He can be reached at mneely@microsoft.com.


Inside Windows 7
Introducing Libraries
Yochay Kiriaty and Alon Fliess

This article discusses:
  • Windows 7
  • Libraries
  • Common File Dialog
  • Shell Programming Model
This article uses the following technologies:
Windows 7
This article is based on a prerelease version of Windows 7. Details are subject to change.
Contents
What Libraries Mean for Users in Windows 7
Library-Aware Applications
Using the Common File Dialog
Select and Consume Libraries Like Folders
Shell Programming Model
Using the Shell Programming Model
Using the New IShellLibrary API
Libraries under the Hood
Support the Full Library Model
Summary
This is the first article in a series of articles about Windows 7. The series focuses on new user experiences that developers can tap into to make their application shine on Windows 7. This article, part one, focuses on a new user profile storage concept in Windows 7, called Libraries. Download Windows 7 Release Candidate now to help you get the most out of this article.

What Libraries Mean for Users in Windows 7
Before we start diving into the Windows 7 Libraries API, we need to better understand why Libraries were introduced in Windows 7 and how they improve the experiences of users to manage their content. To better understand the concept of Libraries in Windows 7, we need to look back in time. Earlier versions of Windows, like Windows Vista and Windows XP, included sets of special folders for storing users content, like My Documents. Windows XP used the Constant Special Item ID List (CSIDL), which is a list of values that provide a unique system-independent way to identify special folders used frequently by applications, but which may not have the same name or location on any given system.
In the Windows Vista timeframe, CSIDL evolved into a new storage system called Known Folder IDs. As of Windows Vista, CSIDL special folders are referenced by a set of GUID values. It is important to note that the CSIDL system is still supported both in Windows Vista and Windows 7 for backwards compatibility. In the Known Folder IDs list you can find the FOLDERID_Documents folder. This folder represents both the My Documents folder as part of the users storage profile and the FOLDERID_Fonts as part of the system special folders. In Windows 7, the list of Known Folder IDs expanded to support Windows 7 Libraries. You can find there, among other things, GUIDs like the FOLDERID_DocumentsLibrary that represents the Documents Library. In Windows Vista, these special folders were automatically indexed to allow users to perform faster, more efficient, search operations on their content. However, we found that many users store files in a variety of places on their PC, not only in the "special" user profile folders, but also in various folders like c:\My Temp Folder, d:\Birthday2008\pictures, or even in remote storage. Storing files outside the users profile storage space affects the indexing, which hinders the search experience of users. We often find ourselves looking for a particular file that we worked on just a few days ago, forgetting where we saved it, and then discover that search is failing to find it because the file was never indexed.
With Windows 7, the concept of Libraries tries to address the problem of users storing content all over their PC by allowing them to have full control over their Documents Library folder structure. In other words, in Windows 7 users can define which folders to include in the Documents Library. Actually, this is true for any Library in Windows 7. We can say that Libraries are user-defined collections of folders that are logical representations of user content. By including folders in Libraries, the user is telling Windows where his or her important data is located. The system will index these folders, enabling faster searching and much richer viewing arrangement capabilities in Windows Explorer based on the file properties and metadata. Figure 1 displays the integration of several folders into a single library view and shows the rich search and pivots of Windows Explorer in Windows 7.
Figure 1 Integration of Several Folders into a Single Library View
Libraries are an integral part of the Windows Shell. This integration is very important because it enables users to browse their files in the same way as they would in a folder, which means there is no new behavior that users need to learn. Clicking on the Documents Library simply shows all your documents in one particular arrangement. Moreover, due to the fact that Libraries are integrated into the Windows Shell, users can perform search operations and filter results by using properties and metadata, such as the date the picture was taken, the genre of the song, and popularity of the item. In other words, by using Libraries users get to enjoy file storage that is both flexible and indexed.

Library-Aware Applications
We want to note that while most applications should work fine with Windows 7 Libraries, there is an opportunity to tap into the Windows 7 Libraries environment to provide richer user experiences. One may wonder what would happen if an application does not support Libraries. Let's imagine an application that, as part of its functionality, needs to save a file to disk. The application prompts the user with the option to select a location for the file to be saved. The user may pick the Documents Library as his or her save "location," since this is where the user usually goes when he or she needs to work on a document.
However, if the application doesn't recognize that the Documents Library is not a regular folder, the application will try to save the file directly to the Documents Library as if it is just another folder on disk. This represents a problem, since by now, as we already know, Libraries are a non–file system location and therefore can't be treated as regular folders. The application needs to become aware of the fact that they are dealing with Libraries. Luckily, Windows 7 includes an updated Shell API, Libraries API, and updated Common File Dialog that allow developers to handle Libraries properly.
A Library-aware application should include mechanisms for handling situations where users inadvertently try to pick Libraries as if they are folders for either saving files to a Library or loading the contents of a Library. Furthermore, most applications allow users to interact with the file system as part of the application experience. An application should provide users with the same familiar entry points and UI offered by the Windows 7 Libraries. By including folders in a Library, users designate where their important data is stored, telling us this is the content they care about. Applications should promote these locations by supporting Libraries in their application.
For developers, there are several integration points with Windows 7 that can help applications become Library-aware. Developers should review the following three integration points and select according to their needs.
  1. The most basic integration point is to simply use the Common File Dialog (CFD) for picking files and folders, and saving directly to a Library.
  2. The second integration point offers applications the option to shine on Windows 7 by enabling applications to select and consume Libraries content.
  3. The last integration point provides the more advanced integration option of supporting the full library programming model.

Using the Common File Dialog
The good news is that Windows 7 Libraries are a first class citizen in the CFD, allowing users to browse through and search libraries. They can even pick a specific library as a save location that is not just one of the folders from within the library, but rather the library itself! Because Windows 7 Libraries are storage-backed, users can save and copy files to any folder they have permissions for that are included in a Library. Every Library has a default save location where files go when users copy and save files directly to the Library. By default, this location will be the known folder that is included in a default Library, or the first folder added to a custom Library.
But (there is always this caveat!), it is highly recommended to use the new CFD interface that was introduced with Windows Vista and not the legacy version of the CFD. It is very important to note that the APIs for using the legacy CFD have not changed since Windows Vista and Windows XP, for application compatibility reasons. However, the legacy version of the CFD doesn't directly support Libraries or the full new user experience offered in Windows 7. Figure 2 displays the legacy and the new CFD side by side.

Figure 2 Old and New View of the New Common File Dialog

In the legacy CFD, even if Libraries are presented in the right navigation pane, they require an extra click to save into one of the included folders, rather than just the Library itself. Users can neither search directly from within the CFD or experience the rich preview handlers, nor select multiple files across folders, because the legacy CFD doesn't support returning multiple files from different folder locations. In contrast, this is a scenario that a Windows 7 Library supports.
It is important to use the proper APIs to show the correct version of the CFD. When it comes to showing a CFD using .NET, developers can either use the System.Windows.Forms.FileDialog or the Microsoft.Win32.FileDialog namespace. Since the latter uses the legacy version of the CFD, .NET developers should always use the WinForms namespace to show the new CFD. Figure 3 shows a code snippet that prompts the user to choose a save location by showing the common save file dialog, enabling the user to select folders or libraries.
Figure 3 SaveFileDialog File
System.Windows.Forms.SaveFileDialog _fd = new System.Windows.Forms.  SaveFileDialog();
_fd.Title = "Please choose a location to save your file";
_fd.FileName = "[Get Folder…]";
_fd.Filter = "Library|no.files";
if (_fd.ShowDialog() == System.Windows.Forms.DialogResult.OK)
{
    string dir_path = System.IO.Path.GetDirectoryName(_fd.FileName);
    if (dir_path != null && dir_path.Length > 0)
    {
        lblResult.Content = dir_path;
    }
}
Native code developers should use the new family of IFileDialog native APIs: IFileDialog, IFileOpenDialog, IFileSaveDialog, IFileDialogCustomize, IFileDialogEvents, IFileDialogControlEvents. These replace the legacy APIs from earlier Windows versions: GetOpenFileName, GetSaveFileName.
Figure 4 shows how to use the new family of IFileDialog native APIs to prompt the user with a save dialog to save to a folder / library.
Figure 4 IFileDialog Native API
*ppsi = NULL;
IFileSaveDialog *pfod;
hr = CoCreateInstance(
CLSID_FileSaveDialog,
NULL,
CLSCTX_INPROC,
IID_PPV_ARGS(&pfod));
if (SUCCEEDED(hr))
{
  hr = pfod->SetOptions(FOS_PICKFOLDERS);
  if (SUCCEEDED(hr))
  {
  hr = pfod->Show(hWndParent);
           if (SUCCEEDED(hr))
           {
           hr = pfod->GetResult(ppsi);
           }
    }
    pfod->Release();
}
The Shell native APIs are COM-based. Before using any COM object we need to initialize the COM object by calling CoCreateInstance. After initializing the *pfod IFileSaveDialog variable, we set the dialog options to pick folders by passing the FOS_PICKFOLDERS flag to the IFileOpenDialog.SetOptions(). This code tells the Open dialog to enable the user to select folders rather than files and allows the user to choose a Library as a save location. In the case of choosing a Library, the CFD will then return the default save location folder that is associated with the chosen Library.
Figure 3 and Figure 4 are very simple and don't introduce any new code. However, it is important to promote consistency among applications running on Windows 7 and in support of Windows 7 Libraries. The CFD is designed to provide a consistent experience with the new Windows Explorer, including Libraries. All improvements in Windows 7 Explorer have been carried over to the CFD. In most cases, it is the ideal way for users to browse and interact with Libraries from inside the application.

Select and Consume Libraries Like Folders
Let's imagine a case of a slideshow application that presents pictures to the user. By using Libraries, the user is essentially telling the system that his or her important pictures are stored in the Picture Library. The application can simply point directly to the Pictures Library and show the entire collection of pictures to the user. Furthermore, from the point of view of the developer, using the Library system can eliminate the need to maintain a separate configuration file or database of pictures, for example, since developers can rely on the Library System. Before we dive into the Shell Libraries programming APIs, we need to understand a few concepts regarding the Shell Programming Model.

Shell Programming Model
A Shell item (IShellItem), commonly referred to as an item, is the currency of the Shell UI and its programming model. Items are individual, self-contained content sources. For example, quite a few of the interface methods used for controlling CFDs use Shell items to refer to folders instead of file system paths. This is important because the CFD can communicate information about both file system folders and other virtual folders that you find in the Shell, such as the Control Panel or the Computer Folder.
Other important Shell COM interfaces are:
  • IShellLink interface represents a link, usually to a file, folder, or an executable
  • IShellFolder interface represents shell folder objects from the shell namespace. Using IShellFolder you can traverse through the contents of a folder, retrieve the display name of an item in a folder, parse a display name relative to the folder, and obtain an item ID list
Windows 7 introduces a new Shell API, called IShellLibrary, that you can query from IShellItem in order to work and support Libraries in Windows 7.
Now that we have defined the different components of the Shell Programming Model, we can see how Libraries fit into this model. Since Libraries are not file system locations, you cannot use file system–specific APIs like FindFirstFile. Instead, you have two main options to consume the contents of a Library.

Using the Shell Programming Model
You can use the IShellItem and IShellFolder interfaces and a number of helper functions to enumerate the contents of Libraries, just as if they were regular folders. This means that applications can consume the content of a Library without using the new Libraries API and with very little change to their existing codebase.
Figure 5 shows how to use the IShellFolder interface to enumerate the entire contents of the Picture Library.
Figure 5 IShellFolder Interface
IShellItem *psi;
HRESULT hr =  SHGetKnownFolderItem(FOLDERID_PicturesLibrary, KF_FLAG_CREATE, NULL, IID_PPV_ARGS(&psi));
if(SUCCEEDED(hr))
{
    IShellFolder *psf;
    hr = psi->BindToHandler(NULL, BHID_SFObject | , IID_PPV_ARGS(&psf));
    if(SUCCEEDED(hr))
         {
               IEnumIDList *penumIDList;
               psf->EnumObjects(NULL, SHCONTF_FOLDERS | SHCONTF_NONFOLDERS , IID_PPV_ARGS(&penumIDList));
               //use penumIDList to enumerate the content of the folder
          }
}
Here you can see that by using the helper function SHGetKnownFolderItem we can retrieve the correct location of the Library by passing the FOLDERID_PicturesLibrary. This is a GUID representing the known folder, which in our case is the Picture Library. A successful call will fill the IShellItem *psi interface with the correct information about the Library, represented as a Shell item. The rest of the code is standard Shell programming, where we use the BindToHandler to bind the previously obtained Shell item to a Shell folder (casting). Next, we enumerate through the different items in the Shell folder, which in the case of a Library can be either files or folders. The SHGetKnownFolderItem is a Shell Helper function and is part of a larger group of helper functions that can be found in the shlobj.h header file in the Windows 7 RC SDK. Note the SHCONTF_FOLDERS | SHCONTF_NONFOLDERS flags that we are passing. This is telling the shell folder that we want to return all of the files and folders in a library. We could pass SHCONTF_NAVIGATION_ENUM to get the library locations instead of the library contents.

Using the New IShellLibrary API
You can achieve the same functionality shown in Figure 5 by using the new Windows 7 IShellLibrary API, as shown in Figure 6.
Figure 6 IShellLibrary API
IShellLibrary *pslLibrary;
HRESULT hr = SHLoadLibraryFromKnownFolder(FOLDERID_PicturesLibrary, STGM_READ, IID_PPV_ARGS(&pslLibrary));
if(SUCCEEDED(hr))
{
       IShellItemArray *psiaFolders;
       hr = pslLibrary->GetFolders(LFF_STORAGEITEMS, IID_PPV_ARGS(&psiaFolders));
       IEnumShellItems *penumShellItems;
       psiaFolders->EnumItems(&penumShellItems);
       //work with penumShellItem to enumerate the items in the library
}
Here you can see that we use another helper function, SHLoadLibraryFromKnownFolder, to create the IShellLibrary object. From this object we can call the GetFolders method to return an IShellitemArray. This return is used later to obtain an enumerator to traverse through the entire contents of the Library.
In the last example, we used the helper function, SHLoadLibraryFromKnowFolder. As mentioned above, this helper function, and others related to Windows 7 Libraries, can be found in the shlobj.h header file from the Windows 7 RC SDK. Here is a list of the important Library helper functions:
  • SHAddFolderPathToLibrary (adds a folder to a library)
  • SHCreateLibrary (creates an IShellLibrary object)
  • SHLoadLibraryFromItem (creates and loads an IShellLibrary object from a specified library definition file)
  • SHLoadLibraryFromKnownFolder (creates and loads an IShellLibrary object for a specified KNOWNFOLDERID)
  • SHLoadLibraryFromParsingName (creates and loads an IShellLibrary object for a specified path)
  • SHRemoveFolderPathFromLibrary (removes a folder from a library)
  • SHResolveFolderPathInLibrary (attempts to resolve the target location of a library folder that has been moved or renamed)
  • SHSaveLibraryInFolderPath (saves an IShellLibrary object to disk)
Let's review the following code (Figure 7) that uses a few of these helper functions to create a new Library, associates a folder with that Library, and "saves" the Library in the Libraries Folder.
Figure 7 IShellLibrary Using the SHCreateLibrary Helper Function
IShellLibrary *pIShelLibrary;
HRESULT hr = SHCreateLibrary(IID_PPV_ARGS(&pIShelLibrary));
if (SUCCEEDED(hr))
{
      IShellItem *pIShellItem;
      SHAddFolderPathToLibrary(pIShelLibrary, L"C:\\Users\\Public\\Documents");
      hr = pIShelLibrary->SaveInKnownFolder(FOLDERID_Libraries, L"My New Library", LSF_MAKEUNIQUENAME,
            &pIShellItem);
      pIShellItem->Release();
      pIShelLibrary->Release();
}
Here you can see that we are creating a new IShellLibrary object using the SHCreateLibrary helper function. Next we add the public Documents folder to the Library object. Then we save the new Library in the Libraries folder, with the rest of the Libraries, and give it the name My New Library.
In Figure 7, we used the SaveInKnownFolder method of the IShellLibrary interface to save the new Library we just created. Most of the methods in the IShellLibrary interface are self-explanatory. However, let's review some that require our attention:
  • The Commit method commits library changes to an existing library file. This means that whenever you programmatically change a library, you need to call the commit method in order to save the changes.
  • The SetIcon method takes a resource DLL name and Icon index to set the library icon.
  • The SetFolderType receives a GUID of known folder-type templates as an input param. This GUID defines the type of the library, which can be one of the following: Generic, Pictures, Music, Video, and Documents. Setting the folder-type template changes the Windows Explorer view of the library and enables search and pivotal view options that are specific to the library type.

Libraries under the Hood
As shown in Figure 7, the code creates a new library file in the Libraries folder. A library in Windows 7 is stored as an XML definition file that has a file extension of .library-ms. The file name is the actual name of the library. For example, the Documents Library is represented by an XML file called Documents.library-ms. Library descriptions are saved on disk in the %appdata%\Microsoft\Windows\Libraries folder (also known as FOLDERID_Libraries).
Let's dig into the Documents Library definition file schema. The XML structure is pretty self-explanatory, but let's explain a few of its elements. As shown in Figure 8, at the top of the file we can find the Library "header" information:
Figure 8 Documents Library Definition File Schema
<libraryDescription xmlns="http://schemas.microsoft.com/windows/2009/  library">
  <name>@shell32.dll,-34575</name>
  <ownerSID>S-1-5-21-2127521184-1604012920-1887927527-4897363</ownerSID>
  <version>4</version>
  <isLibraryPinned>true</isLibraryPinned>
  <iconReference>imageres.dll,-1002</iconReference>
  <templateInfo>
    <folderType>{7D49D726-3C21-4F05-99AA-FDC2C9474656}</folderType>
  </templateInfo>
The root XML element is libraryDescription, which contains child elements that define the library, as follows:
  • <ownerSID> defines the Security ID of the user who created this library to isolate Libraries and protect user data from other users.
  • <isLibraryPinned> is a Boolean element that defines if the library is pinned to the left navigation pane in Windows Explorer and NOT to the Taskbar.
  • <version> defines the content version of this library, which reflects the number of times the library definition file has been changed.
  • <templateInfo> is an optional container element that lets the author specify the folder type (Documents, Pictures, Video) for controlling the arrangements of views in Windows Explorer.
  • <iconReference> defines an icon resource using the standard Windows Shell resource style. For example:
<iconReference> C:\Windows\system32\imageres.dll,-65 </iconReference>.
This icon presents the library in Windows Explorer.
Users can't change the default libraries icon via Windows Explorer, nor can they assign the icon to a new customized user-defined library. However, this can be done programmatically using the API, which we will cover in future parts of this series.
A key part of the XML is the list of locations that the Library represents, as shown in the following code:
<searchConnectorDescriptionList>
    <searchConnectorDescription publisher="Microsoft" product="Windows">
      <description>@shell32.dll,-34577</description>
      <isDefaultSaveLocation>true</isDefaultSaveLocation>
      <isSupported>true</isSupported>
      <simpleLocation>
        <url>knownfolder:{FDD39AD0-238F-46AF-ADB4-6C85480369C7}</url>
      <serialized>MBAAAE…. </serialized>
      </simpleLocation>
    </searchConnectorDescription>
  • <searchConnectorDescriptionList> contains one or more search connectors that map to physical locations included in the library.
  • <searchConnectorDescription> contains a simpleLocation element that describes one location included in the library.
  • <url> defines a URL for this location. For human-readability only, this cannot be used by developers since it is likely to be out of date.
  • <serialized> is the actual representation of a location in a library that is a serialized ShellLink object.
One final note: Applications should never attempt to access or edit Library description files. Instead, applications should always use the Shell programming model or the IShellLibrary API to consume and manipulate the Library content.

Support the Full Library Model
A user can add, remove, and reorder locations, as well as change the default save location through the Library Management UI. This is shown in Figure 9, which is available directly from Windows Explorer.
Figure 9 Library Management UI
In addition, library locations may be modified through the IShellLibrary interface. Any change to a library structure is reflected in the library definition file and persists directly to the underlying .library-ms file. You can be informed of such changes by monitoring any changes to the .library-ms file. Assume your application is relying on the content of a library, or your application is currently displaying the content of a given library. You would like to be notified once the content of the library changes. Applications that wish to be notified about changes to the library definition file can do so by using the native SHChangeNotifyRegister Shell helper function or by using the Managed FileSystemWatcher found in the System.IO namespace. Using these interfaces is out of the scope of this article since these are not new APIs and are well documented.
Another option to consider is the case in which your application needs to manage folders for users, like adding a new folder of pictures into the picture repository. If you are using the Pictures Library, you can use the library management dialog in your application to show the same dialog that Windows 7 offers users to manage their libraries. By doing so, you keep a consistent look and behavior that the user grows to appreciate. If you choose to use the library management dialog interface, changes to the Library will be made as if you were modifying the Library contents directly within Windows Explorer. This dialog will not return any information to your application. In case you are showing the contents of a given library, you will need to register for notification to receive updates, as explained above.

Summary
In this article, you were introduced to the concept and programming model of Libraries in Windows 7. You reviewed the important role that libraries play as part of the Windows 7 user experience. Then you took a deep-dive into Libraries, understanding what Libraries are and exploring their supporting underlying architecture. Then you saw the different opportunities that developers have, to make their applications Library-aware. Finally, you went through a short tour of the different available programming models and APIs.

Yochay Kiriaty is a Technical Evangelist at Microsoft, focusing on Windows 7. He has more than a decade of experience in software development. He has written and taught academic computer science courses and is an active contributor to The Windows Blog.

Alon Fliess is the CTO of Sela Group. Sela has an 18-year track record of success in IT training, and is recognized as Israel’s leader in the training of high quality IT personnel. Alon has a deep understanding of computer technology and holds a BSc degree in electrical and computer engineering from Technion, the Israel Institute of Technology. Alon has been programming and teaching for the past 22 years and is an expert in many Microsoft technologies, especially Windows internals, C++ Win32 programming, and .Net with C# or C++/CLI.


Editor's Note
Architecture and Aggregates
Howard Dierking

As you can see from the article lineup, we're focusing this month on issues related to software architecture. And along those lines, I want to take advantage of this editor's note to talk about something that I've spent some time thinking about recently—aggregates.
Now, those of you who I have had the pleasure of getting to meet (or those of you to whom I am related) can attest to at least two things about me. First, I'm always excited to talk about software development—architecture in particular. Secondly, in such conversations, I tend to talk louder and faster as my excitement grows (this is amplified by adding someone like my good friend Glenn Block to the mix). Aggregates is one of those architectural topics that I get really excited about, so make sure to turn down your speakers.
An aggregate is a term used heavily by practitioners of domain driven design (DDD) and is defined as "a cluster of associated objects that are treated as a unit for the purpose of data changes. External references are restricted to one member of the AGGREGATE, designated as the root. A set of consistency rules applies within the AGGREGATE'S boundaries."
Now let me say that I am not only a believer in this way of decomposing and thinking about a model, but I further believe that the failure to create strong aggregate boundaries is one of the major drivers behind spaghetti-like application designs that are hopelessly coupled to a relational database schema, and too intertwined to easily evolve to support message-based paradigms, such as those used in cloud computing architectures.
However, in some cases, I think we may be going about the process of defining aggregate boundaries in a sub-optimal way. I think at the root of the problem is that regardless of whether you design a database first or an object model first, both representations yield a structure of fine-grained logical entities—and this way of thinking about the abstractions can cause us to overlook opportunities to simplify.
To put this into an example, I've been working on developing a Microsoft Word add-in to tie article metadata to the manuscript. The metadata is managed in SharePoint—which, in database terms, feels more like the star topology of a Kimball-style data warehouse.
I started designing my application by creating a domain model. The initial scenario that I wanted to support was browsing the list of articles and selecting the article that the manuscript should be associated with. Because I wanted to not overwhelm my users with a giant list of articles, I added navigation to the scenario using a natural hierarchy whereby Magazine contains Issues which contain Articles. Sound familiar?
So what's the problem? This model looks similar to models we've all seen countless times over. However, it is also for this type of model where people run into trouble defining aggregate boundaries. Why? Because it is based on structure rather than behavior. To recap my scenario, I want to enable my users to navigate a set of articles using a natural hierarchy. By that logic, my hierarchy is simply a projection over a set of articles—and this changes my model such that I have a clearly defined, single article aggregate, as shown above.
Now, this is not to say that you should never have traditional hierarchies in your object models. Instead, I am trying to say that rather than defining aggregates by attempting to draw a line around a group of classes in a model, first define the course-grained aggregates based on behavior—then let your classes emerge inside the aggregate definition. You will avoid the analysis paralysis of trying to get the aggregate definitions "right" and you'll be starting out on the right foot, building your system based on behavior over structure.

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Jean-Paul Boodhoo, Pablo Castro, Ben Constable, Jeff Derstadt, Chuck Esterbrook, Eric Fabricant, Steve Freeman, Ashok Kamath, Bertrand LeRoy, Nat Pryce, Ashish Shetty, Don Smith, Chris Tavares, Kevin Hoffman, Dave Fugate, Harry Pierson, Curt Hagenlocher, Lynn Langit, Varsha Mahadevan, Stephen Toub, Mark Lawrence, and Henri Socha.


Toolbox
Managing in the Cloud, UX Design Patterns, Anders Hejlsberg’s The C# Programming Language, and More
Scott Mitchell

All prices confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions at Microsoft.

Contents
Project Management in the Cloud
Explore User Experience Design Patterns
The Bookshelf

Project Management in the Cloud
The success or failure of a software project is, to a large degree, a result of the methodologies and tools used to manage the project. Successfully managed software projects use tools such as source code control, issue tracking databases, and document sharing and collaboration among team members and stakeholders. Of course, success or failure is also dependent on whether there is a market for the software and the skill of the developers working on the project, but even an innovative product created by experienced developers will struggle to make it to market if there are serious deficits in how the project is managed.
Unfortunately, the process of selecting, installing, configuring, securing, and maintaining project management tools is a time-consuming and ongoing process. Larger companies have resources dedicated to these endeavors, but for smaller companies these expenses can be prohibitive. Such companies may be better served by offloading these tasks to an application service provider such as ProjectLocker. In a nutshell, ProjectLocker securely hosts your project management tools—like source code control software and an issue tracking database—and provides a Web-based interface for accessing and administering these tools. (ProjectLocker is designed to host closed-source projects where only specified users are allowed access to the source code repository and project management tools. If you are building an open-source application, consider using a hosting provider like CodePlex from Microsoft.
Each project you create has its own source code control repository. ProjectLocker offers support for two source control products: Subversion and Git. To interface with the source control system, you will need to download an appropriate client application. TortoiseSVN is an open source Subversion client that integrates with the Windows shell and uses icon overlays to indicate the status of individual files (checked in, checked out, modified, and so forth). Another option is VisualSVN, which is a third-party plug-in that integrates Subversion and TortoiseSVN into Visual Studio. (Subversion and TortoiseSVN were examined in the April 2007 Toolbox article, "WiX Installation, Creating PowerPoint Decks, and More", and VisualSVN was examined in the Visual Studio 2008 Launch Toolbox article, "SQL Code Completion, Subversion Tools, Agile Development, and more.")
Each ProjectLocker project also includes Trac, a wiki and issue-tracking system. The wiki serves as a virtual whiteboard where team members can discuss ideas and where project managers can post specs and milestones. The issue-tracking system is useful for maintaining information about feature requests and bugs. Tickets can be submitted from the project Web site or via e-mail, and administrators can customize the fields associated with each ticket item. Trac also seamlessly integrates with the source control system; developers can browse the files in the repository from the project Web site and view the history of changes. Furthermore, Trac includes a number of plug-ins that you can enable at the tick of a checkbox, including online forums and time tracking services.
Like most application service providers, ProjectLocker charges its users a monthly fee. ProjectLocker does offer a free account option, but it's limited to two user accounts and 300 MB of disk space. Paying plans start at approximately $5/month. There are additional enterprise features you can add for a monthly fee, which include source code control analytics, document management, and BuildLocker, a hosted build server for performing continuous integration.
Price: Varies by features; includes a free account option.
projectlocker.com
ProjectLocker

Blogs of Note
Discovering interesting and relevant content in the vast sea of developer blogs and resource sites can be like finding a needle in a haystack. Many people refer to one or two trusted resource sites and subscribe to only a few blogs rather than spend time scouring the Net for hidden gems. However, there are blogs that focus on finding the latest "best-of" technical content so that you don't have to.
One such aggregate blog is Chris Alcock's, "The Morning Brew." Each weekday, Chris compiles the latest and greatest technical content into a blog entry. Each installment of "The Morning Brew" includes between 5 and 15 links to recommended blogs entries, articles, tutorials, and other information pertinent to .NET developers. Chris groups each link into categories such as software, information, and community, and includes a synopsis of the recommended article.
Also check out Alvin Ashcraft's blog, "Morning Dew," which offers a daily collection of recommended articles titled "Dew Drop." These entries include a smattering of links covering a wide array of technologies and discussions of interest to .NET developers, including topics on Visual Studio, software design, Web development, community, and database development. Furthermore, most "Dew Drop" entries include a recommended book of the day.
In general, Alvin's entries include more links from a more diverse set of sources. While "The Morning Brew" includes fewer links per entry, each link includes a one or two sentence summary; "Dew Drop" entries in the "Morning Dew" include only a link and a title. Both blogs provide a valuable service by gathering interesting technical content from around the web and putting it all in one place.
The Morning Brew: blog.cwa.me.uk
Morning Dew: alvinashcraft.com

Explore User Experience Design Patterns
Designing and implementing the user experience (UX) is arguably the most important part of software design. It doesn't matter how efficient or how well-architected your application may be if your users cannot effectively use it. Unfortunately, most software developers struggle when it comes to designing the user experience, in part because UX design seems so touchy-feely whereas programming is much more logical and ordered. To help software developers build a better user experience, the Infragistics UX Team has released Quince, an online collection of UX design patterns.
A design pattern is a generic, high-level solution to a regularly occurring software design problem. It does not spell out step-by-step instructions on how to implement the pattern, but rather describes the problem, includes discussion about how and why the pattern solves the problem, offers guidance on implementing the pattern, includes an example of it in action, and cites related patterns. Design patterns are most commonly used in the realm of object-oriented programming languages and use code samples to illustrate the pattern in action. But as Quince illustrates, design patterns can also be used to share solutions to common problems in UX design.
Each UX pattern in Quince includes a succinct description of the problem, posits a solution, provides a context for the problem, describes the rationale behind the solution, and describes steps for implementing the pattern. Each pattern also includes screen shots and discussion of several real-world examples from various desktop and Web applications.
For example, the Alphanumeric Filter Links pattern starts with a problem description: "You have too many items for people to reasonably digest in one screen." It then goes on to offer a general solution, namely to show a list of alphabetic links that enable the user to filter the results. The pattern then explains the rationale behind filtering a long list of items, and includes tips and guidelines for implementing such a list. For instance, if the list being filtered includes entries that start with non-alphabetic characters, then the list of links must include an appropriate filter, such as "0-9". The pattern also includes five screen shots of successful alphanumeric filter lists from a number of applications and Web sites, each accompanied with a paragraph or two of analysis.
Quince offers several ways to browse the myriad of UX design patterns it has catalogued. You can browse all patterns, filtering them by category or keyword. Alternatively, you can see patterns grouped by user tasks, such as search, drill down, data entry, numeric input, display of complex data, and so on. There's also a wireframe view, which shows a standard application user interface with the links to the various design patterns located on-screen where they would typically appear in the UI.
Design patterns have proven to be indispensible tools for software engineers when building object-oriented programs. They can also help you overcome common UX design challenges with tried and tested solutions. Quince is an excellent resource for finding and learning about the most pertinent UX design patterns and should be consulted when you're working on an application's user experience.
quince.infragistics.com
Quince

The Bookshelf
One of the exciting (and terrifying!) things about our field is that there's always something new to learn: new programming languages, new technologies, new tools, and new frameworks. Over the years I have routinely scheduled a few hours a week to read a book or visit a Web site to learn about the latest new thing. However, I recently decided to reappropriate that time to reinforce my knowledge of the technologies that I use every single day—technologies like C#, SQL, and the Microsoft .NET Framework base class libraries (BCL). To this end, I picked up a copy of The C# Programming Language, 3rd Edition, by Anders Hejlsberg et al (Addison-Wesley Professional, 2008). (Anders Hejlsberg is a Microsoft Technical Fellow and the chief designer of the C# language.)
At 754 pages, The C# Programming Language is a dense, technical, and very thorough examination of C# 3.0's syntax and semantics. The first three chapters provide an overview of the language's key features, including types and variables, classes and objects, arrays, interfaces, and events. There's also an overview of C# grammar. The concepts discussed in these introductory chapters are then explored in detail, each in its own chapter. For example, there's a chapter on types, one on variables, one on conversions, another on expressions, and so forth.
The writing style in The C# Programming Language is formal and dry. There are no figures or screen shots; the code samples given are terse. The book reads like a technical specification. However, there are several annotations throughout the book from the authors and other prominent C# software engineers both from within Microsoft and from the wider developer community. Some annotations offer a real-world example to complement the material discussed. Others include pearls of wisdom or point out common pitfalls that may surface when working with a particular feature. These annotations are written in a more conversational voice and help add context to the material.
The C# Programming Language is not for everyone. While the book does include introductory chapters, I would recommend this book only to developers with a working knowledge of the language. This book is best suited for intermediate to advanced developers who want to explore the nitty-gritty details of C# and gain a deeper understanding of the language's syntax and semantics.
Price: $44.99, 754 pages
informit.com/store/product.aspx?isbn=0321562992

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Memory Usage Auditing For .NET Applications
Subramanian Ramaswamy and Vance Morrison
Contents
When Memory Usage Affects Speed
What Can Be Done?
Task Manager
Shared Versus Unshared Memory
Application Size
VADump: A More Detailed View
The .NET Garbage Collector
PerfMon
Conclusion
Performance optimization is about one thing: making computer programs run faster. The execution of instructions is cheap for modern computer hardware while the fetching of instruction operands is expensive. Thus, memory usage can have a direct impact on how fast an application executes and is an important metric to optimize. In this article, we discuss the basics of memory optimization for .NET programs. First, we outline the cases where memory access is a bottleneck and is useful to optimize. Next, we discuss the general breakdown of how memory is used in a typical .NET program. Lastly, we discuss tools and strategies to determine the memory consumption of your .NET application and reduce it.

When Memory Usage Affects Speed
The first case of when memory consumption matters is a CPU-intensive application that is manipulating a large amount of data. A typical PC can execute an instruction in less than half a nanosecond (0.5 ns). However, that speed is limited by how long it takes to fetch the operands from memory. Modern processors have a hierarchy of caches to optimize the cost of hardware. The level-1 (L1) cache is the fastest memory, but is relatively small. Next in the hierarchy is the level-2 cache, followed by the main memory (RAM), and finally the hard disk drive. Figure 1 shows the access time and sizes of the various parts of the memory hierarchy for a typical PC. At every step deeper into the memory hierarchy, the access time (and size) increases by an order of magnitude or more (hard drives are a 10,000 times slower than RAM) while the cost (per byte) decreases.
Figure 1 Size and Access Times with Non-local Storage
 | L1 Cache | L2 Cache | Memory (RAM) | Disk
Size | 64K | 512K | 2000M | 100,000M
Access Time | .4 ns | 4 ns | 40 ns | 4,000,000 ns
If hot data paths access more memory, then the operands will frequently need to be fetched from slower memory. Since the slower memory is slower by an order of magnitude, a few Level-2 cache misses can have a significant performance impact.
The second case of when (some) memory consumption matters is during an application's cold startup. As Figure 1 shows, hard disk access is much slower than main memory access. The operating system tries to mitigate this by caching data from the disk in main memory. This is the reason that an application is faster when launched the second time, during what is called warm startup (the data was cached in faster memory). For the first (cold) startup, caching has not yet happened and data has to be fetched from disk. The only way to improve this is to load less data from the disk. Only memory fetched from the disk (such as the program instructions) affects cold startup; memory initialized by the program itself, including all data on the heap and stack, does not affect cold startup.
The final case when memory consumption matters is during application switching. When your application is reasonably large (larger than 50MB), and a user switches to other applications, these applications steal the physical memory of your application. When the user returns to your app, these stolen pages need to be fetched back from the disk, which makes your app very slow. This is similar to the cold startup case except that it affects not just program instructions, but all memory—including memory that was initialized by your application. Since servers run many unrelated programs simultaneously and continuously, servers are application switching constantly; this means that memory is almost always an issue for servers.

What Can Be Done?
If code could be magically rearranged to ensure all memory requests were satisfied in the fast caches, the program would speed up substantially. In practice, this is only possible in unusual circumstances because ordinarily the program's algorithms dictate the order of memory accesses. A more feasible technique is to minimize the amount of memory used. This reduces the load on the fast caches and makes the program faster. For data structures having frequently accessed (hot) parts that do not fit in CPU caches (typically they would be larger than several megabytes), a 30 percent reduction in the memory size of hot data typically results in a 10 percent improvement in CPU speed.
Memory can be reduced in three ways. First, you can execute less code (which helps cold startup). This applies to obvious cases where something was computed inefficiently in the first place. Second, you can touch less data. This is similar to the first strategy, but it applies to the data structures involved. Finally (and perhaps most commonly), the data structure can be encoded in a different way, making it smaller, or by physically separating the (small) commonly accessed data from the (large) uncommonly accessed part.
These techniques typically require a change in the representation of the data and require changes to a large number of code sites to implement. Thus, it is much easier to make these changes early in the development cycle, so it pays to think about memory early!

Task Manager
The first step in reducing the memory consumption of your application is to understand how much of it is currently used. For that you can use the Windows built-in Task Manager application.
Most users are already familiar with task manager. You can invoke it by typing taskmgr in your run command window (Winkey+R), or by pressing Ctrl+Alt+Del and selecting "Start Task Manager". On the "Processes" tab, you will find information on all the system's currently running processes. If the columns don't include PID, Memory-Working Set and Memory-Private Working Set, use the View | Select Columns' menu option to add them to the display.

Shared Versus Unshared Memory
The working set is the physical memory currently being used by the process. However, the operating system performs optimizations to ensure that all memory is not equally expensive. Much of the memory a process uses holds read-only data (for example, the actual instructions to execute). Because this data is read-only it can be shared among all processes that need it. Since all processes make extensive use of shared, read-only operating system code, a substantial amount of every process's working set is shared. Thus, total working set tends to significantly overestimate the true cost of the memory used by a process.
The operating system also keeps track of unshared (Private) memory. This includes all read-write memory used by the process. While private working set underestimates the true cost of memory used by a process (we will see how when we discuss the tool VADump), it tends to be a better metric to optimize, because unlike optimizing shared memory, any gains in private memory will reduce the total memory pressure on the machine.
Finally, both total and private memory counts miss an important memory used by a process: the file system cache. Because hard disk access is so expensive, even when a file's data is not mapped directly into memory, it is cached by the operating system. This memory use increases memory pressure on the system, and is not included in either of the working set metrics (it is owned by the operating system). There is not much that can be done about file access (if your program needs a file, it can't be avoided), so it can be considered a cost that can't be optimized.

Application Size
An application may be categorized as small, medium, or large depending on its memory usage. A small application has a 20MB or smaller working set size with a smaller than 5MB private working set; a medium application has a working set size of approximately 50MB with about 20MB private working set; a large application typically has working set sizes greater than 100MB, with private working set sizes exceeding 50MB. The larger your application, the more valuable optimizing your application's memory usage is likely to be.
A simple and quick way to monitor memory usage and check for leaks is by running a sniff test on your application. Run the application for a while and monitor its working set usage; if the working set grows unbounded, that can mean a memory leak or other issues.

VADump: A More Detailed View
Task Manager provides only a summary of the memory usage of an application. To get more detail you need a tool called VADump (see the Resources sidebar). This is invoked by typing VADump –sop ProcessID in the command prompt under the directory in which VADump is installed. It prints a breakdown of memory within a single process down to DLL level of granularity. A typical dump is shown in Figure 2.
Figure 2 VADump Output Opened in Notepad
To read the dump, start with grand total working set. This number should agree with the number in Task Manager. This number is then broken down into eight categories. The most interesting of these categories are:
  • Code/Static Data, which represents DLLs that were loaded by the process.
  • Heap, which represents native (not GC) heap memory used.
  • Other Data, which represents memory allocated using the OS VirtualAlloc function. For managed code this is important because it includes the entire garbage-collected heap.
Performance Resources

VA Dump:
go.microsoft.com/fwlink/?LinkId=149683
CLR Perf Team Blog (Instructions on investigating suspicious DLL loads):
blogs.msdn.com/clrperfblog
VS Profiler Team Blog:
blogs.msdn.com/profiler
Improving .NET Application Performance and Scalability:
msdn.microsoft.com/library/ms998530
Windows performance blog: Investigations using Xperf:
blogs.msdn.com/pigscanfly/
Vance Morrison's Blog:
blogs.msdn.com/vancem
Rico Mariani's Blog:
blogs.msdn.com/ricom
Lutz Roeder .NET Reflector for inspecting code:
blog.lutzroeder.com
CLR Inside Out - Investigating Memory Issues:
msdn.microsoft.com/magazine/cc163528
The memory used by DLLs is further broken down by VADump after the summary table. For each DLL, it shows the number of pages (a page is always 4K) that each DLL uses. Thus, one can determine the memory cost of all the code that is loaded.
In Figure 2, there is a row labeled "Grand Total Working Set." The total working set in Kilobytes and Pages is in the first column. Columns 2, 3 and 4 (Private Kbytes, Shareable Kbytes and SharedKBytes) add up to the Total Working Set column. It is Column 2, the Private Kbytes value, that is depicted as Private Working Set in TaskManager, whereas Column 1 is shown as Total Working Set in TaskManager. Thus, VADump allows you to see the separation between private and total working sets, including shareable and shared working sets. This is a more complete picture than what is available through TaskManager.
When .NET applications are large, they are typically large either because they run a lot of code or they use a lot of data.
In this case, you will see a large number of DLLs loaded and the Code/Static Data contribution tends to dominate the total working set. For managed applications, this data is in the GC heap and thus shows up as Other Data dominating the working set.
In the lower part of Figure 2, you see module working sets (in pages) listed. This tells you which modules contribute to the working set of the application and how much working set each module is consuming. Thus, you can very quickly determine how much working set a particular DLL contributes in terms of the DLL's private, shared, and shareable working set. This view unambiguously shows whether a DLL load can be eliminated and how many bytes of private working set can be shaved off the application's working set.
Once a DLL that may not be pay for play is identified (for example, a DLL may be loaded even if it is not used in a particular execution), the next step is to identify why the particular DLL gets loaded and seek to eliminate an unwanted load. Steps for investigating suspicious DLL loads can be found in the CLR and Framework Perf Blog.
The heap data that is shown by the VADump output is for the unmanaged heap—this is memory that will not be managed by the .NET GC. It is important to keep this number small so the GC can manage most of your memory by cleaning up as necessary.
The Other Data category represents calls to a primitive OS memory allocation function (VirtualAlloc) that VADump cannot categorize in any other way. For .NET applications, typically the most important component of Other Data is the garbage-collected heap that holds all user-defined objects.

The .NET Garbage Collector
The .NET runtime supports automatic memory management. It tracks every memory allocation made by the managed program and periodically calls a GC that finds memory that is no longer in use and reuses it for new allocations. An important optimization that the garbage collector performs is that it does not search the whole heap every time, but partitions the heap into three generations (0, 1, and 2).
Generation 0 is the smallest of these, and typically takes only 1/10th of a millisecond to complete but only looks to clean up the allocations that happened after the last GC (and obviously, are not being used). Ideally, the size of a generation is less than the L2 cache size. Generation 1 GCs tackle the allocations that survived one GC; it takes longer to run than Gen 0 GCs, taking about 1 millisecond. Ideally, there should be 10 Gen 0 GCs for every Gen 1 GC.
Gen 2 GCs tackle all objects. Thus, the time taken can be significant. For example, it can take about 160 milliseconds for a 20MB heap, which is a noticeable amount of time. The time grows roughly linearly with the size of the heap (about 8 milliseconds per MB as a very rough estimate). The true cost depends on the amount of memory surviving, the number of GC pointers in surviving memory, and how fragmented the heap is. Ideally, there should be 10 Gen 1 GCs for every Gen 2 GC.
Taken in its entirety, the .NET GC heap looks like a sawtooth with the troughs corresponding to Gen 2 collections, as shown in Figure 3. The typical Gen 2 heap-to-trough ratio is about 1.6, with the ratio being largely independent of heap size (with no fragmentation). In the presence of fragmentation, this number can vary significantly.
Figure 3 GC Heap Sawtooth Waveform

Perfmon
VADump gives the first level of breakdown of memory usage in the process. However, it does not precisely tell us how much GC memory we are using (the Other Data category can include memory other than the GC heap), and it does not tell whether we have a healthy ratio of GC generations. For that we need to use the Windows PerfMon application. You can start it by typing PerfMon in the run command window which should bring up the window shown in Figure 4. PerfMon is able to gather a wealth of performance data, but here we focus in on its use for monitoring the GC heap.
Figure 4 PerfMon Startup Screen
Figure 5 Selecting Counters to Monitor in PerfMon
After PerfMon comes up, we need to configure it to display information about the GC. We do this by first clicking on the Performance Monitor item in the tree control in the left pane. This changes the right pane to display performance counter data. Now click the + sign for adding new counters. Next, select the counters you want to monitor as well as the processes you'd like to watch, as shown in Figure 5.
When you select a few counters, you will notice names of all the applications using the runtime. You may select one or two or however many applications you wish to monitor. In addition, there is an instance named All instances which is to enable monitoring data across all instances shown but the data will be displayed separately. In addition, there is a _Global_ instance, which sums up the data from the different instances.
If an application was started after PerfMon was being used to monitor other applications already, one can add more applications by clicking the + sign and adding counters for the new application (only adding the new instance is required; the other instances will continue to display in PerfMon).
Finally, by default the data is shown graphically, but it is more useful to display it numerically. This can be done by clicking on the report-type toolbar (Figure 6). In one test, it showed us that 7.3MB out of the total 8.6MB of private working set was take up by the GC heap and about 11 percent of time was spent in GC. A healthy number for the time in GC is less than 10 percent of total application time, so this particular application would be on the borderline. Finally, it also tells us the number of Gen0, Gen1 and Gen2 collections. Ideally, we want the number of Gen0 collections to be at least 10 times that of Gen1 collections, and the number of Gen1 collections to be at least 10 times that of Gen2 collections.
Figure 6 PerfMon Report Display

Conclusion
Memory issues are notoriously difficult to debug. If your application is large enough to care about memory, the key is to limit memory usage in the early stages of development. Understanding how application memory is broken down is the first step in this process; monitoring application memory usage is next. Follow that with identifying opportunities by asking questions about which DLLs contribute the most to memory consumption, why Gen2 GC's are happening so frequently, are your memory allocations pay for play, and so forth. Then optimize memory usage appropriately. If you take away only one lesson, it should be that the time spent early in the development cycle on memory issues pays for itself later, so it really pays to think about memory early!

Send your questions and comments to clrinout@microsoft.com.

Subramanian Ramaswamy is the Program Manager for CLR Performance at Microsoft. He holds a PhD in Electrical and Computer Engineering from the Georgia Institute of Technology.

Vance Morrison is the Partner Architect and the Group Manager for CLR Performance at Microsoft. He drove the design of the .NET Intermediate Language (IL) and he has been involved with .NET since its inception.


Data Points
Building An Out-of-Browser Client With Silverlight 3
John Papa

This article is based on prerelease versions of Silverlight 3 and Expression Blend.

Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Eyes On TheTarget
Install To Desktop
Manifest Essentials
Running Offline
Network Connectivity
Storing Data Offline
Silverlight 3 applications are no longer restricted to running in a browser. They can be run in a browser or it they can be detached from the browser and run from the desktop. These out-of-browser applications allow you to bring the richness of Silverlight 3 applications directly to the desktop without the restriction of running within a browser.
In this month's column, I'll discuss how to build a data-driven standalone Silverlight 3 application. I'll begin by demonstrating an overview of the sample application. Next, I'll outline the steps needed to take the application and make it work outside the browser. Being out of the browser requires some additional features such as an ability to detect connectivity. I'll discuss how to detect the status of the offline application and how to handle storing data offline when the network connectivity is interrupted. Other features I'll discuss along the way are building icons into the out-of-browser application, using Isolated Storage to store data on the client computer, showing the status of the out-of-browser application, and handling state change events. The sample application is a lightweight version of a Twitter-like client that sends and retrieves messages from the Twitter servers. All examples are included in this month's code download.

Eyes On TheTarget
Before diving into the out-of-browser features, it is important to understand what features I'll use to build the sample application in this article, so let's begin with an overview. The sample application demonstrates how to build a data-driven application with Silverlight 3 and its out-of-browser features. The goal of the sample application is to create a Twitter client that allows sending of new messages and retrieval of a user's friend timeline, replies, and direct messages. It is a simple, lightweight version of a Twitter client, though many more features could be added following the models I'll discuss here. Figure 1 shows the sample application running out of the browser.
Figure 1 Silverlight Twitter Client at a Glance
The SilverTwit application will request and show the messages from everyone the user is following when the Home button is clicked. When the Replies and Direct buttons are clicked, the replies to the logged-in user and the direct messages to the logged-in user are retrieved and displayed, respectively. The request for the messages is made from the Silverlight out-of-browser application to a Web service (also included in the sample project) and then the Web service relays the requests to the public Twitter API. This relay through a third-party service is required because Twitter does not allow cross-domain communication to be made from a Web client to Twitter directly. However, Twitter does allow calls made from server applications such as the one used in the sample solution. Other alternatives are to use PopFly or Yahoo Pipes to relay the requests.
For more information on cross-domain policies, please refer to my September 2008 Data Points column. In that column, I discuss the file formats and how the policies work.
When a user enters a message and clicks the Post button, the message is sent by way of the relay Web service on to Twitter. If the application cannot detect a network connection and a user tries to post a message, the message will be stored locally until a network connection can be restored. At that time the deferred message will be sent to Twitter.
Notice that the chrome around the sample application shown in Figure 1 contains the application icon, the application's title, and the standard minimize, maximize, and close icons. The title and icons used for the window, the desktop, and other aspects of the application can be customized when you're building the application in Visual Studio or Expression Blend. This window is launched using the sllauncher.exe process and its host the Silverlight 3 out-of-browser application.
The bottom of the SilverTwit sample application contains statistics on whether a network connection is detected or not, if the application is running in or out of a browser, and whether the application has been detached. These features all help the out-of-browser application identify how it should operate. For example, if a network connection is not detected, there is no reason to look for new messages or send new messages to Twitter. When in this disconnected network state, the application must react accordingly. A Detach button also exists in the lower section of the application that will detach the application to run outside of the browser.

Install To Desktop
Silverlight 2 applications are restricted to running inside a browser. However Silverlight 3 applications can run inside the browser or out. The process of taking the application out of the browser is known as "detaching." Once the Silverlight 3 application has been detached, it can be run from the Start menu or from a desktop icon. Out-of-browser applications run on the desktop inside a window that is hosted from the sllauncher.exe process. The out-of-browser applications can still access the network and use all of the .NET framework libraries available to Silverlight while they are detached. There are a few limitations for out-of-browser applications. For example, since they are not hosted in the browser, communications with the browser's DOM are not possible like they are with Silverlight applications hosted in the browser.
Figure 2 outlines how a Silverlight 3 application can be run in a browser and then detached. Clicking the Detach button runs the event handler shown in Figure 2, which ultimately runs the Application.Current Detach() method. Figure 3 shows the architecture. The Application.Current.Detach method will raise an exception if it cannot detach, so some error handling is a good idea.
Figure 2 Detaching from the Browser
private void DetachButton_Click(object sender, RoutedEventArgs e)
{
    try {
        Application.Current.Detach(); // take the app out of browser
    }
    catch (InvalidOperationException opex)    {
        MessageBox.Show("Application is already detached" +
          " to the desktop.");
    }
    catch (Exception ex)    {
        MessageBox.Show("Application could not be detached" +
          " to the desktop."+ Environment.NewLine +
          ex.Message);
    }
}
Figure 3 Detaching a Silverlight Application to Run Outside the Browser
The users can then decide if they are sure they want to detach the application from the browser. Silverlight also gives the user the option of adding a shortcut to the desktop and/or to the Start menu for the out-of-browser application. Once the user clicks OK, the application is installed on the desktop and begins launching (hosted by the sllauncher.exe process). At this point, the browser can be closed and the out-of-browser application continues to run on its own.
The application can be removed by right-clicking on the out-of-browser application and choosing Remove Application from the popup menu. This can be done from either the out-of-browser application or from the same version running in the browser. Once removed, the desktop icon and the Start menu item are both removed. It is worth noting that out-of-browser applications will not show up in the Control Panel's add/remove programs list.

Manifest Essentials
Taking an application offline is a simple process for the user and almost as simple from a development standpoint. Silverlight 3 Beta requires a few simple changes to be made to the appmanifest.xml file in the Silverlight project to allow it to be taken out of browser. If these changes are not made, then the Silverlight application cannot be detached and run out of browser. The AppManifest.xml is located in the Properties folder by default and it can be edited manually to include the out-of-browser capabilities.
Figure 4 shows the AppManifest.xml file from the sample application with the items of interest in bold in the new Deployment.ApplicationIdentity section. The ShortName is used in the shortcuts for the out-of-browser application (in the Start menu and the desktop shortcut). The Title appears in the title bar of the out-of-browser application. The icons are optional; however, either all of the icons must be included or none are included. This requirement may change in the final release of Silverlight 3, but in the beta, the out-of-browser would not operate properly if some icons are specified and not others. The various icons are used for the Start menu shortcut, the desktop icon shortcut, the icon for the window in the out-of-browser application, and in the dialog that prompts users to decide whether they want to install the application outside the browser.
Figure 4 Setting of the AppManifest.xml
<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        EntryPointAssembly="SilverTwit"
        EntryPointType="SilverTwit.App">
  <Deployment.Parts>
    </Deployment.Parts>
  <Deployment.ApplicationIdentity>
    <ApplicationIdentity
         ShortName="SilverTwit"
         Title="SilverTwit - Silverlight Twitter Client">
      <ApplicationIdentity.Blurb>Tweet, tweet and retweet</ApplicationIdentity.Blurb>
      <ApplicationIdentity.Icons>
        <Icon Size="16x16">images/SilverTwit16.png</Icon>
        <Icon Size="32x32">images/SilverTwit32.png</Icon>
        <Icon Size="48x48">images/SilverTwit48.png</Icon>
        <Icon Size="128x128">images/SilverTwit128.png</Icon>
      </ApplicationIdentity.Icons>
    </ApplicationIdentity>
  </Deployment.ApplicationIdentity>
</Deployment>

Running Offline
Another new feature added to Silverlight 3 is the Application.Current.RunningOffline property. This property returns a Boolean value that indicates whether the Silverlight application is running online (in browser) or offline (out of browser). This is useful to know, especially when you might want to run code that communicates with the browser's DOM from Silverlight. If you are running online, the application can communicate with the browser's DOM. However, if you are running offline, the application is hosted in the sllauncher.exe process and does not have a DOM (since it is not a browser). Therefore, before running this code, you could check the RunningOffline property.
When the out-of-browser app first runs, it always opens to a default size. Once open, the window can be resized, maximized, minimized, or closed. The starting size and control of the window itself is not currently a feature in the Silverlight 3 Beta.
The sample application shown in Figure 1 is running outside the browser. This is indicated in the lower portion of the window where it says "Running Out of Browser." This check is done using the code shown in Figure 5. If the application were running in the browser, the message would read "Running In Browser" as shown in Figure 6, where the application is running inside the browser.
Figure 5 Online or Offline?
private void SetOfflineStatus()
{
    if (Application.Current.RunningOffline)
        stats.RunningModeMessage = "Running Out of Browser";
    else
        stats.RunningModeMessage = "Running In Browser";
}
Figure 6 Running Online
Another new feature of Silverlight 3 is that the application is notified by an event (ExecutionStateChanged) whenever the execution state of the application changes. There are five different states (found in the ExecutionStates enumerator) that can cause this to occur:
  • RunningOnline – The application is now running in the browser.
  • Detaching – The application is detaching from the browser.
  • Detached – The application is now detached from the browser and running outside the browser.
  • DetachedUpdatesAvailable – Same as detached, but it sees that updates are available.
  • DetachedFailed – A detach event failed.
As evidenced in both Figure 1 (running offline) and Figure 6 (running online), the lower portion of the application shows the current state as Detached since the application has been detached. As the events change, the message in the application automatically updates itself using data binding.
For more information on Data Binding in Silverlight, please refer to my September 2008 Data Points column ("Service-Driven Apps With Silverlight 2 And WCF"). In that column, I discuss data binding and binding modes.
An event handler can be set up to listen for these states by handling the ExecutionStateChanged event, as shown below:
Application.Current.ExecutionStateChanged += new EventHandler(Network_ExecutionStateChanged);
When the ExecutionStateChanged event fires with a state of DetachUpdatesAvailable, the Silverlight 3 out-of-browser application has detected that a newer version of the Silverlight 3 application can be downloaded. The user can then close the out-of-browser application and reopen it to automatically receive the update. Figure 7 shows that the message in the out-of-browser application was displayed when the ExecutionStateChanged event fired.
Figure 7 Updates Are Available

Network Connectivity
Yet another new feature of Silverlight 3 is that it can detect whether a network connection is available, thanks to the NetworkInterface.GetIsNetworkAvailable() method in the System.Net namespace. This check is essential when running an out-of-browser application that must communicate with Web services. If the network connection is down, the application can perform a workaround. Some workaround might include notifying the user of the problem or possibly storing the data locally until a network connection becomes available.
The NetworkChange.NetworkAddressChanged event (also part of the System.Net.NetworkInformation class) executes when it detects a network address change. This event can be handled as shown below:
NetworkChange.NetworkAddressChanged +=
  new NetworkAddressChangedEventHandler(NetworkChange_  NetworkAddressChanged);
The code in Figure 8 shows the event handler for the NetworkAddressChange event. The code first sets a Boolean status property of IsNetworkAvailable, which is bound to the Silverlight UI and shows the user the status of the network connectivity. Figure 9 shows that the out-of-browser application has detected that the network address has been lost. The bindings updated the message and the status color to red. The simplest way to test this is to open your network connections dialog and disable all network connections. This will cause the NetworkAddressChanged event to fire and the UI to be updated through the data bindings.
Figure 8 Detecting Network Address Changes
private void NetworkChange_NetworkAddressChanged(object sender, EventArgs e)
{
    stats.IsNetworkAvailable = NetworkInterface.GetIsNetworkAvailable();

    if  (NetworkInterface.GetIsNetworkAvailable())
    {
        // get the stored tweet, try to post it
        if (IsolatedStorageSettings.ApplicationSettings.Contains(TWEET_STORAGE))
        {
            PostTweet(IsolatedStorageSettings.ApplicationSettings[TWEET_STORAGE].ToString());
        }
    }
}
Figure 9 Network Address Unavailable
When the application is running and the network connection is lost, a user might still want to post a message to Twitter. (Keep in mind that even though the network connection may be active, the Twitter service may be down. One way to determine if Twitter is running is to do a ping test against it.) The sample application allows the user to post the message still, but because it cannot be sent to Twitter immediately, the message is stored in Isolated Storage (Silverlight's local persistence store). Figure 8 shows that when the network change is detected, if the network just became available and there is a message in Isolated Storage, the message will be posted to Twitter.
For more information on Isolated Storage in Silverlight, please refer to my February 2009 Data Points column "Syndicated Data And Isolated Storage In Silverlight."

Storing Data Offline
If the application cannot detect a network address and a user tries to post a message, the data can be stored in Isolated Storage. The sample application allows only one message to be stored at a time; however, this could be increased by storing each message separately or inside XML. Figure 10 shows the code that checks whether the network is available. If it is available, the message is sent on to Twitter. If not, the code checks to see if there is already a message in Isolated Storage. If there is, the user is notified that only one message can be stored at a time.
Figure 10 Sending a Message to Twitter
private void PostTweet()
{
    string tweetText = TweetTextBox.Text;
    if (!NetworkInterface.GetIsNetworkAvailable())
    {
        if (IsolatedStorageSettings.ApplicationSettings.Contains(TWEET_STORAGE))
        {
            MessageBox.Show(
                "Network connection is not available and only 1 tweet" +
                " can be stored for delayed posting to Twitter.");
        }
        else
        {
            IsolatedStorageSettings.ApplicationSettings.Add(TWEET_STORAGE, tweetText);
            MessageBox.Show(
                "Network connection is not available. The post" +
                " will be stored until connectivity returns.");
        }
        return;
    }

    PostTweet(tweetText);
}
Out-of-browser applications automatically get 25MB of space available in Isolated Storage. Online browser-hosted Silverlight applications get 1MB of storage. Additional storage can be requested through the IsolatedStorageFile.IncreaseQuoteTo API, which will prompt the user to allow or deny the quota increase.
If there is no message in Isolated Storage, the user's new message is stored and the user is notified that the message will be sent when the network connection is reestablished. When the network address is reestablished, the NetworkAddressChanged event handler will run and will pull the message out of Isolated Storage and submit it (as shown in Figure 8).
As You See, Silverlight 3's out-of-browser solution allows Silverlight applications to run on the desktop without a browser and without a network connection. It has the plumbing to detect network connectivity, to take action when network events occur, and to store data locally if needed. Give it a try. You're sure to come up with some great uses for these new features!

Send your questions and comments for John to mmdata@microsoft.com.

John Papa (johnpapa.net) is a Senior Consultant with ASPSOFT and a baseball fan who spends summer nights rooting for the Yankees with his family. John, a C# MVP, Silverlight Insider, and INETA speaker, has authored several books, including his latest titled Data-Driven Services with Silverlight 2 (O'Reilly, 2009). He often speaks at conferences such as Mix, DevConnections, and VSLive.


Cutting Edge
Go Beyond HTML Forms With AJAX
Dino Esposito
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Popup Forms
Just-in-Time Validation
Submission Throttling
Auto-Saving Forms
AJAX And Beyond
Beyond the hype of improved user experience, more interactivity, reduced page flickering, quicker data transfer and all the other improvements AJAX promises, the key change that AJAX brought to the table is that it allows you total control over the submission of data to the server.
In a pure AJAX application, any browser-to-server data submission passes through the developer's code. As the developer, you decide how to post data and when. Furthermore, you can even decide whether or not a form should be used.
To gain such a level of control over the data submission process in a browser-based application, you're required to write a lot of JavaScript code. In fact, in previous installments of this column, I discussed the main features of a very popular JavaScript library—the jQuery library—that can offer you a great deal of help when writing serious JavaScript code.
This month, I'll examine a few scenarios involving forms in the context of AJAX applications. I'll examine various approaches to implementing features such as auto-saving, just-in-time validation, and submission throttling. Let's begin with popup forms.

Popup Forms
Last month I presented a subset of the jQuery library, the jQuery UI Library, which was designed specifically to simplify the building of rich user interfaces through modal and modeless windows and advanced and powerful widgets such as accordions and tabs.
In a Web context, however, a popup form may not be the best solution for displaying data that results from drill-down operations. The popup window is normally created using the browser API—typically, the window.open method—and is subject to the popup blocker policy that the user has in place for that site. The following code snippet shows how to force the browser to display a popup window:
window.open(url, 'Sample', 'width=400,height=300,scrollbars=yes');
The popup will display the page specified in the URL and can post data to any other page in the application. You can pass data to the popup using the query string; as an alternative, you can retrieve input fields in the parent form through the opener property of the window object. You call the window object from within the page displayed in the popup, like so:
var text = window.opener.document.Form1["TextBox1"].value;
From the opener property you access the object model of the opener form and read the content of a particular form via the HTML DOM.
The abuse of pop-up and pop-under forms in Web advertising led browser developers to introduce features to control the proliferation of such windows. As a result, even the legitimate use of popups to convey information to the user is losing appeal and becoming nearly impossible.
Today, popup forms in Web applications are mostly topmost DIV content displayed in a modal way, after disabling the ability to click on elements underneath. The user interface of the popup doesn't necessarily belong to a separate page; more often than not, it is just part of the page that displays it. It can be an initially hidden DIV tag that becomes visible when the proper script code is run.
From such a pseudo popup window, you can easily access any element in the page because the content of the pseudo popup window is already part of the page. Initializing input elements, therefore, is no longer an issue.
Powerful libraries such as the jQuery UI also offer you facilities to control the look of such popups to make them look like classic Windows dialog boxes. (See Figure 1.)
The markup you place in the popup may or may not include a form tag. For sure, it cannot include any additional server-side form elements as long as the host page is an ASP.NET page. An HTML form element (devoid of the runat=server attribute) can be inserted whether the page is classic ASP.NET or ASP.NET MVC. If the popup contains a form tag, then you can take advantage of the browser's submission capabilities to post data to the action URL of choice. Otherwise, posting is still possible, but you must write the code.
Figure 1 A Sample Dialog Box Created with jQuery UI 1.7
You can loop through input fields, prepare the body of an HTTP request, and post it using the XMLHttpRequest object directly or any abstraction of it (such as the $.ajax function in jQuery or the Sys.Net.WebRequest object in the Microsoft AJAX client library.)
Finally, to keep the content of the popup physically separated from the rest of the ASP.NET page, you might want to define it within an ASCX user control, like so:
<div id="DialogBox1">
  <cc1:YourUserControl runat="server"     ID="DialogBox1_Content" />
</div>
This approach can be taken in both classic ASP.NET and in ASP.NET MVC pages.

Just-in-Time Validation
It is fairly common that a Web form is required to notify the user of any incorrect values before submission. Notifying the user of improper input is a job that validator controls in ASP.NET do quite well. The ValidatorCalloutExtender in the AJAX Control Toolkit then can also make invalid input warnings attractive, as you see in Figure 2.
Figure 2 Validator Extenders in a UI
ASP.NET validator controls work by validating the entire contents of the form when it is submitted. In the current version of ASP.NET, there's no asynchronous custom validator that enables client-side code to validate fields against a (custom) server rule. Such a component, though, may be released in future versions of ASP.NET.
In summary, to detect and report incorrect values to the user you have two possible approaches. First, you may associate an asynchronous validator with any input field and report error messages as the value of the field changes. Second, you may validate the contents of the form as a whole. This second approach may be implemented in a couple of different ways: you can simply submit the form in the usual way via the browser and wait for the server response, or you can asynchronously and programmatically submit the contents of the form, even partial contents, to have early feedback. Let's review the various approaches and see how to take advantage of AJAX capabilities.
In ASP.NET, there are a number of predefined validator controls that apply fixed rules to the field contents and determine whether the input is valid or not. Sample validators perform checks such as whether a field is left empty, a date falls within a given range, or some content equals a given value or matches a given regular expression. All these checks cover common validation needs. However, the most common check must be performed on the server and requires a database connection, a workflow, or perhaps a Web service. For all these cases, ASP.NET provides the CustomValidator control.
CustomValidator is an excellent control that allows you to specify both a client JavaScript function and a server method to check the value and report feedback to the UI level. Properly wrapped up in an AJAX update-panel, the CustomValidator control can also return its response asynchronously, avoiding a full-page refresh. The only drawback of the CustomValidator is that, just like any other validator, it won't check the value until the form is submitted. Let's consider the code in Figure 3.
Figure 3 An Update Panel Implementation
<asp:UpdatePanel runat="server" ID="UpdatePanel1">
  <ContentTemplate>
    <asp:TextBox ID="TextBox1" runat="server"
                 AutoPostBack="true" />
    <asp:CustomValidator ID="CustomValidator1" runat="server"
                 Display="Dynamic"
                 ControlToValidate="TextBox1"
                 SetFocusOnError="true"
                 ErrorMessage="Invalid ID"
                 OnServerValidate=" CustomValidator1_Validate" >
   </asp:CustomValidator>
 </ContentTemplate>
</asp:UpdatePanel>
<asp:Button ID="Button1" runat="server" Text="Submit" />
The page posts back as the user edits the contents of the text box. A CustomValidator instance is associated with the text box and knows how to validate the content. The update-panel ensures that a postback will generate only a partial refresh. Unfortunately, this code doesn't produce the expected results. The page posts back as the text box is edited, but no validation ever occurs. To trigger the custom validator, the user needs to push the submit button.
The problem is that no code path leads to the invocation of the Validate method on the Page class outside the classic form submission process. To remedy this, you must check whether an asynchronous postback is happening and invoke Validate yourself, as shown below:
protected void Page_Load(object sender,     EventArgs e)
{
    if (ScriptManager1.IsInAsyncPostBack)
    {
        this.Validate();
    }
}
The Validate method, in turn, will loop through any validators in the page and report about any incorrect values in the page.
If your goal is to report input errors as soon as possible, the only alternative to the previous approach is to add some JavaScript code to critical input fields. Here's an example:
<asp:TextBox ID="TextBox1" runat="server"
              onchange="__doValidation(this)" />
In a textbox, the onchange event is not fired while the user is typing, but only when the user tabs out of the text box and implicitly commits the changes. In the preceding code snippet, a JavaScript handler is executed to perform a remote validation via XMLHttpRequest. Here's the code for the JavaScript handler:
<script type="text/javascript">
    function __doValidation(textBox) {
        var text = textBox.value;
        PageMethods.PerformValidation(text, onComplete);
    }

    function onComplete(results) {
        if (results == false)
            $get("Label1").innerHTML = "Invalid ID";
        else
            $get("Label1").innerHTML = "";
    } </script>
As you can see, the example uses an ASP.NET page method to define the HTTP endpoint to be invoked. Any such endpoint is fine, including a WCF service method, a Web service method, or a REST service. The response you get from the HTTP endpoint may be either a Boolean value or a piece of HTML markup. In the former case, the logic that updates the user interface lives on the client (and it may be really simple); in the latter case, more sophisticated logic is required on the server.
If you don't need to validate input fields individually and can wait for feedback until the entire form is submitted, then the simplest thing you can do is use validators within an ASP.NET UpdatePanel control. In this case, keep in mind that both validators and input controls must be placed in the same panel.
As an alternative to partial rendering for posting and validating the contents of a form, you can consider an AJAX library such as jQuery in combination with the Form and Validation plugin. An example is discussed in the article "Combining JQuery Form Validation and Ajax Submission with ASP.NET."
Overall, I believe that individual checks are preferable because you catch incorrect values immediately and you can adjust the form dynamically to reflect current input. For the time being, you'll need to use JavaScript or perhaps do partial rendering with auto-postback controls.
If you intend to validate the entire form asynchronously, I still believe that using classic ASP.NET validators with partial rendering is the simplest option. Equally effective is any solution that serializes the content of the form to a string and posts that to an HTTP endpoint using the AJAX library of choice. As you'll see in a moment, custom serialization and posting of a form provides for more advanced features.

Submission Throttling
With AJAX, you no longer have just one way to post the contents of a form to the Web server. You can certainly do it explicitly through a submit button, or programmatically via the submit method of the HTML DOM Form object. More importantly, though, you can consider submitting content to the server in chunks and in multiple steps rather than all at once. Pattern-wise, if you opt for sending all the content of a form at once, then you are applying the "Explicit Submission" pattern; otherwise, you are throttling your submission via the "Submission Throttling" (ST) pattern.
The ST pattern proves helpful for smoothing the impact of client operations that put pressure on the server, including operations that result in frequent round-trips. This makes the ST pattern the key strategy for implementing nice features such as auto-saving forms.
Simply put, the ST pattern suggests you cache the content to submit in a browser-side buffer and submit it piecemeal, in multiple steps. The canonical example is the auto-completion engine. As the user types into a textbox, a request is made to the server for suggestions. It's easy to figure out that the number of requests that hit the Web server at peak times is quite high.
To reach a reasonable compromise between responsiveness of the application and Web server workload, you might want to take control over the submission process. To understand throttling, let's examine the source code of the Microsoft's AutoComplete extender, which you'll find in the AJAX Control Toolkit.
The auto-complete extender sends a request for suggestions as the user types in the input box. Based on this, one would expect a handler for the keypress event that calls the specified Web service. However, a closer look at the source code reveals that it works differently, as Figure 4 illustrates.
Figure 4 Postbacks Controlled by a Timer
// Handler for the textbox keydown event
onKeyDown: function(ev)
{
    // A key has been pressed, so let's reset the timer
    this._timer.set_enabled(false);

    // Is it a special key?
    if (k === Sys.UI.Key.esc) {
       ...
    }
    else if (k === Sys.UI.Key.up) {
       ...
    }
       ...
    else {
       // It's a text key, let's start the timer again
       this._timer.set_enabled(true);
    }
}
The extender uses an internal timer configured to an interval of one second (by default). The timer is started when the text box gets the input focus. As soon as a key is pressed down, the timer is stopped and an analysis of the key begins. If the key is one with a special meaning, such as Enter, ESC, Tab, or an arrow key, then the extender proceeds with its own code. If the key pressed indicates content to be entered in the text box buffer, then the timer is started.
The timer is stopped and restarted every time an actual key is typed. If you stop typing for one second (or whatever interval you configure), then on the tick of the timer, the actual content of the input field is submitted to receive suggestions. It's not a big change for users, but it's great news for the server because no postback occurs until the user types. If the user takes a one-second break, the timer expires and the postback is then made.
Timers are a powerful tool in JavaScript because their inherent asynchronous behavior can be leveraged to simulate multithreading. By creating a timer in JavaScript, you are telling the interpreter to run the associated code as soon as possible, but not immediately. If the code is simple enough, and the timer interval appropriate, then the interpreter will be able to schedule execution properly, thus advancing multiple tasks simultaneously.

Auto-Saving Forms
It is not unusual in AJAX applications that the user spends a lot of time working on a single form, but without posting any data to the Web server for a while. The Web server has no way to figure out what's happening on the client. Subsequently, if it doesn't receive requests for a time, it may just have the server session time out. Therefore, when the user finally posts the completed form back to the server, she will get an unpleasant surprise (a timeout message).
To avoid that, the client application sends a "heartbeat" message via XMLHttpRequest at an appropriate frequency. A heartbeat message can take any form and content that is suitable for the particular scenario. For example, it can be an empty request that just shows that the client is still working and has the sole objective of keeping the server session up and running. (In this case, the interval must be of any length that is shorter than the session timeout configured for the application.)
The heartbeat may even contain the current, possibly partial, contents of the form. Such contents are uploaded periodically and cached on the server waiting for the proper time to process it. The overall idea is not much different from the auto-save feature of Microsoft Office Word, where the open document periodically saves back to disk to avoid loss of data.
Over the Web, auto-saving forms have a much less defined behavior, and their internal implementation is largely left to the creativity and needs of the developer. Auto-saving forms are essentially forms endowed with the ability to periodically post their contents to a URL. The receiving URL will then handle data and typically cache it for the current user. Figure 5 shows an example.
Figure 5 Implementation of an Auto-Saving Form
<script type="text/javascript">
    var _myTimerID = null;
    var _userID = null;
    function enableAutoSave(id) {
        _userID = id;
        _myTimerID = window.setInterval(__doAutoSave, 10000);
    }

    function __doAutoSave() {
        var body = "ID='" + _userID + "'&" +
                   "TextBox2='" + $get("TextBox2").value + "'&" +
                   "TextBox3='" + $get("TextBox3").value + "'&" +
                   "TextBox4='" + $get("TextBox4").value + "'";

        PageMethods.AutoSave(_userID, body, onComplete);
    }

    function onComplete(results) {
        $get("Msg").innerHTML = "<i>Form auto-saved for user '" +
              _userID + "' at " + results + ".</i>";
    }
</script>

<body>
    ...
</body>

// In the code-behind class ...
protected void Button1_Click(object sender, EventArgs e)
{
    string user = TextBox1.Text;

    // Populate input fields if any previously entered
    // data is available
    string body = AutoSaveAPI.Read(user);
    if (!String.IsNullOrEmpty(body))
    {
        Dictionary<string, object> data = body.ParseToDictionary();
        TextBox2.Text = data["TextBox2"] as string;
        TextBox3.Text = data["TextBox3"] as string;
        TextBox4.Text = data["TextBox4"] as string;
    }

    // Re-enable auto-save for this editing session
    string script = "enableAutoSave('" + user + "');";
    ScriptManager.RegisterStartupScript(this, this.GetType(),
                  "myTimer", script, true);
}

[WebMethod]
public static string AutoSave(string id, string body)
{
    AutoSaveAPI.Save(id, body);
    return DateTime.Now.ToString();
}
Once the user navigates to the form, the codebehind class loads any data previously entered for that form. Typically, the data is read from the cache or from any other source of choice. In Figure 5, the storage is hidden behind the interface of the AutoSaveAPI class. Any information read is used to populate the input fields in the form.
To enable auto-saving in a new data entry session, you emit a line of script that triggers a timer. When the timer ticks, the callback collects the current values in the various fields and builds a string. The string is then uploaded using XMLHttpRequest. You can upload it to a Web service, a plain URL using a REST approach, or even to a page method. Because an auto-saving form is a specific component of a given page, using a page method is the ideal solution because it is page-limited, efficient, and easy to code. Figure 6 shows an auto-saving form in action.
Figure 6 An Auto-Saving Form in Action

AJAX And Beyond
With AJAX, the old HTML form tag needs some revision. With the upcoming HTML 5.0 standard, a new set of inputs including URL, date, and e-mail will be handled natively by browsers. This will reduce the number of cases for which scripting and validators are required. This evolution of HTML will let you focus on server-side validation, which must be asynchronous in AJAX applications.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an architect at IDesign and the co-author of Microsoft .NET: Architecting Applications for the Enterprise (Microsoft Press, 2008). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Patterns in Practice
The Unit Of Work Pattern And Persistence Ignorance
Jeremy Miller
Contents
The Unit of Work Pattern
Using the Unit of Work
Persistence Ignorance
Can the Business Logic Run Independently of the Database?
Can I Design My Domain Model Independently from the Database Model?
How Does My Persistence Strategy Affect My Business Logic?
More Units of Work
In the April 2009 issue of MSDN Magazine ("Persistence Patterns") I presented some common patterns that you will encounter when using some sort of Object/Relational Mapping (O/RM) technology to persist business entity objects. I think it's unlikely that you or your team will be writing your own O/RM tooling from scratch, but these patterns are important to know to effectively use (or even just to choose) existing tooling.
In this article, I would like to continue the discussion of persistence patterns with the Unit of Work design pattern and examine the issues around persistence ignorance. Throughout most of this article, I will be using a generic invoicing system as the example problem domain.

The Unit of Work Pattern
One of the most common design patterns in enterprise software development is the Unit of Work. According to Martin Fowler, the Unit of Work pattern "maintains a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems."
The Unit of Work pattern isn't necessarily something that you will explicitly build yourself, but the pattern shows up in almost every persistence tool that I'm aware of. The ITransaction interface in NHibernate, the DataContext class in LINQ to SQL, and the ObjectContext class in the Entity Framework are all examples of a Unit of Work. For that matter, the venerable DataSet can be used as a Unit of Work.
Other times, you may want to write your own application-specific Unit of Work interface or class that wraps the inner Unit of Work from your persistence tool. You may do this for a number of reasons. You might want to add application-specific logging, tracing, or error handling to transaction management. Perhaps you want to encapsulate the specifics of your persistence tooling from the rest of the application. You might want this extra encapsulation to make it easier to swap out persistence technologies later. Or you might want to promote testability in your system. Many of the built-in Unit of Work implementations from common persistence tools are difficult to deal with in automated unit testing scenarios.
If you were to build your own Unit of Work implementation, it would probably look something like this interface:
public interface IUnitOfWork {
  void MarkDirty(object entity);
  void MarkNew(object entity);
  void MarkDeleted(object entity);
  void Commit();
  void Rollback();
}
Your Unit of Work class will have methods to mark entities as changed, new, or deleted. (In many implementations the explicit call to MarkDirty would be unnecessary because the Unit of Work itself has some way of automatically determining which entities have been changed.) The Unit of Work will also have methods to commit or roll back all of the changes.
In a way, you can think of the Unit of Work as a place to dump all transaction-handling code. The responsibilities of the Unit of Work are to:
  • Manage transactions.
  • Order the database inserts, deletes, and updates.
  • Prevent duplicate updates. Inside a single usage of a Unit of Work object, different parts of the code may mark the same Invoice object as changed, but the Unit of Work class will only issue a single UPDATE command to the database.
The value of using a Unit of Work pattern is to free the rest of your code from these concerns so that you can otherwise concentrate on business logic.

Using the Unit of Work
One of the best ways to use the Unit of Work pattern is to allow disparate classes and services to take part in a single logical transaction. The key point here is that you want the disparate classes and services to remain ignorant of each other while being able to enlist in a single transaction. Traditionally, you've been able to do this by using transaction coordinators like MTS/COM+ or the newer System.Transactions namespace. Personally, I prefer using the Unit of Work pattern to allow unrelated classes and services to take part in a logical transaction because I think it makes the code more explicit, easier to understand, and simpler to unit test.
Let's say that your new invoicing system performs discrete actions upon existing invoices at various times in the invoice lifecycle. The business changes these actions fairly often and you'll frequently want to add or remove new invoice actions, so let's apply the Command pattern (see "Simplify Distributed System Design Using the Command Pattern, MSMQ, and .NET") and create an interface called IInvoiceCommand that represents a single discrete action against an Invoice:
public interface IInvoiceCommand {
  void Execute(Invoice invoice, IUnitOfWork unitOfWork);
}
The IInvoiceCommand interface has one simple Execute method that is called to perform some type of action using an Invoice and an IUnitOfWork object. Any IInvoiceCommand object should use the IUnitOfWork argument to persist any changes back to the database within that logical transaction.
Simple enough, but this Command pattern plus Unit of Work pattern doesn't get interesting until you put several IInvoiceCommand objects together (see Figure 1).
Figure 1 Using IInvoiceCommand
public class InvoiceCommandProcessor {
  private readonly IInvoiceCommand[] _commands;
  private readonly IUnitOfWorkFactory _unitOfWorkFactory;

  public InvoiceCommandProcessor(IInvoiceCommand[] commands,
    IUnitOfWorkFactory unitOfWorkFactory) {

    _commands = commands;
    _unitOfWorkFactory = unitOfWorkFactory;
  }

  public void RunCommands(Invoice invoice) {
    IUnitOfWork unitOfWork = _unitOfWorkFactory.StartNew();

    try {
      // Each command will potentially add new objects
      // to the Unit of Work for insert, update, or delete
      foreach (IInvoiceCommand command in _commands) {
        command.Execute(invoice, unitOfWork);
      }

      unitOfWork.Commit();
    }
    catch (Exception) {
      unitOfWork.Rollback();
    }
  }
}
Using this approach with the Unit of Work, you can happily mix and match different implementations of the IInvoiceCommand to add or remove business rules in the invoicing system while still maintaining transactional integrity.
In my experience, business people seem to care quite a bit about late, unpaid invoices, so you're probably going to have to build a new IInvoiceCommand class that will alert the company's agents when an Invoice is determined to be late. Here is a possible implementation of this rule:
public class LateInvoiceAlertCommand : IInvoiceCommand {
  public void Execute(Invoice invoice, IUnitOfWork unitOfWork) {
    bool isLate = isTheInvoiceLate(invoice);
    if (!isLate) return;

    AgentAlert alert = createLateAlertFor(invoice);
    unitOfWork.MarkNew(alert);
  }
}
The beauty of this design to me is that the LateInvoiceAlertCommand can be completely developed and tested independently of the database or even the other IInvoiceCommand objects involved in the same transaction. First, to test the interaction of the IInvoiceCommand objects with a Unit of Work, I might create a fake implementation of IUnitOfWork strictly for testing to be precise, I would call the StubUnitOfWork a recording stub:
public class StubUnitOfWork : IUnitOfWork {
  public bool WasCommitted;
  public bool WasRolledback;
  public void MarkDirty(object entity) {
    throw new System.NotImplementedException();
  }
  public ArrayList NewObjects = new ArrayList();
  public void MarkNew(object entity) {
    NewObjects.Add(entity);
  }
}
Now that you've got a nice fake Unit of Work that runs independently of the database, the test fixture for the LateInvoiceAlertCommand might look something like the code in Figure 2.
Figure 2 Test Fixture for LateInvoiceAlertCommand
[TestFixture]
public class
  when_creating_an_alert_for_an_invoice_that_is_more_than_45_days_old {

  private StubUnitOfWork theUnitOfWork;
  private Invoice theLateInvoice;

  [SetUp]
  public void SetUp() {
    // We're going to test against a "Fake" IUnitOfWork that
    // just records what is done to it
    theUnitOfWork = new StubUnitOfWork();

    // If we have an Invoice that is older than 45 days and NOT completed
    theLateInvoice = new Invoice {InvoiceDate =
      DateTime.Today.AddDays(-50), Completed = false};

      // Exercise the LateInvoiceAlertCommand against the test Invoice
      new LateInvoiceAlertCommand().Execute(theLateInvoice, theUnitOfWork);
  }

  [Test]
  public void
    the_command_should_create_a_new_AgentAlert_with_the_UnitOfWork() {

    // just verify that there is a new AgentAlert object
    // registered with the Unit of Work
    theUnitOfWork.NewObjects[0].ShouldBeOfType<AgentAlert>();
  }

  [Test]
  public void the_new_AgentAlert_should_have_XXXXXXXXXXXXX() {
    var alert = theUnitOfWork.NewObjects[0].ShouldBeOfType<AgentAlert>();
    // verify the actual properties of the new AgentAlert object
    // for correctness
  }
}

Persistence Ignorance
When I choose or design a persistence solution for one of my projects, I'm mindful of the impact of the persistence infrastructure on my business logic code—at least in a system with a great deal of business domain logic. Ideally, I'd like to design, build, and test my business logic relatively independently of the database and the persistence infrastructure code. Specifically, I want a solution that supports the idea of persistence ignorance or Plain Old CLR Objects (POCOs).
First, what is persistence ignorance, and where did it come from? In his book Applying Domain-Driven Design and Patterns: With Examples in C# and .NET (Pearson Education, Inc., 2006), Jimmy Nilsson defines POCOs as "…ordinary classes where you focus on the business problem at hand without adding stuff for infrastructure-related reasons. ... The classes should focus on the business problem at hand. Nothing else should be in the classes in the Domain Model."
Now, why should you care? Let's start the discussion by being very clear that persistence ignorance is only a means to various design goals and not an end goal in and of itself. When I evaluate a new persistence tool, I'm generally asking myself the following questions in descending order of importance. Persistence ignorance is not completely required to satisfy these questions, but a solution based on persistence ignorance generally does better than solutions that require infrastructure concerns to be embedded in business objects.

Can the Business Logic Run Independently of the Database?
This is the most important question to me. I think that one of the most important factors in the success of any software project is the ability to use rapid feedback cycles. In other words, I want to shorten the time and effort in between "I just coded something new" and "I have proved that the new code is working so I will go on to something else" or "the new code has a problem, so I will fix it right now."
In my career, I have consistently observed teams to be much more productive in architectures where it is easy to unit test or even just exercise little pieces of new code without running through the entire application. Conversely, I've found architectures that have a tight run time coupling between business logic and infrastructure to be very difficult to develop in.
Let's reconsider the case of the business rule that says you should create a new Agent Alert when you encounter an open invoice older than 45 days. At some later point, the business may decide that you should create alerts at 30 days instead. In order to verify changes to the invoicing alert logic, you have to first put the code into a state that exercises this rule against open and closed invoices and invoices older or newer than the 30-day threshold. This is the point where the architecture starts to matter. Can I walk right up to the new invoicing alert logic and unit test a new business rule, or do I have to jump through technical hoops first?
In terms of a rapid feedback cycle, the worst extreme is a model where the business logic cannot function without the presence of infrastructure. For example, the first homegrown O/RM I created forced you to write Entity classes like this:
public class Invoice : MySpecialEntityType {
  private long _id;
  public Invoice(long id) {
    _id = id;
    // Read in the rest of the Invoice information
    // from the database
    loadDataFromDatabase(id);
  }
  public Invoice() {
    // Hit the database
    _id = fetchNextId();
  }
}
With this design, you literally cannot create an Invoice class without being connected to a database. In my original design, there was no easy way to use or exercise my business entity classes without having the database engine configured correctly and available.
It is certainly possible to write automated tests against code that is accessing the database, but it generally requires significantly more effort in developer time to set up test data than to write automated tests against objects that do not depend on a database. To set up a simple invoice in the database, you would have to contend with non-null fields and data to satisfy referential integrity requirements that do not pertain to the business rule in question.
Instead, wouldn't it be nicer just to say "if I have an open invoice that is 31 days old, then…" in code and be done with it? To that end, how about moving the Invoice class to more of a persistence-ignorant approach like this:
// The Invoice class does NOT depend on
// any sort of persistence infrastructure
public class Invoice {
  private long _id;

  public Invoice() { }

  public bool IsOpen { get; set; }
  public DateTime? InvoiceDate { get; set; }
}
Now, when you want to test the late invoicing alert rule, you can quickly create an open invoice that is 31 days old for the testing scenario with simple code in the test fixture like this:
[SetUp]
public void SetUp() {
  Invoice theLateInvoice = new Invoice() {
    InvoiceDate = DateTime.Today.AddDays(-31),
    IsOpen = true
  };
}
In this approach, I've separated the business logic code for the late invoice rule and the Invoice class itself away from the persistence code. As long as you can easily create invoice data in memory, you can rapidly unit test the business rules.
As an aside, when you're choosing an existing persistence tool, be cautious of the way that lazy loading is implemented. Some tools will implement transparent lazy loading with the Virtual Proxy pattern to make lazy loading effectively invisible to the business entities themselves. Other tools rely on code generation techniques that embed the lazy-loading support directly into the business entities and effectively create a tight run time coupling from the entities to the persistence infrastructure.
Almost as important as the time it takes for developers to author automated tests is the speed with which these tests execute. Automated tests involving data access or Web service access can easily run an order of magnitude or more slowly than tests that run completely within a single AppDomain. This may not seem like much of an issue, but as the size of the project begins to grow over time, slow-running automated tests can easily detract from a team's productivity and even eliminate the usefulness of the automated tests in the first place.

Can I Design My Domain Model Independently from the Database Model?
Now that I've decoupled my business layer from the database at run time, I'm faced with the issue of whether I can design my object structure independently of the database schema (and vice versa). I should be able to design my object model based on the necessary behavior of the business logic. The database, on the other hand, should be designed with efficiency of reading and writing data as well as enforcing referential integrity. (By the way, referential integrity is essential for using an O/RM tool.)
To demonstrate a typical scenario where the domain model can diverge from the database structure, let's switch the example domain to an energy trading system. This system is responsible for tracking and pricing quantities of petroleum sold, bought, and delivered. In a simplistic system, a trade may include several quantities purchased and a record of the quantities delivered. Here's the rub, though: quantity means unit of measure plus the number of those units. For this trading system, let's say that you track quantities in three different units:
public enum UnitOfMeasure {
  Barrels,
  Tons,
  MetricTonnes
}
If you were to lock the domain model to the flat structure of the database schema, you might get classes like this:
public class FlatTradeDetail {
  public UnitOfMeasure PurchasedUnitOfMeasure {get;set;}
  public double PurchasedAmount {get;set;}
  public UnitOfMeasure DeliveredUnitOfMeasure { get; set; }
  public double DeliveredAmount { get; set; }
}
While the structure is suitable for the database structure and convenient for solutions where you generate the code structure from an existing database, this structure does not lend itself to carrying out the business logic.
A large part of the business logic in the energy trading system requires quantities to be compared, subtracted, and added—but you must always assume that the units of measure are different and do the conversions from one to another before making any comparison. From painful experience I can state that using the flat structure generated directly out of the database schema makes this logic laborious to implement.
Instead of the flat structure, let's implement the Money pattern and create a class that models the behavior of a Quantity, as shown in Figure 3.
Figure 3 Quantity
public class Quantity {
  private readonly UnitOfMeasure _uom;
  private readonly double _amount;

  public Quantity(UnitOfMeasure uom, double amount) {
    _uom = uom;
    _amount = amount;
  }

  public Quantity ConvertTo(UnitOfMeasure uom) {
    // return a new Quantity that represents
    // the equivalent amount in the new
    // Unit of Measure
  }

  public Quantity Subtract(Quantity other) {
    double newAmount = _amount - other.ConvertTo(_uom).Amount;
    return new Quantity(_uom, newAmount);
  }

  public UnitOfMeasure Uom {
    get { return _uom; }
  }

  public double Amount {
    get { return _amount; }
  }
}
When you use the Quantity class to model and reuse common behavior around unit of measure quantities, the TradeDetail class might look more like this:
public class TradeDetail     {
  private Quantity _purchasedQuantity;
  private Quantity _deliveredQuantity;
  public Quantity Available() {
    return _purchasedQuantity.Subtract(_deliveredQuantity);
  }
  public bool CanDeliver(Quantity requested) {
    return Available().IsGreaterThan(requested);
  }
}
The Quantity class will definitely make the TradeDetail logic easier to implement, but now the object model has varied from the database structure. Ideally, your persistence tooling should support this sort of mapping.
The issue of having variance between an object model and a database model generally does not arise on systems with simple business logic. In those systems, an Active Record architecture that simply generates the entity classes from the database structure may be the easiest solution. Alternatively, you may use a database mapper tool that allows you to generate a database schema from the entity objects.

How Does My Persistence Strategy Affect My Business Logic?
It's not a perfect world. Any persistence tooling will have some sort of impact on how you shape your entity classes. For example, my team uses NHibernate for persistence. Due to the way that NHibernate implements lazy-loaded property relationships, many of our properties have to be marked as virtual just to enable lazy loading like this new Customer property on Invoice:
public class Invoice {
  public virtual Customer Customer { get; set; }
}
The Customer property is marked as Virtual for no other reason than to enable NHibernate to create a dynamic proxy for the Invoice object that provides a lazy-loaded Customer property
Marking members as virtual strictly to support lazy loading is an annoyance, but there are worse potential problems. Increasing numbers of teams are using Domain Driven Design (DDD). One of the common strategies in DDD is to create domain model classes that cannot be placed into an invalid state. The domain model classes themselves would also jealously guard their internal data to enforce business rules internally. To enact this design philosophy, an Invoice class might look more like Figure 4.
Figure 4 Domain Model Invoice Class
public class Invoice {
  private readonly DateTime _invoiceDate;
  private readonly Customer _customer;
  private bool _isOpen;

  // An Invoice must always have an Invoice Date and
  // a Customer, so there is NO default constructor
  // with no arguments
  public Invoice(DateTime invoiceDate, Customer customer) {
    _invoiceDate = invoiceDate;
    _customer = customer;
  }

  public void AddDetail(InvoiceDetailMessage detail) {
    // determines whether or not, and how, a
    // new Invoice Detail should be created
    // and added to this Invoice
  }

  public CloseInvoiceResponse Close(CloseInvoiceRequest request) {
    // Invoice itself will determine if the correct
    // conditions are met to close itself.
    // The _isOpen field can only be set by
    // Invoice itself
  }

  public bool IsOpen {
    get {
      return _isOpen;
    }
  }

  public DateTime InvoiceDate {
    get { return _invoiceDate; }
  }

  public Customer Customer {
    get { return _customer; }
  }
}
By no means is this approach to crafting a domain model applicable to every type of application, but if you do choose this style of architecture it will affect your choice of persistence tooling.
The version of Invoice in Figure 4 has only one constructor function that enforces the rule that no Invoice can be created without an Invoice Date and a Customer. Many, if not most, persistence technologies require a no-argument constructor function.
Also, this version of the Invoice class jealously guards internal fields like _isOpen without public setter properties. Again, many persistence tools can work only against public properties. If you want to use the more strict DDD style of entities, you will need to research whether your persistence tooling can support mapping fields, private properties, or constructor functions with arguments.

More Units of Work
There are some other important issues with using the Unit of Work pattern that are worth considering. If you are interested in using the Unit of Work pattern explicitly in your application, these issues should be researched in the context of your application:
  • The division of responsibility between your Repositories and the Unit of Work. You may say that all reads happen through Repositories and writes through the Unit of Work. You may also use a Unit of Work implementation that forces you to do reads and queries through the Unit of Work to make state change tracking easier.
  • How to deliver the correct Unit of Work and the underlying Identity Map helpers to the various classes taking part in a logical transaction. Many people use an Inversion of Control container to correctly attach the Unit of Work for the current HttpContext, thread, or other scoping strategy.
Persistence ignorance is somewhat controversial in the .NET community and its importance is heavily disputed. At this time, persistence ignorance is not well supported in the persistence tooling available in the Microsoft .NET Framework itself or in the extended .NET ecosystem. NHibernate is the one tool that comes closest, but even with NHibernate you will have to compromise your domain model somewhat to enable persistence.

Send your questions and comments to mmpatt@microsoft.com.

Jeremy Miller, a Microsoft MVP for C#, is also the author of the open-source StructureMap tool for Dependency Injection with .NET and the forthcoming StoryTeller tool for supercharged FIT testing in .NET. Visit his blog, "The Shade Tree Developer," part of the CodeBetter site.


Test Run
.NET Module Testing with IronPython
James McCaffrey
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
The Module Under Test
Ad Hoc Interactive Module Testing
Lightweight Module Test Automation
Wrapping Up
The Python scripting language has features that make it an excellent choice for performing several types of software testing. There are several implementations of Python available including CPython, the most common implementation on machines running UNIX-like operating systems, and IronPython, an implementation released in late 2006 that targets the Microsoft .NET Framework. In this month's column I'll demonstrate how to use IronPython to test .NET-based modules from both the Python command line and in lightweight Python scripts. I assume you have some experience with a scripting language such as JavaScript, Windows PowerShell, VBScript, Perl, PHP, or Ruby, but I do not assume you have any experience with Python. Take a look at Figure 1 to get an idea of where I'm headed. I am using IronPython to perform a quick, ad hoc test of a method that compares two card hands in a .NET module named TwoCardPokerLib. I describe the commands shown in Figure 1 in detail later in this column, but for now observe that I interactively add a reference to a DLL that houses my .NET module, instantiate two Hand objects from that module, and call a Compare method to determine if one hand beats the other hand. Now take a quick look at Figure 2. That screenshot shows the execution of a lightweight IronPython script that took me only minutes to create. Again, I go over all the details later, but you can see that I perform classic module testing by reading test case data from a text file, instantiate two Hand objects from the TwoCardPokerLib module, call the Compare method, and then compare actual results with expected results to determine test case pass/fail results.
Figure 1 Ad Hoc Testing with IronPython on the Command Line
>>> import sys
>>> sys.path
['C:\\IronPython', 'C:\\IronPython\\Lib']
>>> sys.path.append(r'C:\ModuleTestingWithPython\TwoCardPokerLib\bin\  Debug')
>>> sys.path
['C:\\IronPython', 'C:\\IronPython\\Lib', 'C:\\ModuleTestingWithPython\\  TwoCardPokerLib\\bin\\Debug']
>>>
>>> import clr
>>> dir()
['_', '__builtins__', '__doc__', '__name__', 'clr', 'site', 'sys']
>>>
>>> clr.AddReferenceToFile("TwoCardPokerLib.dll")
>>> from TwoCardPokerLib import *
>>> dir()
['Card', 'Hand', '_', '__builtins__', '__doc__', '__name__', 'clr',   'site', 'sys']
>>>
>>> c1 = Card()
>>> c2 = Card("9d")
>>> print c1,c2
As 9d
>>>
>>> h1 = Hand(c1,c2)
>>> h2 = Hand("Ah","8c")
>>> expected = 1
>>> actual = h1.Compare(h2)
>>>
>>> if actual == expected: print "Pass\n",
... else: print "Fail\n"
...
Pass
>>>
>>> ^Z

C:\IronPython>
Figure 2 Test Automation with an IronPython Script
In the sections that follow, I'll describe the TwoCardPokerLib class library under test so you'll understand exactly what is being tested. Then I'll discuss the interactive IronPython commands I used in Figure 1. Next I present and explain in detail the short Python script that produced the output in Figure 2. The complete source code for the test harness and the library under test are in the download that accompanies this column.
I'll wrap up with a brief discussion of how you can adapt and extend the ideas presented here to meet your own testing needs. I'm confident you'll find that the Python testing techniques presented here will be a great addition to your software testing skill set.

The Module Under Test
Now let's look at the library under test. I decided to create a .NET class library for testing that has enough functionality to reveal the types of issues you face when performing real module testing in a production environment, but not so much complexity that the details of the library obscure the testing issues. I imagined a hypothetical game of two-card poker where each player receives just two cards from four, standard, 52-card decks. This naturally leads to an object-oriented design with a Card class, a Hand class, and a Compare method to determine which of two Hand objects is better. I decided that each two-card hand would be categorized as a Straight Flush (consecutive cards of the same suit), a Pair (two cards with the same rank), a Flush (two cards of the same suit), a Straight (two consecutive cards), and then Ace High down to Four High. Note that a Three High hand is not possible because Three-Two is a Straight and Three-Ace is an Ace High. Even such a simple example is rather interesting to implement. IronPython can be used to effectively test .NET libraries regardless of the implementation language, and IronPython can be used to test classic COM libraries. The source code for the Card class in my TwoCardPokerLib library is listed in Figure 3.
Figure 3 The Card Class of the Library under Test
public class Card
{
  private string rank;
  private string suit;
  public Card() {
    this.rank = "A"; // A,2, . . ,9,T,J,Q,K
    this.suit = "s"; // c,d,h,s
  }
  public Card(string s) {
    this.rank = s[0].ToString();
    this.suit = s[1].ToString();
  }
  public override string ToString() {
    return this.rank + this.suit;
  }
  public string Rank {
    get { return this.rank; }
  }
  public string Suit {
    get { return this.suit; }
  }
  public static bool Beats(Card c1, Card c2) {
    if (c1.rank == "A") {
      if (c2.rank != "A") return true;
      else return false;
    }
    else if (c1.rank == "K") {
      if (c2.rank == "A" || c2.rank == "K") return false;
      else return true;
    }
    else if (c1.rank == "Q") {
      if (c2.rank == "A" || c2.rank == "K" || c2.rank == "Q") return false;
      else return true;
    }
    else if (c1.rank == "J") {
      if (c2.rank == "A" || c2.rank == "K" || c2.rank == "Q" || c2.rank == "J")
        return false;
      else
        return true;
    }
    else if (c1.rank == "T") {
      if (c2.rank == "A" || c2.rank == "K" || c2.rank == "Q" ||
        c2.rank == "J" || c2.rank == "T")
        return false;
      else return true;
    }
    else { // c1.rank is 2, 3, . . 9
      int c1Rank = int.Parse(c1.rank);
      int c2Rank = int.Parse(c2.rank);
      return c1Rank > c2Rank;
    }
  } // Beats()

  public static bool Ties(Card c1, Card c2) {
    return (c1.rank == c2.rank);
  }
} // class Card
In order to keep my code short and the main ideas clear, I have taken some shortcuts you wouldn't take in a production environment, such as omitting all error checks. My Card class was intentionally designed to illustrate many of the common features of a typical .NET module. Notice there is a default Card constructor, which creates an Ace of spades card object. There is a constructor that accepts a string such as Td to create an object that represents a ten of diamonds card. I implement get properties to expose the rank and suit of a Card object. And I implement static Beats and Ties methods that can be used to determine if one Card object beats or ties another Card object. My Hand class is too long to present in its entirety so I'll just describe the key parts of the class. The Hand class has just two member fields:
public class Hand {
  private Card card1;
  private Card card2;
  // constructors, methods, etc.
}
I make the important assumption that card1 is the higher of (or possibly equal to) the two Card objects. This greatly simplifies the logic of my Hand.Compare method. My Hand class has multiple constructors, a typical situation you must take into account when performing module testing. The default Hand constructor creates a hand with two ace–of-spades cards:
public Hand() {
  this.card1 = new Card();
  this.card2 = new Card();
}
I implement two other Hand constructors that allow me to either pass in two Card objects or two strings:
public Hand(Card c1, Card c2) {
  this.card1 = new Card(c1.Rank + c1.Suit);
  this.card2 = new Card(c2.Rank + c2.Suit);
}
public Hand(string s1, string s2) {
  this.card1 = new Card(s1);
  this.card2 = new Card(s2);
}
In Figure 4 I implement four private helper methods. As you'll see, private methods are not exposed to an IronPython test script.
Figure 4 Private Methods
private bool IsPair() { return this.card1.Rank == this.card2.Rank; }
private bool IsFlush() { return this.card1.Suit == this.card2.Suit; }
private bool IsStraight() {
  if (this.card1.Rank == "A" && this.card2.Rank == "K") return true;
  else if (this.card1.Rank == "K" && this.card2.Rank == "Q") return true;
  // etc: Q-J, J-T, T-9, 9-8, 8-7, 7-6, 6-5, 5-4, 4-3, 3-2
  else if (this.card1.Rank == "A" && this.card2.Rank == "2") return true;
  else return false;
}
private bool IsStraightFlush() { return this.IsStraight() &&
  this.IsFlush();
}
As a general rule of thumb, when performing module testing you do not explicitly test private methods. The idea is that any errors in helper methods will be exposed when you test the public method that uses the helper. The Hand.Compare method is surprisingly tricky. I coded private Beats and Ties helper methods and then use them to implement the public Compare method:
public int Compare(Hand h) {
  if (this.Beats(h))
    return 1;
  else if (this.Ties(h))
    return 0;
  else if (h.Beats(this))
    return -1;
  else
    throw new Exception("Illegal path in Compare()");
}
I use the old C language strcmp(s,t) function paradigm for Hand.Compare—if the "left" Hand parameter (the "this" object) beats the "right" parameter (the explicit Hand input parameter), Compare returns 1. If the right parameter beats the left parameter Compare returns -1. If the two Hand objects are equal according to my rules, Compare returns 0. Most of the work is done by the private Beats method, which is shown in Figure 5.
Figure 5 The Beats Method
private bool Beats(Hand h) {
  if (this.IsStraightFlush()) {
    if (!h.IsStraightFlush()) return true;
    if (h.IsStraightFlush()) {
      if (Card.Beats(this.card1, h.card1)) return true;
      else return false;
    }
  } // this.IsStraightFlush()
  else if (this.IsPair())
    // code
  else if (this.IsFlush())
   //  code
  else if (this.IsStraight())
    // code
  else
    // code for Ace-high down to Four-high
  }

  return false;
}
The code for Beats is about a page long and you can check out the details if you're interested by examining the accompanying code download. I deliberately inserted a logic error into the private Ties method:
else if (this.IsFlush() && h.IsFlush() &&
  this.card1.Rank == h.card1.Rank)          // error
    return true;
If the two Hand objects being compared are both Flushes, then I check to see if both hands have the same rank for the high card. However I don't check the second card of each hand as I should have:
else if (this.IsFlush() && h.IsFlush() &&
  this.card1.Rank == h.card1.Rank &&
  this.card2.Rank == h.card2.Rank)         // correct
    return true;
This logic error produces the test case failure shown in Figure 2. I built my library by using Visual Studio to create a Class Library project named TwoCardPokerLib at C:\Module TestingWithPython, resulting in a TwoCardPokerLib.dll file at C:\ModuleTestingWithPython\TwoCardPokerLib\bin\Debug.

Ad Hoc Interactive Module Testing
Now let's see how you can examine and test .NET libraries using IronPython. In particular, let's look at each of the commands shown in the screenshot in Figure 1. The first part of the output shown in Figure 1 indicates that I am using version 1.1.1 of IronPython.
IronPython is a free download available from CodePlex, the Microsoft-sponsored open source project, at codeplex.com/IronPython. It requires the .NET Framework 2.0 and runs on any machine that supports that version of the Framework. I am using Windows Vista. You don't really install IronPython, rather you simply download a single zipped file to your machine and extract its contents to any convenient directory. In my case I stored all the IronPython files and subdirectories at C:\IronPython. I invoked the Python command-line interpreter (ipy.exe) and specified an optional -X:TabCompletion argument to enable tab completion. If you type ipy.exe -h you'll get a list of all options. When IronPython starts up, it will execute a special startup script, named site.py, if such a script exists. I used the standard site.py file that comes with IronPython, without any modifications.
After IronPython initializes, I examine the current IronPython system path information:
>>> import sys
>>> sys.path
['C:\\IronPython', 'C:\\IronPython\\Lib']
First I issue an import sys command so I have access to the methods inside the special IronPython sys module. Next I display the list of paths that IronPython will examine when looking for files that are not in the current directory. You can think of this as a local IronPython mechanism that is somewhat similar to the Windows operating system Path environment variable. Because I invoked IronPython with the TabCompletion feature, if I wanted to know what properties and methods are available to me from the sys module, I could have typed sys. and hit the <tab> key repeatedly. Next I tell IronPython where my .NET module under test is located:
>>> sys.path.append(r'C:\ModuleTestingWithPython\TwoCardPokerLib\bin\Debug')
>>> sys.path
['C:\\IronPython', 'C:\\IronPython\\Lib', 'C:\\ModuleTestingWithPython\\TwoCardPokerLib\\bin\\Debug']
I use the path.append method of the sys module to add a new directory to the IronPython search list. Notice the r character in front of the string argument to append. In Python you can use either single-quotes or double-quotes around strings. However, unlike some languages, Python will evaluate escape characters such as \n inside both single-quoted and double-quoted strings. To ensure that a string is interpreted strictly as a literal (a "raw" string in Python terminology) you can use the r modifier as I've done above. After appending the new directory I verify that I didn't make any typing mistakes by issuing a sys.path command. Next I prepare to load my DLL under test by first enabling methods that can load .NET modules:
>>> import clr
>>> dir()
['_', '__builtins__', '__doc__', '__name__', 'clr', 'site', 'sys']
I issue an import clr (for the common language runtime) command and now I have access to the clr module, which has several methods that can load .NET assemblies. Then I use the dir() Python command to see which modules I currently have access to. Notice I do not currently have access to the TwoCardPokerLib library. Now I can use the clr module to gain access to my library under test:
>>> clr.AddReferenceToFile("TwoCardPokerLib.dll")
>>> from TwoCardPokerLib import *
>>> dir()
['Card', 'Hand', '_', '__builtins__', '__doc__', '__name__', 'clr', 'site', 'sys']
I use the AddReferenceToFile method to enable my current IronPython environment to call into the TwoCardPokerLib DLL. Python is case sensitive, so if I had typed addreferencetofile, for example, I would have gotten a no-such-attribute (method) error.
After adding a reference to my module under test, I need to use the import statement to make the module available. I could have typed import TwoCardPokerLib but I type from TwoCardPokerLib import * instead. If I use the first, simpler form, I would have to fully qualify everything in my module, for example, TwoCardPokerLb.Hand instead of just typing Hand. The from-<module>-import-<classes> form of the import command allows me to omit the parent module name when I type. Notice that after I do a dir() command, I see that the Card and Hand classes inside the TwoCardPokerLib module are now available to me. Now I can exercise my module under test by creating two Card objects:
>>> c1 = Card()
>>> c2 = Card("9d")
>>> print c1,c2
As 9d
I use the default Card constructor to instantiate an object named c1. Recall from the previous section that the default Card constructor creates an Ace of spades object. I use the non-default Card constructor to create an object representing a Nine of diamonds. Observe that I do not use a keyword like "new" to instantiate objects as I would in some languages. If I had used the short "import TwoCardPokerLib" statement earlier, I would have called the constructor as "c1 = TwoCardPokerLib.Card()". I use the intrinsic Python print statement to display my two Card objects. Behind the scenes, the IronPython print statement is actually calling the Card.ToString() method I implemented in my class library. Now I instantiate two Hand objects and call the Hand.Compare method:
>>> h1 = Hand(c1,c2)
>>> h2 = Hand("Ah","8c")
>>> expected = 1
>>> actual = h1.Compare(h2)
I pass the two Card objects I just created into a Hand constructor to create a first hand, and then I use the string-parameter version of constructor to instantiate a second hand. Because a hand of Ace-Nine beats a hand of Ace-Eight, I expect the Compare method to return 1 so I store that value in a variable called expected. Notice that Python is a dynamically typed language, so I don't declare data types. I call the Hand.Compare method and store the result into a variable named actual. I can view the available methods in the Hand class by typing Hand. on the command line and then hitting the <tab> key. I would see the public methods such as Compare and ToString but I would not see private methods such as Beats and Ties. Now I can determine a pass/fail result for my ad hoc interactive test:
>>> if actual == expected: print "Pass\n",
... else: print "Fail\n"
...
Pass
>>>
The Python if-then syntax on a command line is a bit unusual, so I'll explain it in the next section. But essentially I check to see if the value in the variable called actual is equal to the value in the variable called expected. If so, I display a "Pass" message; otherwise I print "Fail". Notice I 've included a newline character in each string. The intermediate "..." response from the IronPython interpreter indicates that I have an incomplete command and the interpreter is waiting for me to complete the command.

Lightweight Module Test Automation
Now let's see how to write lightweight IronPython scripts to test .NET class libraries. The script in Figure 6 produced the output shown in Figure 2.
Figure 6 File harness.py
# harness.py
# test TwoCardPokerLib.dll using data in TestCases.txt

print "\nBegin test run\n"
import sys
print "Adding location of TwoCardPokerLib.dll to sys.path"
sys.path.append(r'C:\ModuleTestingWithPython\TwoCardPokerLib\bin\Debug')

import clr
print "Loading TwoCardPokerLib.dll\n"
clr.AddReferenceToFile("TwoCardPokerLib.dll")
from TwoCardPokerLib import *

print "=================================="
fin = open("TestCases.txt", "r")
for line in fin:
  if line.startswith("$"):
    continue
  (caseID,input,method,expected,comment) = line.split(':')
  expected = int(expected)
  (left,right) = input.split(',')
  h1 = Hand(left[0:2],left[2:4])
  h2 = Hand(right[0:2],right[2:4])
  print "Case ID = " + caseID
  print "Hand1 is " + h1.ToString() + "   " + "Hand2 is " + h2.ToString()
  print "Method = " + method + "()"
  actual = h1.Compare(h2)

  print "Expected = " + str(expected) + " " + "Actual = " + str(actual)
  if actual == expected:
    print "Pass"
  else:
    print "** FAIL **"

  print "=================================="

fin.close()
print "\nEnd test run"
# end script
I begin my IronPython test harness script with comments:
# harness.py
# test TwoCardPokerLib.dll using data in TestCases.txt
The # is the comment character for Python scripts. Python is line-based, so comments run through the end of a line. Python also supports multi-line comments using triple single-quotes. Python scripts use a .py file extension and can be created using any text editor, including Notepad. My test harness reads test case data from an external file named TestCases.txt:
0001:AcKc,AdAs:Compare:1:  "royal flush" > pair Aces
0002:Td9d,Th9h:Compare:0:  straight flush diamonds == straight flush hearts
$0003:Ah2c,As9c:Compare:1:   straight > Ace high
0004:9h6h,9d7d:Compare:-1:  flush 9-6 high < flush 9-7
0005:KcJh,As5d:Compare:-1: King high < Ace high
Here I have just five test cases, but in a production scenario I would have hundreds or thousands. There are 7,311,616 (or 524 ) legal inputs to Hand.Compare, which illustrates the impracticality of exhaustively testing even simple modules.
Each line represents a single test case. Each field is delimited by the colon character. The first field is a test case ID. The second field is a string that contains information for two Hand objects, separated by a comma character. The third field indicates which method to test. The fourth field is the expected value. The fifth field is optional and is a test case comment. Notice test case 0003 is preceded by a $ character. I will scan for this character in my test harness and skip over any such test cases. Test case 0004 generates a fail result because of the deliberate logic error (solely for demonstration purposes) that I placed inside the Hand.Ties method that is called by the Hand.Compare method under test. Storing test case data in a text file is quick and easy. I could also have embedded my test case data directly into my harness, or stored my cases in an XML file, and so on. Python can easily handle any test case storage scheme. Next I display a message to the command shell:
print "\nBegin test run\n"
Because Python strings can be delimited by either double quotes or single quotes, and escape characters are evaluated inside both quote styles, the choice between quote types is really a matter of preference. I generally use double-quotes except when I am going to use the r modifier to make my string a literal, in which case I tend to use single quotes. Next my Python test harness script adds the location of my DLL under test to the Python system path:
import sys
print "Adding location of TwoCardPokerLib.dll to sys.path"
sys.path.append(r'C:\ModuleTestingWithPython\TwoCardPokerLib\bin\Debug')
Here I hardcode the location of the DLL under test. I can pass command-line arguments to my python script and access them with the built-in sys.argv array. For example, if I were to invoke my script as
> ipy.exe   harness.py   C:\Data   MyLib.dll
then sys.argv[0] would hold the name of my script (harness.py), sys.argv[1] would hold the first argument (C:\Data), and sys.argv[2] would hold the second argument (MyLib.dll). Next I tell my harness to load my module under test:
import clr
print "Loading TwoCardPokerLib.dll\n"
clr.AddReferenceToFile("TwoCardPokerLib.dll")
from TwoCardPokerLib import *
There are several ways to load a .NET-based library using IronPython. The clr.AddReferenceToFile method is simple and effective when your sys.path contains the location of the library to add, but the clr module also contains alternatives including clr.AddReference, clr.AddReferenceByName, clr.AddReferenceByPartialName, and clr.AddReferenceToFileAndPath. Here I use the * wildcard to load all classes from the TwoCardPokerLib library, but I can load specific classes by naming them. Python has several ways to process a text file line-by-line. I use the intrinsic file open function, and pass it the name of my test case file and an r argument to indicate that I am opening the file for reading:
print "=================================="
fin = open("TestCases.txt", "r")
for line in fin:
  # process line here
Other common modes include w for writing and a for appending. The mode argument is optional and defaults to r so I could have left it out. A good reference for Python language features, syntax, and intrinsic functions can be found at docs.python.org. The result of the open function is a file handle I assign to a variable named fin. Then I use a for loop to iterate line by line. I named my line-storage variable "line", but I could have used any legal variable name. Notice a syntactic quirk of Python: instead of using begin-end tokens such as {. .. } or begin. .. end as most languages do, Python uses the colon character in conjunction with indentation to indicate the start and finish of statement blocks. Inside my main processing loop, I use the intrinsic startswith string function to check if the current line from my test case file begins with a dollar sign character:
if line.startswith("$"):
  continue
If I find a $, I use the continue statement to skip over the remaining statements in the for loop, and read the next line from my test case file. Python has a complete set of loop and decision control structures. Next I parse my test case data line into its individual fields:
(caseID,input,method,expected,comment) = line.split(':')
expected = int(expected)
I use a neat feature of Python called a tuple and the intrinsic split string function to quickly parse my data. I could have performed the task using a traditional array approach as you see below, but I think the Python tuple approach is both shorter and easier to understand.
tokens = line.split(':')
caseID = tokens[0]
input = tokens[1]
method = tokens[2]
expected = tokens[3]
comment = tokens[4]
I use the int() function to explicitly convert the variable called expected from type string to type int so that I can compare expected with actual. Other useful type conversion functions include str(), float(), long(), and bool(). Next I perform a second parsing operation to extract the two input hands:
(left,right) = input.split(',')
h1 = Hand(left[0:2],left[2:4])
h2 = Hand(right[0:2],right[2:4])
After my first parse with split, the variable input holds a string value like KcJh,As5d. So, I call split again with a, argument and store the two results into the variables I called left and right. So now the string variable called left holds KcJh and right holds As5d. Unlike many languages, Python does not have a substring method. Instead, Python uses array indexing to extract substrings. Array indexing starts at 0, so if variable left holds KcJh then the expression left[0:2] yields Kc and the expression left[2:4] yields Jh. Notice that to extract a substring from a larger string, you specify the index of the first character in the larger string (as you'd expect) but one more than the index of the ending character. Once I have strings that represent individual cards, I pass them to the Hand constructor. Next I echo my input data to the shell:
print "Case ID = " + caseID
print "Hand1 is " + h1.ToString() + "   " + "Hand2 is " + h2.ToString()
print "Method = " + method + "()"
Python uses the + character to perform string concatenation, so in the three print statements above I am printing three strings. The print function can accept multiple arguments separated by the, character so I could have written statements such as the following to produce the same output:
print "Case ID = " , caseID
Now I'm ready to call the method under test:
actual = h1.Compare(h2)
print "Expected = " + str(expected) + " " + "Actual = " + str(actual)
Notice that because I implemented the Hand.Compare method as an instance method, I call it from the context of the h1 Hand object. If I had coded Compare as a static method I would have called it like so:
actual = Compare(h1,h2)
Notice that at this point the variables actual and expected are of type int because actual is the return value from the Compare method that is defined to return an int, and expected was explicitly cast to an int. Therefore I cast actual and expected to strings so that I can concatenate a single output string. Now I display my test case pass/fail result:
if actual == expected:
  print "Pass"
else:
  print "** FAIL **"

print "=================================="
Python uses indentation to indicate the beginning and end of block statements. If you have a lot of experience programming in other languages, Python's use of indentation may seem a bit odd at first. But most testers I've talked to say that Python's unusual syntax issues become familiar quite quickly.
If I want to keep track of the total number of test cases that pass, I can initialize counters outside my main processing loop like so:
numPass = numFail = 0
and then modify my if-then structure:
if actual == expected:
  numPass += 1
  print "Pass"
else:
  numFail += 1
  print "** FAIL **"
Then I can print my results outside the processing loop:
print "Num pass = " , numPass ,  " num fail = " ,  numFail
Note that Python does not support syntax like numPass++ or ++numPass to increment an int variable. Here I am simply printing results to the command shell, but I can easily write results to a text file, XML file, SQL database, or other storage. Now I finish my test harness up:
fin.close()
print "\nEnd test run"
# end script
I close my test case file reference to free up its file handle, and print a message stating that the test is complete. My script structure is very simple but Python has features that allow you to manage complex scripts. For example, in a production environment I would definitely wrap my entire script in an exception handler:
try
  # harness code here
except
  # handle any exceptions here
Additionally, Python supports programmer-defined functions, so I could have structured my harness as a main function plus helper functions.

Wrapping Up
Automated module testing is arguably the most fundamental kind of software test automation. Whenever I want to evaluate a programming language's suitability for test automation, I first check how well the language deals with module testing. In my opinion, IronPython passes this litmus test with flying colors. No programming language is perfect for all testing tasks and scenarios. That said, IronPython has many features that make it an excellent choice as a module testing language.
Let me finish by mentioning how lightweight module test automation with IronPython relates to unit testing. Unit tests with frameworks such as NUnit are most often placed directly inside your module code. Test driven development with unit testing is primarily a developer activity. Using a unit testing approach during development does not absolve you from the responsibility of performing thorough, dedicated module testing. This is where Python comes in. In other words, module testing with Python is a complement to, not a substitute for, unit testing. When the two approaches are used together you can produce better, more reliable software.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes. He can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


The Polyglot Programmer
Reaping The Benefits Of Cobra
Ted Neward
Contents
Beginnings
Trust But Verify
"It's a dessert topping; it's a floor wax. It's both!"
Summing Up
In the first installment of this column, I discussed the importance of understanding more than one programming language, and in particular, more than one kind of programming language; just knowing C# and/or Visual Basic isn't enough to stand out against the mob anymore. The more tools in your toolbox, the better the carpenter you become. (Up to a point, anyway—'tis the poor carpenter who owns every tool ever made and yet still doesn't know how to use any of them.)
In this installment, I'll examine a language that's not too far removed from the object-oriented languages you're familiar with, but has a few new features that will not only offer some new ideas, but potentially kick-start some new ways of thinking about your current tool set.
Allow me to introduce you to Cobra, a descendant of Python that, among other things, offers a combined dynamic and statically typed programming model, built-in unit test facilities, scripting capabilities, and some design-by-contract declarations.

Beginnings
As Lewis Carroll is famous for saying, let us "begin at the beginning." Cobra is a .NET language that can be obtained via your Web browser, at cobra-language.org, or via a Subversion client at the same domain. If building from source, use the install-from-workspace.bat script in the source directory to build the Cobra compiler and install the necessary assemblies into the global assembly cache (GAC). While building from source isn't necessary—prebuilt binaries are also available—it's often interesting to look at the source of Cobra's compiler, which itself is written in Cobra (a significant milestone in any language), as a way of seeing many, if not all, of the language features at work in a nontrivial program.
Once your compiler is built, the first step is to test out the Cobra language via everybody's favorite "test-the-compiler" (or is it "test-the-programmer") program, the ubiquitous Hello World:
class Program
        def main is shared
                print 'Hello, world.'
Cobra offers two different ways to interact with its source code: either the code can be executed directly via a script-like approach by executing "cobra hello.cobra" from the command-line, or the code can be compiled and then executed in the traditional manner of C#/Visual Basic/C++ languages, by compiling it first ("cobra –compile hello.cobra") and executing it as you would any .NET executable. Note that the "cobra" executable is the same in both cases, the one built in the Cobra\Source directory. (Observant readers that examine the contents of the directory after "scripting" the .cobra file will notice that Cobra's "scripting" mode is really just a two-pass compile-then-execute sequence, leaving the compiled "hello.exe" behind after it's finished executing.)
Looking at the Cobra source, it becomes apparent that while there is some distinct similarities to the C# and Visual Basic languages, there are also some distinct differences. In fact, those readers familiar with Python (or its CLR equivalent, IronPython) will quickly notice that Cobra is primarily derived from the Python syntax—Cobra's inventor, Chuck Esterbrook, found the Python syntax appealing and used that as a base from which to derive his language.
For those who've never used Python before, however, one difference immediately stands out: instead of using keywords (like "begin" and "end") or tokens (like "{" and "}") to set blocks of code apart from the surrounding code, Cobra—like Python—uses significant white space, or, as Cobra's inventor prefers, indented blocks, meaning that blocks of code are set apart by lexical indentation. In other words, the end of a decision-making block (an "If" block) is signaled not by a closing brace, but by the fact that the next line is one level of indentation removed from the previous line. Thus, in Figure 1, Cobra knows that the final "print" line is executed regardless of the results of the "if" test, because that line is indented equally to the "if" statement that made the decision.
Figure 1 Indentation Structure in Cobra Code
class Program
        def main is shared
                if 1==1
                        print "Oh, goody, math works!"
                        print "I was beginning to get worried there."
                print "Math test complete"
Because of the problems that emerge if you mix both tabs and spaces in a single file to control the indentation, Cobra mandates either/or, but not both, generating a compiler error where it sees a mix of the two.
Furthermore, Cobra has codified ritualistic tendencies in the name of code clarity:
  • Types such as classes, structs, interfaces, and enums, must be capitalized and cannot start with an underscore, similar to the patterns established by Microsoft in the Base Class Library: String, Control, ApPDomain, and so on.
  • Arguments and local variables must begin with a lowercase letter and cannot begin with an underscore. Examples include x, y, count, and index.
  • In expressions, access to object or class members is always done explicitly with a period (.foo) or implicitly if the member starts with an underscore ("_foo").
  • Although not enforced, object variables are typically prefixed by an underscore, which also implies protected visibility. Examples include _serialNum, _color, and _parent.
Thus, for any given fragment of Cobra code, a Cobra programmer can easily distinguish the elements involved in the fragment or snippet. For example, in the following Cobra statement
e = Element(alphabet[_random.next(alphabet.length)])
we can tell that
  • e and alphabet are local variables or arguments
  • _random is an object member/variable, of (probably) protected visibility
  • Element is a type
Cobra thus follows the basic premise of Python: that for any given scenario, there's only one way to do it, including code naming and formatting conventions.

Trust But Verify
Trust but verify. Those words, famously uttered by former U.S. President Ronald Reagan (in the context of arms reduction negotiations), hold just as strongly for programming as they did for nuclear arms reduction. Although it's easy to believe—and trust—that fellow job-site programmers would always know better than to pass a null reference into a method that clearly documents the fact that doing so will generate a NullReferenceException, it's generally been proven over and over again (particularly during demos before the customer or the Big Boss) that the safer course is to make those assumptions explicit. In fact, had the former President been alive today and taken up programming, it's entirely possible that he would amend his words to read, "Trust, but document, insist, and verify."
In the programming world, that means two basic things: to insist is to program defensively, often using assert statements or methods to ensure (at the risk of a runtime exception) that values passed in match a certain criteria; to verify, is to write unit tests against the method to ensure that the method's implementation handles the pathological programmer who insists on violating those restrictions.
Consider the (relatively simple) task of creating a class that represents a person within the system. Persons generally have a first name, a last name, and an age, and can be married to other Persons. In general, when writing these kinds of classes, you need to establish some invariants—basic principles that will hold for any use of the class. So, for example, a Person class may decide that a last name cannot be null, something that is generally enforced by the developer writing Person and usually done in the property-set construct of the class, as you see in the C# code in Figure 2.
Figure 2 Invariants Enforce Constraints
public class Person
{
  public Person(string fn, string ln, int a)
  {
    this.firstName = fn;
    this.lastName = ln;
    this.age = a;
  }
  public string FirstName
  {
    get { return firstName; }
    set { firstName = value; }
  }
  public string LastName
  {
    get { return lastName; }
    set { if ((value != null) || value.Length > 0) lastName = value; }
  }
  public int Age
  {
    get { return age; }
    set { age = value; }
  }
  public override string ToString()
  {
    string ret = String.Format("Person:
  }
}
The problem with this is that the invariant is enforced only when the property-setter is used; if another developer comes around later and puts a new method on Person that directly references the backing store (the field encapsulated by the property), the invariant is thus silently and subtly broken, and no one will notice it until it is too late. (This is the heart of the debate around whether classes should use properties internally or directly manipulate the fields.) This question, of course, applies equally to both properties and methods. Although it's not often that a property touches more than one backing field, it's not prohibited by the language, and it does happen periodically. Of even greater concern, each and every invariant (first name cannot be null or empty, last name cannot be null or empty, age cannot be negative) will need to be enforced, in each and every method, property, or constructor that could potentially modify the internal state.
Cobra offers a basic design-by-contract system, in that a class or method can declare certain requirements about the class, and the Cobra compiler will silently add verification statements to each and every member (or property or method) that could potentially modify the data in question. In this way, it is guaranteed that last name is never null, regardless of how the class is used and without having to explicitly code it in every method that could possibly change the contents of the instance (see Figure 3).
Figure 3 Verification Statements
class Person
    invariant
        .firstName <> ""
        .firstName.length > 0
        .lastName <> ""
        .lastName.length > 0
        .age >= 0

    var _firstName as String
    var _lastName as String
    var _age as number

    pro firstName from var
    pro lastName from var
    pro age from var

    cue init(first as String, last as String, age as number)
        ensure
            first.length > 0
            last.length > 0
            age > -1
        body
            _firstName = first
            _lastName = last
            _age = age

    def toString as String is override
        return 'Person: [.firstName] [.lastName], [.age.toString] years'
It's relatively easy to verify exactly how far and how often the Cobra language inserts these validation checks, by firing up the IL disassembler (ILDasm.exe) and examining the resulting Person class. Doing so reveals the beauty of the Cobra approach—each and every "modifying" method is appended with the code to verify the invariants, making them class-wide. (There is a compiler option to change how the invariants are generated—not at all, or only in methods, or, the default, appended to all of those operations.)
In and of itself, the design-by-contract feature of Cobra is nice, but it's still not sufficient to guarantee error-free programs; in addition, it's almost always necessary to write a series of unit tests against the code to ensure that it behaves as desired where both good data and bad is passed. Again, given that this is so critical to the software development process, Cobra chooses to make this a first-class construct inside the language—a "test" construct of the language—such that Cobra can execute the tests as part of the compilation process. So, for example, a simple suite of unit tests might look like Figure 4 for the Person class.
Figure 4 Unit Tests for Person
class Person
    invariant
        .firstName <> ""
        .firstName.length > 0
        .lastName <> ""
        .lastName.length > 0
        .age >= 0
    test
        p = Person('Neal', 'Ford', 29)
        assert p.firstName == 'Neal'
        assert p.lastName == 'Ford'
        assert p.age == 29

    var _firstName as String
    var _lastName as String
    var _age as number

    # ... everything else as-is ...
When run with the "-test" option at the command-line, Cobra will execute the unit tests contained in the class, and assuming they all work as expected, Cobra will report the tests pass, as well as compile the code into an executable. (Libraries are compiled using the "-t:lib" command-line switch.)
This is a different way of building unit tests, but it holds several advantages over the traditional NUnit-based approach to unit testing in that the tests are kept close to the code rather than in a separate file where it can get too far away from developers who are maintaining or updating the code. Anything that makes it easier to write (and maintain) the unit tests is generally considered a good thing, and having the unit tests right inside the class to be tested is a trick long ago learned in the Java camp.
Unfortunately, with this approach to tests comes code bloat; if the tests are sufficiently comprehensive (and they should be), including the contractual verification and unit tests as part of the deployed code can double, triple, or even quadruple the size of the code, which can certainly put a crimp in your plans for easy deployment of the finished product. Fortunately, the Cobra language provides a solution to this as well, via the "-turbo" option, which not only turns out every optimization, but also strips out the design-by-contract and unit test code, and is generally intended as a just-before-deployment optimization step.

"It's a dessert topping; it's a floor wax. It's both!"
Ages ago, a Saturday Night Live skit talked about the versatility of a product that was both a dessert topping and a floor wax. "It's both!" was the tag line, and at the time, it was pretty funny. Hey, it was the 70s.
Recently, however, a new flavor of argument has emerged within the programming language community, that of static versus dynamic typing—in other words, we're back to the same arguments that used to be leveled against (classic) Visual Basic and its runtime (IDispatch-based) binding and VARIANTS by the community of developers using C++ and its compile-time template facilities. This time, however, it's the strongly/statically typed community on the defensive, arguing the merits of safety in strongly/statically typed languages, against the productivity benefits cited by users of Ruby and Python (or, if you prefer, IronRuby and IronPython). As with most debates of this nature, there's lots of rhetoric, lots of unsubstantiated claims, and lots of cited statistics (most of which are made up on the spot).
In truth, the argument is a bit spurious and often masks a deeper difference, that of early or late binding of method calls. In "traditional" compiled languages (like C++/CLI or C#), the compiler decides at compile time whether a method call is acceptable based on the presence of a matching method. If one is found on the target class, it emits the necessary instructions (in CIL, this is a "callvirt" instruction with the metadata token of the method in question) into the generated assembly. In contrast, a late-bound call isn't actually resolved until run time, usually via a metadata-based API like Reflection, at which point the run time looks up the method, and if present, invokes it.
The differences here aren't too difficult to spot—in the first case, the method has to be visible at compile time, forcing the programmer, in some cases, to engage in some type navigation to keep the compiler happy. (This is the ubiquitous "if obj is Person" followed by a downcast and method call idiom.) This particular sequence of language statements is often frustrating, particularly when the programmer "knows," by virtue of a wider awareness of the context of the situation, that the object in question is exactly the type it needs to be to make the call succeed. Yet the compiler can't see that far, and forces the developer through a series of steps simply to appease the compilation process.
In the second case, however, the programmer will often discover that what she "knows" doesn't exactly jibe with reality, usually because a run time exception is thrown at the most embarrassing moment possible—during the big demo, the product launch, or at run time on the day of the IPO. Maybe another developer accidentally broke that assumed invariant, maybe it's a code path that was never quite covered during unit tests, but regardless of how it happens, it still leaves the program terminated, and the programmer mortified.
Thus the argument persists: is it better to take the productivity hit to get compiler-assured promises of code correctness, or is it better to trust the programmers' unit test suite and tell the compiler to go hang?
Cobra neatly dodges this argument completely, by being both static and dynamic. In other words, Cobra takes the position that when it can do so, it will bind method calls early, in the same way that C# chooses to, but when it can't, for whatever reason, it will choose instead to use late-bound semantics and resolve the method call at run time. And, while C# 4.0 promises a similar kind of capability (as, by the way, has Visual Basic since it became a .NET language, via the Option Explicit and Option Strict flags), Cobra requires no additional syntactic help from the programmer to decide when to be early-bound and when to be late-bound; it simply makes the decision as it compiles, neatly avoiding the need for the developer to make that call ahead of time, as in Figure 5.
Figure 5 Dynamic Binding
class Person

    get name as String
        return 'Blaise'

class Car

    get name as String
        return 'Saleen S7'

class Program

    shared

        def main
            assert .add(2, 3) == 5
            assert .add('Hi ', 'there.') == 'Hi there.'
            .printName(Person())
            .printName(Car())

        def add(a, b) as dynamic
            return a + b

        def printName(x)
            print x.name  # dynamic binding
As you can see, Cobra supports both the "dynamic" modifier, to indicate that a particular type or method should treat its arguments and types as dynamic, as well as the "dynamic inference" that is used in the "printName" method, where Cobra simply recognizes that name can't be bound statically, and instead seeks to resolve it at run time.

Summing Up
Cobra's unique feature set marks it as an interesting "sum of choices" language. The silent switch between early- and late-binding makes it an idea language for working with .NET components that make heavy use of late-bound APIs, such as the Office Automation model, while still preserving the safety and performance benefits of early-binding where possible. This alone makes Cobra worth considering, particularly for Office (and other COM-based) Automation programming.
When combined with Cobra's ability to act as both scripting and compiled language tool, however, Cobra's advantages really begin to shine.
Good luck with your Polyglot experimentation!

Send your questions and comments for Ted to polyglot@microsoft.com.

Ted Neward is a Principal Consultant with ThoughtWorks, an international consultancy specializing in reliable, agile enterprise systems. He has written numerous books, is a Microsoft MVP Architect, INETA speaker, and PluralSight instructor. Reach Ted at ted@tedneward.com, or read his blog at blogs.tedneward.com.


Foundations
Customizing the New WPF Calendar Controls
Charles Petzold
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Calendar: A Brief Orientation
Styling the Calendar
Setting a New Calendar Template
Replacing the CalendarItem Template
The Powerful CalendarDayButton Template
Deriving from Calendar
The DatePicker Control
When Microsoft released Windows Presentation Foundation (WPF) in 2006, everyone agreed it was missing a couple of pieces, most noticeably calendar controls. It seemed like an odd omission—after all, Windows Forms had MonthCalendar and DateTimePicker controls, so why not just port those to WPF?
But porting a control to WPF isn't a simple matter of just changing property and event names. WPF controls must be "lookless." The visual appearance of the control must be separated into a visual tree of other elements and controls. This default template must be structured and documented in such a way to be completely replaceable. In this way developers and designers can customize controls to give them a whole new appearance.
Designing a control with a replaceable template can be intimidating, as I discovered when I attempted to design a MonthCalendar control for my article "Templates for Uncommon Controls" in the January 2008 issue. You want to make the control as generalized as possible to be amenable to a variety of different appearances, yet without making the interface between the code and template inordinately complex. Calendars introduce another level of complexity because they are heavily reliant on cultural conventions, and it's not obvious to what extent a custom template should be allowed to violate those conventions.
That's possibly why WPF calendar controls took so long. But late last year Microsoft released Calendar and DatePicker controls (with a few subsidiary classes) in a WPF Toolkit. (A new DataGrid is also included, but I won't be discussing that here.) These controls are very much compatible with the same controls in Silverlight, and they will probably end up in the next release of WPF.
For the remainder of this article, I'll assume you have downloaded the WPF Toolkit source code and binaries and run the installer to register WPFToolkit.dll. Once this DLL is registered, you can use it in a WPF project in Visual Studio by right-clicking the References tag in the project, selecting Add Reference, and then selecting the .NET tab. The WPF Toolkit DLL will be found toward the bottom of the list.

Calendar: A Brief Orientation
The Calendar and DatePicker classes (along with some supporting enumerations and delegates) have a namespace of Microsoft.Windows.Controls. Some subsidiary classes, mostly notably CalendarItem, CalendarButton, CalendarDayButton, and DatePickerTextBox, are in the Microsoft.Windows.Controls.Primitives namespace. These are not the customary namespaces for WPF controls, so you'll probably need to add new using directives in your C# code and construct the proper XML namespace declarations in XAML files. Classes related to the new Visual State Manager (which I'll discuss later in this article) have a more normal namespace of System.Windows.
The Calendar class derives from Control. If you simply instantiate Calendar, you get the view shown on the left in Figure 1. Only one month is displayed—never multiple months as the Windows Forms MonthCalendar does. The month always includes six weeks, even if only four or five are required. Click the header (the part at the top indicating the month and year) and you'll switch to the year view in the center. You can click one of the months to return to a view of that month, or click the header again to get the decade view shown at the right. Click one of the years to return to the year view.
Figure 1 The Three Calendar Display Modes)
These three views correspond to the DisplayMode property of Calendar, which is set to one of the three members of the CalendarMode enumeration: Month, Year, or Decade.
The two buttons on each side of the header navigate forward or backward by month, by year, or by decade.
The DisplayDate property is of type DateTime and indicates which month, year, or decade is displayed. DisplayDateStart and DisplayDateEnd properties are of type nullable DateTime and restrict the navigation of the calendar to a particular range.
The IsTodayHighlighted property is true by default and causes a dark square to be drawn behind today's date. Dates can also be selected. Set the SelectionMode property to a member of the CalendarSelectionMode enumeration: SingleDate (the default), SingleRange, MultipleRange, or None. For the SingleDate mode, the SelectedDate property (of type nullable DateTime) indicates the selection; otherwise, the SelectedDates property is of type SelectedDatesCollection, which derives from ObservableCollection of type DateTime.
The BlackoutDates property lets you inhibit selection of various ranges of dates. The property is of type CalendarBlackoutDateCollection, which derives from ObservableCollection of type CalendarDateRange, a class that defines Start and End properties of type DateTime.
The FirstDayOfWeek property is of type DayOfWeek; the default is usually Sunday, but for some locales (most notably France) the default is Monday.
Calendar defines four events: DisplayModeChanged, DisplayDateChanged, SelectionModeChanged, and SelectedDatesChanged.

Styling the Calendar
The remaining public properties of the Calendar class are CalendarItemStyle, CalendarButtonStyle, and CalendarDayButtonStyle, all of type Style, but to understand these properties you need to know how the Calendar control is built and structured, and for that knowledge it helps to examine some XAML.
All the source code for the Calendar control is downloadable from the CodePlex site, but if you're interested in customizing the control, the most important file is undoubtedly Generic.xaml, located in the subdirectory Toolkit-release\Calendar\Themes. The Generic.xaml file contains the default styles (including templates) for the Calendar control itself and the other controls that make up the Calendar control. If you'll be templating Calendar, you'll want to study Generic.xaml very closely.
Although the Calendar class defines all the properties and events for the control, and also performs some of the user input processing, the Calendar template in Generic.xaml reveals that Calendar consists only of an invisible StackPanel and a CalendarItem control.
The CalendarItem class also derives from Control and encompasses the entire visual appearance of the control. This includes the three buttons at the top (which I'll refer to collectively as the navigation buttons). It also includes two grids. One grid shows the days of the month (including the headers for the days of the week). The other grid shows either 12 months or 12 years. At any time only one of these two grids is visible.
CalendarItem is also responsible for populating those two grids. When showing the days of the month, it creates objects of type CalendarDayButton; the 12 months or years are of type CalendarButton. Both classes derive from Button.
The Generic.xaml includes the default styles and templates for Calendar, CalendarItem, CalendarButton, and CalendarDayButton. For these latter three items, you can set new styles (including templates) by setting the CalendarItemStyle, CalendarButtonStyle, or CalendarDayButtonStyle properties defined by Calendar, either directly or through the Style property of Calendar.
Setting a Style for Calendar itself is useful for Calendar-specific properties (such as DisplayMode or SelectionMode) or properties inherited by Calendar (such as HorizontalAlignment or Margin), but I want to focus on more problematic classes of properties: those relating to fonts and brushes.
The Calendar control is very precisely built from its subsidiary controls, and if you start messing around with fonts, the control can quickly become visually lopsided. For example, you can set the FontFamily property on Calendar and it will affect all text except the day-of-week headings. (The FontFamily for those headings is hardcoded in a DateTemplate inside the default template for CalendarItem.) However, if the text of the days of the month becomes a little smaller as a result, the whole month will seem to be shifted to the left.
Setting the FontSize property on Calendar has no effect. If you truly want to change the FontSize you'll have to change it in the CalendarButton style, the CalendarDayButton style, and two places inside the CalendarItem template. Forget about it. If you want to make the Calendar control a little larger or smaller than normal, don't do it with FontSize. I suggest defining a ScaleTransform and setting that to the Calendar's LayoutTransform property, or perhaps putting the Calendar in a Viewbox.
The colors that Calendar uses for marking the current day, selected dates, and blackout dates are hardcoded in the various templates. The only brushes you can easily change are the Background property of the Calendar control and the BorderBrush that surrounds the control. These are both set to linear gradient brushes in the default style of Calendar and then passed to CalendarItem via template bindings. If you want to replace the default Background brush, which I'll demonstrate later in this article, keep in mind that it's responsible for the different background shade behind the navigation buttons that you can see in Figure 1.

Setting a New Calendar Template
The first step in replacing a template for a control is looking at the class definition, either in documentation or the actual source code (if available). There you will see a collection of TemplatePart attributes. These TemplatePart attributes indicate named controls or elements that your new template must have to be entirely functional. For the Calendar control, a new template must have at least two parts: a CalendarItem with the name PART_CalendarItem and a Panel with the name PART_Root. (In the default Calendar template, the PART_Root Panel is a StackPanel.)
It's unlikely that you'll be replacing the template for the Calendar, but I shouldn't try to second guess your needs. Perhaps you'd like to add something to the visual tree that makes up Calendar, such as a scrollable ItemsControl that displays all the selected dates.
Such an application is illustrated by the SelectedDateLister project included with the downloadable source code for this column. Figure 2 shows an excerpt from the MainWindow.xaml file of that project with a new template for Calendar, and Figure 3 shows what it looks like.
Figure 2 A New Template for Calendar
<ControlTemplate TargetType="toolkit:Calendar">
    <StackPanel Name="PART_Root"
                Orientation="Horizontal">
        <primitives:CalendarItem
            Name="PART_CalendarItem"
            Style="{TemplateBinding CalendarItemStyle}"
            Background="{TemplateBinding Background}"
            BorderBrush="{TemplateBinding BorderBrush}"
            BorderThickness="{TemplateBinding BorderThickness}"
            VerticalAlignment="Center" />
        <Border BorderBrush="{TemplateBinding BorderBrush}"
                BorderThickness="{TemplateBinding BorderThickness}"
                Margin="4 4 0 4">
            <ScrollViewer
                    VerticalScrollBarVisibility="Auto"
                    Height="{Binding ElementName=PART_CalendarItem,
                                     Path=ActualHeight}"
                    Width="100">
                <ItemsControl
                    ItemsSource=
                        "{Binding RelativeSource={RelativeSource
                                      AncestorType=toolkit:Calendar},
                                  Path=SelectedDates}" />
            </ScrollViewer>
        </Border>
    </StackPanel>
</ControlTemplate>
Figure 3 Calendar with Selected Dates List
Notice the XML namespace prefix of primitives needed to reference the CalendarItem control. In the root element of the XAML file, that prefix is associated with the .NET namespace of Microsoft.Windows.Controls.Primitives and the WPFToolkit assembly. Similarly, the prefix toolkit is associated with the Microsoft.Windows.Controls namespace in the same assembly. The Calendar is simply instantiated in the XAML like so:
<toolkit:Calendar />
The style containing the new template is applied implicitly.
Notice also in Figure 2 the use of a Binding with RelativeSource to connect the ItemsSource property of the ItemsControl with the SelectedDates property of the Calendar. TemplateBinding can't be used for this job because SelectedDates is not backed with a dependency property.
How did I know what properties of CalendarItem I should set with TemplateBinding? I looked at the default Calendar template in Generic.xaml. In fact, I not only looked at it: I copy-and-pasted it right into my source code. Copy-and-paste is, I'm afraid, the best way to make modifications to existing templates.

Replacing the CalendarItem Template
To have more control over the internal appearance of the Calendar control, it's necessary to replace the template for CalendarItem. The default template is a substantial chunk of XAML—almost 250 lines in the Generic.xaml file—and it includes three embedded templates for the navigation buttons that appear at the top.
Experienced WPF programmers who haven't yet delved into Silverlight will be surprised to see something unfamiliar in the three embedded button templates: references to classes named VisualStateManager, VisualStateGroup, and VisualState. These classes are major components of the Visual State Manager that is already in Silverlight and expected to become part of WPF in the next major release. (The source code for this preliminary Visual State Manager is included in the WPF Toolkit.)
The Visual State Manager is intended to replace many of the triggers currently used in templates. Template triggers are customarily based on values of certain properties, such as IsEnabled, IsMouseOver, and IsSelected. The Visual State Manager is instead based around a more structured concept of visual states, with names such as Disabled, MouseOver, and Selected. At first glance, these visual states seem to have a one-to-one correspondence with triggers, but that's not really so. In practical use, triggers can result in different visual appearances depending on the order in which they appear, and often it becomes clumsy to define multiple triggers when triggers need to work in conjunction with each other. The alternative identification of visual states makes the programmer's intention much more obvious. The association between property settings and visual states occurs in code.
To reference the Visual State Manager classes in XAML, you'll need to define an additional XML prefix (vsm, for example) associated with the System.Windows namespace in the WPFToolkit assembly.
A custom template for CalendarItem should have eight named parts, as shown in Figure 4.
Figure 4 Named Parts for the CalendarItem Template
Part | Type
PART_Root | FrameworkElement
PART_HeaderButton | Button
PART_PreviousButton | Button
PART_NextButton | Button
DayTitleTemplate | Key name of DataTemplate
PART_MonthView | Grid
PART_YearView | Grid
PART_DisabledVisual | FrameworkElement
Notice that one of these names is actually a resource key name. The Resources section of the CalendarItem template should include a DataTemplate with a key name of "DayTitleTemplate" that contains a TextBlock element with the Text property set like so:
Text="{Binding}"
This DateTemplate is used to display the headings with the days of the week. The CalendarDayButton items used for these headers has its Content set to null, but its DataContext set to the text header (Su, Mo, Tu, and so forth), hence the strange binding. (I'll have more to say on the DataContext property of the CalendarDayButton later.)
The default visual tree of the CalendarItem template consists of several nested grids. An outer single-cell Grid is named PART_Root. This Grid contains a Border and a Rectangle named PART_DisabledVisual intended to provide a semi-opaque covering when the control is disabled.
The Border contains another Grid, this one with three columns and two rows providing the main structure to the control. The three named navigation buttons occupy the top row. The bottom row contains two grids (PART_MonthView and PART_YearView) spanning the three columns.
The Grid named PART_MonthView is expected to have seven columns (for the seven days of the week) and seven rows (one for the day-of-week headings and six for the days of the month), while PART_YearView is expected to have four columns and three rows (for 12 months of a year or 12 years in a "decade" view). When the code part of CalenderItem creates the CalendarDayButton and CalendarButton children to populate these grids, it explicitly sets the Grid.Row and Grid.Column attached properties to each child.
Figure 5 A New Bare-Bones CalendarItem Template
For this reason, it is unlikely that a custom CalendarItem template will differ significantly from the existing layout. If you want to take a shot at it, the MainWindow.xaml file in the BareBonesCalendarItem project has pretty much the minimum markup you need to get something complete and functional. The resultant visual is shown in Figure 5. Just to make it a little different, I built the overall structure from a DockPanel, so that the buttons for previous-month and next-month are on the sides. I didn't apply templates to these navigation buttons, which is why they look like actual buttons. Even with these simplifications, the template is almost 80 lines in length. It looks a little odd, but it works.
A safer and more rational approach to defining a new CalendarItem template is to simply copy the default template for CalendarItem from Generic.xaml into your own XAML file, and then modify only certain parts of it.
That's what I did in the ColorfulCalendarItem project. The MainWindow.xaml file defines an XML namespace prefix of local rather than toolkit for referencing the Calendar class to be consistent with the namespace declarations in Generic.xaml.
In the CalendarItem template I added a new Background brush for the outer Border:
<LinearGradientBrush StartPoint="0 0"   EndPoint="0 1">
    <GradientStop Offset="0"   Color="#FFFFC0" />
    <GradientStop Offset="0.5" Color="#FFE0B0" />
    <GradientStop Offset="1"   Color="#FFD0A8" />
</LinearGradientBrush>
I also added a new Border around the ContentPresenter for the center header button and gave that a background as well:
<Border Padding="12 0"
        CornerRadius="6">
    <Border.Background>
        <LinearGradientBrush StartPoint="0 0" EndPoint="0 1">
            <GradientStop Offset="0" Color="#FFC4A0" />
            <GradientStop Offset="1" Color="#FF9450" />
        </LinearGradientBrush>
    </Border.Background>
The result is shown in Figure 6.
Figure 6 New Colors Defined in the CalendarItem Template

The Powerful CalendarDayButton Template
The CalendarItem template contains embedded templates for the three navigation buttons displayed at the top of the calendar, but the template has no references to the CalendarButton and CalendarDayButton objects that populate the lower grids. These buttons are instantiated in code, as you can note by examining the CalendarItem.cs file. However, you can define new templates for these buttons by setting a Style object to the CalendarButtonStyle and CalendarDayButtonStyle properties of Calendar.
A custom template for CalendarDayButton is potentially very powerful. Here you can display additional information for each day beyond the customary two-digit number between 1 and 31. At first it doesn't seem as if you can do this in an intelligent manner, because anything that you add to the content of the CalendarDayButton would probably need to depend on the exact date represented by that button, and that information doesn't seem to be readily available.
Don't fret. Although the Content of the CalendarDayButton is set to a simple text string such as 24, the DataContext property of each CalendarDayButton is set to the actual DateTime object for that date, including the correct month and year. (This feature also helps distinguish the CalendarDayButton objects used for the day-of-week headings; those buttons have a null Content but a DataContext set to the text strings Su, Mo, and so forth. This is why a DataTemplate is required in CalendarItem for these headings.)
The availability of a DateTime object opens up a whole area of possible enhancements that you can implement simply by referencing some custom class in a new CalendarDayButton template. Because the default CalendarDayButton template is about 120 lines of XAML, you probably won't be writing such a template from scratch. As with the CalendarItem template, you'll probably want to copy the default CalendarDayButton template from Generic.xaml into your own project, and then modify it.
One possibility is to display holidays and other important days in a ToolTip displayed when you pass the mouse pointer over each day, as illustrated by the project RedLetterDays. It seemed reasonable to implement this feature as a binding converter: in effect, the converter converts a DateTime object into a String object.
Figure 7 shows the important parts of this binding converter. The constructor sets items for dates between March (when I'm writing this article) and June (when it will be published). In a real program, it would probably access these dates and text strings from a file.
Figure 7 Most of a Binding Converter for RedLetterDays
public class RedLetterDayConverter : IValueConverter {
    static Dictionary<DateTime, string> dict =
        new Dictionary<DateTime, string>();

    static RedLetterDayConverter() {
        dict.Add(new DateTime(2009, 3, 17), "St. Patrick's Day");
        dict.Add(new DateTime(2009, 3, 20), "First day of spring");
        dict.Add(new DateTime(2009, 4, 1), "April Fools");
        dict.Add(new DateTime(2009, 4, 22), "Earth Day");
        dict.Add(new DateTime(2009, 5, 1), "May Day");
        dict.Add(new DateTime(2009, 5, 10), "Mother's Day");
        dict.Add(new DateTime(2009, 6, 21), "First Day of Summer");
    }

    public object Convert(object value, Type targetType,
        object parameter, CultureInfo culture) {

        string text;
        if (!dict.TryGetValue((DateTime)value, out text))
            text = null;
        return text;
    }

    ...

}
The MainWindow.xaml file contains the template for CalendarDayButton with just a few additions. The first is the binding converter stored as a resource:
<ControlTemplate.Resources>
    <src:RedLetterDayConverter x:Key="conv" />
</ControlTemplate.Resources>
The outer Grid in the CalendarDayButton template then references that binding converter to display a ToolTip:
<Grid ToolTip="{Binding Converter={StaticResource conv},
Mode=OneWay}">
If that binding looks like it's missing a Path setting, keep in mind that the DataContext of the button is a DateTime object, and that DataContext is inherited through the visual tree. If you just wanted to display the DateTime in the ToolTip, you would use simply:
<Grid ToolTip="{Binding}">
The template also has an additional Rectangle object among the others used for highlighting the current day and selected days:
<Rectangle x:Name="RedLetterDayBackground"
    IsHitTestVisible="False"
    Fill="#80FF0000" />
A DataTrigger makes this Rectangle invisible when the binding converter returns null:
<DataTrigger
    Binding="{Binding
        Converter={StaticResource conv}}"
    Value="{x:Null}">
    <Setter TargetName="RedLetterDayBackground"
            Property="Visibility"
            Value="Hidden" />
</DataTrigger>
Figure 8 shows the three holidays for the month of March.
Figure 8 The RedLetterDays Display
In the more general case, the CalendarDayButton template visual tree can include any FrameworkElement derivative that has a dependency property of type DateTime, or even any FrameworkElement derivative that assumes its DataContext is of type DateTime.
A few readers of this column may recall the calendar I created for the January 2008 issue with the phases of the moon. The original MoonDisk class derived from Viewport3D to simulate a view of the moon illuminated by the sun. I eliminated the DateTime dependency property from that class and modified it slightly to cast its DataContext to a DateTime. The MoonPhaseCalendar project includes that MoonDisk class and references it in the visual tree of the CalendarDayButton template like so:
<Grid Width="48" Height="48">
    <ContentPresenter
        x:Name="NormalText"
        HorizontalAlignment="Left"
        VerticalAlignment="Top">
        <TextElement.Foreground>
            <SolidColorBrush x:Name="selectedText"
                Color="#FF333333"/>
        </TextElement.Foreground>
    </ContentPresenter>
    <src:MoonDisk Margin="6" />
</Grid>
Notice there's no binding on MoonDisk because it inherits the DataContext of type DateTime. The result for the month of July 2009 is shown in Figure 9.
Figure 9 The MoonPhaseCalendar Display

Deriving from Calendar
CalendarItem, CalendarButton, and CalendarDayButton are all sealed classes—you can't derive from them, and even if you could derive from CalendarButton and CalendarDayButton, you'd need to rewrite parts of CalendarItem to instantiate your new buttons rather than the old ones.
You can, however, derive from Calendar itself to provide some additional functionality. For example, you might want to enhance the Calendar control so that when you click on a particular day, a modeless dialog pops up, as shown in Figure 10, to let you enter and view a schedule for the day. This is the idea behind the DailyReminders project.
Figure 10 The DailyReminders Dialog Box
To implement this feature, the class that derives from Calendar (which I called DailyRemindersCalendar) must be able to detect when the user is clicking a CalendarDayButton. Normally this would be very straightforward: The constructor of the class that derives from Calendar would call AddHandler with arguments indicating the Mouse.ClickEvent and a method to handle these events. As you know, WPF implements routed events, so the Click event from any child button travels up the visual tree. The handler would discriminate among the various child buttons by checking the Source argument of the RoutedEvent object, and also by checking whether the DataContext is an object of type DateTime.
However, this proved not to be possible. The CalendarItem class installs handlers for the CalendarDayButton button-down and button-up events and disables the Click event.
Instead, I wrote DailyRemindersCalendar to override the OnMouseDoubleClick method. However, that raised another problem: for that event in a Calendar derivative, neither the Source nor OriginalSource properties of MouseButtonEventArgs is any type of Button. Source is null and OriginalSource is either a TextBlock or a Path or Rectangle.
Once again, however, the inheritance of the DataContext through the visual tree comes to the rescue. In the OnMouseDoubleClick override, if the OriginalSource object has a DataContext of type DateTime, the method knows that a CalendarDayButton is being clicked, and the DateTime object tells the method exactly which one. The method can then instantiate a DailyRemindersDialog, which builds its user interface (a Grid containing TextBlock and TextBox controls) entirely in code. (In fact, the entire DailyReminders program consists entirely of code with no markup.)
The DailyRemindersDialog is invoked modelessly, so you can simultaneously display dialogs for different days. However, the DailyRemindersDialog prohibits displaying multiple dialogs for the same day. That would raise issues involved with properly storing and retrieving these daily reminders. The DailyRemindersStorage class handles that part of the program by referencing an XML file in isolated storage. Whenever a DailyRemindersDialog closes, it causes that file to be saved with the updated information.

The DatePicker Control
I have focused exclusively on the new Calendar control rather than the DatePicker because DatePicker consists largely of a DatePickerTextBox and a dropdown that invokes the Calendar control.
This DatePickerTextBox is extremely crude. It certainly does not implement the feature I like so much in the DateTimePicker control in Windows Forms—clicking the individual components of the date or time (year, month, day, hour, and so forth) and using the cursor keys or numeric keys to change the values.
But all the styles and templates you can apply to the standalone Calendar control can also be applied to the Calendar control invoked from the dropdown in DatePicker. The DatePicker control has a property named CalendarStyle of type Style, and the Style object you set to this property can contain setters for any property defined by Calendar, including the CalendarItemStyle, CalendarButtonStyle, and CalendarDayButtonStyle properties.
Though a series of embedded styles and templates, you can in effect reach into both the Calendar control and DatePicker control and tailor them to your own needs and preferences.

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a longtime Contributing Editor to MSDN Magazine. His most recent book is The Annotated Turing: A Guided Tour through Alan Turing's Historic Paper on Computability and the Turing Machine (Wiley, 2008). His Web site is www.charlespetzold.com.


Concurrent Affairs
Solving The Dining Philosophers Problem With Asynchronous Agents
Rick Molloy

This article is based on a prerelease version of Visual C++ 2010. All information is subject to change.

Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
The Dining Philosophers
An Actor-Based Approach
A Solution In Five Classes
Message Blocks And Messages
Agents And The Join Message Block
Testing The Philosopher And Displaying State
Implementing The Table Class
Time For Lunch
Enabling C++ developers to write highly concurrent applications is a major focus of Visual Studio 2010. The beta release includes the Concurrency Runtime, parallel libraries, and development tools aimed at addressing several common problems preventing developers from unlocking the performance potential inherent to multicore hardware. Notably, this includes ensuring that developers can identify and take advantage of opportunities for concurrency in their code, productively manage shared state and its side effects, and not having to worry about building low-overhead concurrency infrastructure that is scalable at run time on a variety of hardware.
In this article, I'm going to demonstrate how to use the new Asynchronous Agents Library included as part of Visual C++ 2010 to manage the difficulties that can arise with shared state. To show you how this works, I will walk through an implementation of a classic concurrency problem: Djikstra's Dining Philosophers. You'll see how the actor-based programming construct of an agent in combination with asynchronous message-passing APIs can be used to provide a correct and easy to understand solution to this problem that doesn't rely directly on threading or synchronization primitives.

The Dining Philosophers
For those who aren't familiar with it, the Dining Philosophers problem is intended to illustrate the complexities of managing shared state in a multithreaded environment. Here's the problem:
At a round table sit five philosophers who alternate between thinking and eating from a large bowl of rice at random intervals. Unfortunately for the philosophers, there are only five chopsticks at the table (see Figure 1), and before the philosophers can begin eating, they must acquire a pair of chopsticks. Since the chopsticks are shared between the philosophers, access to the chopsticks must be protected, but if care isn't taken, concurrency problems arise. Most notably, starvation (pun intended) via deadlock can occur: if each philosopher picks up a chopstick as soon as it is available and waits indefinitely for the other, eventually they will all end up holding only one chopstick with no chance of acquiring another.
Figure 1 The Philosophers and Chopsticks
The Dining Philosophers problem is typically represented in code by a thread for each philosopher and some form of shared state used to represent each of the chopsticks. Straightforward solutions to this problem often involve introducing a waiter entity to coordinate access to the chopsticks, introducing lock ordering heuristics, and manually working with threading APIs, critical sections, semaphores, or monitors to manage the state.
Most developers will agree that building correct, nontrivial locking is considered challenging and fragile at best. As a result, most implementations of the Dining Philosophers problem tend to have implementation details leaked into them. For example, the Philosophers may be aware of the synchronization primitives or of their neighbors' chopsticks. This becomes particularly problematic if you want to generalize the problem to an arbitrary number of philosophers or refocus an implementation on the problem domain—such as what each philosopher does—rather than focusing on the synchronization and threading primitives.

An Actor-Based Approach
Let's walk through a design that is built on top of the Asynchronous Agents Library. Dining Philosophers really only has two moderately difficult sections: creating a thread for each philosopher to run in and coordinating the philosophers' access to the chopsticks. The Asynchronous Agents Library provides an actor-based programming model and asynchronous message passing APIs, and you'll need both of these in the implementation.
In the Asynchronous Agents Library, the agent class models an actor pattern and provides an asynchronous run method. I use an agent for each philosopher and take advantage of the run method to fulfill the requirement of each philosopher running in its own thread.
The second portion of a correct solution involves managing concurrent access to the chopsticks and, in contrast to traditional solutions using semaphores or locks, I'll use the message passing APIs in the Asynchronous Agents Library to move the chopsticks between the philosophers' hands and the table.
As each philosopher transitions between thinking and eating, he picks up the chopsticks and returns them to the table by sending messages. You'll see that in addition to providing some abstraction on top of threads and shared state, this will also allow the implementation to stay more focused on the domain of the problem than other solutions enable.

A Solution In Five Classes
The basic design is going to use five types:
  • A Chopstick class that is relatively self explanatory.
  • A ChopstickProvider class that will be used to hold the chopsticks on the table and provide them to the philosophers.
  • A Philosopher class that is responsible for thinking and eating and is aware only of its two ChopstickProviders.
  • A PhilospherState enum that can be thinking or eating.
  • A Table class that contains a set of Philosophers and a set of Chopsticks. Table is responsible for setting the table by placing a single Chopstick in a ChopstickProvider between each Philosopher. Figure 2 shows the relationships between the classes.
Figure 2 Classes Used In The Dining Philosophers Solution
Let's start with the simplest class: Chopstick. The purpose of the Chopstick is to simply exist as a chopstick and be able to identify itself. As a result, the implementation is a simple class that has an identifier member variable, a constructor that initializes the identifier, and a method to return the identifier.
class Chopstick{
  const std::string m_Id;
public:
  Chopstick(std::string && Id):m_Id(Id){};
  const std::string GetID() {
    return m_Id;
  };
};
It's important to note that the identifier field m_Id is a const member variable. I don't want the chopstick to have any state on its own that can get changed accidentally by another thread. Also notice that the constructor is using an r-value reference in its parameter list (the &&). This is a new language construct in Visual Studio 2010 that is part of the draft C++0x standard. An r-value reference here allows the compiler to avoid a copy constructor call and move Id into m_Id within the Chopstick instance if Id is a temporary variable or r-value.

Message Blocks And Messages
ChopstickProvider is also straightforward. It can be thought of as a storage buffer whose purpose is to accept Chopsticks when they are sent to it and to provide Chopsticks to a Philosopher upon request.
Here's the first place where I use the agent APIs. I use what's known as a message block for the ChopstickProvider. A message block is defined as a class that is able to send and receive messages. More specifically, if you're able to send a message to message block, the block is referred to as a target and, if you're able to receive a message from a message block, it's referred to as a source.
In this example, I'll need the ChopstickProvider to be both a source and a target because the Philosophers will be both receiving chopsticks from the ChopstickProviders when the Philosophers are hungry and sending them back to the ChopstickProviders when the Philosophers are ready to think again. Here's an example of putting them to work:
//create a chopstick
Chopstick chopstick("chopstick one");

//create a ChopstickProvider to store the chopstick
ChopstickProvider chopstickBuffer;

//put the chopstick in the chopstick holder
Concurrency::send(chopstickBuffer, &chopstick);

//request a chopstick from the chopstick buffer
Chopstick* result = Concurrency::receive(chopstickBuffer);
You can see that, after declaring a chopstick and a chopstickBuffer (an instance of the still undefined ChopstickProvider), Concurrency::send is used to place the address of chopstick into the ChopstickBuffer and Concurrency::receive returns a pointer to a Chopstick from the chopstickBuffer. Concurrency::send is a template method that synchronously places a message in a message block. Concurrency::receive is a template method that returns a message from a message block. Concurrency::receive blocks until a message is available in the message block.
The Asynchronous Agents Library also provides Concurrency::asend and Concurrency::try_receive. Concurrency::asend asynchronously spawns a task to send the message without waiting for acknowledgement of receipt. Concurrency::try_receive is a non-blocking call that will acquire a message if one is available.
Now let's actually define the ChopstickProvider. Remember I said it was straightforward? It's a typedef:
typedef Concurrency::unbounded_buffer<Chopstick*>
  ChopstickProvider;
The agents library provides a message block called the unbounded_buffer that has the functionality you need. An unbounded_buffer is a template class that is both a source and a target. It stores a memory-limited number of messages in an internal queue. Messages are removed from the queue when they are requested. You won't need the unbounded quality in this implementation of Dining Philosophers as there is a limited number of chopsticks that will be moving between each chopstick holder. The key functionality you're interested in from the unbounded_buffer is its move semantics.

Agents And The Join Message Block
Now let's take a look at implementing the Philosopher class. This is the interesting portion of the Dining Philosophers problem, where you ensure that each philosopher is running on its own thread and that access to the shared resource is coordinated appropriately. The Philosopher class will use two additional constructs from the Asynchronous Agents Library, the abstract agent class and the join construct. First I'm going to describe the agent and how it's used.
The agent class is an abstract class that models what is sometimes referred to as an actor pattern or actor-based programming. Agents are intended to be used to facilitate removing dependencies and shared state between active components of an application by having the agents encapsulate state and communicate with each other solely via message passing through a small set of public interfaces. This sounds more complicated than it is, but hopefully you can see the similarities to the Philosopher class.
As I mentioned earlier, the Philosopher class needs to be running on its own thread. Translating this to the terminology of agents this means it needs to be an active task, so Philosopher is going to derive publicly from Concurrency::agent. The Philosopher class also needs to be aware of two ChopstickProviders (one for each hand) and then use them to transition successfully between thinking and eating. The Philosopher class isn't complete yet, but you can see how this is starting to translate into a class synopsis:
enum PhilosopherState {Thinking,Eating};

typedef Concurrency::unbounded_buffer<Chopstick*> ChopstickProvider;
class Philosopher : public Concurrency::agent {
  ChopstickProvider* m_LeftChopstickProvider;
  ChopstickProvider* m_RightChopstickProvider;

public:
  const std::string m_Name;
  const int  m_Bites;
  Philosopher(const std::string&& name,
    int bites=500):m_Name(name),
    m_Bites(bites){};
...
}
Philosopher has a pointer to two ChopstickProviders, a name, the number of bites or turns the Philosopher will take, and a constructor for initialization.
What is missing, though, is a way to initialize the ChopstickProviders in the agent. You don't want to initialize them at construction time because the Philosophers are independent from the Chopsticks and their ChopstickProviders. You could create an additional public method like AssignChopsticks (ChopstickProvider* left, ChopstickProvider* right), but then you would have to take some care—or a lock—to ensure that this method is thread safe.
Instead, I'm going to create a public interface so that the ChopstickProviders can be assigned asynchronously. I do this by adding two more public unbounded_buffer member variables and use these to send the Philosophers two ChopstickProviders:
Concurrency::unbounded_buffer<ChopstickProvider*>
  LeftChopstickProviderBuffer;
Concurrency::unbounded_buffer<ChopstickProvider*>
  RightChopstickProviderBuffer;
Now I'm ready to start implementing the Philosopher method that will run in its own thread, wait for the ChopstickProviders to be initialized, and then start alternating between thinking and eating. I implement the virtual run method from the agent base class. agent::run will be started in its own task when a call to agent::start is made.
void run() {
The first thing I need to do inside run is initialize the ChopstickProviders by waiting for messages to be sent on the public methods with receive:
//initialize the ChopstickProviders
m_LeftChopstickProvider  =
  Concurrency::receive(LeftChopstickProviderBuffer);
m_RightChopstickProvider =
  Concurrency::receive(RightChopstickProviderBuffer);
Now I need to transition between thinking and eating. To do this I'm going to use two additional methods, PickupChopsticks and PutDownChopsticks:
for(int i = 0; i < m_Bites;++i) {
  Think();
  std::vector<Chopstick*> chopsticks(PickupChopsticks());
  Eat();
  PutDownChopsticks(chopsticks);
}
The only thing remaining to implement in the run method is to clean up by returning the ChopstickProviders so that they can be reused if needed, and to set the agent status to done.
Concurrency::send(LeftChopstickProviderBuffer,
  m_LeftChopstickProvider);
Concurrency::send(RightChopstickProviderBuffer,
  m_RightChopstickProvider);

this->done(Concurrency::agent_done);
}
Moving on, the additional methods Eat and Think are straightforward. In the original problem they are described as random spin loops:
private:
  void Eat() {
    RandomSpin();
  };

  void Think() {
    RandomSpin();
  };
};
Implementing PickupChopsticks is significantly more interesting because it manages acquiring and releasing the chopsticks without introducing deadlocks or race conditions. To implement this I'm going to use another messaging block from the Asynchronous Agents Library called join. Concurrency::join is a message block that waits for messages from multiple sources, potentially of varying types. Once all messages have been received, it produces a conglomerate message. A join can support sources of the same or multiple types and will acquire messages in a greedy or non-greedy manner. This means that there are four variants of join in the agents library. The Philosopher class will use a single-type non-greedy join. Figure 3 shows a simple example of a join in code.
Figure 3 A Simple Join
//create two chopsticks
Chopstick chopstick1("chopstick one");
Chopstick chopstick2("chopstick two");

//create ChopstickProviders to store the chopstick
ChopstickProvider chopstickBuffer1;
ChopstickProvider chopstickBuffer2;

//put a chopstick in each chopstick holder
Concurrency::send(chopstickBuffer1, &chopstick1);
Concurrency::send(chopstickBuffer2, &chopstick2);

//declare a single-type non greedy join to acquire them.
//the constructor parameter is the number of inputs
Concurrency::join<Chopstick*,non_greedy> j(2);

//connect the chopstick providers to the join so that messages
//sent will propagate forwards
chopstickBuffer1.link_target(&j);
chopstickBuffer2.link_target(&j);

//the single type join message block produces a vector of Chopsticks
std::vector<Chopstick*> result = Concurrency::receive(j);
Now I'm ready to implement PickupChopsticks. The first thing to notice is that you return a set of chopsticks by returning a vector. To get a pair of chopsticks from the chopstick provider means using a non-greedy join. The non-greedy variant of join waits for an offer of messages from all linked sources, and then, once it has an offer from each source, confirms that it can actually take ownership of the message. This is what prevents deadlock in this example. If I had used greedy as the template parameter for the join, the message would be taken as soon as an offer is made, and sooner or later each Philosopher would have a single chopstick and they would all be deadlocked.
Here's the code for PickupChopsticks. I create a join, link it to the chopstick providers, and wait for the chopsticks to arrive:
std::vector<Chopstick*> PickupChopsticks() {
  //create the join
  Concurrency::join<Chopstick*,Concurrency::non_greedy> j(2);
  m_LeftChopstickProvider->link_target(&j);
  m_RightChopstickProvider->link_target(&j);

  //pickup the chopsticks
  return Concurrency::receive (j);
}
Implementing PutDownChopsticks is straightforward as well. All I need to do is return the chopsticks that I've acquired from the vector using the asynchronous method asend:
void PutDownChopsticks(std::vector<Chopstick*>& v) {
  Concurrency::asend(m_LeftChopstickProvider,v[0]);
  Concurrency::asend(m_RightChopstickProvider,v[1]);
};

Testing The Philosopher And Displaying State
The Philosopher class can be used as is, but you need to understand how to start the Philosopher class and have a way of reporting on its status—whether it is eating or thinking.
To start the philosopher, call the agent::start method, which spawns a task that invokes the virtual run method. To report on the status, I introduce two more message blocks from the agents library: Concurrency::overwrite_buffer and Concurrency::call.
Concurrency::overwite_buffer<T> is a message block that stores a single value that can be overwritten, and it produces a copy of the value when requested. As with the other message blocks, overwrite_buffer can be linked to additional targets, and message propagation happens in order. I'll add a public member variable to the Philosopher class that is an overwrite_buffer<PhilosopherState> and update it by sending messages to it as the Philosopher transitions from Thinking to Eating. Specifically, this will involve adding a member variable to the Philosopher class:
Concurrency::overwrite_buffer<PhilosopherState> CurrentState;
It also means modifying Eat and Think to update status:
void Eat() {
  send(&CurrentState,PhilosopherState::Eating);
  RandomSpin();
};

void Think() {
  send(&CurrentState,PhilosopherState::Thinking);
  RandomSpin();
};
Concurrency::call<T> is a message block that is constructed with a functor and spawns a task to execute the functor when it receives the message. You can use the call in combination with the newly defined overwrite_buffer on the Philosopher to report on its state, as shown in Figure 4.
Figure 4 Reporting Philosopher State
//create an instance of a Philosopher and have it take 5 bites
Philosopher Descartes("Descartres",5);

//create a pair of chopsticks
Chopstick chopstick1("chopstick1");
Chopstick chopstick2("chopstick2");

//create a pair of ChopstickProviders
ChopstickProvider chopstickBuffer1;
ChopstickProvider chopstickBuffer2;

//associate our chopstick providers with Descartes
Concurrency::send(&Descartes.LeftChopstickProvider, chopstickBuffer1);
Concurrency::send(&Descartes.RightChopstickProvider, chopstickBuffer2);

//start the Philosopher asynchronously
agent::start(&Descartes);

//declare a call that will be used to display the philosophers state
//note this is using a C++0x lambda
Concurrency::call<PhilosopherState> display([](PhilosopherState state){
  if (state ==  Eating)
    std::cout << "eating\n" ;
  else
    std::cout << "thinking\n" ;
});

//link the call to the status buffer on our Philosopher
Descartes.CurrentState .link_target (&display);

//start our Philosopher eating / thinking by sending chopsticks
asend(&chopstickBuffer1,&chopstick1);
asend(&chopstickBuffer2,&chopstick2);
And that's it for the Philosopher class.

Implementing The Table Class
You've already seen everything needed to create the Table class. As mentioned previously, it will contain a set of Philosophers, a set of Chopsticks, and a set of ChopstickProviders. The Table will be responsible for seating the Philosophers at the table, placing a ChopstickProvider between each of them, and placing a Chopstick in each of the ChopstickProviders. For flexibility I've chosen to make it a template class, and it contains a reference to a list of Philosophers, a vector of Chopsticks, and a vector of Chopstick Holders.
template<class PhilosopherList>
class Table {
  PhilosopherList & m_Philosophers;
  std::vector<ChopstickProvider*> m_ChopstickProviders;
  std::vector<Chopstick*> m_Chopsticks;
  ...
The only public method in the Table class is the constructor, which initializes the vectors and assigns ChopstickProviders to each Philosopher (see Figure 5).
Figure 5 Table Constructor
Table(PhilosopherList& philosophers): m_Philosophers(philosophers) {
  //fill the chopstick and the chopstick holder vectors
  for(size_t i = 0; i < m_Philosophers.size();++i) {
    m_ChopstickProviders.push_back(new ChopstickProvider());
    m_Chopsticks.push_back(new Chopstick("chopstick"));

    //put the chopsticks in the chopstick providers
    send(m_ChopstickProviders[i],m_Chopsticks[i]);
  }

  //assign the philosophers to their spots
  for(size_t leftIndex = 0;
    leftIndex < m_Philosophers.size();++leftIndex) {

    //calculate the rightIndex
    int rightIndex = (leftIndex+1)% m_Philosophers.size();

    //send the left & right chopstick provider to the appropriate Philosopher
    Concurrency::asend(& m_Philosophers[leftIndex].LeftChopstickProviderBuffer,
      m_ChopstickProviders[leftIndex]);
    Concurrency::asend(& m_Philosophers[leftIndex].RightChopstickProviderBuffer,
      m_ChopstickProviders[rightIndex]);
  }
}

~Table(){
  m_ChopstickProviders.clear();
  m_Chopsticks.clear();
}

Time For Lunch
Finally, to implement main, I need to declare the list of Philosophers, create a Table, start the Philosophers, hook up a reporting mechanism, and wait for them to eat (see Figure 6).
Figure 6 Mealtime
int main() {
  //create a set of Philosophers using the tr1 array
  std::tr1::array<Philosopher,5> philosophers =
    {"Socrates", "Descartes", "Nietzsche", "Sartre", "Amdahl"};
  Table<std::tr1::array<Philosopher,5>> Table(philosophers);

  //create a list of call blocks to display
  std::vector<Concurrency::call<PhilosopherState>*> displays;

  //start the Philosophers and create display item
  std::for_each(
    philosophers.begin(),philosophers.end(),[&displays](Philosopher& cur) {

    //create a new call block to display the status
    Concurrency::call<PhilosopherState>* consoleDisplayBlock =
      new Concurrency::call<PhilosopherState>([&](PhilosopherState in){

      //cout isn't threadsafe between each output
      if(in == Eating)
        std::cout << cur.m_Name << " is eating\n";
      else
        std::cout << cur.m_Name << " is  thinking\n";
    });

    //link up the display and store it in a vector
    cur.CurrentState.link_target(consoleDisplayBlock);
    displays.push_back(consoleDisplayBlock);

    //and start the agent
    cur.start();
  });

  //wait for them to complete
  std::for_each(
    philosophers.begin(),philosophers.end(),[](Philosopher& cur) {

    cur.wait(&cur);
  });

  displays.clear();

  return 1;
};
Hopefully, it's clear how the agent class and asynchronous message passing that the Asynchronous Agents Library provides can help avoid the difficulties of coordinating access to shared state. I've managed to implement the Dining Philosophers problem without using a single explicit lock and without having to call any threading APIs directly. I've also managed to keep the implementation within the terms of the domain—specifically, I've spent time working with Chopsticks, Philosophers, and Tables as opposed to arrays of integers and semaphores.
What I haven't done yet is added any inherent scalability to the Philosophers. This application as written won't run significantly faster on any more than four cores. But this can be fixed by replacing RandomSpin with something more meaningful, like an algorithm implemented with task-based parallelism using the Parallel Pattern Library (PPL). I encourage you to read about this on the native concurrency blog , where I've got the source code for this article, but where I've also added a few fine-grained parallel thinking algorithms implemented to demonstrate composability between the PPL and the agents library. I've also used the Concurrency Runtime's resource management features to manage the quality of service issues that can arise with an abundance of parallel work. When processing resources get scarce, I can ensure that the Philosophers still are able to get enough to eat.

Send your questions and comments to mmsync@microsoft.com.

Rick Molloy is a Program Manager on the Parallel Computing Platform team at Microsoft.


Windows With C++
Introducing Direct2D
Kenny Kerr

This column is based on a prerelease version of Windows 7. Details are subject to change.
Contents
Architecture And Principals
Factories And Resources
Render Targets
Brushes And Drawing Commands
With the introduction of Windows Vista a few years ago, it was clear that the era of the Windows Graphics Device Interface (GDI) was coming to an end. GDI, even with the help of GDI+, had been showing its age especially when compared to the superior graphics capabilities of Windows Presentation Foundation (WPF). As if that weren't enough, GDI lost its hardware acceleration while WPF took advantage of Direct3D's abundance of hardware acceleration.
However, if you want to develop high-performance and high-quality commercial applications, you'll still look to C++ and native code to deliver that power. That's why few things excite me more about Windows 7 than the introduction of Direct2D and DirectWrite. Direct2D is a brand new 2D graphics API designed to support the most demanding and visually rich desktop applications with the best possible performance. DirectWrite is also a brand-new API that complements Direct2D and provides hardware-accelerated text, when used with Direct2D, and high-quality text layout and rendering with advanced support for OpenType typography and ClearType text rendering.
In this article, I will explore these new technologies and give you an idea of why it matters and how you can start using them today.

Architecture And Principals
On Windows XP, GDI and Direct3D had equal standing as far as the operating system was concerned. The user-mode portions of GDI and Direct3D interfaced directly with their kernel-mode counterparts. As a result, both could be directly hardware accelerated, given suitable display drivers. With Windows Vista, Microsoft delegated the responsibility of controlling video hardware exclusively to Direct3D. GDI suddenly became the legacy graphics API that was supported predominantly through software-based rendering on top of Direct3D.
So GDI is dead and Direct3D is the future, but where does this leave us and how does this all relate to Direct2D and WPF? Well, the first thing to note is that Direct3D is effectively the Windows graphics processing unit (GPU) API, whether you want to render graphics or hotwire your GPU for added computing power. Ignoring OpenGL (which is still supported), Direct3D is the lowest-level graphics pipeline for directly controlling a graphics display adapter from a user-mode application.
Direct3D is what is known as an immediate mode graphics API. This simply means that the API provides a thin layer over any graphics hardware, providing access to the hardware's features while filling in some of the blanks should the hardware be limited in some way. Every time you want to render or update the display, you effectively need to tell Direct3D that you are about to render, supply the Direct3D pipeline with everything it needs to render a frame, and then tell it that you're finished, and it will cause the display to be updated. Although Direct3D provides many advanced 3D functions, it is up to you to directly control all of them and it has relatively few 3D primitives. Needless to say, this is not a trivial task.
Prior to Windows Vista, Direct3D also included a higher-level retained-mode graphics API that was built on top of the immediate-mode API. This API provided direct support for manipulating 3D objects, called scenes—a hierarchy of frames with objects and lighting. It was called retained mode because the API retains a copy of the entire scene graph, so to speak. Applications simply update the scene, and the API automatically takes care of rendering and updating the display. The trouble, of course, is that all of this comes at a cost; if the cost is too high or the abstractions don't quite meet your requirements, you have to discard this technique entirely, rely on the immediate mode API directly, and provide your own geometry algorithms.
If you're at all familiar with WPF, then the previous description of a retained-mode graphics API should sound quite familiar. Although Direct3D discontinued its retained-mode API, WPF includes its own internal retained-mode API, known as the Media Integration Layer (MIL). MIL sits between WPF and Direct3D and retains a copy of the scene graph defined by WPF, allowing managed code to interact with visual objects while remaining in the background, coordinating with MIL to ensure that changes are reflected in the scene graph that it retains. This allows WPF to provide a very rich and interactive experience for developers, but it comes with a price.
Having said all that, it should be fairly obvious where Direct2D comes in. Direct2D provides rich rendering support for simple and complex geometries, bitmaps, and text built directly on top of Direct3D's immediate-mode graphics API for unparalleled performance. The result is a high-performance and low-overhead approach for producing high-quality graphics content for your applications. It provides a dramatically simpler API for producing 2D content compared to using Direct3D directly while adding little overhead in most cases. In fact, with the exception of a few specific operations such as per-primitive anti-aliasing, you'd be hard pressed to outperform Direct2D with Direct3D. Direct2D should also easily outperform the likes of MIL and Quartz2DE.
As if that weren't enough, it goes even further and provides many additional features, such as remote rendering over Remote Desktop Protocol (RDP), software fallback for server-side rendering or to compensate for a lack of hardware, support for rendering ClearType text, and unparalleled interoperability with both GDI and Direct3D. Let's just say there's none of this WPF "airspace" nonsense.
Now let's get down to writing some code! To keep the examples focused, I'm going to use the Active Template Library (ATL) and the Windows Template Library (WTL) to handle all of the boilerplate windowing code for me. Remember, Direct2D is about rendering. If you want it to render to a window, you still need to manage that window. If you want mouse input, you still need to respond to window messages as usual. You get the idea. Figure 1 contains a skeleton of a window that I'll fill out during the rest of this article.
Figure 1 Window Skeleton
#define HR(_hr_expr) { hr = _hr_expr; if (FAILED(hr)) return hr; }
class Window : public CWindowImpl<Window, CWindow, CWinTraits<WS_OVERLAPPEDWINDOW>>
{
public:
    BEGIN_MSG_MAP(Window)
        MSG_WM_DESTROY(OnDestroy)
    END_MSG_MAP()

    HRESULT Create()
    {
        VERIFY(__super::Create(0)); // top-level
        VERIFY(SetWindowText(L"Direct2D Sample"));

        VERIFY(SetWindowPos(0, // z-order
                            100, // x
                            100, // y
                            600, // pixel width,
                            400, // pixel height,
                            SWP_NOZORDER | SWP_SHOWWINDOW));

        VERIFY(UpdateWindow());
        return S_OK;
    }
private:
    void OnDestroy()
    {
        ::PostQuitMessage(1);
    }
};

Factories And Resources
As with various other APIs like Direct3D and XmlLite, Direct2D uses a lightweight version of the COM specification to manage object lifetime through interfaces derived from IUnknown. There's no need to initialize the COM run time and worry about apartments or proxies. It's just a convention to simplify resource management and allow APIs and applications to expose and consume objects in a well-defined way. Keep in mind that just because Direct2D uses COM interfaces does not mean that you can provide your own implementations of those interfaces. Unless otherwise noted, Direct2D will work only with its own implementations. I would suggest that you use ATL's CComPtr smart pointer class for managing the interface pointers, as I do in the examples in this article.
Every Direct2D application begins by creating a factory object. The D2D1CreateFactory function returns an implementation of the ID2D1Factory interface:
CComPtr<ID2D1Factory> factory;
HR(D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, &factory));
Of course, before you can use any of these types, you need to include d2d1.h, the Direct2D header file, which declares the various Direct2D interfaces, structures, and constants as well as the D2D1CreateFactory function. The d2d1.lib file is provided to import the D2D1CreateFactory function. The d2d1helper.h header file also provides a great many helpful functions and classes to simply the use of Direct2D from C++. I normally include Direct2D in a project by adding the following directives in the project's precompiled header:
#include <d2d1.h>
#include <d2d1helper.h>
#pragma comment(lib, "d2d1.lib")
As I mentioned, Direct2D does not rely on COM, but the first parameter to D2D1CreateFactory might lead you to believe otherwise. The D2D1_FACTORY_TYPE enum defines both D2D1_FACTORY_TYPE_SINGLE_THREADED and D2D1_FACTORY_TYPE_MULTI_THREADED constants, but these have nothing to do with COM apartments and don't assume any thread affinity. The D2D1_FACTORY_TYPE_SINGLE_THREADED constant merely specifies that the factory object, as well as any and all objects rooted in that factory, may be accessed by only a single thread at a time. Objects created in this way will provide the best performance for single-threaded access, avoiding any unnecessary serialization into and out of the rendering calls. The D2D1_FACTORY_TYPE_MULTI_THREADED constant, on the other hand, specifies that the factory object, as well as any and all objects rooted in that factory, may be accessed by multiple threads concurrently. Direct2D provides the necessary synchronization with interlocked reference counting. This is useful for rendering to different targets concurrently while sharing certain resources. Keep in mind that this parallelism is specific to the CPU, and instructions sent to the GPU may still be serialized and ultimately parallelized independently.
OK, so what is the factory object used for? Well, it is responsible for creating all device-independent resources, predominantly geometries, as well as creating render targets representing devices. The render target then has the responsibility of creating device-dependent resources. The distinction between device-dependent and device-independent resources is critical for using Direct2D correctly. One of the benefits of a retained-mode graphics API such as WPF is that the intermediate layer between the hardware and the programming model generally takes care of scenarios where the display device may be lost. This may happen for a variety of reasons. The resolution may change in the background, a display adapter may be removed such as when a user undocks a laptop, a remote desktop connection session may be lost, and so on. An immediate-mode graphics API like Direct2D needs to take into account these events. Of course, the benefit is that resources used to render to a particular device may be physically present on the device and thus provide far better performance.
For these reasons, it is good practice to clearly separate the creation of device-independent resources from device-dependent resources. To accommodate this, in Figure 2 I add these two methods to the Window class from Figure 1.
Figure 2 Create Device-Independent and Device-Dependent Resources Separately
HRESULT CreateDeviceIndependentResources()
{
    HRESULT hr;
    HR(D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED, &m_factory));
    // TODO: Create device-independent resources here
    return S_OK;
}
HRESULT CreateDeviceResources()
{
    HRESULT hr;
    // TODO: Create device resources here
    return S_OK;
}
I'll flesh them out in the next few sections, but for the time being, you can call CreateDeviceIndependentResources from the Create method of the Window class because device-independent resources need be created only once. CreateDeviceResources will, however, be called on demand to create device resources as needed. You'll also want to add the m_factory member variable as follows:
CComPtr<ID2D1Factory> m_factory;

Render Targets
A render target is used to represent a device and is itself dependent on the underlying device. With a render target, you can create various resources, such as brushes, and perform the actual drawing operations. Direct2D supports a number of different types of render targets. If you are building a Direct2D application from scratch, you might create a render target to render content to a window (HWND). You can also create a render target to render to a GDI device context (DC) or to a DirectX Graphics Infrastructure (DXGI) surface for use in a Direct3D application. You can even create a render target for various types of bitmap resources for off-screen rendering.
Various factory methods are provided for creating the different types of render targets. The methods themselves are pretty obvious. For example, the CreateHwndRenderTarget method creates a window render target, and the CreateDxgiSurfaceRenderTarget method creates a DXGI surface render target. To render to the window, you're going to need to update the CreateDeviceResources method to create the window render target. At the very minimum, it might look something like Figure 3. Keep in mind that this sample is not DPI-aware, but that is a topic for another article.
Figure 3 Create the Window Render Target
HRESULT CreateDeviceResources()
{
    HRESULT hr;
    if (0 == m_target)
    {
        CRect rect;
        VERIFY(GetClientRect(&rect));
        D2D1_SIZE_U size = D2D1::SizeU(rect.Width(), rect.Height());
        HR(m_factory->CreateHwndRenderTarget(D2D1::RenderTargetProperties(),
                        D2D1::HwndRenderTargetProperties(m_hWnd, size), &m_target));
    }
    return S_OK;
}
You'll also want to add the m_target member variable:
CComPtr<ID2D1HwndRenderTarget> m_target;
The CreateDeviceResources method creates the device resources only if the render target has not yet been created or, as you will see shortly, if the device is lost and the render target needs to be re-created.
The CreateHwndRenderTarget method's first parameter is actually of type D2D1_RENDER_TARGET_PROPERTIES. This parameter is common to all the render target creation functions. The RenderTargetProperties function is actually a helper function defined in the d2d1helper.h header file. It represents a common pattern for initializing many of the data structures used by Direct2D and helps to simplify your code dramatically. The RenderTargetProperties helper function has parameters with default values to account for the most common initialization of the resulting structure. You can override these values to adjust the pixel format and DPI information, as well as additional characteristics of the render target, such as whether to force hardware- or software-based rendering and whether to support GDI interop.
Likewise, CreateHwndRenderTarget's second parameter is actually a D2D1_HWND_RENDER_TARGET_PROPERTIES structure and provides information specific to the window.
Having created the render target, I can now do some actual rendering. To do this, add a Render method to the Window class with the basic outline shown in Figure 4.
Figure 4 Rendering
void Render()
{
    if (SUCCEEDED(CreateDeviceResources()))
    {
        if (0 == (D2D1_WINDOW_STATE_OCCLUDED & m_target->CheckWindowState()))
        {
            m_target->BeginDraw();
            m_target->SetTransform(D2D1::Matrix3x2F::Identity());
            m_target->Clear(D2D1::ColorF(D2D1::ColorF::Red));
            // TODO: add drawing code here

            if (D2DERR_RECREATE_TARGET == m_target->EndDraw())
            {
                DiscardDeviceResources();
            }
        }
    }
}
The logic here is quite important because it enables very efficient updating, thanks to the reuse of device resources and because it must anticipate device loss. The Render method starts by calling the CreateDeviceResources method that I've just described. Of course it won't have to do anything if the device resources are still available. As an optimization, I check the window state to ensure that it is not occluded, which is just a fancy term that indicates the window is obstructed from view and any painting would just be a waste of precious CPU and GPU resources. Although this rarely happens when the Desktop Window Manager (DWM) is handling desktop composition, it doesn't hurt and is a nice optimization when desktop composition is disabled.
Actual drawing must be sandwiched between calls to the render target's BeginDraw and EndDraw methods. Most of the render target's methods have a void return type. Since drawing operations are batched, any failures will be detected only when the drawing operations are flushed and rendered to the device. The EndDraw method thus has an HRESULT return type. It uses the D2DERR_RECREATE_TARGET constant to indicate that the render target has been invalidated and must be recreated. It then calls DiscardDeviceResources, which must release all the device resources.
void DiscardDeviceResources()
{
    m_target.Release();
}
I should also just mention the calls to the render target's SetTransform and Clear methods. SetTransform in this case specifies that any subsequent drawing operations should be transformed using an identity matrix. Of course, an identity matrix is really no translation at all, so this just assures that drawing operations use the literal coordinate space in device-independent pixels. You can, however, call SetTransform repeatedly during rendering to change the transform on the fly. The Clear method simply clears the drawing area for subsequent drawing operations and in this case makes it red.
Of course, if you were to create the window at this point, you would notice that the window remains white. Something still needs to call the Render method to update the display. For this, you can add message handlers for the WM_PAINT and WM_DISPLAYCHANGE window messages. You can do that by adding the following to the message map:
MSG_WM_PAINT(OnPaint)
MSG_WM_DISPLAYCHANGE(OnDisplayChange)
Responding to WM_PAINT is obvious, but handling WM_DISPLAYCHANGE is equally important to ensure that the window is properly repainted should the display resolution or color depth change (see Figure 5).
Figure 5 Repainting
void OnPaint(CDCHandle /*dc*/)
{
    PAINTSTRUCT paint;
    VERIFY(BeginPaint(&paint));
    Render();
    EndPaint(&paint);
}
void OnDisplayChange(UINT /*bpp*/, CSize /*resolution*/)
{
    Render();
}
Notice that the DC returned by BeginPaint is not used at all. Creating the window will now result in a window with a red background, as expected. Resizing the window will however, reveal some flickering. What's happening is that the window background is still automatically being cleared using the window class background brush because of the default handling of the WM_ERASEBKGND message. To avoid this, simply handle this message and return TRUE in the OnEraseBackground method to indicate that you will take care of clearing the window background. You can now resize the window while avoiding any flickering.
While you're at it, you will probably want to handle window resizing. To do so, add a message handler for the WM_SIZE window message, as follows:
MSG_WM_SIZE(OnSize)
The OnSize handle then needs to simply inform the render target that the size has changed:
void OnSize(UINT /*type*/, CSize size)
{
    if (0 != m_target)
    {
        if (FAILED(m_target->Resize(D2D1::SizeU(size.cx, size.cy))))
        {
            DiscardDeviceResources();
            VERIFY(Invalidate(FALSE));
        }
    }
}
If it fails to resize, you simply discard the device resources, and they will automatically be recreated the next time the window is rendered. Notice how the Resize method accepts the new size in device pixels since the WM_SIZE message communicates the size of the client area in pixels. Even though Direct2D uses a device-independent coordinate system, the window render target understands that it ultimately needs to map to device pixels.

Brushes And Drawing Commands
To do any meaningful drawing, you'll want to create some brushes. You will need brushes to paint various geometric shapes as well as text. Brushes fall in the category of device-dependent resources, so their creation should go in the CreateDeviceResources method. As you might expect, Direct2D provides solid and bitmap brushes as well as linear and radial gradient brushes. To create a solid color brush, you can add the following method call right after creating the render target. That way, it is re-created only when the render target is recreated.
HR(m_target->CreateSolidColorBrush(D2D1::ColorF(D2D1::ColorF::Black), &m_brush));
Remember to also release the brush in the DiscardDeviceResources method as follows:
m_brush.Release();
You'll also want to add the m_brush member variable as follows:
CComPtr<ID2D1SolidColorBrush> m_brush;
Figure 7 Basic Drawing
You can now add some basic drawing commands to the Render method between the Clear and EndDraw methods. The code in Figure 6 produces a circle with a diagonal line through it. Although I'll leave a complete discussion of brushes for a future article, it is worth mentioning that Direct2D allows you to efficiently change the color of a brush while rendering, obviating the need to create multiple brushes for different colors.
Figure 6 Circle with Diagonal Line
m_target->DrawLine(D2D1::Point2(10.0f, 10.0f), // start
                   D2D1::Point2(200.0f, 200.0f), // end
                   m_brush,
                   10.0f); // stroke width

const D2D1_POINT_2F center = D2D1::Point2(105.0f, 105.0f);

const D2D1_ELLIPSE ellipse = D2D1::Ellipse(center,
                                           95.0f, // radius X
                                           95.0f); // radius Y

m_target->DrawEllipse(&ellipse,
                      m_brush,
                      5.0f); // stroke width
As I've mentioned, Direct2D uses a device-independent coordinate system so the coordinates expressed don't necessarily map to display pixels but rather will honor the DPI setting for the target device. Figure 7 shows the result of the drawing commands.
Although I've used only an opaque brush, you can also specify an alpha value when creating the brush and alpha blend to your heart's content.
Unfortunately, I've completely run out of space, but you should now have a good idea of the potential that Direct2D and DirectWrite will have on native applications going forward. There are many more reasons to love these new technologies that I hope to dig into in an upcoming column, including rich geometries and operations as well as all the capabilities of DirectWrite. In the meantime, start experimenting with Direct2D. I'm sure you'll find it impressive.
Insights: Direct2D Rendering

Many people will be tempted to contrast the performance of Direct2D and Direct3D applications. This is something of an apples to oranges comparison in that Direct2D provides a set of primitives much more comparable to other 2D rendering APIs such as GDI/GDI+ or WPF. It also is much simpler to use. However, it might be surprising to some customers that Direct2D can actually out-perform a simple, or naïve, Direct3D application in a number of different scenarios. For example, Direct2D can outperform a naïve Direct3D app when drawing large numbers of lines with different colors or when blending repeatedly from a bitmap.
The key to how Direct2D does this is to realize that Direct2D does not maintain a 1:1 relationship between a drawing request issued to it and the primitives that it issues to Direct3D. In fact, Direct2D tries to aggregate its requests to Direct3D in a number of different ways.
Reduced Vertex Buffer Mapping
Consider a naïve Direct3D application. It creates a vertex buffer, maps it, writes its geometry in the buffer, un-maps it and draws it. The problem with this approach is that if the vertex buffer is still in use by the GPU when the next map request comes in, the CPU will have to stall until the GPU is done with it, and then it can only be mapped back into the application memory. This can dramatically decrease the application's performance.
Direct2D solves this problem by leaving the vertex buffer mapped into memory and accumulating many successive primitives' geometry into it. Only when the vertex buffer is full, or the application calls Flush or EndDraw, does Direct2D then send the draw calls down into Direct3D. This minimizes the number of GPU stalls because the number of map and un-map calls is greatly reduced.
Coalescing Draw Calls
Even after geometries are allowed to accumulate in the vertex buffer, a naïve Direct3D application would have to issue a Draw call to Direct3D for most Direct2D Draw calls. Consider two different rectangles rendered with different colors, each color might require writing different shader constant data to represent it. Each time the shader constant data is updated, a different Draw call needs to be issued.
So long as certain other constants do not change. For example, the same type of brush is used consecutively. Or, if a bitmap brush, the same input bitmap is used consistently. Direct2D can use its vertex shaders to continue to accumulate shader constant data and then issue one Direct3D Draw call for a large number of Direct2D Draw calls.
Out-of-Order Text Rendering
The Direct2D text rendering pipeline is executed entirely in hardware, through three stages. The first stage writes the glyphs into a texture, the next stage down-samples these glyphs and the final stage performs a clear-type filtering operation to transfer the text to the render target. A naïve application would render each stage sequentially, changing the render target three times and issuing a Draw call to move the data between each texture. Direct2D will render the text primitives out of order with other primitives. It will first accumulate a large number of glyphs. It then down-samples all of them and blends the final set of glyphs in order with the other geometric primitives issued to Direct2D. Coalescing the text rendering in this way reduces the number of times the render target changes as well as the number of general Direct3D device state changes in order to render the text, even if other primitives are interleaved with the Direct2D text rendering calls.
By combining vertex buffer batching, draw call coalescing, and out-of-order rendering of text, Direct2D is capable of achieving rendering performance that would require extensive amount of development effort to match if the application were instead directly targeting Direct3D.
--Mark Lawrence, Senior Software Development Engineer, Microsoft.

Send your questions and comments to mmwincpp@microsoft.com

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. You can reach Kenny at weblogs.asp.net/kennykerr.


{End Bracket}
Think Before You Speak
Steven M. List

"What we've got here is failure to communicate." Remember that line from Cool Hand Luke? It's one of the great movie lines of all time, I think. And it's something I keep in mind when working with development teams, particularly agile teams.
At the core of any effectively functioning team is an ability to communicate effectively. Effective communication is frequently misunderstood to mean that you should say what you're feeling or thinking, when you're thinking it or feeling it, and let the other guy just deal with it. Or, hold it in until you're ready to explode, and then explode all over everyone. Well, needless to say, neither of these represent effective communication.
So how do I define effective communication? Any communication in which meaning is exchanged, people are treated with respect, and some value is created can be effective.
Exchanging meaning means moving away from pronouncements such as "that won't work"—and engaging in dialogue by tempering the pronouncements—"I think that won't work. Have you considered this?" It means dropping the emotionally charged terms like "that's just stupid!" and moving toward terms that have contextual meaning: "I think that won't work, given the goals of the project. What do you think?"
Treating people with respect is hard for some people. Let me be clear. Whether or not someone has earned your respect is separate from how you behave toward them. I'm saying that your behavior—which includes your words and facial expressions—should always be respectful.
The Golden Rule and the Platinum Rule are good guidelines for communication, although they're open to interpretation. For instance, I have a good friend who says, "Just tell me, give it to me straight, and I won't take it personally." Most people are not like that, so when he treats people the way he'd like to be treated (following the Golden Rule), they feel offended and insulted. By the same token, he doesn't really understand how others like to be treated (the Platinum Rule), and to a certain extent he doesn't care, so he still treats them the way he'd like to be treated, because he assumes that they're like him.
That's not effective communication. It doesn't include treating others with respect, even though my friend would say that it's more respectful to be direct and honest than not.
The last part of the equation—creating value—is also challenging. Do the following statements sound familiar?
"I told him how to do it right, and he's not doing it. So I told him that he's an idiot."
"I told him he's wrong, and now it's up to him to figure it out."
Do these types of exchanges create value? Of course not. They make the other person feel terrible and they don't improve productivity or morale.
Creating value means that at the end of each exchange, there must be more there between you, rather than less. There should be more understanding. There should be more learning. There should be more camaraderie.
Here's the thing—every one of us must learn effective communication. If you don't want to learn how, then find a job that lets you work by yourself, and then deliver a perfect product every time so you don't have to discuss it with anyone. If you're not capable of delivering a perfect product, then you're going to have to talk to someone. Why not talk to them in a way that doesn't upset them?
If you've read my blog, you know that I believe that there are two basic communication patterns: IAAM– and IAAM+. They're the two variations on the pattern "It's All About Me."
In short, consider this thesis: Everything you say and do is about you—revealing your feelings, your motivations, your reactions. They're not about someone else. I believe that the statements:"He made me angry" and "She hurt my feelings" are false. No one is responsible for your feelings and behavior but you. But by the same token, while you are not responsible for anyone else's feelings or behavior, you still have a responsibility to be aware of other people and what leads them to feel certain ways.
That's at the core of effective communication: taking responsibility for your feelings and behavior.
Here are a few things I like to think about when reflecting on my interactions with others:
  • If I was there, I contributed, even if I sat silently by.
  • If someone feels something, while I'm not responsible, I may have contributed to the circumstances that led to them feeling that way.
  • Have I checked my own behavior and words?
  • What's most important: being right or contributing to the relationship?
Whether in meetings or in individual conversation, these are among the things I consider every time. If we all do the same, our interactions will yield better results, productivity will improve, and our team cohesion will be enhanced.

Steven M. List (known as "Doc") is a career technologist, leader, coach, and facilitator and currently works for ThoughtWorks as an organizational transformation consultant. He is a recognized professional Open Space facilitator, including having been embraced by Microsoft and the ALT.Net community. He explores the topics of facilitation, communication, team building, and personal growth in his blog at www.stevenlist.com/blog.