February 2009
Best Practice: An Introduction To Domain-Driven Design
We give you a gentle introduction to designing and evolving rich domain models as part of integrating Domain-Driven Design (DDD) into your coding efforts. David Laribee
.NET Interop: Getting Started With IronRuby And RSpec, Part 1
See how IronRuby lets you employ the straightforward Ruby language to create acceptance tests that interoperate with .NET-compliant code. Ben Hall
"Oslo" Basics: Build Metadata-Based Applications With The “Oslo” Platform
We introduce you to “Oslo” and demonstrate how MSchema and MGraph enable you to build metadata-driven apps. We’ll define types and values in “M” and deploy them to the repository. Chris Sells
Patterns: WPF Apps With The Model-View-ViewModel Design Pattern
In this article we explain just how simple it can be to build a WPF application the right way using the MVVM Pattern. Josh Smith
Silverlight: Build Line-Of-Business Enterprise Apps With Silverlight, Part 2
Here we wrap up the call center client application we began last month. The techniques we illustrate will help you build real-world enterprise solutions using Silverlight. Hanu Kommalapati
MSBuild: Best Practices For Creating Reliable Builds, Part 1
This article describes some basic practices and techniques that you can use in MSBuild to better manage and control your build projects in Visual Studio. Sayed Ibrahim Hashimi
Columns
Editor's Note: Driving Design: Howard Dierking concludes that the business domain must drive software design—even if the natural paradigm is not what is most comfortable. See how he got there.Howard Dierking | Toolbox: The Active Record Pattern, Aspect-Oriented Programming: This month we look at implementing the Active Record Pattern in your .NET application and how aspect-oriented programming yields separation of concerns.Scott Mitchell | CLR Inside Out: Handling Corrupted State Exceptions: There’s a danger in using one-size-fits-all exception handling, and you need to know how to avoid it by being as specific as possible about how you handle errors.Andrew Pardoe
Data Points: Syndicated Data And Isolated Storage In Silverlight: Here we build a syndicated news reader application to illustrate the use of isolated storage and data syndication in Silverlight.John Papa | Cutting Edge: Managing Dynamic Content Delivery In Silverlight, Part 2: This month Dino continues his look at managing dynamic Silverlight content by discussing caching and isolated storage.Dino Esposito | Patterns in Practice: Convention Over Configuration: We look at some techniques you can adopt to reduce the amount of housekeeping code you write so you can focus on the essence of the application.Jeremy Miller
Under The Table: Spatial Data Support In SQL Server 2008: New spatial data support in SQL Server 2008 opens the door to mapping and querying geometric and geographic data, allowing you to build exciting new applications.Bob Beauchemin | Foundations: Error Handling In Workflows: Workflows often define long-running processes and an unhandled failure usually means termination. Avoid this scenario by handling exceptions properly.Matt Milner | Windows With C++: Visual C++ 2010 And The Parallel Patterns Library: The Parallel Patterns Library allows you to more easily take advantage of parallelism. See what this and other Visual C++ 2010 features are in store.Kenny Kerr
.NET Matters: Ordered Execution With ThreadPool: This month we demonstrate how you can use the ThreadPool to support ordered execution without having to build custom thread pools yourself.Stephen Toub | Going Places: Mobile Device Provisioning With SyncML: OMA Device Management (OMA-DM), based on a dialect of XML called SyncML, can be used to provision and manage mobile devices in an enterprise scenario. We'll show you how.Ramon Arjona | Stack Trace: Working With An Incomplete Or Invalid IRP:
Stack Trace: What Are These System PTEs?: | { End Bracket }: Your Innovative Ideas: Having that next great innovative idea is only half the battle—getting it heard is the real challenge.Eric N. Bush


Best Practice
An Introduction To Domain-Driven Design
David Laribee

This article discusses:
  • Modeling from Ubiquitous Language
  • Bounded Contexts and aggregate roots
  • Using the Single Responsibility Principle
  • Repositories and databases
This article uses the following technologies:
Visual Studio
Contents
The Platonic Model
Talk the Talk
Context
Know Your Value Proposition
A System of Single Responsibilities
Entities Have an Identity and a Life
Value Objects Describe Things
Aggregate Roots Combine Entities
Domain Services Model Primary Operations
Repositories Save and Dispense Aggregate Roots
The Thing with Databases
Getting Started with DDD
Domain-Driven Design (DDD) is a collection of principles and patterns that help developers craft elegant object systems. Properly applied it can lead to software abstractions called domain models. These models encapsulate complex business logic, closing the gap between business reality and code.
In this article, I'll cover the basic concepts and design patterns germane to DDD. Think of this article as a gentle introduction to designing and evolving rich domain models. To provide some context to the discussion, I'm using a complex business domain with which I'm familiar: insurance policy management.
If the ideas presented here appeal to you, I highly recommend that you deepen your toolbox by reading the book Domain-Driven Design: Tackling Complexity in the Heart of Software, by Eric Evans. More than simply the original introduction to DDD, it is a treasure trove of information by one of the industry's most seasoned software designers. The patterns and core tenets of DDD that I will discuss in this article are derived from the concepts that are detailed in this book.

Cutting Contexts by Architectural Need
Bounded Contexts needn't be organized solely by the functional area of an application. They're very useful in dividing a system to achieve desired architectural examples. The classic example of this approach is an application that has both a robust transactional footprint and a portfolio of reports.
It's often desirable in such circumstances (which might occur pretty often) to break out the reporting database from the transactional database. You want the freedom to pursue the right degree of normalization for developing reliable reports, and you want to use an Object-Relational Mapper so that you can keep coding transactional business logic in the object-oriented paradigm. You can use a technology such as Microsoft Message Queue (MSMQ) to publish data updates coming from the model and incorporate them into data warehouses optimized for reporting and analysis purposes.
This might come as a shock to some, but it's possible for database administrators and developers to get along. Bounded Contexts give you a glimpse of this promised land. If you're interested in architectural Bounded Contexts, I highly recommend keeping tabs on Greg Young's blog. He's quite experienced with and articulate about this approach and produces a fair amount of content on the subject.
The Platonic Model
Since we're just starting off here, it makes practical sense to define what I mean by model. Answering this question leads us on a short, metaphysical journey. Who better than Plato to guide us?
Plato, that most famous student of Socrates, proposed that the concepts, people, places, and things we intuit and perceive with our senses are merely shadows of the truth. He dubbed this idea of a true thing a Form.
To explain Forms, Plato used what's become known as the allegory of the cave. In this allegory, there exists a people that are bound inside a deep, dark cave. These cave people are chained in such a way that they can only ever see a blank wall of the cave that receives light from the opening. When an animal walks by the opening, a shadow is projected onto the interior wall that the cave dwellers see. To the cave dwellers, these shadows are the real thing. When a lion walks by, they point at the shadow of the lion and exclaim, "Run for cover!" But it is really only a shadow of the real Form, the lion itself.
You can relate Plato's theory of Forms to DDD. Much of its guidance helps us get closer to the ideal model over time. The path to the Form you want to describe with your code is scattered about in the minds of domain experts, the desires of stakeholders, and the requirements of industries in which we're working. These are, in a very real sense, the shadows to Plato's imaginary cave dwellers.
Furthermore, you are often constrained by programming languages and considerations of time and budget in trying to reach that Form. It's not much of a stretch to say these limitations are the equivalent of cave dwellers only ever being able to see the interior wall of shadows.
Good models exhibit a number of attributes independent of their implementation. The fact of the matter is, the mismatch between the model that's in everyone's head and the model you're committing to code is the first thing an aspiring domain modeler should understand.
The software you create is not the true model. It is only a manifestation—a shadow, if you will—of the application Form you set out to achieve. Even though it's an imitation of the perfect solution, you can seek to bring that code closer to the true Form over time.
In DDD, this notion is called model-driven design. Your understanding of the model is evolved in your code. Domain-driven designers would rather not bother with reams of documentation or heavy diagramming tools. They seek, instead, to imbue their sense of domain understanding directly into their code.
The idea of the code capturing the model is core to DDD. By keeping your software focused on the problem at hand and constrained to solving that problem, you end up with software receptive to new insights and moments of enlightenment. I like what Eric Evans calls it: crunching knowledge into models. When you learn something important about the domain, you'll know right where to go.

Talk the Talk
Let's examine some of the techniques DDD provides for achieving this goal. A big part of your job as a developer is working with non-coders to understand what you're meant to deliver. If you're working in an organization with any kind of process, you likely have requirements expressed as a user story, task, or use case. Whatever kind of requirements or specification you receive, is it usually complete?
Typically requirements are somewhat murky or expressed at a high level of understanding. In the course of designing and implementing a solution, it's beneficial when developers have access to the people who bring some expertise in the intended domain. This is exactly the point of user stories, which are typically expressed according to a template like this: as a [role], I want [feature] so that [benefit].
Let's take one example from the domain of insurance policy management: as an underwriter, I want approval control over a policy so that I can write safe exposures and reject risky ones.
Is there anyone who understands what that means? I know I didn't when I saw it written and prioritized. How can you possibly understand everything that will need to go into delivering the supporting software from this abstracted description?
User stories, properly done, are an invitation to have a conversation with their author—the user. This means when you go to work on the approve/deny policy feature, you should ideally have access to an underwriter. Underwriters, for the uninitiated, are domain experts (at least good ones are) who determine whether a certain category of exposure is safe for a carrier to cover.
When you begin discussing the feature with your underwriter (or whomever your project domain expert might be), pay especially close attention to the terminology the underwriter uses. These domain experts use company- or industry-standard terminology. In DDD, this vocabulary is called the Ubiquitous Language. As developers, you want to understand this vocabulary and not only use it when speaking with domain experts but also see the same terminology reflected in your code. If the terms "class code" or "rate sets" or "exposure" are frequently used in conversation, I would expect to find corresponding class names in the code.
This is a pretty fundamental pattern to DDD. At first blush, the Ubiquitous Language seems like an obvious thing, and chances are a good that number of you are already practicing this intuitively. However, it is critical that developers use the business language in code consciously and as a disciplined rule. Doing so reduces the disconnect between business vocabulary and technological jargon. The how becomes subordinate to the what, and you stay closer to the reason for your job: delivering business value.

Context
Developers are, in a sense, organizers. You sling code (hopefully with intent) into abstractions that solve problems. Tools such as design patterns, layered architectures, and object-oriented principles give a framework for applying order to evermore complicated systems.
DDD extends your organizational toolbox and borrows from well-known industry patterns. What I like most about the organizational patterns that DDD lays out is that there are solutions for every level of detail in a system. Bounded Contexts guide you toward thinking of software as a portfolio of models. Modules help you organize a larger single model into smaller chunks. Later, I'll cover aggregate roots as a technique for organizing small collaborations between a few highly related classes.
In most enterprise systems there are course-grained areas of responsibility. DDD calls this top level of organization a Bounded Context.
Workers' compensation insurance policies need to be concerned with elements such as:
  • Quoting and sales
  • General policy workflow (renewals, terminations)
  • Auditing payroll estimation
  • Quarterly self-estimates
  • Setting and managing rates
  • Issuing commissions to agencies and brokers
  • Billing customers
  • General accounting
  • Determining acceptable exposures (underwriting)
Wow. That's a lot of stuff! You could incorporate all of this into a single, monolithic system, but doing so leads you down a foggy, amorphous path. People are talking about two entirely different things when they chat about a policy in the context of a general workflow versus a policy in the context of payroll auditing. If you use the same policy class, you're fattening the profile of that class and getting a long way from tried-and-true best practices such as the Single Responsibility Principle (SRP).
Systems that fail to isolate and insulate Bounded Contexts often slip into an architectural style (amusingly) called The Big Ball of Mud. In 1999, Brian Foot and Joseph Yoder defined the architectural style (or anti-architectural style, as the case may be) in their classic paper of the same name ("Big Ball of Mud").
DDD nudges you toward identifying contexts and constraining your modeling effort within particular contexts. You can use a simple diagram, called a context map, to explore the boundaries of our system. I've enumerated the contexts involved in a fully featured, insurance policy management system, and Figure 1 takes this from a textual description to a (partial) graphical context map.
Figure 1 From Bounded Context to Context Map
Did you notice some key relationships among the various Bounded Contexts? This is valuable intelligence because you can start making informed business and architectural decisions such as packaging and deployment design; choice of technology used to marshal messages between models; and, perhaps most important, where you choose to set milestones and deploy effort, time, and talent.
One last but very important thought about Bounded Contexts: each context owns its own Ubiquitous Language. It's important to differentiate between the notion of a policy in the auditing subsystem versus the policy in core workflow because they're different things. While they may have the same identity, the value objects and child entities (more on these in a bit) are often radically different. Since you're modeling within context, you also want the language to provide precision within that context so you can have productive communication with domain experts and within your teams.
Some areas inside models group more closely together than others. Modules are a means of organizing these groups within a particular context, serving as mini-boundaries where you want to stop and think about associations to other modules. They are also another organization technique that leads you away from "Small Balls of Mud." Technologically, modules are easy to create; in the Microsoft .NET Framework they are simply namespaces. But the art of identifying modules involves spending a little time with your code. Eventually you may see a few things emerge as a mini-model within a model, at which time you may consider dividing things into namespaces.
Teasing apart models into cohesive modules has a nice effect in your IDE. Namely, since you need to use multiple using statements to include modules explicitly, you get a much cleaner IntelliSense experience. They also give you a way of looking at associations between larger conceptual chunks of your system with a static analysis tool such as NDepend.
Introducing an organizational change to your model should prompt you to engage in some pragmatic, cost-benefit thinking. When you use modules (or namespaces) to divide up your model, you really want to question whether you're dealing with a separate context. The cost of cleaving out another context is usually much higher: now you have two models, likely in two assemblies, that you need to connect with application services, controllers, and so on.

Anti-Corruption Layers
An Anti-Corruption Layer (ACL) is another DDD pattern that encourages you to create gatekeepers that work to prevent non-domain concepts from leaking into your model. They keep the model clean.
At their heart, repositories are actually a type of ACL. They keep SQL or object-relational mapping (ORM) constructs outside of your model.
ACLs are a great technique for introducing what Michael Feathers calls, in his book Working Effectively With Legacy Code, a seam. A seam is an area where you can start to cleave off some legacy code and begin to introduce changes.Finding seams, along with isolating your core domain, can be extremely valuable when using DDD techniques to refactor and tighten the highest value parts of your code.
Know Your Value Proposition
Most development shops have a handful of both seasoned businesspeople and top developers who are capable of isolating and describing a problem and building an elegant, maintainable object-oriented solution. To get the biggest bang for your customer's buck, you want to be sure to understand the core domain of your application. The core domain is the Bounded Context that brings the most value to applying DDD.
In any given enterprise system, there are some areas that are more important than others. The more important areas tend to fall into alignment with the core competencies of the client. It's rare that a business will be running custom general ledger software. But if that business is in insurance (going with my previous example) and their money-making offering is managing risk pools where liability is spread among all members, then they had better be darned good at rejecting bad risks and identifying trends. Or perhaps you have a client that's a medical claims processor, and their strategy is to flank their competition on pricing by automating payments to amplify the efforts of their bill payor workforce.
Whatever the industry, your employer or client has some edge in the market, and this edge is usually where you find the custom software. That custom software is where you're likely to find and model the core domain.
We can measure our investment in value on another dimension, namely where we invest our intellectual capital in reaching technical excellence. Too many times senior developers are the kind of people who get obsessed with new technologies. A certain amount of this is to be expected—the industry innovates at a relentless pace, and vendors are compelled to frequently release new technology offerings to satisfy their customers' demands and stay competitive. The challenge, as I see it, is for senior developers to master the fundamental principles and patterns that contribute value to the heart of a system. It's tempting to get wrapped up in a new framework or platform, but we need to remember the reason vendors produce these things is so we can just trust that they work.

A System of Single Responsibilities
I've mentioned that DDD provides a pattern language for structuring rich domain models. By implementing these patterns, you get a certain level of adherence to the SRP for free, and that's certainly valuable.
The SRP encourages getting at the core purpose of an interface or class. It guides you toward high cohesion—a very good thing, as it makes code easier to discover, reuse, and maintain.
DDD identifies certain types of class responsibilities in a core collection of patterns. I'll cover a few of the more primary ones now, but it's worth mentioning that there are many patterns described by Eric Evans in the original book ranging from class level to architectural. For introductory purposes, I'll stay at the class level covering entities, value objects, aggregate roots, domain services, and repositories. Because this is an introduction, I'll only cover the responsibility of each pattern with one to two code examples or tips each.

Entities Have an Identity and a Life
An entity is a "thing" in your system. It's often helpful to think about these in terms of nouns: people, places, and, well, things.
Entities have both an identity and a lifecycle. For example, If I want to access a particular customer in my system, I can ask for her by a number. When I complete a trading order, it's then dead to my system and can go to long-term storage (the historical reporting system).
Think of entities as units of behavior rather than as units of data. Try to put your logic in the entities that own them. Most times there's an entity that should receive some operation you're trying to add to your model, or a new entity is begging to be created or extracted. In more anemic code, you find a lot of service or manager classes that validate entities from the outside. In general, I much prefer to do that from within the entity—you get all the benefits associated with the fundamental principle of encapsulation, and you're making your entities behavioral.
Some developers are bothered by having dependencies in their entities. Obviously you need to create associations between the various entities in your system. For example, you might need to obtain a Product entity from the Policy entity so you can determine sensible defaults on the policy. Where people seem to fall down is when you need some outside service to perform behavior intrinsic to an entity.
I, for one, am not disturbed by the need to involve other, non-entity classes, and I would try to avoid lifting the central behavior outside of my entity. You should always remember that entities are intrinsically behavioral units. Often that behavior will be implemented as a kind of state machine—when you invoke a command on an entity, it is responsible for changing its internal state—but sometimes it's necessary for you to obtain additional data or impose side-effects upon the outside world. My preferred technique for accomplishing this is to provide the dependency to the command method:
public class Policy {
  public void Renew(IAuditNotifier notifier) {
    // do a bunch of internal state-related things,
    // some validation, etc.
    ...
    // now notify the audit system that there's
    // a new policy period that needs auditing
    notifier.ScheduleAuditFor(this);
  }
}
The benefit of this approach is that you don't need an inversion of control (IOC) container to create an entity for you. Another perfectly acceptable approach, in my opinion, would be to use a Service Locator to resolve the IAuditNotifier inside the method. This technique has the benefit of keeping the interface clean, but I find the former strategy tells me a lot more about my dependencies at a higher level.

Value Objects Describe Things
Value objects are descriptors or properties important in the domain you are modeling. Unlike entities, they do not have an identity; they simply describe the things that do have identities. Are you changing an entity called "Thirty-Five Dollars" or are you incrementing the balance of an account?
Part of the beauty of value objects is that they describe the properties of entities in a much more elegant and intention-revealing way. Money, a common value object, looks a lot better as a function parameter on a funds transfer API than a decimal does. When you spot it in an interface or entity method, you know what you're dealing with right away.
Value objects are immutable. They are incapable of change once they are created. Why is it important that they be immutable? With value objects, you're seeking side-effect-free functions, yet another concept borrowed by DDD. When you add $20 to $20, are you changing $20? No, you are creating a new money descriptor of $40. In C# you can use the read-only keyword on public fields to enforce immutability and side-effect-free functions, as shown in Figure 2.
Figure 2 Using Read-Only to Enforce Immutability
public class Money {
  public readonly Currency Currency;
  public readonly decimal Amount;

  public Money(decimal amount, Currency currency) {
    Amount = amount;
    Currency = currency;
  }

  public Money AddFunds(Money fundsToAdd) {
    // because the money we're adding might
    // be in a different currency, we'll service
    // locate a money exchange Domain Service.
    var exchange = ServiceLocator.Find<IMoneyExchange>();
    var normalizedMoney = exchange.CurrentValueFor(fundsToAdd,         this.Currency);
    var newAmount = this.Amount + normalizedMoney.Amount;
    return new Money(newAmount, this.Currency);
  }
}

public enum Currency {
  USD,
  GBP,
  EUR,
  JPY
}

Aggregate Roots Combine Entities
An aggregate root is a special kind of entity that consumers refer to directly. Identifying aggregate roots allows you to avoid over-coupling the objects that comprise your model by imposing a few simple rules. You should note that aggregate roots guard their sub-entities zealously.
The biggest rule to keep in mind is that aggregate roots are the only kind of entity to which your software may hold a reference. This helps avoid The Big Ball of Mud because you now have a constraint that prevents you from creating a tightly coupled system where everything has an association to everything else.
Let's assume I have an entity called Policy. Well, policies get renewed on an annual term, so there's likely an entity called Period. Since a Period cannot exist without a Policy and you can act on Periods through Policy, Policy is said to be an aggregate root and Period is a child of the same.
I like my aggregate roots to just figure it out for me. Consider this consumer code accessing a Policy aggregate root:
Policy.CurrentPeriod().Renew()
I'm trying to renew an insurance policy—recall the class diagram of the core domain of insurance policy management. Note how I'm dotting my way to the behavior I want to invoke?
There are a couple of problems with this approach. First, I'm clearly violating the Law of Demeter. A method M of an object O should invoke only the methods of the following kinds of objects: itself, its parameters, any objects it creates or instantiates, or its direct component objects.
Isn't deep dotting kind of convenient? IntelliSense is one of the coolest, most useful features of Visual Studio and other modern IDEs, but when you start dotting your way to the function you actually want to invoke, you're introducing unnecessary coupling into the system. In the previous example, I'm now dependent upon the Policy class and the Period class.
For further reading, Brad Appleton has an excellent article available on his site that explains the deeper implications, theories, tooling and caveats around the Law of Demeter.
The cliche "death by a thousand cuts" is a good description of the potential maintenance nightmare of an overly coupled system. When you create unnecessary references all over the place, you also create a rigid model where changes in one place result in a cascade of changes all over consumer code. You could accomplish the same goal with, as far as I'm concerned, a much more expressive bit of code:
Policy.Renew()
See how the aggregate just figures it out? Internally it can find the current period and whether or not there's already a renewal period and whatever else you need it to do.
I find that when I'm unit testing my aggregate roots using a technique such as Behavior-Driven Development (BDD), my tests trend toward the more black-box and state-testing paradigm. Aggregate roots and entities often end up as state machines, and the behavior matches accordingly. I end up with state validation, addition, and subtraction. There's quite a bit of behavior going on in the renewal example in Figure 3, and it's pretty clear how you can express this in a BDD style of test.
Figure 3 Testing Aggregate Roots
public class
  When_renewing_an_active_policy_that_needs_renewal {

  Policy ThePolicy;
  DateTime OriginalEndingOn;

  [SetUp]
  public void Context() {
    ThePolicy = new Policy(new DateTime(1/1/2009));
    var somePayroll = new CompanyPayroll();
    ThePolicy.Covers(somePayroll);
    ThePolicy.Write();
    OriginalEndingOn = ThePolicy.EndingOn;
  }

  [Test]
  public void Should_create_a_new_period() {
    ThePolicy.EndingOn.ShouldEqual(OriginalEndingOn.AddYears(1));
  }
}

Domain Services Model Primary Operations
Sometimes you have operations or processes that do not have an identity or lifecycle in your domain. Domain services give you a tool for modeling these concepts. They're typically stateless and highly cohesive, often providing a single public method and sometimes an overload for acting on sets.
I like to use services for a few reasons. When there are a number of dependencies involved in a behavior and I can't find a natural place on an entity to place that behavior, I'll use a service. When my Ubiquitous Language talks about a process or operation as a first-order concept, I'll question whether a service makes sense as a central point of orchestration for the model.
You could use a domain service in the case of renewal. This is an alternative style. Rather than method injecting an IAuditNotifier directly into the method of the Policy entity's Renew method, you might choose to extract a domain service to handle dependency resolution for us; it's more natural for us to resolve a domain service from an IOC container than an entity. This strategy makes a lot more sense to me when there are multiple dependencies, but I'll present the alternative anyway.
Here is a short example of a domain service:
public class PolicyRenewalProcesor {
  private readonly IAuditNotifier _notifier;

  public PolicyRenewalProcessor(IAuditNotifier notifier) {
    _notifier = notifier;
  }
  public void Renew(Policy policy) {
    policy.Renew();
    _notifier.ScheduleAuditFor(policy);
  }
}
The word service is a highly overloaded term in the developer world. Sometimes I think of service as in service-oriented architecture (SOA). Other times I think of services as little classes that don't represent a particular person, place, or thing in my application, but that tend to embody processes. Domain services usually fall in this latter category. They tend to be named after verbs or business activities that domain experts introduce into the Ubiquitous Language.
Application services, on the other hand, are a great way to introduce a layered architecture. They can be used for mapping data inside the domain model to a shape required by a client application. For example, maybe you need to display tabular data in a DataGrid, but you want to maintain a granular and jagged object graph in your model.
Application services are also quite useful for integrating multiple models—for example, translating between policy auditing and core policy workflow. Similarly, I use them to bring infrastructure dependencies into the mix. Imagine a common scenario where you want to expose your domain model with Windows Communication Foundation (WCF). I'd use an application service decorated with the WCF attributes to make this happen rather than letting WCF leak into my pure domain model.
Application services tend to be very broad and shallow, embodying cohesive functionality. Consider the interface and partial implementation you see in the code in Figure 4 as a good example of an application service.
Figure 4 A Simple Application Service
public IPolicyService {
  void Renew(PolicyRenewalDTO renewal);
  void Terminate(PolicyTerminationDTO termination);
  void Write(QuoteDTO quote);
}

public PolicyService : Service {
  private readonly ILogger _logger;
  public PolicyService(ILogger logger, IPolicyRepository policies) {
    _logger = logger;
    _policies = policies;
  }

  public void Renew(PolicyRenewalDTO renewal) {
    var policy = _policies.Find(renewal.PolicyID);
    policy.Renew();
    var logMessage = string.Format(
      "Policy {0} was successfully renewed by {1}.",
      Policy.Number, renewal.RequestedBy);
    _logger.Log(logMessage);
  }
}

Repositories Save and Dispense Aggregate Roots
Where do you go to retrieve entities? How do you store them? These questions are answered by the Repository pattern. Repositories represent an in-memory collection, and the conventional wisdom is that you end up with one repository per aggregate root.
Repositories are a good candidate for a super class or what Martin Fowler refers to as the Layer Supertype pattern. Using generics to derive a base repository interface from the previous example is a simple matter:
public interface IRepository<T>
  where T : IEntity
{
  Find<T>(int id);
  Find<T>(Query<T> query);
  Save(T entity);
  Delete(T entity);
}
Repositories prevent database or persistence concepts, such as SQL statements or stored procedures, from commingling with your model and distracting you from the mission at hand: capturing the domain. This separation of model code from infrastructure is a good attribute. See the sidebar "Anti-Corruption Layers" for a more detailed discussion.
At this point, you have likely noticed that I am not talking about how aggregate roots, their subordinate entities, and the attached value objects actually get persisted to disk. This is intentional. Saving the data required to perform behavior in your model is a concern orthogonal to the model itself. Persistence is infrastructure.
A survey of these technologies is far beyond the scope of an introduction to DDD. Suffice it to say there are a number of suitable and mature options for storing your model's data from Object-Relational Mapping (ORM) frameworks to document-oriented databases to "roll-your-own" data mappers for simple scenarios.

DDD Resources

The Official DDD Site
Dan North on How to Structure User Stories
The Big Ball of Mud Architectural Style
Greg Young's Blog on CodeBetter
Robert C. Martin's Paper on Single Responsibility Principle
Brad Appleton on Law of Demeter
Martin Fowler Describes the Layer Supertype Pattern
Robert C. Martin on the S.O.L.I.D. Principles
The Thing with Databases
By this point, I'm sure you've thought, "This is all well and good, Dave. But where do I save my entities?" Yes, you have to deal with this important detail, but how or where you persist your models is mostly tangential to an overview of DDD.
A number of developers or database admins will make the assertion that the database is the model. In a lot of cases, that's partially true. Databases, when highly normalized and visualized with a diagramming tool, can convey a lot about the information and relationships in your domain.
Data modeling as a primary technique leaves something to be desired, however. When you want to understand the behaviors inherent in the same domain, data-only techniques such as entity-relationship diagrams (ERDs) or entity-relationship models and class diagrams break down. You need to see the application's parts in motion and how types collaborate to get work done.
When I am modeling, I often reach for sequence diagrams on a whiteboard as a communications tool. This approach captures much of the essence of communicating a behavioral design or problem without the ceremony of Unified Modeling Language (UML) or model-driven architecture. I like to avoid unnecessary ceremony, especially when I'm going to erase the diagrams I put on the whiteboard. I don't worry about 100% UML adherence in my diagrams, preferring simple boxes, arrows, and swim lanes that I can sketch quickly.
If you don't already use sequence diagrams on your team, I highly recommend learning the technique. I've observed the powerful effect they have on getting team members to think through issues around the SRP, layered architecture, behavior design over data modeling, and architectural thinking in general.

Getting Started with DDD
Becoming proficient with object-oriented programming is no trivial undertaking. I believe competence is within the reach of most professional developers, but it takes dedication, book learning, and practice, practice, and more practice. It also helps if you adopt an attitude of craftsmanship and continual learning.
How do you get started? Simply put: do your homework. Learn about things like the S.O.L.I.D. principles and study Eric Evans' book. Your time investment will more than make up for itself. InfoQ has produced a smaller version of the DDD book that introduces a few key concepts. If you're on a financial or temporal budget or simply in investigation mode, I recommend starting there. Once you get a solid footing, wander over to the Yahoo! DDD group to see what issues your fellow designers are struggling with and get involved in the conversation.
DDD is not a new doctrine or methodology. It's a collection of time-tested strategies. When you get ready to practice, try adapting those philosophies, techniques, and patterns that make the most sense in your situation. Some elements of DDD apply more universally than others. Understanding the reason for your core domain's existence by uncovering and using the Ubiquitous Language and identifying the contexts in which we're modeling are way more important than nailing that perfectly opaque and one-size-fits-all repository.
When you're designing your solutions, design for value. If designers produce art and developers are a kind of designer, then our medium has to be business value. Value consciousness trumps considerations such as adherence to dogma and choice of persistence technology, as important as these choices sometimes seem.

Dave Laribee coaches the product development team at VersionOne. He is a frequent speaker at local and national developer events and was awarded a Microsoft Architecture MVP for 2007 and 2008. He writes on the CodeBetter blog network at thebeelog.com.


.NET Interop
Getting Started With IronRuby And RSpec, Part 1
Ben Hall

This article is based on a prerelease version of IronRuby. All information is subject to change.

This article discusses:
  • Ruby and Duck Typing
  • Ruby and the Microsoft .NET Framework
  • Using IronRuby and RSpec
This article uses the following technologies:
IronRuby
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Defining Requirements and Examples in Code
Ruby and Duck Typing
Ruby and MetaProgramming
Ruby and the .NET Framework
Ruby and the CLR
Inside IronRuby
Testing a C# App with IronRuby
Moving Forward
"That's not what we asked for!" I'm sure most developers have heard this cry from a customer shortly after delivering the latest build. The customer could be yelling at the poor developer for various reasons—maybe the requirements were not correctly understood or part of the system simply did not work.
Because customers are often unclear as to their own requirements, they will opt for the safety of the 100-page technical document that tries to define everything the system might have to do. Meanwhile, the developer struggles with undocumented legacy code, trying to understand how the app is intended to work while attempting to implement new requirements without breaking anything.
Times are changing. New approaches to software development are aimed at solving these problems, helping you to meet the customer requirements at the first attempt without causing failures in the process. These approaches are taking advantage of languages such as Ruby, allowing you to create easily readable and maintainable code with much shorter development iterations.
In this article, I will introduce you to Ruby and IronRuby and demonstrate some basics of Ruby interoperating with Microsoft .NET Framework-based code. I also explain how frameworks such as RSpec can be used to generate examples of how objects are intended to behave, providing both documentation and verification that the system is built correctly. This will set the stage for a future article in which I will explain acceptance testing in detail and demonstrate creating acceptance tests with IronRuby.

Defining Requirements and Examples in Code
To help write examples and define requirements you need a framework. There are many different approaches to writing automated acceptance tests or executable specifications. Some people use standard xUnit test frameworks successfully, while others use Fit and Fitness frameworks. I have found the best approach is to use Behavior-Driven Development (BDD). Dan North devised a BDD framework called JBehave as a way to define scenarios that describe the application's behavior in a way that can be communicated to the whole team, regardless of technical ability.
JBehave was the result of problems North faced with Test-Driven Development (TDD) and was implemented for the Java language. Later, North created RBehave, which has since been integrated into RSpec, a popular framework for Ruby. RSpec accommodates two different approaches to BDD: Dan North's approach, based on stories and scenarios to describe application's behavior, and Dave Astels' approach, which is more focused on creating examples at an object level.
C# does have some BDD frameworks, such as NSpec and NBehave. The main problem with writing tests in C# is that the true intent of the test is often hidden due to fact that you have extra structural elements and metadata such as braces, public and private. Overall, I don't think what is on offer via C# and NSpec/NBehave can match what is available via IronRuby and RSpec. Previously, this would have been a major problem for C# developers, as you could not have used the Ruby-based RSpec to test C# apps. With IronRuby, this is no longer a problem.
While still early in development, IronRuby is taking advantage of the Dynamic Language Runtime (DLR) to create an implementation of the Ruby language on top of the CLR. With IronRuby, you can employ existing Ruby apps and frameworks together with the .NET Framework and .NET-compliant languages. The result is that you can use the Ruby language and RSpec to test C# applications!
As a language, Ruby is concise, allowing you to write less code and express it in a much more natural manner, making your code easier to maintain. For example, to read all the lines of text from a file and write them out to the console, you would write this:
File.readlines('AboutMicrosoft.txt').map {|line| puts line}
The code opens the file AboutMicrosoft.txt and reads all the lines, passing each line into the block, with the line as a parameter that is then written to the console. In Ruby, a block is the collection of statements between the braces and is similar to invoking a method in C#, which uses the yield statement to return control to the calling method.
Its natural language approach is one of the reasons why Ruby is excellent to use when testing. The tests, or scenarios in this case, are much more readable.

Ruby and Duck Typing
One of the reasons why Ruby and dynamic languages are easier to read is due to how they handle typing. Instead of the developer defining a type, when the Ruby code is interrupted, the type of the variable is inferred. The language is still strongly typed, but it determines the variable's type dynamically.
With C#, interfaces allow for decoupling of objects while still defining a contract for the implementation. With Ruby, there is no need for contracts and interfaces. Instead, if the object has an implementation of the method you are calling, then it's called. If it doesn't, then an error is returned. As such, the contract for the object is defined by its implementation, not its relationship to other parts of the code.
To demonstrate this, I created a method that accepts a value, the type of which will be inferred at run time. It will then output Hello plus the value:
def SayHello(val)
     puts "Hello #{val}"
   end
Because of the way types are handled, you can reuse the same method for both strings and integers without changing any of the code:
SayHello("Test") => "Hello Test"
   SayHello(5) => "Hello 5"
I could have also have called the method without brackets, as these are optional in Ruby and can make the syntax much more readable:
SayHello "Test" => "Hello Test"
This can be achieved in C# using object, but let's look at how this works with more complex objects. I defined a new method to output the result of a call to GetName. As long as the value of the parameter n implements a method called GetName, the code will work:
def outputName(n)
  puts n.GetName()
end
Here are two unrelated classes:
class Person
  attr_accessor :name
  def GetName()
    @name
  end
end
class Product
  def GetName()
    "The product with no name"
  end
end
Person has an accessor allowing the name variable to be set, while Product returns a hardcoded value. In Ruby, there is no need to write the return statement, the last result of the last line of code of a method is returned by default.
If you call the method outputName, the GetName method is called, demonstrating how Ruby's typing can enhance code reusability since we are not fixed to a single type:
outputName(Product.new) => "The product with no name"
$x = Person.new
$x.name = "Ben Hall"
outputName($x) => "Ben Hall"
If you call the method using an argument that doesn't implement GetName, then a NoMethodError is raised:
outputName("MyName") => :1:in 'outputName': \
  undefined method 'GetName' \
  for MyName:String (NoMethodError)
While this concept concerns some people, it can be useful; it allows for improved testability and application design, especially when testing C# applications, which I will discuss in the next section.

Ruby and MetaProgramming
Similar to other dynamic languages, Ruby embraces the concept of meta-programming. This allows you to extend any class or object at run time, enabling you to define methods and behavior at run time, giving you the ability to develop a self-modifying application. This is extremely useful in dynamic languages, especially in unit testing, as it allows you to customize the behavior of objects and methods to your own requirements.
In a similar fashion, you can extend the built-in Ruby classes to provide your own functionally, much like the extension methods in C# 3.0. With C#, there are various restrictions about the methods you can create. For example, they have to be static and in a separate class. This all harms readability. With Ruby, you just create a normal method on an existing class. For example, Integer is a class to handle whole numbers. Integer has a series of methods, but it does not have a method to say whether the number is even.
To create such a method, you simply create a new class with the same name (Integer) and define the new method. The question mark at the end of the method denotes that a Boolean is returned with the aim of making it easier to read:
class Integer
   def even?()
       self.abs % 2 == 0
   end
end
puts 2.even? => true
puts 1.even? => false

Ruby and the .NET Framework
In place of the old debate about whether to use a static or dynamic language, IronRuby lets you use the right language for the job. If it makes more sense to use C#, then you can use C#. If you need a more dynamic approach, you can easily use IronRuby, taking advantage of its interoperability with .NET and the dynamic nature of the Ruby language. I anticipate people using a mixture of different languages and technologies, such as C# for the main application and Ruby for testing.
The DLR provides the foundation to enable .NET interop. With this in place, you can take advantage of UI technology such as Windows Forms, Windows Presentation Foundation (WPF), and Silverlight while writing application code in Ruby.
Taking advantage of WPF from IronRuby is easy. When the following code is executed, you can have a fully functioning WPF window created from IronRuby. You still must reference mscorlib and the two WPF assemblies shipped with the .NET Framework 3.0, as you did with C#, using the require statements:
require 'mscorlib'
require 'PresentationFramework, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
require 'PresentationCore, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
You can now create and interact with the WPF objects. First, create aliases for all the objects you want to use. This makes the code cleaner, as you won't need to access the object via its namespace:
Window = System::Windows::Window
Application = System::Windows::Application
Button = System::Windows::Controls::Button
Next, create a WPF window and give it a title:
win = Window.new
win.title = 'IronRuby and WPF Interop'
Create a button in the same fashion:
mainButton = Button.new
mainButton.content = 'I'm a WPF button — press me'
In both instances I'm using the alias to access the full name of the object. When the button is clicked, I want to display a MessageBox to the user. As you would in C#, you can subscribe to the event and provide a block that is called when the click event is invoked:
mainButton.click do |sender, args|
  System::Windows::MessageBox.Show("Created using IronRuby!")
end
Finally, set the content of the window to be the button, create a new Application object, and launch the window:
win.content = mainButton
my_app = Application.new
my_app.run win
Figure 1 Using IronRuby to Create a WPF App
At this point, a fully interactive WPF window will be displayed with the click event correctly wired up as shown in Figure 1. I interacted with the .NET Framework in the same way I would interact with Ruby libraries. One of the core principals set out by John Lam and team was to stay true to the Ruby language. This is an important principal for adoption by current Ruby developers, who won't want to change the way they create Ruby applications just because they're moving to IronRuby. Instead they can access all the rich .NET code in the same way.

Ruby and the CLR
Here is another example. If you attempt to access AddRange on IEnumerable, in C# the code would look like this:
ArrayList list = new ArrayList();
list.AddRange(new [] { 1,2,3,4 });
However, with Ruby, the accepted convention is for words within method names to be separated by underscores to improve readability. Creating a separate library to follow this convention would be too time-consuming and error-prone, and it would not support additional third-party development.
Instead, for CLR objects, IronRuby translates Ruby method calls into the equivalent method name for the CLR:
$list = ArrayList.new
   $list.add_range([1,2,3,4])
Here I am accessing the AddRange method following Ruby's approach, which is lower case with words separated by an underscore. If you prefer, you can keep with the CLR naming convention as the method name still exists from Ruby:
$list.AddRange([1,2,3,4])
Both work the same; it's just a personal preference which to use.
As I mentioned, Ruby infers the type of objects at run time, and this is the same when handling C# objects. Consider a series of C# methods that return different object types, while the same object is being returned, the method signature returns either the interface or the concrete type. Within my example, I have an interface that defines a single HelloWorld method:
public interface IHello {
  string HelloWorld();
}
I created a Hello4Times class that inherits from Hello3Times, which inherits from the interface but has implemented three extra HelloWorld methods. Within the class I define a new method called HelloMethodOn4Times that calls the base implementation:
public class Hello4Times : Hello3Times {
  public string HelloMethodOn4Times () {
    return base.HelloWorld();
  }
}
I then defined a static class and methods that will return a new instance of the Hello4Times class but return it to the calling code as an interface. This means the calling code should only know about HelloWorld, not any of the additional methods defined:
public static class HelloWorld {
  public static IHello ReturnHello4TimesAsInterface() {
    return new Hello4Times();
  }
In my Ruby code, I have two method calls. The first call is on the interface-defined method, which you would expect to work without a problem. However, the second call is to the method on the concrete class. IronRuby has determined the type of the object being returned and can dispatch the method calls:
puts InteropSample::HelloWorld.ReturnHello4TimesAsInterface.HelloWorld
puts interopSample::HelloWorld.ReturnHello4TimesAsInterface.HelloMethodOn4Times
All of this happens without you ever needing to worry about casting the object to the correct type to call the method.
Following this theme, you can extend .NET objects in exactly the same fashion as you can with Ruby objects. When displaying a MessageBox, I don't want to keep defining which icons and buttons to use. Instead, I just want to provide a message.
Maybe you don't like the built-in functionality and want to extend the actual MessageBox class for seamless interaction. With Ruby, this is simple. You define a new class that is the same as the built in MessageBox within WPF. You then create a new method on the class that simply calls the show method with various defaults:
class System::Windows::MessageBox
  def self.ShowMessage(msg)
    System::Windows::MessageBox.Show(msg, msg, \
      System::Windows::MessageBoxButton.OK, \
      System::Windows::MessageBoxImage.Stop)
  end
end
After executing this block of code
System::Windows::MessageBox.ShowMessage( \
  "I'm going to show you a message")
you can call the ShowMessage method, the result being the message box shown in Figure 2.
Figure 2 Displaying a Custom MessageBox from Ruby

Inside IronRuby
How is interop possible? The answer is the DLR. At a high level, when you execute Ruby code using IronRuby, a lot of work takes place behind the scenes. First, the code you write is tokenized and parsed by the IronRuby engine. The parsed code is then converted into the DLR's Abstract Syntax Tree (AST). This is a standardized AST that all language implementations, such as IronPython, need to provide to the DLR in order to execute the code.
Once the DLR has the AST, it converts the tree into Intermediate Language (IL). All .NET code is compiled down into IL to provide a generic language for the CLR. This is how IronRuby can interoperate with the .NET Framework—behind the scenes, all the code is executed as IL instructions. After the DLR has completed the conversion, it passes the IL to the CLR for execution and the result is returned to IronRuby.
During this process there are additional steps to improve performance and reliability, such as caching. For a deeper explanation, I recommend reading Bill Chiles's CLR Inside Out column "IronPython and the Dynamic Language Runtime" in the October 2007 issue of MSDN Magazine.
Within a separate assembly there is a hosting API that allows you to embed IronRuby within your own applications, allowing you to execute Ruby code from C# or users to execute their own code.
For more about the IronRuby implementation, download the entire source code from RubyForge. IronRuby is an open-source project implemented in C# and is an excellent example of a dynamic language implementation. IronPython is available as an open source project hosted at CodePlex, and it includes a sample language called ToyScript if you want an introduction into how the DLR works.
To ensure continued compatibility with the core Ruby platform, the IronRuby team is using RubySpecs. This is a shared set of examples based around how the Ruby language should be implemented. The aim of RubySpecs is to ensure different implementations including Matz's Ruby Interpreter (MRI), JRuby, MacRuby, and IronRuby have the same behavior. RubySpecs uses a syntax-compatible version of RSpec called MSpec. These are viewed as the acceptance tests for the IronRuby implementation.

Testing a C# App with IronRuby
For many years now, the awareness and practice of TDD has increased as a way to develop software, resulting in higher quality code, in terms of design and maintainability, along with fewer defects along the way. With IronRuby, I can use the RSpec specification framework and runner to provide examples of how my C# objects work, following a BDD approach instead of TDD.
While the scenario framework, which I'll cover in a future article, is more aligned with acceptance testing at the app level, the specification framework is more aligned to developers writing their own specifications about the code they are just about to implement and its expected behavior. The specification framework is based around providing examples describing the behavior at the object level. These examples can be run to verify that the implementation of the system is still working as the developer expects while providing documentation about how the objects are expected to behave.
This is an important distinction compared to unit-testing frameworks such as NUnit and MbUnit. Both of those use test attributes to indicate a method is a test for the system. RSpec takes a different approach. RSpec says that each method is an example of how the code is intended to work. While the distinction is subtle, it changes the way you write the examples, including the terminology you use, the way you organize the methods, and how easily the concept can be understood compared to TDD. With BDD and RSpec, together with the close integration of IronRuby and the .NET Framework, you can start using IronRuby to test C# applications. In terms of an example, the classic RSpec example is the Bowling game.
First, you need to access the bowling game's implementation in a C# assembly:
require File.dirname(__FILE__) + \
  '/InteropSamples/Bowling/Bowling/bin/Debug/bowling.dll'
You then need to access RSpec:
require 'rubygems'
require 'spec'
Now you can begin writing examples. This is an example showing how bowling implementation should work. RSpec has a Domain Specific Language (DSL) that you must follow for the examples to be executable. The first part of the DSL is the describe block. Here you simply state the object you are "describing" together with an optional description. In this case, I define the Bowling object, which will be implemented in C#:
describe Bowling, " defines the bowling game" do
To improve readability, any setup will be performed within a before block:
before(:each) do
    @bowling = Bowling.new
  End
I am now in a position to interact with the object, create the example and verify that it works correctly. I use the "it" block, providing a string stating the context of the example and what it is demonstrating. I then write the section of code that interacts with the system and verifies the correct action has happened:
it "should score 0 for a gutter game" do
    20.times { @bowling.hit(0) }
    @bowling.score.should == 0
  end
end
The C# implementation simply looks like this:
public class Bowling {
  public int Score { get; set; }
  public void Hit(int pins)
    { Score += pins; }
}
Finally, execute this to verify that the bowling example works as expected with RSpec testing C# objects:
>ir bowling_spec.rb
.
Finished in 1.0458315 seconds
1 example, 0 failures
If I wanted a more detailed report, I could select the format to be specdoc. This outputs the object description together with all the examples, stating if they have passed or not:
>ir bowling_spec.rb --format specdoc
Bowling defines the bowling game
- should score 0 for a gutter game
Finished in 1.43728 seconds
1 example, 0 failures
At the time of writing, the IronRuby team is not regularly shipping binaries. The team has said it is waiting until it is happy with the implementation, compatibility, and performance before releasing official binaries. However, because the source code is freely available, you can download the code and build it yourself.
To download the source code, you will need to have a Git source control client, such as msysgit, installed. The IronRuby source control is available online. If you would like more information then I recommend you visit the IronRuby project site or my blog post "Downloading IronRuby from GitHub." Once you have downloaded the source code, you can compile the assembly either with Visual Studio using the IronRuby.sln solution file or, if you have MRI installed, then you can use the command:
rake compile
Once you have IronRuby compiled, you must download various libraries, such as RSpec, to gain the extra functionality. Ruby has a concept of RubyGems where the gems are packages you can download to gain the functionality and the additional dependencies. To download RSpec, type the following at a command prompt:
gem install rspec
You would now be able to access the RSpec library from within your Ruby code. However, at the time of writing, RSpec does not work with IronRuby due to one or two bugs. Hopefully by the time this article is published RSpec should be working with IronRuby. To find out the status of the RSpec support, see the RSpec Web site or the IronRuby mailing list.
If the bug hasn't been fixed, I have made a binary with the bug fixed, together with the RSpec library in place. (Note: don't use this version once the team has fixed the problem.)
Once you have the correct binaries in place, ir.exe is the IronRuby interpreter that can execute your code. If you simply launch the app from the command line, you'll enter the interactive console. In Ruby, code is executed on a line-by-line basis, excellent for learning and debugging, but also for running short commands to solve daily problems. If you provide a file name as a parameter, then the file is executed with the results output to the console.

Moving Forward
In my next article, I will introduce the concept of acceptance testing and how it can improve communication between customer and developer. I will demonstrate how acceptance testing can be automated using IronRuby and RSpec to verify .NET applications and create an executable specification for the system.

Ben Hall is a C# developer/tester with a strong passion for software development and loves writing code. Ben works at Red Gate Software in the U.K. as a Test Engineer and enjoys exploring different ways of testing software, including both manual and automated testing, focusing on the best ways to test different types of applications. Ben is a C# MVP and maintains a blog at Blog.BenHall.me.uk.


"Oslo" Basics
Build Metadata-Based Applications With The “Oslo” Platform
Chris Sells

This article is based on a prerelease version of the "Oslo" platform. All information is subject to change.

This article discusses:
  • ”Oslo” and metadata
  • MGraph and MSchema
  • SQL scripts, packaging, and deployment
  • Visual Studio integration
This article uses the following technologies:
”Oslo,” SQL Server 2008
Contents
Metadata
Building on "Oslo"
MGraph and MSchema
SQL Generation
Packaging and Deployment
Visual Studio Integration
Sample Code
An application based on the Microsoft .NET Framework is traditionally defined by a set of compiled code and a set of associated resources. For example, an ASP.NET application is defined by compiled code (assemblies) that provide the logic of your Web site and the resource files (.aspx files) that define the pages themselves. The code is executed by the CPU (after compilation) and the pages are interpreted by the ASP.NET runtime. Similarly, Windows Workflow Foundation (WF) lets you define behaviors as a series of configured activities, all of which is just data to be interpreted by the WF runtime.
Combine ASP.NET on the UI side and WF on the behavior side and you can start to imagine building entire applications as almost completely data, using code only to fill in behaviors not provided by the runtimes.
The data that drives these runtimes is sometimes called metadata. Metadata can be defined as the data that describes an application, for example, how a home page is rendered or how the purchasing workflow operates. Your application data represents the state of the execution of an application, such as what Bob put in his shopping cart on your Web site or his shipping address in your checkout workflow.
To provide a common set of tools for defining metadata so that it can be stored in a place that provides the same set of features as normal application data, Microsoft is creating "Oslo," a platform for building data-driven applications. "Oslo" is composed of three elements: a family of languages collectively called "M," a visual data manipulation tool called "Quadrant," and a data store called the repository.
In this article, I will introduce you to some of the concepts of the "Oslo" platform. I will also introduce you to the tools and technologies you need to employ to put those concepts into practice. In particular, I will discuss how "Oslo" enables you to build metadata-driven apps, using the MSchema and MGraph tools, and the basics of defining types and values in "M" as well as deploying them to the repository.

“Olso”: A New Way to Implement an Old Idea
The idea of data-driven frameworks is nothing new. Consider another example: Windows Presentation Foundation (WPF). The way most WPF applications are written, much of the application is composed of XAML, which is loaded at run time and interpreted by the XAML parser. This is the metadata that drives the definition of a WPF application. You can even change the definition of a WPF application by loading or generating XAML at run time that the developer didn't know about at compile time.
However, there's nothing magical about the XAML (compiled into a form known as BAML) being bundled into a resource with your application code—it could as easily be loaded into a separate file or even into a relational database. If it were loaded into a database, the WPF loader would perform SQL queries to access application definitions instead of loading resources, but the user would otherwise be unaware of the difference. And if an application definition would loaded from a database, all of that data could be updated centrally, giving the developers of the application the ability to propagate changes without re-deploying the application to every client that wanted to run it. This is, in fact, the killer feature of the Web and one worth driving towards for all kinds of applications, whether they run in the browser or not.
Now, as soon as you load an application into a central location, you're going to want multi-user features such as replication, securit,y and versioning. You'll also want support for disconnected users and for users in different locales around the world.
And finally, application definitions can be used for more than just driving an instance of an application. They can be used at development time to test whether particular parts of an application are running according to specifications. They can be used at run time to show what parts are getting usage or having trouble. They can be used to tag issues so developers can see specifically what's causing the trouble. Having an application's metadata provides all the query power of a modern, indexed, tuned, replicated, secured, versioned, localized, offline-capable database.
And that's essentially what "Oslo" is all about—using the power of SQL Server to store an application's metadata. Using the metadata to drive the behavior of an application has been used on Windows at least since we started tagging binaries with the console bit to get a console window without writing the code. "Oslo" is simply about formalizing and generalizing the metadata of your application and providing the same power and tools used for the data of your applications.
Metadata
What makes metadata distinct from your application data is that the metadata for an application seldom changes—it's typically something that even in the wildest startups only changes once a day when a new version of a Web site is pushed live. On the other hand, the data for an application often changes quite a lot—it's what users are manipulating all day, every day on a popular Web site.
Taking this further, the metadata that describes the word processor I'm using to type this article hasn't changed since 2007 when it was released, but the data has changed thousands of times just today as I type characters and move the mouse to manipulate the documents I'm working on. In fact, metadata is written so seldom that in many cases it's delivered in read-only packages, like the class definitions in a .NET assembly or the precompiled XAML files embedded as resources into an assembly. Since developers often write their applications depending on the static nature of metadata, like the event handlers for a button defined in an ASP.NET page, changing an application's metadata seldomly isn't a bad thing.
However, the thing that is kind of bad about our current metadata infrastructure is that the tools for defining and reading metadata are so different from those you would use for application data.
I can develop an application to read and write data via a relational database, giving me a powerful set of tools for things like querying, securing, replicating, and versioning. But if I want to query .NET Framework metadata, I have a much more limited set of tools: the Reflection API, basically.
Further, the .NET Framework tools for manipulating metadata are very different from the tools I use to crack open SQL metadata. If I want to query between an application's data and metadata—query across the workflows currently being executed with the definitions of those workflows—I've got to build my own tools because the data sources are different: SQL versus Extensible Object Markup Language (XOML) files.

Building on "Oslo"
There are many things that you can build with "Oslo." When we talk about scenarios internally, the discussion often boils down to building applications that involve data. However, you'll notice that in the set of things that I listed as part of "Olso" I didn't mention any runtimes. In fact, .aspx files aren't too useful without ASP.NET, and XOML files without WF would be much less compelling.
There are a number of runtimes and services being developed that will take advantage of "Oslo." They include domain-specific languages (DSLs) and Quadrant designers for for ASP.Net Web applications, WCF Web Services, and EF Databases. You'll be able to load data into the repository and drive those runtimes.
The "Oslo" repository is geared toward apps where enterprise developers have built their own repository-like things for storing metadata. For example, "Oslo" would be a good platform for a repository of machine configuration information that would allow operations staff to query and understand machine configurations easily, or similarly, a repository of operational procedures and scripts. For developers, because the repository provides a unified and queriable store for metadata, it would be possible to answer questions such as "Which applications and classes implement a certain method?" and "Will changing this method impact performance?"
Another potential area for employing "Oslo" could be for understanding distributed applications. Combining profiling, code coverage, load testing, and architectural data within the repository will allow developers to, for example, quickly resolve performance issues by browsing to the related distributed component, examining the settings of various configuration parameters, and viewing relevant implementation details. In addition, it will allow for the development of static analysis tools that automatically report design defects.
One area that particularly tickles my fancy is deployment and installation. By defining setup packages in a domain-specific language and loading them into the repository, a developer gets the language services of any modern language along with query tools for policy compliance. For example, you could query whether setups are protected with appropriate license agreements.
Here's an example: the Microsoft download center Web site provides for highly scalable downloads for customers. For files to be hosted on that site, they must be packaged in a Microsoft Installer (MSI) file and tagged with an end-user license agreement (EULA). After doing this about half a dozen times using the GUI in Visual Studio, I went looking for an automated solution and found one in Windows Installer XML (WiX), which let me use XML to define the setup and compile it into an MSI.
Figure 1 shows an example of the WiX XML. To the untrained eye, this might not seem like an improvement over running a graphical UI, but once you've done the same 27 mouse gestures a few times, the programmer in you will take over.
Figure 1 WiX XML to Create an MSI File
<?xml version='1.0' encoding='Windows-1252'?>
<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
  <Product Name="SuperNotepad" Id="d65d2125-560b-4ba9-bdca-3b3bcd3f7b70"
           UpgradeCode="ac5c9dac-b3f0-469a-8f1a-02c2566eee33"
           Language="1033" Codepage="1252"
           Version="1.0.0.0" Manufacturer="SuperNotepad Corp.">
    <Package Description="A really super pad for notes"
             Manufacturer="SuperNotepad Corp."
             InstallerVersion="200" Languages="1033" Compressed="yes" />
    <Media Id="1" Cabinet="setup.cab" EmbedCab="yes" />
    <Directory Id="TARGETDIR" Name="SourceDir">
      <Directory Id="ProgramFilesFolder" Name="ProgramFilesFolder">
        <Directory Id="Directory3" Name="SuperNotepad Corp.">
          <Directory Id="INSTALLDIR" Name="Better NotePad">
            <Component Id="Component1"
                       Guid="f7554ac8-32cd-44fb-bf85-0559c7d2e15c">
              <File Id="File1" Name="Manual.txt"
                    Source="C:\temp\SuperNotepad\Files\manual.txt" />
              <File Id="File2" Name="dependency.dll"
                    Source="C:\temp\SuperNotepad\Files\dependency.dll" />
              <File Id="File3" Name="BetterNotePad.exe"
                    Source="C:\temp\SuperNotepad\Files\notepad.exe" />
            </Component>
          </Directory>
        </Directory>
      </Directory>
    </Directory>
    <Feature Id="Feature1" Level="1">
      <ComponentRef Id="Component1" />
    </Feature>
  </Product>
</Wix>

MGraph and MSchema
Even though I can automate creation of the WiX XML to some extent, I don't really want to write my setups in a sea of angle brackets. "Oslo" lets me write them in "M", a family of languages including MSchema, MGrammar, and MGraph. (The MSchema and MGrammar specifications can be found at "The "Oslo" Modeling Language Specification" and "MGrammar Language Specification," respectively.)
Let me give you a quick intro to MGraph and MSchema (I'll explore MGrammar in a future article). If you'd like to play along at home, I highly recommend the Mr. Epl tool that comes with the "Oslo" SDK. ("Mr. Epl" is how we pronounce MREPL.EXE, which stands for M Read-Evaluate-Print-Loop.) You can use it from within the text editor called Intellipad that ships in the "Oslo" SDK by choosing Intellipad (Samples Enabled) from the Start menu, pressing Ctrl+/ to bring up the mini-buffer, and typing "SetMode('MScriptMode')" (no quotes, but case matters), followed by clicking Enter.
MGraph is a language for defining instances of structured data (better known as values). In this example, I've defined three pieces of data: an anonymous integer, an anonymous collection of three integers, and a record named "pt1" with two fields, X and Y, both integers:
42
{ 1, 2, 3 }
pt1 { X = 10, Y = 20 }
MSchema is a language for defining constraints over data. In MSchema, a set of constraints is tied together into a type. Here I've defined two named types, a SmallInteger type that constrains the built in Integer32 type to only those values less than 256, and a Point entity type (the name for record types in M) with two fields, X and Y:
type SmallInteger : Integer32 where value < 256;
type Point { X : Integer32; Y : Integer32; };
Both X and Y are constrained to hold integer values between -2,147,483,648 and 2,147,483,647.
Anyone reading this article is already familiar with the CLR. The CLR uses something called "nominal typing," that is, naming the single type to which a value belongs. "M," on the other hand, is based on structural typing, much like XML. In a structurally typed world, a value can belong to any number of types so long as the data meets the constraints set by that type. To check whether a value is in the set defined by a type, you use the "in" operator (shown here from inside the Mr. Epl tool):
M>>> type SmallInteger : Integer32 where value < 256;
M>>> 4 in SmallInteger
true
M>>> 4 in Integer32
true
M>>> 256 in SmallInteger
false
M>>> 256 in Integer
true
Here you can see that 4 is a SmallInteger and an Integer32, but 256 is only an Integer32. Entity values can be checked for type membership as well:
M>>> type Point { X : Integer32; Y : Integer32; };
M>>> { X = 100, Y = 200 } in Point
true
M>>> pt1 { X = 10, Y = 20 }
M>>> pt1 in Point
true
M>>> pt1 in { X : SmallInteger; Y : SmallInteger; }
true
M>>> pt1 in { X; Y; }
true
You can see that the named and unnamed X, Y pairs are both Points and that both fields of the named X, Y pair are SmallInteger values, this time defining an anonymous type to do the checking for you. The last line indicates that the named X, Y pair fits into the set of values that have fields named X and Y, with no restriction on their values—they could be strings or dates or collections or whatever.
Here's another example. In this case, because the named X, Y pair doesn't also have a Z field, it doesn't fit into the anonymous type that requires X, Y, and Z fields:
M>>> pt1 in { X; Y; Z; }
false
M>>> pt1 in { x; y; }
false
Further, the last line shows that M is case-sensitive.
Finally, you'll want to understand the default value and collection notations:
type Product {
    Name : Text;
    ...
};

type Package {
    Product : Product; // required value
    Keywords : Text*; // collection of 0 or more values
    Description : Text?; // optional value
    Comments : Text?; // optional value
    Manufacturer : Text; // required value
    InstallerVersion : Integer32 = 200; // default value
    Language : Integer32 = 1033; // default value
    Compressed : Logical = true; // defaults value
};
Here you see the Package type from the WiX module. Every product (the top-level concept in an MSI setup) has a package, which has an optional set of text keywords, an optional description and comment, a required manufacturer, an installer version that defaults to Windows Installer 2.0, a language the defaults to U.S. English and a compression flag that defaults to true.
The required values have no suffix, like Product and Manufacturer. The default values have an equals sign and a value, such as InstallerVersion and Language. The optional values have a question mark suffix, such as Description and Comments. The collection values (those with zero or more values) end in star. The question mark and star notions (? and *) are meant to remind you of the Kleene operators from your favorite regular expression language.
Here's an example Package value:
Products {
    SuperNotepad : Product* {
        Name = "SuperNotepad",
        ...
    }
}

Packages : Package* where item.Product in Products {
    Package1 {
        Product = Products.SuperNotepad,
        Description = "A really super pad for notes",
        Manufacturer = Products.SuperNotepad.Manufacturer,
        Keywords = { "Installer", "Super", "Notepad", "Sample" }
    }
}
Here I've got part of a Product and an entire Package. Notice that I have two kinds of constraints. The first kind is of the form ": something", as in:
SuperNotepad : Product* {
This is called type ascription and constraints all values inside the curly braces to conform to the constraints of the type.
The second constraint is the where clause that tells the "M" compiler where to find the storage for the Product entity values:
Packages : Package* where item.Product in Products {
Also notice that two of the values have names (SuperNotepad and Package1). The MGraph syntax allows for names to be used in the middle of initializing a graph of values so that one part of the graph can reference another part without duplicating the data. I am doing this so that I can reference my product in the definition of the package.
As useful as Mr. Epl is, you'll want to use a text editor for larger things. Both Visual Studio 2008 and Intellipad provide language services to make editing "M" a pleasant experience. For example, to check that you've got something syntactically valid, you can look for data tips over the red squiggles showing errors (for example, a missing closing double quote).
All "M" code must be inside a module, much like all .NET-based code must be inside a namespace. This scopes the types and values.
If the lack of squiggles isn't enough to make you happy, you can also run the "M" compiler from the command line:
C:\>set path=%path%;"c:\Program Files\Microsoft Oslo SDK 1.0\Bin"
...
C:\>m.exe /t:none wixbits.m
Microsoft (R) "Codename M" Compiler version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.
C:\>
If there are no errors reported by the "M" compiler, your code is clean. Notice here that in addition to passing a file with a .m extension to m.exe, I am also specifying a target (via the /t switch) of none. This means to just do the syntax checking. Without the target switch, the "M" compiler will attempt to use your program to generate a SQL script to create tables and insert values.

SQL Generation
If you try to generate SQL from the sample bit of WiX model data (after adding the closing double quote), you'll get an error report like this one:
C:\>m wixbits.m
Microsoft (R) "Codename M" Compiler version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.

C:\wixbits.m(16,9): error M2010: The referenced type 'Product' does not define an identity constraint.
...
You can also get this error message if you choose M Mode | Reach SQL Preview from inside Intellipad. What's happening is that the "M" compiler is trying to generate a SQL script that matches the types and values defined in the program. It does so by attempting to map top-level named values (called extents) to create table statements and to map the values themselves into insert statements. However, because "M" has all kinds of general-purpose data description uses, not everything in it maps to SQL constructs—for example, entity types without identity fields. It's still valid "M," but not valid for generating SQL.
Generating SQL is the default action of the "M" compiler because a modern relational database is a fabulous environment for storing and manipulating data. It has all kinds of wonderful features such as security, replication, versioning, auditing, and on and on. If you make use of SQL Server for hosting your data, you get all of those features, plus a host of robustness and performance optimizations.
To make your "M" types compatible with SQL Server, the mapping requires an identity column, which you can add using the following code:
type Package {
    Id : Integer32 = AutoNumber();
    ...
} where identity Id;
As you can see, I am creating a field named Id and using the AutoNumber function to set a default value. Then, I add a constraint on the type using the where keyword that marks the Id field as the identity column. With this in place, I have enough data to get valid SQL for the type, as shown in Figure 2.
Figure 2 SQL Generated by "M"
...
create table [WixBits].[Packages](
  [Id] int not null identity,
  [Comments] nvarchar(max) null,
  [Compressed] bit not null default 1,
  [Description] nvarchar(max) null,
  [InstallerVersion] int not null default 200,
  [Language] int not null default 1033,
  [Manufacturer] nvarchar(max) not null,
  [Product] int not null,
  constraint [PK_Packages] primary key clustered ([Id]),
  constraint [FK_Packages_Product_WixBits_Products] foreign key ([Product]) references [WixBits].[Products] ([Id])
);
...

create table [WixBits].[Packages_Keywords](
  [_Id] bigint not null identity,
  [Packages_Id] int not null,
  [Item] nvarchar(max) not null,
  constraint [PK_Packages_Keywords] primary key clustered ([_Id]),
  constraint [FK_Packages_Keywords_Packages_Id_WixBits_Packages] foreign key
  ([Packages_Id]) references [WixBits].[Packages] ([Id]) on delete cascade
);
...
Hopefully the SQL generated by the "M" compiler matches what you'd write yourself. Module names map to SQL schema names. Type names map to table names. AutoNumber maps to "identity" on the field, and the identity constraint maps to the primary key constraint on the field. Intrinsic "M" types such as Integer32 and Text map go corresponding SQL types such as int and nvarchar. Collection fields—the Keywords field of type Text*—maps to another table to hold the zero or more items in that collection, with the corresponding foreign key constraint. Figure 3 illustrates how the values map.
Figure 3 Mapping "M" Types to SQL
insert into [WixBits].[Packages] ([Product], [Description], [Manufacturer])
 values (@WixBits_Products_Id0, N'A really super pad for notes', @WixBits_Products_Manufacturer0);
declare @WixBits_Packages_Id0 bigint = @@identity;

insert into [WixBits].[Packages_Keywords] ([Item], [Packages_Id])
 values (N'Installer', @WixBits_Packages_Id0);

insert into [WixBits].[Packages_Keywords] ([Item], [Packages_Id])
 values (N'Super', @WixBits_Packages_Id0);

insert into [WixBits].[Packages_Keywords] ([Item], [Packages_Id])
 values (N'Notepad', @WixBits_Packages_Id0);

insert into [WixBits].[Packages_Keywords] ([Item], [Packages_Id])
 values (N'Sample', @WixBits_Packages_Id0);

Packaging and Deployment
The default output of the "M" compiler is SQL, which you'll likely want to push into a SQL Server 2008 instance. By default, that SQL is packaged as a single SQL script file:
C:\>m.exe wixbits.m
Microsoft (R) "Codename M" Compiler version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.

C:\>dir wixbits.sql
...
11/22/2008  04:43 PM             2,545 wixbits.sql
...
You are free to use this script file to populate your SQL database instance however you like, such as with sqlcmd.exe or SQL Server Management Studio. However, the SQL script has a lot less of the metadata than the original "M" source code. If you'd like to maintain that metadata—and there is value in doing so, as I will explain in a moment—you can choose the "M" image packaging format with the /p:image command line switch to m.exe:
C:\>m.exe /p:image wixbits.m
Microsoft (R) "Codename M" Compiler version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.
C:\>dir wixbits.mx
...
11/22/2008  04:44 PM             9,073 wixbits.mx
...
The wixbits.mx file is an "M" image file, which contains the generated SQL as well as a manifest and other metadata. Since the file is in OPC (Open Packaging Conventions) format, you can stick a .zip file extension on it and open it just like any other .zip file. More importantly, however, you can use a .mx file for references and for deployment.
When you build a .NET assembly, you can reference that assembly as part of your build process to pull public types and static instances into your .NET-based programs. If you're using the C# or Visual Basic compilers from the command line, you can reference existing .NET assemblies with the /r (reference) switch. Likewise, if you're compiling your "M" programs, you can employ the /r switch on the "M" compiler to pull in other types from "M" images. For that to work, you must first export your types or extents from the module you're referencing:
// WixbitsTypes.m
module WixBits {
    export Product, Package; // export types
    export Products, Packages; // export extents

    type Product {...};
    type Package {...};
    Products : Product*;
    Packages : Package* where item.Product in Products;
}
You can then import types and extents that have been exported:
// WixbitsValues.m
module WixBits {
    import WixBits; // bring in exported types and extents
    ...
}
Once you've built an "M" image file with exports, you can reference that when compiling "M" programs with imports (for more information, see Figure 4).
Figure 4 Compiling with Imported Types and Extents
C:\>m.exe /p:image /t:Repository WixbitsTypes.m
Microsoft (R) "Codename M" Compiler version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.

C:\>dir WixbitsTypes.mx
...
11/22/2008  05:48 PM            28,332 WixbitsTypes.mx
...

C:\>m.exe /p:image /t:Repository /r:WixbitsTypes.mx WixbitsValues.m
Microsoft (R) "Codename M" Compiler version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.

C:\>dir WixbitsValues.mx
...
11/22/2008  05:50 PM             5,606 WixbitsValues.mx
In addition to packaging up types and extents for reference in other "M" programs, an "M" image file can be deployed to a SQL Server database instance using the "M" loader utility, mx.exe, in the following manner:
C:\>mx.exe /db:Repository /i:WixbitsTypes.mx /i:WixbitsValues.mx
Microsoft (R) "Codename M" Command-line Utility version 1.0.0925.0
Copyright (C) Microsoft Corporation. All rights reserved.
Here I am using the /db switch to specify which database to connect to, assuming the local host as the database server and installing the two image files. I load the tables into SQL and populate them with my values, which means I can use the data access technology of my choice to pull them back out again (or update them or delete them or insert new ones). In this case, I'm using the sqlcmd command-line tool:
C:\>sqlcmd -d Repository
1> select Name from WixBitx.Products
2> go
Name
----------------
SuperNotepad
(1 rows affected)
While Intellipad supports "M" language services out of the box, including generating SQL, there is no supported mechanism for building the image or referencing other images. However, Visual Studio supports these features.

Visual Studio Integration
The "Oslo" SDK ships with built-in support for Visual Studio 2008 SP1. After you install the SDK, you will have the "M" Project template installed, as shown in Figure 5.
By default, you'll get a Model1.m file added to your project. This provides the same IntelliSense features you got in Intellipad. When you need more model files, you can right-click on your project in the Solution Explorer, add a new item, and then choose the "M" Model template.
Figure 5 The “M” Project Template in Visual Studio 2008
Further, if you'd like to reference existing "M" images, you can do so by right-clicking on your project and choosing Add Reference, then choosing a .mx file. For example, if you'd like to create a project that builds the values I've been playing with and references the types I've been demonstrating, it will look like Figure 6.
Figure 6 An “M” Project with Source Code and .mx File Reference
When you build, you'll see the errors in the Error List as you'd expect and the output will go into \bin\Debug or \bin\Release (depending on the configuration that you're building), including both the .sql and .mx files for you to do with as you see fit (like run mx.exe on the output .mx file). All of these same things also work in the latest CTP of Visual Studio 2010.
The build process inside Visual Studio is driven by an MSBuild file with the .mproj extension (see Figure 7). This is handy if you'd like to use the "Oslo" msbuild targets file in your own project files.
Figure 7 MSBuild File for "M" Projects
<?xml version="1.0" encoding="utf-8"?>
<Project ...>
  <PropertyGroup>
    <MTarget>Repository</MTarget>
    <MPackageScript>true</MPackageScript>
    <MPackageImage>true</MPackageImage>
    <MTargetsPath Condition="$(MTargetsPath) == ''">
      $(MSBuildExtensionsPath)\Microsoft\M\v1.0</MTargetsPath>
    ...
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="MixbitsValues.m" />
  </ItemGroup>
  <ItemGroup>
    <MReference Include="C:\WixbitsTypes.mx" />
  </ItemGroup>
  <Import Project="$(MTargetsPath)\MProject.targets" />
</Project>

Sample Code
The sample code included with this article is a toy Repository Installer utility (repinst.exe) that understands a slightly more full-featured model for describing MSI files that I've been discussing here. Once the types and values that describe the WiX models have been compiled (using m.exe) and loaded into the repository (using mx.exe), running the repinst.exe utility on the value data will read the data from the repository, create an MSI file from it, and launch the setup, allowing anyone with access to the database to deploy applications thus loaded, as shown in Figure 8.
Figure 8 The repinst Sample in Action
I did some work on this sample, mostly rearranging for exposition, but the bulk of the work (and the credit!) goes to Thomas Delrue, Alejandro Trigo, and Giovanni Della-Libera for their work on modeling WiX, translating "M" into WiX, and producing an MSI from SQL data. I could never have written this article in the time I did without their ground-breaking work. Also, thanks to Josh Williams, Martin Gudgin, and John Doty for a whole other sample built to explore the features of the repository that you'll see on the "Oslo" Developer Center real soon now.

Chris Sells is a Program Manager for the Connected Systems Division at Microsoft where he is working on next-gen application development techniques. More info about Chris and his various projects can be found at sellsbrothers.com.


Patterns
WPF Apps With The Model-View-ViewModel Design Pattern
Josh Smith

This article discusses:
  • Patterns and WPF
  • MVP pattern
  • Why MVVM is better for WPF
  • Building an application with MVVM
This article uses the following technologies:
WPF, data binding
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Order vs. Chaos
The Evolution of Model-View-ViewModel
Why WPF Developers Love MVVM
The Demo Application
Relaying Command Logic
ViewModel Class Hierarchy
ViewModelBase Class
CommandViewModel Class
MainWindowViewModel Class
Applying a View to a ViewModel
The Data Model and Repository
New Customer Data Entry Form
All Customers View
Wrapping Up
Developing the user interface of a professional software application is not easy. It can be a murky blend of data, interaction design, visual design, connectivity, multithreading, security, internationalization, validation, unit testing, and a touch of voodoo. Considering that a user interface exposes the underlying system and must satisfy the unpredictable stylistic requirements of its users, it can be the most volatile area of many applications.
There are popular design patterns that can help to tame this unwieldy beast, but properly separating and addressing the multitude of concerns can be difficult. The more complicated the patterns are, the more likely that shortcuts will be used later on which undermine all previous efforts to do things the right way.
It is not always the design patterns at fault. Sometimes we use complicated design patterns, which require writing a lot of code because the UI platform in use does not lend itself well to a simpler pattern. What's needed is a platform that makes it easy to build UIs using simple, time-tested, developer-approved design patterns. Fortunately, Windows Presentation Foundation (WPF) provides exactly that.
As the software world continues to adopt WPF at an increasing rate, the WPF community has been developing its own ecosystem of patterns and practices. In this article, I'll review some of those best practices for designing and implementing client applications with WPF. By leveraging some core features of WPF in conjunction with the Model-View-ViewModel (MVVM) design pattern, I will walk through an example program that demonstrates just how simple it can be to build a WPF application the "right way."
By the end of this article, it will be clear how data templates, commands, data binding, the resource system, and the MVVM pattern all fit together to create a simple, testable, robust framework on which any WPF application can thrive. The demonstration program that accompanies this article can serve as a template for a real WPF application that uses MVVM as its core architecture. The unit tests in the demo solution show how easy it is to test the functionality of an application's user interface when that functionality exists in a set of ViewModel classes. Before diving into the details, let's review why you should use a pattern like MVVM in the first place.

Order vs. Chaos
It is unnecessary and counterproductive to use design patterns in a simple "Hello, World!" program. Any competent developer can understand a few lines of code at a glance. However, as the number of features in a program increases, the number of lines of code and moving parts increase accordingly. Eventually, the complexity of a system, and the recurring problems it contains, encourages developers to organize their code in such a way that it is easier to comprehend, discuss, extend, and troubleshoot. We diminish the cognitive chaos of a complex system by applying well-known names to certain entities in the source code. We determine the name to apply to a piece of code by considering its functional role in the system.
Developers often intentionally structure their code according to a design pattern, as opposed to letting the patterns emerge organically. There is nothing wrong with either approach, but in this article, I examine the benefits of explicitly using MVVM as the architecture of a WPF application. The names of certain classes include well-known terms from the MVVM pattern, such as ending with "ViewModel" if the class is an abstraction of a view. This approach helps avoid the cognitive chaos mentioned earlier. Instead, you can happily exist in a state of controlled chaos, which is the natural state of affairs in most professional software development projects!

The Evolution of Model-View-ViewModel
Ever since people started to create software user interfaces, there have been popular design patterns to help make it easier. For example, the Model-View-Presenter (MVP) pattern has enjoyed popularity on various UI programming platforms. MVP is a variation of the Model-View-Controller pattern, which has been around for decades. In case you have never used the MVP pattern before, here is a simplified explanation. What you see on the screen is the View, the data it displays is the model, and the Presenter hooks the two together. The view relies on a Presenter to populate it with model data, react to user input, provide input validation (perhaps by delegating to the model), and other such tasks. If you would like to learn more about the Model View Presenter, I suggest you read Jean-Paul Boodhoo's August 2006 Design Patterns column.
Back in 2004, Martin Fowler published an article about a pattern named Presentation Model (PM). The PM pattern is similar to MVP in that it separates a view from its behavior and state. The interesting part of the PM pattern is that an abstraction of a view is created, called the Presentation Model. A view, then, becomes merely a rendering of a Presentation Model. In Fowler's explanation, he shows that the Presentation Model frequently updates its View, so that the two stay in sync with each other. That synchronization logic exists as code in the Presentation Model classes.
In 2005, John Gossman, currently one of the WPF and Silverlight Architects at Microsoft, unveiled the Model-View-ViewModel (MVVM) pattern on his blog. MVVM is identical to Fowler's Presentation Model, in that both patterns feature an abstraction of a View, which contains a View's state and behavior. Fowler introduced Presentation Model as a means of creating a UI platform-independent abstraction of a View, whereas Gossman introduced MVVM as a standardized way to leverage core features of WPF to simplify the creation of user interfaces. In that sense, I consider MVVM to be a specialization of the more general PM pattern, tailor-made for the WPF and Silverlight platforms.
In Glenn Block's excellent article "Prism: Patterns for Building Composite Applications with WPF" in the September 2008 issue, he explains the Microsoft Composite Application Guidance for WPF. The term ViewModel is never used. Instead, the term Presentation Model is used to describe the abstraction of a view. Throughout this article, however, I'll refer to the pattern as MVVM and the abstraction of a view as a ViewModel. I find this terminology is much more prevelant in the WPF and Silverlight communities.
Unlike the Presenter in MVP, a ViewModel does not need a reference to a view. The view binds to properties on a ViewModel, which, in turn, exposes data contained in model objects and other state specific to the view. The bindings between view and ViewModel are simple to construct because a ViewModel object is set as the DataContext of a view. If property values in the ViewModel change, those new values automatically propagate to the view via data binding. When the user clicks a button in the View, a command on the ViewModel executes to perform the requested action. The ViewModel, never the View, performs all modifications made to the model data.
The view classes have no idea that the model classes exist, while the ViewModel and model are unaware of the view. In fact, the model is completely oblivious to the fact that the ViewModel and view exist. This is a very loosely coupled design, which pays dividends in many ways, as you will soon see.

Why WPF Developers Love MVVM
Once a developer becomes comfortable with WPF and MVVM, it can be difficult to differentiate the two. MVVM is the lingua franca of WPF developers because it is well suited to the WPF platform, and WPF was designed to make it easy to build applications using the MVVM pattern (amongst others). In fact, Microsoft was using MVVM internally to develop WPF applications, such as Microsoft Expression Blend, while the core WPF platform was under construction. Many aspects of WPF, such as the look-less control model and data templates, utilize the strong separation of display from state and behavior promoted by MVVM.
The single most important aspect of WPF that makes MVVM a great pattern to use is the data binding infrastructure. By binding properties of a view to a ViewModel, you get loose coupling between the two and entirely remove the need for writing code in a ViewModel that directly updates a view. The data binding system also supports input validation, which provides a standardized way of transmitting validation errors to a view.
Two other features of WPF that make this pattern so usable are data templates and the resource system. Data templates apply Views to ViewModel objects shown in the user interface. You can declare templates in XAML and let the resource system automatically locate and apply those templates for you at run time. You can learn more about binding and data templates in my July 2008 article, "Data and WPF: Customize Data Display with Data Binding and WPF."
If it were not for the support for commands in WPF, the MVVM pattern would be much less powerful. In this article, I will show you how a ViewModel can expose commands to a View, thus allowing the view to consume its functionality. If you aren't familiar with commanding, I recommend that you read Brian Noyes's comprehensive article, "Advanced WPF: Understanding Routed Events and Commands in WPF," from the September 2008 issue.
In addition to the WPF (and Silverlight 2) features that make MVVM a natural way to structure an application, the pattern is also popular because ViewModel classes are easy to unit test. When an application's interaction logic lives in a set of ViewModel classes, you can easily write code that tests it. In a sense, Views and unit tests are just two different types of ViewModel consumers. Having a suite of tests for an application's ViewModels provides free and fast regression testing, which helps reduce the cost of maintaining an application over time.
In addition to promoting the creation of automated regression tests, the testability of ViewModel classes can assist in properly designing user interfaces that are easy to skin. When you are designing an application, you can often decide whether something should be in the view or the ViewModel by imagining that you want to write a unit test to consume the ViewModel. If you can write unit tests for the ViewModel without creating any UI objects, you can also completely skin the ViewModel because it has no dependencies on specific visual elements.
Lastly, for developers who work with visual designers, using MVVM makes it much easier to create a smooth designer/developer workflow. Since a view is just an arbitrary consumer of a ViewModel, it is easy to just rip one view out and drop in a new view to render a ViewModel. This simple step allows for rapid prototyping and evaluation of user interfaces made by the designers.
The development team can focus on creating robust ViewModel classes, and the design team can focus on making user-friendly Views. Connecting the output of both teams can involve little more than ensuring that the correct bindings exist in a view's XAML file.

The Demo Application
At this point, I have reviewed MVVM's history and theory of operation. I also examined why it is so popular amongst WPF developers. Now it is time to roll up your sleeves and see the pattern in action. The demo application that accompanies this article uses MVVM in a variety of ways. It provides a fertile source of examples to help put the concepts into a meaningful context. I created the demo application in Visual Studio 2008 SP1, against the Microsoft .NET Framework 3.5 SP1. The unit tests run in the Visual Studio unit testing system.
The application can contain any number of "workspaces," each of which the user can open by clicking on a command link in the navigation area on the left. All workspaces live in a TabControl on the main content area. The user can close a workspace by clicking the Close button on that workspace's tab item. The application has two available workspaces: "All Customers" and "New Customer." After running the application and opening some workspaces, the UI looks something like Figure 1.
Figure 1 Workspaces
Only one instance of the "All Customers" workspace can be open at a time, but any number of "New Customer" workspaces can be open at once. When the user decides to create a new customer, she must fill in the data entry form in Figure 2.
Figure 2 New Customer Data Entry Form
After filling in the data entry form with valid values and clicking the Save button, the new customer's name appears in the tab item and that customer is added to the list of all customers. The application does not have support for deleting or editing an existing customer, but that functionality, and many other features similar to it, are easy to implement by building on top of the existing application architecture. Now that you have a high-level understanding of what the demo application does, let's investigate how it was designed and implemented.

Relaying Command Logic
Every view in the app has an empty codebehind file, except for the standard boilerplate code that calls InitializeComponent in the class's constructor. In fact, you could remove the views' codebehind files from the project and the application would still compile and run correctly. Despite the lack of event handling methods in the views, when the user clicks on buttons, the application reacts and satisfies the user's requests. This works because of bindings that were established on the Command property of Hyperlink, Button, and MenuItem controls displayed in the UI. Those bindings ensure that when the user clicks on the controls, ICommand objects exposed by the ViewModel execute. You can think of the command object as an adapter that makes it easy to consume a ViewModel's functionality from a view declared in XAML.
When a ViewModel exposes an instance property of type ICommand, the command object typically uses that ViewModel object to get its job done. One possible implementation pattern is to create a private nested class within the ViewModel class, so that the command has access to private members of its containing ViewModel and does not pollute the namespace. That nested class implements the ICommand interface, and a reference to the containing ViewModel object is injected into its constructor. However, creating a nested class that implements ICommand for each command exposed by a ViewModel can bloat the size of the ViewModel class. More code means a greater potential for bugs.
In the demo application, the RelayCommand class solves this problem. RelayCommand allows you to inject the command's logic via delegates passed into its constructor. This approach allows for terse, concise command implementation in ViewModel classes. RelayCommand is a simplified variation of the DelegateCommand found in the Microsoft Composite Application Library. The RelayCommand class is shown in Figure 3.
Figure 3 The RelayCommand Class
public class RelayCommand : ICommand
{
    #region Fields

    readonly Action<object> _execute;
    readonly Predicate<object> _canExecute;

    #endregion // Fields

    #region Constructors

    public RelayCommand(Action<object> execute)
    : this(execute, null)
    {
    }

    public RelayCommand(Action<object> execute, Predicate<object> canExecute)
    {
        if (execute == null)
            throw new ArgumentNullException("execute");

        _execute = execute;
        _canExecute = canExecute;
    }
    #endregion // Constructors

    #region ICommand Members

    [DebuggerStepThrough]
    public bool CanExecute(object parameter)
    {
        return _canExecute == null ? true : _canExecute(parameter);
    }

    public event EventHandler CanExecuteChanged
    {
        add { CommandManager.RequerySuggested += value; }
        remove { CommandManager.RequerySuggested -= value; }
    }

    public void Execute(object parameter)
    {
        _execute(parameter);
    }

    #endregion // ICommand Members
}
The CanExecuteChanged event, which is part of the ICommand interface implementation, has some interesting features. It delegates the event subscription to the CommandManager.RequerySuggested event. This ensures that the WPF commanding infrastructure asks all RelayCommand objects if they can execute whenever it asks the built-in commands. The following code from the CustomerViewModel class, which I will examine in-depth later, shows how to configure a RelayCommand with lambda expressions:
RelayCommand _saveCommand;
public ICommand SaveCommand
{
    get
    {
        if (_saveCommand == null)
        {
            _saveCommand = new RelayCommand(param => this.Save(),
                param => this.CanSave );
        }
        return _saveCommand;
    }
}

ViewModel Class Hierarchy
Most ViewModel classes need the same features. They often need to implement the INotifyPropertyChanged interface, they usually need to have a user-friendly display name, and, in the case of workspaces, they need the ability to close (that is, be removed from the UI). This problem naturally lends itself to the creations of a ViewModel base class or two, so that new ViewModel classes can inherit all of the common functionality from a base class. The ViewModel classes form the inheritance hierarchy seen in Figure 4.
Figure 4 Inheritance Hierarchy
Having a base class for all of your ViewModels is by no means a requirement. If you prefer to gain features in your classes by composing many smaller classes together, instead of using inheritance, that is not a problem. Just like any other design pattern, MVVM is a set of guidelines, not rules.

ViewModelBase Class
ViewModelBase is the root class in the hierarchy, which is why it implements the commonly used INotifyPropertyChanged interface and has a DisplayName property. The INotifyPropertyChanged interface contains an event called PropertyChanged. Whenever a property on a ViewModel object has a new value, it can raise the PropertyChanged event to notify the WPF binding system of the new value. Upon receiving that notification, the binding system queries the property, and the bound property on some UI element receives the new value.
In order for WPF to know which property on the ViewModel object has changed, the PropertyChangedEventArgs class exposes a PropertyName property of type String. You must be careful to pass the correct property name into that event argument; otherwise, WPF will end up querying the wrong property for a new value.
One interesting aspect of ViewModelBase is that it provides the ability to verify that a property with a given name actually exists on the ViewModel object. This is very useful when refactoring, because changing a property's name via the Visual Studio 2008 refactoring feature will not update strings in your source code that happen to contain that property's name (nor should it). Raising the PropertyChanged event with an incorrect property name in the event argument can lead to subtle bugs that are difficult to track down, so this little feature can be a huge timesaver. The code from ViewModelBase that adds this useful support is shown in Figure 5.
Figure 5 Verifying a Property
// In ViewModelBase.cs
public event PropertyChangedEventHandler PropertyChanged;

protected virtual void OnPropertyChanged(string propertyName)
{
    this.VerifyPropertyName(propertyName);

    PropertyChangedEventHandler handler = this.PropertyChanged;
    if (handler != null)
    {
        var e = new PropertyChangedEventArgs(propertyName);
        handler(this, e);
    }
}

[Conditional("DEBUG")]
[DebuggerStepThrough]
public void VerifyPropertyName(string propertyName)
{
    // Verify that the property name matches a real,
    // public, instance property on this object.
    if (TypeDescriptor.GetProperties(this)[propertyName] == null)
    {
        string msg = "Invalid property name: " + propertyName;

        if (this.ThrowOnInvalidPropertyName)
            throw new Exception(msg);
        else
            Debug.Fail(msg);
    }
}

CommandViewModel Class
The simplest concrete ViewModelBase subclass is CommandViewModel. It exposes a property called Command of type ICommand. MainWindowViewModel exposes a collection of these objects through its Commands property. The navigation area on the left-hand side of the main window displays a link for each CommandViewModel exposed by MainWindowViewModel, such as "View all customers" and "Create new customer." When the user clicks on a link, thus executing one of those commands, a workspace opens in the TabControl on the main window. The CommandViewModel class definition is shown here:
public class CommandViewModel : ViewModelBase
{
    public CommandViewModel(string displayName, ICommand command)
    {
        if (command == null)
            throw new ArgumentNullException("command");

        base.DisplayName = displayName;
        this.Command = command;
    }

    public ICommand Command { get; private set; }
}
In the MainWindowResources.xaml file there exists a DataTemplate whose key is "CommandsTemplate". MainWindow uses that template to render the collection of CommandViewModels mentioned earlier. The template simply renders each CommandViewModel object as a link in an ItemsControl. Each Hyperlink's Command property is bound to the Command property of a CommandViewModel. That XAML is shown in Figure 6.
Figure 6 Render the List of Commands
<!-- In MainWindowResources.xaml -->
<!--
This template explains how to render the list of commands on
the left side in the main window (the 'Control Panel' area).
-->
<DataTemplate x:Key="CommandsTemplate">
  <ItemsControl ItemsSource="{Binding Path=Commands}">
    <ItemsControl.ItemTemplate>
      <DataTemplate>
        <TextBlock Margin="2,6">
          <Hyperlink Command="{Binding Path=Command}">
            <TextBlock Text="{Binding Path=DisplayName}" />
          </Hyperlink>
        </TextBlock>
      </DataTemplate>
    </ItemsControl.ItemTemplate>
  </ItemsControl>
</DataTemplate>

MainWindowViewModel Class
As previously seen in the class diagram, the WorkspaceViewModel class derives from ViewModelBase and adds the ability to close. By "close," I mean that something removes the workspace from the user interface at run time. Three classes derive from WorkspaceViewModel: MainWindowViewModel, AllCustomersViewModel, and CustomerViewModel. MainWindowViewModel's request to close is handled by the App class, which creates the MainWindow and its ViewModel, as seen in Figure 7.
Figure 7 Create the ViewModel
// In App.xaml.cs
protected override void OnStartup(StartupEventArgs e)
{
    base.OnStartup(e);

    MainWindow window = new MainWindow();

    // Create the ViewModel to which
    // the main window binds.
    string path = "Data/customers.xml";
    var viewModel = new MainWindowViewModel(path);

    // When the ViewModel asks to be closed,
    // close the window.
    viewModel.RequestClose += delegate
    {
        window.Close();
    };

    // Allow all controls in the window to
    // bind to the ViewModel by setting the
    // DataContext, which propagates down
    // the element tree.
    window.DataContext = viewModel;

    window.Show();
}
MainWindow contains a menu item whose Command property is bound to the MainWindowViewModel's CloseCommand property. When the user clicks on that menu item, the App class responds by calling the window's Close method, like so:
<!-- In MainWindow.xaml -->
<Menu>
  <MenuItem Header="_File">
    <MenuItem Header="_Exit" Command="{Binding Path=CloseCommand}" />
  </MenuItem>
  <MenuItem Header="_Edit" />
  <MenuItem Header="_Options" />
  <MenuItem Header="_Help" />
</Menu>
MainWindowViewModel contains an observable collection of WorkspaceViewModel objects, called Workspaces. The main window contains a TabControl whose ItemsSource property is bound to that collection. Each tab item has a Close button whose Command property is bound to the CloseCommand of its corresponding WorkspaceViewModel instance. An abridged version of the template that configures each tab item is shown in the code that follows. The code is found in MainWindowResources.xaml, and the template explains how to render a tab item with a Close button:
<DataTemplate x:Key="ClosableTabItemTemplate">
  <DockPanel Width="120">
    <Button
      Command="{Binding Path=CloseCommand}"
      Content="X"
      DockPanel.Dock="Right"
      Width="16" Height="16"
      />
    <ContentPresenter Content="{Binding Path=DisplayName}" />
  </DockPanel>
</DataTemplate>
When the user clicks the Close button in a tab item, that WorkspaceViewModel's CloseCommand executes, causing its RequestClose event to fire. MainWindowViewModel monitors the RequestClose event of its workspaces and removes the workspace from the Workspaces collection upon request. Since the MainWindow's TabControl has its ItemsSource property bound to the observable collection of WorkspaceViewModels, removing an item from the collection causes the corresponding workspace to be removed from the TabControl. That logic from MainWindowViewModel is shown in Figure 8.
Figure 8 Removing Workspace from the UI
// In MainWindowViewModel.cs

ObservableCollection<WorkspaceViewModel> _workspaces;

public ObservableCollection<WorkspaceViewModel> Workspaces
{
    get
    {
        if (_workspaces == null)
        {
            _workspaces = new ObservableCollection<WorkspaceViewModel>();
            _workspaces.CollectionChanged += this.OnWorkspacesChanged;
        }
        return _workspaces;
    }
}

void OnWorkspacesChanged(object sender, NotifyCollectionChangedEventArgs e)
{
    if (e.NewItems != null && e.NewItems.Count != 0)
        foreach (WorkspaceViewModel workspace in e.NewItems)
            workspace.RequestClose += this.OnWorkspaceRequestClose;

    if (e.OldItems != null && e.OldItems.Count != 0)
        foreach (WorkspaceViewModel workspace in e.OldItems)
            workspace.RequestClose -= this.OnWorkspaceRequestClose;
}

void OnWorkspaceRequestClose(object sender, EventArgs e)
{
    this.Workspaces.Remove(sender as WorkspaceViewModel);
}
In the UnitTests project, the MainWindowViewModelTests.cs file contains a test method that verifies that this functionality is working properly. The ease with which you can create unit tests for ViewModel classes is a huge selling point of the MVVM pattern, because it allows for simple testing of application functionality without writing code that touches the UI. That test method is shown in Figure 9.
Figure 9 The Test Method
// In MainWindowViewModelTests.cs
[TestMethod]
public void TestCloseAllCustomersWorkspace()
{
    // Create the MainWindowViewModel, but not the MainWindow.
    MainWindowViewModel target =
        new MainWindowViewModel(Constants.CUSTOMER_DATA_FILE);

    Assert.AreEqual(0, target.Workspaces.Count, "Workspaces isn't empty.");

    // Find the command that opens the "All Customers" workspace.
    CommandViewModel commandVM =
        target.Commands.First(cvm => cvm.DisplayName == "View all customers");

    // Open the "All Customers" workspace.
    commandVM.Command.Execute(null);
    Assert.AreEqual(1, target.Workspaces.Count, "Did not create viewmodel.");

    // Ensure the correct type of workspace was created.
    var allCustomersVM = target.Workspaces[0] as AllCustomersViewModel;
    Assert.IsNotNull(allCustomersVM, "Wrong viewmodel type created.");

    // Tell the "All Customers" workspace to close.
    allCustomersVM.CloseCommand.Execute(null);
    Assert.AreEqual(0, target.Workspaces.Count, "Did not close viewmodel.");
}

Applying a View to a ViewModel
MainWindowViewModel indirectly adds and removes WorkspaceViewModel objects to and from the main window's TabControl. By relying on data binding, the Content property of a TabItem receives a ViewModelBase-derived object to display. ViewModelBase is not a UI element, so it has no inherent support for rendering itself. By default, in WPF a non-visual object is rendered by displaying the results of a call to its ToString method in a TextBlock. That clearly is not what you need, unless your users have a burning desire to see the type name of our ViewModel classes!
You can easily tell WPF how to render a ViewModel object by using typed DataTemplates. A typed DataTemplate does not have an x:Key value assigned to it, but it does have its DataType property set to an instance of the Type class. If WPF tries to render one of your ViewModel objects, it will check to see if the resource system has a typed DataTemplate in scope whose DataType is the same as (or a base class of) the type of your ViewModel object. If it finds one, it uses that template to render the ViewModel object referenced by the tab item's Content property.
The MainWindowResources.xaml file has a ResourceDictionary. That dictionary is added to the main window's resource hierarchy, which means that the resources it contains are in the window's resource scope. When a tab item's content is set to a ViewModel object, a typed DataTemplate from this dictionary supplies a view (that is, a user control) to render it, as shown in Figure 10.
Figure 10 Supplying a View
<!--
This resource dictionary is used by the MainWindow.
-->
<ResourceDictionary
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:vm="clr-namespace:DemoApp.ViewModel"
  xmlns:vw="clr-namespace:DemoApp.View"
  >

  <!--
  This template applies an AllCustomersView to an instance
  of the AllCustomersViewModel class shown in the main window.
  -->
  <DataTemplate DataType="{x:Type vm:AllCustomersViewModel}">
    <vw:AllCustomersView />
  </DataTemplate>

  <!--
  This template applies a CustomerView to an instance
  of the CustomerViewModel class shown in the main window.
  -->
  <DataTemplate DataType="{x:Type vm:CustomerViewModel}">
    <vw:CustomerView />
  </DataTemplate>

 <!-- Other resources omitted for clarity... -->

</ResourceDictionary>
You do not need to write any code that determines which view to show for a ViewModel object. The WPF resource system does all of the heavy lifting for you, freeing you up to focus on more important things. In more complex scenarios, it is possible to programmatically select the view, but in most situations that is unnecessary.

The Data Model and Repository
You have seen how ViewModel objects are loaded, displayed, and closed by the application shell. Now that the general plumbing is in place, you can review implementation details more specific to the domain of the application. Before getting deep into the application's two workspaces, "All Customers" and "New Customer," let's first examine the data model and data access classes. The design of those classes has almost nothing to do with the MVVM pattern, because you can create a ViewModel class to adapt just about any data object into something friendly to WPF.
The sole model class in the demo program is Customer. That class has a handful of properties that represent information about a customer of a company, such as their first name, last name, and e-mail address. It provides validation messages by implementing the standard IDataErrorInfo interface, which existed for years before WPF hit the street. The Customer class has nothing in it that suggests it is being used in an MVVM architecture or even in a WPF application. The class could easily have come from a legacy business library.
Data must come from and reside somewhere. In this application, an instance of the CustomerRepository class loads and stores all Customer objects. It happens to load the customer data from an XML file, but the type of external data source is irrelevant. The data could come from a database, a Web service, a named pipe, a file on disk, or even carrier pigeons: it simply does not matter. As long as you have a .NET object with some data in it, regardless of where it came from, the MVVM pattern can get that data on the screen.
The CustomerRepository class exposes a few methods that allow you to get all the available Customer objects, add new a Customer to the repository, and check if a Customer is already in the repository. Since the application does not allow the user to delete a customer, the repository does not allow you to remove a customer. The CustomerAdded event fires when a new Customer enters the CustomerRepository, via the AddCustomer method.
Clearly, this application's data model is very small, compared to what real business applications require, but that is not important. What is important to understand is how the ViewModel classes make use of Customer and CustomerRepository. Note that CustomerViewModel is a wrapper around a Customer object. It exposes the state of a Customer, and other state used by the CustomerView control, through a set of properties. CustomerViewModel does not duplicate the state of a Customer; it simply exposes it via delegation, like this:
public string FirstName
{
    get { return _customer.FirstName; }
    set
    {
        if (value == _customer.FirstName)
            return;
        _customer.FirstName = value;
        base.OnPropertyChanged("FirstName");
    }
}
When the user creates a new customer and clicks the Save button in the CustomerView control, the CustomerViewModel associated with that view will add the new Customer object to the CustomerRepository. That causes the repository's CustomerAdded event to fire, which lets the AllCustomersViewModel know that it should add a new CustomerViewModel to its AllCustomers collection. In a sense, CustomerRepository acts as a synchronization mechanism between various ViewModels that deal with Customer objects. Perhaps one might think of this as using the Mediator design pattern. I will review more of how this works in the upcoming sections, but for now refer to the diagram in Figure 11 for a high-level understanding of how all the pieces fit together.
Figure 11 Customer Relationships

New Customer Data Entry Form
When the user clicks the "Create new customer" link, MainWindowViewModel adds a new CustomerViewModel to its list of workspaces, and a CustomerView control displays it. After the user types valid values into the input fields, the Save button enters the enabled state so that the user can persist the new customer information. There is nothing out of the ordinary here, just a regular data entry form with input validation and a Save button.
The Customer class has built-in validation support, available through its IDataErrorInfo interface implementation. That validation ensures the customer has a first name, a well-formed e-mail address, and, if the customer is a person, a last name. If the Customer's IsCompany property returns true, the LastName property cannot have a value (the idea being that a company does not have a last name). This validation logic might make sense from the Customer object's perspective, but it does not meet the needs of the user interface. The UI requires a user to select whether a new customer is a person or a company. The Customer Type selector initially has the value "(Not Specified)". How can the UI tell the user that the customer type is unspecified if the IsCompany property of a Customer only allows for a true or false value?
Assuming you have complete control over the entire software system, you could change the IsCompany property to be of type Nullable<bool>, which would allow for the "unselected" value. However, the real world is not always so simple. Suppose you cannot change the Customer class because it comes from a legacy library owned by a different team in your company. What if there is no easy way to persist that "unselected" value because of the existing database schema? What if other applications already use the Customer class and rely on the property being a normal Boolean value? Once again, having a ViewModel comes to the rescue.
The test method in Figure 12 shows how this functionality works in CustomerViewModel. CustomerViewModel exposes a CustomerTypeOptions property so that the Customer Type selector has three strings to display. It also exposes a CustomerType property, which stores the selected String in the selector. When CustomerType is set, it maps the String value to a Boolean value for the underlying Customer object's IsCompany property. Figure 13 shows the two properties.
Figure 12 The Test Method
// In CustomerViewModelTests.cs
[TestMethod]
public void TestCustomerType()
{
    Customer cust = Customer.CreateNewCustomer();
    CustomerRepository repos = new CustomerRepository(
        Constants.CUSTOMER_DATA_FILE);
    CustomerViewModel target = new CustomerViewModel(cust, repos);

    target.CustomerType = "Company"
    Assert.IsTrue(cust.IsCompany, "Should be a company");

    target.CustomerType = "Person";
    Assert.IsFalse(cust.IsCompany, "Should be a person");

    target.CustomerType = "(Not Specified)";
    string error = (target as IDataErrorInfo)["CustomerType"];
    Assert.IsFalse(String.IsNullOrEmpty(error), "Error message should
        be returned");
}
Figure 13 CustomerType Properties
// In CustomerViewModel.cs

public string[] CustomerTypeOptions
{
    get
    {
        if (_customerTypeOptions == null)
        {
            _customerTypeOptions = new string[]
            {
                "(Not Specified)",
                "Person",
                "Company"
            };
        }
        return _customerTypeOptions;
    }
}
public string CustomerType
{
    get { return _customerType; }
    set
    {
        if (value == _customerType ||
            String.IsNullOrEmpty(value))
            return;

        _customerType = value;

        if (_customerType == "Company")
        {
            _customer.IsCompany = true;
        }
        else if (_customerType == "Person")
        {
            _customer.IsCompany = false;
        }

        base.OnPropertyChanged("CustomerType");
        base.OnPropertyChanged("LastName");
    }
}
The CustomerView control contains a ComboBox that is bound to those properties, as seen here:
<ComboBox
  ItemsSource="{Binding CustomerTypeOptions}"
  SelectedItem="{Binding CustomerType, ValidatesOnDataErrors=True}"
  />
When the selected item in that ComboBox changes, the data source's IDataErrorInfo interface is queried to see if the new value is valid. That occurs because the SelectedItem property binding has ValidatesOnDataErrors set to true. Since the data source is a CustomerViewModel object, the binding system asks that CustomerViewModel for a validation error on the CustomerType property. Most of the time, CustomerViewModel delegates all requests for validation errors to the Customer object it contains. However, since Customer has no notion of having an unselected state for the IsCompany property, the CustomerViewModel class must handle validating the new selected item in the ComboBox control. That code is seen in Figure 14.
Figure 14 Validating a CustomerViewModel Object
// In CustomerViewModel.cs
string IDataErrorInfo.this[string propertyName]
{
    get
    {
        string error = null;

        if (propertyName == "CustomerType")
        {
            // The IsCompany property of the Customer class
            // is Boolean, so it has no concept of being in
            // an "unselected" state. The CustomerViewModel
            // class handles this mapping and validation.
            error = this.ValidateCustomerType();
        }
        else
        {
            error = (_customer as IDataErrorInfo)[propertyName];
        }

        // Dirty the commands registered with CommandManager,
        // such as our Save command, so that they are queried
        // to see if they can execute now.
        CommandManager.InvalidateRequerySuggested();

        return error;
    }
}

string ValidateCustomerType()
{
    if (this.CustomerType == "Company" ||
       this.CustomerType == "Person")
        return null;

    return "Customer type must be selected";
}
The key aspect of this code is that CustomerViewModel's implementation of IDataErrorInfo can handle requests for ViewModel-specific property validation and delegate the other requests to the Customer object. This allows you to make use of validation logic in Model classes and have additional validation for properties that only make sense to ViewModel classes.
The ability to save a CustomerViewModel is available to a view through the SaveCommand property. That command uses the RelayCommand class examined earlier to allow CustomerViewModel to decide if it can save itself and what to do when told to save its state. In this application, saving a new customer simply means adding it to a CustomerRepository. Deciding if the new customer is ready to be saved requires consent from two parties. The Customer object must be asked if it is valid or not, and the CustomerViewModel must decide if it is valid. This two-part decision is necessary because of the ViewModel-specific properties and validation examined previously. The save logic for CustomerViewModel is shown in Figure 15.
Figure 15 The Save Logic for CustomerViewModel
// In CustomerViewModel.cs
public ICommand SaveCommand
{
    get
    {
        if (_saveCommand == null)
        {
            _saveCommand = new RelayCommand(
                param => this.Save(),
                param => this.CanSave
                );
        }
        return _saveCommand;
    }
}

public void Save()
{
    if (!_customer.IsValid)
        throw new InvalidOperationException("...");

    if (this.IsNewCustomer)
        _customerRepository.AddCustomer(_customer);

    base.OnPropertyChanged("DisplayName");
}

bool IsNewCustomer
{
    get
    {
        return !_customerRepository.ContainsCustomer(_customer);
    }
}

bool CanSave
{
    get
    {
        return
            String.IsNullOrEmpty(this.ValidateCustomerType()) &&
            _customer.IsValid;
    }
}
The use of a ViewModel here makes it much easier to create a view that can display a Customer object and allow for things like an "unselected" state of a Boolean property. It also provides the ability to easily tell the customer to save its state. If the view were bound directly to a Customer object, the view would require a lot of code to make this work properly. In a well-designed MVVM architecture, the codebehind for most Views should be empty, or, at most, only contain code that manipulates the controls and resources contained within that view. Sometimes it is also necessary to write code in a View's codebehind that interacts with a ViewModel object, such as hooking an event or calling a method that would otherwise be very difficult to invoke from the ViewModel itself.

All Customers View
The demo application also contains a workspace that displays all of the customers in a ListView. The customers in the list are grouped according to whether they are a company or a person. The user can select one or more customers at a time and view the sum of their total sales in the bottom right corner.
The UI is the AllCustomersView control, which renders an AllCustomersViewModel object. Each ListViewItem represents a CustomerViewModel object in the AllCustomers collection exposed by the AllCustomerViewModel object. In the previous section, you saw how a CustomerViewModel can render as a data entry form, and now the exact same CustomerViewModel object is rendered as an item in a ListView. The CustomerViewModel class has no idea what visual elements display it, which is why this reuse is possible.
AllCustomersView creates the groups seen in the ListView. It accomplishes this by binding the ListView's ItemsSource to a CollectionViewSource configured like Figure 16.
Figure 16 CollectionViewSource
<!-- In AllCustomersView.xaml -->
<CollectionViewSource
  x:Key="CustomerGroups"
  Source="{Binding Path=AllCustomers}"
  >
  <CollectionViewSource.GroupDescriptions>
    <PropertyGroupDescription PropertyName="IsCompany" />
  </CollectionViewSource.GroupDescriptions>
  <CollectionViewSource.SortDescriptions>
    <!--
    Sort descending by IsCompany so that the ' True' values appear first,
    which means that companies will always be listed before people.
    -->
    <scm:SortDescription PropertyName="IsCompany" Direction="Descending" />
    <scm:SortDescription PropertyName="DisplayName" Direction="Ascending" />
  </CollectionViewSource.SortDescriptions>
</CollectionViewSource>
The association between a ListViewItem and a CustomerViewModel object is established by the ListView's ItemContainerStyle property. The Style assigned to that property is applied to each ListViewItem, which enables properties on a ListViewItem to be bound to properties on the CustomerViewModel. One important binding in that Style creates a link between the IsSelected property of a ListViewItem and the IsSelected property of a CustomerViewModel, as seen here:
<Style x:Key="CustomerItemStyle" TargetType="{x:Type ListViewItem}">
  <!--   Stretch the content of each cell so that we can
  right-align text in the Total Sales column.  -->
  <Setter Property="HorizontalContentAlignment" Value="Stretch" />
  <!--
  Bind the IsSelected property of a ListViewItem to the
  IsSelected property of a CustomerViewModel object.
  -->
  <Setter Property="IsSelected" Value="{Binding Path=IsSelected,
    Mode=TwoWay}" />
</Style>
When a CustomerViewModel is selected or unselected, that causes the sum of all selected customers' total sales to change. The AllCustomersViewModel class is responsible for maintaining that value, so that the ContentPresenter beneath the ListView can display the correct number. Figure 17 shows how AllCustomersViewModel monitors each customer for being selected or unselected and notifies the view that it needs to update the display value.
Figure 17 Monitoring for Selected or Unselected
// In AllCustomersViewModel.cs
public double TotalSelectedSales
{
    get
    {
        return this.AllCustomers.Sum(
            custVM => custVM.IsSelected ? custVM.TotalSales : 0.0);
    }
}

void OnCustomerViewModelPropertyChanged(object sender,
    PropertyChangedEventArgs e)
{
    string IsSelected = "IsSelected";

    // Make sure that the property name we're
    // referencing is valid.  This is a debugging
    // technique, and does not execute in a Release build.
    (sender as CustomerViewModel).VerifyPropertyName(IsSelected);

    // When a customer is selected or unselected, we must let the
    // world know that the TotalSelectedSales property has changed,
    // so that it will be queried again for a new value.
    if (e.PropertyName == IsSelected)
        this.OnPropertyChanged("TotalSelectedSales");
}
The UI binds to the TotalSelectedSales property and applies currency (monetary) formatting to the value. The ViewModel object could apply the currency formatting, instead of the view, by returning a String instead of a Double value from the TotalSelectedSales property. The ContentStringFormat property of ContentPresenter was added in the .NET Framework 3.5 SP1, so if you must target an older version of WPF, you will need to apply the currency formatting in code:
<!-- In AllCustomersView.xaml -->
<StackPanel Orientation="Horizontal">
  <TextBlock Text="Total selected sales: " />
  <ContentPresenter
    Content="{Binding Path=TotalSelectedSales}"
    ContentStringFormat="c"
  />
</StackPanel>

Wrapping Up
WPF has a lot to offer application developers, and learning to leverage that power requires a mindset shift. The Model-View-ViewModel pattern is a simple and effective set of guidelines for designing and implementing a WPF application. It allows you to create a strong separation between data, behavior, and presentation, making it easier to control the chaos that is software development.
I would like to thank John Gossman for his help with this article.

Josh Smith is passionate about using WPF to create great user experiences. He was awarded the Microsoft MVP title for his work in the WPF community. Josh works for Infragistics in the Experience Design Group. When he is not at a computer, he enjoys playing the piano, reading about history, and exploring New York City with his girlfriend. You can visit Josh's blog at joshsmithonwpf.wordpress.com.


Silverlight
Build Line-Of-Business Enterprise Apps With Silverlight, Part 2
Hanu Kommalapati

This article discusses:
  • The Silverlight runtime environment
  • Silverlight asynchronous programming
  • Cross-domain policies
  • A sample enterprise application
This article uses the following technologies:
Silverlight 2
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Integration with Business Services
Service Invocation
Synchronized Service Calls
Message Entity Translation
Silverlight State Change after the Service Calls
Cross-Domain Policies
Cross Domain Policies for Web Services Hosted Outside IIS
Cross-Domain Policies for Services Hosted inside IIS
Application Security
Application Partitioning
Productivity and Beyond
During the first installment of this series, I introduced a call center scenario and showed a screen-population (screen pop) implementation through the connected sockets that utilized the asynchronous TCP sockets supported by Silverlight (please see "Build Line-Of-Business Enterprise Apps With Silverlight, Part 1").
The screen pop was implemented through a simulated call dispatcher that picked up a call from an internal queue and pushed notifications through the previously accepted socket connection cached in a generic list on the server. Here I will conclude by implementing application security, integrating with business services, and implementing cross-domain policies for Web services and application partitioning. The logical architecture of the call center application is shown in Figure 1. The authentication service will be implemented in the utility service while the business services, ICallService and IUserProfile, will be implemented inside the business service project, as the name suggests.
Figure 1 Silverlight Call Center Logical Architecture
Even though the diagram shows event streaming to utility services, in the interest of time, the downloadable demo does not include this functionality. The implementation of the event capture service feature will be similar to the business services implementation. However, business events that are not critical errors can be cached locally into isolated storage and dumped onto the server in a batch mode. I will begin the discussion with the implementation of business services and conclude with application partitioning.

Integration with Business Services
Integration with services is one of the important aspects of a line-of-business (LOB) application, and Silverlight provides ample components for accessing Web-based resources and services. HttpWebRequest, WebClient, and Windows Communication Foundation (WCF) proxy infrastructure are a few of the commonly used network components for HTTP-based interaction. In this article, I will use WCF services to integrate with the back-end business processes.
Most of us use Web services for integrating with the back-end data sources during the course of application development; WCF Web service access with Silverlight is not much different than it is with traditional applications such as ASP.NET, Windows Presentation Foundation (WPF) or Windows Forms applications. The differences are the binding support and the asynchronous programming model. Silverlight will only support basicHttpBinding and PollingDuplexHttpBinding. Note that HttpBinding is the most interoperable binding. For this reason, I will use it for all integration in this article.
PollingDuplexHttpBinding allows the use of callback contracts to push notifications over HTTP. My call center could have used this binding for screen-pop notifications. However, the implementation requires the caching of the HTTP connection on the server, thereby monopolizing one of the two concurrent HTTP connections allowed by browsers such as Internet Explorer 7.0. This can cause performance issues, as all the Web content will have to be serialized through one connection. Internet Explorer 8.0 allows six concurrent connections per domain and will address such performance issues. (Push notifications using PollingDuplexHttpBinding could be a topic for a future article when Internet Explorer 8.0 is widely available.)
Back to the application. When the agent accepts a call, the screen-pop process populates the screen with the caller information—in this case, the order details of the caller. The caller information should contain necessary information to uniquely identify the order in the back-end database. For this demo scenario, I will assume that the order number was spoken into the interactive voice response (IVR) system. The Silverlight application will call WCF Web services with the order number as the unique identifier. The service contract definition and the implementation is shown in Figure 2.
Figure 2 Business Service Implementation
ServiceContracts.cs
[ServiceContract]
public interface ICallService
{
    [OperationContract]
    AgentScript GetAgentScript(string orderNumber);
    [OperationContract]
    OrderInfo GetOrderDetails(string orderNumber);
}

[ServiceContract]
public interface IUserProfile
{
    [OperationContract]
    User GetUser(string userID);
}
CallService.svc.cs
[AspNetCompatibilityRequirements(RequirementsMode =
                            AspNetCompatibilityRequirementsMode.Allowed)]
public class CallService:ICallService, IUserProfile
{
  public AgentScript GetAgentScript(string orderNumber)
  {
    ...
    script.QuestionList = DataUtility.GetSecurityQuestions(orderNumber);
    return script;
  }

  public OrderInfo GetOrderDetails(string orderNumber)
  {
    ...
    oi.Customer = DataUtility.GetCustomerByID(oi.Order.CustomerID);
    return oi;
  }

  public User GetUser(string userID)
  {
    return DataUtility.GetUserByID(userID);
  }
 }
Web.Config
<system.servicemodel>
   <services>
     <endpoint binding="basicHttpBinding"                contract="AdvBusinessServices.ICallService"/>
     <endpoint binding="basicHttpBinding"                contract="AdvBusinessServices.IUserProfile"/>
   </services>
   <serviceHostingEnvironment aspNetCompatibilityEnabled="true" />
<system.servicemodel>
The implementation of these service endpoints is not really very interesting, as these are straightforward WCF implementations. For simplicity's sake, I will not use any database for business entities but will use in-memory List objects for storing Customer, Order, and User objects. The DataUtil class (not shown here but available in the code download) encapsulates access to these in-memory List objects.
Figure 3 Agent Script with Security Questions
WCF service endpoints for Silverlight consumption need access to the ASP.NET pipeline and hence require the AspNetCompatibilityRequirements attribute on the CallService implementation. This has to be matched by the <serviceHostingEnvironment/> setting in the web.config file.
As previously mentioned, Silverlight only supports basicHttpBinding and PollingDuplexHttpBinding. If you use the WCF Service Visual Studio template, it configures the endpoint binding to wsHttpBinding, which must be manually changed to basicHttpBinding before Silverlight can add a service reference for proxy generation. The ASP.NET hosting compatibility changes and binding changes will automatically be taken care of if CallService.svc is added to the AdvBusinessServices project using a Silverlight-enabled WCF Service Visual Studio template.

Service Invocation
Having implemented a Silverlight callable service, now it is time to create service proxies and use them to wire up the UI to the back-end service implementations. You can only generate proxies for WCF services reliably by using the Service References | Add Service Reference sequence in Visual Studio. The proxies in my demo were generated into the namespace CallBusinessProxy. Silverlight only allows asynchronous calls to the network resources, and service invocation is no exception. When a customer call comes in, the Silverlight client will listen to the notification and display an Accept/Reject dialog.
Once a call is accepted by the agent, the next step in the process is to call the Web service to retrieve the agent script that corresponds to the call situation. For this demo, I will only be using one script as displayed in Figure 3. The displayed script contains a greeting as well as a list of security questions. The agent will ensure that a minimum number of questions are answered before moving forward with the assistance.
The agent script is retrieved by accessing the ICallService.GetAgentScript(), providing the order number as the input. Consistent with the asynchronous programming model enforced by the Silverlight Web services stack, the GetAgentScript() is available as CallServiceClient.BeginGetAgentScript(). While making the service call, you will need to provide a callback handler, GetAgentScriptCallback, as shown in Figure 4.
Figure 4 Service Invocation and Silverlight UI Change
class Page:UserControl
{
   ...
   void _notifyCallPopup_OnAccept(object sender, EventArgs e)
   {
     AcceptMessage acceptMsg = new AcceptMessage();
     acceptMsg.RepNumber = ClientGlobals.currentUser.RepNumber;
     ClientGlobals.socketClient.SendAsync(acceptMsg);
     this.borderCallProgressView.DataContext = ClientGlobals.callInfo;
     ICallService callService = new CallServiceClient();
     IAsyncResult result =
        callService.BeginGetAgentScript(ClientGlobals.callInfo.OrderNumber,
                     GetAgentScriptCallback, callService);
     //do a preemptive download of user control
     ThreadPool.QueueUserWorkItem(ExecuteControlDownload);
     //do a preemptive download of the order information
     ThreadPool.QueueUserWorkItem(ExecuteGetOrderDetails,
                ClientGlobals.callInfo.OrderNumber);
   }

   void GetAgentScriptCallback(IAsyncResult asyncReseult)
   {

     ICallService callService = asyncReseult.AsyncState as ICallService;
     CallBusinessProxy.AgentScript svcOutputAgentScript =
                     callService.EndGetAgentScript(asyncReseult);
     ClientEntityTranslator astobas =
                               SvcScriptToClientScript.entityTranslator;
     ClientEntities.AgentScript currentAgentScript =
                             astobas.ToClientEntity(svcOutputAgentScript)
                             as ClientEntities.AgentScript;
     Interlocked.Exchange<ClientEntities.AgentScript>(ref
                   ClientGlobals.currentAgentScript, currentAgentScript);
     if (this.Dispatcher.CheckAccess())
     {
       this.borderAgentScript.DataContext = ClientGlobals.agentScript;
       ...
       this.hlVerifyContinue.Visibility = Visibility.Visible;
     }
     else
     {
       this.Dispatcher.BeginInvoke(
        delegate()
        {
          this.borderAgentScript.DataContext = ClientGlobals.agentScript;
          ...
          this.hlVerifyContinue.Visibility = Visibility.Visible;

        } );
       }
     }
   private void ExecuteControlDownload(object state)
   {
     WebClient webClient = new WebClient();
     webClient.OpenReadCompleted += new
       OpenReadCompletedEventHandler(OrderDetailControlDownloadCallback);
     webClient.OpenReadAsync(new Uri("/ClientBin/AdvOrderClientControls.dll",
                                                     UriKind.Relative));
   }
   ...
}
Since the result of the service call can only be retrieved from the callback handler, any changes to the Silverlight application state will have to happen in the callback handler. CallServiceClient.BeginGetAgentScript() is invoked by _notifyCallPopup_OnAccept running on the UI thread and queues the asynchronous request and immediately returns to the next statement. Since the agent script is not yet available, you have to wait until the callback is triggered before you cache the script and data bind it to the UI.
Successful completion of the service call triggers GetAgentScriptCallback, which retrieves the agent script, populates a global variable, and adjusts the UI by data binding the agent script to the appropriate UI elements. While adjusting the UI, the GetAgentScriptCallback will make sure that it is updated on the UI thread through the use of Dispatcher.CheckAccess().
UIElement.Dispatcher.CheckAccess() will compare the UI thread ID with that of the worker thread and return true if both threads are the same; otherwise, it returns false. When GetAgentScriptCallback executes on a worker thread (actually, since this will always execute on a worker thread you could simply call Dispatcher.BeginInvoke), CheckAccess() will return false and the UI will be updated by dispatching an anonymous delegate through Dispatcher.Invoke().

Synchronized Service Calls
Because of the asynchronous nature of the Silverlight networking environment, it is almost impossible to make an asynchronous service call on the UI thread and wait for it to complete with the intention of changing the application state based on the results of the call. In Figure 4, _notifyCallPopup_OnAccept needs to retrieve order details, transform the output message into a client entity, and save it to a global variable in a thread-safe manner. To accomplish this, one may be tempted to write the handler code as shown here:
CallServiceClient client = new CallServiceClient();
client.GetOrderDetailsAsync(orderNumber);
this._orderDetailDownloadHandle.WaitOne();
//do something with the results
But this code will freeze the application when it hits the this._orderDetailDownloadHandle.WaitOne() statement. This is because the WaitOne() statement blocks the UI thread from receiving any dispatched messages from other threads. Instead, you can schedule the worker thread to execute the service call, wait for the call to complete, and finish the post processing of the service output in its entirety on the worker thread. This technique is shown in Figure 5. To prevent the inadvertent use of blocking calls in the UI thread, I wrapped ManualResetEvent inside a custom SLManualResetEvent and test for UI thread when a call to WaitOne() is made.
Figure 5 Retrieve Order Details
void _notifyCallPopup_OnAccept(object sender, EventArgs e)
{
  ...
  ThreadPool.QueueUserWorkItem(ExecuteGetOrderDetails,
        ClientGlobals.callInfo.OrderNumber);
}
private SLManualResetEvent _ orderDetailDownloadHandle = new
        SLManualResetEvent();
  private void ExecuteGetOrderDetails(object state)
{
  CallServiceClient client = new CallServiceClient();
  string orderNumber = state as string;
  client.GetOrderDetailsCompleted += new
        EventHandler<GetOrderDetailsCompletedEventArgs>
        (GetOrderDetailsCompletedCallback);
  client.GetOrderDetailsAsync(orderNumber);
  this._orderDetailDownloadHandle.WaitOne();
  //translate entity and save it to global variable
  ClientEntityTranslator oito = SvcOrderToClientOrder.entityTranslator;
  ClientEntities.Order currentOrder =
        oito.ToClientEntity(ClientGlobals.serviceOutputOrder)
        as ClientEntities.Order;
  Interlocked.Exchange<ClientEntities.Order>(ref ClientGlobals.
       currentOrder, currentOrder);
}

void GetOrderDetailsCompletedCallback(object sender,
        GetOrderDetailsCompletedEventArgs e)
  {
    Interlocked.Exchange<OrderInfo>(ref ClientGlobals.serviceOutputOrder,
         e.Result);
    this._orderDetailDownloadHandle.Set();
  }
Since SLManualResetEvent is a general-purpose class, you can't depend on the Dispatcher.CheckAccess() of a particular control. ApplicationHelper.IsUiThread() can check Application.RootVisual.Dispatcher.CheckAccess(); however, access to this method will trigger an invalid cross-thread access exception. So the only reliable way of testing this in a worker thread, when there is no access to a UIElement instance, is to use Deployment.Current.Dispatcher.CheckAccess() as shown here:
public static bool IsUiThread()
    {
        if (Deployment.Current.Dispatcher.CheckAccess())
            return true;
        else
            return false;
    }
For the background execution of tasks, instead of using ThreadPool.QueueUserWorkItem, you could use BackGroundWorker, which will also use ThreadPool but allows you to wire handlers that can execute on the UI thread. This pattern allows the execution of several service calls in parallel and waits for all the calls to complete using SLManualResetEvent.WaitOne() before the results are aggregated for further processing.

Message Entity Translation
The GetAgentScriptCallback also translates the output message entities (also known as DataContracts) from the service into a client-side entity that represents the client-side usage semantics. For example, the design of server-side message entities may not care about data binding while paying close attention to the multiuse nature of the service that will have to serve a broad range of uses, not only the call center.
Also, it is a good practice not to have tight coupling with the message entities, because changes to the message entities will not be within the client's control. The practice of translating message entities to client-side entities is not just applicable to Silverlight, but is generally applicable to any Web service consumer when wanting to avoid design-time tight coupling.
I decided to keep the implementation of the entity translators very simple—no exotic nested generics, lambda expressions, or inversion of control containers. ClientEntityTranslator is an abstract class that defines the ToClientEntity() method, which every subclass must override:
public abstract class ClientEntityTranslator
{
  public abstract ClientEntities.ClientEntity ToClientEntity(object
                                                 serviceOutputEntity);
}
Each child class is unique to a service exchange type; hence I will create as many translators as necessary. In my demo, I have made three types of service calls: IUserProfile.GetUser(), ICallService.GetAgentScript(), and ICallService.GetOrderDetails(). So I created three translators, as shown in Figure 6.
Figure 6 Message Entity to Client-Side Entity Translator
public class SvcOrderToClientOrder : ClientEntityTranslator
{
  //singleton
  public static ClientEntityTranslator entityTranslator = new
                                           SvcOrderToClientOrder();
  private SvcOrderToClientOrder() { }
  public override ClientEntities.ClientEntity ToClientEntity(object
                                                  serviceOutputEntity)
  {
    CallBusinessProxy.OrderInfo oi = serviceOutputEntity as
                                         CallBusinessProxy.OrderInfo;
    ClientEntities.Order bindableOrder = new ClientEntities.Order();
    bindableOrder.OrderNumber = oi.Order.OrderNumber;
    //code removed for brevity  ...
    return bindableOrder;
  }
}

public class SvcUserToClientUser : ClientEntityTranslator
{
    //code removed for brevity  ...
}

public class SvcScriptToClientScript : ClientEntityTranslator
{
    //code removed for brevity  ...
    }
}
If you noticed, the above translators are stateless and employ a singleton pattern. The translator must be able to inherit from ClientEntityTranslator for consistency and needs to be a singleton to avoid garbage-collection churn.
I keep reusing the same instance whenever the respective service call is made. I could also create ServiOutputEntityTranslator for service interaction that requires large input messages (which generally is the case for transactional service invocation) with the following class definition:
public abstract class ServiOutputEntityTranslator
{
  public abstract object ToServiceOutputEntity(ClientEntity
                                                      clientEntity);
}
If you notice the return value of the above function, it is "object", as I don't control the base class of the message entities (in this demo I could, but not in the real world). The type safety will be implemented by the respective translators. For the simplicity of the demo, I don't save any data back to the server, so this demo does not include any translators for converting client entities to message entities.

Silverlight State Change after the Service Calls
Silverlight visual state change can only be performed by the code executing on the UI thread. Since the asynchronous execution of the service calls always returns the results on the callback handler, the handler will be the right place to make changes to the visual or non-visual state of the application.
Non-visual state changes should be exchanged in a thread-safe manner if there might be multiple services trying to modify the shared state asynchronously. It is always recommended that you check Deployment.Current.Dispatcher.CheckAccess() before modifying the UI.

Cross-Domain Policies
Unlike the media applications and the applications that show banner ads, real enterprise-class LOB applications require integration with a variety of service-hosting environments. For example, the call center application referenced throughout the article is typical of the enterprise application. This application hosted on a Web site accesses a stateful socket server for screen-pop, WCF-based Web services for accessing LOB data, and it may download additional XAP packages (Zipped Silverlight deployment packages) from a different domain. It will use yet another domain for transmitting instrumentation data.
The Silverlight sandbox doesn't by default allow network access to any domain other than the domain of origin—advcallclientweb as you saw back in Figure 1. The Silverlight runtime checks for the opt-in policies when the application accesses any domain other than the domain of origin. Here is a typical list of the service-hosting scenarios that need to support cross-domain policy requests by the client:
  • Web services hosted in a service process (or a console application for simplicity)
  • Web services hosted on IIS or other Web servers
  • TCP Services hosted in a service process (or a console app)
I discussed cross-domain policy implementation for TCP services last month and hence will focus on Web services hosted in custom processes and inside IIS.
While it is straightforward to implement cross-domain policies for Web service endpoints hosted in IIS, the other cases require knowledge of the nature of the policy requests and responses.

Cross Domain Policies for Web Services Hosted Outside IIS
For effective state management, there may be cases where one may want to host services in an OS process outside IIS. For cross-domain access of such WCF services, the process will have to host policies at the root of the HTTP endpoint. When a cross-domain Web service is invoked, Silverlight makes an HTTP Get request to clientaccesspolicy.xml. If the service is hosted inside IIS, the clientaccesspolicy.xml file can be copied to the root of the Web site and IIS will do the rest in serving the file. In case of custom hosting on the local machine, http://localhost:<port>/clientaccesspolicy.xml should be a valid URL.
Since the call center demo does not use any custom hosted Web services, I will use a simple TimeService in a console application to demonstrate the concepts. The console will expose a representational state transfer (REST) endpoint using the new REST capabilities of the Microsoft .NET Framework 3.5. UriTemplate property has to be precisely set to the literal shown in Figure 7.
Figure 7 Implementation for Custom-Hosted WCF Services
[ServiceContract]
public interface IPolicyService
{
    [OperationContract]
    [WebInvoke(Method = "GET", UriTemplate = "/clientaccesspolicy.xml")]
    Stream GetClientAccessPolicy();
}
public class PolicyService : IPolicyService
{
    public Stream GetClientAccessPolicy()
    {
        FileStream fs = new FileStream("PolicyFile.xml", FileMode.Open);
        return fs;
    }
}
The interface name or the method name has no bearing on the outcome; you can choose anything you like. WebInvoke has other properties such as RequestFormat and ResponseFormat, which are by default set to XML; we don't need to specify them explicitly. We are also relying on the default value of the BodyStyle property to be BodyStyle.Bare, which means that the response won't be wrapped.
The service implementation is very simple; it merely streams the clientaccesspolicy.xml in response to the Silverlight client request. The policy file name can be of your own choosing, and you can call it anything you like. The implementation of the policy service is shown in Figure 7.
Now we have to configure the IPolicyService for REST-style serving of HTTP requests. The App.Config of the console application (ConsoleWebServices) is shown in Figure 8. There are a few things to note about the special configuration need: the binding of the ConsoleWebServices.IPolicyServer endpoint has to be set to webHttpBinding. Also, the IPolicyService endpoint behavior should be configured with WebHttpBehavior as shown in the configuration file. The base address of the PolicyService should be set to the root URL (as in http://localhost:3045/) and the endpoint address should be left empty (such as <endpoint address=" " … contract="ConsoleWebServices.IPolicyService" />.
Figure 8 WCF Settings for Custom Hosting Environment
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>
    <services>
      <!-- IPolicyService end point should be configured with
           webHttpBinding-->
      <service name="ConsoleWebServices.PolicyService">
         <endpoint address=""
               behaviorConfiguration="ConsoleWebServices.WebHttp"
               binding="webHttpBinding"
               contract="ConsoleWebServices.IPolicyService" />
         <host>
           <baseAddresses>
             <add baseAddress="http://localhost:3045/" />
           </baseAddresses>
         </host>
      </service>
      <service behaviorConfiguration="ConsoleWebServices.TimeServiceBehavior"
               name="ConsoleWebServices.TimeService">
         <endpoint address="TimeService" binding="basicHttpBinding"
               contract="ConsoleWebServices.ITimeService">
         </endpoint>
         <host>
            <baseAddresses>
              <add baseAddress="http://localhost:3045/TimeService.svc" />
            </baseAddresses>
         </host>
       </service>
     </services>
     <behaviors>
        <endpointBehaviors>
          <!--end point behavior is used by REST endpoints like
              IPolicyService described above-->
          <behavior name="ConsoleWebServices.WebHttp">
            <webHttp />
          </behavior>
        </endpointBehaviors>
       ...
      </behaviors>
    </system.serviceModel>
</configuration>
Lastly, the console-hosted services, such as the TimeService shown in the code samples as well as the configuration, should be configured to have a URL that looks similar to their IIS counterparts. For example, the URL of an IIS-hosted TimeService endpoint on default HTTP may look like the following: http://localhost/TimeService.svc. In this case, the metadata can be obtained from http://localhost/TimeService.svc?WSDL.
However, in the case of console hosting, the metadata can be obtained by appending "?WSDL" to the base address of the service host. In the configuration shown in Figure 8, you can see that the base address of the TimeService is set to http://localhost:3045/TimeService.svc, hence the metadata can be obtained from http://localhost:3045/TimeService.svc?WSDL.
This URL is similar to what we use in IIS hosting. If you set the host base address to http://localhost:3045/TimeService.svc/, then the metadata URL will be http://localhost:3045/TimeService.svc/?WSDL, which looks a little bit odd. So watch out for this behavior as it can save you time in figuring out the metadata URL.

Cross-Domain Policies for Services Hosted inside IIS
As discussed previously, deploying cross-domain policies for IIS-hosted services is straightforward: you just copy the clientaccesspolicy.xml file to the root of the site on which the Web services are hosted. As you saw in Figure 1, the Silverlight app is hosted on advcallclientweb (localhost:1041) and accesses business services from AdvBusinessServices (localhost:1043). The Silverlight runtime requires clientaccesspolicy.xml to be deployed at the root of the AdvBusinessServices Web site with the code shown in Figure 9.
Figure 9 Clientaccesspolicy.xml for IIS-Hosted Web Services
<?xml version="1.0" encoding="utf-8"?>
<access-policy>
  <cross-domain-access>
    <policy>
      <allow-from http-request-headers="*">
        <!--allows the access of Silverlight application with localhost:1041
           as the domain of origin-->
        <domain uri="http://localhost:1041"/>
        <!--allows the access of call simulator Silverlight application
           with localhost:1042 as the domain of origin-->
        <domain uri="http://localhost:1042"/>
      </allow-from>
      <grant-to>
        <resource path="/" include-subpaths="true"/>
      </grant-to>
    </policy>
  </cross-domain-access>
</access-policy>
If you recall the cross-domain policy format for the socket server (advpolicyserver) from the first installment of this series, the format of <allow-from> is similar. The difference is in the <grant-to> section where the socket server requires a <socket-resource> setting with port range and protocol attributes, as shown here:
<grant-to>
  <socket-resource port="4530" protocol="tcp" />
</grant-to>
If you create the WCF service-hosting site using the ASP.NET Web site template and add WCF endpoints later, the test Web server will map the virtual directory to the name of the project (such as "/AdvBusinessServices"). This should be changed to "/" in the property pages of the project so that the clientaccesspolicy.xml can be served from the root. If you don't change this, the clientaccesspolicy.xml will not be at the root, and Silverlight applications will get server errors when the service is accessed. Note that this will not be a problem for the Web sites created using the WCF Web service project template.

Figure 10 Login Control Using PasswordBox
<UserControl x:Class="AdvCallCenterClient.Login">
  <Border x:Name="LayoutRoot" ... >
    <Grid x:Name="gridLayoutRoot">
     <Border x:Name="borderLoginViw" ...>
       <TextBlock Text="Pleae login.." Style="{StaticResource headerStyle}"/>
       <TextBlock Text="Rep ID" Style="{StaticResource labelStyle}"/>
       <TextBox x:Name="txRepID" Style="{StaticResource valueStyle}"/>
       <TextBlock Text="Password" Style="{StaticResource labelStyle}"/>
       <PasswordBox x:Name="pbPassword" PasswordChar="*"/>
       <HyperlinkButton x:Name="hlLogin" Content="Click to login"
            ToolTipService.ToolTip="Clik to login" Click="hlLogin_Click" />
     </Border>
     <TextBlock x:Name="tbLoginStatus" Foreground="Red" ... />
      ...
</UserControl>

public partial class Login : UserControl
{
  public Login()
  {
    InitializeComponent();
  }
  public event EventHandler<EventArgs> OnSuccessfulLogin;
  private void hlLogin_Click(object sender, RoutedEventArgs e)
  {
    //validate the login
    AuthenticationProxy.AuthenticationServiceClient authService
                  = new AuthenticationProxy.AuthenticationServiceClient();
    authService.LoginCompleted += new
                EventHandler< AuthenticationProxy.LoginCompletedEventArgs>
                                           (authService_LoginCompleted);
    authService.LoginAsync(this.txRepID.Text, this.pbPassword.Password,
                                                          null, false);
  }

  void authService_LoginCompleted(object sender,
                           AuthenticationProxy.LoginCompletedEventArgs e)
  {
    if (e.Result == true)
    {
       if (OnSuccessfulLogin != null)
          OnSuccessfulLogin(this, null);
    }
    else
    {
      this.tbLoginStatus.Text = "Invalid user id or password";
    }

  }
}
Application Security
One of the critical requirements of an LOB application is authentication; before the call center agent can start the shift, he will authenticate by giving a user ID and password. In ASP.NET Web applications, this can easily be done by taking advantage of the membership provider and the server-side ASP.NET login controls. In Silverlight, there are two ways to enforce authentication: authentication outside and authentication inside.
Authentication outside is very straightforward and is similar to the authentication implementation of ASP.NET applications. With this approach, authentication happens in an ASP.NET-based Web page before the Silverlight application is displayed. The authentication context can be transferred into the Silverlight application through InitParams parameter before a Silverlight application is loaded or through a custom Web service call (to extract the authentication state information) after the application is loaded.
This approach has its place when the Silverlight application is part of a larger ASP.NET/HTML-based system. However, in cases where Silverlight is the main driver of the application, it is natural to perform authentication inside Silverlight. I will use the Silverlight 2 PasswordBox control to capture the password and authenticate using the ASP.NET AuthenticationService WCF endpoint for validating the user's credentials. AuthenticationService, ProfileService, and RoleService are part of the new namespace—System.Web.ApplicationServices—which was new with the .NET Framework 3.5. Figure 10 shows the XAML for the Login control created for this purpose. The Login control calls ASP.NET AuthenticationService.LoginAsync() with the user ID and password that was entered.
Figure 11 Login Custom Silverlight Control
The login screen of the call center, shown in Figure 11, is not sophisticated but serves the purpose of the demo. I implemented a handler for dealing with the LoginCompleted event inside the control so that it is self-contained for displaying invalid login messages and password-resetting dialogues for sophisticated implementations. Upon a successful login, the event OnSuccessfulLogin will be triggered to tell the parent control (Application.RootVisual in this case) to display the first application screen populated with the user information.
The LoginCompleted (ctrlLoginView_OnSuccessfulLogin) handler located inside the main Silverlight Page will invoke the profile service hosted on the business services Web site, as shown in Figure 12. AuthenticationService by default is not mapped to any .svc endpoint; therefore, I will map .svc file to the physical implementation, as shown here:
<!-- AuthenticationService.svc -->
<%@ ServiceHost Language="C#" Service="System.Web.ApplicationServices.
    AuthenticationService" %>
Figure 12 Usage of Login.xaml inside the Page.xaml
<!-- Page.xaml of the main UserControl attached to RootVisual-->
<UserControl x:Class="AdvCallCenterClient.Page" ...>
   <page:Login x:Name="ctrlLoginView" Visibility="Visible"
         OnSuccessfulLogin="ctrlLoginView_OnSuccessfulLogin"/>
   ...
</UserControl>
<!-- Page.xaml.cs of the main UserControl attached to RootVisual-->
public partial class Page : UserControl
{
   ...

   private void ctrlLoginView_OnSuccessfulLogin(object sender, EventArgs e)
   {
     Login login = sender as Login;
     login.Visibility = Visibility.Collapsed;
     CallBusinessProxy.UserProfileClient userProfile
                           = new CallBusinessProxy.UserProfileClient();
     userProfile.GetUserCompleted += new
     EventHandler<GetUserCompletedEventArgs>(userProfile_GetUserCompleted);
     userProfile.GetUserAsync(login.txRepID.Text);
   }
   ...
   void userProfile_GetUserCompleted(object sender,
                                             GetUserCompletedEventArgs e)
   {
     CallBusinessProxy.User user = e.Result;
     UserToBindableUser utobu = new UserToBindableUser(user);
     ClientGlobals.currentUser = utobu.Translate() as ClientEntities.User;
     //all the time the service calls will be complete on a worker thread
     //so the following check is redunant but done to be safe
     if (!this.Dispatcher.CheckAccess())
     {
       this.Dispatcher.BeginInvoke(delegate()
       {
         this.registrationView.DataContext = ClientGlobals.currentUser;
         this.ctrlLoginView.Visibility = Visibility.Collapsed;
         this.registrationView.Visibility = Visibility.Visible;
       });
      }
    }
}
Silverlight can only call Web services that are configured to be called by scripting environments such as AJAX. Like all AJAX callable services, the AuthenticationService service needs access to the ASP.NET runtime environment. I provide this access by setting <serviceHostingEnvironment aspNetCompatibilityEnabled="true"/> directly under the <system.servicemodel> node. In order for the authentication service to be callable by the Silverlight login process (or to be called by AJAX), the web.config must be set according to the directions in "How to: Enable the WCF Authentication Service." The services will be automatically configured for Silverlight if they are created using the Silverlight-enabled WCF Service template located in the Silverlight category.
Figure 13 shows the edited configuration with important elements necessary for the Authentication service. In addition to the service configuration, I also replaced the SQL Server configuration setting for aspnetdb that stores authentication information. Machine.config defines a LocalSqlServer setting that expects aspnetdb.mdf to be embedded into the App_Data directory of the Web site. This configuration setting removes the default setting and points it to the aspnetdb attached to the SQL Server instance. This can easily be changed to point to a database instance running on a separate machine.
Figure 13 Settings for ASP.NET Authentication Service
//web.config
<Configuration>
  <connectionStrings>
  <!-- removal and addition of LocalSqlServer setting will override the
   default asp.net security database used by the ASP.NET Configuration tool
   located in the Visul Studio Project menu-->
  <remove name="LocalSqlServer"/>
    <add name="LocalSqlServer" connectionString="Data
             Source=localhost\SqlExpress;Initial Catalog=aspnetdb; ... />
</connectionStrings>
<system.web.extensions>
   <scripting>
     <webServices>
   <authenticationService enabled="true" requireSSL="false"/>
     </webServices>
   </scripting>
</system.web.extensions>
...
<authentication mode="Forms"/>
...
<system.serviceModel>
   <services>
     <service name="System.Web.ApplicationServices.AuthenticationService"
              behaviorConfiguration="CommonServiceBehavior">
    <endpoint
              contract="System.Web.ApplicationServices.AuthenticationService"
              binding="basicHttpBinding" bindingConfiguration="useHttp"
              bindingNamespace="http://asp.net/ApplicationServices/v200"/>
     </service>
   </services>
   <bindings>
     <basicHttpBinding>
    <binding name="useHttp">
          <!--for production use mode="Transport" -->
      <security mode="None"/>
     </binding>
     </basicHttpBinding>
   </bindings>
   ...
   <serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
</system.serviceModel>
</configuration>
To preserve encapsulation of the Login control and to maintain design-time loose coupling with the parent control, success of the login process is communicated by triggering the OnSuccessfulLogin event. The Application.RootVisual (which is a Page class) will execute the necessary business process to display the first screen upon successful login. The first screen displayed after a successful login is the registrationView, as shown in the userProfile_GetUserCompleted method of Figure 12. Before this view is displayed, I will retrieve user information by calling CallBusinessProxy.UserProfileClient.GetUserAsync(). Please take note of the asynchronous service call, which is similar to the business service integration I'll discuss later.
Be aware that the previous configuration does not use secure sockets layer (SSL); you must modify it to use SSL when building for production systems.

Figure 14 OrderDetails.xaml Control with Order Details Populated
Application Partitioning
One of the factors contributing to the Silverlight application startup time is the size of the initial package. The guidelines for the size of the XAP package are no different than the page weight for Web applications. Bandwidth is a limited resource. The stringent response times of Web applications require that you pay close attention to the Silverlight application startup time.
In addition to the processing time spent before the first UserControl is displayed, the size of the application package has direct influence on this important quality of the application. In order to improve startup speed, you must avoid monolithic XAP files that can grow to tens of megabytes in size for complex applications.
The Silverlight application can be broken down into a collection of XAP files; individual DLLs; or individual XML files, images, and any other types with recognized MIME types. In the call center scenario, to demonstrate granular application partitioning, I will deploy the OrderDetail Silverlight control as a separate DLL (AdvOrderClientControls.dll) along with AdvCallCenterClient.xap into the ClientBin directory of the AdvCallClientWeb project (refer back to Figure 1).
The DLL will be downloaded preemptively on the worker thread when the agent accepts the incoming call. The call you saw in Figure 4—ThreadPool.QueueUserWorkItem(ExecuteControlDownload)—is responsible for this. Once the caller answers security questions, I will use reflection to create an instance of the OrderDetail control and add it to the control tree before displaying it on the screen. Figure 14 shows OrderDetail.xaml control loaded into the control tree with order details populated.
The DLL containing the OrderDetail control is deployed to the same Web site as the call center client, which is typical of the DLLs belonging to the same application, so I won't run into any cross-domain issues in this case. However, this may not be the case with services, because Silverlight applications may access services deployed on multiple domains, including local ones and in the cloud, as shown in the architecture diagram (again, refer back to Figure 1).
The ExecuteControlDownload method (see Figure 4) runs on a background worker thread and uses the WebClient class for downloading the DLL. WebClient, by default, assumes that the download happens from the domain of origin and hence only uses relative URIs.
The OrderDetailControlDownloadCallback handler receives the DLL stream and creates the assembly using ResourceUtility.GetAssembly() shown in Figure 15. Because creation of the assembly must happen on the UI thread, I will dispatch the GetAssembly() and the (thread-safe) assignment of the assembly to the global variable to the UI thread:
void OrderDetailControlDownloadCallback(object sender,
       OpenReadCompletedEventArgs e)
  {
    this.Dispatcher.BeginInvoke(delegate() {
    Assembly asm = ResourceUtility.GetAssembly(e.Result);
    Interlocked.Exchange<Assembly>(ref
        ClientGlobals.advOrderControls_dll, asm ); });
  }
Figure 15 Utility Functions to Extract Resources
public class ResourceUtility
{
  //helper function to retrieve assembly from a package stream
  public static Assembly GetAssembly(string assemblyName, Stream
                                                        packageStream)
  {
    StreamResourceInfo srInfo =
    Application.GetResourceStream(
              new StreamResourceInfo(packageStream, "application/binary"),
              new Uri(assemblyName, UriKind.Relative));
    return GetAssembly(srInfo.Stream);
  }
  //helper function to retrieve assembly from a assembly stream
  public static Assembly GetAssembly(Stream assemblyStream)
  {
    AssemblyPart assemblyPart = new AssemblyPart();
    return assemblyPart.Load(assemblyStream);
  }
  //helper function to create an XML document from the stream
  public static XElement GetXmlDocument(Stream xmlStream)
  {
    XmlReader reader = XmlReader.Create(xmlStream);
    XElement element = XElement.Load(reader);
    return element;
  }
  //helper function to create an XML document from the default package
  public static XElement GetXmlDocumentFromXap(string fileName)
  {
    XmlReaderSettings settings = new XmlReaderSettings();
    settings.XmlResolver = new XmlXapResolver();
    XmlReader reader = XmlReader.Create(fileName);
    XElement element = XElement.Load(reader);
    return element;
  }
  //gets the UIElement from the default package
  public static UIElement GetUIElementFromXaml(string xamlFileName)
  {
    StreamResourceInfo streamInfo = Application.GetResourceStream(new
                                  Uri(xamlFileName, UriKind.Relative));
    string xaml = new StreamReader(streamInfo.Stream).ReadToEnd();
    UIElement uiElement = null;
    try
    {
      uiElement = (UIElement)XamlReader.Load(xaml);
    }
    catch
    {
      throw new SLApplicationException(string.Format("Can't create
                                  UIElement from {0}", xamlFileName));
    }
    return uiElement;
  }
}
Since the dispatched delegate runs on a different thread than the callback handler, you have to be conscious of the state of the objects that are accessed from the anonymous delegate. In the previous code, the state of the downloaded DLL stream is really important. You may not write code that reclaims the resources of the stream inside the OrderDetailControlDownloadCallback function. Such code will prematurely dispose of the downloaded stream before the UI thread has a chance to create the assembly. I will use reflection to create an instance of the OrderDetail user control and add it to the Panel as shown here:
_orderDetailContol = ClientGlobals.advOrderControls_dll.CreateInstance
                  ("AdvOrderClientControls.OrderDetail") as UserControl;
spCallProgressPanel.Children.Add(_orderDetailContol);
ResourceUtility in Figure 15 also shows various utility functions to extract UIElement from the XAML and the XML document from the downloaded streams and default packages.

Productivity and Beyond
I have looked at Silverlight from a traditional enterprise application perspective, touching several architectural aspects of the application. Implementation of push notifications with Silverlight sockets is an enabler of LOB scenarios such as call centers. With the upcoming release of Internet Explorer 8.0—which is slated to include six concurrent HTTP connections per host—push notification implementation over the Internet will be more compelling when using duplex WCF binding. Integration with LOB data and processes is as easy as it is in traditional desktop applications.
This will be a huge productivity boost when compared with AJAX and other rich Internet application (RIA) platforms. Silverlight applications can be secured using the WCF authentication and authorization endpoints provided by ASP.NET in its latest release. I hope this little exploration of LOB application development with Silverlight will motivate you to take Silverlight beyond media and advertising scenarios.

Hanu Kommalapati is a Platform Strategy Advisor at Microsoft, and in this role he advises enterprise customers in building scalable line-of-business applications on Silverlight and Azure Services platforms.


MSBuild
Best Practices For Creating Reliable Builds, Part 1
Sayed Ibrahim Hashimi

This article discusses:
  • Defining target dependencies
  • Batching tasks
  • Extending the clean process
  • Organizing targets
This article uses the following technologies:
MSBuild, Visual Studio
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Defining Target Dependencies
Batching Tasks
Defining Dynamic Items and Properties
Extending the Clean Process
Organizing Targets
Using Wildcards
In this article, the first of two parts, I describe several best practices that developers should follow when using MSBuild, the Microsoft build engine used by Visual Studio to build managed projects. In this first part, I'll cover some basic practices and techniques that you can apply to most every project. I'll also discuss topics such as defining task dependencies, batching tasks, organizing targets, and more. In Part 2, I'll discuss some practices related to build configurations that require heaver customization, largely because of their size.
If you are not familiar with MSBuild, take a look at my previous articles published in MSDN Magazine: "Inside MSBuild: Compile Apps Your Way with Custom Tasks for the Microsoft Build Engine" and "WiX Tricks: Automate Releases with MSBuild and Windows Installer XML."

Defining Target Dependencies
When you define a target in MSBuild, you can use the DependsOnTargets attribute to define target dependencies. MSBuild uses this attribute to determine the order in which targets are executed. For example, if you define a target named Deploy that depends on the targets PrepareForBuild and Build, you can express that dependency as follows:
<Target Name="PrepareForBuild">
  <!-- target contents here -->
</Target>

<Target Name="Build">
  <!-- target contents here -->

</Target>

<Target Name="Deploy" DependsOnTargets="PrepareForBuild;Build">
  <!-- target contents here -->
</Target>
Although the previous listing defines the dependencies you want, it does not provide a way to extend the behavior of the Deploy target. It would be better if you could specify additional targets that the Deploy target depends on without having to modify the Deploy target itself. You can implement this behavior by defining properties inside the DependsOnTargets attribute instead of using hardcoded values.
Take a look at the modified Deploy target shown here:
<PropertyGroup>
  <DeployDependsOn>
    $(DeployDependsOn);
    PrepareForBuild;
    Build
  </DeployDependsOn>
</PropertyGroup>
<Target Name="Deploy" DependsOnTargets="$(DeployDependsOn)">
  <!-- target contents here -->
</Target>
In this version, the Deploy target depends on the targets contained in the DeployDependsOn property. (By convention, these types of properties are named with the target name followed by DependsOn, but this is not required.) Consumers can now attach additional targets to the DeployDependsOn property by overriding it. However, notice that the declaration of the DeployDependsOn property contains a reference to itself: $(DeployDependsOn). Including this reference ensures that any existing values for DeployDependsOn are not overridden. Instead, they are simply appended.
This approach is better, but it only supports adding targets that are executed before the target in question. If you want consumers to be able to specify targets that should be executed either before or after a target, you need to declare the Deploy target, as shown in Figure 1.
Figure 1 Deploy Target Declaration
<PropertyGroup>
  <DeployDependsOn>
    $(DeployDependsOn);
    PrepareForBuild;
    Build;
    BeforeDeploy;
    CoreDeploy;
    AfterDeploy
  </DeployDependsOn>
</PropertyGroup>
<Target Name="Deploy" DependsOnTargets="$(DeployDependsOn)"/>
<Target Name="BeforeDeploy"/>
<Target Name="AfterDeploy"/>
<Target Name="CoreDeploy">
  <!-- target contents here -->
</Target>
In this case, the Deploy target is an empty target. The only thing it does is specify the targets that it depends on. The actual work is performed inside the CoreDeploy target. By taking this step, you can extend the DeployDependsOn property to execute targets both before and after CoreDeploy executes.
Additionally, there are two other empty targets here: BeforeDeploy and AfterDeploy. They are declared to allow consumers a quick way to extend the Deploy target. If either of these targets is defined by a consumer, it will be called at the appropriate time. The difference between overriding a target like BeforeDeploy and extending the DeployDependsOn property is that only one definition of the BeforeDeploy target will exist. This means that if the target is declared more than once, the last one processed will be the definition that takes effect.
As an alternative to this, you can append as many target names to a dependency property as needed. For reusable build files the latter approach is the only method that ensures that a target name collision does not occur. As an example of extending the DeployDependsOn property, you could declare the DeployDependsOn target again, as shown here:
<PropertyGroup>
  CustomBeforeDeploy;
  $(DeployDependsOn);
  CustomAfterDeploy
</PropertyGroup>
<Target Name="CustomBeforeDeploy">
  <!-- your steps here -->
</Target>
<Target Name="CustomAfterDeploy">
  <!-- your steps here -->
</Target>
In this example, the CustomBeforeDeploy and CustomAfterDeploy targets have been added to the list of targets to be executed. The targets shipped for Visual Basic .NET and C# projects declare empty targets like BeforeBuild as well as many dependency properties like RebuildDependsOn. You can find more information about both these topics at the MSBuild team blog.
Every target that can be called externally should have a complete list of targets in its DependsOnTargets attribute. This step is important because if a user wants to perform a specific task, let's say UnitTests, they should be able to specify that in the targets switch for msbuild.exe. By properly declaring dependent targets, your project file will support this.

Batching Tasks
One of the common questions about MSBuild is how to execute a loop. Because MSBuild uses a declarative language, you cannot specify an explicit loop. Instead, you must describe what you want the MSBuild engine to perform and let it handle the looping for you.
MSBuild uses a concept known as task batching, which allows a task to be executed once per each unique batch or group of values. Batches are always created based on item metadata. Take a look at a simple example using the contents of the Batching01.proj file, shown in Figure 2.
Figure 2 A Simple Batching Example
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         ToolsVersion="3.5">

  <ItemGroup>
    <SampleItem Include="one">
      <Group>A1</Group>
      <Id>A363BE85-2CB1-4221-A9CB-2881B7699329</Id>
    </SampleItem>
    <SampleItem Include="two">
      <Group>A2</Group>
      <Id>48E171C8-2274-4567-84D5-D20C6B0CB363</Id>
    </SampleItem>
    <SampleItem Include="three">
      <Group>A2</Group>
      <Id>618E5BD8-650F-43c9-855E-259126284004</Id>
    </SampleItem>
    <SampleItem Include="four">
      <Group>A1</Group>
      <Id>65E8E8E7-5A3F-4e02-A1D9-34F797CB68D9</Id>
    </SampleItem>
    <SampleItem Include="five">
      <Group>A1</Group>
      <Id>43D0D1FE-304F-4aff-BE19-67AD2195872B</Id>
    </SampleItem>
  </ItemGroup>

  <Target Name="TaskBatching01">
    <!-- No batching here -->
    <Message Text="SampleItem: @(SampleItem)"/>
    <Message Text=" "/>
    <!-- Batches created based on Group Metadata -->
    <Message Text="SampleItem.Group: %(SampleItem.Group)"/>
    <Message Text=" "/>
    <!-- Batches created based on Id Metadata -->
    <Message Text="SampleItem.Id: %(SampleItem.Id)"/>
  </Target>

</Project>
This project file defines an item type named SampleItem, which includes five values. Each value has metadata values for Group and Id. The metadata for Id is unique for each item value, but Group has only two values, A1 and A2. Batching is triggered when an expression of the form %(ItemType.MetadataName) is encountered.
In the TaskBatching01 target, the Message task is invoked with <Message Text="SampleItem.Group: %(SampleItem.Group)"/>. This expression causes MSBuild to determine the unique batches defined by the Group metadata from the SampleItem item. Because I know that Group has two unique values, I know that two batches are created, one from values that have A1 as the Group metadata and the other from those with A2. Another Message task, <Message Text="SampleItem.Id: %(SampleItem.Id)"/>, creates batches based on the Id metadata value. The result of executing this target is shown in Figure 3.
Figure 3 Batching01.proj Result
C:\Samples\Batching>msbuild Batching01.proj /t:TaskBatching01 /nologo
Build started 10/20/2008 1:27:58 AM.
Project "C:\Samples\Batching\Batching01.proj" on node 0 (TaskBatching01
  target(s)).
  SampleItem: one;two;three;four;five

  SampleItem.Group: A1
  SampleItem.Group: A2
  SampleItem.Id: A363BE85-2CB1-4221-A9CB-2881B7699329
  SampleItem.Id: 48E171C8-2274-4567-84D5-D20C6B0CB363
  SampleItem.Id: 618E5BD8-650F-43c9-855E-259126284004
  SampleItem.Id: 65E8E8E7-5A3F-4e02-A1D9-34F797CB68D9
  SampleItem.Id: 43D0D1FE-304F-4aff-BE19-67AD2195872B
Done Building Project "C:\Samples\Batching\Batching01.proj"
  (TaskBatching01 target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)
As you can see, the statement <Message Text="SampleItem: @(SampleItem)"/> flattens the SampleItem values into a single string passed into the Message task in place of the @(SampleItem) expression, resulting in "one;two;three;four;five." Contrast this with the behavior demonstrated with the <Message Text="SampleItem.Group: %(SampleItem.Group)"/> invocation. In this case, the Message task was executed twice. In the case of the SampleItem.Id instance, the Message task was executed five times.
One common use of batching is to recursively copy files from one location to another. In these cases you have to use the RecursiveDir well-known metadata to determine where the folder resides relative to the parent directory. (The set of MSBuild well-known metadata can be found in the "MSBuild Well-known Item Metadata" article on MSDN.) The project file Copy01.proj, shown in Figure 4, copies files from the src folder to the dest folder.
Figure 4 Copy01.proj
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         ToolsVersion="3.5">

  <ItemGroup>
    <!-- Get all files under src\ except svn files -->
    <SrcFiles Include="src\**\*" Exclude="**\.svn\**\*" />
  </ItemGroup>

  <PropertyGroup>
    <Dest>dest\</Dest>
  </PropertyGroup>

  <Target Name="CopyToDest">
    <Copy SourceFiles="@(SrcFiles)"
          DestinationFolder ="$(Dest)%(SrcFiles.RecursiveDir)" />
  </Target>

  <Target Name="Clean">
    <ItemGroup>
      <_FilesToDelete Include="$(Dest)**\*"/>
    </ItemGroup>

    <Delete Files="@(_FilesToDelete)"/>
  </Target>
</Project>
Inside the CopyToDest target, the Copy task is used along with batching to copy the files. Here are the contents of the src folder:
src
¦   class1.cs
¦   class2.cs
¦   class3.cs
¦   class4.cs
¦
+---Admin
¦       admin_class1.cs
¦       admin_class2.cs
¦
+---Utilities
        util_class1.cs
        util_class2.cs
In this example, the DestinationFolder is specified as $(Dest)%(SrcFiles.RecursiveDir). This statement breaks the SrcFiles item into three groups based on the value for RecursiveDir—an empty value for RecursiveDir, Admin, and Utilities. The Copy task is called three times, once for each group. During each of these calls only the files in the current group are passed to the Copy task as the SrcFiles item.
To see this clearly, take a look at the detailed log of the build shown in Figure 5. From the log, you can verify that the Copy task was executed three times, and each time only the files matching the current value for RecursiveDir were included in the SrcFiles item.
Figure 5 Recursively Copying Files
Build started 10/26/2008 12:09:28 AM.
Project "C:\Samples\Batching\Copy01.proj" on node 0 (default targets).
Building with tools version "3.5".
Target "CopyToDest" in file "C:\Samples\Batching\Copy01.proj" from project "C:\Samples\Batching\Copy01.proj":
Using "Copy" task from assembly "Microsoft.Build.Tasks.v3.5, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".
Task "Copy"
  Copying file from "src\Admin\admin_class1.cs" to "dest\Admin\admin_
  class1.cs".
  Command:
  copy /y "src\Admin\admin_class1.cs" "dest\Admin\admin_class1.cs"
  Copying file from "src\Admin\admin_class2.cs" to "dest\Admin\admin_
  class2.cs".
  Command:
  copy /y "src\Admin\admin_class2.cs" "dest\Admin\admin_class2.cs"
Done executing task "Copy".
Task "Copy"
  Copying file from "src\class1.cs" to "dest\class1.cs".
  Command:
  copy /y "src\class1.cs" "dest\class1.cs"
  Copying file from "src\class2.cs" to "dest\class2.cs".
  Command:
  copy /y "src\class2.cs" "dest\class2.cs"
  Copying file from "src\class3.cs" to "dest\class3.cs".
  Command:
  copy /y "src\class3.cs" "dest\class3.cs"
  Copying file from "src\class4.cs" to "dest\class4.cs".
  Command:
  copy /y "src\class4.cs" "dest\class4.cs"
Done executing task "Copy".
Task "Copy"
  Copying file from "src\Utilities\util_class1.cs" to "dest\Utilities\
  util_class1.cs".
  Command:
  copy /y "src\Utilities\util_class1.cs" "dest\Utilities\util_class1.cs"
  Copying file from "src\Utilities\util_class2.cs" to "dest\Utilities\
  util_class2.cs".
  Command:
  copy /y "src\Utilities\util_class2.cs" "dest\Utilities\util_class2.cs"
Done executing task "Copy".
Done building target "CopyToDest" in project "Copy01.proj".
Done Building Project "C:\Samples\Batching\Copy01.proj" (default
  targets).

Build succeeded.
    0 Warning(s)
    0 Error(s)
This previous example is a common use of batching. Another is creating a .zip file. If you need your build process to create a .zip file, there are some tasks available that can assist you. One such task is the Zip task from the MSBuild Community Tasks project. This task is used in the Zip01.proj file shown in Figure 6.
Figure 6 Zip01.proj
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         ToolsVersion="3.5">

  <PropertyGroup>
    <SrcFolder>src\</SrcFolder>
  </PropertyGroup>

  <ItemGroup>
    <SourceFiles Include="$(SrcFolder)*"
                 Exclude="$(SrcFolder).svn\**\*">
    </SourceFiles>
    <SourceFiles Include="$(SrcFolder)class1.cs;$(SrcFolder)class3.cs">
      <Access>public</Access>
    </SourceFiles>
    <SourceFiles Include="$(SrcFolder)class2.cs;$(SrcFolder)class4.cs">
      <Access>private</Access>
    </SourceFiles>
    <SourceFiles Include="$(SrcFolder)Utilities\*"
                 Exclude="$(SrcFolder)Utilities\.svn\**\*">
      <Group>util</Group>
    </SourceFiles>
    <SourceFiles Include="$(SrcFolder)Admin\*"
                 Exclude="$(SrcFolder)Admin\.svn\**\*">
      <Group>admin</Group>
      <Access>private</Access>
    </SourceFiles>
  </ItemGroup>

  <!--
  Define default values here.
  -->
  <ItemDefinitionGroup>
    <SourceFiles>
      <!-- Default Group value -->
      <Group>common</Group>
      <!-- Default Access value -->
      <Access>public</Access>
    </SourceFiles>
  </ItemDefinitionGroup>

  <!-- Import SDC tasks from known location in source control -->
  <Import
    Project="..\Contrib\CommunityTasks\MSBuild.Community.Tasks.Targets"/>

  <Target Name="ZipFiles">
    <!-- Create an item with output in order to get the full path -->
    <PropertyGroup>
      <_ZipWorkingDir>output\</_ZipWorkingDir>
    </PropertyGroup>
    <MakeDir Directories="$(_ZipWorkingDir)"/>
    <!--
    Zip the files based on what zip file they should be placed in.
    Also this task should be passed a full path for the WorkingDirectory.
    -->
    <Zip Files="@(SourceFiles)"
         ZipFileName=
         "$(_ZipWorkingDir)%(SourceFiles.Group).%(SourceFiles.Access).
         zip"/>
  </Target>
</Project>
This project file declares an item list, SourceFiles, that contains the contents of the src folder. This item list also contains some additional metadata, Group and Access, which is used to determine which .zip file the files in the src folder should be placed in. The project uses the ItemDefinitionGroup element to define default values for Group and Access. All values in the SourceFiles item list that do not have a Group value explicitly defined use the value "common" and those that do not have a value for Access use the value "public".
The Zip task is used inside the ZipFile target, and the value $(_ZipWorkingDir)%(SourceFiles.Group).%(SourceFiles.Access).zip is provided for the ZipFileName property. In cases that include multiple batching expressions from the same item list, batches are created from unique combinations of the metadata used. For this statement, batches will be created from unique values for Group and Access. Because there are three values for Group and two for Access, there will be at most six batches. In this case there are only four batches consisting of the combinations shown in Figure 7.
The other possible combinations (admin/public and util/private) never appear in the item so they will not be used. We now know that four .zip files should be created. If you execute the command msbuild Zip01.proj /t:ZipFiles, you'll see results shown in Figure 8.
Figure 8 Creating Zip Files Based on Metadata
C:\Samples\Batching>msbuild Zip01.proj /nologo
Build started 11/3/2008 11:40:40 PM.
Project "C:\Samples\Batching\Zip01.proj" on node 0 (default targets).
  Creating directory "output\".
  Creating zip file "output\common.public.zip".
    added "src/class1.cs".
    added "src/class3.cs".
  Created zip file "output\common.public.zip" successfully.
  Creating zip file "output\common.private.zip".
    added "src/class2.cs".
    added "src/class4.cs".
  Created zip file "output\common.private.zip" successfully.
  Creating zip file "output\util.public.zip".
    added "src/Utilities/util_class1.cs".
    added "src/Utilities/util_class2.cs".
  Created zip file "output\util.public.zip" successfully.
  Creating zip file "output\admin.private.zip".
    added "src/Admin/admin_class1.cs".
    added "src/Admin/admin_class2.cs".
  Created zip file "output\admin.private.zip" successfully.
Done Building Project "C:\Samples\Batching\Zip01.proj" (default targets).

Build succeeded.
    0 Warning(s)
    0 Error(s)
In this figure, you can see that the Zip task was called four times as expected. The name of each zip file contains both metadata values used for the batch.
The behavior described here is not limited to two metadata values or a single item list. You are free to use as many metadata values as necessary. The behavior for batching on more than one item list is different from what I've described and will not be discussed here.

Defining Dynamic Items and Properties
I've previously discussed items and properties, but I haven't really defined them. In MSBuild, items are ordered lists of objects with metadata, all of which share a particular item type, such as Resource or Compile. Properties are key-value pairs. Dynamic items and properties are those that you create inside a target., Static items and properties are declared outside a target.
In MSBuild 2.0 you had to use the tasks CreateItem and CreateProperty to create dynamic items and properties, respectively. In MSBuild 3.5, you can create them by using the same syntax as you use for static items. The preferred method is to use the ItemGroup and PropertyGroup elements inside a target.
Static values are always evaluated before any target executes. This means that every ItemGroup and PropertyGroup found outside a target must be expanded before a single build step can be executed. Property evaluation is quick, but item evaluation can be time-consuming, depending on the number of values an item contains. For this reason, if an item or property is only used with one target, it should be declared inside that target and not as a static value.
Property and item evaluation occurs in several passes. For static properties and items, properties are evaluated top to bottom (from the start of a file to the end of a file), including entering into any imported projects, and items are evaluated from top to bottom, also including entering into any imported projects. Dynamic properties and items are evaluated as they are executed.
When you create items and properties, the convention is to start the name for "internal" values with an underscore. Items and properties whose names do not start with an underscore indicate to users that they are free to override them. Given this convention, when you create those properties or items, you should first check to see whether they have already been defined with a condition. This is useful in case someone else has already defined the same property or item that should override yours.
To demonstrate this, take a look at the following declarations taken from Microsoft.Common.targets:
<PropertyGroup>
  <AssemblyName
    Condition=" '$(AssemblyName)'=='' ">$(MSBuildProjectName)</AssemblyName>
</PropertyGroup>
From this code, we can see that the property AssemblyName will be set to $(MSBuildProjectName) only if it has not already been previously declared.
MSBuild 3.5 includes new features related to dynamic items. For example, in MSBuild 2.0 you cannot modify the contents of an item list. After you include an item in an item list you cannot remove it. The workaround to this limitation is to create a new item excluding the value you are not interested in. MSBuild 3.5 now supports removing values from items through the Remove attribute. The project file RemoveItems01.proj, shown in Figure 9, demonstrates this.
Figure 9 RemoveItems01.proj
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         ToolsVersion="3.5">

  <ItemGroup>
    <SrcFiles Include="Batching\src\class1.cs"/>
    <SrcFiles Include="Batching\src\class2.cs"/>
    <SrcFiles Include="Batching\src\class3.cs"/>
  </ItemGroup>

  <Target Name="Remove">
    <Message Text="SrcFiles: @(SrcFiles)"/>
    <ItemGroup>
      <!-- Remove using value provided inside -->
      <SrcFiles Remove="Batching\src\class2.cs"/>
      <!--
      Remove all items in SrcFiles that match the condition.
      In this case, a single file.
      -->
      <SrcFiles Remove="@(SrcFiles)"
                Condition="'%(Filename)%(Extension)'=='class3.cs'" />
    </ItemGroup>
    <Message Text="SrcFiles: @(SrcFiles)"/>
  </Target>
</Project>
There are two ways to use the Remove attribute on the ItemGroup. You can remove a value by passing the value that was included inside the Include attribute when the item was created. This approach is shown in the expression <SrcFiles Remove="Batching\src\class2.cs"/>. The other method shown in Figure 9 is to pass the entire item into the Remove attribute while placing a condition on the element.
Figure 10 shows the result of executing the Remove target. Based on the output shown in Figure 10, you can see that the class2.cs and class3.cs files were successfully removed from the SrcFiles item. You can also add and update metadata using the ItemGroup element inside of a target, using the same syntax you do for creating a dynamic item. To see how to update metadata values, take a look at the UpdateMetadata01.proj file shown in Figure 11.
Figure 10 Removing Values from Items
C:\Samples>msbuild RemoveItems01.proj /t:Remove /nologo
Build started 10/26/2008 12:54:11 AM.
Project "C:\Samples\RemoveItems01.proj" on node 0 (Remove target(s)).
  SrcFiles: Batching\src\class1.cs;Batching\src\class2.cs;Batching\src\class3.cs
  SrcFiles: Batching\src\class1.cs
Done Building Project "C:\Samples\RemoveItems01.proj" (Remove target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)
Figure 11 UpdateMetadata01.proj
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         ToolsVersion="3.5">
  <ItemGroup>
    <Reference Include="IronPython, Version= ...">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\shared\IronPython-1.1\IronPython.dll</HintPath>
    </Reference>
    <Reference Include="log4net, Version= ...">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\binaries\log4net.dll</HintPath>
    </Reference>
    <Reference Include="nunit.core, ...">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\shared\nunit2.2\nunit.core.dll</HintPath>
    </Reference>
    <Reference Include="nunit.framework, Version= ...">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\shared\nunit2.2\nunit.framework.dll</HintPath>
    </Reference>
    <Reference Include="nunit.util, ...">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>..\shared\nunit2.2\nunit.util.dll</HintPath>
    </Reference>
  </ItemGroup>

  <Target Name="UpdateSpecificVersion">
    <Message Text="%(Reference.Identity) : %(Reference.
    SpecificVersion)"/>

    <Message Text=" "/><!-- blank line -->
    <Message Text="Update Reference.SpecificVersion to True"/>
    <Message Text=" "/><!-- blank line -->

    <!-- Update all Reference items to be SpecificVersion=True -->
    <ItemGroup>
      <Reference>
        <SpecificVersion>True</SpecificVersion>
      </Reference>
    </ItemGroup>
    <Message Text="%(Reference.Identity) : %(Reference.SpecificVersion)"/>
  </Target>
</Project>
This project file declares an item list named Reference. (In these examples, strong names are truncated to preserve space.) Also, the project contains a target named UpdateSpecificVersion. This target uses the ItemGroup element to update the SpecificVersion metadata for the Reference item list. The values for this metadata are printed before and after the update. Figure 12 shows the results.
Figure 12 Updating Existing Metadata
C:\Samples>msbuild UpdateMetadata01.proj /t:UpdateSpecificVersion /nologo
Build started 10/29/2008 12:07:39 AM.
Project "C:\Samples\UpdateMetadata01.proj" on node 0
  (UpdateSpecificVersion target(s)).
  IronPython, Version= ... : False
  log4net, Version= ... : False
  nunit.core, ... : False
  nunit.framework, Version= ... : False
  nunit.util, ... : False

  Update Reference.SpecificVersion to True

  IronPython, Version= ... : True
  log4net, Version= ... : True
  nunit.core, ... : True
  nunit.framework, Version= ... : True
  nunit.util, ... : True
Done Building Project "C:\Samples\UpdateMetadata01.proj"
  (UpdateSpecificVersion target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)
You can see an example of adding metadata in the AddMetadata01.proj file included with the article's sample code. I won't go into the details here. Along with modifying item list themselves, updating metadata and adding metadata to items while a target is executing is new functionality that has no equivalent in MSBuild 2.0.

Extending the Clean Process
This section relates to using MSBuild when you are building C# or Visual Basic .NET projects in Visual Studio. The behavior described here is contained within the target files used to build these types of projects and not within MSBuild itself.
When you modify the build process to create new files, you are responsible to extend the clean process to clean up those files. MSBuild cleans up files that it is responsible for automatically, but it cannot do so for files generated by custom targets. MSBuild maintains an item list named FileWrites that contains the files that need to be cleaned. This list is persisted to a file inside the obj folder that is referred to as the "clean cache." You can place additional values into the FileWrites item list so that they are removed when the project is cleaned up.
There are two drawbacks to doing this, however. Files to be deleted must reside under the output path, and you must append items to that list before the clean cache is written to disk, which occurs in either the Clean or IncrementalClean target, depending on the type of build that is executed. These targets will be called before the Build target completes.
In the sample code accompanying this article, you will find a Windows Forms project, WindowsFormsApplication1, in which I've defined the BeforeBuild target as follows:
<Target Name="BeforeBuild">
  <ItemGroup>
    <CustomConfigFile Include="$(OutputPath)settings.config" />
  </ItemGroup>
  <!-- Generate config file here -->
  <WriteLinesToFile File="@(CustomConfigFile)"
                    Lines="config entries here"
                    Overwrite="true" />
  <!-- Append to FileWrites so the file will be removed on clean -->
  <ItemGroup>
    <FileWrites Include="@(CustomConfigFile)"/>
  </ItemGroup>
</Target>
In this target, a file named, settings.config is created by using the WriteLinesToFile task. In this case the file contains only the text "config entries here," but it could contain values for the appSettings node as well as other content. Also, the FileWrites list is appended to, so the settings.config file will be removed when a cleanup is executed.
In Figure 13 you can see the result of executing the Build target and then the Clean target (some lines that are not relevant are excluded). You can also see that settings.config was successfully deleted when the Clean target was executed. If the customizations were placed in the AfterBuild target instead of the BeforeBuild target, the file would not have been removed on Clean. To accomplish this, you have to extend the clean process.
Figure 13 Clean Target Results
C:\Samples\WindowsFormsApplication1>msbuild WindowsFormsApplication1.csproj /fl /t:Build;Clean /nologo
...
CoreClean:
  Deleting file "C:\Samples\WindowsFormsApplication1\bin\Debug\settings.config".
  Deleting file "C:\Samples\WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.exe.config".
  Deleting file "C:\Samples\WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.exe".
  Deleting file "C:\Samples\WindowsFormsApplication1\bin\Debug\WindowsFormsApplication1.pdb".
  Deleting file "C:\Samples\WindowsFormsApplication1\obj\Debug\WindowsFormsApplication1.Form1.resources".
  Deleting file "C:\Samples\WindowsFormsApplication1\obj\Debug\WindowsFormsApplication1.Properties.Resources.resources".
  Deleting file "C:\Samples\WindowsFormsApplication1\obj\Debug\WindowsFormsApplication1.csproj.GenerateResource.Cache".
  Deleting file "C:\Samples\WindowsFormsApplication1\obj\Debug\WindowsFormsApplication1.exe".
  Deleting file "C:\Samples\WindowsFormsApplication1\obj\Debug\WindowsFormsApplication1.pdb".
Done Building Project "C:\Samples\WindowsFormsApplication1\WindowsFormsApplication1.csproj" (Build;Clean target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)
There are two targets that can be overridden in your project file that are executed when the Clean target is called. Those targets are BeforeClean and AfterClean. If you define either of these targets in the project file, the target will be called at the appropriate time. These targets must be defined after the Import statement for either Microsoft.CSharp.targets or Microsoft.VisualBasic.targets.
Another approach is to use target dependencies. You can extend the CleanDependsOn property to inject your own target into the cleanup process. Take a look at the customizations to the WindowsFormsApplication2 project shown in Figure 14, which occur after the Import statement for Microsoft.CSharp.targets.
Figure 14 WindowFormsApplication2
<Target Name="AfterClean">
  <Message Text="AfterClean target executed"/>
</Target>
<!-- Inject a custom target into Clean by extending CleanDependsOn -->
<PropertyGroup>
  <CleanDependsOn>
    CustomBeforeClean;
    $(CleanDependsOn);
    CustomAfterClean
  </CleanDependsOn>
</PropertyGroup>
<Target Name="CustomBeforeClean">
  <Message Text="CustomBeforeClean target executed"/>
</Target>
<Target Name="CustomAfterClean">
  <Message Text="CustomAfterClean target executed"/>
</Target>
In this listing, the AfterClean target is overridden and two targets are placed in the CleanDependsOn property. These targets only display a message to demonstrate that the target was executed, but in an actual build process they could clean up resources. Figure 15 shows the results of calling the Clean target (some lines were shortened to preserve space). As expected, the targets were called at the appropriate times.
Figure 15 Clean Target Results
C:\Samples\WindowsFormsApplication2>msbuild WindowsFormsApplication2.csproj /t:Clean /nologo
Build started 10/26/2008 1:17:40 PM.
Project "C:\Samples\WindowsFormsApplication2\WindowsFormsApplication2.csproj" on node 0 (Clean target(s)).
  CustomBeforeClean target executed
CoreClean:
  Deleting file "\bin\Debug\WindowsFormsApplication2.exe".
  Deleting file "\bin\Debug\WindowsFormsApplication2.pdb".
  Deleting file "\obj\Debug\WindowsFormsApplication2.Form1.resources".
  Deleting file "\obj\Debug\WindowsFormsApplication2.Properties.Resources.resources".
  Deleting file "\obj\Debug\WindowsFormsApplication2.csproj.GenerateResource.Cache".
  Deleting file "\obj\Debug\WindowsFormsApplication2.exe".
  Deleting file " \obj\Debug\WindowsFormsApplication2.pdb".
AfterClean:
  AfterClean target executed
CustomAfterClean:
  CustomAfterClean target executed
Done Building Project "C:\Samples\WindowsFormsApplication2\WindowsFormsApplication2.csproj" (Clean target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Organizing Targets
As a build process grows, you need to organize where the different pieces reside. Generally, it is better to use files with dedicated responsibilities and whose contents reflect their purpose. To address this situation when you are defining a build process for a product, it is helpful to create three files:
  • MyProduct.settings.targets
  • MyProduct.targets
  • MyProduct.csproj
You can use MyProduct.setting.targets to contain various properties related to shared utilities used during the build and deployment processes as well as any other common settings. For example, if you are using NUnit, a setting that you might place in this file is the location of the NUnit references. When properties are declared in this file, they should always be declared so that they will not override any existing value. This file contains the default settings. (This is the same situation as discussed in the section "Defining Dynamic Items and Properties.")
MyProduct.targets contains the UsingTask elements that declare the custom tasks that are being used as well as any shared targets. This file defines how the product gets built in the sense that most of the targets as well as their dependencies should be contained in this file.
MyProduct.csproj defines what gets built. This project file declares all the items and properties that the MyProduct.targets file uses within its targets. The contract between these two files is represented by these shared items and properties. This file should also import MyProduct.settings.targets at the top and MyProduct.targets file at the bottom.
By doing so you can override any values in MyProduct.settings.targets and prepare properties and items for MyProduct.targets. If you relate this build process to that for a C# project, MyProduct.targets is equivalent to Microsoft.CSharp.targets (and Microsoft.Common.targets), and the other two files form the actual project file.

Using Wildcards
Visual Studio project files do not use wildcards to populate items that Visual Studio will be interacting with. For example, you could define the Compile item, which contains a list of files that will be sent to the compiler, as follows:
<ItemGroup>
  <Compile Include="src\*"/>
</ItemGroup>
This expression would place all the files in the src folder into the Compile item, and while this is valid and will work, it is not the best approach to take outside casual projects. Any edit made to the project file by Visual Studio will cause this item to expand to list each file individually. Also, this approach makes it easy to forget to check in to source control new or removed files from your local machine. Instead of using wildcards for items maintained by Visual Studio, you should explicitly include each file.
In this article I'll taken a look at a few key recommendations that you can use to create better build process for your products. As with all best practices, there will be situations in which these rules may not apply one hundred percent of the time and need to be bent a little. The best way to learn which of these practices works for you is to employ them and see which ones work and which ones fail. I would be honored to hear your stories, good and bad, about these best practices. You can contact me at sayed.hashimi@gmail.com.
I would like to thank Dan Moseley, from the MSBuild team, and Brian Kretzler for their invaluable help on this article.

Sayed Ibrahim Hashimi has been working with MSBuild since the early preview bits of Visual Studio 2005 were released. He is the author of Inside the Microsoft Build Engine: Using MSBuild and Team Foundation Build (Microsoft Press, 2009), the coauthor of Deploying .NET Applications: Learning MSBuild and ClickOnce (Apress, 2006), and has written for several publications. He works in Jacksonville, Florida, as a consultant and trainer, with expertise in the financial, education, and collection industries. You can reach Sayed at his blog, sedodream.com.


Editor's Note
Driving Design
Howard Dierking

In this issue of MSDN Magazine, we focus our attention on topics related to best practices in software design. This theme is a personal favorite, because good design practices are ubiquitous across all sub-specialties of software development. We have a compelling lineup, with topics ranging from acceptance testing and IronRuby to presentation patterns in WPF—and even an overview of the "Oslo" platform.
The article of greatest interest to me, however, is Dave Laribee's "An Introduction to Domain-Driven Design." I have been a subscriber to and practitioner of domain-driven design (DDD) thinking since I read Eric Evans' book of the same name about four years ago and have grown in this way of thinking with minimal problems. I believe that this has been largely due to the complementary relationship between domain-driven design, the rich domain model pattern, and test-driven development.
But when it comes to building software, there is no silver bullet. I watch technologies such as Windows Workflow Foundation (WF) and Windows Communication Foundation (WCF) continue to evolve; I see the continued shift toward service orientation. And I can't help but wonder whether my staunch adherence to certain practices and heuristics has caused me to recreate a few wheels of my own and subsequently undermine one of the goals of good software design—creating a more maintainable system.
Let's take the insurance problem domain that Dave references in his article. I, too, once worked in this industry and built what I considered to be a rather elegant framework for validating policy configurations and calculating projections known as illustrations. I created my domain model from the outside in using a test-driven approach, mapped my model to a database using a popular object-relational mapper, and projected my model onto multiple presentation layers. As I continued to refactor my domain model, I noticed a couple of things. First, all of the logic (and all of the tests) that I had built began to resemble something I had seen before. And in fact, when I stepped back and looked at my domain model, I realized that I had essentially written my own forward-chaining rules engine.
Keeping in mind that the most maintainable code is code that you don't have to maintain, consider how policy illustrations are calculated. The company's actual calculation logic was created by actuarial scientists using a functional, generally symbolic language such as APL or Mathematica. This logic was then translated into our object-oriented domain model. As I'm sure you can deduce, a great many bugs found during regression testing were introduced in this translation.
In light of my own experience, I wonder what my application would have looked like had I refactored my validation logic into a rules engine and taken a more polyglot-based approach for calculating. Would my application have still been "domain-driven"? Or can DDD philosophy be separated from its more frequent, object-oriented implementations?
I've been reviewing my old code recently, looking to see where my understanding of DDD principles has been biased by my OO coding skills and then refactoring appropriate pieces of the application to more suitable paradigms. What I've concluded thus far is that domain-driven thinking is still an essential part of designing software. (In fact, certain elements of DDD become more meaningful in a multi-paradigm design.) In the end, my experience has been a validation of the fact that the domain must drive the design—even when it challenges what is comfortable for us as developers.

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Surupa Biswas, Mike Calligaro, John Gossman, Tim Heuer, Ed Katibah, Gaurav Khanna, Ravi Krishnamurthy, Mike Magruder, Rick Molloy, Dave Murray, Michael Puleio, Bob Schmidt, Don Smith, and Matt Winkler.


Toolbox
The Active Record Pattern, Aspect-Oriented Programming
Scott Mitchell

All prices confirmed on December 29, 2008, and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions at Microsoft.

Contents
Implement the Active Record Pattern in Your .NET Application
Aspect-Oriented Programming in the .NET Framework

Implement the Active Record Pattern in Your .NET Application
The active record pattern is commonly used in data-driven applications to model a database table or view in terms of a class, where a row of the database table is modeled by an instance of the class. In this pattern, the class's properties map to the corresponding database table's columns.
The class's instance methods perform operations on a database record, whereas its static methods work with the whole table. For example, if the database contained a table named Employee with columns EmployeeId as a uniqueidentifier, Name as an nvarchar(50), and Salary as a money, to implement the active record pattern you'd create a class named Employee with properties named EmployeeId, Name, and Salary. The Employee class would include instance methods such as Create, Save, and Delete, and might include static methods such as DeleteAll, Find, and FindAll.
Figure 1 shows the active record pattern. It offers a simple, intuitive syntax for working with data and is the pattern used by many object-relational mappers (O/RM).
Figure 1 Active Record Pattern in Action
// Add Andrew Fuller as a new employee
Employee emp = new Employee();
emp.Name = "Andrew Fuller";
emp.Salary = 50000.00M;
emp.Create();

// Give a 10% raise to all employees
Employee[] allEmployees = Employee.FindAll();
foreach (Employee current in allEmployees)
{
    current.Salary *= 1.10M;
    current.Save();
}
The Castle ActiveRecord (version 1.0, RC3) project offers a quick and easy way to implement the active record pattern for Microsoft .NET Framework-based apps. It's built on top of NHibernate, which is a free, open-source O/RM tool and was reviewed in the October 2006 Toolbox column .
One of the drawbacks of NHibernate is that it has a bit of a steep learning curve and requires the developer to create and maintain the XML mapping files that tie together the database tables, columns, and relationships with the classes and properties in the app. The ActiveRecord project API encapsulates much of this complexity by using .NET attributes and makes implementing the active record pattern easy.
To get started, download the installation package from the project's home page. This installs the ActiveRecord and NHibernate assemblies and makes them visible to Visual Studio; add these assemblies to your project. Next, create the classes in your app that will use the active record pattern to model a database table.
This is the code for the Employee class referenced in Figure 1:
[ActiveRecord]
public class Employee : ActiveRecordBase<Employee>
{
    [PrimaryKey] public Guid EmployeeId { get; set; }
    [Property] public string Name { get; set; }
    [Property] public decimal Salary { get; set; }
}
Note that the Employee class extends the ActiveRecordBase class and uses attributes to flag which properties map to columns in the database table and which column serves as the primary key. The ActiveRecordBase class defines instance methods such as Create and Save and static methods including Find, FindAll, and DeleteAll.
You can provide more detail about each member in the Employee class by passing in parameters through the PrimaryKey and Property attributes, including the length of the string columns, whether the column requires unique values, if the column can be assigned a database NULL value, and whether to ignore the column when inserting or updating a class instance.
Along with creating the Employee class, you also need to define a few configuration settings, such as the database connection string and a handful of NHibernate-specific options. But there's no need to write any data-access code. That's handled for you automatically by the ActiveRecord project and NHibernate. In fact, you don't even need to have created the database tables at this point, as the ActiveRecord project can auto generate the tables for you based on the classes you've created.
The attribute syntax of the ActiveRecord project is also used to establish relationships among classes. For example, the HasMany and BelongsTo attributes can be used in a parent and child class, respectively, to indicate a one-to-many relationship. Imagine that the data model was expanded to include a Department table with columns DepartmentId and Name, and a DepartmentId column in the Employee table establishing a one-to-many relationship between departments and employees. This relationship would be modeled in the application by adding a Department class and updating the Employee class to include a new property (see Figure 2).
Figure 2 Adding Employee Class
[ActiveRecord]
public class Department : ActiveRecordBase<Department>
{
    [PrimaryKey] public int DepartmentId { get; set; }
    [Property] public string Name { get; set; }
    [HasMany(typeof(Employee))] public IList Employees { get; set; }
}

[ActiveRecord]
public class Employee : ActiveRecordBase<Employee>
{
    ...
    [BelongsTo("DepartmentId")] public Department Department { get; set; }
}
Notice that the Department class has an Employees property, which returns the set of Employee objects for a specific Department instance. Similarly, the Employee object has a Department property that returns information about the department to which the employee belongs.
The ActiveRecord project makes it easy to get started building your application's architecture on top of a proven framework, NHibernate. Within a few minutes you can craft classes that use the active record pattern without having to master NHibernate's XML configuration syntax.
Price: Free, open-source.
castleproject.org/activerecord

Aspect-Oriented Programming in the .NET Framework
Separation of concerns is a central tenet of software engineering that encourages separating an application's features and functionality into non-overlapping modules. Each programming paradigm has its own techniques for encouraging this. Procedural programming languages allow developers to encapsulate functionality in subroutines; object-oriented programming languages use classes.
Security, logging, and caching are all examples of concerns that typically apply to multiple modules and, when implemented, result in code scattered across those modules to which the rules apply. These types of concerns are said to be cross cutting and are referred to as aspects.
Aspect-oriented programming (AOP) is a programming paradigm that focuses on handling the separation of cross-cutting concerns. AOP allows developers to create aspects in a single module and then specify when these run—prior to the execution of a method, when a method completes with success, and so on.
For example, with AOP a developer could build an aspect that runs prior to a method's execution and verifies that the user belongs to a particular role and then specifies to what methods the aspect applies. As a result, whenever one of these specified methods is about to be executed during run time, the corresponding aspect code executes first. AOP is typically implemented through code weaving, which involves injecting the aspect code into the appropriate spots in the application code—either at the end of compile time or during run time.
Check out PostSharp (version 1.5), a free, open-source library created by Gael Fraiteur that you can use to implement AOP techniques in the .NET Framework. At its core, PostSharp analyzes and transforms .NET assemblies, allowing for the code weaving necessary to implement AOP.
Built atop this core is PostSharp Laos, a lightweight aspect-oriented system that enables developers to create and use aspects in terms of custom attributes. The aspects can then be applied to classes or methods by decorating them with the custom attribute. Alternatively, aspects can be applied to all classes or methods that have a certain naming pattern through syntax in the project's AssemblyInfo file.
Figure 3 shows an example of a PostSharp aspect for error notification. This aspect extends the OnExceptionAspect, which means that the aspect can execute in response to an unhandled exception, perhaps e-mailing an administrator details of the exception. This aspect can be applied to a particular method by decorating it with the OnException attribute, as shown here:
[OnException("admin@contoso.com")]
private void MyMethod()
{
    // ...
}
Figure 3 A PostSharp Aspect for Error Notification
public class OnExceptionAttribute : OnExceptionAspect
 {
    public string ToAddress { get; set; }

    public OnExceptionAttribute(string to)
    {
        ToAddress = to;
    }

    public override void OnException(MethodExecutionEventArgs e)
    {
        // TODO: Send ToAddress an e-mail detailing the exception details.
        // Exception details available in the MethodExecutionEventArgs object
    }
}
Immediately after the code has been compiled, PostSharp modifies the IL code of MyMethod, surrounding the existing code with a try...catch with a call to the OnException method within the catch block. As a result, the OnException aspect is invoked whenever there is an unhandled exception in MyMethod.
You'll find extensive documentation at the PostSharp Web site, along with a helpful message board. There are also community-created plug-ins, libraries, and other tools that use or extend PostSharp.
Price: Free, open-source.
postsharp.org

Send your questions and comments to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Handling Corrupted State Exceptions
Andrew Pardoe

This column is based on a prerelease version of Visual Studio 2010. All information is subject to change.

Contents
What Exactly Are Exceptions?
Win32 SEH Exceptions and System.Exception
Managed Code and SEH
Corrupted State Exceptions
It's Still Wrong to Use Catch (Exception e)
Code Wisely
Have you ever written code that isn't quite right but is close enough? Have you had to write code that works fine when everything goes well, but you're not quite sure what will happen when something goes wrong? There's a simple, incorrect statement that's probably sitting in some code that you wrote or have had to maintain: catch (Exception e). It seems innocent and straightforward, but this little statement can cause a lot of problems when it fails to do what you expect.
If you've ever seen code that uses exceptions the way the following code does, then you need to read this column:
public void FileSave(String name)
{
    try
    {
      FileStream fs = new FileStream(name, FileMode.Create);
    }
    catch (Exception)
    {
      throw new System.IO.IOException("File Open Error!");
    }
}
The error in this code is common: it is simpler to write code to catch all exceptions than it is to catch exactly the exceptions that could be raised by code executing in your try block. But by catching the base of the exception hierarchy, you will end up swallowing an exception of any type and converting it to an IOException.
Exception handling is one of those areas about which most people have a working knowledge rather than an intimate understanding. I'll begin with a bit of background information explaining exception handling from the CLR point of view for those who may be more familiar with exception handling from native programming or an old college textbook. If you're an old pro at managed exception handling, feel free to skip ahead to the section on different kinds of exceptions or the one on managed code and structured exception handling (SEH). Make sure to read the last few sections, though.

What Exactly Are Exceptions?
An exception is the signal raised when a condition is detected that was not expected in the normal execution of a program thread. Many agents can detect incorrect conditions and raise exceptions. Program code (or the library code it uses) can throw types derived from System.Exception, the CLR execution engine can raise exceptions, and unmanaged code can raise exceptions as well. Exceptions raised on a thread of execution follow the thread through native and managed code, across AppDomains, and, if not handled by the program, are treated as unhandled exceptions by the operating system.
An exception indicates that something bad has happened. While every managed exception has a type (such as System.ArgumentException or System.ArithmeticException), the type is only meaningful in the context in which the exception is raised. A program can handle an exception if it understands the conditions that caused the exception to occur. But if the program doesn't handle the exception, it could indicate any number of bad things. And once the exception has left the program, it only has one very general meaning: something bad has happened.
When Windows sees that a program doesn't handle an exception, it tries to protect the program's persisted data (files on disk, registry settings, and so on) by terminating the process. Even if the exception originally indicated some benign, unexpected program state (such as failing to pop from an empty stack) it appears to be a serious problem when Windows sees it because the operating system doesn't have the context to properly interpret the exception. A single thread in one AppDomain can bring down an entire CLR instance by not handling an exception (see Figure 1).
Figure 1 One Thread’s Unhandled Exception Causes the Entire Process to Terminate
If exceptions are so dangerous, why are they so popular? Like motorcycles and chain saws, the raw power of exceptions makes them very useful. Normal dataflow on a program thread goes from function to function through calls and returns. Every call to a function creates a frame of execution on the stack; every return destroys that frame. Aside from altering global state, the only flow of data in a program is achieved by passing data between contiguous frames as function parameters or return values. In the absence of exception handling, every caller needs to check for the success of the function that it called (or just assume everything is always OK).
Most Win32 APIs return a non-zero value to indicate failure because Windows doesn't use exception handling. The programmer has to wrap every function call with code that checks the return value of the called function. For example, this code from MSDN documentation on how to list files in a directory explicitly checks each call for success. The call to FindNextFile(...) is wrapped in a check to see if the return is non-zero. If the call is not successful then a separate function call—GetLastError()—provides details of the exceptional condition. Note that every call must be checked for success on the next frame because return values are necessarily limited to the local function scope:
// FindNextFile requires checking for success of each call
while (FindNextFile(hFind, &ffd) != 0);
dwError = GetLastError();
if (dwError != ERROR_NO_MORE_FILES)
{
  ErrorHandler(TEXT("FindFirstFile"));
}
FindClose(hFind);
return dwError;
An error condition can only pass from the function containing the unexpected condition to that function's caller. Exceptions have the power to pass the results of a function's execution out of the current function's scope to every frame up the stack until it reaches the frame that knows how to handle the unexpected condition. The CLR's exception system (called a two-pass exception system) delivers the exception to every predecessor on the thread's call stack, beginning with the caller and proceeding until some function says it will handle the exception (this is known as the first pass).
The exception system will then unwind the state of each frame on the call stack between where the exception is raised and where it will be handled (known as the second pass). As the stack unwinds, the CLR will run both finally clauses and fault clauses in each frame as it is unwound. Then, the catch clause in the handling frame is executed.
Because the CLR checks every predecessor on the call stack there's no need for the caller to have a catch block—the exception can be caught anywhere up the stack. Instead of having code to immediately check the result of every function call, a programmer can handle errors in a place far away from where the exception was raised. Using error codes requires the programmer to inspect and pass on an error code at every stack frame until it reaches the place where the erroneous condition can be handled. Exception handling frees the programmer from inspecting the exception at every frame on the stack.
For more on throwing custom exception types, see "Error Handling: Throwing Custom Exception Types from a Managed COM+ Server Application".

Win32 SEH Exceptions and System.Exception
There's an interesting side effect that comes from the ability to catch exceptions far away from where the exception was raised. A program thread can receive program exceptions from any active frame on its call stack without knowing where the exception was raised. But exceptions don't always represent an error condition that the program detects: a program thread can also cause an exception outside of the program.
If a thread's execution causes a processor to fault, then control is transferred to the operating system kernel, which presents the fault to the thread as an SEH exception. Just as your catch block doesn't know where on the thread's stack an exception was raised, it doesn't need to know exactly at what point the OS kernel raised the SEH exception.
Windows notifies program threads about OS exceptions using SEH. Managed code programmers will rarely see these because the CLR typically prevents the kinds of errors indicated by SEH exceptions. But if Windows raises an SEH exception, the CLR will deliver it to managed code. Although SEH exceptions in managed code are rare, unsafe managed code can generate a STATUS_ACCESS_VIOLATION which indicates that the program has attempted to access invalid memory.
For details on SEH, see Matt Pietrek's article "A Crash Course on the Depths of Win32 Structured Exception Handling" in the January 1997 issue of Microsoft Systems Journal.
SEH exceptions are a different class from those exceptions raised by your program. A program might raise an exception because it tried to pop an item from an empty stack or tried to open a file that didn't exist. All of these exceptions make sense in the context of your program's execution. SEH exceptions refer to a context outside of your program. An access violation (AV), for example, indicates an attempted write to invalid memory. Unlike program errors, an SEH exception indicates that the integrity of the runtime's process may have been compromised. But even though SEH exceptions are different from exceptions that derive from System.Exception, when the CLR delivers the SEH exception to a managed thread it can be caught with a catch (Exception e) statement.
Some systems attempt to separate these two kinds of exceptions. The Microsoft Visual C++ compiler distinguishes between exceptions raised by a C++ throw statement and Win32 SEH exceptions if you compile your program with the /EH switch. This separation is useful because a normal program doesn't know how to handle errors that it did not raise. If a C++ program tries to add an element to a std::vector, it should expect that the operation may fail due to lack of memory, but a correct program using well-written libraries shouldn't be expected to deal with an access violation.
This separation is useful for programmers. An AV is a serious problem: an unexpected write to critical system memory can affect any part of the process unpredictably. But some SEH errors, such as a divide-by-zero error resulting from bad (and unchecked) user input, are less serious. While a program with a divide-by-zero is incorrect, it's unlikely that this will affect any other part of the system. In fact, it's likely that a C++ program could handle a divide-by-zero error without destabilizing the rest of the system. So while this separation is useful, it doesn't quite express the semantics managed programmers need.

Managed Code and SEH
The CLR has always delivered SEH exceptions to managed code using the same mechanisms as exceptions raised by the program itself. This isn't a problem as long as code doesn't attempt to handle exceptional conditions that it cannot reasonably handle. Most programs cannot safely continue execution after an access violation. Unfortunately, the CLR's exception handling model has always encouraged users to catch these serious errors by allowing programs to catch any exception at the top of the System.Exception hierarchy. But this is rarely the right thing to do.
Writing catch (Exception e) is a common programming error because unhandled exceptions have serious consequences. But you might argue that if you don't know what errors will be raised by a function, you should protect against all possible errors when your program calls that function. This seems like a reasonable course of action until you think about what it means to continue execution when your process is possibly in a corrupted state. Sometimes aborting and trying again is the best option: nobody likes to see a Watson dialog, but it's better to restart your program than to have your data corrupted.
Programs catching exceptions arising from contexts they don't understand is a serious problem. But you can't solve the problem by using exceptions specifications or some other contract mechanism. And it's important that managed programs be able to receive notification of SEH exceptions because the CLR is a platform for many kinds of applications and hosts. Some hosts, such as SQL Server, need to have total control of their application's process. Managed code that interoperates with native code sometimes must deal with native C++ exceptions or SEH exceptions.
But most programmers who write catch (Exception e) don't really want to catch access violations. They'd prefer that execution of their program stops when a catastrophic error occurs rather than letting the program limp along in an unknown state. This is especially true for programs that host managed add-ins such as Visual Studio or Microsoft Office. If an add-in causes an access violation and then swallows the exception, the host could be doing damage to its own state (or user files) without ever realizing something went wrong.
In version 4 of the CLR, the product team is making exceptions that indicate a corrupted process state distinct from all other exceptions. We are designating about a dozen SEH exceptions to indicate corrupted process state. The designation is related to the context in which the exception is raised as opposed to the exception type itself. This means that an access violation received from Windows will be marked as a corrupted state exception (CSE), but one raised in user code by writing throw new System.AccessViolationException will not be marked as a CSE. If you attended PDC 2008, you received a Community Technology Preview of Visual Studio 2010 which includes these changes.
It's important to note that the exception doesn't corrupt the process: the exception is raised after corruption is detected in the process state. For example, when a write through a pointer in unsafe code references memory that doesn't belong to the program, an access violation is raised. The illegal write didn't actually occur—the operating system checked the memory's ownership and prevented the action from taking place. The access violation indicates that the pointer itself was corrupted at an earlier time in the thread's execution.

Corrupted State Exceptions
In version 4 and later, the CLR exception system will not deliver CSEs to managed code unless the code has expressly indicated that it can handle process corrupted state exceptions. This means that an instance of catch (Exception e) in managed code won't have the CSE presented to it. By making the change inside of the CLR exception system you don't need to change the exception hierarchy or alter any managed language's exception handling semantics.
For compatibility reasons, the CLR team provided some ways to let you run your old code under the old behavior:
  • If you want to recompile your code created in the Microsoft .NET Framework 3.5 and run it in the .NET Framework 4.0 without having to update the source, you can add an entry in your application configuration file: legacyCorruptedStateExceptionsPolicy=true.
  • Assemblies compiled against the .NET Framework 3.5 or an earlier version of the runtime will be able to process corrupted state exceptions (in other words, maintain the old behavior) when running on the .NET Framework 4.0.
If you want your code to handle CSEs, you must indicate your intention by marking the function containing the exceptions clause (catch, finally, or fault) with a new attribute: System.Runtime.ExceptionServices.HandleProcessCorruptedStateExceptions. If a CSE is raised, the CLR will perform its search for a matching catch clause but will only search in functions marked with the HandleProcessCorruptedStateExceptions attribute (see Figure 2).
Figure 2 Using HandleProcessCorruptedStateExceptions
// This program runs as part of an automated test system so you need
// to prevent the normal Unhandled Exception behavior (Watson dialog).
// Instead, print out any exceptions and exit with an error code.
[HandledProcessCorruptedStateExceptions]
public static int Main()
{
    try
    {
        // Catch any exceptions leaking out of the program
        CallMainProgramLoop();
    }
    catch (Exception e) // We could be catching anything here
    {
        // The exception we caught could have been a program error
        // or something much more serious. Regardless, we know that
        // something is not right. We'll just output the exception
        // and exit with an error. We won't try to do any work when
        // the program or process is in an unknown state!
        System.Console.WriteLine(e.Message);
        return 1;
    }
    return 0;
  }
If a suitable catch clause is found, the CLR will unwind the stack as normal but will only execute finally and fault blocks (and in C#, the implicit finally block of a using statement) in functions marked with the attribute. The HandleProcessCorruptedStateExceptions attribute is ignored when encountered in partially trusted or transparent code because a trusted host wouldn't want an untrusted add-in to catch and ignore these serious exceptions.

It's Still Wrong to Use Catch (Exception e)
Even though the CLR exception system marks the worst exceptions as CSE, it's still not a good idea to write catch (Exception e) in your code. Exceptions represent a whole spectrum of unexpected situations. The CLR can detect the worst exceptions—SEH exceptions that indicate a possibly corrupt process state. But other unexpected conditions can still be harmful if they are ignored or dealt with generically.
In the absence of process corruption, the CLR offers some pretty strong guarantees about program correctness and memory safety. When executing a program written in safe Microsoft Intermediate Language (MSIL) code you can be certain that all the instructions in your program will execute correctly. But doing what the program instructions say to do is often different from doing what the programmer wants. A program that is completely correct according to the CLR can corrupt persisted state, such as program files written to a disk.
Take as a simple example a program that manages a database of test scores for a high school. The program uses object-oriented design principles to encapsulate data and raises managed exceptions to indicate unexpected events. One day the school secretary hits the Enter key one too many times when generating a grade file. The program tries to pop a value from an empty queue and raises a QueueEmptyException which goes unhandled through frames on the callstack.
Somewhere near the top of the stack is a function, GenerateGrades(), with a try/catch clause that catches Exception. Unfortunately, GenerateGrades() has no idea that students are stored in a queue and doesn't have any idea what to do with a QueueEmptyException. But the programmer who wrote GenerateGrades() doesn't want the program to crash without saving the data that's been computed so far. Everything is written safely to the disk and the program exits.
The problem with this program is that it makes a number of assumptions that could be incorrect. What's to say that the missing entry in the student queue is at the end? Maybe the first student record got skipped, or the tenth. The exception only tells the programmer that the program is incorrect. Taking any action—saving data to disk or "recovering" and continuing execution—is just plain wrong. No correct action is possible without knowledge of the context in which the exception was raised.
If the program had caught a specific exception close to where the exception was raised, it might have been able to take proper action. The program knows what a QueueEmptyException means in the function that attempts to dequeue students. If the function catches that exception by type—rather than catching a whole class of exception types—it would be in a much better position to try to make the program state correct.
In general, catching a specific exception is the correct thing to do because it provides the most context to your exception handler. If your code can potentially catch two exceptions, then it has to be able to deal with both. Writing code that says catch (Exception e) has to be able to deal with literally any exceptional situation. That's a promise that is very hard to keep.
Some languages try to prevent programmers from catching a broad class of exceptions. For example, C++ has exception specifications, a mechanism that allows a programmer to specify what exceptions can be raised in that function. Java takes this a step further with checked exceptions, a compiler-enforced requirement that a certain class of exceptions be specified. In both languages, you list the exceptions that can flow out of this function in the function declaration and callers are required to handle those exceptions. Exception specifications are a good idea, but they have had mixed results in practice.
There's vigorous debate as to whether or not any managed code should be able to handle CSEs. These exceptions normally denote a systems-level error and should only be handled by code that understands the systems-level context. While most people don't need the ability to handle CSEs, there are a couple of scenarios where it's necessary.
One scenario is when you are very close to where the exception happened. For example, consider a program that calls native code that's known to be buggy. Debugging into the code you learn that it sometimes zeros a pointer before accessing it, which causes an access violation. You might want to use the HandleProcessCorruptedStateExceptions attribute on the function which calls native code using P/Invoke because you know the cause of the pointer corruption and are comfortable that the process integrity is maintained.
The other scenario that may call for the use of this attribute is when you're as far as you can be from the error. In fact, you're almost ready to exit your process. Say you've written a host or a framework that wants to perform some custom logging in case of an error. You might wrap your main function with a try/catch/finally block and mark it with HandleProcessCorruptedStateExceptions. If an error unexpectedly makes it all the way up to your program's main function, you write some data to your log, doing as little work as you have to, and exit the process. When the integrity of the process is in question any work you do could be dangerous, but if the custom logging fails sometimes it's acceptable.
Take a look at the diagram shown in Figure 3. Here function 1 (fn1()) is attributed with [HandleProcessCorruptedStateExceptions] so its catch clause catches the access violation. The finally block in function 3 does not execute even though the exception will be caught in function 1. Function 4 at the bottom of the stack raises an access violation.
Figure 3 Exception and Access Violation
There is no guarantee in either of these scenarios that what you're doing is completely safe, but there are scenarios where just terminating the process is unacceptable. However, if you decide to handle a CSE there is a huge burden on you as the programmer to do it correctly. Remember that the CLR exception system won't even deliver a CSE to any function that is not marked with the new attribute either during the first pass (when it searches for a matching catch clause) or the second pass (when it unwinds each frame's state and executes finally and fault blocks).
The finally block exists in order to guarantee that code always runs, whether or not there is an exception. (Fault blocks run only when an exception occurs, but they have a similar guarantee of always being executed.) These constructs are used to clean up critical resources such as releasing file handles or reversing impersonation contexts.
Even code that is written to be reliable through the use of constrained execution regions (CER) won't be executed when a CSE has been raised unless it is in a function that has been marked with the HandleProcessCorruptedStateExceptions attribute. It's very difficult to write correct code that handles a CSE and continues running the process safely.
Look closely at the code in Figure 4 to see what could go wrong. If this code is not in a function that can process CSEs, then the finally block will not run when an access violation happens. That's fine if the process terminates—the open file handle will be released. But if some other code catches the access violation and tries to restore state, it needs to know that it has to close this file as well as restore any other external state this program has changed.
Figure 4 The Finally Block May Not Run
void ReadFile(int index)
    {
      System.IO.StreamReader file =
        new System.IO.StreamReader(filepath);
          try
          {
            file.ReadBlock(buffer, index, buffer.Length);
          }
          catch (System.IO.IOException e)
          {
            Console.WriteLine("File Read Error!");
          }
          finally
          {
            if (file != null)
                {
                    file.Close()
                }
          }
    }
If you decide that you're going to handle a CSE, your code needs to expect that there is a ton of critical state that has not been unwound. Finally and fault blocks have not been run. Constrained execution regions have not been executed. The program—and the process—are in an unknown state.
If you know your code will do the right thing, then you know what to do. But if you're not sure of the state your program is executing in, then it's better to just let your process exit. Or, if your application is hosted, invoke the escalation policy your host has specified. See Alessandro Catorcini and Brian Grunkemeyer's CLR Inside Out column from December 2007 for more information on writing reliable code and CERs.

Code Wisely
Even though the CLR prevents you from naively catching CSEs, it's still not a good idea to catch overly broad classes of exceptions. But catch (Exception e) appears in a lot of code, and it's unlikely that this will change. By not delivering exceptions that represent a corrupted process state to code that naively catches all exceptions, you prevent this code from making a serious situation worse.
The next time you write or maintain code that catches an exception, think about what the exception means. Does the type you caught match what your program (and the libraries it uses) is documented to throw? Do you know how to handle the exception such that your program can correctly and safely continue execution?
Exception handling is a powerful tool that should be used carefully and thoughtfully. If you really need to use this feature—if you really need to handle exceptions that may indicate a corrupt process—the CLR will trust you and let you do so. Just be careful and do it correctly.

Send your questions and comments to clrinout@microsoft.com.

Andrew Pardoe is a program manager for CLR at Microsoft. He works on many aspects of the execution engine for both the desktop and Silverlight runtimes. He can be reached at Andrew.Pardoe@microsoft.com.


Data Points
Syndicated Data And Isolated Storage In Silverlight
John Papa

Code download available from the MSDN Code Gallery
Browse the Code Online

Contents
Getting Finished
HTTP Web Requests and Threading
Adding a Feed
Parsing Feeds
Cross-Domain Requests for Feeds
Basic Isolated Storage
Organized Isolated Storage
Wrap-Up
Silverlight is ideal for building a syndicated news reader application. It can read RSS and AtomPub syndicated formats, communicate with Web services via HTTP requests, and handle cross-domain policies. Once the syndicated data has been received, it can be read into a class structure, parsed with LINQ, and presented to the user via XAML-based data binding.
In this column I will demonstrate how to utilize these unique features to build a syndicated news reader. I will also show you how to debug problems with Web service communications from Silverlight and how to store data locally using isolated storage. All of the examples are available in the MSDN code download.

Getting Finished
Feed syndication provides access to the RSS and AtomPub formats through Web services. Each feed is accessed through a Uri that returns XML containing the feed items using the RSS or AtomPub formats. The sample application provided with this column demonstrates techniques that read syndicated data using Web service requests into a Silverlight application. Before diving into the details, it might be helpful if I show you the finished version of the application. Then I will discuss the logic and various other aspects of the application's code.
Figure 1 shows the Silverlight application reading these two syndicated feeds:
http://pipes.yahooapis.com/pipes/pipe.run?_id=957
    d9624940693fb9f9644d7b12fb0e9&_render=rss
http://pipes.yahooapis.com/pipes/pipe.run?_id=057559bac7aad6640b
    c17529f3421db0&_render=rss
Figure 1 The SilverlightSyndication Application
A Web request is made to gather the feed data when the user clicks the Add button. The feed's Uri is stored locally on the client computer, and the title of the feed service is displayed in the upper DataGrid control.
Storing the feed's Uri on the client computer allows the Silverlight application to get the feed data for those feeds when the application starts. When the feed data is gathered, the items are placed into a list of objects and then parsed using LINQ. The results are then bound to the lower DataGrid control in sorted order.

HTTP Web Requests and Threading
Consuming syndicated feeds from Silverlight applications starts with being able to make Web requests. Both the WebClient and the HttpWebRequest classes can make HTTP Web requests from Silverlight applications. The first step in the process is deciding which technique to use to make HTTP Web requests. In most cases, the WebClient class is all that is needed since it is simpler to use (it also calls the HttpWebRequest under the covers). HttpWebRequest allows for more customization of the requests.
This code snippet demonstrates how to make a request through WebClient:
//feedUri is of type Uri
WebClient request = new WebClient();
request.DownloadStringCompleted += AddFeedCompleted;
request.DownloadStringAsync(feedUri);
The following code, on the other hand, shows how to make a similar call using HttpWebRequest:
//feedUri is of type Uri
WebRequest request = HttpWebRequest.Create(feedUri);
request.BeginGetResponse(new AsyncCallback(ReadCallback), request);
The WebClient class is easier to use because it wraps some of the functionality of the HttpWebRequest class.
Since all HTTP Web requests from Silverlight are made asynchronously through both WebClient and HttpWebRequest, it is important to understand how to process the data when the calls return. When HttpWebRequest makes its asynchronous calls and it completes, the completed event handler is not guaranteed to be operating on the UI thread. If data were retrieved and meant to be displayed in a UI element, the call to the UI element must be made using a technique that transfers control back to the UI thread from the background thread. This can be done with the Dispatcher or through the SynchronizationContext class. The following code shows how to make a call using the UI thread with the Dispatcher class's BeginInvoke method:
Deployment.Current.Dispatcher.BeginInvoke(() =>
     {
         MyDataGrid.DataContext = productList;
     });
This code example takes the productList variable (which was filled from the data returned from a Web service call) and sets it to a UI element's DataContext. In this case, a DataGrid will now be bound to the list of products. The Dispatcher is not needed, however, if the call was made through the WebClient class. In that case, the code could simply set the product list directly to the UI element's DataContext.
To use the SynchronizationContext class, an instance of the SynchronizationContext class must be created in a place where the UI thread is known to be available. The constructor and the load event handler are good places to create the instance of a SynchronizationContext. This code sample shows the _syncContext field being initialized in the class constructor:
public Page() {
_syncContext = SynchronizationContext.Current;
}
And this code shows the SynchronizationContext instance using its Post method to make the call to the LoadProducts method. It makes sure that the LoadProducts method has access to the UI thread:
if (_syncContext != null) {
  _syncContext.Post(delegate(object state){ LoadProducts(products); }
  ,null);
}
Besides being easier to use, WebClient requests always come back on the UI thread. This means that any results from the WebClient's request can be easily bound to the UI elements without having to involve the Dispatcher (or, optionally, the SynchronizationContext class instead of the Dispatcher). For reading syndicated data, the WebClient class is adequate and will be used for this column's sample application.

Adding a Feed
In the sample application, when the user enters a feed address and clicks the Add button, the code shown in Figure 2 executes. First the code attempts to create a Uri from the address, using the Uri.TryCreate method if possible. If it can create a Uri, the Uri is returned to the local variable feedUri. Otherwise, feedUri remains null and the code exits.
Figure 2 Adding a Feed
private void btnAdd_Click(object sender, RoutedEventArgs e)
{
    Uri feedUri;
    Uri.TryCreate(txtAddress.Text, UriKind.Absolute, out feedUri);
    if (feedUri == null)
        return;

    LoadFeed(feedUri);
}

public void LoadFeed(Uri feedUri)
{
    WebClient request = new WebClient();
    request.DownloadStringCompleted += AddFeedCompleted;
    request.DownloadStringAsync(feedUri);
}
Once a valid Uri is created, the LoadFeed method executes, making the HTTP request to gather the feed data using the WebClient class. The WebClient instance is created, and an event handler is assigned to the DownloadStringCompleted event. When the DownloadStringAsync method is executed and is ready to return its data, it needs to know what event handler to go to. That is why the event handler (in this case, AddFeedCompleted) must be assigned before the asynchronous event is executed.
Once the request has completed, the AddFeedCompleted event handler will execute (see Figure 3). The DownloadStringCompletedEventArgs parameter contains a Result property and an Error property, both of which are important to check after each Web request. The e.Error property will be null if there were no errors during the request. The e.Result property contains the results of the Web request. For the sample application, e.Result will contain the XML representing the feed data.
Figure 3 AddFeedCompleted
private void AddFeedCompleted(object sender,
    DownloadStringCompletedEventArgs e)
{
    if (e.Error != null)
        return;
    string xml = e.Result;
    if (xml.Length == 0)
        return;
    StringReader stringReader = new StringReader(xml);
    XmlReader reader = XmlReader.Create(stringReader);
    SyndicationFeed feed = SyndicationFeed.Load(reader);
    if (_feeds.Where(f => f.Title.Text == feed.Title.Text).ToList().Count > 0)
        return;
    _feeds.Add(feed); // This also saves the feeds to isolated storage
    ReBindAggregatedItems();
    txtAddress.Text = string.Empty;
}
Once the feed data is gathered, it can be read into the System.ServiceModel.SyndicationFeed class using the SyndicationFeed class's Load method. Note that when retrieving feed data and using it in a read-only manner, using LINQ to XML to retrieve the feed and loading it in a custom object may be a better option than SyndicationFeed. SyndicationFeed has more features, but if they are not being used it may not be worth the additional size added to the XAP—SyndicationFeed adds about 150KB to the XAP while LINQ to XML adds about 40KB. With the additional power of SyndicationFeed you also have some cost in size.
SyndicationFeed is a special class that knows how to represent feed data (both RSS and AtomPub) as an object. It has properties that describe the feed itself, such as Title and Description, as well as an Items property that contains an IEnumerable<SyndicationItem>. Each SyndicationItem class instance represents a feed item for the feed. For example, the feeds are represented by instances of the SyndicationFeed class, and their Items collections contain the individual posts from the feeds.
Once the SyndicationFeed class is loaded with the feed and its items, the code shown in Figure 3 checks to see if the same feed has already been gathered. If so, the code exits immediately. Otherwise, the feed is added to the local ObservableCollection<SyndicationFeed> called _feeds. Through the ReBindAggregatedItems method, the feed items from all of the loaded feeds are then filtered, sorted, and bound to the lower DataGrid. Since the WebClient class made the HTTP Web request, the AddFeedCompleted event handler will have access to the UI thread. This is why the code inside the ReBindAggregatedItems method can bind the data to the DataGrid UI element without the Dispatcher's help.

Parsing Feeds
When the ReBindAggregatedItems method executes, the feed data is stored in a collection of SyndicatedFeed instances and their respective collections of SyndicatedItem instances. LINQ is ideal for querying the feed data since it is now in an object structure. The data did not need to be loaded into SyndicatedFeed objects. Instead it could have been kept in its native XML format (as RSS or AtomPub), and it could have been parsed using an XmlReader or LINQ to XML. However, the SyndicatedFeed class makes it easier to manage, and LINQ can still be used to query the data.
Displaying the feed items for several feeds requires that the feed items are all mashed together. The LINQ query shown in Figure 4 demonstrates how to grab all of the feed items (SyndicationItem instances) for all of the feeds (SyndicationFeed instances) and sort them by their publication date.
Figure 4 Querying Feeds with LINQ
private void ReBindAggregatedItems()
{
    //Read the feed items and bind them to the lower list
    var query = from f in _feeds
                from i in f.Items
                orderby i.PublishDate descending
                select new SyndicationItemExtra
                        { FeedTitle = f.Title.Text, Item = i };

    var items = query.ToList();
    feedItemsGridLayout.DataContext = items;
}
Notice in Figure 4 that the query returns a list of SyndicationItemExtra classes. The SyndicationItemExtra class is a custom class that has FeedTitle property of type string and an Item property of type SyndicationItem. The application displays the feed items in the DataGrid, and most of the data for this can be found in the SyndicationItem class.
However, since the application mashes together items from several feeds, displaying the title of the feed for each feed item makes it clear which feed each item is from. The title for the feed is not accessible from the SyndicationItem class, so the application uses a custom class called SyndicationItemExtra, which will store the SyndicationItem and the feed's title.
The feed items are then bound to the Grid panel feedItemsGridLayout in the Silverlight application. The Grid panel contains the DataGrid as well as other UI elements (such as the number of items displayed in a TextBlock) that are involved in data-binding operations to display information about the feed items.

Cross-Domain Requests for Feeds
Requests to gather feeds are HTTP Web requests that generally make requests to a different Web domain. Any Web request from Silverlight that communicates with a different domain than the one that hosts the Silverlight application must comply with the remote domain's cross-domain policy. The diagram in Figure 5 demonstrates this situation.
Figure 5 Cross-Domain Call for a Feed
For more on cross-domain policies, please refer to my September 2008 Data Points column. In that column I discuss the file formats and how the policies work.
When an HTTP Web request is made across domains, Silverlight preempts the request by first requesting the cross-domain policy file from the remote Web server. Silverlight first looks for the clientaccesspolicy.xml file (the Silverlight cross-domain policy file), and if it is not found, it then looks for the crossdomain.xml file (the Flash cross-domain policy file). If neither file is found, the request fails and an error is thrown. This error can be caught in the DownloadStringCompleted event handler and presented to the user, if desired.
For example, if the Uri http://johnpapa.net/feed/default.aspx is entered into the sample application, Silverlight will first look for one of the cross-domain policy files on the johnpapa.net Web server's root. If neither of the files are found, then an error is returned to the application, at which point the application can notify the user if desired. Figure 6 shows the FireBug plug-in, which is tracking all requests from the browser. It shows the browser looking for the cross-domain policy files, not finding them, and returning without actually making the request for the RSS feed.
Figure 6 Debugging Cross-Domain Feed Calls
FireBug is a great tool for watching HTTP requests in Firefox, and Web Development Helper is a great tool when using Internet Explorer. Another option is Fiddler2, which is a standalone application that can watch all traffic on your computer.
One solution to this problem is to ask the Web administrator for the feed to put a clientaccesspolicy.xml file in the Web server's root. This may not be realistic, since you most likely do not control the remote Web server nor do you know who does. Another option is to see if the feed uses an intermediary service such as Yahoo Pipes. For example, the main feed at johnpapa.net can be retrieved through Yahoo Pipes using the Uri http://pipes.yahooapis.com/pipes/pipe.run?_id=057559bac7aad6640bc17529f3421db0&_render=rss. Since there is a cross-domain policy file located at http://pipes.yahooapis.com/clientaccesspolicy.xml that allows open access, this is a good alternative.
A third option is to use a service such as Popfly or FeedBurner to aggregate the feeds, effectively relaying them through a service that also has an open cross-domain policy. Finally, a fourth option would be to write your own custom Web service that gathers the feeds and then relays them to the Silverlight application. Using a service such as Popfly or Yahoo Pipes offers the simplest solutions.

Basic Isolated Storage
The sample app lets a user add several feeds and view all of the items for each of those feeds. If a user enters 10 feeds and decides that he needs to close the app and come back later to read them, he would likely expect that the feeds would be remembered by the application. Otherwise, he'd have to enter the Uri for each feed every time the app is opened. Since these feeds are specific to a user, they can be stored either on the server using some token to identify the user that entered them or on the user's computer.
Silverlight allows data to be stored to a protected area of the user's computer using the classes in the System.IO.IsolatedStorage namespace. Silverlight Isolated Storage is like cookies on steroids: it allows you to store simple scalar values or even store serialized complex object graphs on the client computer. The simplest way to save to isolated storage is to create an ApplicationSettings entry and stuff your data in it, as shown here:
private void SaveFeedsToStorage_UsingSettings()
{
    string data = GetFeedsFromStorage_UsingSettings() + FEED_DELIMITER +
        txtAddress.Text;
    if (IsolatedStorageSettings.ApplicationSettings.Contains(FEED_DATA))
        IsolatedStorageSettings.ApplicationSettings[FEED_DATA] = data;
    else
        IsolatedStorageSettings.ApplicationSettings.Add(FEED_DATA, data);
}
This can be called every time a SyndicationFeed is added or removed from the ObservableCollection<SyndicationFeed> field instance called _feeds. Since the ObservableCollection exposes a CollectionChanged event, a handler can be assigned to the event that performs the save, as shown here:
_feeds.CollectionChanged += ((sender, e) => {
                               SaveFeedsToStorage_UsingSettings(); });
When the SaveFeedsToStorage_UsingSettings method is executed, it first calls the GetFeedsFromStorage_UsingSettings method, which grabs the addresses of all of the feeds from isolated storage and puts them in a single string delimited by a special character.
When the app first starts, the LoadFeedsFromStorage_UsingSettings method retrieves the feeds from isolated storage:
private void LoadFeedsFromStorage_UsingSettings()
{
    string data = LoadFeedsFromStorage_UsingSettings();
    string[] feedList = data.Split(new string[1] { FEED_DELIMITER },
      StringSplitOptions.RemoveEmptyEntries);
    foreach (var address in feedList)
        LoadFeed(new Uri(address));
}
The code first reads the list of Uri addresses for each feed from isolated storage. Then it iterates through the addresses and loads each individual feed one at a time, using the LoadFeed method.

Organized Isolated Storage
This feature allows the application to remember the feed addresses for the user and load them when the user runs the application. Packing Uri addresses into a delimited string is simple but neither elegant nor expansive. For example, if you wanted to store more than just a single scalar value, this would get complicated using this technique.
Another way to store data in isolated storage is to use the IsolatedStorageFile and IsolatedStorageFileStream classes, which let you store more complex data structures, including serialized objects, per user. The data can even be segmented into different files and folders in isolated storage. This is ideal for organizing data that will be saved in isolated storage. For example, a folder could be created for all static lists of data, and a separate file for each list could be created. So within a folder in isolated storage, a file might exist for name prefixes, another for gender, and yet another for U.S. states.
The sample application could create a file in isolated storage to contain the list of Uri addresses. The data must first be serialized and then sent to the file in isolated storage (as shown in Figure 7). First, an instance of the IsolatedStorageFile class for the current user is created using the GetUserStoreForApplication method. Then a file stream is created so the application can write the Uri address. The data is then serialized and written to the IsolatedStorageFileStream instance. The example for this application serializes a string, but any serializable object can be written to isolated storage as well.
Figure 7 Saving Serialized Data to an Isolated Storage File
private void SaveFeedsToStorage_UsingFile() {
    using (var isoStore = IsolatedStorageFile.GetUserStoreForApplication())
{
        List<string> data = GetFeedsFromStorage_UsingFile();
        if (data == null)
            if (txtAddress.Text.Length == 0)
                return;
            else
                data = new List<string>();
         using (var isoStoreFileStream =
                new IsolatedStorageFileStream(FEED_FILENAME,
                   FileMode.Create, isoStore)) {
            data.Add(txtAddress.Text);
            byte[] bytes = Serialize(data);
            isoStoreFileStream.Write(bytes, 0, bytes.Length);
        }
    }
}
Reading the serialized data out of a file from isolated storage is a little more involved than the previous example. Figure 8 shows that first you must get an instance of the IsolatedStorageFile class for the user, and then check if the file exists before you read it. If the file exists, the file is opened for read access, which allows the data to be read via a stream of type IsolatedStorageFileStream. The data is read from the stream, put together, and then deserialized so it can be used to load the syndicated feeds.
Figure 8 Reading Serialized Data from an Isolated Storage File
private List<string> GetFeedsFromStorage_UsingFile() {
    byte[] feedBytes;
    var ms = new MemoryStream();
    using (var isoStore =
      solatedStorageFile.GetUserStoreForApplication())
    {
        if (!isoStore.FileExists(FEED_FILENAME)) return null;
        using (var stream = isoStore.OpenFile(FEED_FILENAME,
          FileMode.Open, FileAccess.Read))  {
            while (true) {
                byte[] tempBytes = new byte[1024];
                int read = stream.Read(tempBytes, 0, tempBytes.Length);
                if (read <= 0) {
                    //feedBytes = ms.ToArray();
                    break;
                }
                ms.Write(tempBytes, 0, read);
            }
        }
        feedBytes = ms.ToArray();
        List<string> feedList = Deserialize(typeof(List<string>),
            feedBytes) as List<string>;
        return feedList;
    }
}

private void LoadFeedsFromStorage_UsingFile() {
    var feedList = GetFeedsFromStorage_UsingFile();
    foreach (var address in feedList) {
        Uri feedUri;
        Uri.TryCreate(address, UriKind.Absolute, out feedUri);
        if (feedUri != null)
            LoadFeed(feedUri);
    }
}
For simpler data structures, using serialized objects and files in isolated storage may not be necessary. However, when isolated storage is used for several types of local storage, it can help organize the data and provide easy access to read and write to it. Isolated storage should be used wisely to store data that should be cached locally.
Also note that users can ultimately clear the storage at any time since they have full control over their settings. This means that data stored in isolated storage should not be considered guaranteed persistent storage. Another good example is storing a list of U.S. states in isolated storage so a Web request (and database hit) does not have to be made every time a combobox needs to be filled with a list of U.S. states.

Wrap-Up
The sample application demonstrates how easy it is to load RSS and AtomPub feeds into a Silverlight application. Silverlight makes it possible to make a Web request, accept its results, handle cross-domain policy calls, load feed data into SyndicationFeed classes, query them with LINQ, bind them to UI elements, and store feed data in isolated storage.
Both last month and this month, Hanu Kommalapati covered building a line-of-business application with Silverlight, which you can read about in his articles "Silverlight: Build Line-Of-Business Enterprise Apps with Silverlight, Part 1" and "Silverlight: Build Line-Of-Business Enterprise Apps with Silverlight, Part 2."

Send your questions and comments for John to mmdata@microsoft.com.

John Papa (johnpapa.net) is a Senior Consultant with ASPSOFT and a baseball fan who spends summer nights rooting for the Yankees with his family. John, a C# MVP and INETA speaker, has authored several books and is now working on his latest, titled Data-Driven Services with Silverlight 2. He often speaks at conferences such as DevConnections and VSLive.


Code download available from the MSDN Code Gallery
Browse the Code Online

Cutting Edge
Managing Dynamic Content Delivery In Silverlight, Part 2
Dino Esposito
Contents
Motivation for a Permanent Cache
Foundations of Isolated Storage
Isolated Storage API
Building a Permanent Cache of Packages
Expiration Policies
Putting It All Together
Some Final Notes
Last month I discussed how to serve a Silverlight application some dynamically generated content at startup and even in an on-demand scenario. Many examples exist out there to show you how to use the WebClient class and its asynchronous call model to download a URL-based resource. In particular, I focused on what it takes to download a XAP package that includes XAML and managed code (see the January 2009 installment of Cutting Edge.
The basic idea is that you download a zipped stream and then extract any assembly you need. Next, you instantiate any class you need that is contained in the assembly. The class is a XAML user control—an entire piece of a XAML visual tree—that you can then append to any placeholders in the current XAML document object model (DOM).
To the browser, any downloaded XAP resources are totally indistinguishable from any other type of resource. Hence, the browser caches a XAP package just as any other downloaded resource. This built-in mechanism offers a first level of optimization that saves you repeated round-trips to get the same package over and over again. The WebClient class—the core of the downloader component discussed in last month's column—is based on the browser's connectivity engine and doesn't download resources available locally and not expired yet.
At the end of the day, you have a downloader component to defer the loading of any external packages you may need. In addition, you also have free caching capabilities for any dynamically downloaded resources. Among other things, you might like to add a permanent local cache of packages that is also able to deal with dynamic changes to the visual tree. Let's see how to achieve this.

Motivation for a Permanent Cache
When it comes to dynamic Silverlight content, there are two main aspects over which you might want to gain some more control.
The first is the expiration policy of the downloaded content. You might want to control exactly when a given package expires and needs to be downloaded again. Furthermore, you might want to tie the expiration to some external event such as a particular user action or changes to other cached resources. If you know how the ASP.NET Cache works, you know what I mean.
The ASP.NET Cache, in fact, allows you to cache data and give each cached item its own expiration policy—based on file changes, date/time, or even changes to other cached items. A similar engine doesn't exist in Silverlight 2, but large, dynamic, and highly customizable applications would gain significant benefits from it.
The second aspect of standard Silverlight resource caching that you might want to change regards the exposure of your packages to the user's activity. In other words, any XAP package saved in the browser's cache is at the mercy of the user. If the user clears the cache acting on the browser's interface, all of your XAP packages will be inevitably lost.
An application-managed permanent cache addresses both issues. XAP packages stored in this permanent cache would not be affected by the user clearing the browser's cache. To permanently store Silverlight XAP packages, you need to gain access to the local file system. For security reasons, Silverlight doesn't let applications access the entire local file system. However, the Isolated Storage API is here to help. For more on Silverlight security, see "CLR Inside Out: Security in Silverlight 2."

Foundations of Isolated Storage
Isolated storage was not specifically created for Silverlight. Isolated storage has been part of the Microsoft .NET Framework since version 1.0. Aimed at partially trusted applications, isolated storage enables such apps to store data on the local computer in full respect of any ongoing security policy. A classic fully trusted .NET application probably has no need to ever go through the isolated storage layer to save its own data, but for a partially trusted application, isolated storage is the only option for saving data on the client.
From a Silverlight perspective, isolated storage is a powerful tool and the only possible way to persist relatively large chunks of data in a cross-browser manner and without any of the restrictions that affect, for example, HTTP cookies. It is important to understand this point: in Silverlight, isolated storage is the only possibility you have to cache data on the local machine. If a Silverlight application needs to save some data—any kind of data—locally, then it can only do so through isolated storage. In addition, with isolated storage each application can keep its own data isolated from any other applications or from any other applications outside the site.
If you want a general, .NET-oriented introduction to isolated storage and its most common usage scenarios, you should read the .NET Framework Developer's Guide to Isolated Storage. The article mentions a couple of scenarios where using isolated storage is not appropriate. In particular, the guidelines say that you should not be using isolated storage to store sensitive data, code, or configuration settings (other than user preferences). Such guidelines stem from a general security awareness and do not necessarily imply any hazards inherent in the use of isolated storage.
So, can you safely store in XAP packages you download into Silverlight isolated storage? In Silverlight, unlike in the desktop CLR, any piece of executable code is untrusted by default and is not allowed to invoke critical methods or elevate privileges of the calling stack. In Silverlight, whatever code you store for later execution will be unable to do anything dangerous. This is no more risky than executing any other piece of Silverlight code. By building a permanent cache of Silverlight packages, you end up storing locally a segment of the Silverlight application you are consciously executing.
In Silverlight, the role of isolated storage is analogous—as far as persistence is concerned—to the role of HTTP cookies in classic Web applications. In Silverlight, you should look at isolated storage as a set of larger cookies that can contain any sort of data, including executable code. In this case, though, the Silverlight core CLR provides protection. According to the Silverlight security model, in fact, the core CLR will throw an exception any time the application code wants to execute critical methods. Unlike HTTP cookies, isolated storage in Silverlight is not linked to network I/O and no content is transmitted with requests.
Data in isolated storage is isolated by application and no other Silverlight application can access the store. The data is stored on the local file system, however, so an administrator of the machine is certainly able to access it.
Again, the overall model is not really different from what happens with HTTP cookies. An administrator can always locate and even alter the content of cookies. If it's worthwhile in your context, you can use encryption to add another level of data protection.
If you're still worried about having some downloaded executable code lying around your machine, you should refresh your understanding of the Silverlight security model. In brief, the Silverlight core CLR throws an exception any time application code attempts to execute a critical method. In the Silverlight Base Class Library (BCL), methods and classes that perform operations requiring high privileges are marked with a special SecurityCritical attribute. Note that this is the case with most of the content of the System.IO namespace.
The Silverlight security acknowledges that some platform classes may need to place safe calls to critical methods. Such classes and methods are then marked with the SecuritySafeCritical attribute. This is the case with classes in the System.IO.IsolatedStorage API (see Figure 1). The key point about Silverlight security is that no piece of application code can ever be marked with the SecurityCritical or SecuritySafeCritical attribute. This attribute is reserved for classes in assemblies digitally signed by Microsoft and loaded into memory from the Silverlight installation directory.
Figure 1 Browsing inside the Isolated Storage API
As you can see, even in the very unfortunate (and unlikely) circumstance that some malicious guy penetrates your computer and replaces downloaded Silverlight content, the damage is limited to regular operations executable in transparent mode.

Isolated Storage API
The Silverlight BCL comes with its own implementation of the isolated storage that is tailor-made for the Web scenario. Isolated storage provides access to a subtree of the whole local file system, and no method or property ever allows running code to figure out where the file store is physically located on the user machine. A Silverlight application is not allowed to use absolute file system paths through isolated storage. Likewise, drive information is not available and is unsupported and the same applies to relative paths that contain ellipsis like this one:
\..\..\myfile.txt
The isolated storage subtree is rooted in a folder located under the current user path. For example, on Windows Vista, the root of the isolated storage folder is located under the Users directory.
A Silverlight application gains access to the application-specific isolated storage entry point through a method call:
using (IsolatedStorageFile iso =
       IsolatedStorageFile.GetUserStoreForApplication())
{
  ...
}
The static method GetUserStoreForApplication returns the token you use for any further access to the isolated storage. When the first call to GetUserStoreForApplication is made, the application-specific subtree is created, if it doesn't exist already.
The Silverlight isolated storage API supplies classes to work with files and directories within the protected file system subtree. Happily, the list of classes you need to know is fairly short; you'll find them listed in Figure 2.
In the IsolatedStorageFile class, there are a number of methods to create and delete files and directories, to check for the presence of files and directories, and to read and write new files. You employ streams to work with files. Optionally, you can wrap up streams with stream readers, which are much more comfortable objects to work with. Figure 3 shows a brief example of how to create an isolated storage file using a stream writer.
Figure 3 Creating an Isolated Storage File
using (IsolatedStorageFile iso =
      IsolatedStorageFile.GetUserStoreForApplication())
{
    // Open or create the low level stream
    IsolatedStorageFileStream fileStream;
    fileStream = new IsolatedStorageFileStream(fileName,
        FileMode.OpenOrCreate, iso);

    // Encapsulate the raw stream in a more convenient writer
    StreamWriter writer = new StreamWriter(stream);

    // Write some data
    writer.Write(DateTime.Now.ToString());

    // Clean up
    writer.Close();
    stream.Close();
}
Once you wrap a low-level stream in a more convenient stream writer or reader, the code you use to write (or read) some data is nearly identical to the code you would use in a classic .NET application. Let's see how to take advantage of the isolated storage API to save any downloaded XAP package locally and reload it later.

Building a Permanent Cache of Packages
In last month's column, I used a downloader wrapper class to hide some of the boilerplate code you need to download a XAP package and extract assemblies and other resources. The Downloader class, however, is not just a helper class. Conceptually, it represents a nontrivial piece of logic that you might want to isolate from the rest of the application code for a number of reasons.
The first reason that springs to mind is testability. By exposing the functionality of a downloader component through an interface, you gain the possibility of quickly and effectively mocking the downloader for testing purposes. In addition, an interface represents the tool you leverage to replace a simpler downloader with a more sophisticated one that, perhaps, just happens to support package caching. Figure 4 shows the architecture of the design you should aim for.
Figure 4 The Downloader Component and the Rest of the Application
Figure 5 Extracting an Interface

In last month's source code, the Downloader class was a monolithic piece of code. For a more flexible design, let's extract an interface out of it. As Figure 5 shows, Visual Studio has a context menu that, although not as rich as that of commercial refactoring tools, does provide some help with the extraction of an interface out of a class.
Now that the core of your Silverlight application talks to the IDownloader interface, all the logic for package caching must go only inside the actual downloader class:
interface IDownloader
{
    void LoadPackage(string xapUrl, string asm, string cls);
    event EventHandler<Samples.XapEventArgs> XapDownloaded;
}
In particular, the LoadPackage method will be rewritten to incorporate the logic that would check for the existence of the specified XAP package within the isolated storage and download it from the Internet otherwise. Figure 6 shows a large section of the code for the Downloader class. The method first attempts to get the stream for the XAP package from the internal cache. If this attempt fails, the method proceeds and downloads the package from the host server. (This is just what I discussed in detail last month.)
Figure 6 Cache Support for the Downloader Component
public void LoadPackage(string xap, string asm, string cls)
{
    // Cache data within the class
    Initialize(xap, asm, cls, PackageContent.ClassFromAssembly);

    // Have a look in the cache
    Stream xapStream = LookupCacheForPackage();
    if (xapStream == null)
        StartDownload();
    else
    {
        // Process and extract resources
        FindClassFromAssembly(xapStream);
    }
}

protected Stream LookupCacheForPackage()
{
    // Look up the XAP package for the assembly.
    // Assuming the XAP URL is a file name with no HTTP information
    string xapFile = m_data.XapName;

    return DownloadCache.Load(xapFile);
}

protected void StartDownload()
{
    Uri address = new Uri(m_data.XapName, UriKind.RelativeOrAbsolute);
    WebClient client = new WebClient();

    switch (m_data.ActionRequired)
    {
        case PackageContent.ClassFromAssembly:
            client.OpenReadCompleted +=
                new OpenReadCompletedEventHandler(OnCompleted);
            break;
        default:
            return;
    }
    client.OpenReadAsync(address);
}

private void OnCompleted(object sender, OpenReadCompletedEventArgs e)
{
    // Handler registered at the application level?
    if (XapDownloaded == null)
        return;

    if (e.Error != null)
        return;

    // Save to the cache
    DownloadCache.Add(m_data.XapName, e.Result);

    // Process and extract resources
    FindClassFromAssembly(e.Result);
}

private void FindClassFromAssembly(Stream content)
{
    // Load a particular assembly from XAP
    Assembly a = GetAssemblyFromPackage(m_data.AssemblyName, content);

    // Get an instance of the specified user control class
    object page = a.CreateInstance(m_data.ClassName);

    // Fire the event
    XapEventArgs args = new XapEventArgs();
    args.DownloadedContent = page as UserControl;
    XapDownloaded(this, args);
}
In Silverlight, downloading is an asynchronous process, so the internal method StartDownload raises a "completed" event when the package is entirely available on the client. The event handler first saves the content of the XAP package stream to a local file and then extracts resources from it. Note that in the sample code I'm only extracting assemblies; in a more general component you might want to extend caching capabilities to any other type of resource such as XAML for animations, images, or other auxiliary files.
The LoadPackage method in the Downloader class is used to download Silverlightuser controls to insert in the current XAML tree. Because the XAP package is a multi-file container, you have to specify which assembly contains the user control and the class name. The code in Figure 6 simply extracts the specified assembly from the package, loads it into the current AppDomain, and then creates an instance of the specified contained class.
What if the assembly has some dependencies? The code in Figure 6 just doesn't cover this scenario. As a result, if the assembly passed as an argument to LoadPackage has a dependency on another assembly (even in the same XAP package), you get an exception as soon as the execution flow reaches a class in the dependency assembly. The point is that all assemblies in the package should be loaded in memory. For this to happen, you need to access the manifest file, read about deployed assemblies, and process them. Figure 7 shows how to load into memory all assemblies referenced in the manifest file.
Figure 7 Loading All Assemblies in the Manifest
private Assembly GetAssemblyFromPackage(
     string assemblyName, Stream xapStream)
{
    // Local variables
    StreamResourceInfo resPackage = null;
    StreamResourceInfo resAssembly = null;

    // Initialize
    Uri assemblyUri = new Uri(assemblyName, UriKind.Relative);
    resPackage = new StreamResourceInfo(xapStream, null);
    resAssembly = Application.GetResourceStream(
                              resPackage, assemblyUri);

    // Extract the primary assembly and load into the AppDomain
    AssemblyPart part = new AssemblyPart();
    Assembly a = part.Load(resAssembly.Stream);

    // Load other assemblies (dependencies) from manifest
    Uri manifestUri = new Uri("AppManifest.xaml", UriKind.Relative);
    Stream manifestStream = Application.GetResourceStream(
        resPackage, manifestUri).Stream;
    string manifest = new StreamReader(manifestStream).ReadToEnd();

    // Parse the manifest to get the list of referenced assemblies
    List<AssemblyPart> parts = ManifestHelper.GetDeploymentParts(manifest);

    foreach (AssemblyPart ap in parts)
    {
        // Skip over primary assembly (already processed)
        if (!ap.Source.ToLower().Equals(assemblyName))
        {
            StreamResourceInfo sri = null;
            sri = Application.GetResourceStream(
                resPackage, new Uri(ap.Source, UriKind.Relative));
            ap.Load(sri.Stream);
        }
    }

    // Close stream and returns
    xapStream.Close();
    return a;
}
The manifest file is an XML file, as shown here:
<Deployment EntryPointAssembly="More" EntryPointType="More.App"
            RuntimeVersion="2.0.31005.0">
  <Deployment.Parts>
    <AssemblyPart x:Name="More" Source="More.dll" />
    <AssemblyPart x:Name="TestLib" Source="TestLib.dll" />
  </Deployment.Parts>
</Deployment>
To parse this file, you can use LINQ-to-XML. The source code contains a sample ManifestHelper class with a method that returns a list of AssemblyPart objects (see Figure 8). It is worth noting that in Beta versions of Silverlight 2, you could use the XamlReader class to parse the manifest file to a Deployment object:
// This code throws in Silverlight 2 RTM
Deployment deploy = XamlReader.Load(manifest) as Deployment;
Figure 8 Parsing the Manifest Using LINQ-to-XML
public class ManifestHelper
{
   public static List<AssemblyPart> GetDeploymentParts(string manifest)
   {
      XElement deploymentRoot = XDocument.Parse(manifest).Root;
      List<AssemblyPart> parts =
          (from n in deploymentRoot.Descendants().Elements()
           select new AssemblyPart() {
                Source = n.Attribute("Source").Value }
          ).ToList();

          return parts;
   }
}
In the release version, the Deployment object has been transformed into a singleton and subsequently it can't be used to load assemblies dynamically. The XML in the manifest therefore must be parsed manually.

Expiration Policies
As implemented thus far, the assembly cache is permanent and there's no way for the user to get updated packages. The only possible workaround is to have a machine administrator locate isolated storage files for the application and delete them manually through Windows Explorer.
Let's see what it would take to add a simple expiration policy that makes any cached XAP file obsolete after a given amount of time after download. The proper place to set an expiration policy is the DownloadCache class, as in Figure 6. When you add a XAP file to the cache, you need to store some information about the download time. When you access the cache to pick up a package, you should verify whether the package is expired (see Figure 9).
Figure 9 Test for Expiration
public bool IsExpired(string xapFile)
{
    bool expired = true;
    if (m_ItemsIndex.ContainsKey(xapFile))
    {
        DateTime dt = (DateTime)m_ItemsIndex[xapFile];

        // Expires after 1 hour
        expired = dt.AddSeconds(3600) < DateTime.Now;
        if (expired)
            Remove(xapFile);
    }

    return expired;
}
The implementation of such an apparently simple algorithm is hindered by one notable fact. In Silverlight, you have no way to access file attributes like last update or creation time. This means that you are responsible for managing time information. In other words, when you add a XAP to the cache you also create an entry in some custom and persistent dictionary that tracks when the package was downloaded. Needless to say, this information has to be persisted in some way to the isolated storage. Figure 10 illustrates this concept.
Figure 10 Persisting Download Details to Isolated Storage
public static Stream Load(string file)
{
    IsolatedStorageFile iso;
    iso = IsolatedStorageFile.GetUserStoreForApplication();

    if (!iso.FileExists(file))
    {
        iso.Dispose();
        return null;
    }

    // Check some expiration policy
    CacheIndex m_Index = new CacheIndex();
    if (!m_Index.IsExpired(file))
        return iso.OpenFile(file, FileMode.Open);

    // Force reload
    iso.Dispose();
    return null;
}
CacheIndex is a helper class that uses the Silverlight native application settings API to persist a dictionary of XAP names and download times to the isolated storage. The m_ItemIndex member is a plain Dictionary object instantiated in CacheIndex constructor, as shown in Figure 11.
Figure 11 The CacheIndex Class
public class CacheIndex
{
    private const string XAPCACHENAME = "XapCache";
    private Dictionary<string, object> m_ItemsIndex;
    public CacheIndex()
    {
      IsolatedStorageSettings iss;
      iss = IsolatedStorageSettings.ApplicationSettings;
      if (iss.Contains(XAPCACHENAME))
         m_ItemsIndex = iss[XAPCACHENAME] as Dictionary<string, object>;
      else
      {
         m_ItemsIndex = new Dictionary<string, object>();
         iss[XAPCACHENAME] = m_ItemsIndex;
         iss.Save();
      }
   }
  ...
}
ApplicationSettings is a very nice feature of Silverlight 2. It basically consists of a string/object dictionary and is automatically read from storage upon application loading and saved back to the storage upon shutdown. Any (serializable) object you add to the dictionary is automatically persisted.
By creating a XAPCACHENAME entry in the dictionary, you arrange to persist the content of the XAP dictionary. The XAP dictionary contains one entry for each downloaded package paired with the time of the download, as you see in Figure 12. Note that the ApplicationSettings API also offers you a Save method to force persistence before application shutdown.
Figure 12 Adding Download Information
public void Add(string xapFile)
{
    m_ItemsIndex[xapFile] = DateTime.Now;
    IsolatedStorageSettings iss;
    iss = IsolatedStorageSettings.ApplicationSettings;
    iss.Save();
}

public void Remove(string xapFile)
{
    m_ItemsIndex.Remove(xapFile);
    IsolatedStorageSettings iss;
    iss = IsolatedStorageSettings.ApplicationSettings;
    iss.Save();
}

Putting It All Together
All the changes and details discussed so far occur behind the curtain of a public class—the Downloader class. By incorporating this class in your Silverlight application, you gain multiple levels of caching in a single shot. With proper coding you can cache downloaded user controls for the duration of the application session. If you download content that goes to, say, a tab item, you can then hide and show the tab item repeatedly without the need to download the package over and over again.
If you download through the WebClient class (as in last month's column), you pass through the browser engine and get browser-level caching. Any downloaded XAP package lives on the user's machine until the user clears the browser cache. Finally, the Downloader class you get with this column supports a permanent cache through isolated storage. This means that any time you download via WebClient, the XAP package is also saved to the local storage.
The Downloader class, however, also offers to pick up the XAP file from the storage if a valid package can be found. Note that this works across application sessions. You download the package once, you work and then you shut down the application. When you resume, the package is reloaded from the storage if it has not expired. What if the package expires? In this case, the downloader passes through WebClient. But at this point, WebClient might just return a previously browser-cached copy of the same package.
This is by design. To bypass the browser-level caching, you have to disable it in the original HTTP request, as discussed last month. You must cache properties at the page level or get the package through an HTTP handler where you can set expiration policies more precisely without affecting other resources that go with the page.

Some Final Notes
Caching the XAP package doesn't mean caching individual resources such as DLLs, XAML animations, or multimedia content. In the current implementation, resources are extracted from the XAP package every time they are used. However, this is an aspect you can further improve on with the Downloader class. Likewise, the package delivers a user control but doesn't track changes the user can force on its user interface. Tracking dynamic changes to the XAML tree is another topic and deserves an article of its own.
There are two ways to access a Silverlight private file system on the local user's machine. In all the code snippets for this column, I used the method GetUserStoreForApplication on the IsolatedStorageFile class. This method returns a token to access a section of the file system that is isolated per application, meaning that all and only the assemblies associated with an application will use the same store. You can also select a store and share it across all applications hosted on the same site. In this case, you get the token through the method GetUserStoreForSite.
Note too that local storage can be administered through the Silverlight configuration dialog box (right-click on a Silverlight app) and even disabled completely. In this case, when attempting to get the token, an exception will be raised. A disk quota also applies to local storage on a per-domain basis; its default value is 1MB. Remember this when planning for a permanent Silverlight cache.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an architect at IDesign and the co-author of Microsoft .NET: Architecting Applications for the Enterprise (Microsoft Press, 2008). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Patterns in Practice
Convention Over Configuration
Jeremy Miller
Contents
The Impact of Language Innovation
Say It Once and Only Once
Sensible Defaults
Convention over Configuration
Next Steps
Have you ever thought about how much time on your project is spent on the core problems versus the time you spend wrestling with purely technical concerns? Some may say that the most important goal of all new software technologies and techniques is to reduce the distance between the developer's intent for the software and the realization of that intent in code. And the industry has continuously raised the level of abstraction to allow developers to spend more time delivering functionality and less time writing low-level infrastructure—but there's still a long way to go.
Think about this for a minute. If you were to show your code to your business representatives—assuming that they were actually willing to read the code with you—how much of that code would they care about? Those business representatives will probably only care about the code that expresses the business functionality of the system. That code is the "essence" of the system. On the other hand, they probably wouldn't have the slightest interest in code such as type declarations, configuration settings, try/catch blocks, and generic constraints. That code is the infrastructure, or "ceremony," that you, the developer, have to go through just to ship code.
In previous installments of this column I've largely explored fundamental design concepts and principles, most of which have been part of the design canon for quite some time. This time around, I'd like to look at some newer techniques that you can adopt to reduce the amount of ceremony in your code. Some of these concepts may be new to you, but I believe that all of this will be part of the .NET mainstream within a few years.

The Impact of Language Innovation
The first factor I'd like to consider is the choice of programming language and how you use programming languages. To illustrate the impact of a programming language on the amount of ceremony in the code, let's take a little detour into the musty pages of history.
Earlier in this decade I was building a large system in Visual Basic 6.0. Each method would look something like what you see in Figure 1. Every single bit of that code was ceremony.
Figure 1 Standing on Ceremony
Sub FileOperations()
    On Error Goto ErrorHandler

    Dim A as AType
    Dim B as AnotherType

    ' Put some code here

    Set A = Nothing
    Set B = Nothing
    ErrorHandler:
        Err.Raise Err.Number, _
            "FileOperations" & vbNewLine & Err.Source, _
            Err.Description, _
            Err.HelpFile, _
            Err.HelpContext
Exit Sub
I used a fair amount of boilerplate on each method to create an equivalent of a stack trace for easier debugging. I also had to dereference variables inside the method (Set A = Nothing) to release those objects. I had a strict standard for code reviews just to verify that the error handling and object cleanup was coded correctly on each method. The actual code, the essence of the system, floated somewhere in the middle of all that ceremony.
Flash forward to today and contemporary programming languages such as C# or Visual Basic .NET. Today, garbage collection eliminated most of the explicit memory cleanup that you used to face in Visual Basic 6.0 programming. Stack traces in exceptions are built into the Microsoft .NET Framework itself, so you no longer have to do that for yourself. If you think about all the rote boilerplate code that was eliminated when you moved to the .NET Framework, you can say that the .NET-compliant languages are more productive and readable than Visual Basic 6.0 because of the reduction in ceremony code.
The .NET Framework was a big jump, but the evolution in languages isn't finished yet. Let's consider a simple property in C# implemented the classical way:
public class ClassWithProperty {
  // Higher Ceremony
  private string _name;
  public string Name {
    get { return _name; }
    set { _name = value; }
  }
}
The essence of this code is simply that the class ClassWithProperty has a string property named Name. Let's fast forward to C# 3.0 and do the same thing with an auto property:
public class ClassWithProperty {
  // Lower Ceremony
  public string Name { get; set; }
}
This code has the exact same intent as the classic style property, but it required noticeably less "compiler noise" code.
Generally, though, software developers do not have absolute control over programming languages. While I definitely think that we should take advantage of new programming language innovations or even alternative languages, it's time to talk about design ideas that you can use today with mainstream C# and Visual Basic.

Domain-Centric Validation
The .NET Framework makes it nearly trivial to add declarative field-level validation in the user interface with tools such as the ASP.NET Validator controls. All the same, I think it's advantageous to place validation logic on the actual domain model classes, or at least close by in domain services, for these reasons:
  1. Validation logic is a business logic concern, and I'd prefer that all business logic be contained in the business logic classes.
  2. Putting the validation logic into the domain model or domain services disconnected from the user interface potentially reduces duplication across screens and allows the same validation logic to be exercised in non-user interface services (Web services, for example) exposed by the application (Saying It Once and Only Once, yet again).
  3. It is far easier to write unit and acceptance tests against validation logic in the model than it is to test that same logic implemented as part of the user interface.
Say It Once and Only Once
What happens when you find out midway through a project that something about the definition of a single data field needs to be changed? In far too many cases, that small change to the database will ripple through your application as you make an analogous change to various parts of the middle tier, data access code, and even into the user interface layer in order to accommodate a single logical change.
At a past employer, we called that rippling effect from small data field changes the Wormhole Anti-Pattern. You want to get away from the need for a small change in one place being teleported all through the layers. You can slow down the wormhole effect by reducing unnecessary layering, and that's the first step. The harder but more rewarding step is to find a way to say it once and only once.
The Say It Once and Only Once Principle states that there should only be a single authoritative source for any fact or policy in the system as a whole. Let's take the example of building a Web application with create, read, update, and delete (CRUD) functionality. This kind of system is largely concerned with editing and storing data fields. These fields need to be edited in screens, validated on the server, stored in the database, and hopefully validated on the client as well for a better user experience—but you'd like to specify that "this field is required and/or this field must be no more than 50 characters" in the code only once.
There are a couple of different approaches you could take. You could make the database schema the master definition and generate both middle-tier and user-presentation code from the schema. You could also define the data fields in some sort of external metadata storage such as an XML file, then use code generation to build the database schema, any middle-tier objects, and the user interface screens. Personally, I'm not a fan of large-scale code generation, so my team chose another direction.
We generally design the domain model classes first, and we consider the validation logic to be the responsibility of the domain model entity classes. For simple validation rules such as required fields and maximum string length rules, we decorate properties with validation attributes such as the Address class shown in Figure 2.
Figure 2 Using Validation Attributes
public class Address : DomainEntity {
  [Required, MaximumStringLength(250)]
  public string Address1 { get; set; }

  [Required, MaximumStringLength(250)]
  public string City { get; set; }

  [Required]
  public string StateOrProvince { get; set; }

  [Required, MaximumStringLength(100)]
  public string Country { get; set; }

  [Required, MaximumStringLength(50)]
  public string PostalCode { get; set; }

  public string TimeZone { get; set; }
}
Using attributes is a simple and fairly common technique in order to specify validation rules. You are also able to say that since the validation rules are expressed declaratively instead of implemented with imperative code, you have passed the essence-versus-ceremony test.
Now, though, you need to replicate the rules for required fields and maximum string lengths to the database. In my team's case, we use NHibernate for our persistence mechanism. One of the powerful capabilities of NHibernate is to generate data definition language (DDL) code from the NHibernate mappings that you can then use to create the database schema and keep it in sync with the domain model (this strategy obviously works best in new projects). In order for this strategy of generating the database from the domain model to be useful, it was necessary for us to add additional information to the NHibernate mappings to mark non-null fields and specify string lengths.
We used the new Fluent NHibernate mechanism to define our object mappings. In our setup code for Fluent NHibernate, we established automatic conventions in the mapping by teaching Fluent NHibernate how to handle the presence of the [Required] and [MaximumStringLength] attributes in our model classes with the code shown in Figure 3.
Figure 3 Handling Attributes in NHibernate
public class MyPersistenceModel : PersistenceModel {
  public MyPersistenceModel() {
    // If a property is marked with the [Required]
    // attribute, make the corresponding column in
    // the database "NOT NULL"
    Conventions.ForAttribute<RequiredAttribute>((att, prop) => {
      if (prop.ParentIsRequired) {
        prop.SetAttribute("not-null", "true");
      }
    });

    // Uses the value from the [MaximumStringLength]
    // attribute on a property to set the length of
    // a string column in the database
    Conventions.ForAttribute<MaximumStringLengthAttribute>((att, prop) => {
      prop.SetAttribute("length", att.Length.ToString());
    });
  }
}
These conventions will now be applied to all mappings in the project. For the Address class, I simply tell Fluent NHibernate which properties are persisted:
public class AddressMap : DomainMap<Address> {
  public AddressMap() {
    Map(a => a.Address1);
    Map(a => a.City);
    Map(a => a.TimeZone);
    Map(a => a.StateOrProvince);
    Map(a => a.Country);
    Map(a => a.PostalCode);
  }
}
Now that I've taught Fluent NHibernate about the validation attributes, I can generate the DDL for the Address table (see Figure 4). Note in the SQL in Figure 4 that the string lengths match the definition from the [MaximumStringLength] attributes on the Address class. Likewise, the NULL / NOT NULL values follow from the [Required] attributes on the Address class.
Figure 4 Generating DDL Code
CREATE TABLE [dbo].[Address](
  [id] [bigint] IDENTITY(1,1) NOT NULL,
  [StateOrProvince] [nvarchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
  [Country] [nvarchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
  [PostalCode] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
  [TimeZone] [nvarchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
  [Address1] [nvarchar](250) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
  [Address2] [nvarchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
  [City] [nvarchar](250) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
PRIMARY KEY CLUSTERED
(
  [id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
We've invested in some infrastructure that derives the database structure from the validation attributes on the Domain Model objects, but we still have the matter of client-side validation and client-side appearance. We'd like to perform the simple input validations inside the browser and mark the required field elements in some way for a better user experience.
The solution that my team arrived at was to make the HTML rendering of input elements aware of the validation attributes. (For context, we're using the ASP.NET Model View Controller, or MVC, Framework Beta1 with the Web Forms engine as our view engine.) Figure 5 shows what the markup might look like for an Address View in our system.
Figure 5 Address View Markup
<div class="formlayout_body">
  <p><%= this.TextBoxFor(m => m.Address1).Width(300)%></p>
  <p><%= this.TextBoxFor(m => m.Address2).Width(300) %></p>
  <p>
    <%= this.TextBoxFor(m => m.City).Width(140) %>
    <%= this.DropDownListFor(m =>
        m.StateOrProvince).FillWith(m => m.StateOrProvinceList) %>
  </p>
  <p><%= this.TextBoxFor(m => m.PostalCode).Width(80) %></p>
  <p><%= this.DropDownListFor(m =>
         m.Country).FillWith(m => m.CountryList) %></p>
  <p><%= this.DropDownListFor(m =>
         m.TimeZone).FillWith(m => m.DovetailTimeZoneList) %></p>
</div>
TextBoxFor and DropDownListFor are little HTML helpers in the common base view that are used for all views in our MVC architecture. The signature of TextBoxFor is shown here:
public static TextBoxExpression<TModel> TextBoxFor< TModel >(
  this IViewWithModel< TModel > viewPage,
  Expression<Func< TModel, object>> expression)

  where TModel : class {
    return new TextBoxExpression< TModel >(
    viewPage.Model, expression);
  }
The important thing to note in this code is that the input argument is an Expression (Expression<Func<TModel, object>> to be precise). While constructing the actual HTML for the textbox, the TextBoxExpression class will:
  1. Parse the Expression and find the exact PropertyInfo object for the bound property.
  2. Interrogate that PropertyInfo for the existence of the validation attributes.
  3. Render the HTML for the textbox accordingly.
We simply added a class called required to all HTML elements that are bound to properties marked with the [Required] attribute. Likewise, if we found a [MaximumStringAttribute] on a bound property, we set the maxlength attribute of the HTML textbox to match the attribute and limit the user input to allowable lengths. The resulting HTML looks something like this:
<p><label>Address:</label>
<input type="text" name="Address1" value=""
       maxlength="250" style="width: 300px;"
       class="required textinput" /></p>
The appearance of required fields is easy to control simply by editing the appearance of the CSS class required (we set the color of required fields on the screen to a light blue). We did the actual client-side validation with the jQuery Validation plug-in, which, conveniently enough, simply looks for the existence of the required class on input elements. The maximum length of a text element is enforced simply by setting the maxlength attribute on the input elements.
By no means was this a complete implementation. Building the actual implementation over time wasn't that difficult. The hard part was thinking through ways to eliminate repetitive metadata and coding in the multiple layers. I'm sure that many teams will not be keen on the way that my team generated the database from the object model, but that's OK because my real goal is just to give you some ideas about how you might use the Say it Once and Only Once Principal to streamline your own development efforts.

Sensible Defaults
In the previous section, I showed a very small example (via an AddressMap class) of expressing object relation mapping (ORM) with Fluent NHibernate. Here's a slightly more complicated example that expresses a reference from a Site class to Address objects:
public SiteMap() {
  // Map the simple properties
  // The Site object has an Address property called PrimaryAddress
  // The code below sets up the mapping for the reference between
  // Site and the Address class
  References(s => s.PrimaryAddress).Cascade.All();
  References(s => s.BillToAddress).Cascade.All();
  References(s => s.ShipToAddress).Cascade.All();
}
When you configure ORM tools, you generally have to:
  1. Specify the table name to which an Entity class maps.
  2. Specify the primary key field of an Entity and usually also specify some sort of strategy for assigning the primary key values. (Is it an auto number/database sequence? Or does the system assign the primary key values? Or do you use GUIDs for the primary key?)
  3. Map object properties or fields to table columns.
  4. When making a "to one" relationship from one object to another (such as the mapping from Site to Address), you need to specify the foreign key column that the ORM tool can use to join parent and child records.
All of that is generally tedious work. It's ceremony that you have to follow to get the ORM to persist objects. Fortunately, you can eliminate some of the tedium by embedding some Sensible Defaults in our code (note the presence of capital letters).
You might notice in the mapping examples that I did not specify a table name, a primary key strategy, or any foreign key field names. In my team's Fluent NHibernate mapping superclass, we set some defaults for our mapping:
public abstract class DomainMap<T> : ClassMap<T>,
  IDomainMap where T : DomainEntity {

  protected DomainMap() {
    // For every DomainEntity class, use the Id property
    // as the Primary Key / Object Identifier
    // and use an Identity column in SQL Server,
    // or an Oracle Sequence
    UseIdentityForKey(x => x.Id, "id");
    WithTable(typeof(T).Name);
  }
}
Opinionated Software
You might notice that I described the adoption of conventions as "restrictive." Part of the philosophy behind convention over configuration is to make "opinionated software" that creates artificial constraints on the design.
An opinionated framework expects developers to do things in a certain way almost to the point of eliminating flexibility. Proponents of opinionated software feel that these constraints make development more efficient by removing decisions from the developers and promoting consistency.
One opinion used by my team is that all domain model classes are completely identified by a single long property called Id:
public virtual long Id { get; set; }
It's a simple rule, but it has had a number of profound impacts on design. Because all entity classes are identified in the same way, you have been able to use a single repository class instead of writing specialized repository classes for each top-level entity. In the same vein, URL handling in a Web application is consistent across entity classes without your having to register special routing rules for each entity.
Following this opinion reduces the cost of infrastructure for adding a new entity type. The downside of this approach is that it would be very difficult to accommodate a natural key or even a composite key or to use a GUID for an object identifier. That's not an issue for my team, but it could easily block another team from adopting our opinion.
Now, how do you enforce these opinions? The first step is simply creating a common understanding and agreement within a team about these opinions. Informed developers stand the best chance of effectively using these opinionated design choices to their advantage. Likewise, convention over configuration can be a near disaster if the developers aren't familiar with the existing conventions or the conventions are confusing.
You might also consider using a static code analysis tool as part of your continuous integration build to automatically enforce your project's conventions.
In this code we are setting a policy that all classes that subclass DomainEntity will be identified by the Id property that is assigned with the identity strategy. The table name is assumed to be the same as the name of the class. Now, we can always override these choices on a per-class basis, but we've rarely had to do this (a class named User had to be mapped to a table named Users just to avoid a conflict with a reserved word in SQL Server). In the same way, Fluent NHibernate assumes a foreign key name based on the property name that references another class.
Granted, this isn't saving many lines of code per mapping class, but it makes the parts of the mapping that do vary easier to read by reducing the overall noise code in the mapping.

Convention over Configuration
Software developers have sought to gain more productivity and make systems more dynamic by moving behavior out of imperative code and into declarative XML configuration. Many developers felt that the proliferation of XML configuration went too far and was becoming a harmful practice. The strategy of defaults over explicit configuration is also known as convention over configuration.
Convention over configuration is a design philosophy and technique that seeks to apply defaults that can be implied from the structure of the code instead of requiring explicit code. The idea is to simplify development by allowing the developer to worry only about the unconventional parts of the application and architecture.
Right now, many people are eagerly playing with the ASP.NET MVC Framework and experimenting with different ways to use it. In the MVC model of Web development there are a couple of sources of repetitive code that might be a great opportunity to apply conventions over configuration.
Here are the five steps in the basic flow of a single request in the MVC model:
  1. Receive a URL from the client. The routing subsystem will parse the URL and determine the name of the controller that handles this URL.
  2. From the controller name determined by the routing subsystem, build or locate the proper controller object.
  3. Invoke the correct controller method.
  4. Select the proper view, and marshal the model data generated from the controller method to this view.
  5. Render the view.
Out of the box, there is some repetitive ceremony in building Web pages with the ASP.NET MVC Framework that you can mitigate by adopting some restrictive conventions.
The first task is to connect an incoming URL to the Web site with the proper controller class. The Routing library in the MVC Framework can interrogate a URL and determine the name of the controller. The MVC Framework will then ask the registered IControllerFactory object for the controller object that matches the controller that matches the controller name determined from the incoming URL.
Many teams simply delegate controller construction to an inversion of control (IOC) tool. In my team's case, we use the open source StructureMap tool for resolving controller instances by name:
public class StructureMapControllerFactory
  : IControllerFactory {

  public IController CreateController(
    RequestContext requestContext, string controllerName) {

    // Requests the named Controller from the
    // StructureMap container
    return ObjectFactory.GetNamedInstance<IController>(
      controllerName.ToLowerInvariant());
  }
}
Requesting the controller is fairly simple, but first you need to register all of the controller classes by name with the IOC container. Wait! Isn't that adding some ceremony to the architecture? A year or two ago I would have plunged ahead with explicit IOC configuration of the controller classes like this:
public static class ExplicitRegistration {
  public static void BootstrapContainer() {
    ObjectFactory.Initialize(x => {
      x.ForRequestedType<IController>().AddInstances(y => {
        y.OfConcreteType<AddressController>().WithName("address");
        y.OfConcreteType<ContactController>().WithName("contact");

        // and so on for every possible type of Controller
      });
    });
  }
}
This code represents pure tedium and ceremony that exists for no other reason than to feed the IOC tool. If you look closer at the registration code, you'll notice that it's following a consistent pattern. AddressController is registered as address and ContactController is registered as contact. Instead of explicitly configuring each controller, you could simply create a convention for automatically determining the routing name of each controller class.
Fortunately, there's direct support in StructureMap for convention-based registration, so you can create a new ControllerConvention that automatically registers any concrete type of IController:
public class ControllerConvention : TypeRules, ITypeScanner {
  public void Process(Type type, PluginGraph graph) {
    if (CanBeCast(typeof (IController), type)) {
      string name = type.Name.Replace("Controller", "").ToLower();
      graph.AddType(typeof(IController), type, name);
    }
  }
}
Next, you need some code that bootstraps the StructureMap container with the new convention, as shown in Figure 6. Once the new ControllerConvention is in place and part of the IOC container bootstrapping, any new controller classes that you add to the application are automatically added to the IOC registration without any explicit configuration on the part of the developer. So there are no more errors and bugs because a developer forgot to add new configuration elements for new screens.
Figure 6 New Conventions for StructureMap
/// <summary>
/// This code would be in the same assembly as
/// the controller classes and would be executed
/// in the Application_Start() method of your
/// Web application
/// </summary>
public static class SampleBootstrapper {
  public static void BootstrapContainer() {
    ObjectFactory.Initialize(x => {
      // Directs StructureMap to perform auto registration
      // on all the Types in this assembly
      // with the ControllerConvention
      x.Scan(scanner => {
        scanner.TheCallingAssembly();
        scanner.With<ControllerConvention>();
        scanner.WithDefaultConventions();
      });
    });
  }
}
As a side note, I want to explain that this strategy of auto registration is possible in all of the IOC containers that I'm aware of for the .NET Framework as long as the IOC container exposes a programmatic registration API.

Next Steps
In the end, it's all about reducing the distance and friction between your intention and the code that makes that intention happen. A lot of the techniques I showed in this column are really about letting the code "just figure it out" from using naming conventions instead of explicit code or finding ways to avoid duplicating information in the system. I also used some reflective techniques to reuse information buried in attributes to reduce the mechanical effort.
All of these design ideas can reduce the repetitive ceremony of the development effort, but it comes at a price. Detractors of convention over configuration complain about the inherent "magic" of the approach. Embedding opinions into your code or framework will detract from potential reuse in new scenarios where those opinions aren't as favorable.
There were a lot of other topics that I was unable to cover this time that I may cover in a later column. I definitely want to explore how language-oriented programming; alternative languages such as F#, IronRuby, and IronPython; and internal domain-specific language usage affects the software design process.

Send your questions and comments to mmpatt@microsoft.com.

Jeremy Miller, a Microsoft MVP for C#, is also the author of the open source StructureMap tool for Dependency Injection with .NET and the forthcoming StoryTeller tool for supercharged FIT testing in .NET. Visit his blog, The Shade Tree Developer, part of the CodeBetter site.


Under The Table
Spatial Data Support In SQL Server 2008
Bob Beauchemin

Code download available from the MSDN Code Gallery
Browse the Code Online

Contents
Using the Geography Data Type
Spatial Indexes
Visualizers
The Geometry Data Type
Spatial Visualization
The SQL Server Spatial Data Library
Wrapping Up
SQL Server 2008 offers a number of enhancements, including new data types, new T-SQL statements, and incremental changes to Microsoft .NET Framework support, XML support, and the Service Broker feature. By far, the biggest and most exciting change for developers is the SQL Server 2008 support for spatial data—a powerful addition to the SQL Server programmer's toolbox. In this column, I'm going to present a short overview of spatial data and illustrate some interesting uses.
Spatial data is defined as data that is represented by 2D or 3D images. SQL Server 2008 supports 2D images, and 3D may be added in a future release. Spatial data can be further subdivided into geometric data (data that can use calculations involving Euclidian geometry) and geographic data (data that identifies geographic locations and boundaries on the earth).

Using the Geography Data Type
Let's start with a simple application of spatial data, which could be useful to most traditional online transaction processing (OLTP) systems that involve customers. Spatial data is useful in this case because every customer has an address. You usually think of addresses as street, city, state, country, and ZIP code, but an address is a point on the earth and can also be represented as a latitude/longitude pair. Technically, an address can comprise a parcel of land that can be represented as a polygon, but let's stick to a single point in this case, just for simplicity. Knowing where a customer lives as latitude/longitude allows you to answer questions such as:
  • What are the three closest bank branches for a specific customer?
  • Which salesperson lives closest to the customer?
  • How many customers does your company have within a 10 mile radius of, for instance, Seattle, Washington?
  • How many customers live more than 2 miles from your nearest branch location?
The process of converting a street address to a latitude/longitude pair is called address geocoding. A number of online services (including MapPoint, Virtual Earth, and Google Earth) provide geocoding as a service. To convert a street address in the U.S. to a point, you can encapsulate a call to the MapPoint geocoding Web service in a SQLCLR function. This is illustrated in the code sample. But where should you store the latitude/longitude inside your SQL Server database?
SQL Server 2008 ships with the two necessary data types for storing spatial data: geometry and geography. Both data types are implemented using the .NET architecture for user-defined types, meaning that they can have properties and methods. Let's stick with the geography data type for now because this maps more closely to the problem at hand.
You can define an instance of a geography type with a simple SQL variable declaration (DECLARE @g geography) or as a column in a table, and you can initialize this data type in a variety of ways. For a geography type representing an instance of a point, the simplest way is to use the STPointFromText static method of the geography type. The STPointFromText method requires not only a textual representation of a point in the Open Geospatial Consortium's well-known text format (WKT)—that is, POINT(x,y)—but also a spatial reference identifier (SRID). The SRID identifies the spatial reference system used for either a round-earth or flat-earth mapping and, for now, it's enough to know that the MapPoint geocoder Web service uses GPS coordinates that correspond to SRID 4326. This represents the World Geodetic System 1984 (WGS 84). (To find all the SRIDs that SQL Server 2008 supports, simply query the metadata table sys.spatial_reference_systems.)
So your code to initialize the address geography would look something like this:
DECLARE @addr nvarchar(256) = 'Some sample address, City, State, Zip';
DECLARE @addr_as_xy nvarchar(30);
DECLARE @g geography;
SET @addr_as_xy = dbo.Geocoder(@addr);
SET @g = geography::STPointFromText(@addr_as_xy, 4326);
Note that because geography is implemented as a .NET-based data type, there are some interesting points to mention about the code. First, STPointFromText is a static method, so it must be called with the datatype::method syntax. In addition, the method names of .NET-based data types are case sensitive so, STPointFromText must use exact capitalization.
There's one more important thing to point out: the geocoder function must be coded to return the location as "xy" rather than latitude/longitude. The Open Geospatial Consortium defines their WKB and well-known binary (WKB) formats as using x,y coordinate pairs. Therefore, what's being passed in to STPointFromText must be POINT(Longitude/Latitude), not POINT(Latitude/Longitude). If you'd rather use latitude/longitude, SQL Server will construct geography instances using Geographic Markup Language (GML), an XML vocabulary that uses Latitude/Longitude. Or, if you have a point, there is a special static method on the geometry data type that takes three parameters: latitude, longitude, and SRID.
Now that you've geocoded your customer information, you can also go ahead and geocode your bank branch locations (or other branch office information, warehouse information, or anything related to location) and your salesperson locations, as well. Suppose you finish with tables named customer, salesperson, and branch. Each one would have a column (let's call it geog for simplicity) of type geography that indicates its location.
You can use this information to answer the questions mentioned earlier. For this, you need to use methods on the geography data type. The method for calculating the distance between two geography objects is, not surprisingly, called STDistance. The methods you could use to answer the third question (how many customers there are within a 10-mile radius of Seattle) would be STBuffer and STIntersects. SQL Server also has a method that's used to calculate a more approximate buffer named BufferWithTolerance (it's a little faster than calculating an exact buffer).
To answer the first question, you would use a query that looks like this:
SELECT TOP(3) b.name,
 c.geog.STDistance(b.geog)/1000 [Distance in km]
FROM customer c, branch b
WHERE c.customerid = '12345' -- this query looks at customer 12345
ORDER BY c.geog.STDistance(b.geog)
The query to calculate the closest salesperson would be done in approximately the same way.
How about the people who live within a 10-mile radius of Seattle? For this query, you would put a buffer around the city limits of Seattle (represented as either a polygon of the city limits or a point that you designate as downtown Seattle) and select all points (customers) that intersect that buffer. Represented in code, this is:
-- or declare POINT for "downtown Seattle"
-- 1609.344 meters per mile
DECLARE @Seattle geography = 'POLYGON(....)'; SELECT c.customerid FROM
 customer c WHERE c.geog.STIntersects(@Seattle.STBuffer(10 * 1609.344));
These examples illustrate more interesting points about how you interact with instances of the geography type. When you use STBuffer, for instance, which is an instance method on the geography data type, you use the instance.method syntax rather than the type::method syntax you would use for static methods like STPointFromText. In addition, notice that when you're calculating the distance between two geographies or determining whether one geography instance intersects another, you use the syntax Instance1.method(Instance2).
To answer the question about the number of customers who live more than two miles from the nearest bank branch, you must be able to aggregate the geographies of all the bank branches, with a buffer of two miles around each one, and then look for customers that don't intersect with this collection of geographies. The geography data type does have a method named STUnion, but it's specified as a method on a geography instance that returns the union between that instance and another instance.
This is not the set-based aggregation I had in mind. Luckily, SQL Server 2005 introduced user-defined aggregates that can be written in .NET code, and you can use one here to get an aggregate union of a set of values. Without going too far into the specifics of coding user-defined aggregates, they require four method initializations: Init (answer initialization), Terminate (return answer), Accumulate (called for each row), and Merge (called when merging processing from multiple threads).
In the aggregate, the Accumulate method can simply unite the current row geography with the answer geography each time it's called. In fact, you don't have to write this aggregate yourself; it's part of the SQL Server Spatial Tools project on CodePlex. This aggregate will even allow you to pass in a buffer for each row, which solves the two-mile buffer problem. The query can then be written like this:
SELECT COUNT(*)
FROM customer c
WHERE
c.geog.STIntersects(
SELECT dbo.GeographyUnionAggregate(b.geog,2*1609.344) FROM branch b)=0
As you can see by these examples, geocoding addresses and storing them along with the post office's form of the address can bring business value to an application.

Spatial Indexes
Once your business gets bigger, calculating the distance between every customer and every salesperson or every customer and every bank branch might become too slow. SQL Server 2008 spatial data support includes spatial indexing. The spatial indexes are ordinary B-tree indexes that are meant to make spatial queries run faster, just like the relational indexes in SQL Server make relational queries run faster.
Mapping two-dimensional spatial data to a one-dimensional B-tree is accomplished by means of tessellation; that is, dividing the area up into small subareas and recording the subareas that intersect each spatial instance. For the geography data type, that means dividing the entire globe into hemispheres and projecting each hemisphere onto a plane. Each geography instance then covers one or more subsections (tiles) of this plane; the spatial index would contain a row for each tile that an instance covers. For the geometry type, because you're specifying your own rectangular coordinate system, you can specify the boundaries (bounding box) that your spatial index covers.
There are a number of rules that govern how spatial instances that intersect tiles are mapped to rows in a spatial index, and you can define how fine the grid system will be for your spatial index at multiple levels. For more information about the specifics of spatial indexes, SQL Server Books Online is your best reference.
To return to the customer system, you could define a spatial index on the geog column in your customer table with the following data definition language (DDL):
CREATE SPATIAL INDEX cust_geog_idx
ON dbo.customer(geog)
GRIDS =(LEVEL1=HIGH,LEVEL2=HIGH,LEVEL3=HIGH,LEVEL4=HIGH));
Defining this index with a high-granularity grid would be best for your index because each customer address is a point, which would only appear in one tile, as opposed to lines or polygons, which could intersect many tiles.
You should note that spatial indexes are only used with certain spatial methods. Currently, they are used with the following geography predicates:
instance1.STIntersects(instance2) = 1
instance1.STEquals(instance2) = 1
instance1.STDistance(instance2) < number
instance1.STDistance(instance2) <= number
When you're dealing with geometry instead of geography, the spatial index also works with the STContains, STOverlaps, STTouches, and STWithin methods, but only when checking for 1 (True). Based on the way the spatial indexes work, you would really want to rephrase your spatial query about "customers who live more than two miles from a bank branch" to count the customers where STIntersects returns True and subtract that number from the total number of customers.

Visualizers
Notice that in the few examples I've worked through so far, it would be really helpful to be able to see the spatial data in a form other than rows and columns. In fact, you might have noticed that selecting the spatial data type itself returns its binary representation. Using the ToString method or the STAsText method returns the WKB format. That's a little better, but it still doesn't give you any idea about geographic locations unless you can project longitude/latitude pairs onto a map in your head. Spatial data almost always needs a graphic visualizer to make it more useful, and when you're talking about geographic data, that visualizer would normally include a map.
SQL Server Management Studio 2008 includes a simple visualizer in the form of a Spatial results tab in the Query results window. This visualizer works with a geography or geometry column in the query results and plots the spatial data types on a grid. If multiple spatial columns appear in the query results, you can choose the one to visualize. The column to be displayed must be in the SQL Server binary format; using ToString() or STAsText() won't work with the visualizer.
For a geography data type, you even get a choice of map projections such as Mercator or Equirectangular, but the data is not overlaid on a map by default. One cheap and easy way to overlay a map if you have a table that contains map outline data is to do a UNION ALL between your rowset and a rowset that contains the map geographies. Figure 1 shows points representing a set of more than 700 cities from the Mondial database in the SQL Server Management Studio visualizer. The SQL Server query that generated it was:
SELECT geog, name
FROM Mondial.dbo.city
WHERE geog IS NOT NULL
Figure 1 A Rowset of Points from the Mondial Database City Table
An even better choice is to use a commercial or shareware visualizer that provides map overlay by default, as shown in Figure 2. You'll note that the Spatial Results tab in SQL Server Management Studio shows a rowset of more than 700 points with a map overlay. The SQL Server query that produced it looks like this:
SELECT geog, name
FROM Mondial.dbo.city
WHERE geog IS NOT NULL
UNION ALL
SELECT geog, cntry_name
FROM SpatialSamples.dbo.cntry00
Figure 2 A Rowset of Points with a Map Overlay
One visualizer you might use is Craig Dunn's Geoquery 2008 program. This is a free visualizer for SQL Server 2008 query results that includes map projections and also lets you choose the color and line thickness of your spatial data objects. You could also write code to display your data in Microsoft Virtual Earth or Google Earth, but that's beyond the scope of this column.

The Geometry Data Type
Let's get back to the other spatial data type that I touched on briefly, the geometry data type, and discuss where you would use geometry as opposed to geography. The geometry data type represents a flat plane with x and y coordinates, as opposed to geography that represents latitude and longitude, angles from the center to the surface of the earth (ellipsoidal coordinates). The geometry data type could be used for problems that don't need to take the earth's shape into consideration, relatively small planar surfaces such as office cubical layouts or warehouses. When you're trying to locate a coworker's rectangular office in cubicle land, the curvature of the earth doesn't matter, so using the geometry data type and linear calculations will suffice. When you're dealing with geometry, you set your unit of measure and your point of origin (let 0,0 be the bottom-left corner of your warehouse, for example).
Although sometimes you can use the geometry type for spatial locations on the earth, the process is a little bit more complicated. Because there are a number of different ways to map the earth to a coordinate system, and because these affect calculation methods such as STArea and STDistance, your spatial data instances need to specify an SRID.
Whether you are using geometry or geography, an SRID is required. For the geometry data type, if you've mandated the coordinate system and unit of measure yourself (for example, when mapping your office layout) you would choose SRID 0, which specifies an unknown or local spatial reference system. SRID 0 is the default for the geometry data type in SQL Server. For geography, it is important that you choose one of the SRIDs that SQL Server is able to use. SRID 4326 is the default and the one that is used by GPS systems.
The geographic methods built into SQL Server can use any of the 390 spatial reference systems that are enumerated in the system metadata table, sys.spatial_reference_systems. These spatial reference systems come from the European Petroleum Survey Group's (EPSG) geodetic parameter registry. EPSG no longer exists—its functions have been absorbed into the European Petroleum Survey Group. You can assume that an association of oil and gas producers would want to make sure that they have the most accurate positions on the earth when they go exploring.
Each SRID not only names a unit of measure (most use meters) but also include a string that specifies a datum (set of reference points on the earth), geoid, coordinate system, and map projection. For more information about coordinate systems, you should read Isaac Kunen's white paper "Introduction to Spatial Coordinate Systems: Flat Maps for a Round Planet."
Because the SQL Server geography data type's calculation is sensitive to the SRID that is used, attempting to use spatial methods (for example, Distance and Intersects) between instances of spatial types that do not have the same SRID will return the value NULL. I'll talk more about converting between SRIDs later.

Spatial Visualization
I've covered some of the most obvious applications for spatial data and the fact that many of them use visualizers to provide a pictorial view of the data. Suppose that, instead of just using a map of the countries of the world or the states in the U.S., I wanted to visualize my data by projecting it against county boundaries or correlate it with congressional districts or with census data? In this case, I'd have to acquire what I call "spatial reference data." Spatial reference data can either be downloaded from public Web sites or purchased from companies such as Environmental Systems Research Institute (ESRI). Note that the availability of free public spatial reference data varies widely depending on local custom. In the U.S., most of the data I've talked about, such as county boundaries and census data, is more freely available than in many other countries.
So you've gone out to your favorite public Web site, for example the U.S. Census Bureau's Tiger Web site, and downloaded geographic reference files. But the files are in ESRI Shapefile format, not SQL Server geography columns in tables.
How do you import these files into SQL Server? And many publicly available files use SRID 4269, not the SRID 4326 that your geocoder emitted. How do you convert between different SRIDs so that all of your spatial methods won't return NULL? The problem that needs to be solved here is more of an extract-transform-load (ETL) problem than a simple conversion problem. You do want to bring along non-spatial data like census track numbers or population figures. And, although you could (in theory) transform your data between SRIDs on the fly, this would slow your queries down considerably. This is definitely an ETL problem.
SQL Server 2008 doesn't really ship anything in the box to help you here. There are programs such as Morten Nielsen's Shape2SQL tool that loads Shapefiles into SQL Server tables. But Shapefiles aren't the only kind of third-party spatial file, and you may want to perform transformations between SRIDs or other more specialized transformations. There are a variety of third-party commercial products to specialize in spatial data transformations and bulk loading. They include SAFE Software's FME for SQL Server, Manifold's Spatial Extender for SQL Server, and, of course, ESRI's line of geographic information system (GIS) products. These can move data into SQL Server or move data from SQL Server into a full-fledged GIS. I've successfully used SAFE Software's product called FME for SQL Server, which includes a library of transformations, supports almost every spatial data format, and even provides a series of components for the SQL Server ETL system: SQL Server Integration Services.

The SQL Server Spatial Data Library
The format that you will choose to use to move data in and out of SQL Server will depend on the specific needs of your organization and also on the needs of your specific application. SQL Server data types intrinsically support the WKB and WKT formats, as well as geographic markup language (GML) format. SQL Server exposes the geometry and geography types and the associated methods in a .NET-based library named Microsoft.SqlServer.Types.dll. Not only does this library ship with SQL Server, but because it's based on .NET, you can download it as part of the SQL Server 2008 Feature Pack and move the computations and transformations to the middle-tier or client if necessary. Bear in mind that only the server side provides spatial indexes to optimize your spatial data queries.
SQL Server is really storing instances of the .NET classes SqlGeometry and SqlGeography; the classes and methods are intrinsic to the library. You can use spatial methods in SQL Server queries or encapsulate your operations in T-SQL or SQLCLR stored procedures, user-defined functions, and triggers. You can extend the base functionality by using SqlGeometry and SqlGeography as members in .NET user-defined types and user-defined aggregates, like the GeographyUnionAggregate that I used earlier.
The spatial data library also includes a builder API that you can use to optimize loading and for custom transformations and aggregations. This consists of a SqlGeometryBuilder and SqlGeographyBuilder and sink interfaces (IGeometrySink and IGeographySink) that you can use to do optimized construction of spatial data instances. You implement one of the sink interfaces, which allow you to begin or end figures, add lines, and set the appropriate SRID. Once you have the data in place, you simply call the Populate method to create the instances that you've set up.

Wrapping Up
The elements I've covered here represent just a little taste of the functionality that's been added to SQL Server with the inclusion of spatial data types and an industry standard spatial library. Data that you gather through geocoding, public reference data, or your own spatial data for warehouses and offices can be stored in SQL Server, managed with the same administrative tools that you use for other kinds of business data, indexed, queried, and analyzed to bring a whole new dimension to your business.
To get you started, remember that any sort of data that you will be able to capture with a GPS system can be imported and can be used with the rest of your database data. With GPS capabilities becoming more commonplace in phones, cameras, automobiles, and other devices and appliances, and with so many information systems now containing address data, the potential for integrating GPS data into interesting features—those that really meet the needs of the user—is truly unlimited. So stay tuned for some exciting developments.

Bob Beauchemin is a database-centric application practitioner and architect, course author and instructor, writer, and Developer Skills Partner at SQLskills. He's written books and articles on SQL Server, data access and integration technologies, and database security. You can reach him at bobb@sqlskills.com.


Foundations
Error Handling In Workflows
Matt Milner

Code download available from the MSDN Code Gallery
Browse the Code Online

Contents
Handling Faults in Workflows
Handling Faults in the Host Process
Handling Faults in Custom Activities
Using Compensation
Retry Activity
Windows Workflow Foundation (WF) provides the tools to define rich business processes and a runtime environment to execute and manage those processes. In any business process, exceptions to the expected flow of execution occur, and developers need to be able to write robust application logic to recover from those exceptions. Most samples of any technology tend to overlook fault handling and proper recovery. In this month's installment, I will show you how to handle exceptions properly at several levels of the WF programming model and how to build rich exception handling capabilities into your workflows and hosts.

Handling Faults in Workflows
Developers building business processes must be able to handle exceptions to the business case to ensure that the process itself is resilient and can continue after failures occur. This is especially important with workflows, as they often define long-running processes, and an unhandled failure, in most cases, means having to restart the process. The default behavior for a workflow is that if an exception happens in the workflow and is not handled, the workflow will be terminated. Thus, it is crucial for workflow developers to correctly scope work, handle errors, and build into the workflows the ability to retry work when failures occur.
Handling faults in workflows has many things in common with handling faults in Microsoft .NET Framework-targeted code and a few new concepts. To handle a fault, the first step is to define a scope of execution. In .NET code, this is accomplished with the try keyword. In workflows, most composite activities can be used to create an exception handling scope. Each composite activity has the main view showing the child activities and also has alternate views. In Figure 1 the context menu on a Sequence activity shows how various views can be accessed and the result of selecting the View Fault Handlers option.
Figure 1 Alternate Views Menu and Selecting View Fault Handlers
When switching to the Fault Handlers view, a FaultHandlers activity is added to the activities collection of the Sequence activity. Within the FaultHandlers activity, individual FaultHandler activities can be added. Each of the FaultHandler activities has a property to define the fault type and acts like a catch expression in .NET.
The FaultHandler activity is a composite activity that allows workflow developers to add child activities that define how to handle the exception. These activities can provide functionality to log errors, contact administrators, or any other actions that you would normally take when handling exceptions in your code.
The FaultHandler activity also has a Fault property that contains the exception being caught. Child activities can bind to this property to get access to the exception. This is illustrated in Figure 2, where a custom logging activity has its exception property bound to the Fault property on the FaultHandler activity. The logging activity can now write the exception information to a logging API, the Windows Event Log, Windows Management Instrumentation (WMI), or any other destination.
Figure 2 Binding to Faults
Like catch blocks, the FaultHandler activities are evaluated based on their fault types. When defining the workflow, the FaultHandler activities should be added to the FaultHandlers in order from the most specific fault to the least specific, left to right.
Figure 3 Execution Continues after the Composite
When an exception occurs and is caught in .NET code, after the catch block finishes the execution continues after the try scope. So in a workflow, execution continues with the next activity after the composite activity that handles the exception (see Figure 3).
There are two key concepts about how the FaultHandler activities are evaluated and executed. When the Throw activity executes (as in Figure 3), or another activity throws an exception, the runtime puts the activity in the faulted state and schedules the HandleFault method on the activity to execute. I will go into more detail on how activities implement this method shortly, but for now it is enough to know that this is the chance for the activity to clean up.
When the activity finishes cleaning up and moves to the closed state, the parent activity is put into the faulting state and is likewise given the chance to clean up any child activities and signal that it is ready to move to the closed state. It is at this point, when the composite activity signals that it is ready to move to the closed state, that the runtime checks the status and, when it﻿ is﻿ Faulting, examines the FaultHandlers collection. If a FaultHandler activity is found with a Fault type that matches the current exception, that FaultHandler is scheduled and the evaluation is halted. Once the FaultHandler closes, execution can then continue to the next activity.
When an exception occurs, the runtime will attempt to find fault-handling activities on the immediate parent composite activity. If no matching handlers are found, that composite is faulted and the exception bubbles up to the next composite activity in the tree. This is similar to how .NET exceptions bubble up the stack to the calling methods when they are unhandled. If the exception bubbles all the way to the root activity of the workflow and no handler is found, then the workflow is terminated.
Note that the workflow itself is a composite activity and, therefore, can have fault-handling logic defined to deal with exceptions that reach the top level. This is the last chance for a workflow developer to catch and handle exceptions in the business process.

Handling Faults in the Host Process
While creating robust workflows is important, having a host that can deal with exceptions is equally important to the stability of your application. Fortunately, the workflow runtime is robust in handling these exceptions out of the box and shields the host process from most exceptions bubbling up.
When an exception occurs in a workflow, bubbles up through the hierarchy, and is uncaught, the workflow will terminate and an event is raised on the runtime. The host of the runtime can register a handler to get notified when these exceptions occur, but the exceptions do not cause the host to crash. To get notified about these terminations, the host process can use code like the following to get information about the exception from the event arguments:
workflowRuntime.WorkflowTerminated += delegate(
  object sender, WorkflowTerminatedEventArgs e)
{
  Console.WriteLine(e.Exception.Message);
};
In addition to dealing with terminated workflows, the host process also has the ability to get notified about exceptions that occur in runtime services. For example, if SqlWorkflowPersistenceService is loaded into the runtime, it will poll the database and may try to load workflows periodically when they have further work to do. When attempting to load a workflow, the persistence service may throw an exception when trying to deserialize the workflow, for example. When this happens, it is again important that the host process not fail, which is why these services don't rethrow these exceptions. Instead, they raise an event to the workflow runtime. The runtime in turn raises a ServicesExceptionNotHandled event that can be handled in code, as shown here:
workflowRuntime.ServicesExceptionNotHandled += delegate(
  object sender, ServicesExceptionNotHandledEventArgs snhe)
{
  Console.WriteLine(snhe.Exception.Message);
};
In general, developers of runtime services have to make a choice when catching an exception about whether the exception is critical. In SqlWorkflowPersistenceService, not being able to load a single workflow does not mean the service cannot function. Therefore, it makes sense in this case to simply raise an event to allow the host process to determine whether further action is needed. However, if the persistence service cannot connect to the SQL Server database, then it cannot function at all. In that case, rather than raise an event, it makes more sense for the service to throw an exception and bring the host to a halt so that the issue can be resolved.
When developing custom runtime services, the recommended approach is to have those services derive from the WorkflowRuntimeService base class. This base class provides both access to the runtime and a protected method to raise a ServicesExceptionNotHandled event. When an exception occurs in the execution of a runtime service, the service should only throw that exception if it is truly an unrecoverable error. If the error is related to a single workflow instance and not the general execution of the service, then an event should be raised instead.

Handling Faults in Custom Activities
For activity authors, exception handling takes on a slightly different meaning. The goal with exception handling in activities is twofold: handle exceptions that occur to keep them, when possible, from bubbling up and disrupting the workflow and clean up properly in cases where an unhandled exception bubbles out of the activity.
Because an activity is simply a class, handling exceptions within the activity is no different than in any other class. You use try/catch blocks when calling other components that may throw errors. However, once you catch an exception in an activity, you must decide whether to rethrow the exception. If the exception is something that will not affect the outcome of the activity, or your activity has a more controlled way of indicating that it was not successful, this is the preferred way to provide that feedback. If, however, the exception means that your activity has failed and cannot complete its processing or provide an indication of the failure, then you should throw an exception so the workflow developer can design the business process to handle the exception.
The other facet of handling exceptions in activities is dealing with cleaning up activity resources. Unlike in a workflow, where fault handling is focused on the business process, logging, and notification, handling faults in activities is primarily focused on cleaning up the resources used in the activity execution.
The way you handle faults will also depend on whether you are writing a leaf activity or a composite activity. In a leaf activity, the HandleFault method is called when an unhandled exception is caught by the runtime in order to allow the activity to free up any resources that might be in use and clean up any execution that has begun. For example, if the activity is using a database during execution, in the HandleFault method it should make sure to close the connection if it isn't already closed and dispose of any other resources that might be in use. If the activity has begun any asynchronous work, this would be the time to cancel that work and free up the resources being used for that processing.
For a composite activity, when the HandleFault method occurs, it might be because of a logic error in the activity itself, or it may be because a child activity has faulted. In either case, the intent in calling the HandleFault method on a composite activity is to allow the activity to clean up its child activities. This cleanup involves making sure the composite doesn't request any more activities be executed and canceling any activities that are executing. Fortunately, the default implementation of the HandleFault method, defined in the CompositeActivity base class, is to call the Cancel method on the composite activity.
Cancellation is another mechanism that allows activities that have started some work asynchronously and are currently waiting for that work to complete to be notified that they should cancel the work they have started and clean up their resources so they can close. An activity may be canceled if another activity has thrown a fault, or under normal circumstances if the control flow logic of the parent composite activity decides to cancel the work.
When an activity is to be canceled, the runtime sets the status of that activity to Canceling and calls the Cancel method on the activity. For example, the Replicator activity can start several iterations of a child activity, one for each piece of data supplied, and schedule those activities to run in parallel. It also has an UntilCondition property that will be evaluated as each child activity closes. It is possible, and likely, that evaluation of the UntilCondition will cause the activity to determine that it should complete.
In order for the Replicator to close, it must first close all child activities. Since each of those activities has already been scheduled and is potentially executing, the Replicator activity checks the current value of the ExecutionStatus property and, if it is﻿ Executing, makes a request for the runtime to cancel that activity.

Using Compensation
Handling faults in workflows allows developers to deal with immediate exception conditions. The use of transactions also provides the ability to scope work together to ensure consistency. However, in long running workflows, it is possible that two units of work require consistency, but cannot use a transaction.
For example, once a workflow starts it may update the data in a line-of-business application, perhaps adding a customer into the CRM system. This work may even be part of a transaction to provide consistency across several operations in the CRM and with the state of the workflow. Then, after waiting for further input from a user, which may takes days to happen, the workflow updates an accounting system with the customer information. It is important that both the accounting system and the CRM system have consistent data, but it is not possible to use an atomic transaction for those resources across such a large time span. So the question becomes, how do handle exceptions that occur when updating the second system to ensure consistency with the changes already committed to the first system?
Figure 4 While Activity as Retry Logic
Because the work in the two systems cannot be made consistent with a transaction, what you need is a mechanism to detect errors that occur when updating the second system and provide an opportunity to go back and undo the work applied in the initial system, or otherwise make changes to ensure consistency. While the act of detecting this change and initiating this process can be automatic, the work of fixing the initial system obviously has to be specified by the developer.
In WF this process is referred to as Compensation and several activities are provided to help develop workflows that use compensation. For more information on compensation and how to use the compensation related activities, see Dino Esposito's Cutting Edge column on transactional workflows in the June 2007 issue of MSDN Magazine ("Transactional Workflows").

Retry Activity
One of the problems with dealing with exceptions in workflows is that, when an exception occurs, even if you catch it, the execution moves on to the next step in the process. In many business processes, execution really should not continue until the business logic defined in the workflows executes successfully. Developers often deal with this by using a While activity to provide retry logic and defining the condition of the activity to indicate that the activity should continue to execute as long as an error has occurred. Further, a Delay activity is often used to keep the retry logic from happening immediately.
To enable this retry model, you can employ a Sequence activity as the child of a While activity. Further, a specific unit of work in the sequence is often wrapped in another sequence or composite activity to handle the exceptions, acting as the fault-handling scope with all fault handlers defined in the Fault Handlers view. Then an IfElse activity is usually used to modify the state of the workflow to influence the condition on the While activity.
In the case where no exception occurs, the logic sets a property or flag of some sort so the While activity can close. If an exception did occur, then the flag is set to cause the while activity to execute again, and a Delay activity is used to pause before making the next attempt. Figure 4 shows one example of using the While activity to retry activities in a workflow.
While this particular pattern works in many scenarios, imagine a workflow with 5 or 10 different operations that need to be retried. You will quickly realize that it is a lot of work to build the retry logic for each activity. Fortunately, WF enables developers to write custom activities, including custom composite activities. That means I can write my own Retry activity to encapsulate executing the child activities again when an exception occurs. For this to be valuable, I want to provide two key inputs for users: a delay interval between retries, and a maximum number of times to retry the work before letting the exception bubble up and be handled.
In the remainder of this column, I will detail the logic in the Retry activity. For background information on creating custom activities, see my previous article ("Windows Workflow: Build Custom Activities to Extend the Reach of your Workflows", and for more information on using the ActivityExecutionContext to create activities that can iterate over a child activity, see the June 2007 installment of this column ("ActivityExecutionContext in Workflows").
To manage the child activity correctly, it is important to be able to monitor the activity to know when errors occur. Thus, when executing the child activity, the retry activity not only registers to get notified when the child activity closes, but also registers to get notified when the child activity is put into a Faulting state. Figure 5 shows the BeginIteration method used to start each iteration of the child activity. Before scheduling the activity, the Closed and Faulting events have handlers registered.
Figure 5 Executing Child Activities and Registering for Faults
Activity child = EnabledActivities[0];
ActivityExecutionContext newContext =
  executionContext.ExecutionContextManager.CreateExecutionContext(child);

newContext.Activity.Closed +=
  new
EventHandler<ActivityExecutionStatusChangedEventArgs>(child_Closed);

newContext.Activity.Faulting +=
  new
EventHandler<ActivityExecutionStatusChangedEventArgs>(Activity_Faulting);

newContext.ExecuteActivity(newContext.Activity);
Normally if a child activity faults, the parent activity would also be put in the faulting state. In order to avoid that situation, when the child activity faults, the Retry activity checks to see whether the activity has already been retried the maximum number of times. If the retry count has not been reached, then this code nulls out the current exception on the child activity, thus suppressing the exception:
void Activity_Faulting(object sender,
  ActivityExecutionStatusChangedEventArgs e)
{
  e.Activity.Faulting -= Activity_Faulting;
  if(CurrentRetryAttempt < RetryCount)
e.Activity.SetValue(
    ActivityExecutionContext.CurrentExceptionProperty, null);
}
When the child activity closes, the logic must determine how the activity got to the closed state and uses the ExecutionResult property to do so. Since all activities end in the Closed state, the ExecutionStatus does not provide the information needed to determine the actual outcome, but the ExecutionResult indicates whether the activity faulted, succeeded, or was canceled. If the child activity succeeded, then no retry is needed and the Retry activity simply closes:
if (e.ExecutionResult == ActivityExecutionResult.Succeeded)
{
  this.SetValue(ActivityExecutionContext.CurrentExceptionProperty, null);
  thisContext.CloseActivity();
  return;
}
If the result from the closing activity is not success, and the retry count hasn't been reached, then the activity must be executed again, but not before the retry interval has expired. In Figure 6, instead of beginning another iteration directly, a timer subscription is created using the interval configured on the activity.
Figure 6 Creating a Timer Subscription
if (CurrentRetryAttempt++ < RetryCount &&
    this.ExecutionStatus == ActivityExecutionStatus.Executing) {

  this.SetValue(ActivityExecutionContext.CurrentExceptionProperty, null);

  DateTime expires = DateTime.UtcNow.Add(RetryInterval);
  SubscriptionID = Guid.NewGuid();

  WorkflowQueuingService qSvc =
    thisContext.GetService<WorkflowQueuingService>();
  WorkflowQueue q = qSvc.CreateWorkflowQueue(SubscriptionID, false);
  q.QueueItemAvailable += new EventHandler<QueueEventArgs>(TimerExpired);

  TimerEventSubscription subscription = new TimerEventSubscription(
    SubscriptionID, WorkflowInstanceId, expires);
  TimerEventSubscriptionCollection timers =
    GetTimerSubscriptionCollection();
  timers.Add(subscription);

  return;
}
When the timer expires, the TimerExpired method will be invoked, as shown here:
void TimerExpired(object sender, QueueEventArgs e)
{
  ActivityExecutionContext ctx =
    sender as ActivityExecutionContext;
  CleanupSubscription(ctx);
  BeginIteration(ctx);
}
Figure 7 The Retry Activity in a Workflow
This will begin the next iteration of the child activity. By using the TimerEventSubscription class and adding the timer to the workflow's timer collection, the activity is able to correctly participate in persistence and resumption with whatever persistence service is currently configured in the runtime. If the retry interval is long, the entire workflow can be taken out of memory until the timer expires.
The key behavior of the workflow activity has been met at this point. If a child activity faults, the retry activity will not fault. Instead it will pause for the retry interval, then attempt to execute the child activity again.
The final step is to deal with the case where the activity has reached the retry count and the child activity has continued to fail. In this case, the Activity_Faulting method does not clear the exception on the child activity, as the goal is to let that activity fault as normal. And when the child activity closes, the Retry activity also closes.
When the Retry closes after all retry attempts have failed, the result is the same as if the original work had failed in a sequence. The Retry activity can have FaultHandler activities defined and those fault handlers will only execute after all retries have been executed. Using this model simplifies the development of workflows with actions that may need to be retried, yet maintains the same development experience for workflow developers in regard to handling faults as shown in Figure 7.
In addition, the fault handlers will be executed for the child activity when the retry attempts have failed, so workflow developers can choose to handle the faults on either activity. The HandleFault method gets called on the child activity for each failure, ensuring that the activity has a chance to clean up on each iteration.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is a member of the technical staff at Pluralsight, where he focuses on connected systems technologies. Matt is also an independent consultant specializing in Microsoft .NET technologies with a focus on Windows Workflow Foundation, BizTalk Server, ASP.NET, and Windows Communication Foundation. Matt lives in Minnesota with his wife, Kristen, and his two sons. Contact Matt via his blog at pluralsight.com/community/blogs/matt.


Windows With C++
Visual C++ 2010 And The Parallel Patterns Library
Kenny Kerr

This column is based on a prerelease version of Visual Studio 2010. All information is subject to change.

Contents
Language Enhancements
Parallel Algorithms
Tasks and Task Groups
Visual C++ is getting a major upgrade in the 2010 release of Visual Studio. Many of the new language and library features are designed purely to make it easier and more natural to express your desires in code. But as has always been the case with C++, the combination of these features is what makes C++ such a powerful and expressive language.
So this month I am going to introduce some of the additions to the C++ language that Visual C++ has added as part of the forthcoming C++0x standard. I'll then look at the Parallel Patterns Library (PPL) that Microsoft has developed over and above the C++0x standard to introduce parallelism to your applications in a manner that naturally complements the standard C++ library.

Language Enhancements
In the May 2008 article "C++ Plus: Beef Up Windows Apps with the Visual C++ 2008 Feature Pack," I introduced the additions to the Standard C++ Library as part of Technical Report 1 (TR1) that was originally introduced with the Visual C++ 2008 Feature Pack and is now included with Visual Studio 2008 SP1. In that article, I demonstrated the support for function objects through the function template class and the bind template function. Being able to treat functions polymorphically solved many of the awkward problems C++ developers often ran into when writing or using generic algorithms.
As a recap, here's an example of how you initialize a function object with the standard plus algorithm:
function<int (int, int)> f = plus<int>();
int result = f(4, 5);
With the help of the bind function, you can transform a function that provides the necessary functionality but doesn't quite have the right signature.
In the following example, I'm using the bind function with placeholders to initialize the function object with a member function:
struct Adder
{
   int Add(int x, int y, void* /*reserved*/)
   {
       return x + y;
   }
};

Adder adder;
function<int (int, int)> f = bind(&Adder::Add, &adder, _1, _2,
    static_cast<void*>(0));
int result = f(4, 5);
There are two problems that arise with the use of these library additions that cannot be easily overcome without language enhancements. For starters, it's often inefficient to explicitly define a function object, as it adds certain overhead that the compiler could otherwise have avoided. It can also be quite redundant and tedious to re-declare the function prototype when the compiler clearly knows the signature that best matches the initialization expression.
This is where the new auto keyword helps you. It can be used in lieu of explicitly defining the type of a variable, which is useful in template metaprogramming where the specific types are either hard to define or complicated to express. Here's what it looks like:
auto f = plus<int>();
The definitions of the functions themselves can also benefit from some improvement. Often you can reuse helpful algorithms, such as those in the Standard C++ Library. More often than not, however, you need to write some domain-specific function for a very specific purpose that is not generally reusable.
But because the function must be defined elsewhere, you must consider both logical and physical design. Wouldn't it be nice if the definition could be placed in the very location where it is needed, simplifying the understanding of the code by improving the locality of the logic and improving the encapsulation of the overall design? The addition of lambda expressions allows exactly that:
auto f = [](int x, int y) { return x + y; };
Lambda expressions define unnamed function objects, sometimes called closures. The [] is the hint that tells the compiler that a lambda expression is beginning. This is known as the lambda introducer, and it is followed by a parameter list. This parameter declaration can also include a return type, although it is often omitted when the compiler can deduce the type unambiguously, as is the case in the previous code snippet. In fact, the parameter list itself can be omitted if the function is to accept no parameters. The lambda expression concludes with any number of C++ statements within braces.
Lambda expressions can also capture variables for use within the function from the scope in which the lambda expression is defined. Here's an example that uses a lambda expression to calculate the sum of the values in a container:
int sum = 0;
for_each(values.begin(), values.end(), [&sum](int value)
{
    sum += value;
});
Granted, this could have been performed more succinctly with the accumulate function, but that would be missing the point. This example demonstrates how the sum variable is captured by reference and used within the function.

Parallel Algorithms
The PPL introduces a set of task-oriented parallelism constructs as well as a number of parallel algorithms similar to what is available with OpenMP today. The PPL algorithms are, however, written using C++ templates rather than pragma directives and, as a result, are far more expressive and flexible. The PPL is, however, fundamentally different from OpenMP in the sense that the PPL promotes a set of primitives and algorithms that are more composable and reusable as a set of patterns. Meanwhile, OpenMP is inherently more declarative and explicit in matters such as scheduling and ultimately is not part of C++ proper. The PPL is also built on top of the Concurrency Runtime, allowing for greater potential interoperability with other libraries based on the same runtime. Let's look at the PPL algorithms and then see how you can use the underlying functionality directly for task-oriented parallelism.
In the October 2008 installment of this column ("Exploring High-Performance Algorithms"), I demonstrated the benefit of efficient algorithms and the effects of locality and cache-conscious designs on performance. I showed how an inefficient, single-threaded algorithm for converting a large image to grayscale took 46 seconds, while an efficient implementation still only using a single thread took just 2 seconds. With a little sprinkling of OpenMP I was able to parallelize the algorithm over the Y axis and reduce the time even further. Figure 1 shows the code for the OpenMP algorithm.
Figure 1 Grayscale Algorithm Using OpenMP
struct Pixel
{
    BYTE Blue;
    BYTE Green;
    BYTE Red;
    BYTE Alpha;
};

void MakeGrayscale(Pixel& pixel)
{
    const BYTE scale = static_cast<BYTE>(0.30 * pixel.Red +
                                         0.59 * pixel.Green +
                                         0.11 * pixel.Blue);

    pixel.Red = scale;
    pixel.Green = scale;
    pixel.Blue = scale;
}

void MakeGrayscale(BYTE* bitmap,
                   const int width,
                   const int height,
                   const int stride)
{
    #pragma omp parallel for
    for (int y = 0; y < height; ++y)
    {
        for (int x = 0; x < width; ++x)
        {
            const int offset = x * sizeof(Pixel) + y * stride;

            Pixel& pixel = *reinterpret_cast<Pixel*>(bitmap + offset);

            MakeGrayscale(pixel);
        }
    }
}
The PPL includes a parallel for algorithms that can very naturally, with a little help from lambda expressions, replace the use of OpenMP in the MakeGrayscale function from Figure 1:
parallel_for(0, height, [&] (int y)
{
    for (int x = 0; x < width; ++x)
    {
        // omitted for brevity
    }
});
As you can see, the for loop as well as the OpenMP pragma have been replaced with the parallel_for function. The function's first two parameters define the range of iteration, just like for the previous for loop. Unlike OpenMP, which places heavy restrictions on the for directive, parallel_for is a template function so you can, for example, iterate over unsigned types or even complex iterators from the standard containers. The last parameter is a function object, which I define as a lambda expression.
You'll notice that the lambda introducer includes only an ampersand without explicitly declaring any variables to capture. This tells the compiler to capture all possible variables by reference. Since the statements within the lambda expression use a number of variables, I used this as a shorthand. Be careful, however, because the compiler is unable to optimize away all the unused variables, resulting in poor runtime performance. I could have explicitly captured the variables I needed with the following capture list:
[&bitmap, width, stride]
Just as the parallel_for function is a parallel alternative to the for loop providing parallel iteration over a range of indices, the PPL also provides the parallel_for_each template function as a parallel alternative to the standard for_each function. It provides parallel iteration over a range of elements defined by a pair of iterators, such as those provided by the standard containers. Although it made more sense for the previous example to use the parallel_for function with explicit indices, it is often more natural to use iterators to define a range of elements. Given an array of numbers, you could square their values using the parallel_for function as follows:
array<int, 5> values = { 1, 2, 3, 4, 5 };

parallel_for(0U, values.size(), [&values] (size_t i)
{
    values[i] *= 2;
});
But this approach is overly verbose, requires the array itself to be captured by the lambda expression, and, depending on the type of container, may be inefficient. The parallel_for_each function solves these problems nicely:
parallel_for_each(values.begin(), values.end(), [] (int& value)
{
    value *= 2;
});
If you just want to run a number of functions in parallel, or as parallel as is possible based on the number of hardware threads available, you can use the parallel_invoke template function. There are overloads available to accept anywhere from 2 to 10 function objects. Here's an example of running 3 lambda functions in parallel:
combinable<int> sum;

parallel_invoke([&] { sum.local() += 1; },
                [&] { sum.local() += 2; },
                [&] { sum.local() += 3; });

int result = sum.combine([] (int left, int right)
{
    return left + right;
});

ASSERT(6 == result);
This example also illustrates another helper class provided by the PPL. The combinable class makes it incredibly easy to combine the results of a number of parallel tasks with minimal locking. By providing a local copy of the value for each thread and then only combining the results of each thread after the parallel work has concluded, the combinable class avoids much of the locking that normally occurs in such cases.

Tasks and Task Groups
The actual parallelism in the algorithms I've discussed is achieved by means of a simple task-oriented API that you are free to use directly. Tasks are defined with the task_handle class initialized with a function object. Tasks are grouped together with a task_group class that runs the tasks and waits for them to complete. Of course, the task_group provides helpful overloads so that in many cases you don't even have to define the task_handle objects yourself and can let the task_group object allocate and manage its lifetimes for you. Here's an example of how to use a task_group to replace the parallel_invoke function from the previous example:
task_group group;
group.run([&] { sum.local() += 1; });
group.run([&] { sum.local() += 2; });
group.run([&] { sum.local() += 3; });
group.wait();
In addition to the algorithms and APIs I've discussed here, other parallel algorithms and helper classes may well be included when the Parallel Patterns Library is finally released with Visual C++ 2010. To keep up to date on the latest in concurrency, visit Parallel Programming in Native Code.

Send your questions and comments to mmwincpp@microsoft.com.

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. Reach Kenny at weblogs.asp.net/kennykerr.


.NET Matters
Ordered Execution With ThreadPool
Stephen Toub
Q Many components in my system need to execute work asynchronously, which makes me think that the Microsoft .NET Framework ThreadPool is the right solution. However, I have what I believe is a unique requirement: each component needs to ensure that its work items are processed in order and that, as a result, no two of its work items are executed at the same time. It's OK, though, for multiple components to execute concurrently with each other; in fact, that's desired. Do you have any recommendations?
A This isn't as unique a predicament as you might think, as it occurs in a variety of important scenarios, including ones based on message passing. Consider a pipeline implementation that obtains parallelism benefits by having multiple stages of the pipeline active at any one time.
For example, you could have a pipeline that reads in data from a file, compresses it, encrypts it, and writes it out to a new file. The compression can be done concurrently with the encryption, but not on the same data at the same time, since the output of one needs to be the input to the other. Rather, the compression routine can compress some data and send it off to the encryption routine to be processed, at which point the compression routine can work on the next piece of data.
Since many compression and encryption algorithms maintain a state that affects how future data is compressed and encrypted, it's important that ordering is maintained. (Never mind that this example deals with files, and it'd be nice if you could decrypt and decompress the results in order to get back the original with all of the data in the correct order.)
There are several potential solutions. The first solution is simply to dedicate a thread to each component. This DedicatedThread would have a first-in-first-out (FIFO) queue of work items to be executed and a single thread that services that queue. When the component has work to be run, it dumps that work into the queue, and eventually the thread will get around to picking up the work and executing it. Since there's only one thread, only one item will be run at a time. And as a FIFO queue is being used, the work items will be processed in the order they were generated.
As with the example I provided in the January 2008 .NET Matters column, I'll use a simple WorkItem class to represent the work to be executed, shown in Figure 1. An implementation of DedicatedThread that uses this WorkItem type is shown in Figure 2. The bulk of the implementation is in a naive BlockingQueue<T> implementation (the .NET Framework 4.0 includes a BlockingCollection<T> type that would be a better fit for an implementation like this). The constructor of DedicatedThread simply creates a BlockingQueue<T> instance, then spins up a thread that continually waits for another item to arrive in the queue and then executes it.
Figure 1 Capturing a Work Item
internal class WorkItem {
  public WaitCallback Callback;
  public object State;
  public ExecutionContext Context;

  private static ContextCallback _contextCallback = s => {
    var item = (WorkItem)s;
    item.Callback(item.State);
 };

  public void Execute() {
    if (Context != null)
      ExecutionContext.Run(Context, _contextCallback, this);
    else Callback(State);
  }
}
Figure 2 DedicatedThread Implementation
public class DedicatedThread {
  private BlockingQueue<WorkItem> _workItems =
    new BlockingQueue<WorkItem>();

  public DedicatedThread() {
    new Thread(() => {
      while (true) { workItems.Dequeue().Execute(); }
    }) { IsBackground = true }.Start();
  }

  public void QueueUserWorkItem(WaitCallback callback, object state) {
    _workItems.Enqueue(new WorkItem {
      Callback = callback, State = state,
      Context = ExecutionContext.Capture() });
  }

  private class BlockingQueue<T> {
    private Queue<T> _queue = new Queue<T>();
    private Semaphore _gate = new Semaphore(0, Int32.MaxValue);

    public void Enqueue(T item) {
      lock (_queue) _queue.Enqueue(item);
      _gate.Release();
    }

    public T Dequeue() {
      _gate.WaitOne();
      lock (_queue) return _queue.Dequeue();
    }
  }
}
This provides the basic functionality for your scenario and it may meet your needs, but there are some important downsides. First, a thread is being reserved for each component. With one or two components, that may not be a problem. But for a lot of components, this could result in a serious explosion in the number of threads. That can lead to bad performance.
This particular implementation is also not extremely robust. For example, what happens if you want to tear down a component—how do you tell the thread to stop blocking? And what happens if an exception is thrown from a work item?
As an aside, it's interesting to note that this solution is similar to what Windows uses in a typical message pump. The message pump is a loop waiting for messages to arrive, dispatching them (processing them), then going back and waiting for more. The messages for a particular window are processed by a single thread. The similarities are demonstrated by the code in Figure 3, which should exhibit behavior very much like the code in Figure 2. A new thread is spun up that creates a Control, ensures that its handle has been initialized, and uses Application.Run to execute a message loop. To queue a work item to this thread, you simply use the Control's BeginInvoke method. Note that I'm not recommending this approach, but rather just pointing out that, at a high level, it's the same basic concept as the DedicatedThread solution already shown.
Figure 3 Similarities with a UI Message Loop
public class WindowsFormsDedicatedThread {
  private Control _control;

  public WindowsFormsDedicatedThread() {
    using (var mre = new ManualResetEvent(false)) {
      new Thread(() => {
        _control = new Control();
        var forceHandleCreation = _control.Handle;
        mre.Set();
        Application.Run();
      }) { IsBackground = true }.Start();
      mre.WaitOne();
    }
  }

  public void QueueUserWorkItem(WaitCallback callback, object state) {
    _control.BeginInvoke(callback, state);
  }
}
A second solution involves using the ThreadPool for execution. Rather than spinning up a new, custom thread per component that services a private queue, we'll keep just the queue per component, such that no two elements from the same queue will ever be serviced at the same time. This has the benefits of allowing the ThreadPool itself to control how many threads are needed, to handle their injection and retirement, to handle reliability issues, and to get you out of the business of spinning up new threads, which is infrequently the right thing to do.
An implementation of this solution is shown in Figure 4. The FifoExecution class maintains just two fields: a queue of work items to be processed, and a Boolean value that indicates whether a request has been issued to the ThreadPool to process work items. Both of these fields are protected by a lock on the work items list. The rest of the implementation is simply two methods.
Figure 4 Implementing FifoExecution
public class FifoExecution {
  private Queue<WorkItem> _workItems = new Queue<WorkItem>();
  private bool _delegateQueuedOrRunning = false;

  public void QueueUserWorkItem(WaitCallback callback, object state) {
    var item = new WorkItem {
      Callback = callback, State = state,
      Context = ExecutionContext.Capture() };
    lock (_workItems) {
      _workItems.Enqueue(item);
      if (!_delegateQueuedOrRunning) {
        _delegateQueuedOrRunning = true;
        ThreadPool.UnsafeQueueUserWorkItem(ProcessQueuedItems, null);
      }
    }
  }

  private void ProcessQueuedItems(object ignored) {
    while (true) {
      WorkItem item;
      lock (_workItems) {
        if (_workItems.Count == 0) {
          _delegateQueuedOrRunning = false;
          break;
        }
        item = _workItems.Dequeue();
      }
      try { item.Execute(); }
      catch {
        ThreadPool.UnsafeQueueUserWorkItem(ProcessQueuedItems,
          null);
        throw;
      }
    }
  }
}
The first method is QueueUserWorkItem, with a signature that matches that exposed by the ThreadPool (the ThreadPool also provides a convenience overload that accepts just a WaitCallback, an overload you could choose to add). The method first creates a WorkItem to be stored and then takes the lock. (No shared state is accessed while creating the WorkItem. Thus, in order to keep the lock as small as possible, this capturing of the item is done before taking the lock.) Once the lock is held, the created work item is enqueued onto the work item queue.
The method then checks whether a request has been made to the ThreadPool to process queued work items, and, if one hasn't been made, it makes such a request (and notes it for the future). This request to the ThreadPool is simply to use one of the ThreadPool's threads to execute the ProcessQueuedItems method.
When invoked by a ThreadPool thread, ProcessQueuedItems enters a loop. In this loop, it takes the lock and, while holding the lock, it checks whether there are any more work items to be processed. If there aren't any, it resets the request flag (such that future queued items will request processing from the pool again) and exits. If there are work items to be processed, it grabs the next one, releases the lock, executes the processing, and starts all over again, running until there are no more items in the queue.
This is a simple yet powerful implementation. A component may now create an instance of FifoExecution and use it to schedule work items. Per instance of FifoExecution, only one queued work item will be able to execute at a time, and queued work items will execute in the order they were queued. Additionally, work items from distinct FifoExecution instances will be able to execute concurrently. And the best part is that you're now out of the business of thread management, leaving all of the hard (but very important) work of thread management to the ThreadPool.
In the extreme case, where every component is keeping the pool saturated with work, the ThreadPool will likely ramp up to having one thread per component, just like in the original DedicatedThread implementation. But that will only happen if it's deemed appropriate by the ThreadPool. If components aren't keeping the pool saturated, many fewer threads will be required.
There are additional benefits, such as letting the ThreadPool do the right thing with regard to exceptions. In the DedicatedThread implementation, what happens if the processing of an item throws an exception? The thread will come crashing down, but depending upon the application's configuration, the process may not be torn down. In that case, work items will start queueing up to the DedicatedThread, but none will ever get processed. With FifoExecution, the ThreadPool will just end up adding more threads to compensate for those that have gone away.
Figure 5 shows a simple demo application that utilizes the FifoExecution class. This app has three stages in a pipeline. Each stage writes out the ID of the current piece of data it's working with (which is just the loop iteration). It then does some work (represented here by a Thread.SpinWait) and passes data (again, just the loop iteration) along to the next stage. Each step outputs its information with a different number of tabs so that it's easy to see the results separated out. As you can observe in the output shown in Figure 6, each stage (a column) is keeping the work ordered correctly.
Figure 5 Demonstration of FifoExecution
static void Main(string[] args) {
  var stage1 = new FifoExecution();
  var stage2 = new FifoExecution();
  var stage3 = new FifoExecution();

  for (int i = 0; i < 100; i++) {
    stage1.QueueUserWorkItem(one => {
      Console.WriteLine("" + one);
      Thread.SpinWait(100000000);

      stage2.QueueUserWorkItem(two => {
        Console.WriteLine("\t\t" + two);
        Thread.SpinWait(100000000);

        stage3.QueueUserWorkItem(three => {
          Console.WriteLine("\t\t\t\t" + three);
          Thread.SpinWait(100000000);
        }, two);
      }, one);
    }, i);
  }

   Console.ReadLine();
}
Figure 6 Output from Demo Application
It's also interesting to note that there's a lack of fairness between the stages of the pipeline. You can see, for example, that stage1 in Figure 6 is already up to iteration 21, while stage2 is still back on 13 and stage3 is on 9. This is largely due to my implementation of ProcessQueuedItems. The sample app is very quickly pushing 100 work items into stage1, and thus the thread from the pool that services stage1 will likely sit in the ProcessQueuedItems loop and not return until there's no more stage1 work. This gives it an unfair bias over the other stages. If you see similar behavior in your app, and it's a problem, you can increase fairness between the stages by modifying the implementation of ProcessQueuedItems to one more like the following:
private void ProcessQueuedItems(object ignored) {
  WorkItem item;
  lock (_workItems) {
    if (_workItems.Count == 0) {
      _delegateQueuedOrRunning = false;
      return;
    }
    item = _workItems.Dequeue();
  }
  try { item.Execute(); }
  finally {
    ThreadPool.UnsafeQueueUserWorkItem(ProcessQueuedItems,
      null);
  }
}
Now, even if there are more items to be processed, ProcessQueuedItems won't loop around but rather will recursively queue itself to the ThreadPool, thus prioritizing itself behind items from other stages. With this modification, the output from the application in Figure 5 now looks like that shown in Figure 7. You can see in this new output that scheduling is indeed treating stage2 and stage3 with more fairness than before (there's still some lag between the stages, but that's to be expected given that this is a pipeline).
Figure 7 New Output with Fairer Scheduling
Of course, this increased fairness doesn't come for free. Each work item now incurs an extra trip through the scheduler, which adds some cost. You'll need to decide whether this is a trade-off you can make for your application; for example, if the work you're doing in your work items is at all substantial, this overhead should be negligible and unnoticeable.
This is just one more example of how it's possible to build systems on top of the ThreadPool that add functionality without having to build custom thread pools yourself. For other examples, see previous editions of the .NET Matters column in MSDN Magazine.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.


Going Places
Mobile Device Provisioning With SyncML
Ramon Arjona
Contents
What Is OMA?
SyncML: XML for Synchronization
SyncML Elements
The SyncBody Element
But, Wait! There's More!
Putting SyncML to Work
In the April 2008 issue of MSDN Magazine, Mike Calligaro wrote a Going Places column about provisioning a Windows Mobile device using XML files and the DMProcessConfigXML API. The XML files Mike used were based on the Open Mobile Alliance (OMA) Client Provisioning (OMA-CP) standard for device management, formerly known as Wireless Application Protocol (WAP).
OMA-CP works fine for managing a few devices, and in his column Mike has suggestions for ways to push provisioning information to multiple devices at once. But what if you need to provision thousands of devices in a systematic way over a long period of time? And what if you need to know what software and registry settings are already set on the devices you're trying to manage?
This sort of large, coordinated provisioning effort is the domain of OMA Device Management (OMA-DM). The OMA-DM protocol is based on a dialect of XML called SyncML, and it can be used to provision and manage devices in an enterprise scenario. This column will discuss the structure of a SyncML document used in the OMA-DM protocol. I will cover specific usage scenarios for the Windows Mobile platform, including remotely wiping the device. I will also demonstrate how you can set a browser favorite on the device using the XML from Mike's column in conjunction with OMA-DM.

What Is OMA?
OMA is an industry group comprising more than 200 mobile operators, mobile device manufacturers, and software vendors (including Microsoft). OMA was formed in 2002 to create a single group to manage the burgeoning number of mobile device standards that were appearing in the marketplace, including WAP, Device Management, and SyncML. This multiplicity of groups was creating duplication of work, so the concerned industry partners got together and brought all of these important initiatives together under a single umbrella group.
OMA standards help make it easier to work with mobile devices and networks by creating a nonproprietary way to communicate between technologies. SyncML and OMA-DM are two aspects of this open protocol family. Like all protocols, they are subject to a certain amount of interpretation, and every vendor is free to supply its own value-added extensions. But working with them is easier and more straightforward than trying to negotiate a proprietary, vendor-specific format.

SyncML: XML for Synchronization
SyncML is an XML-based markup that is used as the basis for most protocols defined by OMA. A SyncML document is called a message. The message must be well-formed XML but not necessarily valid XML. That is, it must have matching open and close tags for all elements, must have quotes around all attributes, and otherwise conform to the basic definition of well-formedness that is essential to any document calling itself XML.
While there is a SyncML document type definition (DTD), there is no requirement that the sender or receiver validate against it. One of the reasons for this is that validation requires extra memory and processor time, and mobile devices tend to be short on both of these resources. The message, in turn, contains command elements that do the heavy lifting of specific OMA protocols.
SyncML and OMA-DM are transport independent. There are transport bindings defined for HTTP and Object Exchange (OBEX), and it would be possible for other bindings to be defined. This makes SyncML and OMA-DM useful for provisioning devices over the air. Using SyncML and OMA-DM with a compliant device-management server, you can push configurations to a device without using a memory card, without tethering the device, and without asking the end user for interaction, such as visiting a Web site.
One or more messages is contained, conceptually, in a SyncML package. The SyncML package encompasses all of the messages sent between a sender and a receiver.

SyncML Elements
A SyncML document consists of a root SyncML element, a header defined by the SyncHdr element, and a body defined by the SyncBody element. The root of a SyncML document is always a SyncML element, which looks like this:
<SyncML xmlns='SYNCML:SYNCML1.1'>
...
</SyncML>
The SyncML element contains the SyncHdr and SyncBody child elements. A SyncHdr element looks like the markup in Figure 1. This short header tells you everything that a receiver would need to know to process the message.
Figure 1 SyncHdr
<SyncHdr>
  <VerDTD>1.2</VerDTD>
  <VerProto>DM/1.2</VerProto>
  <SessionID>1</SessionID>
  <MsgID>1</MsgID>
  <Target>
    <LocURI>https://mgmt.contoso.com/ </LocURI>
  </Target>
  <Source>
    <LocURI>urn:uuid:6D67E196-D082-4c91-A0DD-DEB3D1D58EB5</LocURI>
    <LocName>MyDeviceName</LocName>
  </Source>
  <Cred>
    <Meta>
      <Format xmlns="syncml:metinf">b64</Format>
      <Type xmlns="syncml:metinf">syncml:auth-md5</Type>
    </Meta>
    <Data>ODZlMDNiYmM1MjA1YTI3MDc5MDk2MDcwYTA4OGM2Zjg=</Data>
  </Cred>
</SyncHdr>
The VerDTD element indicates that this message conforms to version 1.2 of the SyncML Representation Protocol, as defined by OMA. The VerProto element indicates something similar: that you're looking at a message that deals with version 1.2 of the OMA-DM protocol. The version numbers are the same, but the two things are different. SyncML is used for different OMA protocols, including a device management protocol (which I am discussing in this column) and a data synchronization protocol (for which SyncML was originally designed).
The SessionID element indicates at which SyncML session you're looking. The value of this ID can be, at most, 4 bytes long.
The MsgID is an ID that is unique within the session. It's used to keep track of the conversation between sender and receiver. When the receiver replies with results, the results refer to the message being responded to by including the MsgID in a MsgRef element.
The Target element says who the receiver of the message is meant to be. The Source element says who the sender of the message is meant to be. These elements both contain a LocURI element that contains the string uniquely identifying the sender or receiver. The LocURI of the Target and Source must be a URL, a URI, or a URN. Since a management server will usually already have a unique DNS name, it is common to see the fully qualified domain name of the management server in the LocURI when the sender or receiver is a management server.
Note that neither the Target nor the Source element is directly related to addressing or routing the message. These tasks are left to the device-management application and the supporting transport protocols.
Mobile devices are frequently referred to by a URN that refers to a GUID—as defined by RFC 4122—that refers uniquely to the specific device being addressed. It is up to the application to map this GUID back to an address that can be reached over the network. Since the GUID is not an immediately intuitive way to tell to which specific device you are talking, the application has added a LocName element to the Source element, which holds the name of the device that originated the SyncML message.
Finally, the Cred element contains information that identifies the originator. It contains a pair of Meta elements that inform the recipient this message uses the MD5 authentication scheme that is defined in the SyncML standard, and that the security token is base-64 encoded. The Data element that is the sibling of the Meta elements contains the authentication token that can be used by the recipient to confirm the sender's identity.

The SyncBody Element
A SyncBody element is shown in Figure 2. Notice that the Target elements here, just like in the SyncHdr, are identified by LocURI. Nearly everything in SyncML is identified by LocURI. The LocURI is built according to a nested tree structure, similar to the nested tree structure of an XML document, but is somewhat less expensive to process than a series of nested XML nodes. The root of the conceptual tree is identified by the " . " character, which must always be specified.
Figure 2 SyncBody Element
<SyncBody>
  <Add>
    <CmdID>2</CmdID>
    <Item>
      <Target>
        <LocURI>./Vendor/MSFT/SwMgmt/Download/SetBrowserFavorite/PkgID
        </LocURI>
      </Target>
      <Data>pkg id setbrowserfavorite</Data>
    </Item>
    <Item>
      <Target>
        <LocURI>./Vendor/MSFT/SwMgmt/Download/SetBrowserFavorite/PkgURL
        </LocURI>
      </Target>
      <Data>http://content.contoso.com/download/SetBrowserFavorite.cab
      </Data>
    </Item>
  </Add>
  <Exec>
    <CmdID>3</CmdID>
    <Item>
      <Target>
        <LocURI>./Vendor/MSFT/SwMgmt/Download/SetBrowserFavorite/Operations/DownloadInstall        </LocURI>
      </Target>
    </Item>
  </Exec>
<Final/>
    </SyncBody>
The tree contains information about the mobile device, including details such as the amount of memory it has or what executables are available to be invoked over OMA-DM. When a LocURI is used to refer to this tree of device information, it must always be a full URI. That is, it must always be specified from the device root down to the specific leaf to be addressed. Partial or relative URIs are not allowed in this part of the OMA-DM specification.
Note that I refer to this as a "conceptual" tree. That is because the SyncML and OMA-DM specifications do not impose any implementation details on the mobile device or on the device management server for how this information is actually persisted to a backing store. SyncML wants to address the data as though it were a tree, but you could store it in a relational database, in the device or server registry, in volatile memory, in a series of flat files, or in some other way. The specification is completely agnostic to the mechanism used to physically persist this data.
Beneath the root node are three important child nodes: DevInfo, DevDetail, and Vendor. The DevInfo node contains information about the device manufacturer, the language set on the device's UI, and the device ID. The DevDetail node contains other vendor-independent information about the device, including its hardware version and the maximum length of URI supported by the device. Most other interesting details about the device are stored under the Vendor node. By specification, implementers must park their extensions to the OMA-DM protocol under the Vendor node, and this is where you will spend much of your time when provisioning or managing Windows Mobile devices.
Remote Wipe
The Exec command always tells the device to do something, but there is a limited set of things the device can be commanded to do by OMA-DM. One other thing that can be triggered on a Windows Mobile device by OMA-DM is a remote wipe. This is similar to the wipe feature offered for Windows Mobile clients by Exchange and is really useful if a device under your control has been lost or stolen. The command to wipe a device with OMA-DM looks like this:
<Exec>
  <CmdID>3</CmdID>
  <Item>
    <Target><LocURI>./Vendor/MSFT/RemoteWipe/doWipe</LocURI></Target>
  </Item>
</Exec>
It's possible to wipe a device with OMA-CP, too. So you could, if you really wanted to, create a CAB file with provisioning XML in it and distribute it to the device via OMA-DM download. The CAB file's contents would then get routed to the same RemoteWipe CSP that would be invoked by OMA-DM, and the results would be the same.
The SyncBody element shown earlier contains one Add element. The Add element tells the receiver to add a node or series of nodes to the device-information tree. The Microsoft implementation of OMA-DM supports something called implicit add, which means that if an Add command refers to a LocURI with nodes that are not present on the device, all of the nodes from the root to the leaf are inserted into the device-information tree. Without this extension, nodes would have to be added to the device tree one at a time, which would rapidly consume bandwidth, processor time, and—eventually—the user's patience.
This Add command is adding something to the Download node for software management on the device having the URL content.contoso.com/download/SetBrowserFavorite.cab. Ultimately, this URL will be routed to a configuration service provider (CSP) on the device—coincidentally called the Download CSP—that will try to fetch the content specified by this URL.
The Add command is followed by an Exec command. The Exec command tells the device to go do something. In this case, it's telling the device to go download and install the package with the ID SetBrowserFavorite.cab.
The Exec command is followed by a Final element, which tells the receiver that this is the last SyncML message that should be expected for this session. Without the Final element, the receiver would expect that there would be more SyncML messages to follow.
If you can install a CAB file onto the device by copying it onto the device via ActiveSync, you can usually install that CAB file onto the device using OMA-DM. The CAB file must be signed with a certificate that the device trusts. If the file is not signed with a valid certificate, the installation of the CAB file will fail. This differs from the behavior you'll see when installing an unsigned CAB file over ActiveSync. When you install an unsigned file over ActiveSync the device might prompt you for confirmation of your intention to install the unsigned CAB, assuming policy on the device is set to allow this.
The OMA-DM protocol on a Windows Mobile device does not prompt you. It just fails, notifies you of the failure with a message containing an error code in the Managed Programs applet, and sends a related error code back to the device management server. This design makes sense because ActiveSync is being initiated by you, or hopefully someone you trust, while the device is physically in your hands. OMA-DM is initiated by a remote agent, at a time that is completely out of your control.
First, the Download CSP fetches the content from the specified URL. Then it looks at the CAB file and fails if it is unsigned. If the CAB file is signed, the Download CSP unpacks the CAB file and routes the contents of the file appropriately. If the CAB contains software, the software is installed onto the device. If the CAB contains OMA-CP provisioning XML—such as the CAB file Mike created in his column—then the provisioning XML is applied to the device. If the CAB file contained plain content, such as a movie or a home screen, then this content is stored on the device file system.
The Managed Programs applet records every attempt, successful or failed, to install a CAB file onto the device via OMA-DM. After the installation has completed, the device sends a code defined by the OMA-DM standard back to the management server in a new OMA-DM session.

But, Wait! There's More!
Windows Mobile devices respond to a subset of the command elements specified in SyncML. Following are a few command elements I haven't discussed yet.
The Alert command allows the sender to notify the recipient. For instance, an alert from a management server to a device might tell the device to wake up and start a session. Or an alert could contain information that should be displayed to the end user through the UI.
The Atomic element is used to group other commands that must all succeed or fail as a unit. This is important when a group of commands is related and partial success would leave the device in an undesirable or unknown state. Unless grouped in an Atomic element, three separate Add commands would succeed or fail independent of each other and generate three separate response messages for the management server.
Delete is the opposite of Add. The Delete command removes a node from the device tree. If the node has child nodes, those are removed, too. Some nodes, such as the built-in DevInfo node and its children, can't be removed, and the attempt to remove them generates an error code. The Replace command replaces a given node in the device tree.
The Get command is used to query the device tree for information and returns that information in a SyncML message to the sender. For instance, to get the amount of storage available currently on the device, the following Get command would be sent:
<Get>
  <CmdID>3</CmdID>
  <Item>
    <Target>
      <LocURI>./Vendor/MSFT/DeviceInformation/TotalStorage</LocURI>
    </Target>
  </Item>
</Get>
The Result command is sent in response to a Get, containing the value of the node the Get requested.
The Sequence element is used to group nodes in a specific order. If you expect to have commands processed one after another, you must group them in a Sequence element. Otherwise the order of execution is not guaranteed.
And finally, the Status element contains the success or failure code returned by a given operation. A Status of 200 generally indicates success.

Putting SyncML to Work
SyncML began as a protocol not for device management but for device synchronization. Implementations of the OMA sync protocol based on SyncML are used frequently to allow devices to synchronize calendar and contact information. The base protocol specification for SyncML addresses the requirements for synchronization as well as the requirements for management. This means the base SyncML representation protocol contains elements that are never used in OMA-DM and other elements that are never used in synchronization.
First-time readers of the SyncML specification may find themselves a bit confused, because the specification is trying to address two divergent problem domains. While there is some overlap between synchronization and management, the two things are definitely not the same. It's helpful to segregate the elements of SyncML that are used in management conceptually from those that are used in synchronization when reviewing the protocol representation specification.
You need to have a management server to provision your devices via OMA-DM. There are several commercial offerings available, including Microsoft System Center Mobile Device Manager 2008. And there are noncommercial SyncML libraries and implementations available as well. The amount of flexibility allowed to vendors and implementers means that making devices and servers talk using OMA-DM is not always as straightforward as one would want. But the difficulties presented by integration of different devices that use OMA standards is immensely preferable to the difficulty of making devices interoperate using the hodgepodge of unrelated, proprietary communication methods that dominated the mobile marketplace before the advent of OMA.

Send your questions and comments to goplaces@microsoft.com.

Ramon Arjona is a Senior Test Lead working on the Windows Mobile team at Microsoft.


Stack Trace
Working With An Incomplete Or Invalid IRP
Bob Golding and David Butler

How many times have you looked at a hang and found an IRP that looks like this:
0: kd> !irp 8667a928
Irp is active with 10 stacks 5 is current (= 0x8667aa28)
 Mdl = 86041108 Thread 8a613768:  Irp stack trace.
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

                        Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

                        Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

                        Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

                        Args: 00000000 00000000 00000000 00000000
>[  4,34]   c e0 8a58b030 00000000 f74c95d0-8a58be00 Success Error Cancel   <------ Where's the device and
               \Driver\Disk     PartMgr!PmIoCompletion                              the rest of the irp?
                        Args: 00010000 00000000 be82ee00 00000000
 [  4, 0]   c e1 8a58be00 00000000 f73fc5dc-8a430e68 Success Error Cancel pending
               \Driver\PartMgr  ftdisk!FtpRefCountCompletionRoutine
How do you find the physical device? Starting in 2003, the classpnp driver allocates a second IRP and sends it to the port driver. Where can you find them? Well, if you have the FDO for the partition, you can find all of the transfer packets:
Device object (8a58b030) is for:
 DR2 \Driver\Disk DriverObject 8a58c550
Current Irp 00000000 RefCount 0 Type 00000007 Flags 00002050
Vpb 8a439758 Dacl e13f624c DevExt 8a58b0e8 DevObjExt 8a58b548 Dope 8a439700
ExtensionFlags (0000000000)
AttachedDevice (Upper) 8a58be00 \Driver\PartMgr
AttachedTo (Lower) 8a43a028 \Driver\raidisk
Device queue is not busy.
0: kd>
   Take the device extension and find the private data.
0: kd> dt classpnp!_FUNCTIONAL_DEVICE_EXTENSION  8a58b0e8
   +0x000 Version          : 3
   +0x004 DeviceObject     : 0x8a58b030
   +0x000 CommonExtension  : _COMMON_DEVICE_EXTENSION
   +0x100 LowerPdo         : 0x8a43a028
   +0x104 DeviceDescriptor : 0x8a4371e0
   +0x108 AdapterDescriptor : 0x8a575f90
   +0x10c DevicePowerState : 1 ( PowerDeviceD0 )
..................
 +0x204 ScanForSpecialFlags : 0
+0x208 PowerRetryDpc    : _KDPC
+0x228 PowerRetryTimer  : _KTIMER
+0x250 PowerContext     : _CLASS_POWER_CONTEXT
+0x2b4 PrivateFdoData   : 0x8a437350
+0x2b8 Reserved2        : 0
Then display the Private data:
0: kd> dt classpnp!_CLASS_PRIVATE_FDO_DATA 8a437350
   +0x000 AllFdosListEntry : _LIST_ENTRY [ 0x8a193008 - 0x8a588008 ]        <------ They're linked to make things easy, maybe
   +0x008 Perf             : __unnamed
   +0x014 HackFlags        : 0
   +0x018 HotplugInfo      : _STORAGE_HOTPLUG_INFO
   +0x020 Retry            : __unnamed
   +0x088 TimerStarted     : 0 ''
   +0x089 LoggedTURFailureSinceLastIO : 0 ''
   +0x08a LoggedSYNCFailure : 0 ''
   +0x08b ReleaseQueueIrpAllocated : 0x1 ''
   +0x08c ReleaseQueueIrp  : 0x8a437268
   +0x090 AllTransferPacketsList : _LIST_ENTRY [ 0x8a434f10 - 0x860e3308 ]  <------ All the xfer packets for that FDO
   +0x098 FreeTransferPacketsList : _SLIST_HEADER
   +0x0a0 NumFreeTransferPackets : 0
   +0x0a4 NumTotalTransferPackets : 0x41
Just dump the packet list. The original IRP (8667a928 above) is offset 14 in one of the packets on the list. Just find that IRP and dump the packet to find other info:
0: kd> dl 0x8a434f10
8a434f10  8a332308 8a4373e0 00000000 8a6188d0
8a332308  8a2e7008 8a434f10 00000000 8a155008
8a2e7008  89ec9be0 8a332308 00000000 8a267678
89ec9be0  89e31950 8a2e7008 00000000 89f18008
89e31950  89e33858 89ec9be0 00000000 89e55a48
89e33858  892bb2f8 89e31950 00000000 89eef578
892bb2f8  8a318678 89e33858 00000000 89563b28
.....................................                            <----- For space, the whole list was not dumped

: kd> dt classpnp!_TRANSFER_PACKET 87652c80
  +0x000 AllPktsListEntry : _LIST_ENTRY [ 0x8a251f10 - 0x862a81b0 ]
  +0x008 SlistEntry       : _SINGLE_LIST_ENTRY
  +0x00c Irp              : 0x86025d98
  +0x010 Fdo              : 0x8a58b030
  +0x014 OriginalIrp      : 0x8667a928                           <----- This is the IRP that we started with
  +0x018 CompleteOriginalIrpWhenLastPacketCompletes : 0x1 ''
  +0x01c NumRetries       : 8
  +0x020 RetryTimer       : _KTIMER
  +0x048 RetryTimerDPC    : _KDPC
  +0x068 RetryIntervalSec : 0
  +0x06c SyncEventPtr     : (null)
  +0x070 InLowMemRetry    : 0 ''
  +0x074 LowMemRetry_remainingBufPtr : (null)
  +0x078 LowMemRetry_remainingBufLen : 0
  +0x080 LowMemRetry_nextChunkTargetLocation : _LARGE_INTEGER 0x0
  +0x088 BufPtrCopy       : 0x8a3ee000  "RCRD("
  +0x08c BufLenCopy       : 0x10000
  +0x090 TargetLocationCopy : _LARGE_INTEGER 0xbe82ee00
  +0x098 SrbErrorSenseData : _SENSE_DATA
  +0x0ac Srb              : _SCSI_REQUEST_BLOCK
   0: kd> !irp 0x86025d98
Irp is active with 4 stacks 3 is current (= 0x86025e50)
 Mdl = 86041108 Thread 00000000:  Irp stack trace.  Pending has been returned
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

                        Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000

                        Args: 00000000 00000000 00000000 00000000
>[  f, 0]   0 e1 8a58d030 00000000 f721c30a-8a3264d0 Success Error Cancel pending     <----  This is the device we want!
 \Driver\CPQKGPSA RAIDISK
                        Args: 87652d2c 00000000 00000000 00000000
 [  f, 0]   c e1 8a43a028 00000000 f732e540-87652c80 Success Error Cancel pending
               \Driver\raidisk  CLASSPNP!TransferPktComplete
                        Args: 87652d2c 63d46296 01c49a05 00000000

Bob Golding has been with Microsoft since 1997. He is a Senior Escalation Engineer on the Global Escalation Services team where he supports Microsoft's largest customers with their most critical issues.

David Butler has been with Microsoft since 2000. He is an Escalation Engineer on the Global Escalation Services team where he supports Microsoft's largest customers with their most critical issues.


Stack Trace
What Are These System PTEs?
Bob Golding and David Butler
Contents
What System APIs Are Used to Map These Addresses?
Stop 3F Track PTEs Are Not Always Needed
Kernel Stacks
System PTEs, like all PTEs, represent addresses in the system address space. What makes these PTEs different is that they are a system resource. There are only so many per system. These "system PTEs" are "dynamic slots" in the system address space map, which means that a range of addresses are reserved in the system address space and are used to map pages on demand and assign an address to a buffer, or they are used for system thread stacks. The address range lives between nonpaged pool and paged pool. Below is from mi386.h.
E1000000 | Start of paged system area         |
         |   Kernel mode access only.         |
         |                                    |
         |                                    |
         +------------------------------------+
         |                                    |
         | System PTE area - for mapping      |
         |   kernel thread stacks and MDLs    |
         |   that require system VAs.         |<----- Range reserved for dynamic mapping.
         |   Kernel mode access only.         |
         |                                    |
         +------------------------------------+
         |                                    |
         | NonPaged System area               |
         |   Kernel mode access only.         |
         |                                    |
         +------------------------------------+
FFBE0000 | Crash Dump Driver area             |
         |   Kernel mode access only.         |
         +------------------------------------+
So since its size is between paged and nonpaged pool, adjustments to these memory pools have an affect on the size of the mapping window. Also, since this is part of the address space, factors that affect the size of the memory pools (that is, /3GB) affect the size of the mapping window.

What System APIs Are Used to Map These Addresses?
Some APIs that map these addresses are MiMapLockedPagesInUserSpace, MiUnmapLockedPagesInUserSpace, MmMapLockedPages, and MmUnmapLockedPages. These APIs map a buffer into the dynamic address range. They will map the entire buffer, so the buffer size determines how many contiguous pages get used.
As you can see, there is an unmap API for every map API. This is important to note. Buffers should be unmapped; otherwise, the mapping window will close. In other words: no more system PTEs. The more pages that get mapped the more system PTEs get used. As I mentioned before, system stacks use this range. Video does as well in order to map video buffers. So there must be enough left over to map buffers. When system PTEs get low, performance issues will occur. Also, a Stop 3F can occur, but these should be rare, as NT code is more tolerant of low resource conditions.

Stop 3F Track PTEs Are Not Always Needed
A lack of system PTEs is commonly troubleshot using TrackPtes to find the consumer of a high number of the PTEs. However, sometimes this is not useful because TrackPtes is not implemented by the sysptes.c code itself. So it's possible to reserve PTEs without being tracked. TrackPtes is only implemented by some of the callers of MiReserveSystemPtes.

Kernel Stacks
One common consumer of sysptes that is not tracked by TrackPtes is kernel stacks. Fortunately, the memory manager offers us several globals, which give insight into the sysptes being used to map kernel stacks.
In the example below, the memory manager has only created 23,000 sysptes. That's a fairly small number, but it's not uncommon with /3GB. Of these sysptes, 17,860 are being used to map kernel stacks. Analysis of the processes and threads did not find an apparent leak; the customer's application architecture required many GDI threads. After analyzing the machine's memory usage, we were able to tune the memory manager to increase the total system PTEs and make enough room for things to run.
1: kd> !sysptes

System PTE Information
  Total System Ptes 23006

1: kd> dc nt!MmKernelStackPages l 1
80483680  000045c4                             .E..
1: kd> ?45c4
Evaluate expression: 17860 = 000045c4

1: kd> dc MmLargeStacks l 1
8048368c  0000037b
1: kd> dc MmSmallStacks l 1
8048367c  00000385
1: kd> ?37b*f
Evaluate expression: 13365 = 00003435
1: kd> ?385*3
Evaluate expression: 2703 = 00000a8f
1: kd> ?3435+a8f+37b+385
Evaluate expression: 17860 = 000045c4

Bob Golding has been with Microsoft since 1997. He is a Senior Escalation Engineer on the Global Escalation Services team where he supports Microsoft's largest customers with their most critical issues.

David Butler has been with Microsoft since 2000. He is an Escalation Engineer on the Global Escalation Services team where he supports Microsoft's largest customers with their most critical issues.


{ End Bracket }
Your Innovative Ideas
Eric N. Bush
Perhaps you have had the experience of talking to an executive about an idea that you thought was so fantastic it would be your ticket to fame and fortune. Much to your amazement, you were told, "Thanks for coming to me with that idea, but we can't even include half of the features we originally planned for our most recent release! We don't need any more ideas. Besides, if your idea was really so fabulous, my top-level architects would have already thought of it. Now run along back to your team and do what you have been told, and leave the innovating to others who are more experienced."
You would then come to realize, as I did, that having that next great innovative idea is only half the battle. Perhaps the greater challenge is in getting someone to actually listen to you and provide the support needed in order to implement it.
Surprise! It is not the idea that gets people's attention; it's your credentials. So if you don't already have an established track record as an innovator, you will have a harder time getting people to listen to you. Interestingly, a venture fund company is much more likely to fund someone who has been a failure at several startups than a newcomer who has never been involved in a startup before.
The value of the idea is almost secondary. Why is this, you ask? This concept has been rooted in standard business management rules that have existed for years. One of those rules says that it is better to ride out the good and bad times with a few smart people you know and trust, because they will grow from their mistakes and get things right eventually.
Somewhere in MBA school case study folklore there is a story about a corporate vice president who just lost a million dollars for his company. He promptly goes to the office of the president and hands over his resignation letter. The president rips up the letter and tells the VP that he is not about to give up on him, especially since he has just gone through such a costly and valuable learning experience.
If you are not already one of those known and trusted, don't give up. There is still hope for you to take your great idea and get someone to listen to it. However, if you are going to present your idea to business executives, realize that these people deal at a whole different level of engagement every single day. Therefore, you have to be very professional in your execution if you want to be taken seriously.
When I joined Microsoft in 1997, part of my job was to visit IT organizations and discover the issues they faced when managing such extensive equipment. While visiting a 10,000-machine datacenter, I was shown how workers had to cycle the power every three hours on every machine because of software and hardware reliability issues.
Later, I went back to my office and started thinking about creating health monitoring software automated actions that could address all of the problems I had just learned about in the datacenter. After speaking with various executives, I finally found myself giving a presentation to Bill Gates. My idea became a reality; today, it is sold as Microsoft System Center Operations Manager.
Based on my experiences, here is a checklist of important questions you need to answer before you consider approaching anyone about your innovation:
  • What is the market opportunity?
  • Who is the competition?
  • Who are the customers and what are their scenarios?
  • How do we differentiate our offering?
  • What are the risks?
  • What key partnerships need to be established?
  • How does it fit in with the overall company strategy?
  • What is the timeline and cost?
The next step is to find the right people to influence, give them a brief sales pitch, and then ask for a more appropriate time to give them a complete demonstration.
At the presentation, you need to command attention right from the start. To do this, just keep the presentation short and simple. Begin by framing the problem and then propose your solution. Conclude with your requests: Do you want funding? Do you want them to point you to other people with whom you can collaborate? Be thoroughly prepared to address all of their concerns, and don't skimp on your salesmanship skills. The art of getting people to listen is as important a part of innovating as having a great idea.

Eric N. Bush is a Software Development Lead in the Engineering Excellence group at Microsoft. He works as an internal consultant and classroom instructor driving engineering and organizational improvements in the areas of people skills, process, and technology. You can reach Eric at ericbush@microsoft.com.