August 2007 August 2007
Gadgets: Build Your Own Windows Vista Sidebar Gadget
A Sidebar gadget is a powerful little too that's surprisingly easy to create. Get in on the fun with Donavon West. Donavon West
WPF: Flexible Content Display With Flow Documents
Markus Egger discusses all the flexibility and power of flow documents in Windows Presentation Foundation Markus Egger
OPC: A New Standard For Packaging Your Data
Open Packaging Conventions (OPC) defines a structured means for storing application data together with related resources. Learn about the benefits here. Jack Davis and Andrey Shur
Office Apps: Extend Your VBA Code With VSTO
VSTO brings you the full feature set of Visual Studio including LINQ, WPF, WCF, and the .NET Framework 3.5. Paul Stubbs and Kathleen McGrath
Excel Services: Develop A Calculation Engine For Your Apps
The Excel Services architecture lets users design their own algorithms and share workbooks on a server. Vishwas Lele and Pyush Kumar
Columns
Editor's Note: Thanks for the Memories!: Stephen Toub gets nostalgic as he prepares to leave MSDN Magazine.Stephen Toub | Toolbox: Create Graphs, Send E-Mail, Spell-Check, and More: Turn boring data into stunning graphs, create, parse, send, and receive mail messages from code, and more.Scott Mitchell | CLR Inside Out: Collections Best Practices: This month Inbar Gazit shows readers how to get the most out of the System.Collections namespace.Inbar Gazit
Basic Instincts: Query Comprehensions: Scott Wisniewski digs into LINQ and builds a custom query provider to create queries in real-world apps.Scott Wisniewski | Office Space: Solution Deployment with SharePoint 2007: WSS 3.0 introduces a new deployment mechanism that lets you move your development efforts into a staging or production environment.Ted Pattison | Cutting Edge: Canceling Server Tasks with ASP.NET AJAX: This month Dino explains how to remotely cancel tasks running on the server using ASP.NET AJAX.Dino Esposito
Extreme ASP.NET: Web Client Software Factory: The Web Service Software Factory is designed to provide guidance and enhanced tools for building Web services using ASMX or WCF.Fritz Onion | Foundations: Declarative WCF Security: Juval Lowy designs easily configured security settings for applications built on Windows Communication Foundation.Juval Lowy | Netting C++: Configuration with XML: Stanley B. Lippman explores the use of C++/CLI as a pure Microsoft .NET Framework-compliant language.Stanley B. Lippman
Windows with C++: Windows Vista Control Enhancements: With the release of Windows Vista, C++ developers will be happy to know there's plenty of magic left to perform. This new column will provide the necessary insight.Kenny Kerr | { End Bracket }: Dependency Management: Dependency management, in which one group provides a software component to be utilized by others, can really benefit from some structure and collaborative efforts.Eric N. Bush


Gadgets
Build Your Own Windows Vista Sidebar Gadget
Donavon West

This article discusses:
  • Anatomy of a Sidebar gadget
  • Coding and debugging JavaScript
  • Flyouts and settings
  • Localization
This article uses the following technologies:
Windows Vista, JavaScript
Contents
What Is a Sidebar Gadget?
XML Definition File
The Main HTML File
JavaScript
Docked and Floating
Flyouts
Options Dialog
Reading and Writing Settings
Localization
What Happened to Alert and Confirm?
Debugging
Packaging Your Gadget
The MSDN Magazine Ticker
You’ve probably heard a lot of buzz lately about gadgets—these are the lightweight applications that you’ve seen, for example, sitting on a Windows Vista™ desktop. There are actually three different types of gadgets supported by Microsoft. Sidebar gadgets are the ones that run on the Windows Vista desktop. Web gadgets are similar, but run on Live.com and Windows Live Spaces pages. SideShow gadgets run on devices such as secondary displays on laptops, remote controls, and computer keyboards.
In this article, I explore the major parts of a Sidebar gadget and then demonstrate how these elements come together to create a sample gadget: an MSDN® Magazine Ticker for the Windows Vista Sidebar. Note that I focus strictly on developing Windows Vista Sidebar gadgets. Thus, when I use the term "gadget," I am referring specifically to a Sidebar gadget. (If you’re interested in developing SideShow gadgets, see Jeffrey Richter’s article in the January 2007 issue of MSDN Magazine.)

What Is a Sidebar Gadget?
A Sidebar gadget can be a powerful and handy little tool. So you might be surprised by how easy they are to create. In fact, if you know HTML, CSS, and JavaScript (and I suspect many of you already do), you’re well on your way.
In its simplest form, a gadget is made up of nothing more than an HTML file and an XML definition file. Of course, most gadgets include other files, such as image files (PNG and JPG), style sheets (CSS), and scripting (JavaScript and VBScript source files). All of the content is stored in a ZIP file that is renamed with a .gadget extension. If you want, you can grab a gadget online, rename it with a .zip extension, and easily explore its contents. I recommend this as a good way to peek into other gadgets and see what they’re made up of.

XML Definition File
The XML definition file, or manifest, is the glue that holds a gadget together. I am not sure about calling this file a manifest since it does not contain links to all of the files in the gadget; it only contains links to the main HTML file (which has links to the other files), a few icon files, and the gadget author’s Web site.
Figure 1 shows a typical skeleton XML definition file. As you can see, it is a standard XML file with a base element of gadget. Here is a list of the elements you should be most concerned with:
Figure 1 Typical XML Definition File
<?xml version=”1.0” encoding=”utf-8” ?>
<gadget>
  <name>Gadget Name Here</name>
  <namespace>YourCompanyNameHere</namespace>
  <version>1.0.0.0</version>
  <author name=”Company Name Here”>
    <info url=”http://contoso.com” text=”Vist our Web site” />
    <logo src=”logo.png” />
  </author>
  <copyright>&#0169; 2007</copyright>
  <description>your gadget description</description>
  <icons>
    <icon width=”64” height=”64” src=”icon.png” />
  </icons>
  <hosts>
    <host name=”sidebar”>
      <base type=”HTML” apiVersion=”1.0.0” src=”gadget.html” />
      <permissions>full</permissions>
      <platform minPlatformVersion=”0.3” />
    </host>
  </hosts>
</gadget>
  • name: Title of your gadget.
  • version: Version number of your gadget.
  • author: Your name or your company’s name.
  • info url: Web site address.
  • info text: Friendly name for your Web site.
  • logo src: Name of company’s logo image file.
  • copyright: Copyright notice.
  • description: Description of the gadget.
  • icon src: Name of icon image file for the gadget.
  • base src: Name of gadget’s main HTML file.
Most of the elements in the definition file are used for displaying the gadget in the gallery. The one truly functional element is the src attribute of the base element—this points to the HTML file that will kickstart the gadget. I make it a practice to name this file gadget.html, but any valid filename will do.

The Main HTML File
The implementation of a gadget is nothing more than an HTML page that is a maximum of 130 pixels wide. Though this isn’t readily evident, the main HTML file (the one referenced in the XML definition file) is actually loaded into an Internet Explorer® 7 window. There is, of course, no chrome surrounding this window and its location is controlled by Sidebar, but everything inside is basically a Web application. You have access to the DOM and most of the APIs as you would with a standard Web page. This means you can use AJAX techniques, create dynamic HTML elements, trap events, and so on. I will do all of these things, and more, as I walk you through the process of building my sample gadget.
You can also use APIs from the Sidebar Gadget Object Model. These APIs provide a way for your gadget to interface with the system. For example, you can read the signal strength of your wireless network card, play a sound file, or determine the CPU usage.
As you look at the HTML for a gadget, you’ll notice that there is absolutely nothing that distinguishes it from HTML that you would code for a regular Web page. Here is the HTML code I use to start practically all of my Sidebar gadget projects:
<!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN”
   “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>
<html xmlns=”http://www.w3.org/1999/xhtml”>
<head>
   <title></title>
   <meta http-equiv=”Content-Type” content=”text/html; charset=utf-8” />
   <link href=”style.css” rel=”stylesheet” type=”text/css” />
   <script src=”local.js” type=”text/javascript”></script>
   <script src=”gadget.js” type=”text/javascript”></script>
</head>
<body>
   <div id=”gadget” class=”gadget”></div>
</body>
</html>
In the sample presented here, all of the HTML for the gadget is created dynamically from within the gadget.js JavaScript file. You could include most of the markup in the main HTML file, but what fun would that be?
Notice that I use CSS style sheets. This is what will drive the look of my gadget. As with any Web page, style sheets are an essential part of designing good-looking gadgets. Thus a good understanding of CSS is critical when producing a professional-looking gadget.
I should point out that while developing the main HTML file for a gadget is entirely like building a page that will be deployed online, the gadget implementation has a major advantage. You don’t have to worry about cross-browser issues. Since your gadget will always be run under Internet Explorer, you can rely on support for Internet Explorer features, such as native PNG alpha support, and there’s no need to write code to account for browser incompatibilities.
When I started developing Sidebar gadgets, I found this very refreshing. Over time, I began to take this for granted and when I recently had to design a Web page, I found myself getting very aggravated when I had to deal with cross-browser issues once again.

JavaScript
If you plan on doing anything interesting within your gadget, you’d better brush up on your JavaScript. You can code with any scripting language that is supported by Internet Explorer 7, but you will find that most of the gadget samples on "the Internets" are written in JavaScript.
If you have worked with JavaScript for any time at all, you have probably heard the phrase "eval is evil." If not, go ahead and look it up online. I’ll wait.
Now that you understand why you should never use eval, I’m going to amend that rule to include that you should never place JavaScript text in an attribute inside of your HTML nor pass a string to setInterval or setTimeout. And when I say that you should never place JavaScript text in an attribute inside your HTML, this includes the body onload attribute, as this is interpreted internally just as an eval statement would be.
I’m sure some of you are saying, "No body onload? How will my code execute?" I suggest that you get in the habit of attaching a function to the onload event of the window object. This code snippet shows how to attach to the onload and onunload events:
function pageLoad() {
    window.detachEvent(“onload”, pageLoad);
    window.attachEvent(“onunload”, pageUnload);
    //page initalization here
}

function pageUnload() {
    window.detachEvent(“onload”, pageUnload);
    //gadget is closing, clean up
}

window.attachEvent(“onload”, pageLoad);
By using this method, function pageLoad is called well after the page loads (when the DOM is complete). Notice that the first thing I do in pageLoad is detatch the event. It is always a good idea to clean up after yourself in JavaScript to prevent memory leaks.
Next, I set up the onunload event that will be called as the gadget is destroyed—either as the user closes the gadget or when Sidebar is shut down. Again, this is done to free up any references that may cause memory leaks. It is also your last chance to save any settings if need be. I’ll talk all about settings a little later in the article.
You may have noticed by now that I’m a bit of a stickler for writing good, clean JavaScript, so please indulge me one more time on the subject. I strongly suggest that you "lint" your code before deployment. This will not only make your code better, but it may even root out some bugs before they happen.
Lint was the original name given to a tool that flagged problem areas in C source code. It is now a general term applied to many source languages. One of the better lint applications for JavaScript is a free online tool called JSLint (available at www.jslint.com).
The May 2007 issue of MSDN Magazine has an excellent feature on JavaScript titled "Create Advanced Web Applications with Object Oriented Techniques" by Ray Djajadinata. You should check it out if you haven’t already done so. This article will go a long way to helping you write a better gadget.

Docked and Floating
When a gadget is nestled in the Sidebar, it is considered to be in the docked state. Once dragged onto the desktop, it is floating or undocked. As I mentioned, the maximum width for a docked gadget is 130 pixels. An undocked gadget has no restriction on width, allowing you to design an expanded view. You can take advantage of this extra room to provide a richer experience. (Microsoft recommends that you don’t exceed 400 pixels square for an undocked gadget.)
Figure 2 shows my MSDN Magazine Ticker gadget (that shows scrolling article headlines from the most recent issue) and a weather gadget (that ships with Windows Vista) in both their docked and floating states. The weather gadget is an excellent example of a gadget that provides a richer experience when floating—it goes from giving just the current temperature to giving a detailed three day forecast. The MSDN Magazine Ticker gadget doesn’t display any additional information; it simply expands to show the same information in a wider format.
Figure 2 Docked and Floating Gadgets
Figure 2 Docked and Floating Gadgets
Note that there is an undocumented minimum height of 57 pixels for a gadget (whether docked or floating). The reason for this minimum is simple: the height of the toolbar when a gadget is in its floating state is 57 pixels (see Figure 3).
Figure 3
You can query the current state of a gadget with System.Gadget.docked. It returns true if docked, false if undocked. There are also two events that you can monitor, System.Gadget.onDock and System.Gadget.onUndock, to determine when the docking state changes.
You cannot simply change the class of the document.body from within an onDocked or onUndocked event to change the height, width, or background image of a gadget (this is contrary to my initial expectations). Instead, you must specifically set the properties of the style object of the body element. To change the background image, use the System.Gadget.background API to set the filename of the background image. Here’s an example:
// called when the docked state changes
function dockStateChanged() {
    if (System.Gadget.docked) {
        System.Gadget.background = “images/background.png”;
        document.body.style.width = “130px”;
    } else {
        System.Gadget.background = “images/background-undocked.png”;
        document.body.style.width = “230px”;
    }
}

Flyouts
A flyout extends the user interface outside of the gadget’s own borders. You can use this window for anything you wish. (There is no maximum size imposed for flyout windows.) A good example of a gadget that uses a flyout is the Stocks gadget that ships with Windows Vista. Clicking on a stock causes a window to fly out to the side of the gadget, displaying a graph of that stock’s activity (see Figure 4). The Live Search gadget also uses a flyout, displaying search results for the query you entered. Figure 5 shows a flyout from our MSDN Magazine Ticker gadget. Notice that when a headline is clicked, the UI extends out from the docked gadget to show the article title, a description, and the author’s name.
Figure 4 Stocks Gadget Displaying a Flyout
Figure 4 Stocks Gadget Displaying a Flyout
Figure 5 Ticker Gadget Displaying a Flyout
Figure 5 Ticker Gadget Displaying a Flyout
Note that Sidebar gadget flyouts are system modal—only one can be displayed at a time. If the user clicks on Gadget A, causing a flyout, and then clicks on Gadget B, causing another flyout, Gadget A’s flyout will close. Furthermore, when a gadget loses focus, its flyout will close.
The flyout APIs are exposed from the System.Gadget.Flyout object. A flyout lives in a completely isolated window (with its own DOM) from that of the main HTML window. As such, you need to provide a separate flyout HTML file, along with any other necessary supporting files. The flyout file is specified by setting System.Gadget.Flyout.file to the name of the flyout HTML file.
There is an API you can use to facilitate communication between the main and flyout windows. System.Gadget.Flyout.document returns the Document object of the flyout window. You can use this from the main gadget window to dynamically generate HTML in the flyout window.
There are also events that can be captured by both the main and flyout code. These include onShow, which is called just after the flyout document has been created, and onHide, which is called just before the flyout document is destroyed.
You can programmatically show or hide the flyout by setting System.Gadget.Flyout.show to true or false, respectively. And reading this value will tell you the current state of the flyout window.
Keep in mind that you cannot control the location of the flyout. Sidebar determines the location based on screen real estate and the position of the gadget itself. For example, the flyout could be displayed to the left or to the right of the gadget. Or even below it.

Options Dialog
The options dialog allows you to present the user with a list of gadget preferences. The user experience can be most anything you want. The dialog can be just like a regular Web page with radio buttons, checkboxes, and textboxes.
A user can access the options dialog by clicking on the wrench icon in the gadget’s toolbar (see Figure 3), which appears when you hover over the gadget. When a user opens the options dialog of a gadget, Sidebar takes a snapshot of the gadget and displays the image as a small icon (see Figure 6).
Figure 6 Options Dialog
Figure 6 Options Dialog
I would like to point out that there is some ambiguity when it comes to the name of this dialog. When you right-click on a gadget, the gadget UI presents a menu that refers to this as "Options." The API, however, refers to this as "settings."
I was a bit surprised to find that user settings are stored in a circa 1990 INI file instead of a more modern XML file. Fortunately, there are gadget APIs to read and write settings, so where and how the settings are stored is of little concern to you or your application.
There are a few caveats when it comes to creating your options UI. The width of the user area of the dialog is limited to a maximum of 300 pixels. There is no height limit, but the Microsoft UX Guide for Windows Vista Gadgets recommends a maximum width of 278 pixels and a height of no more than 400 pixels. If you need more room, you should probably create a tabbed dialog. I won’t talk about how to implement a tabbed UI in this article, but there are plenty of resources on the Web to learn how you can do this.
You need to instruct the gadget to enable the options icon. This is done by setting System.Gadget.settingsUI to the name of the HTML file, generally in the gadget initialization area of your script. You also need to set up a callback function for when the options dialog closes (so your gadget can read the new user preferences). You do this by setting System.Gadget.onSettingsClosed to the name of your handler, as shown here:
System.Gadget.settingsUI = “settings.html”;
System.Gadget.onSettingsClosed = settingsClosed;

function settingsClosed(p_event) {
    //OK clicked?
    if (p_event.closeAction == p_event.Action.commit) {
        //yes, read settings here
    }
}
As you can see from this example, when the handler is called, it is passed an event object— specifically the System.Gadget.Settings.ClosingEvent event object. If the closeAction property of the ClosingEvent object contains a value of commit, this means the user selected OK and you will likely want to read the new settings. Otherwise, the options dialog was canceled and you can bypass reading of the preferences.

Reading and Writing Settings
Settings are written using System.Gadget.Settings.write or System.Gadget.Settings.writeString. Both are passed a key/value pair.
Conversely, settings are read with System.Gadget.Settings.read or System.Gadget.Settings.readString. Both of these functions take a key and return a value. If the key does not exist (for instance if it has never been written) both will return a value of undefined.
JavaScript is not a strongly typed language and if you use write and read, Sidebar will attempt type conversion. If you want to be absolutely sure what is written and read, consider using writeString and readString as these will assume strings. Depending on the type of data in question, you will have to determine which methods will work best.

Localization
Sidebar gadgets support localization by way of "localized folders." Whenever Sidebar tries to load an asset (a gadget manifest, style sheet, image file, JavaScript file), it searches for the file in folders in the following order:
  • Full locale (en-us, es-us, ja-jp)
  • Language portion of the locale (en, es, ja)
  • Gadget root folder
For example, if you are running a US version of Windows Vista and your preferences are set to Spanish, Sidebar will first look in the folder es-us. If the file is not found there, Sidebar will then search the es folder. And finally, if the file is still not found, Sidebar will search the gadget root folder.
Language is obviously important, but why is location important? Location is actually very important for certain gadgets. Consider the weather gadget. You may display the word sunny to users in both the United States and the United Kingdom, but the location will determine whether you should display the temperature in Fahrenheit (US) or Celsius (UK).
Many developers will likely support an English-only gadget, but if locale is important to you I recommend that you put all of your language-specific strings and location-specific variables in a single JavaScript file called local.js and place this file in the gadget root folder. Then create a folder for each locale that you are going to support and copy the translated versions of local.js into their respective folders. Here is an example of a local.js file in the root folder, representing en-us:
var L_Hello = “Hello”;
var L_Degrees = 0;
The same file in the \es-es folder may look like this:
var L_Hello = “Hola”;
var L_Degrees = 1;
When you want to display Hello, you would use the variable L_Hello rather than the hardcoded string. And when querying the weather feed, you would use L_Degrees to request the proper format. This produces a greeting in the appropriate language and gives the temperature according to the user’s preferences. In your main JavaScript code, when you want to use a string or determine what to use for degrees, you would do something like this:
element.innerHTML = L_Hello;
if (L_Degrees === 0) {
    //load the Fahrenheit feed
} else {
    //load the Celsius feed
}
As you can see, your code will react differently depending on which local.js file is loaded by Sidebar. Cool, huh?
Note that with the English local.js in the gadget root, the gadget will still function for non-supported locales, albeit in the fallback language (in this case English). It is very important to support a default language in your gadget’s root folder. If you don’t provide a default language and someone uses a language that isn’t supported by your gadget (meaning a language for which you have not created a subfolder), your gadget will display blank strings.

What Happened to Alert and Confirm?
We’ve all been on Web sites and received the "Invalid input, please try again" alert and the "Delete record. Are you sure?" confirmation dialog. Developers often want to pop up informational messages like these.
Sidebar, however, has disabled these JavaScript functions. Using popup dialogs goes against the Windows Vista UX guidelines for Sidebar gadgets. If you still feel compelled to use popups, you can emulate these functions.
To do this, first insert a simple one line tag into the head element of your HTML:
<script src=”alert.vbs” type=”text/vbscript”></script>
Then create a file that contains the code shown in Figure 7 and name it alert.vbs. Now you can continue to use alert and confirm as you wish.
Figure 7 Emulating Alert and Confirm
‘simulate JavaScript alert() function
sub alert(prompt)
    MsgBox prompt, 48 , “Sidebar Gadget”
end sub

‘simulate JavaScript confirm() function
function confirm(prompt)
    dim res
    res = MsgBox (prompt, 33, “Sidebar Gadget”)
    if res=1 then
        confirm = true
    else
        confirm = false
    end if
end function

Debugging
Debugging JavaScript has always been tricky. Many developers have resorted to placing alerts in their code to display the values of certain variables. This, however, is not an elegant solution. A better method involves using Visual Studio® (or even the free Visual Web Developer™ 2005 Express Edition).
Simply place debugger statements in your code, wherever you need to check the value of a variable, then run the gadget. When JavaScript executes the debugger statement, you should see a popup that asks if you want to debug the application (see Figure 8). Choosing Yes will allow you to browse the entire gadget environment, including the DOM, and view the value of any variable created. Your gadget is essentially frozen in time.
Figure 8 Do You Want to Debug Your Gadget?
Figure 8 Do You Want to Debug Your Gadget?
Figure 9 shows a section of debugged code where the value of the variable refreshRate is 24. This form of debugging is very powerful. Once you try it, you’ll never go back to spattering your code with alert statements. Note that JavaScript debugging is vastly improved in the next version of Visual Studio code-named "Orcas." You can read more about these new features at JScript Debugging in Visual Web Developer Orcas, and you can download "Orcas" Beta 1 at msdn2.microsoft.com/aa700831.
Figure 9 Debugging a Gadget in Visual Studio
Figure 9 Debugging a Gadget in Visual Studio

Packaging Your Gadget
The simplest way to create a package is with Windows Explorer. Select the files that make up your gadget, right-click, and select Send To | Compressed (zipped) Folder.
You can also package your gadget as a CAB file, which is the Microsoft native compressed archive format. Just generate the CAB file and then rename it with a .gadget extension. (By the way, if you ever try renaming a .gadget file with a .zip extension and Windows Explorer complains when you try to open this ZIP file, try renaming the file with a .cab extension instead.)
There are a few different methods you can use if you want to programmatically generate a gadget. In my projects, I create the following batch file called make.bat:
@echo off

rem ** remove/create a test gadget folder
rd “%LOCALAPPDATA%\Microsoft\Windows Sidebar\
    Gadgets\MSDNSample.gadget\” /s /q
md “%LOCALAPPDATA%\Microsoft\Windows Sidebar\Gadgets\MSDNSample.gadget\”

rem ** copy all of the files into test area
xcopy . “%LOCALAPPDATA%\Microsoft\Windows Sidebar\
    Gadgets\MSDNSample.gadget\” /y /s /q /EXCLUDE:exclude.txt

cd “%LOCALAPPDATA%\Microsoft\Windows Sidebar\Gadgets\MSDNSample\”
cabarc -r -p n “%HOMEPATH%\Documents\MSDNSample.gadget” *
This uses the Cabarc.exe utility to generate a CAB file. (Cabarc.exe is a free Microsoft tool that is installed with Visual Studio. It’s also available separately as part of the Cabinet SDK, which you can download from support.microsoft.com/kb/310618.)
From within Visual Studio 2005, I run make.bat, which I have set up as an external tool. The batch file creates a folder under the user gadgets folder (which is where gadgets are created when installed) and copies all of the gadget files into the new folder. My batch file also generates a .gadget file that is ready for distribution and places it in the Documents folder. When you use this method, there is no need to double-click on the .gadget file to install the gadget on your development machine. I love the technique as it allows me to keep other source files in my Visual Studio project folder (such as Photoshop PSD files) that I don’t want to package with the gadget. If you do this, just be sure to place the names of the files you want to exclude in the exclude.txt file.

The MSDN Magazine Ticker
Now that I’ve covered all the basic components of a gadget, I’d like to present a sample gadget that pulls all the parts together. The MSDN Magazine Ticker gadget is available as a download so you can install it or simply explore its contents.
I wanted my sample gadget to use all of the major features of a Sidebar gadget (flyouts, docked versus floating states, options, localization, and so on). But I also wanted the sample gadget to be simple. And it had to be somewhat useful.
I settled on the idea for an RSS reader that displays news headlines. As this sample is for an MSDN Magazine article, it seemed like an obvious choice to use the RSS feed for all articles in the current issue of the magazine (this is available at msdn.microsoft.com/msdnmag/rss/rss.aspx). Imagine how dynamic and informative you could make this gadget if you were to use a feed of news or sports headlines, or even multiple feeds federated into one data source.
Windows Vista Sidebar already ships with a feed headlines gadget. What makes my sample unique is the way the information is presented. The headlines are displayed in a news ticker fashion, like you see at the bottom of your TV screen on news channels.
My sample gadget is also a lot more compact that the feed gadget that ships with Windows Vista—it’s a mere 57 pixels high (versus 175 pixels for the default gadget). This is great for on a smaller screen (or a cluttered Sidebar) where real estate is at a premium.
The gadget follows all of the recommendations given in this article: it uses a flyout to display detailed information when you click on a headline, it lets you select from three different feed reload times in the Options dialog, it changes size when moved to a floating state, and it supports several languages through the use of localization folders.
I’ve only scratched the surface of Sidebar gadgets to demonstrate how easy it is to get started. There’s so much more you can do. The Additional Resources sidebar has more information.

Donavon West is an independent consultant, CTO for LiveGadgets.net, and a Microsoft MVP for Windows Live Development. Originally from Chicago, he now lives in the Baltimore/Washington DC area. He is very active in the gadget development community, both with Windows Vista Sidebar and Web gadgets. You can reach him through his Web site at www.livegadgets.net.


WPF
Flexible Content Display With Flow Documents
Markus Egger

This article discusses:
  • Windows Presentation Foundation
  • Flow Documents
  • Flow Document rendering
  • Flow Document editing
This article uses the following technologies:
.NET Framework 3.0
Contents
Flow Document Basics
Viewing Flow Documents
Displaying Flow Documents
Creating Flow Documents
Exploring Layout Possibilities
Above and Beyond
Windows® Presentation Foundation (WPF) provides a great set of features. So many features, in fact, that even some very significant ones don’t get nearly the attention they deserve. A perfect example is the Flow Documents feature, which allows developers to create documents natively in WPF. In "XPS Documents: A First Look at APIs For Creating XML Paper Specification Documents" in the January 2006 issue of MSDN® Magazine, Bob Watson took a close look at XPS Documents in WPF, but Flow Documents are different. XPS (the XML Paper Specification) is geared towards printing and page-oriented content, while Flow Documents are aimed at screen reading and provide a more dynamic and arguably more sophisticated model. Flow Documents work for almost everything related to text content, from product descriptions to entire books.
The representation of text is undoubtedly one of the more important UI features. In WPF interfaces, you often use controls such as Label to display text. In many scenarios, however, you need more than a simple display of a few words. Flow Documents provide a more sophisticated approach, though they are fundamentally very simple. They define text flows in a fashion reminiscent of HTML documents, but they are more capable and provide significantly more advanced layout options.
Flow Documents are typically defined using Extensible Application Markup Language (XAML), the XML-based standard markup language. XAML is particularly intuitive for Flow Documents, mainly because of the similarity to HTML. The following Flow Document sample creates a single paragraph of text with simple bold formatting applied to a few words:
<FlowDocument
  xmlns=’http://schemas.microsoft.com/winfx/2006/xaml/presentation’
  xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>
  <Paragraph>The quick <Bold>brown fox</Bold> jumps over the lazy dog.
</Paragraph>
</FlowDocument>
As you can see, the similarities to HTML are even more apparent than in other XAML UIs. The actual element names are different, but at least for simple documents, the paradigm is very similar. Flow Documents generally start with a FlowDocument root element that contains a number of blocks. Blocks are elements within the flow, typically paragraphs of text as in the example above (although there are other block types as well). Paragraphs can in turn contain other elements, such as the two bolded words in this example. Note that as with any other XAML document, the root element must have the XAML-specific namespace definitions in order to be recognized. This is an implementation detail specific to XAML and has nothing to do with Flow Documents. Note that the namespace definition is only necessary in standalone Flow Documents. (Flow Documents can be part of a larger XAML UI, in which case the UI’s root element gets the namespace definitions.)
Of course, users never see the XAML for Flow Documents (unlike HTML source that can be viewed in the browser), just as they don’t see the XAML for any other UI element. Instead, users see the final rendering of the document. For this particular example, you can see the result in a number of ways. Perhaps the simplest is to type it into XamlPad, a utility that ships with the Windows SDK (see Figure 1).
Figure 1 Very Simple Flow Document Displayed in XamlPad
Figure 1 Very Simple Flow Document Displayed in XamlPad
Of course, this is a very simple example; the definition of a document and the embedded layout can be much more sophisticated. Flow Documents support all the formatting you’d expect, such as italic, underline, font colors and typefaces, and much more. Figure 2 shows a slightly more advanced example, the result of which can be seen in Figure 3.
Figure 2 More Formatting and a Bulleted List
<FlowDocument
  xmlns=’http://schemas.microsoft.com/winfx/2006/xaml/presentation’
  xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>
  <Paragraph FontFamily=”Calibri” FontWeight=”Bold” FontSize=”24”>
    WPF Flow Documents</Paragraph>
  <Paragraph>WPF Flow Documents are <Italic>quite sophisticated</Italic>.
    They support all common layout options, as well as many you probably
    do <Span Foreground=”Red”>not expect</Span>.</Paragraph>
  <List>
    <ListItem><Paragraph>First List Item</Paragraph></ListItem>
    <ListItem><Paragraph>Second List Item</Paragraph></ListItem>
    <ListItem><Paragraph>Third List Item</Paragraph></ListItem>
  </List>
  <Paragraph>Of course, Flow Documents also support the definition
    of <Bold><Span FontFamily=”Comic Sans MS” FontSize=”24”
    Foreground=”Blue”>in line</Span></Bold> font sizes and font faces.
  </Paragraph>
</FlowDocument>
Figure 3 Flow Document with Slightly More Advanced Formatting
Figure 3 Flow Document with Slightly More Advanced Formatting
This example shows several paragraphs with inline formatting. It also provides a first example of another type of block, namely a List, which, not surprisingly, contains a number of list items. Note that each list item in turn is also just a container for more blocks. So rather than simply putting text inside of a list item, I added a paragraph element to each. I could have added multiple paragraphs to each item or any other block type for that matter. This lets you create advanced layouts within a single bullet point of a list, something that is generally a problem in formats such as HTML, which only flow simple text strings into each list element.

Flow Document Basics
Now that you’ve seen some basic Flow Documents, let’s take a step back and look at some of the fundamentals again. As you’ve already seen, Flow Documents are collections of blocks. Internally, all blocks are WPF classes that derive from the System.Windows.Documents.Block class. Block in turn derives (a few steps further up the chain) from ContentElement, which is a fairly low-level class in WPF specifically optimized for document definition. This approach is somewhat similar to the controls you use to define WPF interfaces, which all derive from UIElement. Both inheritance trees are similar in concept, but not exactly the same. This means that WPF controls and blocks cannot be combined directly. For example, a button’s caption can’t be set to a paragraph of text, nor can a paragraph directly contain a button. There are some subtle differences in these controls and blocks, which result from the fact that layout within a content control and layout within a block work quite a bit differently. Luckily, the gap that needs to be bridged between the two types of WPF elements is rather narrow. Buttons, for instance, can contain TextBlock objects, which are made up of formatted text, and blocks can contain any WPF control by means of the special BlockUIContainer block class. This means that Flow Documents can contain all kinds of WPF elements (including interactive user interfaces, media, and 3D elements), and, looking at it from the other angle, Flow Documents can become part of any WPF user interface either as an advanced layout element for control contents or as a true document, such as the description of an item in a point of sale application.
The list of available blocks is theoretically open-ended, since developers can derive their own block classes and thus create their own enhancements to the document-rendering engine. This provides freedom that is not available in any other document-rendering engines that I am aware of. However, the number of blocks the average document creator is exposed to is typically limited. A list of the most important types of blocks is shown in Figure 4.
Figure 4 Important Block Types

Block | Description
Paragraph | Contains (potentially richly formatted) text.
List | Contains lists of various kinds (numbered, bulleted, and so on).
Table | Contains tables similar to those in Microsoft Word or HTML.
BlockUIContainer | Contains various UI elements that are made part of the overall flow.
Section | Contains a group of other blocks. Sections are handy for applying common attributes to a group of blocks, such as the same font attributes to multiple paragraphs.
When creating WPF Flow Documents in XAML, you really just instantiate certain types. Consider the following XAML snippet (from here on, I’ll omit the namespace definition to keep the examples simple):
<FlowDocument>
  <Paragraph>Hello World!</Paragraph>
</FlowDocument>
This instantiates a FlowDocument class and a Paragraph class (which has its text set to "Hello World!"). The paragraph is added to the FlowDocument’s blocks collection. Note that as with all XAML, the element names are case-sensitive and map exactly to the classes that are available as part of the WPF. You could also create the same document programmatically, like so:
FlowDocument doc = new FlowDocument();
Paragraph para = new Paragraph();
para.Inlines.Add(“Hello World!”);
doc.Blocks.Add(para);
Of course, this is much less intuitive than the declarative approach XAML provides, so the programmatic approach is only taken in special scenarios. (I sometimes use this approach when I need to create a richly formatted report that is more of a real document, rather than the tabular output created by many reporting engines.)
In many scenarios, paragraphs have richly formatted content themselves, which is also done by instantiating classes, like this:
<Paragraph>Hello <Bold>World!</Bold></Paragraph>
Here, the paragraph contains two segments of text—"Hello" (which uses the default format) and "World!" (which is bold). This is interesting because it means that this XAML doesn’t just instantiate a paragraph and set its text as a simple string; instead it creates a paragraph with two child segments, each of which contains text with different formatting. In WPF, these segments are called inlines. Just like a Flow Document can contain multiple blocks of various types, paragraphs can contain inlines of various types. Inlines come in a number of variations. Some inlines are so-called Spans, which represent a segment of text with certain formatting options applied. The Bold element used in this example is a special case of a Span, with the default font weight set to bold. Another type of an inline is a Run, which is a segment of text with default formatting. Therefore, the XAML above is really just shorthand for this:
<Paragraph>
  <Run>Hello </Run>
  <Bold>World!</Bold>
</Paragraph>
Of course, it is much more convenient that you do not have to define each inline in XAML, but if you were to create the same example programmatically, it is important to understand the concept of inlines, since they cannot be omitted in code. Here is the programmatic equivalent of the previous two XAML examples:
Paragraph para = new Paragraph();
para.Inlines.Add(new Run(“Hello “));
Bold b = new Bold();
b.Inlines.Add(“World!”);
para.Inlines.Add(b);
Bold is a special version of a Span, with its default font weight set to bold; the Bold type is subclassed from Span and overrides the FontWeight property. There are similar special Spans, such as Italic and Underline. However, these special Spans are not strictly needed, because you could also use a default Span and set the appropriate properties instead:
<Paragraph>Hello <Span FontWeight=”Bold”>World!</Span></Paragraph>
Of course, the ability to directly specify attributes such as bold and italic by wrapping a certain section of text into Bold and Italic tags is very convenient and intuitive, so it is much more common to use <Bold> than <Span FontWeight="Bold">. Nevertheless, the <Span> element is very useful since there are numerous properties that can be set beyond simple font bolding, and most of those formatting options do not have individual Span types. In fact, for many very common formatting options, there is no special Span. A typical example is setting font faces. Unlike HTML, Flow Documents do not have <Font> elements. Instead, fonts are set like so:
<Paragraph>Hello <Span FontFamily=”Comic Sans MS” FontSize=”24”>
    World!</Span></Paragraph>
Many properties, such as FontFamily, can be found consistently on all Flow Document classes. For instance, if you wanted to set the font for a complete paragraph rather than just an inline, you can do it without a Span:
<Paragraph FontFamily=”Comic Sans MS” FontSize=”24”>Hello World!</Paragraph>
There are also inlines other than Spans and Runs. Here some of the other more interesting inlines:
Figure Figures are somewhat unusual inlines, because they contain blocks. Thus, in a way, Figures are almost like mini Flow Documents within Flow Documents. Figures are often used for advanced layout features, such as images within paragraphs where the normal text flows around the image.
Floater Floaters are lightweight figures. They do not support all the figure positioning options, but if all you need is the ability to do simple alignment outside the standard paragraph alignment, floaters may be useful.
LineBreak LineBreak elements do exactly what their name suggests: they introduce line breaks within paragraphs.
InlineUIContainer The InlineUIContainer is the inline equivalent of the BlockUIContainer. If you need to combine any sort of WPF control with your other inlines (such as having a button flow inside a paragraph’s text), the InlineUIContainer is what you need.
Figures are used all the time in Flow Documents (as are LineBreaks, but they hardly require a detailed discussion). The following example uses a figure to display an image as part of a larger flow:
<Paragraph>
  <Figure Width=”200”>
    <BlockUIContainer>
      <Image Source=”Pictures\Humpback Whale.jpg” />
    </BlockUIContainer>
    <Paragraph Foreground=”Blue” FontFamily=”Consolas”>
        The Whale</Paragraph>
  </Figure>
  The quick brown fox jumps over the lazy dog. The quick brown...
</Paragraph>
Note that there is no Image block in WPF Flow Documents. Instead, images are embedded as BlockUIContainers with standard WPF Image controls. (The same approach is used for content such as videos or interactive 3D models inside of Flow Content). Figure 5 shows the rendering of a document similar to this one.
Figure 5 Text Flows around Image and Caption
Figure 5 Text Flows around Image and Caption

Viewing Flow Documents
You have now seen how to create some simple Flow Documents and view them in XamlPad. What I’ve ignored so far is how you would view Flow Documents in the wild. After all, you wouldn’t expect your users to bring out XamlPad and paste the document’s XAML. One way of viewing a XAML Flow Document is to save it as a file with a .xaml extension and double-click it in Windows Explorer. This launches the default application associated with XAML files (typically Internet Explorer®), which shows the document. Figure 6 shows the result.
Figure 6 XAML Flow Document in Internet Explorer
Figure 6 XAML Flow Document in Internet Explorer
The fact that Internet Explorer (and other browsers) can display XAML content is of particular interest, since this is your ticket to displaying Flow Documents as part of your Web applications. In other words, if you upload XAML Flow Documents to your Web server and someone browses to the file, he will see something similar to Figure 6 (assuming the user has the Microsoft® .NET Framework 3.0 installed). Of course, this also works dynamically. If your ASP.NET Web application (or any other server-side technology) generates a XAML Flow Document on the fly and returns it as its output (and assuming the content type is set appropriately to "application/xaml+xml"), the user will see a Flow Document as part of your Web application, which of course is extremely useful in many scenarios. Figure 7 shows a simple ASP.NET page that generates a Flow Document.

Displaying Flow Documents
You may have noticed that whenever a Flow Document is displayed (either in the browser or in XamlPad), there seems to be a tad more in the display than just the document itself. In particular, there are controls rendered along the bottom of the document. As you can see in Figure 8, by default Flow Documents are rendered by means of a FlowDocumentReader control, which provides a set of standard features such as zoom, paging, various view modes, and even a find feature. As it turns out, Flow Documents need to be hosted by some sort of control that is capable of displaying them. The default viewer for Flow Documents is a FlowDocumentReader control, which is instantiated automatically unless you explicitly use a different control. WPF currently provides three different controls for viewing Flow Documents:
Figure 8 Control Buttons in the FlowDocumentReader Control
Figure 8 Control Buttons in the FlowDocumentReader Control
FlowDocumentScrollViewer This control displays documents in a continuous flow with a scrollbar, similar to Web pages or the Web Layout in Microsoft Word. Figure 9 shows a document in a scroll viewer.
Figure 9 Using the FlowDocumentScrollViewer Control
Figure 9 Using the FlowDocumentScrollViewer Control
FlowDocumentPageViewer This control displays Flow Documents in individual pages, with page flipping instead of scrolling. This is similar to the Full Screen Reading mode in Word. Figure 10 shows a page viewer. There the document from Figure 9 is rendered in a FlowDocumentPageViewer control and the scrollbar is replaced by a paging mechanism. The simple flow layout approach has been replaced with an advanced, multi-column, paged layout.
Figure 10 Using the FlowDocumentPageViewer Control
Figure 10 Using the FlowDocumentPageViewer Control
FlowDocumentReader This control combines the scroll viewer and the page viewer and allows the user to switch between the two approaches. This is the default control used for Flow Documents, and it is often a great choice for applications that feature sophisticated text display. In Figure 11, the same document shown in Figures 9 and 10 was rendered in a FlowDocumentReader, which combines the scroll viewer and page viewer approaches. In addition, it enables the search feature which is hidden by default in the other controls (the other viewers do support find, and it can be shown either by executing the ApplicationCommands.Find command or by using Ctrl+F from the keyboard). The reader control also supports a multi-page view, which slightly alters the page-based rendering and the way columns and figures are rendered.
Figure 11 Using the FlowDocumentReader Control
Figure 11 Using the FlowDocumentReader Control
The control you choose will depend on your situation, though the FlowDocumentReader is compelling for all but the most basic uses. It is versatile and powerful and it supports paged layouts, which is superior to scrolling in many scenarios. A detailed discussion of the topic is beyond the scope of this article, but it turns out that scrolling and associated effects, such as doubling, is one of the main reasons people prefer print over digital text. The paged approach is more natural in many situations and should help to make digital reading more widely accepted.
So how do you define which control to use? A simple—though rather brute-force—approach is to add the desired control to the document’s XAML:
<FlowDocumentScrollViewer
  xmlns=’http://schemas.microsoft.com/winfx/2006/xaml/presentation’
  xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>
  <FlowDocument>
    <Paragraph>The quick <Bold>brown fox</Bold> jumps over the lazy
        dog.</Paragraph>
  </FlowDocument>
</FlowDocumentScrollViewer>
In this example, the document root has been made a FlowDocumentScrollViewer tag. This means you are not just defining a pure document anymore. Instead, you are defining a complete XAML interface, which happens to be using a scroll viewer as its root. The content of the scroll viewer is the Flow Document from the very first example. (Note that the namespace definitions are now with the scroll viewer tag, rather than the Flow Document tag). Figures 9 through 11 were created using this approach, with different viewer controls used as the root element
Why do I call this a brute force approach? Because, from an architectural point of view, it has some problems caused by mixing the user interface definition with its actual data. It is much more desirable to keep the document separate from its interface. Mixing the reader with the document is a little like creating a SQL Server™ table and somehow defining that it can only be displayed in a Windows Forms DataGrid. There are several ways of keeping the document separate from the UI definition. If you want to display Flow Documents as part of a Web application using the demonstrated ASP.NET approach, you can define the ASP.NET page with the desired viewer control and simply merge in the actual content (which is stored separately, perhaps in a database) using standard ASP.NET code.
In a typical WPF application, on the other hand, you can simply define your user interface using standard WPF, Windows, and XAML Browser Application (XBAP) approaches, and then load your document dynamically. Figure 12 shows a simple example that uses a fictitious library of my articles, which are displayed in a listbox in the top-left corner. When the user selects an article from the list, the document is loaded dynamically into the Flow Document Reader control that takes up most of the form. Note that standard WPF techniques, such as alpha blending, work in this setup. You’ll notice that the actual Flow Document is semi-transparent and the photo of me in the background shines through Also note that the app uses a listbox, imagery, a label, and a FlowDocumentReader control to create a library of fictitious articles.
Figure 12 Using Listbox, Image, Label, and FlowDocumentReader Controls
Figure 12 Using Listbox, Image, Label, and FlowDocumentReader Controls
The trickiest part of this example is loading the actual document into the viewer control. This is done by means of the System.Windows.Markup.XamlReader class, which allows for the dynamic loading of any XAML content, including, but not limited to, Flow Documents. Here is the line of code I wired up to the selection change event of the listbox:
documentReader.Document =
    (FlowDocument)XamlReader.Load(
File.OpenRead(fileName));
The Load method returns an object, since the root element in the XAML file could represent many different types. In my example, I know that the return value is a FlowDocument, so I simply perform a cast and assign that document to the FlowDocumentReader control’s Document property (I named the control instance documentReader in this example). Keep in mind that this is just an example. Production-quality code definitely needs some error handling here as well.
Note that all the things you know about WPF apply to this example. For instance, the reader controls are just standard WPF controls that support styling. This means you can completely change the appearance of all UI elements such as the zoom bar, the view mode switches, or the paging controls. (The only element you have limited control over is the search box, though you do not have to use it at all if you don’t like it.)
Also, while my example shows a Windows-based application, the same application could be deployed as an XBAP and run inside a Web browser (again, of course, assuming the user has the .NET Framework 3.0 installed). Note that Microsoft Silverlight™ (formerly code-named "WPF/E") is not enough as Silverlight supports only a subset of WPF and does not support Flow Documents.

Creating Flow Documents
How are Flow Documents authored? Of course developers can always author Flow Documents using low-level tools such as XamlPad. However, in real-life settings, this is unlikely. Typically, Flow Documents are either created using WYSIWYG editors or by means of a content transformation from an existing document format. Since Flow Documents can be defined in XAML, it is particularly easy to convert existing XML content. However, it is also possible to convert HTML and Word documents with a reasonable amount of effort (although coding is required, as no out-of-the-box tools have emerged for this as of yet).
For WYSIWYG editing, WPF provides a ready-made control. The WPF RichTextBox control can edit XAML Flow Documents natively. The control’s name incorrectly suggests that it is made for Rich Text Format (RTF). Though it supports RTF, this control really operates mostly on FlowDocuments. In fact, the control really mirrors the Flow Document view controls, except that it supports editing. Some would even argue that the RichTextBox control should be considered another way of displaying Flow Documents.
Type the following example into XamlPad to see the RichTextBox control in action:
<RichTextBox
  xmlns=’http://schemas.microsoft.com/
      winfx/2006/xaml/presentation’
  xmlns:x=’http://schemas.microsoft.com/winfx/2006/xaml’>
  <FlowDocument>
    <Paragraph>
      The quick brown fox jumps over the lazy dog.
    </Paragraph>
  </FlowDocument>
</RichTextBox>
Just like the reader controls, the RichTextBox has a Document property, which you can automatically populate with a Flow Document in this session. This actually creates a UI that looks very similar to the FlowDocumentScrollViewer control, except that the text is editable. Note that this textbox control always handles Flow Documents in a scroll fashion. There is no way to edit Flow Documents in a RichTextBox in paged or multi-column mode. However, the result of the editing operation is a standard Flow Document that can be displayed in any of the viewer mechanisms you’ve seen already, including multi-column and paged modes.
One of the features of RichTextBox worth mentioning is the integrated spell checking. You can enable it like so:
<RichTextBox SpellCheck.IsEnabled=”true”>
  <FlowDocument>...</FlowDocument>
</RichTextBox>
Figure 13 shows the spell checker in action.
Figure 13 RichTextBox Control with Spell-Checking
Figure 13 RichTextBox Control with Spell-Checking
The only tricky part about using the control is loading and saving. In most scenarios, you probably won’t code the content of the RichTextBox into the UI XAML as in the earlier example. Instead, you will load and save the document dynamically. The load operation for text in a RichTextBox is identical to loading a Flow Document for a viewer control (see above). Saving a document is essentially the opposite: you take the document object and serialize it back to XAML, which can be done like so:
   System.Windows.Markup.XamlWriter.
   Save(richTextBox.Document)
This returns the XAML as a string, which you can then store to a file or a database or use any other way you can imagine.
The RichTextBox is very handy, but a word of caution is in order. While Flow Documents represent the most sophisticated technology available for rendering on-screen documents, the RichTextBox control is nowhere near as sophisticated. It is great for editing small documents and snippets of text, but you won’t be writing a book, magazine, or marketing brochure with it. For such long formats, it renders too simplistically, since it doesn’t support anything but a scroll layout (which means that there is no good visual way to create the advanced layouts I will discuss shortly). Also, the approach to saving documents is often unsatisfactory. The XmlWriter class simply uses a live, in-memory document and turns it into XAML, but unfortunately, it isn’t aware of many concepts that are important for large-scale Flow Document efforts, such as styles. As a result, the XAML faithfully preserves the look of the document, but it is often nasty and very large. The RichTextBox control is certainly useful, but don’t expect it to be a desktop publishing solution for on-screen content (although such an app is sorely needed).

Exploring Layout Possibilities
Now that you understand how to author and view Flow Documents, let’s return to the documents themselves and look at a few more features. Flow Documents are very sophisticated, and exploring all available features is beyond the scope of this article, but there are a few more things I would like to discuss.
One feature that always amazes me is called optimal paragraph. When enabled, this feature aims to distribute white space as evenly as possible within a given paragraph, resulting in a much improved reading experience. Optimal paragraph works particularly well in combination with hyphenation, another built-in feature that (surprise, surprise), performs on-the-fly hyphenation of either entire Flow Documents or individual paragraphs.
Turning optimal paragraph and hyphenation on is a straightforward operation:
<FlowDocument IsOptimalParagraphEnabled=”true”
    IsHyphenationEnabled=”true”>
Figure 14 shows the same document rendered with these features enabled and disabled. The difference between the two versions is subtle, but significant. Note that the left version appears much calmer, mainly due to more evenly distributed but also overall reduced white space between words. Especially when reading large amounts of text on screen, this seemingly small difference is of crucial importance.
Figure 14 Optimal Paragraph and Hyphenation
Figure 14 Optimal Paragraph and Hyphenation
As you’ve seen already, the FlowDocumentReader control takes a multi-column approach to text rendering. This is another very important readability feature because people do not enjoy reading a single line across the entire width of a wide-screen display. The width of the actual columns varies depending on a number of factors, such as total available width for content display, zoom factor, and defined column width. By default, a Flow Document’s column width is 20 times the font size, which with the default font size is approximately 300 device-independent pixels (3 1/8 inch on a size-accurate display). You can override this default setting easily:
<FlowDocument ColumnWidth=”400”>
This results in columns with a width of roughly 400 pixels. However, there are other factors that impact the actual width. If the zoom is at 50 percent for instance, the actual column width is only 200 pixels. Also, the column width as it stands is to be seen more as a minimum column width. This means that if the total width available is 900 pixels, the rendered result contains two columns that are wide enough to fill up the entire 900 pixels, making each column wider than the defined 400 pixels. This is usually desirable as it results in a very appealing rendering. However, if you don’t like that behavior and want columns that are truly 400 pixels wide, you can make sure that the column widths are not flexible:
<FlowDocument ColumnWidth=”400” IsColumnWidthFlexible=”false”>
Now, all the columns are exactly 400 pixels wide (at 100 percent zoom), and remaining space is just left as white space.
Another column-related setting you may want to play with is the gap between columns. This can be adjusted through the ColumnGap property (the setting is also based on device-independent pixels):
<FlowDocument ColumnGap=”25”>
A related setting is the column rule, which allows for the definition of a visual element between the columns. Consider this example (the result of which can be seen in Figure 15):
Figure 15 Flow Document with a Simple Rule between Columns
Figure 15 Flow Document with a Simple Rule between Columns
<FlowDocument ColumnRuleWidth=”5” ColumnRuleBrush=”Red”>
Of course in many publications, documents are not just laid out in simple columns. Often, other elements exist that are taken out of the regular flow. You have already seen such examples with the images placed inside documents. Figure 12 shows an arrangement that is popular with graphic artists. The image sits between two columns, with the text flowing around it, putting the image squarely in the middle of the content without interrupting the flow much in either column. This is a common layout choice that simply hasn’t been available in dynamic on-screen reading environments I’m aware of prior to Flow Documents.
The key to the creation of such layouts is the figure block, which allows for the definition of content that does not flow with the rest of the document. Putting an image inside of a figure tag is one example, but there are many other uses for figures. For instance, you could use a figure to define a heading that stretches across the entire width of the document:
<Paragraph>
  <Figure HorizontalAnchor=”ContentLeft” VerticalAnchor=”ContentTop”
          Width=”1Content”>
    <Paragraph FontSize=”36” FontWeight=”Bold”>Go With
        The Flow</Paragraph>
  </Figure>
  Windows Presentation Foundation in Windows Vista provides a great set
    of features.
</Paragraph>
In this code the figure contains another paragraph, which is the text used as the headline. Note that there are some handy properties you can use to create advanced, flexible documents. Consider the width of the figure, for instance. Rather than setting the width to a certain number of pixels, I set it to be exactly the width of the content, which will automatically adjust the figure’s width to however wide the entire content is.
Take a look at Figure 16. There you’ll notice that the heading (positioned through a figure) is set to span across the width of the entire content, which pushes all four columns down. The image is horizontally and vertically anchored to the page center.
Figure 16 Heading Spans Four Columns
Figure 16 Heading Spans Four Columns
Note that figures with a width relative to the content do not always have to be as wide as the content. The following example for instance sets the width of the figure to be 75 percent of the width of the content:
<Figure Width=”0.75Content”>
The width can also be relative to other things, for instance, to the width of columns. The following example figure is always two columns wide (unless only one column is displayed, in which case the width is reduced to one column):
<Figure Width=”2Column”>
Of course, figure heights can be defined in a similar fashion (although often figures just grow vertically with the content).
Another important aspect is the position of the figure. In the code snippet, it is set to be horizontally anchored to the left and vertically to the top. This means that the figure appears at the top-left corner of the current page of content, regardless of where it is actually defined. In this example, the figure was defined as the first element of the document anyway, but even if there would have been paragraphs before that heading, it would have been moved up and to the left due to these settings. The photos in Figure 12 and Figure 16 have been moved between the columns in a similar fashion, by setting its horizontal anchor to "PageCenter". (The available property values for all these settings can be found in the WPF documentation).
You may have noticed that there was a lot of manual coding. For instance, whenever the font face needs to be changed, you add that information to a block or an inline. So far, this has not been a huge problem since most of the examples were small. However, if you have a 50-page chapter of a book and would like to change the font in every paragraph, then doing so manually every time becomes tedious. Luckily, there is a better approach: like everything else in WPF, Flow Documents support styling. Styles can be defined as named resources in the actual Flow Document. Here is a style that defines font information:
<FlowDocument>
  <FlowDocument.Resources>
    <Style x:Key=”MyStyle”>
      <Setter Property=”TextElement.FontSize” Value=”12” />
      <Setter Property=”TextElement.FontFamily” Value=”Bodoni MT” />
    </Style>
  <FlowDocument.Resources>
  ...
</FlowDocument>
The style can then be applied to paragraphs (and other elements) in the following fashion:
<Paragraph Style=”{StaticResource MyStyle}”>The quick... </Paragraph>
Due to the nature of Flow Documents, styles are very commonly used. I recommend that for anything but the simplest scenarios, you define most of your formatting options in styles, rather than with attributes on individual inlines. Styles keep your documents compact and also make them easier to maintain.

Above and Beyond
I hope this article provided you with a bit more than just a fundamental understanding of Flow Documents and their capabilities, and piqued your interest. There are many more advanced features that are definitely worth looking into, including sophisticated styling of viewer controls, subclassing and extending documents, blocks, and inlines, digital rights management, the text and ink annotation feature, and advanced font formatting.

Markus Egger is an international speaker and the publisher of CoDe Magazine. Markus is also the President and Chief Software Architect of EPS Software Corp., consulting for object-oriented development, Internet development, B2B, and Web services. See his blog at www.markusegger.com/blog.


OPC
A New Standard For Packaging Your Data
Jack Davis and Andrey Shur

This article discusses:
  • How content is organized in a package
  • Parts and relationships
  • Addressable URIs
  • Creating your own OPC file format
This article uses the following technologies:
Open Packaging Conventions, .NET Framework 3.0
Contents
File Format Organization
Packages
Package Parts
Package Relationships
Package Digital Signatures
Package Properties
Creating and Accessing Packages and Parts
Creating and Accessing Relationships
Pack URI
Authentication and Validation
Rights Management and Encryption
Packaging a Web Page
Exploring a Package
Reading a Package
Signing and Validating a Package
Designing a File Format
M any applications integrate content with various additional resources. A Web browser, for example, displays a page that integrates HTML, image files, style sheets, and other types of content. Similarly, a word processor builds a document that combines text, style definitions, image files, and other elements. For the most part, applications use one of two approaches to organize the content: a flat-file organization where content is stored as separate files organized on disk, or binary container files where all the content is packaged in a single custom file. More and more, applications are tending towards the latter.
In the move toward open standards, a new file packaging technology has evolved as part of the 2007 Microsoft® Office System Open XML specification that was recently approved by the ECMA International standards organization. An underlying component of this standard is Open Packaging Conventions (OPC), which defines a structured means to store application data together with related resources using a standard ZIP file. This new packaging technology is already being used in several Microsoft products, including the 2007 Office System applications. The XML Paper Specification (XPS), which defines the new print-spool and document presentation format for Window Vista™, also implements the storage and transport of high-fidelity documents based on OPC. See the "Online Resources" sidebar for more information about the technologies discussed in this article.
So what makes the portable container technology offered by Open Packaging Conventions different? Since it is an open standard, OPC provides a container technology you can use without having to code your own custom binary container files. And it supports a number of enhanced features, including content addressable URIs, MIME types, relational structuring, and authentication and validation. With the Microsoft .NET Framework 3.0, the packaging APIs also offer options for encryption with rights management. Moreover, because they adhere to an open standard, package-based files can be accessed through high-level services such as workflow applications and virus scanners.
API support for the ECMA Open Packaging Conventions is built into Windows Vista and included as part of the .NET Framework 3.0 for use with Windows® XP and Windows Server® 2003. In this article, we’ll examine the new standard, showing how you can use the .NET Framework 3.0 APIs to organize your application’s storage of multiple data streams in a single portable package.

File Format Organization
Microsoft Word 2007 and Excel® 2007 both use Open Packaging for document storage, however, they use different schemas and file organizations. The specific organization of a package’s contents defines its format, which is typically reflected in the extension (like .docx or .xlsx).
Using packaging, you can define your own file organization, filename extension, and file type association for your application. As an example, we’ll create our own custom package type, .htmx, that will store a Web page along with the local files it depends on (style sheets, scripts, image files, and so on).

Packages
A Package is the basic storage unit of the Open Packaging Conventions standard. In the implementation for the .NET Framework 3.0, System.IO.Packaging.Package is defined as an abstract class from which specific physical implementations are derived. The default and primary physical implementation of a Package in the .NET Framework 3.0 is the ZipPackage derived class that uses a standard ZIP archive (see Figure 1). The following code shows the basic steps used to create and open a new package:
Figure 1 File Formats Using Open Packaging Conventions
Figure 1 File Formats Using Open Packaging Conventions
using System.IO.Packaging;
...
// Path and name for the package file.
string packageFile = @”C:\webpages\packaging.htmx”;

// Create and open the specified package file for writing.
// (The ‘using’ statement ensures that package is
//  closed and disposed when it goes out of scope.)
using (Package package = Package.Open(packageFile, FileMode.Create))
{
    ... // Store content files as parts in the package.
}
Open Packaging uses ZIP files because they are a known industry format that is easy to work with, inspect, and access. In fact, you can take a package-based file (a .docx file, for example), rename it with a .zip extension, and access its contents using a standard ZIP utility, such as the Compressed Folders feature built into Windows Explorer.
A package stores two basic elements: PackageParts (which are referred to simply as parts) and PackageRelationships (which are referred to as relationships). Parts represent the actual content being packaged (see Figure 2). Higher-level components are provided by building upon the basic part and relationship elements. These include PackageDigitalSignatures, PackageProperties, and components to support encryption and rights management.
Figure 2 Basic Elements of a Package
Figure 2 Basic Elements of a Package

Package Parts
A part is a data stream, analogous to a file in a file system or ZIP archive. Parts can hold any type of data—binary, text, images, and so on. Figure 3 outlines a sample package that contains several parts with different types of content.
Figure 3 Packaging Various Forms of Content
Figure 3 Packaging Various Forms of Content
When a part is stored in a package, it is defined with a unique URI-formatted PartName along with a MIME ContentType. Part names start with a forward slash, defining an absolute path based from the package root. You can easily add or remove a part to or from the package, write to a part, update it, or read the part from the package, simply by referencing its unique URI part name.
While the PartName URIs can appear to represent a folder hierarchy, there are actually no separate folders in a package. A package cannot contain empty folders and there is no concept of a default current directory. Because there can be no empty folders, there is no need to create, delete, or otherwise manage folders separately. If the package is opened in a graphical ZIP utility, the utility will often display the paths of the part names in a folder representation. Similarly if the package is unzipped, the utility will recreate the apparent folder hierarchy. The folder representation, however, is an artifact of the ZIP utility and is not inherent in the package itself.
The code sample shown in Figure 4 illustrates the process of creating a part and then writing data from a disk file to the part stored in the package.

Package Relationships
A relationship defines an association between two items: a specified source and a specified target. The source of a relationship can be either the package itself (a package-level relationship) or a specified part in the package (a part-level relationship). The target of a relationship can be either a specified part in the package or a specified external resource. An external resource can be another package, a part within another package, or any other type URI-addressable data entity. Thus, there are four combinations of relationships that can be defined:
  • Between a package (source) and a specified part (target) contained in the package.
  • Between a package (source) and a specified resource (target) external to the package.
  • Between a specified part (source) within the package and a second specified part (target) also contained within the package.
  • Between a specified part (source) within the package and a specified resource (target) external to the package.
The source part is considered the owner of the relationship. If the source part is deleted, all the relationships owned by that part are also deleted. Relationships are defined and stored separately in special relationship parts in the package. Note that creating or deleting a relationship does not physically alter the source or the target elements in any way.
Relationships offer a number of advantages:

Relational Structuring Relationships allow associations between content and resources to be determined without needing to read or parse content streams. Preprocessing tasks can aggregate, cache, and access needed resources without having to understand the details of the content in use.
Content Discoverability Relationship types combined with part MIME content types enable quick discoverability of the structure, associations, and content of the parts contained in the package.
Schema Independence Content and resource associations defined by relationships are schema independent. This is especially helpful for content defined in XML markup where parsing would typically require knowledge of the specific schema in use.
Reference Integrity For XML-based content, you can specify resources in markup by a reference to a relationship ID, rather than embedding resource URI references directly in markup. This simplifies the XML markup and eliminates the possibility that content references and relationship references might become out of sync.
Figure 5 illustrates a sample HTML page packaged with its style sheets and image file resources, as well as the relationships that define the associations. A package-level relationship is used to identify the URI of the PackagePart that contains the base HTML content for the page. By querying for the package’s root-html relationship, the URI of HTML Web page part can be directly determined. With the HTML part as the source, part-level relationships for the style sheet, scripts, and image resources can be easily located and resolved. (For each resource referenced in the HTML page, there is a corresponding part-level relationship.)
Figure 5 Sample HTML Page Package
Figure 5 Sample HTML Page Package
The code shown in Figure 6 demonstrates how to create a package, create and add a part, and then create a package relationship.
Figure 6 Create a Package, Part, and Package Relationship
using System.IO.Packaging;

// Path and name of the package file.
string packageFile = @”C:\webpages\packaging.htmx”;

// Path and name of the file to store in the package.
string partFile = @”C:\inetpub\wwwroot\packaging.htm”;

// Part name URI, MIME content type, and compression option.
Uri partNameUri = PackUriHelper.CreatePartUri(
    new Uri(“/packaging.htm”, UriKind.Relative));
string partType = “text/html”; // MIME type
CompressionOption compression = CompressionOption.Normal;

 for the package relationship type.
string packageRelationshipType = “http://schemas.openxmlformats.org/” +
    “package/2007/relationships/htmx/root-html”;

// Create and open the package file.
using (Package package = Package.Open(packageFile, FileMode.Create))
{
    // Create the part.
    PackagePart part = package.CreatePart(
        partNameUri, partType, compression);

    // Write the data from the file to the part.
    using (FileStream fileStream = File.OpenRead(partFile))
    {
        CopyStream(fileStream, part.GetStream()); // Copy data to part.
    }

    // Create a package-level relationship to the Web page part.
    package.CreateRelationship(
        part.Uri, TargetMode.Internal, packageRelationshipType);
}
Using relationships makes it easy to open and discover the associations between the content and resource items stored in the package. Figure 7 is an example of opening a package and then using the root-html package relationship to locate the root content part contained in the package. The root part can then be queried to return relationships that identify additional resources that the root element needs.
Figure 7 Locating the Root Content Part in a Package
using System.IO.Packaging;

// Path and name of the package file.
string packageFile = @”C:\webpages\packaging.htmx”;

// Name for the root package relationship type.
string rootHtmlRelationshipType = “http://schemas.openxmlformats.org/” +
    “package/2007/relationships/htmx/root-html”;

string resourceRelationshipType = “http://schemas.openxmlformats.org/” +
    “package/2007/relationships/htmx/required-resource”;

// Open the package for reading. - - - - - - - - - - - - - - -
using (Package package =
    Package.Open(packageFile, FileMode.Open, FileAccess.Read))
{
    // A package can contain multiple root items, iterate
    // through each.  Get the “root-html” package relationship.
    foreach (PackageRelationship relationship in
        package.GetRelationshipsByType(rootHtmlRelationshipType))
    {
        // Get the part referenced by the relationship TargetUri.
        PackagePart rootPart = package.GetPart(relationship.TargetUri);

        // Open and access the part data stream.
        using (Stream dataStream = rootPart.GetStream())
        {
            ... //  Access the root part’s data stream.
        }

        // A part can have associations with other parts.
        // Locate and iterate through each associated part.
        // Iterate through each “required-resource” part.
        foreach (PackageRelationship resourceRelationship in
            rootPart.GetRelationshipsByType(
                resourceRelationshipType))
        {
            // Open the Resource Part and write the contents to a file.
            PackagePart resourcePart = package.GetPart(
                resourceRelationship.TargetUri);
            ... // Party with the resource part.
        }
    }
}

Package Digital Signatures
Built upon a composition of parts and relationships, a package can also contain PackageDigitalSignature items (digital signatures for short). A digital signature uses an X.509 certificate to securely sign parts and relationships in the package. The signature provides two features. It identifies and authenticates the individual or entity that has signed a given set of parts and relationships, and it validates that the signed parts and relationships have not been modified.
The digital signature does not prevent a part or relationship from being changed, but a validation check against the signature will fail if the signed item has been altered. The application can then take appropriate action—for example, prevent the part from being opened or notify the user that the data is not secure.
The diagram in Figure 8 illustrates a package that contains parts and relationships along with a digital signature that has been used to sign specific items. If any of the signed items are modified in any way, validation against the digital signature will fail.
Figure 8 Signing Parts and Relationships
Figure 8 Signing Parts and Relationships
For more information about package digital signatures, see the MSDN® article "The Digital Signing Framework of the Open Packaging Conventions".

Package Properties
A package can also store a set of public metadata as PackageProperties. This set includes 16 common properties that describe the package and its contents. Use of these properties is optional. Figure 9 summarizes the properties available with a package.
Figure 9 Properties Available with a Package

Property | Description
Category | A categorization of the package content (example: Letter, Proposal, Resume)
ContentStatus | The status of the package content (example: Draft, Reviewed, Final).
ContentType | The type of content that is contained in the package (example: Whitepaper, Bulletin).
Created | The date and time the package was created (see also Modified).
Creator | The name of the individual or entity that created the package (see also LastModifiedBy).
Description | A description of the package content.
Identifier | A unique identification assigned to the package and its content.
Keywords | A delimited set of keywords to support search and indexing.
Language | An RFC 3066 tag that identifies the language for which the package content is written.
LastModifiedBy | The name of the individual or entity that last modified the package (see also Creator).
LastPrinted | The date and time the package content was last printed.
Modified | The date and time the package content was last modified (see also Created).
Revision | The revision number of the package content (see also Version).
Subject | The topic of the package content.
Title | A name given to the package content.
Version | The version number of the package content (see also Revision).

Creating and Accessing Packages and Parts
You can use the Package.Open method to create a new package or open an existing package. Package.Open provides overloads that instantiate packages, either through a specified filename or by a given file stream. In addition, Package.Open includes settings that let you specify options for file mode (System.IO.FileMode) and file access (System.IO.FileAccess).
After a package is instantiated and opened, you can use the part management methods to create and access the parts contained in the package (see Figure 10). After obtaining a part instance—either through the CreatePart method or the GetPart method—you can call the part GetStream method to return the stream for reading or writing the content data. We discuss how to access the data content of a part in more detail later.
Figure 10 Package Part Methods

Package Method | Description
CreatePart | Creates a new part in the package.
DeletePart | Deletes a specified part from the package.
GetPart | Returns the part with a specified name.
GetParts | Returns a collection of all the parts contained in the package.
PartExists | Determines whether a specified part exists in the package.

Creating and Accessing Relationships
When using the System.IO.Packaging APIs, you can create package-level and part-level relationships through the CreateRelationship methods of the Package and PackagePart classes. The CreateRelationship methods use five items to define a relationship:
  • The source package or part
  • A URI for the target part or external resource
  • A targetMode that identifies if the target is internal or external to the package
  • A URI-formatted relationshipType
  • A unique relationshipID
Once created, relationships can be accessed through the GetRelationshipsByType, GetRelationship, or GetRelationships methods of the Package and PackagePart classes (see Figure 11). We provide more detail about how relationships are stored in a package in the section "Exploring a Package."
Figure 11 Relationship Methods

Package/Part Method | Description
CreateRelationship | Creates a package-level or part-level relationship to a specified target.
DeleteRelationship | Deletes a specified relationship owned by this package or part.
GetRelationshipsByType | Returns all relationships owned by this package or part that match a specified relationshipType.
GetRelationship | Returns the relationship owned by this package or part that matches a specified relationshipID.
GetRelationships | Returns all relationships owned by this package or part.
RelationshipExists | Determines whether a specified relationship owned by this package or part exists.

Pack URI
In order to access a part contained in a package, an address for both the package and the selected part needs to be specified. A package can be identified through conventional URI schemes, such as http, ftp, or file. However, conventional URI schemes do not provide a means to specify an individual part within the package. Based on the extensible architecture for URI schemes (RFC 3986), the Open Packaging Conventions standard defines a pack URI scheme that can address individual parts contained in a package.
Creating a pack URI is fairly simple. Given a conventional URI for a package file and the name of a part contained in that package, a fully-qualified pack URI address that identifies the part can be created in two steps.
Say, for instance, the URI is http://www.proseware.com/mypackage.docx and the part name is /images/chocolate.jpg. The first step is to encode the URI of the package to form a pack authority. This is done by replacing "?", "@", ":", "%", and "," characters with their percent-encoded equivalents ("?"="%3f", "@"="%40", ":"="%3a", "%"="%25", ","="%2c") and replacing all forward slash characters with commas. Thus, the URI would look like this:
http%3a,,www.proseware.com,mypackage.docx
The second step is to combine the "pack://" scheme prefix, pack authority, and part name to form the pack URI for the specified part. The result for our sample would be:
pack://http%3a,,www.proseware.com,mypackage.docx/images/chocolate.jpg
There are additional steps to handle other forms of URIs, but understanding the basic process outlined here is sufficient for most uses.
There are some important classes you ought to be familiar with. These include:
PackUriHelper The PackUriHelper class provides methods that make it easy to create pack URIs, as well as to extract and return the original package URI or part name from a given pack URI.
PackWebRequest To access data through URIs, .NET-based applications can use the abstract WebRequest and WebResponse classes from which scheme-specific subclasses are implemented and registered. Based on the scheme of the URI specified to the WebRequest.Create method, the appropriate subclass is automatically selected and used. With the .NET Framework 3.0, WebRequest and WebResponse directly support pack URIs. A feature specific to pack URIs is that streams returned by the GetResponseStream method are fully seekable (if seek is not supported by the network protocol, seek operations are handled client-side transparently by the pack implementation). Through the use of pack URIs your application can directly access any part contained in a package on any URI accessible data source.

Authentication and Validation
Packaging services include support for digital signatures that can be used to securely sign and validate content within a package. Key PackageDigitalSignatureManager class methods include VerifyCertificate, Sign, VerifySignature, and RemoveSignature.

Rights Management and Encryption
Beyond the functionality defined by the Open Packaging Conventions, the System.IO.Packaging APIs in the .NET Framework 3.0 offer support for protecting packaged content both for privacy and security. Using encryption with rights management in combination with a Windows Rights Management server, packages can be encrypted to limit access to specific individuals or groups. For more information on rights management and encryption, see the "Rights Managed Package Publish" and "Rights Managed Package Viewer" .NET Framework 3.0 samples provided with the Windows SDK.

Packaging a Web Page
We’ll use a simple Web page with its local resources to illustrate some of the basic operations with packages. Note, however, that this example will limit us to some extent since Web pages are based on a fixed HTML schema and we want to avoid altering the existing page content. An extensible XML-based schema provides more flexibility for additional packaging features. We’ll discuss these and other options in the "Package File Format Design" section.
Now that we’ve covered the basics of packaging, we’ll write a sample program, PackageHtmxWrite, for creating our own custom .htmx package (the full source for PackageHtmxWrite is available on the MSDN Magazine Web site). We will store in the package a Web page together with its local style sheets, scripts, and image file resources. To identify the starting part that contains the HTML page, we’ll create a package-level relationship that points to the HTML part that’s stored in the package. We’ll also create part-level relationships that define associations between the Web page and each local resource stored in the package. The relationships will make the associations between each of the components directly discoverable and eliminate the need to search the package to locate the Web page or to parse the HTML content to locate the related resources.
Since the Web page and its local resources are typically defined through relative paths, we can use the relative path and filename of each item as the part name for storing the data stream in the package. For example, given the HTML tag <img src="images/packaging-sign.png">, we can use the path and filename of the src attribute to define the URI name /images/packaging-sign.png for storing the image as a part in the package. The local resources of the Web page are specified within the HTML, therefore we need to parse the HTML once to identify all of the resource files used in the page.
PackageHtmxWrite’s File | Open command lets the user choose a Web page to load and display in the Web browser control. After the page is loaded, the File | Save As command allows the users to specify the location and filename for storing the Web page and its local resources in an .htmx file. Specifically, File | Save As invokes methods that parse the page’s HTML and create a list of the local resource files to be included in the package.
The WritePackage method then performs a series of steps to create the package. First it creates the package with the user-specified path and filename. Then, for the root HTML Web page, it creates a part for storing the root page; this involves storing the HTML data into the Web page part and creating a package-level relationship that identifies the root HTML Web page part. For each local resource referenced in the Web page, the method creates a part for storing the resource, stores the resource’s data into the part, and creates a part-level relationship from the Web page part that targets the resource part. The WritePackage code shown in Figure 12 illustrates the process.
Figure 12 Creating a Package
private void WritePackage(
    string packageFilepath, string parsedHtml, Hashtable resourceHash)
{
    // Create and open the specified package file for writing.
    // (The ‘using’ statement ensures that the package is
    // closed and disposed when it goes out of scope.)
    using (Package package = Package.Open(
        packageFilepath, FileMode.Create))
    {
        string docPath =
            webBrowser.Url.AbsoluteUri.Substring(rootPrefix.Length);
        string docPartName = Path.GetFileName(packageFilepath);
        docPartName = Path.ChangeExtension(docPartName, “htm”);

        if (!docPath.EndsWith(“/”))
        {
            // docPath is in the format of “/folder1/folder2/1.html”
            // Make it “/folder1/folder2/”
            docPath = docPath.Remove(docPath.LastIndexOf(‘/’)+1);
        }
        docPartName = docPath + docPartName;

        // The Web page and its resource files will likely be an absolute
        // path with a common folder prefix.  Determine the common folder
        // prefix to remove when the parts are stored.
        string commonFolderString =
            GetCommonFolderString(resourceHash.Values, docPartName);

        // Remove prefix of common folders from the Web page part name.
        docPartName = docPartName.Substring(commonFolderString.Length);

        // Create a Uri name for the Web page part.
        Uri webpagePartUri = PackUriHelper.CreatePartUri(
            new Uri(docPartName, UriKind.Relative));

        // Create a package part to store the Web page HTML.
        PackagePart webpagePart = package.CreatePart(webpagePartUri,
            System.Net.Mime.MediaTypeNames.Text.Html,
            CompressionOption.Normal);

        // Write the Web page HTML to the Web page part.
        using (StreamWriter sw = new StreamWriter(
            webpagePart.GetStream(), System.Text.Encoding.UTF8))
        {
            sw.Write(parsedHtml); // Write the Web page part.
        }

        // Create a package-level relationship to the Web page part.
        package.CreateRelationship(webpagePart.Uri,
            TargetMode.Internal, _packageRelationshipType);

        // Create and write each of the Web page’s local resource parts.
        foreach (Uri resourceUri in resourceHash.Keys)
        {
            string contenttype;
            // Open the stream for each resource file.
            using (Stream s = GetResourceStream(
                resourceUri, out contenttype))
            {
                // Nothing to do if the resource file has no stream.
                if (s == null) continue;

                // Get the package part name for the resource.
                ResourceInfo ri = (ResourceInfo)resourceHash[resourceUri];
                string partName = ri.partName;

                // Remove the leading common folders.
                partName = partName.Substring(commonFolderString.Length);

                // Create the part name URI for the resource.
                Uri partUri = PackUriHelper.CreatePartUri(
                    new Uri(partName, UriKind.Relative));

                // Use normal compression unless the data has already
                // been compressed or is an octet-stream.
                CompressionOption compression = CompressionOption.Normal;
                if (   contenttype.StartsWith(“image”)
                    || contenttype.StartsWith(“video”)
                    || contenttype.StartsWith(“audio”)
                    || contenttype.StartsWith(“application/octet-stream”))
                    compression = CompressionOption.NotCompressed;

                // Create the part for storing the resource file
                // in the package.
                PackagePart resourcePart =
                    package.CreatePart(partUri, contenttype, compression);

                // Copy the data from the file to the package part.
                CopyStream(s, resourcePart.GetStream());

                // Create a part-level relationship from the Web page part
                // (owner) to the resource part (relationship target).
                webpagePart.CreateRelationship(partUri,
                    TargetMode.Internal, resourceRelationshipType);
            }
        }
    }
}

Exploring a Package
When executed, the PackageHtmxWrite sample creates an .htmx package that contains the original Web page along with its local resources. Since the package is a ZIP file, we can use the standard Windows ZIP functionality to examine the components stored in the package. Figure 13 illustrates the organization of the parts contained in the packaging.htmx sample included in the download accompanying this article. Here’s a quick tour of the contents.
Figure 13 Organization of Parts in Packaging.htmx
Figure 13 Organization of Parts in Packaging.htmx
A [Content_Types].xml part is stored in all packages. This part contains a list of the MIME types and extensions for all of the other parts in the package. In our sample, [Content_Types].xml contains the following information:
<?xml version=”1.0” encoding=”utf-8” ?>
<Types xmlns=
    “http://schemas.openxmlformats.org/package/2006/content-types”>
  <Default Extension=”xml” ContentType=”text/xml” />
  <Default Extension=”htm” ContentType=”text/html” />
  <Default Extension=”html” ContentType=”text/html” />
  <Default Extension=”rels” ContentType=
    “application/vnd.openxmlformats-package.relationships+xml” />
  <Default Extension=”jpg” ContentType=”image/jpeg” />
  <Default Extension=”png” ContentType=”image/png” />
  <Default Extension=”css” ContentType=”text/css” />
</Types>
Within a package, relationships are defined in XML parts with a .rels extension; package-level relationships are defined in a part named /_rels/.rels. A package can have any number of package-level relationships. The relationship with the type /root-html is stored in the part /_rels/.rels. This is the relationship we created to identify the HTML Web page part name. We can use this relationship type in our programs later to return the part name of any HTML Web page we store. In our sample, the /_rels/.rels contains the following XML:
<?xml version=”1.0” encoding=”utf-8” ?>
<Relationships xmlns=
 “http://schemas.openxmlformats.org/package/2006/relationships”>
  <Relationship Type=
   “http://schemas.microsoft.com/opc/2007/relationships/htmx/root-html”
   Target=”/packaging.htm” TargetMode=”Internal”
   Id=”Rf29a606b57094466” />
</Relationships>
The .htm (or .html) file is an embedded copy of the HTML file that the user selected in the application.
Within a package, part-level relationships are stored in a part with a special name. First the source part’s PartName is divided into its path and filename components. The relationship part name is then formed by appending _rels/ to the path and .rels to the filename. So, if the source PartName is /aaa/bbb/mypage.htm, the path is /aaa/bbb/ and the filename is mypage.htm. The resulting relationship part name is /aaa/bbb/_rels/mypage.htm.rels.
A part can have any number of relationships. In the packaging.htmx package, the packaging.htm part defines its relationships in the part /_rels/packaging.htm.rels.

Reading a Package
The accompanying PackageHtmxRead sample application makes use of a Web browser control to display a Web page packaged in an HTMX file. Since the Web browser doesn’t understand HTMX package files, PackageHtmxRead must first unzip the HTML file and its resources to a local temporary directory. The OpenHtmxPackage method shown in Figure 14 demonstrates how to unzip the HTMX content and then display the Web page.
Figure 14 Unzip Content and Display Web Page
public bool OpenHtmxPackage(string filepath)
{
    // Extract the Web page and its local resources to the temp folder.
    _tempFolder = GetTempFolder();

    // Create a new tempFolder directory.  If the tempFolder
    // exists, delete it and create a new empty one.
    DirectoryInfo directoryInfo = new DirectoryInfo(_tempFolder);
    if (directoryInfo.Exists) directoryInfo.Delete(true);

    // Extract the Web page and its local resources to the temp folder.
    _htmlFilepath = ExtractPackageParts(filepath, _tempFolder);

    // Check that the Web page has a valid path and filename.
    if (_htmlFilepath == null)
    {
        WritePrompt(“  Error: web page not found”);
        return false;
    }

    // Convert the path and filename to a URI for the browser control.
    Uri webpageUri;
    try
    {
        webpageUri = new Uri(_htmlFilepath, UriKind.Absolute);
    }
    catch (System.UriFormatException)
    {
        string msg = _htmlFilepath + “\n\nThe specified path and “ +
            “filename cannot be converted to a valid URI.\n\n”;
        System.Windows.MessageBox.Show(msg, “Invalid URI”,
            MessageBoxButton.OK, MessageBoxImage.Error);
        return false;
    }

    // Load the Web page.
    webBrowser.ScriptErrorsSuppressed = false;
    webBrowser.Url = webpageUri;
    return true;
}
To extract the individual files, the OpenHtmxPackage method calls the ExtractPackageParts method, which queries the package for the relationship that identifies the root HTML part. The root HTML part is then copied to its relative location in the target directory.
Next, the HTML part is queried for part-level relationships that identify the associated resource parts. The resource part for each part-level relationship is then copied to its relative location in the target directory. Again, all this is done simply through the use of relationships—there’s no parsing of any part content! The sample code in Figure 15 shows the ExtractPackageParts and ExtractPart methods used to unzip the HTML file and its local resource files.
Figure 15 Unzip HTML File and its Resource Files

ExtractPackageParts
private string ExtractPackageParts(
    string packageFile, string targetDirectory)
{
    Uri uriDocumentTarget = null;

    // Open the Package.
    using (Package package =
        Package.Open(packageFile, FileMode.Open, FileAccess.Read))
    {
        PackagePart documentPart = null,  resourcePart = null;

        // Examine the package-level relationships and look for
        // the relationship with the "root-html" RelationshipType.
        foreach (PackageRelationship relationship in
            package.GetRelationshipsByType(_packageRelationshipType))
        {
            // Resolve the relationship target URI so
            // the root-html document part can be retrieved.
            uriDocumentTarget = PackUriHelper.ResolvePartUri(
                new Uri("/", UriKind.Relative), relationship.TargetUri);

            // Open the document part and write its contents to a file.
            documentPart = package.GetPart(uriDocumentTarget);
            ExtractPart(documentPart, targetDirectory);
        }

        // Examine the root part’s part-level relationships and look
        // for relationships with "required-resource" RelationshipTypes.
        Uri uriResourceTarget = null;
        foreach (PackageRelationship relationship in
            documentPart.GetRelationshipsByType(
                resourceRelationshipType))
        {
            // Resolve the Relationship Target Uri so the resource part
            // can be retrieved.
            uriResourceTarget = PackUriHelper.ResolvePartUri(
                documentPart.Uri, relationship.TargetUri);

            // Open the resource part and write the contents to a file.
            resourcePart = package.GetPart(uriResourceTarget);
            ExtractPart(resourcePart, targetDirectory);
        }

    }

    // Return the path and filename to the file referenced
    // by the HTMX package’s "root-html" package-level relationship.
    return targetDirectory + uriDocumentTarget.ToString().TrimStart(‘/’);
}
...
private const string _packageRelationshipType =
    "http://schemas.openxmlformats.org/package/2007/" +
    "relationships/htmx/root-html";

private const string _resourceRelationshipType =
    "http://schemas.openxmlformats.org/package/2007/" +
    "relationships/htmx/required-resource";

ExtractPart
private static void ExtractPart(
    PackagePart packagePart, string targetDirectory)
{
    // Remove leading slash from the Part Uri and make a new
    // relative Uri from the result.
    string stringPart = packagePart.Uri.ToString().TrimStart(‘/’);
    Uri partUri = new Uri(stringPart, UriKind.Relative);

    // Create an absolute file URI by combining the target directory
    // with the relative part URI created from the part name.
    Uri uriFullFilePath = new Uri(
        new Uri(targetDirectory, UriKind.Absolute), partUri);

    // Create the necessary directories based on the full part path
    Directory.CreateDirectory(
        Path.GetDirectoryName(uriFullFilePath.LocalPath));

    // Write the file from the part’s content stream.
    using (FileStream fileStream =
        File.Create(uriFullFilePath.LocalPath))
    {
        CopyStream(packagePart.GetStream(), fileStream);
    }
}

Signing and Validating a Package
The PackageHtmxSign sample program shows how digital signatures can be used to sign parts and relationships contained in a package. Using an X.509 certificate, you can identify the entity signing the content elements and create an encrypted hash that can then be used to validate that the content has not changed.
Typically an application would define a policy that lists the specific parts and relationships to be signed. To keep things simple for our .htmx format, our policy is to simply sign all parts and relationships. The SignAllParts method shows how to digitally sign all the parts and relationships in a package. Meanwhile, the ValidateSignatures method could be used to validate that all the signed elements have not been altered.

Designing a File Format
When planning your own package file format, there are several considerations you should keep in mind. First, use a package-level relationship to identify a starting part in the package. While you can use a fixed and mandatory part name, hardcoded part names can limit the flexibility of your package design. Relationships and relationship types enable easy discoverability without relying on predefined part names.
For parts that reference other resources, consider creating a relationship for each target resource. Building a tree of relationships and relationship types makes it easy to discover the structure of the items contained in the package without having to parse the content of each part.
For cases where all resources referenced by a part are defined with relationships, you may want to replace the references in the part content with the corresponding relationship ID values. By using relationship IDs, resource references can be changed simply by updating the appropriate relationship without having to parse and modify the content of the part itself. Note that this may not be practical if a part will be extracted or separated from the relationships that reference the actual target resources.
Try to avoid using relative references to resources outside the package. If the package is moved, the relative references to external resources will likely break.
Decide in advance whether the presence of unknown parts or relationships in your package format should be allowed. It’s easy to simply ignore unknown parts and relationships when they appear, but keep in mind that third-parties could add items into your package for undesirable purposes, such as tracking. To improve security, your application should regard unknown parts or relationships as invalid formats (or at least display an alert). And if the content is not intended to be changed, seriously consider signing the parts and relationships in your package.

Online Resources
  • Office Open XML File Formats
  • The Addressing Model of the Open Packaging Conventions
  • Pack URIs in Windows Presentation Foundation
  • Walkthrough: Word 2007 XML Format

Jack Davis is a Program Manager at Microsoft on the Windows Documents and Printing team. In a previous life he worked as a programmer/writer with the Windows Presentation Foundation SDK team. Jack can be reached at jack.davis@microsoft.com.

Andrey Shur is a Program Manager at Microsoft on the Windows Documents and Printing team.


Office Apps
Extend Your VBA Code With VSTO
Paul Stubbs and Kathleen McGrath

This article is based on a prerelease version of Visual Studio "Orcas." All information herein is subject to change.
This article discusses:
  • Language differences between VSTO and VBA
  • Advantages of VSTO over VBA
  • Calling VSTO from VBA and VBA from VSTO
This article uses the following technologies:
VBA, VSTO
Contents
Language Differences
Controlling the User Interface
Deploying VSTO Solutions
Extending VBA with VSTO
If you develop for Microsoft® Office using Visual Basic® for Applications (VBA) and have not yet made the jump to the Microsoft .NET Framework, Visual Studio® Tools for Office (VSTO) in the next version of Visual Studio, code-named "Orcas," makes the move easier and more compelling. VSTO brings you the full feature set of Visual Studio, including language-integrated query (LINQ), Windows® Presentation Foundation (WPF), Windows Communications Foundation (WCF) and the .NET Framework 3.5. In addition, VSTO has many new features targeting the 2007 Microsoft Office system, including visual designers, which make it easy to create custom form regions and customize the Ribbon. You can create custom document-level and application-level task panes and easily create and debug Office SharePoint® Server 2007 workflow solutions. VSTO solutions are simple to deploy thanks to full support for ClickOnce. In addition, VSTO allows you to leverage your existing investments in VBA by extending your existing solutions with managed code.

Language Differences
There are a number of differences in the language features available in VBA and Visual Basic. Some of the biggest concern data types, statements, and new language features. One example is that an Integer in VBA is equivalent to a Short in Visual Basic and a Long in VBA is equivalent to a Visual Basic Integer. There are also some data types that are no longer supported in Visual Basic, such as the Variant—instead, you can use the Object data type.
The default properties, popular in VBA, are not supported in Visual Basic or VSTO. For example, you cannot assign a String to the Range property of a paragraph as you can in VBA; instead, you must specify the Text property, as you see in the following code:
‘ VBA
Me.Application.ActiveDocument.Paragraphs(1).Range = _
    “This is my first paragraph.”

‘ Visual Basic
Me.Application.ActiveDocument.Paragraphs(1).Range.Text = _
    “This is my first paragraph.”
You still have access to many of the VBA-specific language features, such as MsgBox, although Visual Basic also supports the use of MessageBox. And although the OnError statement is still supported in Visual Basic, you should consider using the structured error handling that try/catch statements provide.
The differences between VBA and Visual Basic grow with each new version of Visual Basic that is released. Some of the new features of Visual Basic 9.0 include local type inference, object initializers, anonymous types, extension methods, XML integration, and LINQ.
For example, Visual Basic can now determine the type of a local variable by evaluating the value that is assigned to it, known as local type inference. Although the data type isn’t specified, do not confuse this with late binding, where the data type is determined at run time. Local type inference occurs at design time (early binding) and all of the benefits, such as IntelliSense® are available, as you see in Figure 1.
Figure 1 IntelliSense Is Available with Local Type Inference, Thanks to Early Binding
Figure 1 IntelliSense Is Available with Local Type Inference, Thanks to Early Binding
With LINQ, you can create queries using a familiar syntax that is integrated into the Visual Basic language, rather than having to learn a completely different query language, such as SQL or XQuery.

Controlling the User Interface
In typical VBA solutions, the entry points into the code are made available through UI elements, such as toolbars and menus, or by creating UserForms to gather input from end users. With VSTO, you can customize similar UI elements and, depending on the Office application, you can customize additional UI elements. Whether you create a solution using VBA or VSTO, you collect user input through controls that you provide in the UI. You can add controls to toolbars, menus, forms, documents, task panes, the Ribbon, and so on. The difference, then, really lies in the type and number of controls and the types of UI surfaces available that can contain these controls.
When you create an Office application using VBA, the forms are different from the forms you create with VSTO. With VBA, your only option is to display a UserForm, and the ActiveX® controls available for use on this type of form are limited. VSTO, on the other hand, supports a wide variety of Windows Forms controls. You can always find a Windows Forms control that matches the functionality of the controls available to a UserForm (although the name of the control might differ). For example, the equivalent of an OptionButton on a Windows Form is a RadioButton in VSTO. And although you will not find a ToggleButton on the Toolbox in a VSTO project, you can use a CheckBox control and set its Appearance property to Button. The properties, methods, and event names may differ between these types of controls as well, but you can typically find an equivalent. In VSTO, instead of setting the Caption property of a Label as you would in VBA, you would set the Text property of the Label.
The numerous Windows Forms controls available provide advanced functionality and allow you to build a much more professional UI for your application than you can build with UserForms. Because VSTO solutions are created in Visual Studio, you have access to many features that are not available in the VBA code editor, such as the Data Sources window, which enables you to add data-bound controls directly to a Windows Form. You can also use some of the new language features of Visual Basic and Visual C#® that enable the use of LINQ queries to fill a ListBox or ComboBox control.
You can add WPF controls to your application with VSTO by selecting a WPF user control item from the Add New Item dialog box and designing it by either dragging WPF controls from the Toolbox or adding XAML markup to the XML window. Then add a user control to your project and drag an ElementHost control from the WPF Interoperability tab of the Toolbox to the control. When you set the Child property of this control to the WPF user control, all of the WPF controls become hosted on the Windows Form. Now you can add this control to a Windows Form, the document’s surface, or a task pane.
Task Panes Since Office 2003, you’ve been able to customize the Office task pane to provide the same type of UI that a Windows Form enables, but with the added benefit of being able to dock the task pane outside the document surface. Customization of the task pane is not possible with VBA. Using VSTO, there are two types of task panes that can be customized: Document Actions task panes and Custom task panes.
The Document Actions task pane is a document-level task pane visible only to a document or workbook that is associated with the VSTO customization. You can only have one Document Actions task pane per document or workbook. You customize this document-level task pane through the ActionsPane object when you create a Microsoft Word or Excel® document or template project in VSTO.
To customize the document-level task pane, you add an instance of the user control to the Controls collection of the ActionsPane object. You cannot change the name that appears as the title of the task pane:
Dim TaskPaneControl As New UserControl1
Me.ActionsPane.Controls.Add(TaskPaneControl)
The 2007 Microsoft Office system introduces a new type of task pane, known as the custom task pane, which is customizable when you create application-level add-ins with VSTO. You can customize the actions pane in both the 2003 and 2007 versions of Word and Excel, but custom task panes are available (and thus customizable) in the 2007 version of five applications: Word, Excel, InfoPath®, Outlook®, and PowerPoint®.
A custom task pane can display the same type of Windows Forms controls as a document-level task pane; however, a custom task pane is available at the application level, and you can display multiple custom task panes within your solution. Whether you choose to customize an application-level or document-level task pane, the process is very similar.
To customize the application-level custom task pane, add an instance of the user control to the CustomTaskPanes collection and pass the title that you would like the task pane to display:
Dim CustomTaskPaneControl As New UserControl1
Me.CustomTaskPanes.Add(CustomTaskPaneControl, “Format Document”)
Whether you use VBA or VSTO to create Office 2003 solutions, you can customize the toolbars and menus. Typically, you would call a macro from the OnAction property of a button on the toolbar or in a menu. Using VSTO, you must create the new menu or toolbar items programmatically, rather than using the OnAction property (which cannot be set to call managed code). Additionally, you must create an event handler for the button. The 2007 Office system introduces the Office Fluent interface, which includes the Ribbon. The Ribbon replaces toolbars and menus in most 2007 Office system applications, although you will still find menus and toolbars in the main window of Outlook 2007.
You can use VBA to customize the Ribbon using RibbonX, but VSTO makes it even easier by providing a Ribbon designer that makes adding new tabs, groups, and controls to the Ribbon as intuitive as adding a control to a Windows Form. You can drag buttons from the Toolbox (in the Ribbon Designer tab) to the Ribbon Designer, set properties of the button such as its size, and create event handlers for the button’s Click event. Alternatively, you can customize the Ribbon with several other controls using the Ribbon XML project item. Figure 2 shows the "Ribbon (Visual Designer)" and "Ribbon (XML)" project items that are available in the Office tab of the Add New Item dialog box.
Figure 2 Ribbon Project Items in VSTO
Figure 2 Ribbon Project Items in VSTO
These Ribbon customization options are available to both application-level add-ins and document-level customizations that support the Ribbon. For example, you can create a document-level Ribbon customization for a Word template solution that assembles Professional Services Proposals for an engineering company.
There are three parts to a Ribbon: a tab, a group, and the controls you add to a group. Using the Ribbon designer in VSTO, you can quickly customize an existing tab or create a separate tab that contains all the functionality for your application. The Toolbox in Figure 3 lists all of the Office Ribbon controls available in a Word solution. When you drag a control to the designer, you can set properties, such as its size, its text, and whether it contains an image, in the Properties window. Some controls can contain other controls. For example, Figure 4 shows an Employees Menu control that contains a number of SplitButton controls, which, in turn, contain a number of Buttons.
Figure 3 Employees Menu Control in the Designer
Figure 3 Employees Menu Control in the Designer
Figure 4 Menu Images Rendered
Figure 4 Menu Images Rendered
You can use a built-in image by specifying its name in the OfficeImageId property of a control, or you can easily add your own custom image. You can add custom images to a control by specifying an image that has been added to the project as a resource in the Image property. When you add an image from a resource, it is displayed in the designer, as shown in the PEPs group in Figure 3. However, when you use built-in images, a placeholder image is displayed within the designer instead. When you run the application, the images are displayed as in Figure 4.
One of the advantages of using the Ribbon Designer over Ribbon XML is that you can create the event handlers for these controls in the same way you create them for controls on a Windows Form. For example, if you double-click the Title combobox, you could add the code shown in Figure 5 to the Title_TextChanged event handler to insert the title into the document. Ribbon XML enables advanced customizations not available in the Ribbon designer. You can export the Ribbon customization to Ribbon XML to further customize your solution; but you cannot import the Ribbon XML back into the designer. You must also create a callback method for each event handler that was in the Ribbon designer code.
Figure 5 Inserting Title
Private Sub Title_TextChanged(ByVal sender As System.Object, _
    ByVal e As Microsoft.Office.Tools.Ribbon.RibbonControlEventArgs) _
        Handles Title.TextChanged

    With Globals.ThisDocument.Paragraphs(1)
        .Range.Text = Me.Title.Text & “ Proposal”
        .Range.Font.AllCaps = True
        .Range.Font.Bold = True
        .Range.ParagraphFormat.Alignment = _
            Word.WdParagraphAlignment.wdAlignParagraphCenter
    End With

End Sub

Deploying VSTO Solutions
The VSTO security and deployment model is very different from that of VBA. In VBA, the source code is stored inside of the document itself, so deploying the code is as simple as copying the document. Although this model makes deployment very easy, trusting the VBA code becomes more difficult, as evidenced by the high number of VBA macro viruses that have developed. Another problem with the VBA embedded-code model becomes apparent when you try to roll out a new version—you can’t update the VBA code in your documents. You could open each document to change the VBA code inside,but how do you guarantee that you have updated every copy? Even in a small group you may not find every instance because documents are saved in e-mail and on servers, and they can be renamed.
In VSTO the code is actually linked to the document, where it is deployed to a central location and all of the documents just link to the code. This makes updating quite easy. VSTO supports publishing to local drives, UNC paths, HTTP sites, and removable media such as CDs, DVDs, and USB drives. The developer or administrator publishes a new version of the code and every user gets the new code when the document is opened again. Additionally, VSTO adds full support for ClickOnce, the technology that allows VSTO to publish, update, and cache the code locally. Although a full discussion of ClickOnce is outside the scope of this article, you will learn about some of the features as they apply to VSTO solutions.
You install an add-in the same way you install any other application by running an installer program called setup.exe that prompts the user to allow the setup program to install, register, and set the trust level. You no longer need to remember obscure registry keys or complex caspol.exe commands to install the add-in. Instead, you can use the publish feature of VSTO to create the setup.exe program and the supporting ClickOnce files.
The deployment feature is supported across all 2007 Office system applications that use VSTO. As an example of using VSTO to publish a solution, you can start by creating a standard Word 2007 add-in using Visual Basic. Double-click ThisAddin.vb to open the code window. (You’ll also need to add a reference to the System.Deployment assembly. This is not required by VSTO in order to deploy, but for the sample code to display the ClickOnce version information, you need to use some functions in this assembly.)Add the code in Figure 6 to the Startup event of the add-in. The code uses the IsNetworkDeployed to determine if this is running from the ClickOnce cache. If so, then you want to display the published version number of the application which is returned by the CurrentVersion property of the CurrentDeployment object. The CurrentVersion number matches Publish Version fields of the Publish tab in the property pages of the project in Figure 7. This also displays the application version, which is the version of the assembly set under the Application tab. Start by verifying that the application works by pressing F5 to run it. Word should open and display a dialog with the text "Not a ClickOnce Application". This is correct because you ran the application from VSTO. Close Word to stop debugging.
Figure 6 Is This a ClickOnce Application?
If Not System.Deployment.Application.ApplicationDeployment _
        .IsNetworkDeployed Then

    ‘This is a ClickOnce Application
    Dim currentVersion As String
    currentVersion = System.Deployment.Application _
        .ApplicationDeployment.CurrentDeployment _
            .CurrentVersion.ToString()

    MsgBox(“Started “ & vbCrLf & “ App Version:” _
        & My.Application.Info.Version.ToString() & vbCrLf & _
        “ Published Version “ & currentVersion)
Else
    MsgBox(“Not a ClickOnce Application”)
End If
Figure 7 The Add-in Project
Figure 7 The Add-in Project
Open the Project property page and click the Publish tab to see the Publish Settings which contains two properties—publish location and install location. VSTO will build the published project at the publish location. Think of this as the bin directory for publishing—the final deployed location of the solution. The administrator can change the publish location and re-sign the manifest using the ClickOnce Manifest Generation and Editing Tool (Mage.exe). In the Install Settings section, you can set the prerequisites and the update frequency. For performance reasons, by default the update frequency checks for new versions every seven days. Change this to "Check every time the customization runs" so that you can see the changes immediately. You can set the publish version using the standard major, minor, build, and revision format. By default, it is set to automatically increment the revision number every time you successfully publish. With all of these properties set correctly, just press the Publish Now button at the bottom of the page.
The publish wizard creates a setup.exe bootstrapper file and a VSTO deployment manifest file in the publish location. It also creates a subdirectory named with the version number and places the assemblies and the application manifest file in this location. When you publish again, another subdirectory will be created and a new VSTO deployment manifest file will be built that points to the current version of the application manifest. You can see in Figure 8 that there are two published versions of the add-in: 1.0.0.0 and 1.0.0.1. There are also two back-up copies of the deployment manifest files: DeployWordAddin_1_0_0_0.vsto and DeployWordAddin_1_0_0_1.vsto. These are only there to help you roll back to a particular version. For example, if you want to roll back to version 1.0.0.0, just rename DeployWordAddin_1_0_0_0.vsto to DeployWordAddin.vsto and the next time the application checks for an update it will get version 1.0.0.0.
Figure 8 Word Add-In Versions
Figure 8 Word Add-In Versions
Document solutions work the same way, except that the document is also published to the publish folder and a copy is published to the version subdirectory. There is no need to run setup.exe—to install the document solution you can just open the document, and you will be prompted to allow the installation of the solution. This enables you to publish the solution to a central location and then distribute the document to the users.
Once you publish the solution, users can install it by running the setup.exe bootstrapper, which will start the ClickOnce installation. The security model is more robust than it is in VBA and much easier to use than previous versions of VSTO. The first check looks for the solution to be signed by a trusted certificate. If the certificate is trusted, the solution is installed. If not,the user will be prompted to allow the installation. As I mentioned, document solutions are almost identical to add-ins except that you can install the solution by simply opening the document. One caveat with document solutions is that if the document is not on the local machine or contains VBA code, then the document location must be added to the Office trusted folders list. Once installed, the solution is available offline without running the solution first.

Extending VBA with VSTO
Another great feature of VSTO is the ability to work side-by-side with VBA, allowing you to keep your existing investment in VBA and utilize VSTO where it makes sense. For example, VSTO makes it easy to extend your VBA code to customize the Ribbon, create a custom task pane, or call a WCF service.
The first way for VSTO and VBA to communicate is for VSTO to call VBA functions. You use the object model of the Office application to call the function. For example, Excel provides the Run method to call VBA subroutines and functions. To use the Run method, you pass the name of the subroutine or function as the first parameter. The remaining optional parameters are used to pass data to the functions. Imagine you have a VBA function called AddNumbers, which takes two parameters and returns an integer. The VSTO code to call that VBA function would be:
Dim answer as integer = Me.Application.Run(“AddNumbers”, 10, 2)
One thing to note is that this code will fail if the appropriate access is not granted to the VBA macros. By default, VBA macros are not enabled. But then again, this scenario assumes you have existing VBA code that has permission to run and you’re now calling those existing macros from VSTO. This solution is also fragile because the user can easily change VBA code, which will cause the Run method to fail. Additionally, you will not have IntelliSense to guide you, which makes errors more common. You need to write your code defensively to handle these cases.
The second way for VSTO and VBA to communicate is for VBA to call VSTO functions. This was previously possible only in a very limited and fragile way, but is now built into VSTO. The common scenario for this feature is that you have an existing Excel document containing VBA code and now you want to add additional functionality written in managed code using VSTO.
Another common scenario is that the IT department creates a library of functionality in VSTO to be consumed by the VBA user. You might want to simply create an Excel 2007 workbook project with macros enabled, but first you must have an existing workbook that contains VBA code. Create a new workbook, add some VBA code, and save it as a macro-enabled workbook called VBAInterop.xlsm. Open Visual Studio "Orcas" and create a new Visual Basic Excel 2007 Project.
When the new project dialog prompts you to select a document for your application, select "Copy an existing document" and browse to the VBAInterop.xlsm you just created. This allows you to add VBA macros that call functionality in the VSTO solution attached to the document. If you are using a new installation of Visual Studio, you may get a warning dialog telling you that you need to allow access to the VBA project system. Click OK to allow and the project will be created.
After the project is created, save it by clicking File | Save All; the default location is fine for this example. Double-click on ThisWorkbook.vb in Solution Explorer to open the designer for the workbook. You can expose any view to VBA that you choose. A view in VSTO is the top-level object, which is represented in Solution Explorer as a file under the document node. Excel has four views by default, one for the workbook and one for each worksheet. Word, on the other hand, has only one—the document. You will expose the workbook view to VBA by setting a property in the property grid of the ThisWorkbook.vb file.
There are two properties that control the VBA interop behavior: EnableVbaCallers and ReferenceAssemblyFromVbaProject. You only need to set EnableVbaCallers to True; ReferenceAssemblyFromVbaProject will automatically be set to True also. ReferenceAssemblyFromVbaProject supports advanced scenarios around VBA interop and is beyond the scope of this article.
When you set the EnableVbaCallers to true, you will receive a warning dialog to let you know that VBA code that you write when debugging the project will not be saved in the project. Your VSTO project contains the Office document. When you build and debug/run the project, VSTO makes a copy of the document in the output location. It is this copy that is running, so changes you make will not be persisted back into the project’s copy of the document. If you want your VBA code persisted in the document, you must open the document in the project outside of VSTO and add the VBA code.
Now that you have enabled VBA interop, you can add a function for VBA to call. Right-click on the ThisWorkbook.vb and choose View Code. Create a function which will add two numbers together and return the result:
   Public Function AddNumbers (ByVal _
     x As Integer, _
     ByVal y As Integer) As Integer
       Return x + y
   End Function
You can now run the solution by pressing F5. After the workbook opens, press Alt+F11 to open the VBA editor shown in Figure 9. You will see the VBA code that was generated by VSTO to connect VBA to the VSTO assembly. You can now call the VSTO functions from the CallVSTOAssemby object with full IntelliSense that supports the parameters and return types.
Figure 9 VBA Editor
Figure 9 VBA Editor
You can see how easy it is to extend your VBA code with VSTO and managed code. This is a great technique for adding WPF dialogs to your VBA solution or calling WCF services and other .NET Framework 3.5 features. You can also use this with user-defined functions to call managed code. You can even fully debug this solution by setting breakpoints in both the VBA code and the VSTO code, allowing you to step back and forth between the two development environments.
While VBA support is not going away in the near future, now is a perfect time to make the transition to managed Office development with VSTO. There are many new features available in the latest version of Office and VSTO that make this combination the most productive way to create great Office apps. For more information about VSTO, you can check out the VSTO forum on MSDN® online.

Paul Stubbs is a Senior Program Manager with the Visual Studio Tools for Office and Visual Tools for Applications team. He coauthored VSTO for Mere Mortals with Kathleen McGrath (Addison-Wesley, 2006). He has spoken at Tech•Ed and TechReady and participates in the developer community on the Microsoft forums. Read Paul’s blog at blogs.msdn.com/pstubbs.

Kathleen McGrath is a programming writer at Microsoft. She has written documentation for Visual Studio 2005 Tools for the Microsoft Office System (VSTO), Visual Studio Tools for Applications (VSTA), and Visual Basic. She has created video demonstrations of the features of VSTO and Visual Basic on her blog at blogs.msdn.com/kathleen.


Excel Services
Develop A Calculation Engine For Your Apps
Vishwas Lele and Pyush Kumar

This article discusses:
  • Excel as a server-based application
  • The Excel Services architecture and APIs
  • Creating managed user-defined functions
  • Building custom solutions with Excel Services
This article uses the following technologies:
Excel Services
Contents
Excel Services Architecture
The Excel Services API
A Custom Solution with Excel Services
The Excel Pre-Compiler
The Excel Web Service Client
Digging into the Code
Performance and Scalability
Organizations use Microsoft® Excel® to perform complex calculations and to visualize information using charts, pivot tables, and the like, and to perform many other custom tasks. But in the past, if you wanted to implement a calculation engine, you needed to enlist the services of a developer who would use algorithms provided by your business analysts to design the code. Now, with the Excel Services technology in Office SharePoint® Server 2007, business analysts themselves can implement the calculation engine formulas they need, reducing the cost of implementation and making maintenance of the calculation algorithms easier than before. In addition, with Excel Services the custom algorithms in an Excel workbook can run on a Web server, allowing users to access them remotely. As you might imagine, this means many more users can take advantage of the software from many more locations.

Excel Services Architecture
Let’s see how the Excel Services architecture enables such flexibility. Excel Services consists of three tiers—a Web front end, an application server, and a database (see Figure 1). The SharePoint content database forms the database tier. To enable the server-side Excel behavior, you place the workbook at a trusted SharePoint location or on a network file share. Some functionality (such as additional security features) is only available through SharePoint.
Figure 1 Excel Web Services Architecture
Figure 1 Excel Web Services Architecture
The application server consists of Excel Calculation Services, which is responsible for loading a given workbook and performing any required calculations. Workbook instances can be connected to external data sources.
The Web front end is responsible for rendering the relevant portions of the workbook in HTML via a SharePoint Web Part. The Web front end is also responsible for exposing the Web service endpoints that allow remote access to the Excel Calculation Services.
An important aspect of the Excel Services architecture is that it is integrated with SharePoint 2007. As noted earlier, to enable some of the server-side behavior the workbook needs to be stored inside a SharePoint content database. This makes it possible to take advantage of SharePoint content management features such as versioning, check-in/check-out, and security roles and permissions in the context of Excel workbooks.
Similarly, the Excel Calculation Services is based on the SharePoint Shared Service Provider (SSP) model. SSP is a mechanism for packaging SharePoint functionality as a service that makes it easy to administer and use across different sites. As a result, it is possible to reuse an Excel Calculation Service instance across SharePoint sites, as well as manage it via a SharePoint administration site.
You should also note that Excel Services imposes some restrictions over Excel. Macros and unmanaged code-based add-ins, such as Visual Basic for Applications (VBA) code are not supported by Excel Services. Instead, Excel Services supports managed server-side user-defined functions (UDF), an interface that allows invocation of custom calculations from inside a server-side workbook.
Later in this article, we will look at a UDF code example. The restriction related to unmanaged code add-ins can be overcome by building a managed UDF to wrap unmanaged code. The restriction on the use of VBA and macros is hard, but it may be a boon in disguise as it prevents the server-side calculation logic from becoming unwieldy.

The Excel Services API
Now let’s look at the Web service-based Excel Services API used to interact with a server-side workbook, using the code snippet in Figure 2 as the basis for the discussion that follows. Note that code has been elided for clarity.
Figure 2 Open a New Workbook Session
string sheetName = “Input”;
string targetWorkbookPath = “workbook.xlsx”;
Excel.Status[] outStatus;

// #1 Excel Service Namespace
// using Excel = Microsoft.Office.Excel.Server.WebServices;

//#2 ExcelService
Excel.ExcelService service = new Excel.ExcelService();

//#3 Open a session with the workbook
string sessionID = service.OpenWorkbook(targetWorkbookPath,”en-US”,
    “en-US”, out outStatus);

// #4 Prepare the range
Excel.RangeCoordinates rangeCoordinates = new Excel.RangeCoordinates();
rangeCoordinates.Column = 8;
rangeCoordinates.Row = 7;
rangeCoordinates.Height = 1;
rangeCoordinates.Width = 9;

object[] rangeValues = new object[rangeCoordinates.Width];
// Populate rangeValues
...

// #5 Set range values
service.SetRange(sessionID, sheetName, rangeCoordinates,
    rangeValues, out outStatus);

// #6 Calculate the formulas in the workbook
service.CalculateWorkbook(sessionID,
    Excel.CalculateType.CalculateFull, out outStatus);

// #7 Get the calculated values
object[] rangeResult = service.GetRange(sessionID,
    sheetName, rangeCoordinates, false, out outStatus);

Console.WriteLine(“Total Rows: “ + rangeResult.Length);

// #8 Close the session
service.CloseWorkbook(sessionID, out outStatus);
To begin, we need to make the workbook accessible to the clients. Any workbook saved to a location on the server can be accessed through the Excel Service API, a part of Microsoft.Office.Excel.Server.WebServices. The Excel 2007 client makes publishing the workbook easier through publish functionality. The benefit of using the publish mechanism is that you can control which parts of a workbook (sheets, views, pivot tables, and so on) are accessible via the Excel Services API. The primary class in the API is the ExcelService class, shown in Figure 2. This class represents an in-memory, server-side instance of a workbook. To enable multiple users to interact concurrently with a workbook, a session-based access model has been implemented. Each user opens a separate session with a workbook using the OpenWorkbook method of the ExcelService class. The OpenWorkbook method returns a unique session ID associated with the opened session. This session identifier needs to be supplied when invoking any subsequent methods to interact with the opened workbook. To set a named range inside a workbook, you can use the RangeCoordinates class to define the boundaries of the named range. The SetRange takes RangeCoordinates and the corresponding array containing values to be passed in as parameters. A variation of the SetRange method is the SetRangeA1 method, which uses Excel range specification "A1" instead of the range coordinates used by SetRange. Once all the required range values have been specified, CalculateWorkbook can be invoked to force the workbook to compute the formulas. It is possible to cancel the most recent CalculateWorkbook method by invoking a CancelRequest method. You can use the GetRange method to obtain calculated values from a range in the open workbook. Once all the calculated values have been retrieved, you close the workbook session using the CloseWorkbook method.
Excel Services can be extended by adding UDFs, which are accessible as cell formulas similar to the built-in Excel functions. To create a UDF, you need to create a Microsoft® .NET Framework assembly that contains at least one class that is marked with the UdfClassAttribute and at least one method marked with the UdfMethodAttribute. Please refer to the code snippet below. Here we define ConvertToUpper as a UDF method. After appropriately registering the UDF, the ConvertToUpper function can be inside an Excel Services workbook instance:
using Microsoft.Office.Excel.Server.Udf;

[UdfClass]
public class Util
{
    [UdfMethod]
    public string ConvertToUpper(string name)
    {
        return name.ToUpper();
    }
}
We’ve covered only a small portion of the Excel Services API here. For additional details, refer to the MSDN® documentation.

A Custom Solution with Excel Services
The primary motivation for developing a custom solution is to allow business analysts to author calculations (such as financial models) directly as Excel formulas. Until now, business analysts have mostly relied on documenting algorithms as pseudocode in text. The pseudocode was then translated into code by developers. Using Excel Services, we were able to overcome some of the limitations inherent in the process of creating formulas in Excel and, as a result, eliminated the need for developers to convert the pseudocode into real code.
The key challenge in allowing non-developers to author calculation logic was to strike a balance between flexibility and ease of authoring, with the ability to enforce a robust structure. To provide the structure, we needed a way to define an input and output "interface" that represents the data contract for a calculation algorithm. Business analysts would be limited to the named ranges that are part of the data contract for flowing data in and out of the calculation instance.
The obvious choice was to define the data contract using named cells or ranges constructs within Excel. Named ranges not only accord the required level of granularity for authoring calculations within Excel, they are also the fundamental data structure on which Excel Services API methods such as SetRange and GetRange are based. However, the challenge with using named ranges is that there is no standard format or language, such as XML Schema Definition (XSD) or Web Services Description Language (WSDL), to define the interface. Moreover, named ranges (and consequently Excel Service API methods) are inherently type unsafe. For instance, there is no way to enforce datatype checking on a given named range. Finally, there is no built-in way to enforce the contract across the calculations (inside Excel) and the Excel Services client program.
To overcome these limitations, we developed a custom two-part solution. The first part is an Excel pre-compiler designed to generate the named ranges based on a defined interface. The second part is a generic Excel Web service client that invokes the calculation inside a workbook, while adhering to the interface.

The Excel Pre-Compiler
XML Schema Definition with all its semantic richness and simplicity, seemed the ideal choice for defining the interface. We decided to use the XML Schema constructs to define the input and output contracts. Next, we needed a way to translate the XML Schema into named ranges. We first looked at the possibility of using the XMLMap feature introduced with Excel 2003. XMLMap allows cells inside Excel to be mapped to the elements of an imported XML Schema. Unfortunately, the XMLMap capability is not available to Excel Services. So the alternative was to create the named ranges inside an Excel workbook. We developed a pre-compiler component that generated a template workbook with the required named ranges based on the schema. The generated template workbook has three sheets—one each for input, output, and calculation. The input sheet contains named ranges that correspond to the input for the calculation. Similarly, the output sheet contains named ranges that correspond to the output of the calculation, and the calculation sheet is where the calculations are placed (see Figure 3).
Figure 3 Workbook Input, Calculation, and Output Sheets
Figure 3 Workbook Input, Calculation, and Output Sheets
As we stated earlier, programming constructs such as looping are not available to Excel Services. The pre-compiler compensates for such limitations by transforming the input XML fields into a format that is accessible without the need for complex programming constructs. For example, XSD element collections can be transformed into dimensions of the named range. Figure 4 depicts an XSD snippet that is part of the input data contract for a calculation engine.
Figure 4 Input Data Contract
<?xml version=”1.0” encoding=”utf-8” ?>
<xs:schema id=”Input” ...>
  <xs:complexType name=”TypeA”>
    <xs:sequence>
      <xs:element name=”ValueA1” type=”xs:int”  minOccurs=”0” />
      <xs:element name=”ValueA2” type=”xs:int” minOccurs=”0” />
    </xs:sequence>
    <xs:attribute name=”RangeHeight” type=”xs:int” default=”1” />
    <xs:attribute name=”RangeWidth” type=”xs:int” default=”10” />

  </xs:complexType>
  <xs:complexType name=”InputType”>
    <xs:sequence>
      <xs:element name=”InputElementA” type=”TypeA”    />
      <xs:element name=”InputElementB” type=”TypeB”    />
    </xs:sequence>
  </xs:complexType>
  <xs:element name=”InputDataset” type=”InputType” msdata:IsDataSet=”true” />
</xs:schema>
Elements TypeA and TypeB are part of the input to the calculation. Note the custom attributes RangeHeight and RangeWidth that define the dimensions of the named ranges. The pre-compiler uses this information to generate the named range dimensions. The pre-compiler can also dereference index fields into separate columns—where each column represents an index value.
A noteworthy aspect of the pre-compiler is the ability to preserve the existing calculations while regenerating the input and output sheets. As depicted in Figure 5, developing a calculation algorithm is an iterative process. Business analysts and developers work together to define the initial data contract. During the course of development of the workbook, input and output contracts may need to be modified, and the workbook must be regenerated for these changes to be applied. The pre-compiler supports such iterative development by preserving the calculations sheet while regenerating the workbook.
Figure 5 Generating a Workbook Using the Pre-Compiler
Figure 5 Generating a Workbook Using the Pre-Compiler
The input sheet shown in Figure 3 has the pre-generated named ranges. The calculations sheet has calculations that reference the named ranges defined on the input sheet. The output sheet, in turn, references the calculations from the calculation sheet.

The Excel Web Service Client
The primary role of the Excel Web service client is to invoke calculations inside the workbook using the Excel Services API. In doing so, it interprets the XSD-based input contract, mapping the schema elements into appropriate named ranges. Once the calculation is complete, the client maps the output named ranges back into an XSD-based output contract. Figure 6 depicts the role of the Excel Web service client. A typed DataSet (based on the input schema contract) is passed in as input. Data contained within the DataSet is mapped to input named ranges. Once the calculation is complete, the output named ranges is used to populate the output DataSet. The Excel Web service client is responsible for applying the rules defined for the pre-compilers. It is also possible to inject custom data transformations to alter the aforementioned mapping between XSD and named ranges. Recall our earlier discussion on the need for compensating the lack of programming constructs available to workbook authors. Custom data transformations allow mapping to be altered to make it easy for business analysts to author the calculation logic.
Figure 6 Invoking the Custom Calculation Engine
Figure 6 Invoking the Custom Calculation Engine

Digging into the Code
The solution we’ve built is composed of four projects. The PreCompiler project houses all the code to generate a workbook with the required named ranges based on the input and output schema. It takes into account the aforementioned custom attributes such as RangeHeight while generating the named ranges. The PreCompiler project in turn relies on SpreadsheetML (an XML-based dialect used to represent information inside a spreadsheet) for generating the workbook. The SpreadsheetML project contains simple classes that wrap the SpreadsheetML components such as Workbook, Worksheet, and so forth.
The Client project, as the names suggests, is the Excel Web service client code. It sets the values for the input named ranges, forces the workbook to recalculate, and retrieves the values for output named ranges. You will recall that we talked about the need to transform the data to make it easier for business analysts to develop the calculations. In order to allow data transformation to be customized for each calculation engine, we externalized the data transformation logic by defining an IDataTransformer Interface, as shown here:
public interface IDataTransformer
{
    object    getRangeData(string RangeName);
    object[]  getRangeData(string RangeName, int width);
    object[,] getRangeData(string RangeName, int width, int height);

    string getInputSchema();
    string getOutputSchema();

    string getInputSchemaPrefix();
    string getOutputSchemaPrefix();
}
The name of an assembly that contains an implementation of the IDataTransformer interface is passed in as input to the client program. The client program in turn calls back the appropriate methods on the class implementing the IDataTransformer interface. In this way, the client program obtains the values that are used to populate the named ranges. The implementation logic inside the IDataTransformer methods is responsible for converting the data residing inside the input DataSet to appropriate named range values. For example, some rows from a DataTable may need to be filtered before populating the appropriate named ranges. Or the rows within the DataTable might need to be sorted before passing them along to the workbook. All such data transformation needs can be met using the IDataTransformer interface.
One other class that is important to discuss here is the ExcelServiceFacade. This class hides the Excel Service API details from the caller. The other important function of this class is to combine individual SetRange calls into one aggregated SetRange call. This is crucial for reducing the network latency as each invocation of SetRange causes a round-trip to the server. By exposing a local SetRange call that is ultimately converted into one aggregated SetRange call, ExcelServiceFacade can dramatically improve the response times. Figure 7 depicts the relevant ExcelServiceFacade code. An internal buffer is maintained by the ExcelServiceFacade class that is appended each time a "local" SetRange call is invoked. Once all the input named ranges have been populated, the internal buffer is sent up to the server as part of a single call. A similar mechanism is used when retrieving the output named values after the calculation is complete. Rather than retrieve the values from the output named ranges individually, we simply retrieve all the values on the output sheet in one fell swoop.
Figure 7 ExcelServiceFacade Class Methods
public class ExcelServiceFacade
{
    ...

    public void OpenWorkbook()
    {
        if (this.SessionID.Equals(String.Empty))
        {
            Status[] status = null;
            this.Excel.Credentials =
                System.Net.CredentialCache.DefaultCredentials;
            this.m_SessionID = this.Excel.OpenWorkbook(
                m_Url, CULTURE, CULTURE, out status);
        }
    }

    public void CalculateWookbook()
    {
        RangeCoordinates coorInput = new RangeCoordinates();
        coorInput.Row = 0;
        coorInput.Column = 1;
        coorInput.Width = 25;
        coorInput.Height = m_RowIndex;

        // Set Range
        object[] inputValues = (object[])
            m_Input.ToArray(typeof(object[]));
        SetRange(“Input”, coorInput, inputValues);

        this.Excel.CalculateWorkbook(
            this.SessionID, CalculateType.CalculateFull);
    }
}

Performance and Scalability
We found the actual execution of the calculations within the workbook to be very fast. In an unscientific test conducted with a workbook containing a nontrivial set of calculations involving close to one thousand named ranges, the response time was under one second; the majority of that time was spent on round-trips to the Excel server. As the load on the system grows in terms of the complexity of the calculation and the number of concurrent executions, it is possible to scale the solution by leveraging the various topology options offered by Excel Services. Different topology options allow you to select where each of the logical Excel Services layers (presentation, application, and database) are placed. For smaller setups (mainly used for testing purposes), it is possible to deploy all three layers on a single server. For a medium setup, the presentation and application layers can be installed on a single server, and the database layer on a separate server. For large setups, you can install each of the three layers on separate servers. Additionally, you can scale out the presentation layer by adding more servers using a network load balancer. The application layer comprising the Excel Calculation Services can also be scaled out using the load balancing schemes supported by the SSP framework. Figure 8 depicts a large setup where each layer is installed on a separate server. Furthermore, presentation and application layers are scaled out using load-balancing schemes.
Figure 8 Excel Web Services Large Setup
Figure 8 Excel Web Services Large Setup
For computationally intensive workloads, it is also possible to combine Excel Services with the Compute Cluster Server to seamlessly distribute work to compute nodes, as shown in Figure 9.
Figure 9 Excel Services High-Performance Setup
Figure 9 Excel Services High-Performance Setup
As you can see, a custom solution for implementing calculation engine components using Excel Services is a great productivity boon. It allows you to provide your users with anywhere-access to custom workbook functions, eliminates the need for a developer to implement the logic, and lets you scale out your solution as needed. Give it a try. We’re sure you’ll enjoy the increased flexibility and productivity you’ll gain. For more information, see the "Resources" sidebar.
Resources
  • Services Technical Overview
  • Determine Resource Requirements to Support Excel Services
  • Creating XML Mappings in Excel 2003

Vishwas Lele is a CTO at Applied Information Sciences (AIS) in Reston, VA. He assists organizations in envisioning, designing, and implementing enterprise solutions that are based on Microsoft .NET technologies. Vishwas is the Microsoft Regional Director for the Washington, DC, area. He can be reached at vlele@acm.org.

Pyush Kumar is a Lead Systems Architect for Watson Wyatt Worldwide. He’s been working most recently on grid computing and large-scale software design for the .NET Framework. You can reach him at pyush.kumar@watsonwyatt.com.


Editor's Note
Thanks for the Memories!
Stephen Toub

Writing an editor’s note is not an easy task. Based on my extensive research into many development magazines, I’ve found that to do it correctly you need to start by writing about a topic completely unrelated to anything development-focused. You then wax poetic about the weather, political affairs, or the latest fad in high-tech gadgetry. And you need to come up with a few minimally thought-out recommendations for strategies that will solve all of the world’s problems. To make it a truly outstanding editor’s note, however, you also need to invent a new word for something that already has several well-known names, "proclamize" your publication as the best thing since sliced bread, include several acronyms without definition, and throw in a few trite lines of code (which, ideally, will have a few lurking bugs).
When I set out to write this page, I intended to do none of this, but the pattern is difficult to avoid, IMHO. This being my first editor’s note, and most likely my last, I feel no compulsion to continue the pattern with the non sequitur that is supposed to follow—inevitably an attempt at using misdirection to convince the reader that everything discussed up to this point is crucial to her ability to implement the next big software application, or to even comprehend the pages that follow. Instead, I plan to use the remaining words allotted to me to say thank you.
I’ve had a thrilling ride at MSDN Magazine. I’ve been privileged to work on close to 50 issues, providing technical reviews for over 700 cutting-edge articles and writing more than 40, including the .NET Matters column. I’ve had the opportunity to speak at conferences on behalf of Microsoft and MSDN, and I’ve worked hard to facilitate MSDN Magazine’s rise in prominence to where we are today—serving more than a million developers every month. Truly it’s been a wonderful experience, and an honor, to be able to bring to you each month the best information we can find, from some of the brightest minds in the industry, to aid and entertain you.
After almost four years, though, I’m looking forward to my next big challenge. My interests have always been aligned with those of software developers and researchers, and I’m excited to get back to my roots. Working at Microsoft affords so many great opportunities, and I’ve chosen to apply myself in the Developer Division, where I’ll be working with the Parallel Computing Platform team to help make the "many-core" era of software development a reality.
So, thank you, all of you, for everything. Thank you for being loyal readers. Thank you for all of your questions, suggestions, comments, criticisms, and praise. Thank you to those who have written articles for the magazine, to our wonderful columnists (with whom the opportunity to work was one of the primary reasons I took this job), and to everyone from Microsoft who has graciously proffered their time and resources to make the magazine the success it is. Most importantly, thank you to the entire magazine team at Microsoft for being such a dedicated and passionate group of folks! It’s been my pleasure working with you every day, and I look forward to continuing to work with you when time allows for the occasional future installment of .NET Matters.
We’re all part of something amazing, and I can’t wait to see the progress we in the software industry experience in the next 5 years, in the next 10 years, and beyond. I know that MSDN Magazine will be there helping us out each step of the way.
With that, there’s only one thing left to do:
public static void UntilNextTime() {
    System.Windows.Forms.Application.Exit();
    System.Environment.Exit(0);
    System.Diagnostics.Process.GetCurrentProcess().Kill();
    System.Environment.FailFast(“Thanks for the memories!”);
}
—Stephen Toub


Toolbox
Create Graphs, Send E-Mail, Spell-Check, and More
Scott Mitchell

Contents
Turn Boring Raw Data Into Stunning Graphs and Charts
Create, Parse, Send, and Receive E-Mail Messages from Code
Add Spell-Check to Your Web Forms
Build an ASP.NET E-Commerce App

Turn Boring Raw Data Into Stunning Graphs and Charts
As developers, we work with raw data on a daily basis, be it XML, the results from a SQL query, or information in a text file. However, most people are not as proficient with raw information. A well-designed and well-placed chart can greatly reduce the amount of time needed to understand the data in a report, especially when studying long-term trends or patterns.
The Microsoft® .NET Framework provides a variety of techniques that enable developers to programmatically create charts, but writing the code to turn raw data into eye-catching graphs is typically more hassle than it’s worth. I have learned firsthand that the cost and time to learn how to use a third-party charting control is far less than the struggles, challenges, and frustrations encountered when trying to create your own charts. One third-party charting control that I use is SharpGraph for .NET 1.2 by Data Dynamics. Like most charting controls, SharpGraph for .NET provides a bevy of chart types, including all of the common 2D and 3D types: area, bar, line, doughnut, pie, plot, and so forth. Moreover, the chart’s colors, legends, titles, and axes are highly customizable.
Create a Chart in Minutes with the SharpGraph Wizard
Create a Chart in Minutes with the SharpGraph Wizard
What impressed me most about SharpGraph was its ease of use. Starting from an empty form, you can add a chart, bind it to data, and fine-tune its appearance all within a matter of minutes by using the control’s intuitive wizard. The wizard walks through the core chart settings, starting with the chart type, then the appearance, then the series data, and so on. The chart’s data can be manually entered, bound to a DataAdapter control or DataSet, or programmatically specified. There’s also a Chart Designer tool that simplifies customizing the chart’s appearance.
Moreover, these aesthetic settings can be saved to an XML file that can be loaded and used by other charts, either through the Chart Designer tool at design-time, or programmatically at run time. In addition to the wizard and Chart Designer, the chart’s data and appearance settings can also be set through the Properties window or with code.
SharpGraph for .NET also includes a number of bells and whistles. There are predefined color palettes, which are ideal for artistically-challenged developers like yours truly. It is also easy to specify transparency levels for the chart elements. And all SharpGraph charts can be programmatically exported to a variety of file formats, including BMP, GIF, JPG, PNG, and SVG, or to the Windows® clipboard as a BMP.
Price: $349 per developer license (includes royalty-free distribution).
www.datadynamics.com

Create, Parse, Send, and Receive E-Mail Messages from Code
E-mail messages are sent using the decades-old Simple Mail Transfer Protocol (SMTP). That is, the headers and body of the e-mail message are sent to an SMTP server using a standard protocol, which then ensures the e-mail message reaches its destination. Once an e-mail message reaches the intended recipient’s inbox, it is typically retrieved or viewed from an e-mail program like Microsoft Outlook® using either the Post Office Protocol (POP3) or Internet Message Access Protocol (IMAP) standards. The .NET Framework provides a simple and straightforward way to programmatically send e-mail to a specified SMTP server, but lacks classes for accessing e-mail messages from a POP3 or IMAP server. While the .NET Framework low-level networking classes can be used to communicate directly with a POP3 or IMAP server, such tasks are best left to a third-party component.
Enhanced E-Mail Functionality for .NET
Enhanced E-Mail Functionality for .NET
One such component that offers functionality for both sending and downloading e-mail is Mail for .NET 2.1 from Dart Communications. Like the SMTP-related classes in the .NET Framework System.Net.Mail namespace, Mail for .NET can construct plain-text and HTML-formatted e-mail messages, include attachments, and send e-mail to the specified SMTP server synchronously or asynchronously. Mail for .NET sets itself apart from the .NET Framework classes with a number of additional features. For example, when sending an e-mail message asynchronously, a Progress event is periodically raised that reports the current progress.
The POP3- and IMAP-related classes make downloading or reading e-mail from a POP3 or IMAP server as easy as writing a few lines of code. As with the SMTP-related classes, e-mail can be accessed from POP3 and IMAP servers synchronously or asynchronously. The Progress and Trace events make it possible to track the progress as e-mail messages are downloaded or as data is sent or received. Furthermore, these events can be used to display a progress bar or for logging.
The Mail for .NET classes do not include support for communicating with SMTP, POP3, or IMAP servers over SSL. If you need to interface with secure mail servers, be sure to check out Dart Communication’s Secure Mail for .NET product.
Price: $499 per developer license (includes royalty-free distribution).
www.dart.com

Add Spell-Check to Your Web Forms
While spell-checkers are a fundamental component of word processors, e-mail clients, and other desktop-based applications, they are rarely found within Web applications. For Web applications with short and simple data entry forms, users may be willing to overlook the lack of a spell-checker. But as the amount of text input grows, users will, naturally, want to ensure their entries are free of spelling mistakes.
Fortunately, adding an integrated spell-checker to your Web application is as easy as drag and drop with UltimateSpell 3.0 from Karamasoft. To add spell-checking to a Web page, simply drag the UltimateSpell custom ASP.NET server control from the Toolbox onto the Web page. The UltimateSpell control renders as a button that, when clicked, opens a new window that identifies and offers corrections for spelling errors in any single-line or multi-line textboxes or content-editable <div> tags on the page. In addition to the custom ASP.NET control, UltimateSpell includes client-side and server-side APIs.
Spell-Checking for Web Apps
Spell-Checking for Web Apps
UltimateSpell’s spell-checker can be configured to bypass checking words that match specific patterns, such as words that contain all uppercase letters, words in mixed case, words with numbers, HTML tags, e-mail addresses, and so forth. These ignore options can be assigned to the control declaratively or programmatically, or specified by the end user as the spell-checker is running.
When spell-checking, UltimateSpell sends back bite-sized chunks of text to the Web server using AJAX; any spelling corrections are updated in the textboxes using client-side script. This functionality allows for a rich spell-checking experience without the need for a full postback, thereby providing a more responsive user experience. Moreover, UltimateSpell works in all major browsers, including Internet Explorer®, Netscape, Mozilla, and Opera.
UltimateSpell offers a bounty of bells and whistles, including spell-checking as you type, where misspelled words are underlined with a red squiggly as in Microsoft Word, and auto-correct, where misspelled words with only one suggested fix are automatically replaced with the suggestion. While these features are impressive, they do require an increased number of asynchronous callbacks to the Web server. In my testing, these frequent communications introduced quite a noticeable lag when users typed at a high word-per-minute rate.
Price: $299 for a developer’s license.
www.karamasoft.com

Build an ASP.NET E-Commerce App
Virtually all businesses today have a Web presence of some sort. Many small- to medium-sized businesses have a simple, mostly static Web site that provides little more than a description of the company’s products or services, contact information, and hours of operation. While such companies may want to upgrade their Web site to a full-featured e-commerce application, many hesitate because of limited developer resources to tackle major challenges like payment collection and processing, order fulfillment, and security.
Building a secure, fully-functional e-commerce Web application with ASP.NET, however, is easier than you may think and can be accomplished without hiring a team of expensive consultants. In Pro ASP.NET 2.0 E-Commerce in C# 2005 (Apress, 2006), author Paul Sarknas walks readers through the key steps of building an e-commerce ASP.NET Web application. Rather than focusing on just the e-commerce related tasks, Paul takes readers through the entire application building process. For example, there are high-level discussions on the application’s design and motivation, a look at creating the data model, techniques for building a tiered application architecture, deployment options, and so forth. This background, which encompasses nearly a third of the book, is especially helpful to intermediate ASP.NET developers who may not have had much exposure to building large and complex Web applications; expert ASP.NET developers can skim through these early chapters.
Once the foundation has been laid, the book turns to the e-commerce specific aspects of the Web application: creating the product catalog, implementing a shopping cart, and using the PayPal SDK for both payment processing and completing the checkout process. There are also chapters on order fulfillment, techniques for promoting the site, and ways to upsell customers.
While the book assumes a certain level of ASP.NET experience, Paul’s clear and concise writing style and his decision to build the e-commerce application from the ground up makes Pro ASP.NET 2.0 E-Commerce in C# 2005 an interesting, useful, and accessible read for both intermediate and expert ASP.NET developers alike.
Price: $59.99.

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.

All prices were confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


CLR Inside Out
Collections Best Practices
Inbar Gazit

Contents

Most introductory textbooks on computer science and programming include a chapter about collections. They may be called arrays or data structures, but the concept remains the same. The ability to tie a set of elements to one another in a formal data object is essential to modern programming techniques.
In the Microsoft® .NET Framework, a lot of effort went into creating collection classes that are powerful and that address a variety of needs and styles. These collections are simple to use, intuitive, and have adequate performance—all very important characteristics. In this month’s installment, I’ll look at collections in .NET, how they work, when to use them, and some best practices.

When to Use a Collection
The first decision a developer faces seems quite simple: should I use a collection? Sometimes the answer is obvious. Sometimes it’s a matter of preference. The basic purpose of collections is to tie certain objects together as a group because they have some logical connection to each other. As an example, you may have a Book class that represents a book and you may have a BookCollection class that represents a library or a shelf of books in a library. By using a collection in this case, you can quickly add or remove books from the shelf, move books around, find a book based on title or ISBN or some other descriptive data, count the number of books, iterate over the books, and much more. I’ll cover the details later, but in this scenario using a collection is an obvious choice.
Now, let’s take a look at a case where you would almost certainly choose not to use collections—rational numbers. A rational number is a number that can be expressed as the ratio of two integers. In my description of collections, I proposed that a collection should be used to tie together a set of elements, and in this scenario, you could use a collection with two elements—one for the numerator and one for the denominator. However, the purpose of the numerator and denominator are very different (unlike in the books example, where all elements served a similar purpose). One of the primary reasons for storing items in a collection is to enable easy iteration through all elements so as to allow an operation to be performed on each individual element, a task that would rarely make sense for the integers composing a rational number. Moreover, a rational number has a fixed, very small number of elements. As a result, it doesn’t make sense to use a collection to store the elements of a rational number; here you should use two separate integral fields.
Now, consider a scenario in which the use of a collection could be debated. Imagine a data structure used to represent the teams in a sporting event, such as baseball. There are operations that would need to be performed on each team participating in the game, and each team serves a similar purpose (unlike the numerator and denominator in a rational number). However, in a game like baseball, there are only two teams, so while iteration support might be beneficial, the number of items to be enumerated is very small (and in prototypical implementation, fixed). Some might choose to use a TeamCollection or an array of Team instances to represent both sides, and some might choose to use two individual Team fields. There are pros and cons to both implementations.
Nevertheless, there are some simple guidelines to keep in mind when considering a collection. You should use a collection when at least one of following statements holds true:
  • Individual elements serve similar purposes and are of equal importance.
  • The number of elements is unknown or is not fixed at compile time.
  • You need to support iteration over all elements.
  • You need to support sorting of the elements.
  • You need to expose the elements from a library where a consumer will expect a collection type.
Note that for the purposes of this column, I consider arrays (System.Array) to be a special kind of collection.

When to Create a Custom Collection
The .NET Framework includes a variety of collection types, the most general of which are under the System.Collections namespace and subnamespaces (except for System.Array). I would strongly suggest reviewing what’s available there before writing a custom collection as the existing types will cover the large majority of your collection needs.
A case where you might need to create a custom collection is when you are trying to implement a special data structure that does not yet exist in the Framework. Graphs, trees, and other special data structures come to mind. They can be very useful in many situations. When you want a variant of an existing collection type, you must also create a custom collection. It could be that you want the collection to host only certain type(s) and that the specific implementation of this collection is dependent on these types. You may need a custom collection for performance; if you have special performance needs that cannot be addressed with one of the provided general-purpose collections, writing a custom collection may be your only option. For example, you may need a list where you can insert items into the middle very quickly while at the same time you can find items just as quickly without having to iterate over the list.

Existing Collections
If you use an existing collection, you’ll still need to choose the right one. Depending on what you are building, there are a variety of different collections that you can choose from. First, you need to determine what is going to be stored in the collections and then decide how the collection is going to be used. Are you going to be adding all the elements at once? Are you going to be dynamically adding and removing elements? Do you only need to iterate over the entire collection, or do you also need to locate and use specific elements? Are you exposing your collection through a public API? What are the performance characteristics and requirements of your application? Asking these questions will help you decide which collection to use, and if the options don’t suit your needs, consider extending the collection rather than starting from scratch.

Generic or Non-Generic
The .NET Framework 2.0 introduced the notion of generics. In the .NET Framework 1.x, most of the collection classes stored System.Object references. This meant that every time you extracted an item out of a collection you would have to cast it back to its original type (and if you developed a library, the consumer of that library was required to know what those types were). This cast had a performance impact, especially when storing value types, as adding one to a collection required it first to be boxed into a System.Object and later unboxed when pulled out of the collection and cast back to its original type.
By using generic collection types (found under the System.Collections.Generic namespace), you can avoid these issues. For example, if you need a list of strings in your C# application, you can use System.Collections.Generic.List<String> (List(Of String) in Visual Basic® and List<String^> in C++). This ensures that when you extract elements, they only come in the System.String type (meaning you don’t need to cast from Object to String). It also allows the compiler to enforce that behavior by providing a compile-time error if you try to store anything else into the list. Moreover, if you are developing a library to be consumed by someone else and you don’t know the contained type he’ll be using, you can continue to use a generic type T that the consumer of the library will later define. You can also constrain this T so that you know something about the types that will be used (that they derive from a specific set of interfaces, that they implement a parameterless constructor, and so forth).
If you’re using the .NET Framework 2.0 or later, using the old non-generic collections is strongly discouraged. The only reason you may need to keep them is for legacy purposes or if you need your code to work with older versions of the Framework. Using the old non-generic collections may slow down your code and also make it less readable. Even if you need to store more than one type in a collection, you can still use generic collections. You just need to find a common base-type for the objects you are about to store; in the worst case, you can use System.Object as the generic type parameter to be able to store anything (such as List<Object>), since every type derives from System.Object.
If you have legacy code using non-generic collections, I strongly recommend you consider changing your code to use generic collections. The table in Figure 1 shows recommended types and interfaces you should use. In addition, the .NET Framework 2.0 includes new generic collections that don’t have direct counterparts in the .NET Framework 1.x—for example, LinkedList<T> and SortedDictionary<TKey, TValue>. The .NET Framework 3.5 also includes a HashSet<T> class that provides new standard set functionality (in previous versions, the need for a set was typically satisfied using a Hashtable or Dictionary<TKey,TValue>).

Performance Implications
Designs that require collections frequently come from the need to work with a large number of items, and the number of items you work with will affect the performance of your application. To make sure you choose the right collection type for your needs, it is important to understand the usage patterns that are expected from your collection. For example, you may be inserting items to the collection only on initialization of your application, or you may be doing so throughout the lifetime of your program. Your only interaction with the collection may be based on iterating through all of the elements, or you may need random access capabilities. Understanding your usage profile allows you to find the best collection that fits your needs.
When you want additions, removals, and lookups to be very quick, and when you are not concerned about the order of the items in the collections, your first inclination should be to use a System.Collections.Generic.Dictionary<TKey, TValue> (or if you’re using the .NET Framework 1.x, a Hashtable). The three basic operations (Add, Remove, and Contains) all operate quickly even if the collection contains millions of items. On the other hand, with a List<T> (or ArrayList in the .NET Framework 1.x), inserting and removing items can take a variable amount of time. (Both List<T> and ArrayList store items in an underlying array that maintains order. Adding items may require existing items in the underlying array be moved to make room. Adding items at the end will not require any moves and will be very quick).
If your usage pattern requires few deletions and mostly additions, and if it is important for you to keep the collection in order, you may still want to choose a List<T>. Lookup could be slow (since the underlying array will need to be traversed while searching for the target item), but you are guaranteed that your collection maintains a specific order. Alternatively, you can choose Queue<T> to implement a first-in-first-out (FIFO) order or a Stack<T> to implement a last-in-first-out (LIFO) order. While both Queue<T> and Stack<T> support enumeration of all items in the collection, the former only supports insertion at the end and removal from the beginning, while the latter only supports insertion and removal from the beginning.
Using the new LinkedList<T> collection can potentially help you improve performance in scenarios where you need to maintain order yet still achieve fast inserts. Unlike List<T>, LinkedList<T> is implemented as a chain of dynamically allocated objects. In comparison to List<T>, inserting an object in the middle only requires updating two connections and adding the new item. The downside of a linked list from a performance point of view is increased activity by the garbage collector as it has to traverse the entire list to make sure objects were not disposed of. Additionally, performance problems can arise with large linked lists due to the overhead associated with each node as well as where in memory each node lives. And while the actual act of inserting an item into a LinkedList<T> is much faster than doing so into a List<T>, finding the particular location at which you want to insert a new value still requires traversing the list to find the correct location.
As mentioned, for scenarios where you need fast insertions and lookups, a Dictionary<TKey, TValue> may be most appropriate. However, lookup is only fast in the average case, and certain datasets could cause that performance to degrade quickly. A different implementation, SortedDictionary<TKey,TValue>, uses a balanced tree implementation as the underlying data store; this provides relatively fast lookups and maintains items in a sorted order, but insertions will most likely be slower (and will vary based on the number of items in the collection). Alternatively, you can also use SortedList<Tkey,TValue>, which uses two separate arrays to maintain the keys and the values separately and to maintain them both in order (in the worse case you would have to shift all the keys and values).

A Weather Almanac Using a List<T>
The following example shows a simple application to calculate the average temperature and humidity for a list of weather reports from various cities (see Figure 2). The WeatherReport class is used to store a single weather event for a selected city. The List<WeatherReport> collection is used to store the information that I’ll be averaging. I chose a List<T> collection in this case because the pattern is that I only add items and never remove them. Also, I do not care about sorting or inserting items in the middle, nor do I care about looking up individual items in the list. The only other thing I do with the list is iterate over the items. If you so choose, you can extend this program to find the hottest or coldest city and many additional weather almanac type operations.

Custom Collections
In certain situations, you may not find that all of your requirements are implemented in any of the existing collection types in the Framework. It may be that you have a unique set of requirements or that you can still use an existing collection with minor changes. If you decide you want to write a custom collection, you should first look into extending an existing collection type.
Start by looking at the System.Collections.ObjectModel namespace. These collections already implement the most needed interfaces and give you the baseline functionality you expect. It’s easy to add methods to inherited classes that enable your specific needs.
As an example, let’s say you want to implement a simple collection of System.Double values, but you also want to support an Add(string) method that uses Double.TryParse, only adding the result if the value can be parsed to a Double successfully. The System.Collections.ObjectModel.Collection<T> class implements all of the standard collection interfaces, and your custom collection type can simply derive from Collection<Double> in order to provide the additional overload of Add:
class DoubleCollection : Collection<double>
{
    public void Add(string st)
    {
        Double d;
        if (Double.TryParse(st, out d)) base.Add(d);
        else throw new ArgumentException(
            “Cannot parse string to a double. “ +
            “Item was not added to collection.”);
    }
}
The .NET Framework 3.5 with its introduction of extension methods provides an additional mechanism for creating methods that extend existing types. For example, you could rewrite this method as an extension method in C# 3.0 as follows:
class CollectionExtensions
{
    public static void Add(this Collection<Double> c, string st)
    {
        Double d;
        if (Double.TryParse(st, out d)) c.Add(d);
        else throw new ArgumentException(
            “Cannot parse string to a double. “ +
            “Item was not added to collection.”);
    }
}
Given a Collection<Double> that holds named values, this static method can be used to add a new value to the collection using traditional syntax:
CollectionExtensions.Add(values, “3.14”);
However, as this is an extension method (evident from the "this" keyword attributing the first parameter to the static Add method), you can rewrite this as follows:
values.Add(“3.14”);
This is the exact same method call you would write if you’d extended Collection<Double> with a new type, but you can now use this new Add method with any instance of Collection<Double> (or any type that derives from it), rather than just with your custom type. Alternatively, you can start building your own collection without inheriting from any of the existing ones. In that case you can take advantage of the appropriate interfaces that are provided in the .NET Framework.

Implementing Collection Interfaces
If you do decide that you want to write your own custom collection from scratch, you should implement all of the appropriate interfaces provided with the .NET Framework. These interfaces expose common ways to interact with collections that most people find very intuitive and easy to use. Here is a short overview of the various interfaces you can choose from.
IEnumerable This interface allows the collection to be enumerated by returning an enumerator for the collection. The interface exposes a single method, GetEnumerator, that returns an IEnumerator object. The IEnumerator object allows for quick traversal of the collection. Any collection that implements this interface can be accessed using the foreach language construct in C# (For Each in Visual Basic).
IEnumerable<T> This interface extends the non-generic IEnumerable by adding an additional GetEnumerator method that returns an IEnumerator<T> rather than IEnumerator. This means you must also implement IEnumerable if you plan to implement IEnumerable<T>. The difference is that this additional GetEnumerator returns the generic version of IEnumerator (IEnumerator<T>) instead of the non-generic one. It is a common practice to provide your collection’s enumeration logic in IEnumerable<T> and then have the non-generic GetEnumerator method delegate to the generic GetEnumerator method to assure that both of them return the same enumerator (and to prevent you from needing to implement the same logic twice).
IEnumerator This interface is not typically implemented by the collection itself. I mention it here as it is a useful interface to keep in mind in the context of building a custom collection. This interface is used to allow the enumeration of the items in a collection. It exposes three methods: Reset, Current, and MoveNext. The Reset method is used to initialize the enumeration. The Current method is used to get the current item, and MoveNext is used to move to the next item. If you can get an enumerator for the collection using GetEnumerator, then you can implement a foreach loop. For example, if Words is an ArrayList of strings, the following would do the same:
foreach (string word in Words)  Console.WriteLine (word);
or
IEnumerator enumerator = Words.GetEnumerator();
while(enumerator.MoveNext())
    Console.WriteLine(enumerator.Current);
IEnumerator <T> The generic version of the IEnumerator interface extends the non-generic version by adding an overload of the Current property that returns a T. This overload makes enumerating the collection easier by removing the need to cast.
ICollection This interface is an extension of IEnumerable. It also exposes three additional properties and one additional method: Count is a read-only property that returns the number of elements in the collection; IsSynchronized indicates that this collection is thread safe so that you can use it in a multithreaded application; SyncRoot is used to allow implementers to implement a thread-safe collection by providing an object that can be used for this purpose; and CopyTo is used to copy items from the collection to an array at a given index. For the most part, I would suggest avoiding the non-generic ICollection interface as the generic one is much more useful.
ICollection<T> This interface is an extension of IEnumerable<T> (and thus also of IEnumerable). It also exposes five new methods and two new properties. The methods are Add, which is used to add a new element to a collection; Remove, which is used to remove an element; Clear, which is used to remove all elements; Contains, which tells us if an item is in the collection or not; and CopyTo, which is used to copy items to an array. The two read-only properties are Count (as in the non-generic ICollection) and IsReadOnly, which tells us if the collection is read-only, meaning that items cannot be removed or added (mutable items in the collection can still be modified even if the collection is read-only).
This interface is very useful as a starting point for any custom collection. If you don’t know the type of the items you want to store in your collection (or you want to store more than one type), just use ICollection<Object> and you essentially get a non-generic extended ICollection interface.
IList and IList<T> IList is an extension of ICollection. In addition to everything that is in ICollection (the generic or non-generic version), it has IndexOf, Insert, and RemoveAt methods as well as an indexer that allows you to get and set any item in the collection, just as you can when using arrays. The non-generic IList also includes the methods that exist in the generic ICollection and not in the non-generic one (Add, Remove, Clear, and Contains). The IList interface is very useful when creating custom collections that allow you to add and remove items at will and get and set values of any item. The RemoveAt method allows you to remove an item based on its index (whereas the regular Remove only removes based on the value). The Insert method allows you to add an item at any position in the list, whereas the regular Add only adds it last.
IDictionary and IDictionary<TKey, TValue> IDictionary is an extension of ICollection. It supports the concept of using a key to store and locate items in the collection. In addition to all the methods that ICollection supports, it also exposes another Add overload that associates a key with the item that is added. The ContainsKey method will check for the existence of a particular key. There’s a Keys and Values properties that returns an ICollection of all the keys or values in the IDictionary, a remove overload based on the key instead of the value, and an indexer using the key. The generic version of this interface is a little more complex, as it allows you to specify the type for the value and the type for the key separately.

Generic or Non-Generic Again
When creating a custom collection, it may not be necessary to make it generic. If you know the types of objects that your collection is going to store, and you know that these types will not change in the future, using a fixed-type collection makes sense. You can still extend a generic framework collection or use generic interfaces, but your class will always store objects of a particular type.
If you are building a library, or a more general-purpose collection such as a tree or a graph, it’s probably a good idea to make it generic. This will enable the consumer of this type to use it with various types without the penalty of using an Object and casting it back and forth. Here is what a general-purpose directional graph class might look like:
class DirectionalGraph<T> :
    IEnumerable<T>, ICollection<T>, IList<T>
{
    public void Clear() { }
    public void Add(T item, IEnumerable<T> inNodes,
        IEnumerable<T> outNodes) { }
    public void Remove(T item) { }
    public bool Contains(T item) { }
    static public DirectionalGraph<T> EmptyGraph { get{}}

    // implement the interfaces...
}

A Family Tree
In the next example, I’ll implement a custom collection to store a family tree for genealogical purposes (see Figure 3). For simplicity, I’ll assume the following: each person has a sex and an age, and each person has two biological parents. Let’s call them Mother and Father.
The tree is a special type of collection that consists of nodes. Each node (the Person class, as in Figure 4) stores information about other nodes in the tree. There’s a special node called Root, which is the base of the tree. All other nodes could be assessed from the root traversing the tree. I also implemented IEnumerable<T> to allow enumerating all the members in the FamilyTree in a foreach loop. Once all persons have been entered into the tree you should be able to answer the question: what is the relationship between person A and B?
Note that not all the possible relationships are implemented in this example, but you can add more relationships to the FindRelationship method yourself. Also note that for clarity of the example, not all methods have adequate parameter checking. It is recommended that you make sure that every method only accepts valid values for its arguments before implementing it.
When you run the example in Figure 5, you should see the output that is shown in Figure 6.
Figure 6 Output from the Family Tree Sample
Figure 6 Output from the Family Tree Sample
By now you should have a pretty good understanding of collections in the .NET Framework, what they are and when and why you would want to use them. Collections can be a powerful tool in your programming arsenal so it’s important to know when they will benefit you and how to use them wisely. If you’re looking for more information, check out the resources in the "Further Reading" sidebar.

Send your questions and comments to clrinout@microsoft.com.

Inbar Gazit is a Program Manager focusing on the base class libraries for the CLR team. He can be reached at inbar@microsoft.com.


Basic Instincts
Query Comprehensions
Scott Wisniewski

Code download available at: BasicInstincts2007_08.exe (165 KB)
Browse the Code Online
Contents
Basics of Query Translation
Delayed Evaluation
Query Operators
Expression Trees
Sample LINQ Provider

In the June 2007 issue of MSDN® Magazine, Kit George and Ting Liang provided an introduction to some of the new features being introduced in Visual Basic® 9.0 to support Language Integrated Query, or LINQ. In particular, they showed how LINQ is a universal data platform designed to revolutionize the way data-intensive applications are written.
In this column I am going to expand on what Kit and Ting wrote by providing some background on the key concepts that drove the design of LINQ. To accompany this column, I’ve provided a custom query provider that showcases how this information can be put to use in a real-world applications; you can download it from the MSDN Magazine Web site. The material outlined here is far from introductory and assumes a working knowledge of database programming and a passing familiarity with LINQ. For more information on this subject, see the "Resources" sidebar.

Basics of Query Translation
LINQ brings both simplicity and flexibility to query translation. It is simple because the translation used by the compiler is straightforward and easy to understand. It is flexible because third-party developers can easily plug in their own behaviors. To get a feel for how this all works, let’s consider a brief example:
Imports System.Linq

Module SimpleExample
    Sub Main()
        Dim names() As String = {“Fred”, “George”, “Scott”, “Steve”}
        Dim letter = “S”
        Dim q = From name In names _
                Where name(0) = letter _
                Select name
    End Sub
End Module
Here I define a simple query that selects all strings that start with the letter S from an array. The compiler evaluates this by first splitting the query up into its component parts, then taking the clauses embedded in each part of the query and converting them into inline functions. It then spits everything back out as a series of chained method calls off of the object being queried. The final result should look like this:
Dim Clause1 as Func(of String, Boolean) _
    = Function (name) name(0) = letter
Dim Clause2 as Func(of String, String) _
    = Function (name) name

Dim q = names.Where(Clause1).Select(Clause2)
The Microsoft® .NET Framework 3.5 includes default implementations of the standard query operators. However, a library author or component vendor can provide his own implementations if he desires different or augmented functionality. For example, to modify the behavior of this specific query, a library author or component vendor need only write two methods: one named Where that is responsible for filtering the collection it was provided and another named Select that performs a projection or mutation of the objects in its collection. Similarly, if a library author wanted to support more advanced queries, such as those involving Group By, Order By, or Join clauses, she would simply need to define extra methods corresponding to each operator she wished to enable.
The Clause1 and Clause2 parameters passed to the Where and Select methods are delegate values created using inline functions. Inline functions are a new feature introduced in Visual Basic 9.0 that enable delegates to be constructed on the fly by simply writing expressions. In addition to offering convenient syntax, they also enable the constructed delegates to access the symbols (including local variables) visible in the scope in which they were created. This makes them suitable for use in translating queries, which naturally need to have access to the variables declared around them.
In the Where example, the provided expression compares the values in the collection being processed against the local variable letter. Without inline functions and their ability to capture the values of their containing scopes, implementing this type of functionality would be much more difficult. Because of inline functions, the rules used to translate queries into methods remain relatively simple. The compiler simply needs to gather all the expressions embedded in a query, convert them into inline functions, and then pass them off as delegate arguments to the appropriate query operator methods.

Delayed Evaluation
One of the most fundamental concepts underlying LINQ is the idea of delayed evaluation. In particular, the execution of queries in LINQ is significantly different from the execution of queries in traditional database management systems because database systems tend to have a fundamental understanding of the data they query over. LINQ has no such understanding. Instead, it simply plays dumb, converting each query into a series of chained method calls. The implementations of particular query operators are then responsible for intelligently and efficiently executing the query. Database systems, on the other hand, utilize significant knowledge about both the structures of queries and the nature of the data they are querying over (such as index availability and distribution statistics). Consequently, they take a holistic approach to query evaluation rather than the compositional approach used by LINQ.
When a DBMS sees a query, it always sees the entire query and, therefore, executes it immediately. LINQ sees queries in pieces, processing one operator at a time. If LINQ were to take the same approach as a DBMS and fully execute each piece as it is processed, the resulting performance would be rather poor. To see why this is true, consider the following example:
Dim q = From x In Xs _
        Where x.Value > 10 _
        Select x * 3.14
Here I simply take all the items in an array with a value greater than 10 and multiply them by pi. When translated by the compiler, this code looks something like the following:
Dim q = xs.Where(Function(x) x.Value > 10).Select( _
    Function (x) x  * 3.14)
If these query operator methods were implemented natively and completely executed the desired operation on their input collection, computing the whole query would require two passes over the original collection: one to filter it with the Where clause and another to project out the desired multiplication in the Select clause. This would be inefficient in both time and space as it would process its input twice (when clearly the combined operation of filtering and projection can be done in one pass). It would also need to store a copy of the collection in order to contain the results of the intermediate operation. Obviously this is undesirable.
To get around this, all built-in LINQ providers utilize a concept known as delayed execution. Rather than having query operators execute immediately, they all simply return types that implement the IEnumerable(of T) interface. These types then delay execution until the query is actually used in a for each loop. In this way many of the performance gains available in the holistic approach used by database systems are also available in LINQ. Effectively, each operator simply injects information indicating its presence into a runtime representation of the query. When iterated over, these queries are then executed as a single unit.
In the case of LINQ to SQL, this means that the query provider can delay generation of SQL code until the entire query has been processed, allowing it to integrate well with the holistic approach used by DBMSs. In the case of LINQ to Objects, this means that the underlying collections are processed a minimal number of times and that costly intermediate storage is only allocated when absolutely necessary.

Query Operators
The third and most important stop in this incursion into LINQ is a study of the query operators that make it up. Altogether, LINQ defines 48 distinct query operators that are collectively referred to as the standard query operators. I don’t, however, explore all of them here. Many of them are designed to simply be called as methods defined on collection objects. Others either give little insight into the way LINQ works or are very similar to other operators. Also, due to space constraints, I can’t give in-depth explanations of them all. As a result, let’s focus attention on the most interesting of the operators for which the Visual Basic compiler defines query comprehensions.
Select The Select clause is used to project items out of a collection. For example, it can be used to select a subset of the columns in a database table or to transform the elements of a collection. It is implemented by a method named Select and must be defined by every query provider. This is because all queries end in an implicit Select clause even if one is not explicitly provided in code. Consequently, the Visual Basic compiler will not allow a type to be queried if it does not define one. It should be defined using a signature similar to this:
Public Sub [Select](of OutputElementType)( _
    selector as Func(of InputElementType, OutputElementType)) _
        as IEnumerable(of OutputElementType)
End Sub
The selector parameter must be a delegate that maps from the source collection’s element type to the out collection’s element type. In general, the method should be generic and selector should be parameterized on its return type. If it is not, then any query features that rely on anonymous types will be disabled. This includes multiple element Select lists, Join clauses, and queries with multiple From clauses.
SelectMany The SelectMany operator is used to implement cross joins. Every query containing multiple From clauses will result in one or more calls to the SelectMany method. As an example, consider the following queries:
Dim q0 = From x In xs, y In ys
Dim q1 = From x In xs From y In ys
Here we define two queries that perform a cross join between an inner collection xs and an outer collection ys. When compiled they will look something like this:
q0 = xs.SelectMany(Function(ByVal x) ys, _
    Function(ByVal x, ByVal y) _
        New With {.x = x, .y = y}).Select(Function(ByVal it) it)

q1 = xs.SelectMany(Function(ByVal x) ys, _
    Function(ByVal x, ByVal y) _
        New With {.x = x, .y = y}).Select(Function(ByVal it) it)
A sample definition of the SelectMany method is shown here:
Public Function SelectMany(of InnerElementType, ResultType) _
    (collectionSelector as Func(of OuterElementType, _
        IEnumerable(of InnerElementType)), _
     resultSelector as Func(of InnerElementtype, OuterElementType, _
        ResultType)) as IEnumerable(of ResultType)
End Function
SelectMany takes two delegates as parameters and returns a single queryable collection as a result. The first delegate parameter, collectionSelector, is used to calculate an inner collection given an element of the outer collection. The second delegate parameter, resultSelector, is used to combine an element from the outer collection with an element from the inner collection to produce an element in the final resultset. The method should also be generic and parameterized on both the inner collection element type and the element type of the resultset.
Because the method takes in a collection selector (rather than a collection) as an argument, it is possible to do several interesting things with the From clause in Visual Basic. One particularly useful technique is to join a collection of objects with child collections of its elements. This can be used to "flatten" an object graph into a relational representation suitable for storage in a database, or for interoperability with legacy systems. For example, the query shown here will take a collection of Customer objects with embedded Order collections and convert it into a flat representation that looks similar to the results of an outer join between disparate Customers and Orders collections.
Dim q = From c In customers, o In c.Orders _
        Select c, o
Where The Where operator is used to filter a collection based on a logical condition. Its signature should look something like the following:
Public Function Where( _
    ByVal predicate As Func(Of ElementType, Boolean)) _
        As IEnumerable(Of ElementType)
End Function
Join The Join operator is used to perform an inner join between two collections. A sample usage of it is shown here:
Dim q = From usCust In usCustomers _
        Join euCust In europeCustomers On _
            usCust.CompanyName = euCust.CompanyName
        Select euCust.CompanyName
In this case, I simply join together a set of United States Customers with a set of European Customers where both have identical company names.
In general, Join is similar to the SQL inner join operator, but is slightly more restrictive with the set of expressions that may appear inside an On clause. In particular, LINQ only allows equality comparisons to appear in the Join clause, and requires all compound logical operations to be connected using the AND operator. The equality comparisons must utilize the new Equals operator, each side of the operator must reference only one of the two collections involved in the Join, and each side must reference a different one of the two collections. For example, the On clauses shown here are all legal Visual Basic syntax:
On c.CustomerId Equals o.CustomerID

On usCustomer.LastName Equals euCustomer.LastName and _
   usComster.FirstName Equals euCustomer.FirstName
Whereas the On clauses shown here are not legal Visual Basic:
On c.CustomerID = o.CustomerID

On usCustomer.CompanyName Equals euCustomer.ComapnyName or _
    usCustomer.FirstName Equals euCustomer.FirstName

On usCustomer.CompanyName <> euCustomer.CompanyName and _
    usCustomer.LastName = euCustomer.LastName

On usCustomer.LastName & “, “ & euCustomer.FirstName Equals _
    euCustomer.LastName & “, “ & usCustomer.FirstName
The operator itself is implemented by a method called Join with a signature similar to this:
Public Function Join(Of InnerElementType, KeyType, ResultType)( _
    ByVal inner As IEnumerable(Of InnerElementType), _
    ByVal outeKeySelector As Func(Of ElementType, KeyType), _
    ByVal innerKeySelector As Func(Of InnerElementType, KeyType), _
    ByVal resultsSelector As Func( _
        Of ElementType, InnerElementType, ResultType)) _
    As IEnumerable(Of ResultType)
End Function
The inner parameter represents the inner collection with which to join the current (or outer) collection. The outerKeySelector and innerKeySelector parameters are delegates used to extract the join key from the outer and inner collections, respectively. The resultSelector is used to combine an element from the outer collection and an element from the inner collection to form an element in the resultset.
The compiler is able to determine both the type of the join key and the expressions used to build the key selector arguments by examining the Equals operator used in a join’s On clause. In particular, the side of an Equals operator used to reference the inner collection is associated with the inner key selector, and the side used to reference the outer collection is associated with the outer key selector. When multiple Equals operators are connected using AND operators, the compiler will combine the selector expressions for each Equals instance into a single anonymous type constructor. For example, say you execute this On clause:
On usCompany.CompanyName Equals euCompany.CompanyName and _
   usCompany.StockSymbol Eqauls euCompany.StockSymbol
The clause will result in the key selectors shown here:
Dim innerKeySelector = Function (usCompany) _
    New With {.CompanyName = usCompany.CompanyName, _
              .StockSymbol = usCompany.StockSymbol}

Dim outerKeySelector = Function (euCompany) _
    New With {.CompanyName = euCompany, _
              .StockSymbol = euCompany.StockSymbol}
The implementation of the Join operator will then use these key selectors to uniquely identify the elements of each collection and to associate related elements with each other.

Expression Trees
So far, all of the query operator implementations shown in this column have operated using methods that take delegates as arguments. The delegates are then used to provide executable representations of the expressions they were generated from. For in-memory collections of Objects, this is sufficient. For example, when walking over a source collection, a Where implementation can simply invoke its supplied delegate on each element it encounters. For query providers that need to efficiently execute queries on remote database servers, such as LINQ to SQL, a delegate representation of a query clause expression is not sufficient, however. In particular, in order to generate remotely executable SQL code, a LINQ provider needs to have a program-readable representation of the clauses actually provided by the user. Without them, implementing LINQ to SQL efficiently would be largely impossible.
Fortunately, LINQ has support for this in the form of expression trees. In addition to allowing query operator implementations to take in delegates as parameters, the Visual Basic compiler also allows them to declare parameters as types derived from Expression(of T), where T corresponds to some delegate type. For these parameters the compiler converts the inline functions it generates into trees that describe the actual expression typed by the user. By reading these trees, and appropriately translating them to SQL code, LINQ providers are then able to marshal queries written in Visual Basic over to a database server where they can be executed efficiently. Figure 1 shows a diagram representing the layout of the expression tree generated for the following inline function:
Figure 1 Expression Tree for a Simple Inline Function
Figure 1 Expression Tree for a Simple Inline Function
Dim e As Expression(Of Func(Of Integer, Boolean)) = _
    Function(ByVal i) i > 5 AndAlso i Mod 2 = 1
And here is the code generated by the compiler for it:
Dim param = Expression.Parameter(GetType(Integer), “i”)
Dim body = Expression.AndAlso( _
    Expression.GreaterThan(param, Expression.Constant(5)), _
    Expression.Equal( _
        Expression.Modulo(param, Expression.Constant(2)), _
        Expression.Constant(1)))
Dim e = expression.Lambda(body, param)

Sample LINQ Provider
Walking through the actual implementation of a sample LINQ provider is outside the scope of this column. Fortunately, you can download a working sample from the MSDN Magazine Web site. The sample defines several custom query providers that enable LINQ query execution to be measured using PerfMon. In addition to augmenting the information presented here, the sample is also a very useful tool for optimizing the performance of your LINQ-enabled applications.
Resources
  • LINQ: .NET Language-Integrated Query
  • Standard Query Operators
  • Database Programming Developer Center

Send your questions and comments to instinct@microsoft.com.

Scott Wisniewski is a Software Design Engineer at Microsoft, where he works on the Visual Basic compiler. For the upcoming Visual Studio "Orcas" release, he worked on several features, including nullable types, error correction, and extension methods. You can contact Scott at scottwis@microsoft.com or through the Visual Basic Team blog.


Office Space
Solution Deployment with SharePoint 2007
Ted Pattison

Code download available at: OfficeSpace2007_08.exe (229 KB)
Browse the Code Online
Contents
Solution Packages
Deploying the OfficeSpaceFeature Component
Generating the CAB File for a Solution Package
Deployment
Summary

In the May 2007 issue of MSDN® Magazine, I discussed the fundamental concepts involved in developing custom features and application pages to create business solutions with Windows® SharePoint® Services (WSS) 3.0. Features and application pages represent two types of WSS components that are the primary building blocks a developer can use to extend a SharePoint 2007 site. There are several other WSS component types used in SharePoint development, including site definitions, page templates, Web Parts, event handlers, and workflows.
While my May column focused on how to create, test, and debug WSS components, this month I’ll look at the deployment side. Once you have fully tested a set of components, WSS provides an approach to move your development efforts into a staging or production environment. More specifically, WSS 3.0 introduces a new deployment mechanism, known as a solution package, that allows you to create an installation procedure that is supported, testable, and repeatable. What’s really nice about solution packages is that they allow you to create a single distributable file that abstracts away the details of whether the deployment target is a single SharePoint server or an enterprise-scale server farm with multiple front-end Web servers.

Solution Packages
Solution packages are the preferred mechanism for deploying WSS components. The solution package itself is a compressed CAB file with a .wsp extension, and it contains one or more WSS components along with any dependent files that need to be deployed on each front-end Web server.
A simple solution package might contain just the files needed to deploy a single feature. A more complex package could contain the files for multiple features, applications pages, Web Parts, list definitions, event handlers, and a site definition. You decide what goes into a solution package according to what set of components it makes sense to deploy as a single unit.
The WSS runtime provides a built-in installer component that runs on each front-end Web server and is responsible for uncompressing the files inside a solution package and properly installing its components. The WSS installer requires each solution package to carry additional metadata inside a file named manifest.xml. When the installer is called upon to deploy a solution package, it reads the metadata in manifest.xml to determine exactly which components and files from inside the CAB file need to be extracted and deployed.
Manifest.xml contains metadata to instruct the installer which files need to be extracted from the solution packages and copied into various WSS system directories. But beyond that, manifest.xml also carries metadata that tells the installer to perform other important deployment procedures, such as registering features with the WSS runtime, adding assembly DLLs to the Global Assembly Cache (GAC), and updating the web.config file with SafeControl entries required in Web Part deployment.

Deploying the OfficeSpaceFeature Component
In the May column, I presented a business solution composed of a feature named OfficeSpaceFeature that works together with a custom application page named LetterGenerator.aspx. These two components were designed to work together to create a single business solution.
The sample code that accompanies this month’s column contains a Visual Studio® project named OfficeSpaceFeature. This project contains the source code for both the feature and the application page. While the sample code that accompanied the May column had an earlier version of this project, the download has now been updated to include the extra files needed to build and deploy all the required components and files as a solution package. Note that the project contains a \Solution directory and a \Package directory as shown in Figure 1.
Figure 1 Solution Package Components
Figure 1 Solution Package Components
When the OfficeSpaceFeature is activated within the context of a SharePoint site, it provisions a document library named Customer Letters. It also provisions a custom template file named LetterTemplate.docx, which serves as the document template behind the Customer Letters document library. This means that LetterTemplate.docx becomes a dependent file that must be deployed along with the feature within the \FEATURES directory on each front-end Web server.
The feature named OfficeSpaceFeature has also been created with an event handler that fires during feature activation. This event handler is defined within a SPFeatureReceiver-derived class and is compiled into an assembly named OfficeSpaceFeature.dll. WSS requires that any assembly DLL containing event handlers be compiled with a strong name and installed into the GAC to work properly.
Now that you have a basic idea of what components and files make up this business solution, let’s examine the feature.xml file for the feature named OfficeSpaceFeature (see Figure 2). As many other XML-based files used in WSS development, the feature.xml file contains XML that conforms to a WSS-specific language known as Collaborative Application Markup Language (CAML).
Figure 2 Feature.xml for OfficeSpaceFeature
<Feature Id=”AAEC2E08-1CCF-4712-AE5E-A33BEA53A325”
  Title=”A Sample Office Space Feature”
  Description=”Demoware from Ted Pattison’s Office Space column”
  Version=”1.0.0.0”
  Scope=”Web”
  ImageUrl=”OfficeSpace/PithHelmet.gif”
  ReceiverAssembly=”OfficeSpaceFeature, [full four-part assembly name]”
  ReceiverClass=”OfficeSpaceFeature.FeatureReceiver”
  xmlns=”http://schemas.microsoft.com/sharepoint/”>

  <ElementManifests>
    <ElementManifest Location=”elements.xml” />
    <ElementFile Location=”LetterTemplate.docx” />
  </ElementManifests>

</Feature>
The Feature element contains an ElementManifests element, which includes both an inner ElementManifest element and an ElementFile element. ElementManifest references a file named elements.xml that contains the CAML to define ListInstance elements, which provision a new document library and a new list. There is also a Module element used to copy LetterTemplate.docx into the context of the current site during Feature activation so it can serve as the document template behind the Customer Letters document library.
The ElementManifest element is an essential part of the Feature definition because it references elements.xml. That file, in turn, adds provisioning instructions that are carried out during Feature activation. But what is the purpose of the ElementFile element? It doesn’t really change the behavior of the Feature itself. Instead, its purpose is to add metadata used by the installer when the Feature has been included within a solution package. The ElementFile element informs the installer that LetterTemplate.docx must be copied to the OfficeSpaceFeature directory as part of the deployment process.
Now let’s turn our attention to the CAML defined in the manifest.xml file (see Figure 3). The manifest.xml file contains a top-level Solution element, which contains inner elements that reference components and files.
Figure 3 Manifest.xml
<Solution
  SolutionId=”848ACD1F-DEBC-4078-B21D-56CECE3F499B”
  xmlns=”http://schemas.microsoft.com/sharepoint/”>

  <FeatureManifests>
    <FeatureManifest Location=”OfficeSpaceFeature\feature.xml” />
  </FeatureManifests>

  <TemplateFiles>
    <TemplateFile Location=”LAYOUTS\OfficeSpace\LetterGenerator.aspx” />
    <TemplateFile Location=”IMAGES\OfficeSpace\PithHelmet.gif”/>
  </TemplateFiles>

  <Assemblies>
    <Assembly DeploymentTarget=”GlobalAssemblyCache”
      Location=”OfficeSpaceFeature.dll” />
  </Assemblies>

</Solution>
Note that the Solution element contains a SolutionId attribute whose value is assigned a unique GUID used to identify the solution package. There is also a FeatureManifest element that references the feature.xml file for the feature named OfficeSpaceFeature. Feature.xml references elements.xml and LetterTemplate.docx so that these two files are implicitly included as part of the solution package.
Next, take a look at the two TemplateFile elements nested inside the TemplateFiles element. The first TemplateFile element references the application page named LetterGenerator and specifically defines its path within the custom \OfficeSpace directory, which is located inside the \LAYOUTS directory. The second TemplateFile element references a graphic image file named PithHelmet.gif that will be deployed inside the \IMAGES directory. You should observe that the path assigned to the Location attribute within a TemplateFile element is relative to the \TEMPLATE directory.
The last element inside manifest.xml we need to examine is the Assembly element used to deploy OfficeSpaceFeature.dll into the GAC. Note that many types of WSS components that are compiled into assembly DLLs must be installed into the GAC as shown in Figure 3. Assembly DLLs containing Web Parts provide the option of installation in either the GAC or the bin directory of the hosting Web Application.

Generating the CAB File for a Solution Package
Now that you have seen what files need to be added to a solution package, let’s look at the steps involved in building the CAB file. Today, this is unfortunately a manual and somewhat tedious process. Given the lack of productivity tools, the most common approach involves using the makecab.exe tool. When you use this tool, you must define a .ddf file that tells makecab.exe what files to include in the output CAB file. Take a look at the file named CAB.ddf (see Figure 4), which is used to generate the solution package named OfficeSpaceFeature.wsp.
Figure 4 CAB.ddf
.OPTION EXPLICIT
.Set CabinetNameTemplate=OfficeSpaceFeature.wsp
.Set DiskDirectory1=Package
.Set Cabinet=on
.Set MaxDiskSize=0
.Set CompressionType=MSZIP;
.Set DiskDirectoryTemplate=CDROM;

Solution\manifest.xml manifest.xml

TEMPLATE\FEATURES\OfficeSpaceFeature\feature.xml
  OfficeSpaceFeature\feature.xml
TEMPLATE\FEATURES\OfficeSpaceFeature\elements.xml
  OfficeSpaceFeature\elements.xml
TEMPLATE\FEATURES\OfficeSpaceFeature\LetterTemplate.docx
  OfficeSpaceFeature\LetterTemplate.docx

TEMPLATE\LAYOUTS\OfficeSpace\LetterGenerator.aspx
  LAYOUTS\OfficeSpace\LetterGenerator.aspx
TEMPLATE\IMAGES\OfficeSpace\PithHelmet.gif
  IMAGES\OfficeSpace\PithHelmet.gif
bin\Debug\OfficeSpaceFeature.dll OfficeSpaceFeature.dll
As you can see, CAB.ddf begins with header attributes such as CabinetNameTemplate, which defines the name of the output CAB file. After that there is a separate line for each file that needs to be compiled into OfficeSpaceFeature.wsp. The first element in each line points to the location where the makecab.exe utility can find the file so it can copy it into the CAB file. For example, the path to feature.xml as it exists within the directory structure of the Visual Studio project is defined with this path:
TEMPLATE\FEATURES\OfficeSpaceFeature\feature.xml
The second entry in each line points to the location the file will have once it has been compressed into the CAB file. For example, the feature.xml file must be added to a directory named OfficeSpaceFeature:
OfficeSpaceFeature\feature.xml
Now that you have seen what CAB.ddf looks like, it’s time to put this file to work and use it to build the solution package using the makecab.exe utility. Note that the project structure shown in Figure 1 shows a separate directory named \Package that includes four batch files. These batch files script the important interaction between the makecab.exe and the stsadm.exe command line utilities. Let’s look at the batch file named CreateSolutionPackage.cmd that automates the call to makecab.exe and builds the solution package named OfficeSpaceFeature.wsp:
@echo off
if EXIST OfficeSpaceFeature.wsp del OfficeSpaceFeature.wsp

cd ..

makecab.exe /f Solution\cab.ddf

cd Package
You can see this batch file deletes the previous build of OfficeSpaceFeature.wsp if one already exists. It then changes directories to the main project directory so that when makecab.exe runs, the file references in CAB.ddf are correct. After running this batch file, you should have a newly generated build of the solution package named OfficeSpaceFeature.wsp.
If you are curious and you want to look at what has been compressed inside, you can change the file extension from .wsp to .cab and double-click on the file. Windows will then open the file as it would any other CAB file and display an internal view of its contents, as shown in Figure 5.
Figure 5 Looking Inside OfficeSpaceFeature.wsp
Figure 5 Looking Inside OfficeSpaceFeature.wsp

Deployment
Once you have built a solution package such as OfficeSpaceFeature.wsp, you deploy it within a SharePoint farm in two distinct steps. The first step is installation in which WSS copies the .wsp file to the configuration database. The second step involves the actual deployment in which WSS creates a timer job to be processed by all front-end Web servers in the server farm. The important observation here is that WSS automates solution package deployment across all the front-end Web servers in the farm to ensure a consistent deployment.
Let’s look at the next batch file named InstallSolutionPack age.cmd. It included a single command to the stsadm.exe command-line utility to run an operation named addsolution:
@SET STSADM=”c:\program files\common files\microsoft shared\
 web server extensions\12\bin\stsadm.exe”

%STSADM% -o addsolution -filename OfficeSpaceFeature.wsp
This is the operation that copies the solution package file into the configuration database and makes it available for deployment. At this point you have the option of deploying the solution package with another batch file or by using the standard user interface provided by the SharePoint 3.0 Central Administration Web site.
Keep in mind that deploying a solution package has a definite impact on a SharePoint farm. The deployment process on each front-end Web server involves copying files into WSS system directories, installing features, and adding assembly DLLs to the GAC. Furthermore, WSS always runs a command to restart all the IIS worker processes that might be effected by the new components that have just been installed. For this reason, in production environments you should deploy solution packages during off hours when the fewest number of users are using the system. Fortunately, WSS provides the convenience to schedule deployment jobs for a future time and date. For example, an administrator can schedule a deployment job for 1 A.M. to reduce or eliminate its impact on users.
My sample solution uses a batch file named DeploySolutionPackage.cmd. This batch file makes several calls to the stsadm.exe command-line utility. The first call runs the addsolution operation and is followed by a second call that runs the deploysolution operation:
@SET STSADM=”c:\program files\common files\microsoft shared\web server
    extensions\12\bin\stsadm.exe”

%STSADM% -o addsolution -filename OfficeSpaceFeature.wsp
%STSADM% -o deploysolution -name OfficeSpaceFeature.wsp -immediate -
    allowGacDeployment
%STSADM% -o execadmsvcjobs
In the call to stsadm.exe that runs the deploysolution operation, I have passed the parameters -immediate and -allowGacDeployment. The -immediate parameter instructs WSS to run the job right away. There are other parameters that can be passed to schedule a timer job for this deployment at some time in the future. The -allowGacDeployment parameter is required in this case because the solution package is installing an assembly DLL in the GAC.
Now look at the final call to the stsadm.exe command-line utility to run the execadmsvcjobs operation. This is a handy call to use when you are testing a solution package. This call has the effect of forcing WSS to run any pending timer jobs such as a solution package deployment job. Therefore, you don’t have to sit around and wait for WSS to start the deployment job.
Another powerful aspect of solution package deployment is that it supports retraction. That gives you the ability to effectively remove any WSS components and files that have been installed using a solution package.
Let’s take a look at the batch filed named DeleteSolutionPackage.cmd, which lets you undo a deployment.
@SET STSADM=”c:\program files\common files\microsoft shared\
 web server extensions\12\bin\stsadm.exe”

%STSADM% -o retractsolution -name OfficeSpaceFeature.wsp -immediate
%STSADM% -o execadmsvcjobs
%STSADM% -o deletesolution -name OfficeSpaceFeature.wsp
As you can see, the retractsolution operation is used to roll back a solution package deployment. In this case, running this batch file removes the assembly DLL from the GAC, uninstalls the Feature, and removes all files that were initially copied to the file system of the front-end Web server during solution package deployment.

Summary
This month’s column stepped through the concepts and basic steps of deployment using solution packages. Packaging up and deploying your WSS development efforts using solution packages is a best practice, and knowing how to do this should be considered an essential skill. It is also recommended that you test your solution package in an environment that is not the same as your development environment such as a clean Virtual PC image or a multi-server SharePoint farm that your company uses for staging and quality assurance testing. Using a disciplined approach like this will allow you to roll out custom WSS components such as features, site definitions, Web Parts, and workflows with confidence.

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. Ted has just completed his book Inside Windows SharePoint Services 3.0 for Microsoft Press. He also delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group.


Cutting Edge
Canceling Server Tasks with ASP.NET AJAX
Dino Esposito

Code download available at: CuttingEdge2007_08.exe (167 KB)
Browse the Code Online
Contents
Formalizing Remote Tasks
Canceling Tasks the Easy Way
Inside the abortPostBack Method
Designing an Interruptible Task
The Client Code
Transactions
Generating the Bar

Last month I built a framework to monitor ongoing server-side tasks from the client. Using this framework, which I’ll call Progress Monitor Framework (or PMF), you can provide Web users with information about the progress of operations running on the server, something that typically requires a lot of custom code to do. With PMF, you can have a server-side task register the current state of its operation (as a percentage or as an estimate of the time left) and a client service to ping the server to read that state information. Once the state information is downloaded to the client, which happens out-of-band, updating the UI is a breeze.
Some early feedback about last month’s column pointed out two potential enhancements. The first concerns whether PMF can be used to stop ongoing server tasks, and the second looks for a better way to generate the markup for the progress bar.

Formalizing Remote Tasks
A remote task is a piece of code that executes on the server in response to a client event. There are three possible ways for an ASP.NET AJAX client page to trigger a remote task: by causing a postback managed by an UpdatePanel control, by invoking a method directly on the application’s back end exposed through a local Web service, and by using a page method. Soon there will be a fourth method: a Windows® Communication Foundation (WCF) service.
Once the task on the server has been triggered, the client no longer has control over it. The client page regains control over the operation only after the response generated by the task has been downloaded to the client and parsed. With PMF you can read the status of the task on the fly, but there is no mechanism for passing data to the server task dynamically.

Canceling Tasks the Easy Way
ASP.NET AJAX can make canceling remote operations really easy, but there are two restrictions. First, the task must have been started using UpdatePanel. Second, no extra work should be required on the server to compensate for the abrupt interruption of the task. Figure 1 shows the source code of a sample UpdatePanel-based page that pops up a progress template with a Cancel button, as shown in Figure 2. By clicking the button, you cancel the operation. Or do you?
Figure 1 Canceling a Task Started via an UpdatePanel Postback
<html xmlns=”http://www.w3.org/1999/xhtml” >
<head runat=”server”>
    <title>Canceling tasks</title>
    <style type=”text/css”>
        #UpdateProgress1  {
            width: 270px; background-color: #ffff99; height:120px;
            top: 40%; left: 35%; position: absolute;
            border: solid 1px black;
        }
        #ProgressTemplate1  {
            font-size: 9pt; color: navy; font-family: verdana;
        }
    </style>
</head>

<script language=”javascript” type=”text/javascript”>
function abortTask()  {
    var obj = Sys.WebForms.PageRequestManager.getInstance();
    if (obj.get_isInAsyncPostBack())
        obj.abortPostBack();
}
</script>

<body>
<form id=”form1” runat=”server”>
    <asp:ScriptManager ID=”ScriptManager1” runat=”server” />
    <asp:UpdatePanel ID=”UpdatePanel1” runat=”server”
            UpdateMode=”Conditional”>
        <ContentTemplate>
            <asp:Button runat=”server” ID=”Button1” Text=”Start Task ...”
                onclick=”Button1_Click” />
            <hr />
            <asp:Label runat=”server” ID=”Label1” /><br />
        </ContentTemplate>
    </asp:UpdatePanel>
    <hr />

    <asp:UpdateProgress runat=”server” ID=”UpdateProgress1”>
        <ProgressTemplate>
            <div ID=”ProgressTemplate1”><p style=”margin:5px;”>
                <img alt=”” src=”Images/indicator.gif”
                    align=”left” />&nbsp;&nbsp;
                <span id=”Msg”>Your request has been submitted and
                    it may take a while to complete.
                <br /><br />Please, wait ... </span>
                <p align=”center”>
                <input type=”button” value=”Cancel”
                    onclick=”abortTask()” /></p>
            </div></p>
        </ProgressTemplate>
    </asp:UpdateProgress>
</form></body></html>
Figure 2 Progress Template with a Cancel Button
Figure 2 Progress Template with a Cancel Button
As you can see in the abortTask function in Figure 1, the progress template contains a client button bound to some JavaScript code. The first thing the function does is retrieve the page request manager. In the Microsoft® AJAX client library, the PageRequestManager object is the nerve center of partial rendering. Upon page initialization, the page request manager registers a handler for the form’s submit event. In this way, the request manager is involved each time the page is going to post back. At this point, the request manager makes a copy of the request’s body as prepared by the browser and runs it through the current HTTP executor—by default, the popular XMLHttpRequest object.
The page request manager sets up the eventing model of partial rendering and tracks the ongoing operation. If there’s any pending operation, then the Boolean property isInAsyncPostBack will return true.
When the user clicks the Cancel button shown in Figure 1, the page request manager aborts the current request through its abortPostBack method. The page request manager is a singleton object, meaning that all calls are conveyed to just one instance. The reason this is so is tightly related to the mechanics of partial rendering. Partial rendering consists of sending a page request that goes through the usual processing on the server (except for the rendering stage). Among other things, this means that the viewstate is sent over and used to rebuild the last known good state of server controls. Postback and state-changed events are fired regularly, and the viewstate is updated based on these operations. The updated viewstate is then sent back along with the partially modified markup.
Because of the viewstate, two asynchronous postback calls from the same page need to be serialized, and only one at a time is allowed to run. For this reason, the abortPostBack method on the page request manager has no need to figure out which request you want to stop—there’s at most one pending request.

Inside the abortPostBack Method
Let’s briefly take a look at the source code of the abortPostBack method on the PageRequestManager class:
function Sys$WebForms$PageRequestManager$abortPostBack()
{
    if (!this._processingRequest && this._request)
    {
        this._request.get_executor().abort();
        this._request = null;
    }
}
If there’s a pending request, the manager instructs the executor of the request to abort. The executor is a JavaScript class that inherits from Sys.Net.WebRequestExecutor and takes care of sending the request and receiving the response.
In the Microsoft AJAX client library, there’s only one executor class—the Sys.Net.XMLHttpExecutor class—and it uses the XMLHttpRequest object to execute a request. In brief, when the preceding code calls the abort method, it basically tells the XMLHttpRequest object to abort. Put another way, it simply tells the socket through which the executor receives response data that it must close.
Now, let’s suppose that the remote task performs a disruptive action on the server. For example, let’s say that the user is given a chance to delete a few records from a database table by clicking a button. Attempts to cancel the operation via the process described above doesn’t actually stop the server operation. All it does is close the socket through which you would receive a confirmation message. The abortPostBack method on the PageRequestManager object is merely a client-side method that has no effect on what’s going on in the server.

Designing an Interruptible Task
For the abort request to be effective on the server operation, the task must be interruptible. In other words, the task has to periodically check for instructions from the client that tell it to exit. A bidirectional version of PMF would help.
When I first implemented PMF, the client and server elements of the framework shared a common data container that the server used to write data about its progress and the client used to read that data, in order to update the user interface. Some enhancements are required to make the server code receive and process dynamic client feedback such as a click on the Cancel button.
The progress server API is now based on this contract:
public interface IProgressMonitor
{
    void SetStatus(int taskID, object message);
    string GetStatus(int taskID);

    bool ShouldTerminate(int taskID);
    void RequestTermination(int taskID);
}
I’ve added two new methods: ShouldTerminate and RequestTermination. The former returns a Boolean value indicating whether the ongoing task should be terminated. The RequestTermination method represents the entry point in the API for clients wanting to stop a task. When invoked, the method creates a task-related entry in the data container (the ASP.NET Cache) that ShouldTerminate checks to determine whether interruption was requested.
The IProgressMonitor interface defined above dictates the expected behavior of an application on the server. You can implement it in a variety of classes that may use distinct data containers. I created a sample class using the ASP.NET cache named InMemoryProgressMonitor (see Figure 3). Figure 4 provides a brief description of each method. As I discussed last month, the remote task calls SetStatus repeatedly to track its current execution status and mark its progress. To support dynamic interruption, the same task will periodically invoke ShouldTerminate to be informed of client requests to quit. Figure 5 shows the typical structure of a monitorable and interruptible task.
Figure 5 Monitorable and Interruptible Remote Method
[WebMethod]
public static string ExecuteTask(int taskID)
{
    InMemoryProgressMonitor progMonitor = new InMemoryProgressMonitor();

    // Preliminary check
    if (progMonitor.ShouldTerminate(taskID))
        return “Task aborted--0% done”;

    // First step
    progMonitor.SetStatus(taskID, “0”);
    DoStep(1);
    if (progMonitor.ShouldTerminate(taskID))
        return “Task aborted--5% done”;

    // Second step
    progMonitor.SetStatus(taskID, “5”);
    DoStep(2);
    if (progMonitor.ShouldTerminate(taskID))
        return “Task aborted--45% done”;

    // Third step
    progMonitor.SetStatus(taskID, “45”);
    DoStep(3);
    if (progMonitor.ShouldTerminate(taskID))
        return “Task aborted--69% done”;

    // Final step
    progMonitor.SetStatus(taskID, “69”);
    DoStep(4);

    if (progMonitor.ShouldTerminate(taskID))
        return “Task aborted--100% done”;

    return “Task completed at: “ + DateTime.Now.ToString();
}
Figure 4 IProgressMonitor Interface

Method | Description
GetStatus | Reads the current status of the specified task from the internal data storage and returns it to the client as a string.
SetStatus | Invoked from the task, it writes the current status of the task to the internal data storage. The status is expressed as an object.
RequestTermination | Creates a task-related entry in the internal data storage to indicate that the client placed a request for termination.
ShouldTerminate | Returns true if the client placed a request to terminate the specified task.
Figure 3 InMemoryProgressMonitor
public class InMemoryProgressMonitor : IProgressMonitor
{
    public const int MAX_TIME_MINUTES = 5;

    // Sets the current status of the task
    public void SetStatus(int taskID, object message)
    {
        HttpContext.Current.Cache.Insert(
            taskID.ToString(), message, null,
            DateTime.Now.AddMinutes(MAX_TIME_MINUTES),
            Cache.NoSlidingExpiration);
    }

    // Reads the current status of the task
    public string GetStatus(int taskID)
    {
        object o = HttpContext.Current.Cache[taskID.ToString()];
        return o == null ? string.Empty : (string)o;
    }

    // Captures any user feedback (i.e., abort button clicked)
    public bool ShouldTerminate(int taskID)
    {
        string taskResponseID = GetSlotForResponse(taskID);
        return HttpContext.Current.Cache[taskResponseID] != null;
    }

    // Sets the task for termination
    public void RequestTermination(int taskID)
    {
        string taskResponseID = GetSlotForResponse(taskID);
        HttpContext.Current.Cache.Insert(
            taskResponseID, (object) false, null,
            DateTime.Now.AddMinutes(MAX_TIME_MINUTES),
            Cache.NoSlidingExpiration);
    }

    private string GetSlotForResponse(int taskID)
    {
        return String.Format(“{0}-Response”, taskID);
    }
}
The method featured in Figure 5 orchestrates the various steps that make up the remote task. The task can be part of the application’s middle tier and can be implemented as a workflow. It must be articulated in individual steps so that the client can plug into it to read status and request termination.

The Client Code
The client JavaScript code that triggers the remote task remains mostly unchanged from last month’s. You can use a page or a Web service method to start a task such as the ExecuteTask method in Figure 5, or run the server code through an UpdatePanel region:
<asp:UpdatePanel runat=”server” ID=”UpdatePanel1”>
    <ContentTemplate>
        <asp:Button runat=”server” ID=”Button1” Text=”Start Task ...”
             OnClick=”Button1_Click” />
        <hr />
        <asp:Label runat=”server” ID=”Label1” /><br />
    </ContentTemplate>
</asp:UpdatePanel>
In the Button1_Click event handler, you define the remote task and make it call into a progress monitor object and the SetStatus and ShouldTerminate methods. To abruptly terminate a remote task, you add a Cancel button to the progress template—either an UpdateProgress control or a user-defined <div> block. This time, though, the click handler of the Cancel button doesn’t point to the abortPostBack method on the page request manager. Instead it points to your own abort method in the client progress API:
<script type=”text/javascript”>
var progressManager = null;
var taskID = null;

function pageLoad() {
   progressManager = new Samples.PMF2.Progress();
}

function abortTask() {
    progressManager.abortTask(taskID);
}

...
</script>
Let’s take a look at the modified client progress API. Coded in the progress.js file, this API must be linked to each ASP.NET AJAX page where you plan to use interruptible or monitorable tasks:
<asp:ScriptManager ID=”ScriptManager1” runat=”server”
    EnablePageMethods=”true”>
    <Scripts>
        <asp:ScriptReference path=”random.js” />
        <asp:ScriptReference path=”progress.js” />
    </Scripts>
</asp:ScriptManager>
The random.js file is a dependency of progress.js and defines a method that generates random numbers for tasks. To track the status of a remote task from the client, you poll the server at regular intervals. To stop an ongoing task or, more precisely, to place a request to stop a task, you invoke a server method exposed by the progress monitor server API as part of the application’s back end:
// Cancel the operation
function Samples$PMF2$Progress$abortTask() {
    PageMethods.TerminateTask(_taskID, null, null, null);
}
I opted for page methods to expose this client-callable function. Figure 6 provides an architectural view of the overall solution.
Figure 6 Bidirectional Progress Monitor Framework
Figure 6 Bidirectional Progress Monitor Framework
When the user clicks the Cancel button, an out-of-band call is triggered to execute the TerminateTask method defined as a page method on the codebehind class of the page. The TerminateTask method creates a task-related entry in the internal data storage (the ASP.NET cache). This entry is named after the task ID appended with a "Quit" suffix. A task designed to be interruptible checks this entry at various stages during its execution. If the entry is found, the server task aborts (see Figure 7).
Figure 7 The User Clicked the Cancel Button, Ending the Server Task
Figure 7 The User Clicked the Cancel Button, Ending the Server Task
Implemented in this way, task cancellation is more effective. If you simply abort the client postback during an UpdatePanel refresh, all that happens is that the client socket through which you receive responses is closed down. There is no impact on the code running on the server. There’s also no built-in way to programmatically stop a remote call to a Web service or page method. In this case, the JavaScript proxy class completely hides the request object being used to push the call. Although the request object and its executor have an abort method, there’s no way to find a reference to it in the context of service method call.
In the end, if you need to allow for remote task control, the Progress Indicator pattern is the only way to go. You set up a parallel channel to monitor the status and to pass further information such as a quit command, to the running task. The same architecture allows the client to change parameters on the fly or to request additional operations. The bidirectional progress monitor framework is a duplex channel that a server task and its JavaScript client can use to exchange data in the form of messages.

Transactions
So far I created a framework to monitor and stop ASP.NET AJAX tasks. It is essential to note that the framework simply notifies the task that the user requested that it terminate. If properly designed, the task promptly stops and returns. But what happens to the work it has done already?
In general, when a task is abruptly interrupted, it should undo any changes it has made and return. But the progress monitor framework can’t make that happen. If you wrap the remote task in a transaction, however, you can roll it back once the task has been interrupted, assuming the work done only touches transactional resources, like a SQL Server™ database. Another option is to use a workflow. In that case, you’d wrap the task in a TransactionScope activity and use a Code activity to set the current status and check for any termination request. If the task has to terminate, you throw an exception and automatically cause the transaction to roll back. (See the June 2007 Cutting Edge column for more information about transactional tasks in Windows Workflow Foundation.)
Unfortunately, not all operations can easily be rolled back automatically. In general, you can implement the task within a TransactionScope block and safely and effectively use any objects that implement the ITransaction interface. If you do so, all of these objects will roll back or commit accordingly. And each of these objects will know how to undo its changes.
The bottom line is that monitoring the progress of a remote task from the client is relatively easy and doesn’t have significant side effects. The PMF adds some good abstraction on top of it and provides some readymade programming tools. Making a task interruptible poses additional issues especially when the task has an inherent transactional semantics. Writing the code that just notifies the task that the user requested it to terminate is the simplest part of the game. The real complexity lies in the task implementation and its compensation policy.

Generating the Bar
To top off the article, let’s see how you can use JavaScript to easily generate progress bar markup and make it more maintainable. The progress bar you see in Figure 7 is obtained by composing HTML tables, as shown here:
<table width=”100%”>
  <tr>
    <td>69% done</td>
  </tr>
  <tr>
    <td bgcolor=”blue” width=”69%”>&nbsp;</td>
    <td width=”31%”></td>
  </tr>
</table>
The table contains two rows—for the companion text and the gauge. The gauge is rendered using a two-cell row where cells are given background colors and proportional widths.
If you take a close look at the preceding markup, you can identify at least three parameters—the message to the user, the value to represent, and the colors to use for the "done" and "to-do" regions. Instead of generating the markup as a string, wouldn’t it be neater to create a JavaScript class? The code in Figure 8 shows the primary method of the Samples.GaugeBar class.
Figure 8 Samples$GaugeBar$generateMarkup
function Samples$GaugeBar$generateMarkup(text, perc) {
    var builder = new Sys.StringBuilder(“”);
    builder.append(“<table width=’100%’><tr><td colspan=’2’>&nbsp;”);
    builder.append(text);
    builder.append(“</td></tr><tr><td bgcolor=”);
    builder.append(this._doneBackColor);
    builder.append(“ width=’”);
    builder.append(perc + “%’>”);
    builder.append(“&nbsp;</td><td bgcolor=”);
    builder.append(this._todoBackColor);
    builder.append(“ width=’”);
    builder.append(100-perc + “%’>”);
    builder.append(“</td></tr></table>”);

    return builder.toString();
}
The method takes text and a percentage and returns an HTML table with two rows. The top row just displays the text; the bottom row is articulated in two cells of different colors.
The markup string is built using the JavaScript version of the Microsoft .NET Framework StringBuilder object. Defined in the Sys namespace, the JavaScript StringBuilder object has a programming interface similar to its .NET Framework counterpart. You send text to StringBuilder’s internal buffer, and then output the text using the toString method.
The Samples.GaugeBar class has one method, generateMarkup, and properties such as background color for the "done" and "to-do" areas and the foreground color for the companion text. For performance reasons, the class operates as a singleton. The class is not very big, but still there’s no reason to create a new instance of it every time you need to update the bar. As a result, you define a static instance of the class and add a few static methods and properties:
Samples.GaugeBar.registerClass(‘Samples.GaugeBar’);
Samples.GaugeBar._staticInstance = new Samples.GaugeBar();
Samples.GaugeBar.generateMarkup = function(text, perc) {
   return Samples.GaugeBar._staticInstance.generateMarkup(text, perc);
}
You have a number of options for customizing the gauge bar. To change colors, do the following:
Samples.GaugeBar.set_DoneBackColor(“#ff00ee”);
Samples.GaugeBar.set_TodoBackColor(“#ffccee”);
Likewise, you can add a nice 3D effect by defining an outset border style on the "done" cell of the table, like so:
if (this._effect3D)
    builder.append(“ style=’border:outset white 2px;’”);
Creating a class to expose functionalities makes JavaScript programming significantly more manageable. If you ever had to deal with Dynamic HTML behaviors a few years ago, you know what I mean. The Microsoft client AJAX library is a big step forward as it enables you to write complex JavaScript code much more easily. Most AJAX professionals would probably agree that for powerful AJAX programming, richer JavaScript capabilities are a must.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or subscribe to his blog.


Extreme ASP.NET
Web Client Software Factory
Fritz Onion

Contents
What Is a Software Factory?
Composite Web Application Block
Building Multi-Module Sites
Views, Presenters, and Controllers
Modular Site Maps
Conclusion

One of the first steps in building a long-lived ASP.NET project is to define standards for how to use the many features of the platform. Without such standards, it is easy for the project to degenerate into a jumble of different development techniques and styles. One developer may use Master Pages extensively in her part of the project, for example, while another favors nested user controls. The end result is a tangle of different techniques applied across the application, making it difficult to make high-level architectural decisions or changes.
Enforcing these standards for developing applications can be difficult, however, due to the lack of tools. The wizards and designers in Visual Studio® are typically focused at a smaller level of granularity, with convenient tools for creating Web pages with codebehind, Web sites with code directories, and so on. If you decide on a set of standards that are not directly supported by the normal tools within your development environment, you have to either build your own tools or create very specific instructions for developers to follow when performing certain tasks.
This is precisely the problem that the patterns & practices team at Microsoft set out to solve with the Web Client Software Factory. It created a toolset designed specifically for the task of building large Web sites with ASP.NET that are worked on by many developers concurrently.

What Is a Software Factory?
Before I delve into the details of what the Web Client Software Factory provides, it makes sense to describe what a software factory is. The patterns & practices group at Microsoft builds a different kind of product than many other Microsoft product groups, such as the ASP.NET, CLR, and SQL Server teams. It creates packages of tools, documentation, and reference implementations for product groups that are intended as guidance for how to best use a particular technology in an enterprise environment. For example, the Web Service Software Factory is designed to provide guidance and enhanced tools for building Web services using either ASMX or Windows Communication Foundation (WCF). Instead of starting with the somewhat low-level Web service features exposed by the WCF libraries, the Web Service Software Factory provides guidance and tool support for creating message-based systems that follow commonly used patterns of design and are built for longevity, interoperability, and maintainability. Similarly, there are software factories for smart clients, mobile clients, and Web clients—the focus of this column. (For more information on the Web Service Software Factory, see the December 2006 Service Station column.
While a software factory may sound like a single component you could incorporate into a project, it actually describes an entire collection of assets you can use in developing applications with a specific technology. Each of the software factories available contains:
  • Application blocks and libraries—actual pieces of software that build on top of the target technology, with the intent of providing a prebuilt infrastructure that follows recommended best practices.
  • Guidance package recipes and templates—typically plug-ins for Visual Studio that generate initial solution templates and/or augment an existing project with features built using recommended best practices. These plug-ins rely upon the application blocks and libraries for much of their implementation.
  • Designers—additional designer support for Visual Studio to work with pieces of the application.
  • Reference implementation—a complete execution of a somewhat real-world application using all of the other assets of the software factory.
  • Architecture guidance and patterns—comprehensive documentation on the patterns and best practice choices used by the software factory.
  • How-to topics—detailed step-by-step directions for using parts of the software factory (or even just directions for how to use a particular technology in the way recommended by the software factory).
The Web Client Software Factory in particular introduces two application blocks: the Composite Web Application Block and the Page Flow Application Block. The goal behind the Composite Web Application Block is to create a framework for building modular Web applications with ASP.NET, and it addresses many of the architectural concerns outlined in the introduction. The Page Flow Application Block provides a framework for incorporating Windows® Workflow Foundation into your Web applications to manage the page navigation in a reliable and enforceable way. Each of these application blocks comes with a collection of recipes and templates that extend Visual Studio, a full set of how-to documents describing specific tasks along with descriptions of the patterns involved with each, and a complete reference implementation of an e-banking site that uses both application blocks to their full extent.
For the remainder of this column, I will focus on the composite application block, both to narrow the scope of the topic and also because I think developers will find it the most immediately compelling.

Composite Web Application Block
Once you have installed the Web Client Software Factory (installation binaries and complete instructions for setup can be found at codeplex.com/websf, you will find a new project template in Visual Studio 2005 called Web Client Solution, listed under the Guidance Packages project types as shown in Figure 1.
Figure 1 Web Client Solution Project Template
Figure 1 Web Client Solution Project Template
The first thing you will notice about the solution that is created with the Web Client Development Guidance Package project, shown in Figure 2, is that it is split into two distinct projects: a library project called Shell listed under a directory called Modules, and a Web site called DevelopmentWebsite listed under a directory called WebSites. These are actually just the first two of several projects you will add to your application with the goal of creating multiple independent modules that all work together to define a single Web site. The general layout of this solution is to create a separate module (class library project) for each section of the site. Each module will be independently compilable and will have a separate directory of pages and codebehind files in the main Web site directory. The goals of this project layout are to make it feasible for multiple developers to work independently on different portions of the site without interfering with each other and to be able to deploy modules independently by pushing out a new binary and a fresh directory of pages to the deployment server without redeploying all modules together.
Figure 2 Project Layout
Figure 2 Project Layout
In order to prepare for multiple modules to be incorporated into a single site, the Web Client Solution project template implicitly makes several decisions about the structure of the Web site. First, it defines a single top-level Master Page under the /Shared subdirectory called Default.master. As you’ll see later, every page created on behalf of a module will automatically be linked to this shared Master Page using the standard MasterPageFile property of the @Page directive. This Master Page by default contains a tree view control tied to a SiteMapDataSource to display the navigable pages on the site as well as a SiteMapPath control to display the current location with breadcrumb links back up the tree. This SiteMapDataSource is not tied to the standard Web.sitemap file, but rather to a custom SiteMapProvider that collects navigation information from each of the modules independently.
The initial application template also creates a default theme and implicitly associates each page in the site with that theme using the global styleSheetTheme attribute of the <pages> element in the top-level web.config file.
There are several other pieces of infrastructure put in place by the project template, which you will notice as you poke around the generated files. Most of these are in anticipation of managing multiple modules or have to do with the authorization features of this application block. For example, you will notice in web.config there is an HttpModule registered called WebClientAuthorizationModule to perform custom client authorization. There are also several custom configuration section handlers registered including compositeWeb/authorization and compositeWeb/modules for specifying settings for module authorization and managing multiple modules, which I’ll turn to next.

Building Multi-Module Sites
One of the core benefits of working with the Composite Web Application block is the ability to split a site into multiple independent modules. As mentioned before, each module is responsible for a subdirectory in the main site and all of its content. Typically this is combined with the View-Presenter pattern that I’ll discuss next, to isolate all of the page logic in a separate module assembly and remove it completely from the page or even its codebehind class.
There is a recipe for adding a module to a Web Client Solution that you can access by right-clicking on the Modules directory within the Solution Explorer and selecting Web Client Factory | Add Business Module. Figure 3 shows the elements that will be added using this template with a new business module named Customers.
Figure 3 Add a Module
Figure 3 Add a Module
This recipe creates a new top-level subdirectory in your Web site with the same name as the business module, as well as a new class library project under the Modules directory. The class library project is intended to contain any business logic associated with the set of pages in the /Customers directory. Initially it contains a definition for a presenter class (DefaultViewPresenter) and an interface (IDefaultView), defining methods for the View to be implemented by the Default.aspx page in the /Customers directory. A top-level Controller class is also defined in the project that combined with the view and presenter classes will complete the page logic for this module.
The last piece of interest is the CustomerModuleInitializer.cs file, which contains a CustomersModuleInitializer class that inherits from ModuleInitializer. This class defines initialization methods that will be called as the Web site first loads, and it gives the module an opportunity to initialize any data, register site map information, and so on. The module is hooked into the Web site’s loading through the web.config file that was added to the /Customers subdirectory, which contains an entry under compositeWeb/modules describing the name, assembly, and URL of this module:
<compositeWeb>
  <modules>
    <module name=”Customers” assemblyName=”Customers”
            virtualPath=”~/Customers”>
      <dependencies>
        <dependency module=”Shell” />
      </dependencies>
    </module>
  </modules>
</compositeWeb>
There is also a recipe for adding a new foundational module to your application (rather than a business module), which creates a new class library project complete with a ModuleInitializer class but without any explicit correspondence to a directory or page within the Web site. This can be useful for supplemental business logic that does not tie directly to any particular set of pages in the site.

Views, Presenters, and Controllers
One of the difficulties in trying to separate an ASP.NET Web site into multiple modules is that, by default, most of the programmatic logic associated with a page resides in the page’s codebehind file. It is generally a futile effort to try and separate codebehind pages into a separate assembly as the tools don’t support it and it makes working with the site rather cumbersome.
In order to truly create independent modules of functionality associated with pages in a site, all of the page logic, event-handling logic, and navigation logic needs to be somehow extracted from the page and hosted in a separate assembly. The default solution in the Composite Application Block is to use a View-Presenter pattern to separate the page logic into a distinct class (the Presenter) that responds to any events forwarded by the View (the Web page). The Presenter class is implemented entirely in the business module, keeping the application logic out of the Web site, and an interface is defined in the business module that defines the methods implemented by the View. This way, the Web page ends up forwarding all events to the Presenter and acts as more of a shell without any real application-specific responsibilities. It also makes it much easier to design tests for the Presenter without actually involving the front-end Web pages. (Using View-Presenter does add the complexity of having extra classes and files, so if you are not looking for this separation, you might choose to forgo the pattern altogether and write ASPX classes the usual way).
As mentioned earlier, the initial Default.aspx page generated by the Add Business Module recipe contained a corresponding DefaultViewPresenter class and an IDefaultView interface for implementing the View-Presenter pattern. To add a new .aspx page to your module with a corresponding View and Presenter, there is another recipe you can access by right-clicking on the subdirectory created for your module and selecting Web Client Factory | Add View (with presenter). If you were to create a new form for inserting customers, for example, you might use this recipe to create a new AddCustomer.aspx page as shown in Figure 4.
Figure 4 Adding a View with Presenter
Figure 4 Adding a View with Presenter
The recipe will add a new Web page to the /Customers directory called AddCustomer.aspx, with a corresponding codebehind file. To the Customers business module project, it adds a new class called AddCustomerPresenter and a corresponding IAddCustomer interface, which is already implemented by the codebehind class in your AddCustomer.aspx.cs file. The codebehind class also contains a property declaration for the corresponding AddCustomerPresenter class to which the page should forward events. For example, if your new AddCustomer.aspx page had a button for inserting a new customer, the handler of that button might look something like:
protected void _insertCustomerButton_Click(
      object sender, EventArgs e)
{
  Presenter.InsertCustomer(
    new Customer(_firstName.Text, _lastName.Text));
}
The logic for inserting the customer, and what to do next, would be defined in the InsertCustomer method of your AddCustomerPresenter class, leaving the page itself devoid of any real business logic. The interface implemented by the page (IAddCustomer in this case) will typically contain properties defining data sources used by the page and other attributes that might influence the appearance of the page itself. The presenter is defined with a property bound to the page’s implementation of the interface so that it can set the data properties as appropriate when the page is navigated to.
The final piece in the picture is the controller class that implements the Application Controller pattern. When you first create your business module, the recipe will generate a controller class (CustomersController in this case) to which you add methods for the presenter classes to call in response to events of pages in the site. The controller is then responsible for controlling navigation among the pages. For example, once a new customer is inserted through the AddCustomer.aspx page, you may want to navigate back to the Default.aspx page in the /Customers directory. The controller would define a method to be called once a customer was inserted, which would then redirect to the Default.aspx page. Each presenter class that is added to your site has a local reference to the controller class of the module so that it can invoke navigation methods when needed.
The recipes and classes provided by the software factory take care of many of the details of implementing these two patterns for you. You don’t need to worry about creating the corresponding Presenter class from your View or allocating a controller when needed in your Presenter. Your work is entirely devoted to building the application logic and interface for your business module. The samples, how-tos, and reference implementation included with the Web Client Software Factory are replete with examples of site building that use View-Presenter and Application Controller patterns, which I encourage you to inspect in detail if this model sounds useful in your application development.

Modular Site Maps
Another detail that is taken care of nicely when you use this model of Web site development is the aggregation of site map information across all modules in your site. If you have used the SiteMapDataSource before to populate a menu or tree view for your site, you’re aware that by default it relies only on the web.sitemap file located at the root of your application directory. Trying to manually keep a single XML file up to date with all of the page navigation information of your site is just asking for trouble for large multi-module sites. To deal with this, the Web Client Software Factory includes a custom SiteMapProvider class called ModuleSiteMapProvider that will explicitly ask each module for its site map information as the application starts. By default, this SiteMapProvider is registered as the default provider and will be used by the SiteMapDataSource for all navigation controls.
To populate the ModuleSiteMapProvider with your particular module’s site map information, you override the RegisterSiteMapInformation method in your ModuleInitializer-derived class. The recipe you used to create the business module will already have overridden this method and inserted the Default.aspx page into the collection of SiteMapNodes, but it is up to you to add any other pages to the collection as you add them to your site. For example, in your Customers module, you might create one top-level node for the default page in our module, and one nested page for the Add Customer page, as shown in Figure 5.
Figure 5 Adding Pages in the Customer Module
protected virtual void RegisterSiteMapInformation(
    ISiteMapBuilderService siteMapBuilderService)
{
  SiteMapNodeInfo moduleNode = new SiteMapNodeInfo(“Customers”,
    “~/Customers/Default.aspx”, “Customers”);
  siteMapBuilderService.AddNode(moduleNode);

  SiteMapNodeInfo addCustomerNode = new SiteMapNodeInfo(“Add customer”,
    “~/Customers/AddCustomer.aspx”, “Add customer”,
    “Add customer page”);

  siteMapBuilderService.AddNode(addCustomerNode, moduleNode, 0);
}

Conclusion
The Web Client Software Factory offers a compelling solution for developers trying to manage large Web sites built with ASP.NET. With support for the creation of modules associated with individual subdirectories in the site, and for patterns such as View-Presenter and Application Controller, this software factory provides a framework that enables independent development, testing, and deployment of a Web site in a modular fashion. I have barely scratched the surface of features offered by this software factory, but I hope it’s been enough to give you a taste of the possibilities and compel you to explore the remainder of the factory. Finally, the patterns & practices team is starting work on the release of the second version of the Web Client Software Factory; you can get involved at codeplex.com/websf.

Send your questions and comments for Fritz to xtrmasp@microsoft.com.

Fritz Onion is a cofounder of Pluralsight, a Microsoft .NET training provider, where he heads the Web development curriculum. Fritz is the author of Essential ASP.NET (Addison Wesley, 2003) and Essential ASP.NET 2.0 (Addison Wesley, 2006). You can reach him at pluralsight.com/fritz.


Foundations
Declarative WCF Security
Juval Lowy

Code download available at: Foundations2007_08.exe (231 KB)
Browse the Code Online
Contents
Security Scenarios
Intranet Application
Internet Application
Business-to-Business Application
Anonymous Application
No Security
The Declarative Security Framework
Implementing SecurityBehaviorAttribute
Client-Side Declarative Security
SecurityHelper and SecureClientBase<T>
What It All Means

Security is by far the most intricate area of the Windows® Communication Foundation (WCF). In every WCF operation call on the service side, security is governed by the service contract, the operation contract, the fault contract (if present), the service behavior, the operation behavior, the host configuration, and the method configuration and its code. Each of these items may have as many as a dozen or more security-related properties, as illustrated in Figure 1, depicting the security properties of ServiceHostBase (the base of ServiceHost).
Figure 1 Security-Related Properties of ServiceHostBase
Figure 1 Security-Related Properties of ServiceHostBase
Not only are there a daunting number of details to master, there are also complicated relationships between the various parts, producing an overwhelming number of possible permutations. Further complicating matters, not all combinations are allowed or supported, and not all of the allowed combinations are consistent or make sense.
It’s not surprising that the programming model is very complex and there are severe implications, both at the application level and at the business level, for getting things wrong. This is the result of the underlying complexity of the objective—enabling secure communication across vendor, trust, platform, and technology boundaries. It’s not a trivial task by any stretch. But unlike the other aspects of WCF, such as transactions, synchronization, and instancing, security doesn’t have attributes or a single one-stop way to do configuration. Out of the box, developers simply have to master the raw security model.
There is good reason for not including a high-level programming model for security: any such model could not possibly address all applications and scenarios, yet it would have been unwise to leave out any applications as security is important in all scenarios.
Recognizing that it’s not feasible to come up with a unified high- level security model optimized for all applications, I asked myself this question: is there a model that is good enough for most applications? Although a thorough framework that addresses all scenarios is unlikely, I wanted to create a solution that would make it easy to configure security settings for the majority of cases. In this column, I present my declarative security framework.
For the service, I’ve provided a security attribute and for the client, a few helper classes and secure proxy classes. My declarative framework makes security configuration on par with the other aspects of WCF configuration. I wanted a declarative model that would be simple to use and would minimize the need to understand the many details of security. As a developer, all you need to do is select the correct scenario and my framework will automate the configuration. In addition, my framework mandates correct options and enforces best practices. At the same time, I wanted the model to provide granularity and control of the underlying configuration in case the need for such control arises.
In this article, I focus on the target scenarios, how my framework was implemented, and how to use it. I also present some interesting WCF programming techniques that demonstrate WCF extensibility. (You can download the security framework from the MSDN® Magazine Web site.)

Security Scenarios
My framework supports five key scenarios, along with some slight variations, that address the security needs of the majority of applications today. These scenarios are:
  • Intranet application
  • Internet application
  • Business-to-business application
  • Anonymous application
  • No security
If you need to address a different scenario, you can follow my approach to derive the required security aspects and settings using the accompanying source code.

Intranet Application
In the intranet application, both the client and service use WCF and both are deployed on the same intranet. No firewall separates the client and service, and you can use Windows-based security for transfer security and authentication. You can rely on Windows accounts and groups to store the client’s credentials and you can use principal-based, role-based security verifying membership against Windows groups.
For this scenario, my framework allows the use of the intranet bindings—namely, NetTcpBinding, NetNamedPipeBinding, and NetMsmqBinding. The bindings that are used are configured for the highest protection level (encrypted and signed). You can rely on Transport mode for Transfer security because the calls are invariably point-to-point. In addition, you can control the impersonation level and even request automatic impersonation for all operations.

Internet Application
In the Internet scenario, the client or service may not use WCF or even Windows. In addition, an Internet application typically has a relatively large number of clients calling the service. These client calls originate from outside the firewall. You need to rely on HTTP for transport and multiple intermediaries are likely.
In an Internet application, you typically do not want to use Windows accounts and groups for credentials. Instead, the application will need to access a custom credentials store. (That said, you could still use Windows security, as I’ll demonstrate later on.) Here, my framework uses message security for transfer security, providing security across all intermediaries. The client provides credentials in the form of username and password.
For this scenario, you should use WSHttpBinding or WSDualHttpBinding. In addition, if you have an intranet application that uses NetTcpBinding but do not wish to use Windows security for user accounts and groups, you can use message security and custom username and password in the intranet.
Since in the Internet scenario the client’s message that is sent to the service is transferred over plain HTTP, it is critical to protect the message’s content (both the client’s credentials and the body of the message) by encrypting it using a certificate provided by the service. WCF supports several options for obtaining that certificate. For instance, the client developer can obtain the service certificate using any out-of-band mechanism, such as an e-mail message or via a public Web page.
Using the WCF configuration schema, the client can include in its config file (specifically, in the endpoint behavior section) detailed information about the service certificate, such as where it’s stored on the client-side and how to find it. This is by far the most secure option from the client’s perspective because any attempt to subvert the client’s address resolution and redirect the call to a malicious service will fail since the other service will not have the correct certificate. This, however, is also the least flexible option since every time the client needs to interact with a different service, the client administrator will need to rework the client’s config file.
A reasonable alternative to explicitly referencing the certificates of all services the client may interact with is to store those certificates in the client’s Trusted People certificate folder and instruct WCF to allow calls only to a service whose certificate is in that folder. The client in that case will need to obtain the service certificate at run time as part of the initial pre-call negotiation, check to see if it is in the Trusted People store, and, if so, proceed to use it to protect the message. This form of certificate negotiation is the default behavior of the WS bindings. For the Internet scenario, I use certificate negotiation coupled with validating the service certificate in the client’s Trusted People store. This is called peer-trust validation.
Once the client’s user name and password credentials are received by the service, the host authenticates them. By default, the credentials are authenticated as Windows credentials (against the host domain or machine), but you can configure the service to use the ASP.NET membership providers.
The service can use principal-based, role-based security: if the credentials are Windows credentials, then the roles specified have to be Windows groups. When the ASP.NET providers are used, then the roles are verified using ASP.NET roles providers.

Business-to-Business Application
In the business-to-business scenario, the service and its clients are disparate business entities. They do not share credentials or accounts and the communication between them is typically closed to the public. There are relatively few clients interacting with the service and the client can only interact with the service after an elaborate business agreement and other conditions have been met.
Rather than using Windows accounts or usernames, the clients identify themselves to the service using X509 certificates, which typically are already known to the service. The client or service may not use WCF or even Windows. Client calls originate from outside the firewall, you need to rely on HTTP for transport, and multiple intermediaries are possible.
For the business-to-business scenario, my framework allows the use of the BasicHttpBinding, WSHttpBinding, or WSDualHttpBinding. You must use message security for transfer security to provide security across all intermediaries. The message will be protected using a service-side certificate, just as in the Internet scenario. However, unlike in the Internet scenario, the clients in this scenario provide credentials in the form of a certificate. The client supplies the certificate in the config file (or programmatically) to the proxy, and the proxy bundles the certificate in the message and sends it to the service.
WCF offers the service administrator a number of options for validating the certificates sent by the client. If the certificate is validated, the client is considered authenticated. My framework uses peer-trust validation on the service side so the service administrator should install all the certificates of the clients allowed to interact with the service in the Trusted People store on the service’s local machine.
When the client’s certificate is received by the service, if the certificate is found in the trusted store, the client is authenticated. By default, the service cannot employ principal-based, role-based security. This is because the credentials provided, namely the client’s certificate, do not map to either Windows or ASP.NET user accounts. Since business-to-business endpoints are often dedicated to a small set of clients or even to a particular client, this lack of authorization support may not pose a problem. If, on the other hand, you would still like to authorize the clients, my framework takes advantage of the ASP.NET roles provider for authorization, even though the membership provider was not used for authentication. This ability to use the providers separately was a core design goal for the ASP.NET provider model in WCF.

Anonymous Application
In the anonymous scenario, the clients access the service without presenting any credentials—they are unidentified. On the other hand, the clients and the service do require secure message transfer that is protected against tampering and sniffing. Both an Internet-facing and intranet-based application may need to provide for anonymous yet secure access.
The anonymous scenario can have any number of clients. The clients may connect over HTTP or over TCP. The need to secure the message and the fact that the clients may be calling over the Internet with multiple intermediaries means you should use message security. For this scenario, you can use the NetTcpBinding, WSHttpBinding, WSDualHttpBinding, and NetMsmqBinding—a mixture of both Internet and intranet bindings. Note that you cannot use the BasicHttpBinding, NetNamedPipeBinding, NetPeerTcpBinding or WSFederationHttpBinding, as those bindings either do not support message security or do not support having no credentials in the message.
Note again that no client authentication is done in the anonymous scenario and the client need not provide credentials to the proxy. For service authentication towards the client and for message protection, the service needs to provide its certificate. Since the clients are anonymous (and unauthenticated) authorization and role-based security are precluded.

No Security
In this last scenario, your application turns off security completely. The service does not rely on any transfer security and it does not authenticate or authorize its callers. The service can accept any number of clients and clients need not provide any credentials to the proxy. Since the clients are anonymous (and unauthenticated), authorization and role-based security are precluded. Both Internet and intranet services can be configured for no security, but it goes without saying that such a service is completely exposed and you generally need a business justification for relinquishing security.
Figure 2 and Figure 3 serve as a summary of the key elements of the security scenarios. Figure 2 lists the bindings used in each scenario and Figure 3 shows how each of the security aspects relates to each scenario.
Figure 3 Security Scenarios Aspects

Aspect | Intranet | Internet | Business-to-Business | Anonymous | No Security
Transport | Yes | No | No | No | No
Message | No | Yes | Yes | Yes | No
Service Authentication | Windows | Certificate | Certificate | Certificate | No
Client Authentication | Windows | ASP.NET | Certificate | No | No
Authorization | Windows | ASP.NET | No/ASP.NET | No | No
Impersonation | Yes | No | No | No | No
Figure 2 Security Scenarios and Bindings

Binding | Intranet | Internet | Business-to-Business | Anonymous | No Security
BasicHttpBinding | No | No | Yes | No | Yes
NetTcpBinding | Yes | Yes | No | Yes | Yes
NetPeerTcpBinding | No | No | No | No | Yes
NetNamedPipeBinding | Yes | No | No | No | Yes
WSHttpBinding | No | Yes | Yes | Yes | Yes
WSDualHttpBinding | No | Yes | Yes | Yes | Yes
NetMsmqBinding | Yes | No | No | Yes | Yes

The Declarative Security Framework
Figure 4 lists the definition of SecurityBehaviorAttribute and the ServiceSecurity enum. ServiceSecurity defines the five scenarios supported by my framework.
Figure 4 SecurityBehaviorAttribute
public enum ServiceSecurity
{
   None,
   Anonymous,
   BusinessToBusiness,
   Internet,
   Intranet
}

[AttributeUsage(AttributeTargets.Class)]
public class SecurityBehaviorAttribute : Attribute,IServiceBehavior
{
   public SecurityBehaviorAttribute(ServiceSecurity mode);
   public SecurityBehaviorAttribute(ServiceSecurity mode,
      string serviceCertificateName);
   public SecurityBehaviorAttribute(ServiceSecurity mode,
      StoreLocation storeLocation,
      StoreName storeName,
      X509FindType findType,
      string serviceCertificateName);
   public bool ImpersonateAll {get;set;}
   public string ApplicationName {get;set;}
   public bool UseAspNetProviders {get;set;}
}
When applying SecurityBehaviorAttribute, you need to provide the target scenario in the form of a ServiceSecurity value. You can use just the constructors of SecurityBehaviorAttribute or you can set the properties. Unset, the properties default to reasonable values in the context of the target scenario. When selecting a scenario, the configured behavior follows to the letter my previous description of the individual scenarios.
SecurityBehaviorAttribute yields a composable security model, allowing a few permutations and sub-scenarios. When using the attribute, you can even have a security-free host config file or you can combine settings from the config file with values driven by the attribute. Much the same way, your hosting code can be free of security or you can combine programmatic host security with the attribute.
To configure a service for the intranet security scenario, apply SecurityBehavior with ServiceSecurity.Intranet, like this:
[ServiceContract]
interface IMyContract
{
   [OperationContract]
   void MyMethod();
}
[SecurityBehavior(ServiceSecurity.Intranet)]
class MyService : IMyContract
{
   public void MyMethod() {...}
}
Even though the service contract used may not constrain the protection level, the attribute programmatically adds that demand to enforce message protection. You can use Windows NT groups for role-based security, like so:
[SecurityBehavior(ServiceSecurity.Intranet)]
class MyService : IMyContract
{
   [PrincipalPermission(SecurityAction.Demand,
        Role = @”<Domain>\Customer”)]
   public void MyMethod() {...}
}
The service can programmatically impersonate the callers or use the operation behavior attribute for individual methods impersonation. You can also configure the service to automatically impersonate all callers in all methods via the ImpersonateAll property. ImpersonateAll defaults to false, but when set to true the attribute impersonates all callers in all operations without the need to apply any operation behavior attribute:
[SecurityBehavior(ServiceSecurity.Intranet,ImpersonateAll = true)]
class MyService : IMyContract {...}
With the Internet scenario, you need both to configure with ServiceSecurity.Internet and to select the service certificate to use. Note in Figure 4 that the ServiceBehavior attribute constructor may take the service certificate name. Unspecified, the service certificate is loaded from the host config file as with raw WCF security:
[SecurityBehavior(ServiceSecurity.Internet)]
class MyService : IMyContract {...}
You can also specify the service certificate name, in which case the specified certificate is loaded from the LocalMachine store from the My folder by name:
[SecurityBehavior(ServiceSecurity.Internet,”MyServiceCert”)]
class MyService : IMyContract {...}
If the certificate name is set to an empty string, the attribute will infer the certificate name by using the hosting machine name (or domain) for the certificate name and load the certificate from the LocalMachine store from the My folder by name:
[SecurityBehavior(ServiceSecurity.Internet,””)]
class MyService : IMyContract {...}
Finally, the attribute lets you explicitly specify the store location, the store name, and the lookup method:
[SecurityBehavior(ServiceSecurity.Internet,
    StoreLocation.LocalMachine,StoreName.My,
    X509FindType.FindBySubjectName,”MyServiceCert”)]
class MyService : IMyContract {...}
Note that you can combine an explicit location with an inferred certificate name, like this:
[SecurityBehavior(ServiceSecurity.Internet,
    StoreLocation.LocalMachine,StoreName.My,
    X509FindType.FindBySubjectName,””)]
class MyService : IMyContract {...}
Which credentials store to authenticate the client against is indicated by the UseAspNetProviders property. This property defaults to false, meaning the default is to authenticate the client’s username and password as Windows credentials:
[SecurityBehavior(ServiceSecurity.Internet,”MyServiceCert”)]
class MyService : IMyContract
{
   [PrincipalPermission(
      SecurityAction.Demand,Role = @”<Domain>\Customer”)]
   public void MyMethod() {...}
}
You can even impersonate all callers:
[SecurityBehavior(
   ServiceSecurity.Internet,”MyServiceCert”,ImpersonateAll = true)]
class MyService : IMyContract {...}
If UseAspNetProviders is set to true, the attribute will use the ASP.NET membership and role providers as prescribed for the Internet scenario:
[SecurityBehavior(
   ServiceSecurity.Internet,”MyServiceCert”,UseAspNetProviders = true)]
class MyService : IMyContract
{
   [PrincipalPermission(SecurityAction.Demand,Role = “Manager”)]
   public void MyMethod() {...}
}
And the attribute will programmatically enable the role manager section in the config file so there is no need for anything besides endpoints in the host config file.
Next is the issue of supplying the application name for the ASP.NET providers using the ApplicationName property. When unassigned, the attribute looks up the application name from the config file as with raw WCF. If no value is found in the host config file, the attribute will not default to using the meaningless / from machine.config. Instead, it will default to the host assembly name. If the ApplicationName property is assigned a value, it will override whatever application name is present in the host config files:
[SecurityBehavior(ServiceSecurity.Internet,”MyServiceCert”,
  UseAspNetProviders = true,ApplicationName = “MyApplication”)]
class MyService : IMyContract {...}
Configuring for the business-to-business scenario requires setting ServiceSecurity to ServiceSecurity.BusinessToBusiness, just as with ServiceSecurity.Internet. For example:
[SecurityBehavior(ServiceSecurity.BusinessToBusiness)]
class MyService : IMyContract {...}

[SecurityBehavior(ServiceSecurity.BusinessToBusiness,””)]
class MyService : IMyContract {...}

[SecurityBehavior(ServiceSecurity.BusinessToBusiness,”MyServiceCert”)]
class MyService : IMyContract {...}
By default, when using ServiceSecurity.BusinessToBusiness, the service cannot authorize its callers. However, setting the UseAspNetProviders property to true enables you to employ the ASP.NET role providers:
[SecurityBehavior(ServiceSecurity.BusinessToBusiness,
   UseAspNetProviders = true)]
class MyService : IMyContract {...}
When using the ASP.NET role providers, the application name will be looked up and decided upon just as it would be with the ServiceSecurity.Internet setting:
[SecurityBehavior(ServiceSecurity.BusinessToBusiness,”MyServiceCert”,
   UseAspNetProviders = true,ApplicationName = “MyApplication”)]
class MyService : IMyContract {...}
To allow callers according to the anonymous scenario, you need to configure the attribute with ServiceSecurity.Anonymous. Configuring the service certificate is done just as with the ServiceSecurity.Internet:
[SecurityBehavior(ServiceSecurity.Anonymous)]
class MyService : IMyContract {...}
To turn off security completely, as in the no security scenario, just provide the attribute with ServiceSecurity.None:
[SecurityBehavior(ServiceSecurity.None)]
class MyService : IMyContract {...}

Implementing SecurityBehaviorAttribute
Figure 5 is a partial listing of the implementation of SecurityBehaviorAttribute. The three public properties of the attribute correspond to three private members where the attribute saves the configured values.
Figure 5 Implementing SecurityBehaviorAttribute
[AttributeUsage(AttributeTargets.Class)]
class SecurityBehaviorAttribute : Attribute,IServiceBehavior
{
   SecurityBehavior m_SecurityBehavior;
   string m_ApplicationName = String.Empty;
   bool m_UseAspNetProviders;
   bool m_ImpersonateAll;

   public SecurityBehaviorAttribute(ServiceSecurity mode)
   {
      m_SecurityBehavior = new SecurityBehavior(mode);
   }

   public SecurityBehaviorAttribute(ServiceSecurity mode,
      string serviceCertificateName)
   {
      m_SecurityBehavior =
         new SecurityBehavior(mode,serviceCertificateName);
   }

   public bool ImpersonateAll {get;set;} // m_ImpersonateAll

   public string ApplicationName {get;set;} // m_ApplicationName

   public bool UseAspNetProviders {get;set;} // m_UseAspNetProviders

   void IServiceBehavior.AddBindingParameters(
      ServiceDescription description,
      ServiceHostBase serviceHostBase,
      Collection<ServiceEndpoint> endpoints,
      BindingParameterCollection parameters)
   {
      m_SecurityBehavior.AddBindingParameters(
         description,serviceHostBase, endpoints,parameters);
   }
   void IServiceBehavior.Validate(ServiceDescription description,
      ServiceHostBase serviceHostBase)
   {
      m_SecurityBehavior.UseAspNetProviders = UseAspNetProviders;
      m_SecurityBehavior.ApplicationName = ApplicationName;
      m_SecurityBehavior.ImpersonateAll = ImpersonateAll;
      m_SecurityBehavior.Validate(description,serviceHostBase);
   }

   ... //Rest of the implementation
}
SecurityBehaviorAttribute is a service behavior attribute so you can apply it directly on the service class. The attribute supports the WCF extension interface IServiceBehavior, defined as:
public interface IServiceBehavior
{
   void AddBindingParameters(ServiceDescription serviceDescription,
      ServiceHostBase serviceHostBase,
      Collection<ServiceEndpoint> endpoints,
      BindingParameterCollection bindingParameters);

   void Validate(ServiceDescription serviceDescription,
      ServiceHostBase serviceHostBase);

   void ApplyDispatchBehavior(...);
}
IServiceBehavior is a special interface. If an attribute on the service class supports it, when the host is launched and later when the service is instantiated, WCF calls the various methods of IServiceBehavior, enabling the service to hook and extend WCF and affect such things as configuration of the binding, the dispatcher, and the host.
When the AddBindingParameters method of IServiceBehavior is called, SecurityBehaviorAttribute enforces the binding configuration that matches the requested scenario. The Validate method of IServiceBehavior is where SecurityBehaviorAttribute configures the host. The actual configuration is accomplished using a helper class called SecurityBehavior. In general, I recommend separating the actual code of an extension from the attribute so you can use the code in other places. For example, the code accompanying this article also contains a host class that can apply declarative security on the hosted service class, as if that class used SecurityBehaviorAttribute. In a similar way, the client-side part of my framework uses SecurityBehavior, as well. SecurityBehaviorAttribute constructs an instance of SecurityBehavior, providing it with the scenario (the mode parameter), as well as the certificate name in the matching constructor. SecurityBehavior provides systematic, meticulous, programmatic setting of all security scenarios. By doing so, it frees the config file and the hosting code from any shred of security.
SecurityBehavior is a service behavior in its own right and is designed to be used independently of the attribute. Figure 6 contains a partial listing of SecurityBehavior.
Figure 6 Implementing SecurityBehavior
class SecurityBehavior : IServiceBehavior
{
   ServiceSecurity m_Mode;
   StoreLocation m_StoreLocation;
   StoreName m_StoreName;
   X509FindType m_FindType;
   string m_SubjectName;
   bool m_UseAspNetProviders;
   string m_ApplicationName = String.Empty;
   bool m_ImpersonateAll;

   public SecurityBehavior(ServiceSecurity mode) :
      this(mode,StoreLocation.LocalMachine,StoreName.My,
         X509FindType.FindBySubjectName,null) {}

   public SecurityBehavior(ServiceSecurity mode,
      StoreLocation storeLocation,StoreName storeName,
      X509FindType findType,string subjectName)
        {...} //Sets the corresponding members

   public bool ImpersonateAll {get;set;} // m_ImpersonateAll

   public bool UseAspNetProviders {get;set;} //m_UseAspNetProviders

   public string ApplicationName {get;set;} // m_ApplicationName

   public void Validate(ServiceDescription description,
     ServiceHostBase serviceHostBase)
   {
      if(m_SubjectName != null)
      {
         switch(m_Mode)
         {
            case ServiceSecurity.Anonymous:
            case ServiceSecurity.BusinessToBusiness:
            case ServiceSecurity.Internet:
                 string subjectName = m_SubjectName != String.Empty ?
                    m_SubjectName :
                    description.Endpoints[0].Address.Uri.Host;
               serviceHostBase.Credentials.ServiceCertificate.
                  SetCertificate(m_StoreLocation,m_StoreName,
                    m_FindType,subjectName);
               break;
         }
      }
      ...
   }

   public void AddBindingParameters(ServiceDescription description,
      ServiceHostBase serviceHostBase,
      Collection<ServiceEndpoint> endpoints,
      BindingParameterCollection parameters)
   {
      ...
      switch(m_Mode)
      {
         case ServiceSecurity.Intranet:
            ConfigureIntranet(endpoints);
            break;

         case ServiceSecurity.Internet:
            ConfigureInternet(endpoints,UseAspNetProviders);
            break;
         ...
      }
   }

   internal static void ConfigureInternet(
      Collection<ServiceEndpoint> endpoints)
   {
      foreach(ServiceEndpoint endpoint in endpoints)
      {
         Binding binding = endpoint.Binding;
         if(binding is WSHttpBinding)
         {
            WSHttpBinding wsBinding = (WSHttpBinding)binding;
            wsBinding.Security.Mode = SecurityMode.Message;
            wsBinding.Security.Message.ClientCredentialType =
               MessageCredentialType.UserName;
            continue;
         }
         ...
         throw new InvalidOperationException(binding.GetType() +
            “is unsupprted with ServiceSecurity.Internet”);
      }
    }

    ... //Rest of the implementation
}
The SecurityBehavior constructors store their parameters, such as the security mode and the details of the certificate, in member variables. The Validate method is a decision tree that configures the host according to the scenario and the provided information, supporting the behavior of SecurityBehaviorAttribute described earlier.
AddBindingParameters calls a dedicated helper method for each scenario to configure the host’s collection of endpoints. Each helper method (such as ConfigureInternet) iterates over that collection, and for each endpoint, verifies that the binding being used matches the scenario and then configures the binding according to the scenario.

Client-Side Declarative Security
WCF does not support applying an attribute on the proxy class. Even if it did, the client may need to provide its credentials and other settings at run time. To compensate, the first step in supporting declarative security on the client side is my SecurityHelper static helper class, defined in Figure 7.
Figure 7 SecurityHelper Helper Class
public static class SecurityHelper
{
   public static void UnsecuredProxy<T>(ClientBase<T> proxy)
      where T : class;

   public static void AnonymousProxy<T>(ClientBase<T> proxy)
      where T : class;

   public static void SecureProxy<T>(ClientBase<T> proxy,
      string userName,string password) where T : class;

   public static void SecureProxy<T>(ClientBase<T> proxy,
      string domain,string userName,string password) where T : class;

   public static void SecureProxy<T>(ClientBase<T> proxy,
      string clientCertificateName) where T : class;

   public static void SecureProxy<T>(ClientBase<T> proxy,
      StoreLocation storeLocation,StoreName storeName,
      X509FindType findType,string clientCertificateName)
      where T : class;
   ... //More members
}
SecurityHelper lets you configure a plain proxy according to the desired security scenario and behavior. You can only configure the proxy before opening it. There is no need for any security settings in the client’s config file or elsewhere in the client’s code. SecurityHelper is smart and it will select the correct security behavior based on provided parameters and the method invoked. For example, here is how to secure a proxy for the intranet scenario and provide it with the client’s Windows credentials:
MyContractClient proxy = new MyContractClient();
SecurityHelper.SecureProxy(proxy,”MyDomain”,”MyUsername”,”MyPassword”);
proxy.MyMethod();
proxy.Close();
For the Internet scenario, the client only needs to provide the username and the password (remember that the determination of whether those are Windows or ASP.NET provider credentials is a service-side decision):
MyContractClient proxy = new MyContractClient();
SecurityHelper.SecureProxy(proxy,”MyUsername”,”MyPassword”);
proxy.MyMethod();
proxy.Close();
For the business-to-business scenario, the client can specify a null or an empty string for the client certificate name if it wants to use the certificate in its config file as in raw WCF or it can list the certificate name explicitly:
MyContractClient proxy = new MyContractClient();
SecurityHelper.SecureProxy(proxy,”MyClientCert”);
proxy.MyMethod();
proxy.Close();
SecurityHelper will load the certificate from the client’s LocalMachine store from the My folder by name. The client can also specify all the information required to find and load the certificate. For an anonymous client, use the AnonymousProxy method:
MyContractClient proxy = new MyContractClient();
SecurityHelper.AnonymousProxy(proxy);
proxy.MyMethod();
proxy.Close();
And for no security, use the UnsecuredProxy method:
MyContractClient proxy = new MyContractClient();
SecurityHelper.UnsecuredProxy(proxy);
proxy.MyMethod();
proxy.Close();

SecurityHelper and SecureClientBase<T>
Internally, SecurityHelper uses SecurityBehavior to configure the proxy’s endpoint, as well as to set the credentials (see Figure 8).
Figure 8 Implementing SecurityHelper
public static class SecurityHelper
{
   public static void SecureProxy<T>(ClientBase<T> proxy,
      string userName,string password) where T : class
   {
      if(proxy.State == CommunicationState.Opened)
      {
         throw new InvalidOperationException(
            “Proxy channel is already opened”);
      }

      Collection<ServiceEndpoint> endpoints =
         new Collection<ServiceEndpoint>();
      endpoints.Add(proxy.Endpoint);

      SecurityBehavior.ConfigureInternet(endpoints);

      proxy.ClientCredentials.UserName.UserName = userName;
      proxy.ClientCredentials.UserName.Password = password;
      proxy.ClientCredentials.ServiceCertificate.Authentication.
      CertificateValidationMode = X509CertificateValidationMode.PeerTrust;
   }
   ... //Rest of the implementation
}
The advantage of using SecurityHelper is that it can operate on any proxy—even a proxy the client developer is not responsible for creating. If you are responsible for generating the proxy, you can use the SecureClientBase<T> class, defined in Figure 9.
Figure 9 SecureClientBase<T> Class
public abstract class SecureClientBase<T> : ClientBase<T> where T : class
{
   //These constructors target the default endpoint
   protected SecureClientBase();
   protected SecureClientBase(ServiceSecurity mode);
   protected SecureClientBase(string userName,string password);
   protected SecureClientBase(
      string domain,string userName,string password);
   protected SecureClientBase(string clientCertificateName);
   protected SecureClientBase(StoreLocation storeLocation,
      StoreName storeName,X509FindType findType,
      string clientCertificateName);

   ... //More constructors for other types of endpoints
}
SecureClientBase<T> derives from the conventional ClientBase<T> and adds the declarative security support. You need to derive your proxy from SecureClientBase<T> instead of ClientBase<T>, provide constructors that match your security scenario, and call the base constructors of SecureClientBase<T> with the supplied credentials and endpoint information, as shown here:
class MyContractClient : SecureClientBase<IMyContract>,IMyContract
{
   public MyContractClient(ServiceSecurity mode) : base(mode) {}

   public MyContractClient(string userName,string password) :
     base(userName,password) {}

   ... // More constructors

   public void MyMethod()
   {
      Channel.MyMethod();
   }
}
Using the derived proxy is straightforward. For example, for the Internet scenario, it would look like this:
MyContractClient proxy = new MyContractClient(“MyUsername”,
                                              ”MyPassword”);
proxy.MyMethod();
proxy.Close();
For the anonymous scenario, it would look like this:
MyContractClient proxy = new MyContractClient(ServiceSecurity.                                                        Anonymous);
proxy.MyMethod();
proxy.Close();
The implementation of SecureClientBase<T> simply uses SecurityHelper as shown in Figure 10.
Figure 10 Implementing SecureClientBase<T>
public class SecureClientBase<T> : ClientBase<T> where T : class
{
   protected SecureClientBase(ServiceSecurity mode)
   {
      switch(mode)
      {
         case ServiceSecurity.None:
            SecurityHelper.UnsecuredProxy(this);
            break;

         case ServiceSecurity.Anonymous:
            SecurityHelper.AnonymousProxy(this);
            break;

         ...
      }
   }

   protected SecureClientBase(string userName,string password)
   {
      SecurityHelper.SecureProxy(this,userName,password);
   }

   ... //More constructors
}
When not using a proxy class, you can use my SecureChannelFactory (included in the source code for this article), just like the WCF-provided channel factory. But with my SecureChannelFactory, you can utilize declarative security.

What It All Means
By enabling the widest possible set of interactions between clients and services, WCF security introduces a degree of complexity that is difficult to master. My aim with the declarative security framework is to eliminate that complexity without decreasing security or configuration flexibility for the supported scenarios. All you need to do is select the correct security scenario for your application and you can still have as much control as with raw WCF security. If you want details on how raw WCF security works, see the August 2006 Security Briefs column.
The tradeoff with a framework like mine is that it doesn’t cover all scenarios (federated security, for instance). But you can use my framework as a starting point for enabling declarative security and then customize it to your specific scenario.

Send your questions and comments to mmnet30@microsoft.com.

Juval Lowy is a software architect with IDesign providing WCF training and WCF architecture consulting. This article contains excerpts from his recent book Programming WCF Services (O’Reilly, 2007). He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at idesign.net.


Netting C++
Configuration with XML
Stanley B. Lippman

Contents
XmlReader: the Firehose
Document Object Model

This is the second column in a series that explores using C++/CLI as a pure Microsoft® .NET Framework-compliant language rather than as a transitional bridge in order to bring native C++ code into the managed environment. My application, EEK!, as described in the last column, is a simulation of how a mouse might behave in an experimental environment over some fixed length of time. If the premise of the application leaves you disquieted, you can think of it instead as the logic programming of Non Player Characters (NPCs) in an online game. The task in this column is to initialize the virtual world by reading in a world configuration file stored as XML. To do this, I will use the .NET XML namespace.
Just one note before I start. In the initial EEK! column, I indicated that I would attempt to simulate the actual fish tank environment in which I kept mice while I observed their behaviors. But reality, I discovered, offers a rather tedious constraint to a virtual world; it just didn’t make for an interesting simulation, and so I’ve removed that constraint from the application. Instead, the environment is simply a terrain limited only by one’s imagination. I need it big and complex enough to make it a challenge both for the mouse to explore and for me to program. I apologize for changing the specification, but if you’re a software professional, you should be somewhat used to that by now.
Here’s what I have to do. First, I need to lay out the virtual world in which to run the simulation. In a game studio, this is done with some GUI-driven world-building tool. When the designer presses Save, the geometry and associated state values of everything that has been placed in the world are persisted into an XML file. Second, I need to be able to read the XML file and recreate that virtual world. I’ll call the file, EEKWorld.xml, and write it to a default directory. If I can poke some of my more artistic colleagues, maybe we can produce a simple EEK!-builder as well. For this column, we’ll just assume it exists.
So I’ll read in the generated XML file and store it within a hierarchy of class abstractions that represent each entity or relationship between entities within the world. I will look at those classes in the next column, and I’ll also cover in detail the nature of the EEK! world and how I have designed its internal representation. That’s the fun part, but first I have to get the XML itself into the application before I can make sense of it.
So, in the rest of this column, I’ll review the two primary strategies for parsing an XML file—the firehose and the Document Object Model (DOM) models. Firehose is meant to suggest a one-way stream of data; you can’t retrieve the data once it flows by. The DOM is a traversable in-memory representation of the XML file. My actual implementation will use the DOM, but it’s important to be aware that both are available.

XmlReader: the Firehose
One way to process an XML document is to read it one node at a time. In other words, I grab the node, process it, maintain the necessary state context, and proceed to the next node. I do that under .NET with the abstract XmlReader class (which has several non-abstract derived types, such as XmlTextReader). This allows me to skip nodes that are not of interest to me. The XmlReader is defined within the System.Xml namespace. Figure 1 represents the general code engine to pump through each node.
Figure 1 General Code Engine
using namespace System;       // for String, Console
using namespace System::Xml;  // for XML classes and enums
using namespace System::IO;   // for FileStream

int main()
{
    String^ fname = “EEKWorld.xml”;

    FileStream^ fs = File::OpenRead( fname );
    XmlReader^ xrd = XmlReader::Create( fs );

    while ( xrd->Read() )
    {
        // process each node ...
    }

    xrd->Close();
    fs->Close();
}
The FileStream object reads the actual text. The XmlReader provides the smarts to understand and structure what it all means. Each invocation of Read reads the next node from the FileStream. It evaluates to false when it reaches the end-of-stream. I access the content of the node through the reader object. Some of the properties available for the current node include Name, NodeType, HasValue, Value, HasAttributes, AttributeCount, and NamespaceURI. Figure 2 illustrates how I might program that—specifically, it fills in the while loop of Figure 1.
Figure 2 Pumping through the Nodes
while ( xrd->Read() )
{
    Console::WriteLine( “node type is {0}”, xrd->NodeType );

    if ( xrd->Name != String::Empty )
        Console::WriteLine( “\tname is {0}”, xrd->Name);
    else Console::WriteLine(“\tThis node has no name”);

    if ( xrd->HasValue )
        Console::WriteLine(“\tvalue is {0}”, xrd->Value );

    if ( xrd->NodeType == XmlNodeType::Element &&
         xrd->HasAttributes )
    {
        Console::Write( “\thas {0} attributes: “, xrd->AttributeCount );

        while ( xrd->MoveToNextAttribute() )
            Console::Write( “{0} = {1} “, xrd->Name, xrd->Value );

        Console::WriteLine();
    }
    else Console::WriteLine( “\thas no attributes” );
}
To discover the type of the current node, just examine the NodeType property. For example:
if ( xrd->NodeType == XmlNodeType::Whitespace ) continue;
The XmlNodeType enum serves as a kind of isA flag to identify the type of the current node. For example:
enum class XmlNodeType{ None, Element, Attribute, ... };
Note that attributes are not treated as children of an element. Rather, I must explicitly access them. MoveToAttribute moves to a specified attribute, using one of the following overloaded instances:
// move to the attribute with the specified index
void MoveToAttribute( int index )

// move to the attribute with the specified name
bool MoveToAttribute( String^ name )

// move to the attribute with these characteristics
bool MoveToAttribute( String^ localName, String^ namespaceURI )
To iterate across all the attributes of an element, I can also use the MoveToNextAttribute method. If the current node is an element node, this method moves to the first attribute. Each subsequent invocation moves to the next attribute in turn until there are no more attributes. To get back to the Element node containing the attributes, invoke MoveToElement. So although this represents a forward-only, non-cached reading of the document, it affords some flexibility.
Even though it’s efficient, I don’t personally like using the firehose model. It has two primary drawbacks. First, there is no way to move backwards. Therefore, anything you think you might have need for has to be extracted and stored locally. Second, this switch-on-a-node-type-and-process coding style is essentially typeless in terms of the individual nodes; everything is through the text reader element. My preferred model is the DOM, in which the entire document is stored in memory as a hierarchy of navigable typed class node representing the types supported by XML.
That said, of course, reading the entire document into memory prior to processing is not always the best solution, especially if the document is humongous or you are constrained on available memory. Thus, the firehose model is sometimes the right choice.

Document Object Model
Under DOM, I still use a FileStream and XmlReader pair to read and structure the XML. The difference is that I do not directly manipulate the XmlReader object. Rather, I pass the XmlReader object to the Load method of the XmlDocument class. Load builds the in-memory tree representation of the document, which I can navigate. The individual node types are represented by the XmlNode class hierarchy, which contains such derived classes as XmlElement and XmlAttribute. Figure 3 shows an example of setting up the XML file in memory.
Figure 3 Setting Up the XML File in Memory
using namespace System;
using namespace System::Xml;
using namespace System::IO;

int main()
{
    String^ fname = “EEKWorld.xml”;
    FileStream^ fs = File::OpenRead( fname );
    XmlReader^ xrd = XmlReader::Create( fs );

    // build the in-memory representation
    XmlDocument^ xd = gcnew XmlDocument;
    xd->Load( xrd );

    ProcessDom(xd); // work with the DOM

    xrd->Close();
    fs->Close();
}
The XmlDocument class provides a set of properties supporting querying about the in-memory representation of the document. For example, Figure 4 illustrates the ProcessDom function invoked from the main function in Figure 3. The interior nodes of the document are each represented by an individual object, all of which are rooted in the abstract XmlNode base class. In fact, the XmlDocument class itself inherits from XmlNode.
Figure 4 ProcessDom
void ProcessDom( XmlDocument^ xd )
{
    Console::WriteLine( “XmlDocument {0} :: {1} “, xd->Name, xd->Value );

    XmlAttributeCollection^ xac = xd->Attributes;
    if ( xac->Count != 0 )
    {
        // process attributes ...
    }

    Console::WriteLine(“Retrieving the {0} XmlDocument Children\n”,
        xd->ChildNodes->Count );

    XmlNodeList^ children = xd->ChildNodes;
    for each ( XmlNode^ node in children )
    {
      Console::WriteLine( “Child node: {0} of type {1}”,
          node->Name, node->NodeType );
      Console::WriteLine( “Child has children? {0} :: Node’s parent:
        {1}”, node->HasChildNodes, node->ParentNode->Name );
    }
}
Each of the node types is represented by a class derived from the abstract XmlNode base class. XmlElement represents an element, and XmlAttribute represents an attribute, XmlComment, a comment, and so on. The ChildNodes property returns the node’s children maintained within an XmlNodeList object.
The GetElementsByTagName method allows me to retrieve all child elements matching the tag name. In the code in Figure 5, for example, its invocation retrieves all the tile elements within the world configuration file listed in the order they occur within the XML document.
Figure 5 Processing Tile Elements
// root node of the document
XmlElement^ xelem = xd->DocumentElement;

// returns a list of all elements with a tag name of tile
XmlNodeList^ tiles = xelem->GetElementsByTagName( “tiles” );
Console::WriteLine( “there are {0} tiles to process: “, tiles->Count );

// now let’s process each node ...
for each ( XmlNode^ tile in tiles )
    ProcessTile( static_cast<XmlElement^>( tile ));
C++/CLI is the only .NET-compliant language that provides a compile-time cast through a static_cast operator; all other downcasts of a base class are performed dynamically at run time—within C++/CLI, this includes the c-style cast, the safe_cast, and the dynamic_cast of a base class. The safe_cast is the preferred cast operator for C++/CLI—it will do what’s necessary, and avoids a runtime cast whenever possible. But to suppress a derived-to-base class dynamic cast, you must explicitly use the static_cast operator.
Why do you want to avoid a runtime cast? Isn’t it dangerous not to? You do it for performance when you are absolutely certain from the semantics of the program that the base class you are manipulating is definitely the derived type you need to manipulate. In this case, I have an XmlNode and I need an XmlElement, and I know that is what the XmlNode really is.
Why do I have an XmlNode instead? I suppose it is because the designers of the XML namespace did not wish to provide an XmlElementList, an XmlDocumentList, and so on, and there were no generics available then, so since all these different XML types are derived from XmlNode, they can all be contained by a list of XmlNodes. That would seem to be the only reason to get an inappropriate node type and have to perform a downcast before I can access the correct interface. I think of it as the use tax of an object-oriented design.
To offset that sort of unnecessary performance penalty, we decided within the group working on C++/CLI (and it was a broad group effort) to provide one compile-time hook: the static_cast operator. It was something we all felt was needed in writing real-world code. It just seems naive to think that sort of unnecessary runtime overhead isn’t going to impact performance adversely. The counter argument is that I’m exhibiting a native mindset in believing that, and this is not a significant overhead within a managed environment. That’s the argument of those who think we erred in providing a compile-time cast operator. Ultimately, we don’t really know if it was the right decision; it seems to go against the .NET philosophy, but we did it and it’s there. And since it’s there, I’m going to take advantage of it.
In the next column, I’ll look at the transformation of the XML file being parsed into a collection of C++/CLI classes that represents the world of the simulation. And thus I’ll arrive at the heart of the application.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman currently works for Perpetual Entertainment, a company specializing in the development of massively multiplayer online games and the infrastructural technology to develop and support them. Stan is best known for working on the original C++ cfront compiler with its creator, Bjarne Stroustrup. His all-time favorite job was Software Technical Director on the Firebird segment of Fantasia 2000. He also worked with the Visual C++ team at Microsoft in the invention of C++/CLI.


Windows with C++
Windows Vista Control Enhancements
Kenny Kerr

Code download available at: WindowsWithC++2007_08.exe (165 KB)
Browse the Code Online
Contents
Button Controls
Edit and Combobox Controls
Tree View Controls
Header and List View Controls
Wrapping Up

Welcome to the first installment of Windows with C++. My focus will be on Windows Vista™ and Windows Server® 2008 development using native C++. This column will showcase new and exciting features introduced in the latest version of Windows®.
There are two main reasons why this column was born. First, despite the phenomenal success of the Microsoft® .NET Framework, there are still plenty of scenarios where it makes more sense to use native C++. Adoption of .NET will continue to grow, but C++ is here to stay. Second, with the launch of a major new operating system it is unreasonable to expect that all the various application frameworks—from MFC and the Active Template Library (ATL) to .NET and more—will be able to wrap up all the new features and enhancements in the short term. C++ remains the only language that provides unfettered access to every corner of the Windows SDK, and there certainly are many new corners to explore in Windows Vista and beyond! For more info about the future of .NET, see the "What About .NET?" sidebar.
To start off this series, I’ll be writing about controls. Windows Vista doesn’t introduce many new controls (though it does provide the new Network Address Control, which allows input and validation of IPv4, IPv6, and DNS names), but it offers a number of enhancements and new features to many of the existing standard and common controls. This is good news in a way because it means you can take advantage of many of these new features in your applications with very little effort. In this column, I am going to take you on a whirlwind tour through some of the most commonly used controls and look at various new and exciting features that are provided by Windows Vista and beyond. The download for this column, available from the MSDN® Magazine Web site, provides examples of most of the control features described here.

Button Controls
The button control is responsible for a few of the seemingly new controls in Windows Vista. New style flags introduce noticeably different appearances compared to traditional buttons. Figure 1 illustrates some of the new button styles.
Figure 1 New Windows Vista Button Styles
Figure 1 New Windows Vista Button Styles
The BS_SPLITBUTTON style creates a split button. The button’s parent window receives the BCN_DROPDOWN notification message indicating that the user clicked the dropdown arrow on the button. You are then responsible for displaying a shortcut menu. Your dropdown handler might be implemented as follows:
CRect rectangle;
VERIFY(m_splitButton.GetWindowRect(
    &rectangle));

TPMPARAMS params = { sizeof(TPMPARAMS) };
params.rcExclude = rectangle;

CMenuHandle menu = m_menu.GetSubMenu(0);

VERIFY(menu.TrackPopupMenuEx(TPM_LEFTBUTTON,
    rectangle.left, rectangle.bottom,
    m_hWnd, &params));
This snippet first calculates the bounds of the button and then populates a TPMPARAMS structure to inform TrackPopupMenuEx that, if the menu cannot be displayed in the desired location, it should not overlap the button but instead be displayed on the next available edge. You can also use the BCM_GETSPLITINFO and BCM_SETSPLITINFO messages to query and adjust the appearance of the dropdown arrow, but in most cases the defaults should serve you well.
The BS_COMMANDLINK style creates a command link. Command links feature an arrow next to the button’s caption as well as an optional note displayed in a smaller font. The button’s caption is set as usual with the SetWindowText function, but the new BCM_SETNOTE message sets the note appearing underneath the caption.
Finally, you can instruct Windows to display the now famous "elevation required" shield on a button by sending it the BCM_SETSHIELD message with the LPARAM set to TRUE to include the icon and FALSE to remove it. Here’s a simple example:
button.SendMessage(BCM_SETSHIELD, 0, required);

Edit and Combobox Controls
Windows XP introduced the ability to display a textual cue on edit controls with the EM_SETCUEBANNER message. This cue prompts the user for information that disappeared when the control got the focus. Windows Vista extends this with a subtle but valuable addition, allowing you to show the cue text even when the control has the focus. This seemingly small improvement removes the necessity for static controls in many cases. Simply use the previously unused WPARAM and set it to TRUE to take advantage of this feature. The Edit_SetCueBannerTextFocused macro is provided for simplicity:
VERIFY(Edit_SetCueBannerTextFocused(m_edit, L”Cue text”, TRUE));
Windows Vista also brings this feature to comboboxes in much the same way with the CB_SETCUEBANNER message. It does not, however, provide the option of showing the cue text when the control has the focus and only the "Drop List" style comboboxes continue to show the cue text until a selection is made, as illustrated in Figure 2.
Figure 2 Combo Box Sample
Figure 2 Combo Box Sample
The ComboBox_SetCueBannerText macro is provided for simplicity and convenience:
VERIFY(ComboBox_SetCueBannerText(m_comboBox, L”Cue text”));

Tree View Controls
The Windows developers finally bit the bullet and introduced extended styles for the tree view control. Figure 3 illustrates some of the new tree view control styles and themes in action.
Figure 3 Tree View
Figure 3 Tree View
The TVM_GETEXTENDEDSTYLE and TVM_SETEXTENDEDSTYLE messages are provided for getting and setting the bitmask of extended styles. The TreeView_GetExtendedStyle macro wraps the TVM_GETEXTENDEDSTYLE message and can be used as follows:
DWORD style = TreeView_GetExtendedStyle(m_treeView);
Usually you won’t need to get the extended styles, however, since the TVM_SETEXTENDEDSTYLE message allows you to update a subset of style flags using a bitmask. For example, you can add the TVS_EX_AUTOHSCROLL extended style while at the same time removing the TVS_EX_MULTISELECT extended style by using the TreeView_SetExtendedStyle macro that wraps the TVM_SETEXTENDEDSTYLE message as follows:
TreeView_SetExtendedStyle(m_treeView,
                          TVS_EX_AUTOHSCROLL,
                          TVS_EX_AUTOHSCROLL | TVS_EX_MULTISELECT);
Undoubtedly the most exciting addition to the tree view control is support for multiple selections. Finally, developers can ditch all that code to fake multiple selections and simply use the TVS_EX_MULTISELECT extended style and let the system handle the rest. Of course, you will want to enumerate the selected items. This is possible with the introduction of a new flag for the TVM_GETNEXTITEM message. Start by getting the currently selected item as before with the TVGN_CARET flag and then use the TVGN_NEXTSELECTED flag to get remaining selected items. Here is a simple example using the Windows Template Library CTreeViewCtrlEx and CTreeItem classes:
   for (CTreeItem item =
        m_treeView.GetSelectedItem();
        0 != item;
        item = item.GetNext(TVGN_NEXTSELECTED))
        {
        CString text;
        item.GetText(text);
        TRACE(L”%s\n”, text);
        }
CTreeViewCtrlEx’s GetSelectedItem method sends the TVM_GETNEXTITEM message with the TVGN_CARET flag to get the first selected item and CTreeItem’s GetNext method sends the TVM_GETNEXTITEM message again with the specified flag and the tree item’s handle to provide message context.
If all you need to know is the number of selected items, then the TVM_GETSELECTEDCOUNT message will do the trick. The associated TreeView_GetSelectedCount macro wraps the message and is used as follows:
DWORD count = TreeView_GetSelectedCount(m_treeView);
There are a few more extended styles that are worth checking out. The TVS_EX_DOUBLEBUFFER extended style tells the control to paint via double-buffering. This avoids the flicker when the control is resized. The TVS_EX_AUTOHSCROLL extended style allows the control to automatically scroll a selected tree item into view. This is often combined with the TVS_NOHSCROLL style, which hides the horizontal scroll bar. Here is a simple example:
DWORD treeViewStyle = m_treeView.GetStyle();
treeViewStyle |= TVS_NOHSCROLL;
m_treeView.SetWindowLong(GWL_STYLE, treeViewStyle);

TreeView_SetExtendedStyle(m_treeView,
                          TVS_EX_AUTOHSCROLL,
                          TVS_EX_AUTOHSCROLL);
Although the default behavior is usually sufficient, you can control the auto-scroll animation characteristics using the TVM_SETAUTOSCROLLINFO message.
You may also have noticed that Windows Explorer on Windows Vista now sports new fading "expando" buttons with a new arrow bitmap in place of the older plus/minus boxes. To switch to arrows, you need to use the SetWindowTheme function to instruct Windows to use the visual styles consistent with Windows Explorer for the specified control:
COM_VERIFY(::SetWindowTheme(m_treeView, L”explorer”, 0));
Request the fade effect by using the TVS_EX_FADEINOUTEXPANDOS extended style.

Header and List View Controls
Although relatively few developers use the header control directly, virtually all Windows developers are familiar with the list view control, which relies heavily on the header control for many of its features. Both header and list view controls have received many enhancements in Windows Vista. In this section, I’ll focus on those features from the perspective of list view controls since that’s the control most developers are familiar with. Many of the features can, however, be employed without a list view control and some require direct interaction with the header control.
The list view control in Windows Vista has received a major upgrade with many new features, styles, and improvements. Unlike the tree view control, extended styles have been available for quite some time for the list view control. Windows Vista does, however, define a handful of new extended styles that enable significantly richer user interactions.
The LVS_EX_AUTOSIZECOLUMNS extended style automatically resizes the view’s columns as one column is being resized to best fit the various columns into view. This is a nice touch that users will appreciate and you can add it with no trouble at all.
The LVS_EX_AUTOCHECKSELECT style provides the kind of Select All checkbox common in tabular controls in Web applications. This corresponds to the new HDS_CHECKBOXES header control style. A checkbox is placed in the column header above the item checkboxes. Clicking the checkbox selects or deselects all the list view item checkboxes at once. Figure 4 provides an example of what this looks like.
Figure 4 List View Select All Control
Figure 4 List View Select All Control
As with the tree view control, to get that Windows Explorer theme for your list view all you need to do is call the SetWindowTheme function to instruct Windows to use the visual styles consistent with Windows Explorer.
The LVS_EX_COLUMNSNAPPOINTS extended style works in conjunction with additions to the LVCOLUMN structure and snaps a particular column to a minimum width when the user resizes it. This behavior is also provided when using the LVS_EX_AUTOSIZECOLUMNS extended style. To take advantage of this, add either extended style and prepare the LVCOLUMN structure appropriately. For example, the following example inserts a column with an initial width of 200 pixels and a minimum width of 100 pixels:
LVCOLUMN column = { 0 };
column.mask = LVCF_TEXT | LVCF_WIDTH | LVCF_MINWIDTH;
column.pszText = L”Name”;
column.cx = 200; // initial width
column.cxMin = 100; // minimum width
m_listView.InsertColumn(0, &column);
The new LVCF_MINWIDTH mask is used to indicate that the cxMin field is populated and represents the minimum width for the column.
Another feature in Windows Vista that you may have noticed is that Windows Explorer shows the column header in all views, not just in details view mode. By default the header control is only visible in the details view mode. Simply use the LVS_EX_HEADERINALLVIEWS extended style to show the column headers in all the view modes. Although it takes a bit of getting used to on the user’s part, it is quite a handy feature since it allows the user to sort the view in different view modes.
You may also have noticed that when in a different view mode, such as the tiles view mode, the columns in Windows Explorer cannot be resized and a horizontal scroll bar is not used to scroll any overflowing column headers into view. Instead, an overflow button is presented in the header control. Clicking the button presents a pop-up menu with a list of the header columns that are not visible and thus overflowed. There are a few steps involved in reproducing this behavior.
First, you need to set the view mode to something other than details, and you need to use the LVS_EX_HEADERINALLVIEWS extended style to show the column headers. Second, you need to add the HDS_NOSIZING and HDS_OVERFLOW styles to the list view control’s header control as follows:
CHeaderCtrl header = m_listView.GetHeader();
DWORD headerStyle = header.GetStyle();
headerStyle |= HDS_NOSIZING | HDS_OVERFLOW;
header.SetWindowLong(GWL_STYLE, headerStyle);
The HDS_NOSIZING style disables resizing of all columns, and the HDS_OVERFLOW style displays an overflow button as needed. Finally, you need to handle the HDN_OVERFLOWCLICK header control notification message. It is here that you may display a pop-up window or menu of some kind to present the overflowed columns to the user. You may want to use the TrackPopupMenu function to accomplish this. The iItem member of the NMHEADER structure provided through the notification message’s LPARAM gives you the index of the first column that overflowed.
Another new feature provided by the header control is the split button. The column header appears much like the split buttons described in the section on button controls with a dropdown button appearing when the mouse is over the column header. Use the LVCFMT_SPLITBUTTON format flag when preparing the LVCOLUMN structure for a new column. This corresponds to the HDF_SPLITBUTTON format flag used with the HDITEM structure when working directly with a header control. The LVN_COLUMNDROPDOWN notification message is sent by the list view control when the dropdown buttons is pressed. Again, this corresponds to the HDN_DROPDOWN notification message if you’re working directly with a header control. The LVN_COLUMNDROPDOWN message’s LPARAM is a pointer to a NMLISTVIEW structure and the iSubItem field indicates the index of the column whose dropdown was clicked.
Another area of the list view control that received an upgrade in Windows Vista is the grouping features. Figure 5 illustrates some of the new labels that can be applied to a group. The LVGROUP structure used to define list view groups has more than doubled in size.
Figure 5 List View Grouping
Figure 5 List View Grouping
The group illustrated in Figure 5 can be created as follows:
LVGROUP group = { sizeof(LVGROUP) };

group.mask = LVGF_GROUPID | LVGF_HEADER | LVGF_SUBTITLE | LVGF_TASK |
             LVGF_FOOTER | LVGF_STATE | LVGF_ALIGN;

group.iGroupId = 1;
group.pszHeader = L”Header”;
group.pszSubtitle = L”Subtitle”;
group.pszTask = L”Task”;
group.pszFooter = L”Footer”;
group.state = LVGS_COLLAPSIBLE;
group.uAlign = LVGA_FOOTER_RIGHT | LVGA_HEADER_CENTER;

m_listView.InsertGroup(-1, &group);
What about .NET?
Future versions of the .NET Framework will undoubtedly wrap up many of the features I’ve described in this column. You don’t, however, need to wait until that happens to take advantage of these features in your .NET Framework-based applications as Windows Forms provides the necessary hooks in most cases to extend them with new functionality.
Many of the features I described require sending a control a particular message. Messages are typically sent using the SendMessage function exported by the user32.dll system library and you can import it into your managed application as follows:
[DllImport(&quot;user32.dll&quot;)]
static extern int SendMessage(IntPtr window, int message,
As an example, the section on tree view controls in this column described how to enable multiple selections. You can enable this same thing given a Windows Forms TreeView class using the following code:
int TVM_SETEXTENDEDSTYLE = 4396;
int TVS_EX_MULTISELECT = 2;

SendMessage(m_treeView.Handle, TVM_SETEXTENDEDSTYLE,
            new IntPtr(TVS_EX_MULTISELECT),
Here’s another example illustrating how to handle notification messages. The Windows Forms Control class provides the protected WndProc method that allows you to process specific messages as needed. Many of the messages sent by controls to inform the control’s owner of various events are sent in the form of a WM_NOTIFY message where the LPARAM is a pointer to a NMHDR structure. The NMHDR structure’s code member identifies the specific notification message being sent. Here is how you can handle a notification message in managed code:
struct NMHDR
{
    public IntPtr hwndFrom;
    public IntPtr idFrom;
    public int code;
}

const int WM_NOTIFY = 0x004E;

protected override void WndProc(ref Message message)
{
    bool handled = false;

    if (WM_NOTIFY == message.Msg)
    {
        NMHDR header = (NMHDR)Marshal.PtrToStructure(message.LParam,
                                                     typeof(NMHDR));
        if (&lt;someCode&gt; == header.code)
        {
            // TODO: handle notification here
            handled = true;
        }
    }

    if (!handled)
    {
        base.WndProc(ref message);
    }
}

The LVGS_COLLAPSIBLE state flag allows the group to be collapsed and expanded. Collapsing a group hides the list view items that belong to the group. The LVGS_COLLAPSED state flag can be used to collapse a group programmatically. The LVGA_ flags can be used to override the default alignment for the group labels. The header and footer text are usually left-aligned; the subtitle always appears under the heading. The task link is always right-aligned. The LVN_LINKCLICK notification message is sent when the user clicks the task link. The group that the task link belongs to is identified by the iSubItem field of the NMLVLINK pointer provided as the message’s LPARAM.
Finally, you may optionally respond to the LVN_GETEMPTYMARKUP notification message to provide some text to display when the list view control has no items. You may use the anchor tag to create links within the text. Here is an example of a function that handles the notification message:
LRESULT OnGetEmptyMarkup(LPNMHDR notifyData)
{
    NMLVEMPTYMARKUP* markupInfo =
        reinterpret_cast<NMLVEMPTYMARKUP*>(notifyData);

    markupInfo->dwFlags = EMF_CENTERED;

    wcscpy_s(markupInfo->szMarkup,
             _countof(markupInfo->szMarkup),
             L”Link <A>one</A> and <A>two</A>.”);

    return TRUE; // set the markup
}
By default the message is displayed in the top-left corner of the list view. The EMF_CENTERED flag centers it both horizontally and vertically. The list view reports a user clicking a link in the same way as group task links. A given LVN_LINKCLICK notification message handler can, however, be shared. In the case of an empty markup link, the iItem and iSubItem members of the NMLVLINK structure will be set to -1, the nested LITEM structure will be populated with the link identifier set to EmptyMarkup, and its iLink member set to the index of the link within the text.

Wrapping Up
There are many more small features and enhancements that you will discover as you explore the Windows Vista controls with the help of the Windows SDK. Many scenarios that previously required owner or custom drawing are now provided out of the box. Additionally, upcoming versions of Windows Template Library and MFC feature many improvements that allow you to more easily take advantage of the Windows Vista control improvements. For more information, see "MFC Updates for Vista Common Controls".

Send questions and comments for Kenny to mmwincpp@microsoft.com mmwincpp@microsoft.com.

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. You can reach Kenny at weblogs.asp.net/kennykerr.


{ End Bracket }
Dependency Management
Eric N. Bush

The many layers of dependencies in today’s complex projects create plenty of opportunities for collaboration. Unfortunately, there are often no guidelines in place for facilitating this cross-group collaborative work. Dependency management, in which one group provides a software component to be utilized by others, is one key area where collaboration skills play an important role. Some simple rules can facilitate effective dependency management skills.
The concept of dependencies is controversial among developers. Development teams are often reluctant to take on dependencies because they feel they will lose control of their projects. No one wants to partner with a team that won’t listen, has produced poor quality in the past, or is constantly delivering late.
Another roadblock is the "not invented here" syndrome. This happens when individual developers think that only they are capable of writing exactly the code that suits their needs. They believe they can duplicate any code that a dependency might create, and do it fairly quickly.
Despite these concerns, dependency management can have enormous benefits if handled properly and integrated wisely into your project. Consider the following scenarios: there may be more than one group within your company duplicating certain functionality. Thus, consolidating duplicate efforts frees up your resources to work on higher-level features. Moreover, a focused and dedicated dependency team can produce a component superior in quality and functionality because it has the time to do a complete review of security, reliability, internationalization, and performance. Don’t forget that the support and maintenance of shipped code can be a greater cost than the initial development. Happily, your dependency can take all that off your hands.
I recently participated in a project where one of the features required files to be downloaded across the Internet in the background so as to not disrupt ongoing critical processes. We needed to throttle the network bandwidth usage and also withstand intermittent connections. After some investigation, we found another group producing such a component, so we took that on as a dependency, saving ourselves a great deal of time and cost.
Dependency-taking requires clear communication, however. When you first set up a dependency, all teams involved have to decide how they will balance the governing of the joint venture. This will reduce the cost of that interaction and help things proceed in an organized way. The following are some guidelines that can help in creating effective dependency management:
Authority Who makes what decisions? Does anyone have the final say? What is the escalation process?
Roles Clarify the expectations and determine who is responsible for what.
Goals/Value Define what success looks like. Drive for alignment. Enumerate and explain the risks.
Communication How and what will be communicated? Establish meeting schedules, e-mail protocols, and Web site access.
Accountability How will you track progress? What is the process for fixing mistakes?
Engineering System Create an issue database, source code control, build, setup, drop points, and quality measures.
One of the biggest problems with dependencies is making sure that you get what you really need, when you need it, and at the level of quality you expect. If you are on the client side of the dependency, you should make sure to get your requests (features, bugs, issues) placed at a priority you are comfortable with and that both sides agree upon. Get a commitment of support and maintenance from your dependency’s management. You should also participate in their internal reviews and status meetings so you can track their quality.
Then once the dependency gets started, be careful not to incorporate any aspect into your build until it has reached an acceptable level of maturity and stability. Nothing is worse than constant change underneath you. Supplying your own test suite helps your dependency verify it is not breaking your code. In fact, the ideal situation is when dependencies are more stable than your code. If a problem arises with your dependency, make clear to them the impact it is having on your schedule.
You can make dependencies work for your organization by practicing proper management of the process. Never assume that the dependency can read your mind. Keep things transparent throughout the process.

Eric N. Bush is a software development lead in the Engineering Excellence group at Microsoft. He works as an internal consultant and classroom instructor driving engineering and organizational improvements in the areas of people skills, process, and technology. You can reach Eric at ericbush@microsoft.com.