 MSDN Magazine February 2007

MSDN Magazine February 2007

RibbonX API: Extend The 2007 Office System With Your Own Ribbon Tabs And Controls

If you want to create your own professional looking tabs and controls in Office, check out the RibbonX API of the 2007 Microsoft Office system.

Eric Faller Smarter Art: Create Custom SmartArt Graphics For Use In The 2007 Office System

SmartArt incorporates a gallery of templates and predefined shapes that can quickly be inserted and configured in your Microsoft Office documents. Find out how.

Janet Schorr Office Add-Ins: Develop Add-Ins For PowerPoint And Visio Using VSTO

Microsoft Visual Studio 2005 Tools for the 2007 Microsoft Office System is more powerful than ever, allowing you to create add-ins for Word, Excel, Outlook, PowerPoint, Visio, and InfoPath.

Paul Stubbs Data Joins The Team: Introducing Visual Studio 2005 Team Edition for Database Professionals

Here Brian Randell presents everything you need to know to get started with Visual Studio 2005 Team Edition for Database Professionals.

Brian A. Randell SQL Server 2005: Regular Expressions Make Pattern Matching And Data Extraction Easier

Now you can perform efficient, sophisticated text analysis using regular expressions in SQL Server 2005.

David Banister Editor's Note: Hello, Office

Nothing’s more disappointing than clever design paired with poor implementation. Bluetooth headsets come to mind: they’re tiny, wireless, and cute. But, they don’t pair properly, they turn on when you want them off, or off when you want them on.

Joshua Trupin Toolbox: Rich Web App UIs, Tools for Regular Expressions, and More

In this issue, create rich Web app UIs, explore tools for regular expressions, and create screen mockups.

Scott Mitchell CLR Inside Out: .NET Application Extensibility

This month the CLR team introduces the new System.AddIn namespace in the Base Class Library, which will be available in the next release of Visual Studio.

Jack Gudenkauf and Jesse Kaplan Data Points: Data Bound Applications with ADO.NET and Custom Objects

In this column see how to bind a custom list of business entities using the binding tools in the .NET Framework 2.0.

John Papa Office Space: Building Office Open XML Files

This new column explores how you can extend and customize Microsoft Office System applications and file formats.

Ted Pattison Test Run: AJAX Test Automation

This month James McCaffrey presents a technique that allows you to write lightweight test automation to verify the functionality of AJAX Web applications.

Dr. James McCaffrey Cutting Edge: Perspectives on ASP.NET AJAX

AJAX allows you to build rich browser applications using powerful combinations of existing client-side Web technologies. This month Dino delves into AJAX

Dino Esposito Service Station: The Service Factory for WCF

This month Aaron Skonnard continues his exploration of software factories with a look at the Web Service Software Factory for Windows Communication Foundation.

Aaron Skonnard Foundations: Build a Queued WCF Response Service

With queued calls in Windows Communication Foundation you can perform work in a disconnected manner by sending messages through Microsoft Message Queuing. Juval Lowy explains.

Juval Lowy Netting C++: Mapping Native C++ to the Common Type System

This month Stanley Lippman begins translating the Text Query Language Query class hierarchy from C++ to the .NET Common Type System.

Stanley B. Lippman C++ at Work: Rationales, Highlights, and a Farewell

What's the deal with const functions, and lots more on the reasoning behind the design of the C++/CLI.

Paul DiLascia { End Bracket }: Let My People Code

Charles Petzold discusses why limiting resources can spawn great creativity.

Charles Petzold


RibbonX API: Extend The 2007 Office System With Your Own Ribbon Tabs And Controls

RibbonX API

Extend The 2007 Office System With Your Own Ribbon Tabs And Controls

Eric Faller
This article discusses:
  • An introduction to the Ribbon
  • RibbonX controls and features
  • Upgrading add-ins to use RibbonX
  • Building add-ins for Word and Excel
This article uses the following technologies:
The 2007 Microsoft Office system
Code download available at: RibbonX2007_02.exe (201KB)
Contents
Support for Existing Add-Ins
New Features in RibbonX
StartFromScratch Mode
Command Disabling and Repurposing
Demand-Loading
Upgrading a Word Template to RibbonX
Adding XML and Images
A COM Add-In for Excel
Implementing the Callback Functions
Invalidating getVisible Callback Values
Other Callbacks
Wrap-Up

I
f you've seen the 2007 Microsoft Office system, you will have noticed that the new Microsoft® Office user interface is radically different from that of previous Office releases. So you probably won't be surprised to hear that the extensibility model behind the UI is also completely new.

Much like the old toolbars and menus it extended, the CommandBars object model (available since Office 97) was difficult to use effectively. The new model is known as RibbonX and claims many of the same attributes of the new UI itself: ease of use, consistency, modernity, and predictability for end users.

RibbonX disentangles the design of the UI from the code behind it by using XML to specify its contents and structure, with a dynamic callback mechanism behind it. I'll go into all the details later in this article, but first there are a few things that seasoned Office developers may want to know about the new UI.

In moving away from the old UI, Microsoft is also leaving behind a lot of familiar terminology and replacing it with new and unfamiliar concepts. Let's start by going over the basics of the new UI to ensure we're all on the same page. Figure 1 shows Microsoft Office Word 2007 with each of the major UI components labeled.

Figure 1 Elements of the New Word 2007 Interface
Figure 1 Elements of the New Word 2007 Interface(Click the image for a smaller view)
Figure 1 Elements of the New Word 2007 Interface
Figure 1 Elements of the New Word 2007 Interface (Click the image for a larger view)

1. Ribbon The large rectangular region above the document is known as the ribbon. It contains the title bar, the Office Button, the Quick Access Toolbar, and the Tabs. RibbonX applies mainly to the ribbon and everything inside of it.

2. Office Button This button drops the Office Menu, which is the rough equivalent of the File menu in previous releases of Office. The Office Menu contains commands that act on documents rather than on the contents of documents. RibbonX add-ins can freely alter the contents of the Office Menu (though they cannot customize the Office Button itself).

3. Quick Access Toolbar This toolbar contains commonly used commands and is the main location for end-user customizations. Users can right-click on any ribbon control and add it to the Quick Access Toolbar (including custom RibbonX controls). Because it's meant to be the space that "belongs" to the end user, RibbonX add-ins are not usually allowed to alter the Quick Access Toolbar unless they have enabled StartFromScratch mode.

4. Tabs The tabs make up the main content of the ribbon and contain UI controls that deal with the contents of the document at hand. RibbonX add-ins can create their own custom tabs and alter the visibility and labels of the built-in tabs.

5. Contextual Tabsets When objects such as pictures or tables are selected inside the document, contextual tabsets appear and contain all the UI elements for dealing with those objects. RibbonX add-ins can alter the visibility of built-in tabsets and add custom tabs to them. One feature not supported in the 2007 Office release is the creation of custom contextual tabsets. Tabsets contain contextual tabs, which otherwise behave the same way as regular tabs.

6. Groups Tabs contain sets of groups, which in turn contain individual UI controls. RibbonX add-ins can alter the visibility of built-in groups and create their own custom groups. One thing they cannot do is alter the contents of built-in groups. This restriction protects the UI layout and guards add-ins against conflicts with each other and future versions of Office. Groups optionally have dialog box launchers in the corner that display dialogs relevant to the group (such as the Font or Paragraph dialogs).

7. Task Panes Several task panes still exist in the 2007 Office system, and it's now possible to have more than one open at a time. COM add-ins can now create CustomTaskPanes that host content such as ActiveX® controls or Windows® Forms controls. (The CustomTaskPane feature is distinct from RibbonX and is not covered in this article.)

8. MiniToolbar The MiniToolbar is a collection of common formatting commands that appears above text selections and right-click context menus. RibbonX add-ins cannot modify the contents of the MiniToolbar, but they can disable or repurpose the built-in commands on it.

9. Context Menus These are the same right-click context menus that we all know and love from previous versions of Office. In the 2007 Office release, RibbonX does not apply to context menus, but they can be extended and customized using the CommandBars object model just as they could in the past.

10. Status Bar The status bar contains several handy new controls, such as word count and the view slider. The status bar cannot be customized by add-ins in the 2007 Office system, though it can be hidden.

Support for Existing Add-Ins

If you already have add-ins written against Office 2003, you might wonder whether they will still work in the 2007 release. Happily, the answer is yes, they will still work just fine (unless they do crazy things like checking the version of Office and refusing to run on newer releases). All of the old menus and toolbars still exist under the covers for the simple reason that legacy add-ins that manipulate them still need to run on the 2007 Office system. How do they show up in the new UI? Let's take a look at an example.

Figure 2 shows a hypothetical Word 2003 add-in that creates a custom toolbar and adds a few buttons to the old menus and toolbars, along with how it looks when loaded in Word 2007.

Figure 2a The Same Add-Ins in Word 2003 and Word 2007
Figure 2a The Same Add-Ins in Word 2003 and Word 2007

Figure 2b

As you can see, when a legacy CommandBar add-in is loaded, the Add-Ins tab shows up in the ribbon. The Add-Ins tab contains three groups: Menu Commands, Toolbar Commands, and Custom Toolbars. The Menu Commands and Toolbar Commands groups contain custom controls that were added to the legacy built-in menus and toolbars, respectively. They will also display any legacy built-in controls that were repurposed by add-ins to perform custom actions (by setting the OnAction or Hyperlink properties). The Custom Toolbars group displays any legacy custom toolbars as horizontal sections. The tooltips of the controls show the name of the legacy toolbar they are from.

In the event a misbehaved add-in leaves behind buttons after it is uninstalled, the user can right-click on the controls in the Add-Ins tab and delete them from the legacy CommandBar structure.

New Features in RibbonX

RibbonX brings Office programming into the modern age with XML-based UI declaration. Instead of writing complicated code that builds up the UI using a series of object model calls, you can create an XML file that specifies the appearance of the UI in a structured markup form. This has numerous advantages for the add-in writer.

First, you're separating the UI from the code. The UI designer doesn't need to know how to update the code manually in order to try out new designs. In fact, if the add-in chooses to load its XML from an external file, the UI can be modified and perfected without even recompiling the add-in. This eliminates the middle step from the iterated Edit, Compile, Run cycle required by hardcoded UI systems.

Second, the add-in's complete UI is known by Office. This may not sound significant until you realize that with CommandBars, previous versions of Office had no idea which toolbars and menus belonged to which add-in. All of the UI modifications came in over COM calls, which were not traceable back to any individual add-ins. This led to several annoying problems, which can now be avoided completely.

Third, Office can automatically clean up an add-in's UI. Previously add-ins had to have special code that went through and deleted their menus and toolbars on shutdown or uninstall. If the add-in crashed or failed to clean itself up properly (as many did), it would leave useless buttons in the UI. Since the 2007 Office system cleans up add-in UI automatically, there's no need to write any cleanup code and no possibility that there will be left-over UI.

Versioning between different releases of Office is another issue you'll avoid. Valid XML files are required to contain an xmlns namespace declaration that identifies which schema version they comply with. This provides RibbonX with information about which version of Office a particular add-in targets and allows it to map the UI appropriately, given any changes that might have occurred between versions. The lack of versioning and ownership in the CommandBars object model are two of the main reasons that legacy add-ins often broke when upgrading to new releases of Office and that the Add-Ins tab is unable to better distinguish between the jumble of controls added to it by the OM calls.

Finally, XML has good tool support. Given the RibbonX schema definition file (XSD), a wide variety of XML editors (including Word 2007 itself) can be used to create valid RibbonX UI files with drag-and-drop simplicity.

The CommandBars object model can be described as using a "push" model, where add-ins are pushing all of the UI data to Office by setting various properties in the object model. Each property of the controls in the UI must be explicitly set by the add-in at startup, even if the menu or toolbar where that control lives is never shown. If this involved loading up a large number of image files, it could be quite expensive, performance-wise.

RibbonX uses a "pull" model, where Office pulls the data from the add-in only when it's needed. Instead of setting properties like CommandBar.Name or CommandBarButton.Picture, add-ins provide get callbacks, such as getLabel or getImage. Office calls these callbacks when it needs to know what the label or image of a control is.

The main advantage of the pull model is performance. In the 2007 Office release, it's probable that most of an add-in's UI will not be visible when the application starts, especially if that add-in has added its own top-level tab or a group to the Add-Ins tab. There is no immediate need to incur the performance penalty of loading up all of the images for those controls. Only when the user clicks that tab do the images need to be loaded. RibbonX delays the calling of callbacks as much as possible in order to amortize the cost of resource loading across the entire session, and keeps Office applications starting up snappily even when several add-ins are installed.

If Office is deciding when to call these callbacks, you might wonder how to change control properties on the fly. RibbonX caches the callback return values and won't call back until the add-in invalidates them. Add-ins can do this using the Invalidate and InvalidateControl methods on the IRibbonUI interface. Once a control's properties have been invalidated, RibbonX will know to call back the next time they are needed. If that control is currently on the screen, it will call back and update the value immediately. If it's not on the screen, RibbonX won't call back until it is. We'll get into all the nitty-gritty details of how to implement callback functions and do invalidations in the sample add-ins later in the article.

RibbonX provides many different control types for extensibility, including those shown in Figure 3. Only the first six of the items in this list were available in CommandBars, so there are many new UI options available to RibbonX add-ins. Perhaps the most compelling new control types are a splitButton that includes both a button and dropdown menu (see Figure 4), and a gallery that expands to display a selection of images. They are both used extensively in the 2007 Office release to reduce UI clutter and easily preview visual operations. Add-in writers are encouraged to use these new control types to give their UI the same impact.

Figure 3 RibbonX Controls (Close Figure 3)

RibbonX Controls
1. button
2. toggleButton
3. editBox
4. menu
5. comboBox
6. dropDown
7. dialogBoxLauncher
8. gallery
9. splitButton
10. label
11. checkBox
12. group
13. tab
14. superTip
Figure 4 SplitButton
Figure 4 SplitButton

StartFromScratch Mode

Many developers use Office as a platform for building their own self-contained apps. While running, these apps often remove all of the built-in Office UI and replace it with their own. There was no standard mechanism for doing this using CommandBars, so it was fraught with difficulty and error.

RibbonX introduces StartFromScratch mode, which makes hiding the Office UI as easy as writing one line of XML. Setting startFromScratch="true" on the root <ribbon> tag will do the following things:

  • Hide all of the main top-level built-in tabs (but not the contextual tabsets)
  • Hide the contents of the Quick Access Toolbar
  • Enable the add-in to add its own buttons to the Quick Access Toolbar (this is normally disallowed)
  • Hide all the commands on the Office Menu except for New, Open, and Save

Once these changes are applied, add-ins can make further UI changes, including revealing some of the built-in tabs or hiding further TabSets or Office Menu items.

Most of StartFromScratch mode's functionality can be duplicated by manually hiding all of these UI elements, but StartFromScratch does have a couple of advantages over doing this by hand. First, it's a whole lot easier to write one line of XML than fifty. Second, it's designed to be forward-compatible with future versions of Office. If the next version of Word adds new top-level tabs or Office Menu items, StartFromScratch mode should automatically hide them, but add-ins that hide the UI by hand will need to be updated to hide these new elements.

Another great feature of StartFromScratch mode is that it works on a per-document basis. If the user is editing multiple documents in Multiple Document Interface (MDI) mode, one of those documents can enable StartFromScratch mode, and all the UI will automatically hide and show as the user switches between that document and others. No code required!

Command Disabling and Repurposing

Another operation commonly performed by Office add-ins is disabling built-in commands. RibbonX also makes this as easy as writing one line. Developers can disable built-in commands using lines like this in the <commands> section of their XML:

<command idMso="Bold" enabled="false"/>

This will disable the Bold button anywhere it appears in the UI, which in the default configuration includes the Font group in the ribbon and also in the MiniToolbar.

Because this didn't happen with CommandBars, it's important to note that this disables the Bold button in both locations. Add-ins were required to enumerate through the entire UI and manually disable every copy of the button, in case the user had customized their toolbars and placed a copy somewhere else. They also had to search for commands by their ID number (43, anyone?), instead of human-readable names like "Bold". A getEnabled callback can also be used if the add-in wants to conditionally disable the command only at certain times.

The <commands> XML element is also the home to the RibbonX Command Repurposing feature. Add-ins often enhance or extend the built-in functionality of Office, take over existing UI buttons, and repurpose them. Alternatively, IT managers sometimes choose to restrict functionality using repurposing (for example, showing a password dialog box when the Print button is clicked).

Any button that performs an action on click can be repurposed using RibbonX (though more complicated control types such as galleries or comboboxes cannot be repurposed). Here's an example of the XML required to repurpose the Save button:

<command idMso="Save" onAction="MySaveFunction"/>

When Save is clicked, the add-in's MySaveFunction will run, giving the add-in the opportunity to perform its own actions and choose whether to allow the built-in Save function to execute.

Once again, all copies of the Save button will be repurposed by this one line of XML (both on the Office Menu and on the Quick Access Toolbar), and if an individual document is doing the repurposing, it won't apply to any other documents which are also open (unless that document is loaded as a global template or add-in).

Demand-Loading

One problem with Office add-ins is that their DLLs often take a while to start up, especially if they are written in managed code and the common language runtime (CLR) is not already loaded. Users with several add-ins installed can see a significant performance hit every time they launch an Office application. The irony is that the user probably won't even use most of the add-ins' functionality in any particular session, but they pay the price in all of them.

Since slowdowns reflect poorly on both Office and the add-ins, Office provides a feature for COM add-ins known as demand-loading. This feature has been upgraded in the 2007 Office release and extended to include RibbonX support. An outline of the process follows. The add-in writer sets the add-in's LoadBehavior registry key to ConnectFirstTime (16). On the first boot of the application since the add-in was installed, the add-in is started up. Office queries the add-in for its RibbonX XML and caches it away on disk. RibbonX then calls all of the add-in's Get callbacks in order to load all of its images and cache the initial state of the add-in.

On shutdown, the add-in's registry key is changed to DemandLoad. On subsequent boots of the application, the add-in is not started, but its RibbonX UI is shown as if it had been loaded. As soon as the user clicks one of the add-in's buttons, the add-in is loaded in order to execute the user's command.

Demand-loading is transparent to the end user and, aside from setting the registry key, developers don't have to do any work either. (Compare this to the difficulty of setting up <!ProgID> strings to use demand-loading with CommandBars.) Obviously, demand-loading won't be useful for every add-in, but it should benefit the majority of add-ins that just add a single button or menu item to the UI and then activate their functionality when it's clicked. More complicated add-ins that need to run elaborate code before the user interacts with their UI won't be able to take advantage of demand-loading, however, since they need to be loaded in order to run code.

Upgrading a Word Template to RibbonX

Now that we've covered all the major features of RibbonX, let's dive into the details with some example add-ins and code snippets. First I'll take a look at a scenario many developers will encounter when switching to the 2007 Office system: upgrading a legacy add-in from CommandBars to RibbonX.

The add-in I'm going to upgrade is a fairly simple Word template (.dot) that provides functionality for inserting blank pages into a legal document, complete with the standard (and self-contradictory) "This page intentionally left blank" text.

The add-in is implemented with a button on the Insert menu in the document's attached CommandBars. The button's OnAction property is set to InsertBlankPage, which is a Visual Basic® for Applications (VBA) macro that contains the following code:

Sub InsertBlankPage()
    ' Insert a blank page
    Selection.InsertBreak Type:=wdPageBreak
    Selection.ParagraphFormat.Alignment = wdAlignParagraphCenter
    Selection.Font.Size = 14
    Selection.Font.Color = wdColorGray50
    Selection.Font.Name = "Arial Black"
    Selection.TypeText Text:="This Page Intentionally Left Blank"
    Selection.InsertBreak Type:=wdPageBreak
End Sub

It's obviously not too complicated, but it works as a demo. The file is available for download from the MSDN®Magazine Web site. You can install it as a global template using the Templates and Add-Ins dialog.

The first step when upgrading a legacy CommandBar add-in to the new 2007 Office system UI is to map the old UI into the new model. With all of the toolbars and menus replaced by the ribbon, it's not immediately obvious where the add-in's UI will fit best.

Before you even get to that, take a step back and ask the question, "Do we even need to upgrade this add-in at all?" In many cases, if the budget is tight, the answer might be "No." If you load the add-in in Word 2007, it works just fine in the Add-Ins tab (see Figure 5).

Figure 5 Legacy Add-In
Figure 5 Legacy Add-In

There's nothing particularly wrong about how it works here, but you might not like how it's stuck in the Menu Commands group with all the other legacy add-ins, and it isn't using any of the new features of RibbonX. It also wouldn't make a very good demo to just quit at this point, so let's continue.

Since the add-in inserts pages into a document, a good place for the UI would probably be on the Insert tab. If the add-in dealt with manipulating the document as a whole, you might choose to put its UI into the Office Menu. If there was no good place for it in the existing UI, a custom group on the Add-Ins tab would be best (to avoid creating a whole tab with just one button).

The built-in Insert tab for Word contains a group for inserting pages and, lo-and-behold, it already contains a button for inserting blank pages (a new feature in the 2007 Office release). Since the legacy add-in also inserts "Intentionally Blank" placeholder text, I'd like to distinguish it from the built-in functionality. Let's add a custom group named "Legal Pages" next to the other group, and have a big button with an instructive icon. Figure 6 shows what it should look like when it's finished.

Figure 6 Insert Page Add-In
Figure 6 Insert Page Add-In

The next step is to come up with the XML file that will create the desired result. In demos it's often quickest just to start with the final product and then explain how it works. In that spirit, Figure 7 shows the final XML.

Figure 7 Insert Blank Page XML (Close Figure 7)
<customUI xmlns="http://schemas.microsoft.com/office/2006/01/customui">
  <ribbon>
    <tabs>
      <tab idMso="TabInsert">
        <group id="LegalGroup" label="Legal Pages"
               insertAfterMso="GroupInsertPages">
          <button id="BlankPageButton" label="Intentionally Blank Page"
                  size="large" image="BlankIcon"/>
    </group>
   </tab>
  </tabs>
 </ribbon>
</customUI>

As you can see, it's pretty straightforward. The root tag is <customUI> and specifies the 2007 Office system RibbonX XML namespace to start things off. Then it goes through a series of tags that mirror the structure of the UI: Ribbon > Tabs > Tab > Group > Button.

The first interesting item is the idMso="TabInsert" property on the <tab> tag. All controls in the XML must have ID properties to identify them. Built-in controls use idMso to show that they are Office controls and to distinguish them from custom controls, which use the id property. Thus the XML file is essentially saying "Find the built-in Insert tab and create a custom group inside of it." The insertAfterMso="GroupInsertPages" property says "Insert this group after the built-in Pages group." Finally, button id="BlankPageButton" says "Create a custom button inside the group."

In this example, the labels of the group and button are specified statically in the XML, so they can never change. That's fine in this case, but if you wanted to dynamically change the labels (for localization purposes, perhaps), you could instead use the getLabel property to specify a function that will return the label.

The last interesting bit is the image="BlankIcon" property. This instructs RibbonX to search for an image in the Word template file with that ID. You'll need to stick this image into the file at the same time you insert the XML.

Adding XML and Images

Attached RibbonX XML is only supported in the new file format for the 2007 Office release, so the first thing you'll need to do is open up the .dot file and save it as a .dotm file (the added "m" means that the file has macros). Next, rename the file with a .zip extension and open it up to check out the contents, shown in Figure 8.

Figure 8 Inside the Document Template
Figure 8 Inside the Document Template(Click the image for a smaller view)
Figure 8 Inside the Document Template
Figure 8 Inside the Document Template (Click the image for a larger view)

The details of the Open XML file format are a topic for another article, but it suffices here to say that the files are essentially compressed packages containing a bunch of files that are related to each other in various ways. In this example, document.xml is the main Word document, and it's related to various things such as the style information, the VBA code and the attached CommandBars. Since I don't want the file to have the attached CommandBars anymore, let's just delete attachedToolbars.bin file out of the .zip file and then pack it back up.

We could manually edit the .zip file to insert the RibbonX XML and the icon image, but it would be much easier to just use a tool, so let's do that. Over at openxmldeveloper.org there are a whole bunch of tools for dealing with the Open XML file format. The one you want is called Custom UI Editor.

Custom UI Editor is a handy little tool for inserting RibbonX XML and associated images into Open XML format files. Use it to open the .dotm file, then paste in the XML and insert the image (see Figure 9). Save the file. If you're curious, you can open it back up as a .zip file and see that the tool created the XML file at \customUI\customUI.xml and the image at \customUI\images\BlankIcon.png. It also modified a couple of rels relationships to point to these new files.

Figure 9 Editing the Template
Figure 9 Editing the Template(Click the image for a smaller view)
Figure 9 Editing the Template
Figure 9 Editing the Template (Click the image for a larger view)

If you open up the file in Word now you'll see the new button, but it won't do anything when clicked. That's obvious, since you haven't told it which macro to call. To do that, add the properties shown here in red to the XML:

<button id="BlankPageButton" label="Intentionally Blank Page"
        size="large" image="BlankIcon"
        onAction="RibbonXOnAction" tag="InsertBlankPage" />

At first it might be strange to introduce a new macro called RibbonXOnAction instead of just reusing the InsertBlankPage macro. The reason I did it this way is that RibbonX OnAction macros have different signatures than their CommandBar counterparts do. If you had lots of macros, you would need to change them all by hand, which could be time consuming and error prone. It would also involve breaking the code so that it would not run on previous versions of Office.

You can get around both of these problems by having the new RibbonX macro just call the old CommandBar macros. This is where the tag property comes in: it's a string property that isn't used in the UI, but is passed on to the add-in's code for use there. Here's how it's used in the RibbonXOnAction macro, which you can paste into a new VBA module:

Sub RibbonXOnAction(button As IRibbonControl)
    Application.Run button.Tag
End Sub

The RibbonX OnAction macro takes an IRibbonControl object as a parameter, which represents the button that was clicked. In CommandBars, OnAction macros did not have parameters, which sometimes made it difficult to figure out which button was being clicked. The IRibbonControl object provides ID and Tag properties to solve this problem in RibbonX.

Once you have that macro in place, the fancy new button starts working! That's all it took to upgrade this add-in's code to run using RibbonX: one new one-line macro and no changes to the existing code. Slick.

A COM Add-In for Excel

As a second example, let's make things a bit more complicated and write a COM add-in for Excel®. The add-in will be a miniature line-of-business (LOB) app for filling out and submitting timecards. I'll only be able to show snippets of the code here, but the full source is available in the code download.

Once again the first step is to figure out what sort of UI is needed. The add-in allows users to create timecards and then edit and submit them. Since the only timecard UI that you need to show at first is a New Timecard button, place it by itself on the Office menu (see Figure 10).

Figure 10 A New Button
Figure 10 A New Button

Once the New Timecard button is clicked (or an existing timecard is opened), display the rest of the timecard UI in a custom tab. Since various Excel tabs such as Page Layout, Formulas, Review, and Data don't apply to filling out timecards, hide those at the same time (see Figure 11).

Figure 11 A Custom Timecard Ribbon Tab
Figure 11 A Custom Timecard Ribbon Tab(Click the image for a smaller view)
Figure 11 A Custom Timecard Ribbon Tab
Figure 11 A Custom Timecard Ribbon Tab (Click the image for a larger view)

The timecard UI is clearly a lot more complicated than the previous add-in's was, and so is the XML to create it, as shown in Figure 12. The first thing COM add-ins need to use RibbonX is to implement the IRibbonExtensibility interface. Fortunately, the Visual Studio® IntelliSense® feature makes this easy. Create an add-in by selecting New Project in Visual Studio, going to the Other Project Types category, and selecting Shared Add-in in the Extensibility section.

Figure 12 Timecard UI XML (Close Figure 12)
<customUI xmlns="http://schemas.microsoft.com/office/2006/01/customui"
          onLoad="OnLoad" loadImage="LoadImage">
  <ribbon>
    <officeMenu>
      <button id="NewTimecard" insertAfterMso="FileNew"
              label="New Timecard" imageMso="StartAfterPrevious"
              onAction="NewTimecard" />
    </officeMenu>
    <tabs>
      <tab id="TimecardTab" label="Timecard"
           getVisible="IsCustomTabVisible">
        <group id="EditingTools" label="Editing Tools">
          <button id="ClearTimecard" label="Clear Timecard" size="large"
                  imageMso="TableDeleteRowsAndColumnsMenuWord"
                  onAction="ClearTimecard"/>
          <button id="InsertDay" label="Insert New Day" size="large"
                  imageMso="CellsInsertDialog" onAction="InsertNewDay"/>
          <button id="CalculateTotal" label="Calculate Hours Worked"
                  size="large" imageMso="SlideShowRehearseTimings"
                  onAction="CalculateHours"/>
        </group>
        <group id="WageTools" label="Wage Tools">
          <editBox id="Wage" label="Hourly wage:"
                   image="dollar1.png" onChange="OnWageChanged"
                   getText="GetWage"/>
          <editBox id="Overtime" label="Overtime bonus:"
                   image="dollar2.png" onChange="OnOvertimeChanged"
                   getText="GetOvertime"/>
          <button id="CalculatePay" label="Calculate Pay"
                  size="large" imageMso="AcceptInvitation"
                  onAction="CalculatePay"/>
        </group>
        <group id="SubmissionTools" label="Submission Tools">
          <button id="SignTimecard" label="Sign Timecard"
                  size="large" image="signature.png"
                  onAction="SignTimecard"/>
          <button id="SubmitTimecard" label="Submit Timecard"
                  size="large" image="submit.png"
                  onAction="SubmitTimecard" getEnabled="IsTimecardSigned"
                  screentip="Make sure to sign your timecard before
                            submitting" />
        </group>
      </tab>
      <tab idMso="TabPageLayoutExcel" getVisible="IsBuiltinTabVisible"/>
      <tab idMso="TabData" getVisible="IsBuiltinTabVisible" />
      <tab idMso="TabReview" getVisible="IsBuiltinTabVisible" />
      <tab idMso="TabFormulas" getVisible="IsBuiltinTabVisible" />
    </tabs>
  </ribbon>
</customUI>

Click through that dialog and the wizard that comes after it. You'll need to select various options such as which Office apps your add-in should be installed in, and which programming language to use. For this particular example I chose to use C#.

Once your add-in is created, you should see the Connect class, which implements the standard COM add-in IDTExtensibility2 interface. This is the object you need to implement IRibbonExtensibility on. Add that interface and have Visual Studio automatically implement its methods for you. You should end up with something like this:

using Office = Microsoft.Office.Core;
[GuidAttribute("E8407539-C642-43BB-8FCB-2E27A46179DE"),
 ProgId("TimecardAddin.Connect")]
public class Connect : Object, Extensibility.IDTExtensibility2,
    Office.IRibbonExtensibility
{
    public string GetCustomUI(string RibbonID)
    {
        throw new Exception("Not implemented.");
    }

    // IDTExtensibility methods
}

The only method of IRibbonExtensibility is GetCustomUI, which returns the RibbonX XML as a string. It also takes a RibbonID parameter, which is a string like "Microsoft.Word.Document" or "Microsoft.Excel.Workbook" and is mainly useful when writing add-ins that run in multiple applications or applications with multiple types of ribbons (such as Microsoft Outlook).

If you add the XML file to the project as an Embedded Resource named customui.xml, you can return it from GetCustomUI using code like this:

public string GetCustomUI(string RibbonID)
{
    Assembly assembly = Assembly.GetExecutingAssembly();
    using(Stream stream = assembly.GetManifestResourceStream(
        "TimecardAddin.customui.xml"))
    {
        return new StreamReader(stream).ReadToEnd();
    }
}

At this point you can build the add-in and install it. Upon starting Excel you should see the customized UI, with the exceptions being that the custom images are not loaded and the buttons don't do anything when clicked.

All of the callback functions referenced in the XML will fail to be called since you haven't implemented them yet. If the "Show add-in user interface errors" option in the Advanced section of the application options dialog is enabled you will see a sequence of error messages about the failing callbacks. You should make sure to check this option while developing RibbonX add-ins. It enables an extensive set of error messages that alert you when anything goes wrong in your add-in. The first error message that shows up explains that the "onLoad" callback failed to run, so let's implement that one first.

Implementing the Callback Functions

The root <customUI> tag in the Ribbon has an optional onLoad callback that add-ins can implement to get a reference to the IRibbonUI object. The IRibbonUI object is used to invalidate cached callback return values whenever the state of the add-in's UI needs to change. I'll get into the details of how to use it soon, so just stuff it into a member variable for now. Add this code to the Connect class:

Office.IRibbonUI m_Ribbon;

public void OnLoad(Office.IRibbonUI ribbon)
{
    m_Ribbon = ribbon;
}

Another optional callback on the <customUI> tag is loadImage, which is how COM add-ins load up the icons for their controls. This is where the image="dollar1.png" lines in the XML come into play. For each string specified in an image property, RibbonX calls loadImage to load that image.

Images can be stored however an add-in wants, but in this example let's add the images as embedded resources just as with the XML. Assuming you have added dollar1.png as a resource in the TimecardAddin assembly, here's how to implement the loadImage callback:

public Bitmap LoadImage(string imageName)
{
    Assembly assembly = Assembly.GetExecutingAssembly();
    Stream stream = assembly.GetManifestResourceStream(
        "TimecardAddin." + imageName);
    return new Bitmap(stream);
}

There are a couple of interesting things to note about this function. First, it's returning a System.Drawing.Bitmap object as the image. Support for System.Drawing.Bitmap is new to the 2007 Office release with RibbonX and is the simplest and recommended way to return images for managed add-ins. Unmanaged (C++ and Visual Basic 6.0) add-ins can return IPictureDisp objects for their images, which is the same format that was used in CommandBars.

The second thing to note is that you're returning just one image that contains both the bitmap data and the transparency data. That's right-no more needing to create two separate Picture and Mask icons for your Office add-ins! With RibbonX, Office now supports full 32-bit icons with alpha channels. Currently the PNG format has the best tool support for alpha channels and is the recommended format for your RibbonX icons.

The next set of callbacks you need to implement are the getVisible callbacks on the built-in tabs and the custom tab. The built-in tabs all use IsBuiltinTabVisible and the custom tab uses IsCustomTabVisible, since these two functions need to return opposite values (the built-in tabs should hide when a timecard is being edited, which is when the custom tab should show up).

The add-in can keep track of whether a particular Excel workbook is a timecard by setting the Category property of the file to Timecard. Files without that property will be ignored.

The IsBuiltinTabVisible and IsCustomTabVisible functions will both use a helper IsTimecard function:

public bool IsCustomTabVisible(Office.IRibbonControl tab) {
    return IsTimecard(tab.Context as Excel.Window);
}

public bool IsBuiltinTabVisible(Office.IRibbonControl tab) {
    return !IsTimecard(tab.Context as Excel.Window);
}
private bool IsTimecard(Excel.Window window)
{
    if (null == window) return false;

    Excel.Workbook workbook = (Excel.Workbook)window.Parent;
    Office.DocumentProperties properties =
        (Office.DocumentProperties)workbook.BuiltinDocumentProperties;
    Office.DocumentProperty category = properties["Category"];

    return (null != category && null != category.Value &&
            category.Value.Equals("Timecard"));
}

This code uses the Context property of the IRibbonControl that is passed to RibbonX callbacks. The Context property is equal to the Window object in most Office applications. (Access does not have Window objects so it's NULL there, and in Outlook it's equal to the Inspector objects.) Add-ins should use the Window object to determine which document this RibbonX callback is referring to, because it's not always equal to the Application.ActiveDocument property. The Window object can also be used to distinguish between two different windows that are viewing the same document.

From the Window object you can get the Workbook object, and from there you can get the Category property and check whether it's equal to Timecard.

Invalidating getVisible Callback Values

If you start Excel now, you'll see that the custom tab hides itself and the built-in tabs are showing up. This is correct since the default document isn't a timecard. But if you open a Timecard document, the tabs don't show up. Why is that?

For performance reasons, RibbonX caches the return values from get callbacks. If it didn't, it would be calling back to the add-in all the time just in case it wants to switch something in its UI. The add-in needs to let Office know when it wants to change its UI.

This is where the IRibbonUI object discussed earlier comes into play. It contains two methods: Invalidate and InvalidateControl. The Invalidate method will invalidate all cached callbacks, while the InvalidateControl method takes a control ID parameter and invalidates the cached callbacks just for a specific custom control.

You need to invalidate the tab's getVisible callback values whenever a document-switch happens in Excel since that is when the UI might need to become visible. Excel provides the WorkbookActivate event that you can use for this. Add a handler for that event in the OnStartupComplete method:

public void OnStartupComplete(ref System.Array custom)
{
    excelApplication.WorkbookActivate +=
       new Excel.AppEvents_WorkbookActivateEventHandler(
           OnWorkbookActivate);
}

void OnWorkbookActivate(Excel.Workbook Wb)
{
    // the custom tab might need to become visible on workbook
    // switch, so invalidate the cached getVisible values
    m_Ribbon.Invalidate();
}

Fortunately Visual Studio IntelliSense can automatically implement the event handler for you as soon as you type the += characters, so this is easy.

Now if you open Excel and switch between a Timecard document and another document, you'll see that the custom tab shows up only for the timecard and the built-in tabs hide at the same time. Perfect. Now you just need to be able to use the add-in to create timecards in the first place.

The New Timecard button's callback is pretty straightforward: it uses the same OnAction signature shown earlier. It creates a new workbook with the Category property equal to Timecard and sets up the default sheet:

public void NewTimecard(Office.IRibbonControl button)
{
    Excel.Workbook timecard =
        excelApplication.Workbooks.Add(Missing.Value);
    Office.DocumentProperties properties =
        (Office.DocumentProperties)timecard.BuiltinDocumentProperties;
    properties["Category"].Value = "Timecard";
    Excel.Worksheet timesheet =
        (Excel.Worksheet)timecard.Worksheets[1];
    NewTimesheet(timesheet);
}

It uses a helper function to initialize the timesheet in the first worksheet of the workbook (the same function is called by the Clear Timecard button).

Other Callbacks

The rest of the callbacks are similar to that of the New Timecard button, so I won't include all their code here, but there are a couple of interesting ones worth mentioning. The EditBoxes that hold the hourly wage and overtime multiplier are slightly more complicated than simple buttons since they require two callbacks in order to behave properly. Here are the callbacks for the wage box:

double m_Wage = 5.25;

public void OnWageChanged(Office.IRibbonControl c, string wage)
{
    m_Wage = double.Parse(wage);
}

public string GetWage(Office.IRibbonControl c)
{
    return m_Wage.ToString();
}

OnWageChanged gets called whenever the user types a new value into the wage box. This example just converts and records the new value, but a more realistic add-in would probably want to do some extra bounds-checking and validation on the value before accepting it.

The GetWage method supplies the initial value for the EditBox. It's important to note that it also gets called when the UI is invalidated by the IRibbonUI object. Add-ins shouldn't rely on the fact that the user's input will stick around inside the EditBox, so I recommend that they keep track of the current value using a pair of onChanged and getText like in this example (OnWageChanged and GetWage).

The add-in stores the wage value as a member variable in the add-in, which is OK since this value is global to the add-in and not specific to the document. If this value was tied to each individual timecard, you would need to store the value in memory associated with the timecard, so that the value would properly update when the user switches between two timecards.

The other control of interest is the Sign Timecard button. It pops up a dialog for the user to enter their name and then writes that name into the timecard:

public void SignTimecard(Office.IRibbonControl button)
{
    SignatureDialog sd = new SignatureDialog();
    if (DialogResult.OK == sd.ShowDialog())
    {
        string signature = sd.GetSignature();

        timesheet.get_Range("E" + (daycount + 5),
        Missing.Value).Formula = signature;

        m_Ribbon.InvalidateControl("SubmitTimecard");
    }
}

It also calls IRibbonUI.InvalidateControl on the SubmitTimecard button. The Submit button is disabled until the timecard is signed using a getEnabled function that checks for a signature. Once that signature is in place, you need to invalidate that getEnabled return value in order to re-enable the button.

Since the add-in only calls InvalidateControl when the Sign Timecard button is clicked, the Submit button won't become enabled if the user manually types their signature into the timecard's cell (or become disabled if the user deletes their signature). If you wanted to get that working we could hook the SheetChanged event and call InvalidateControl whenever the signature cell is modified.

Wrap-Up

That's about all there is to the Timecard add-in. The whole add-in is only a few hundred lines of code, and that's including all the code autogenerated by Visual Studio. Admittedly, it's pretty simplistic and needs a lot more error checking and validation code before it could actually be used, but it demonstrates how easy it is to use RibbonX to leverage Office as a platform for your own applications. If you're interested in trying it out, download the source code.

This article has only scratched the surface of the functionality in RibbonX, so if you're looking for more details and example code, check out the official page on MSDN. There are links to the full RibbonX documentation, lists of control IDs, sample add-ins, and RibbonX-related blogs.

Eric Faller is a software developer on the Office User Experience team at Microsoft, working on making the new UI extensible to developers. Reach him at eric.faller@microsoft.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Smarter Art: Create Custom SmartArt Graphics For Use In The 2007 Office System

Smarter Art

Create Custom SmartArt Graphics For Use In The 2007 Office System

Janet Schorr
This article discusses:
  • SmartArt graphic data model
  • Creating XML for custom SmartArt graphics
  • Algorithms, constraints, and rules
  • Testing and debugging custom SmartArt graphics
This article uses the following technologies:
The 2007 Microsoft Office System
Code download available at: SmartArt2007_02.exe (152KB)
Contents
Planning a Graphic Layout
Layout Nodes and Algorithms
Layout Tree and Data Model Mapping
Shape Properties
Constraints, Rules, and Text Properties
Packaging the Layout Definition File
Testing the Graphic Layout
Error Types and the Error Log
Valid File with Design Errors
Modifying Existing Graphic Layout Definitions
What's Next?

T
he 2007 Microsoft Office system offers a new way to quickly add polished graphics and diagrams to your Office files, including Word documents, Excel® spreadsheets, PowerPoint® presentations, and Outlook® e-mail messages. The new feature, called SmartArt™ graphics, incorporates a gallery (library) of templates and predefined shapes that can be quickly inserted and configured. It provides automatic sizing and alignment, while allowing you to edit objects and properties (see Figure 1 for several examples). But as you work with these graphics, experimenting with all the possibilities, it's easy to imagine additional graphics you'd like to see in the gallery. This isn't a problem-SmartArt graphics is completely extensible, allowing you to create your own layouts.

Each graphic layout in the SmartArt layout gallery has its own underlying XML file that defines how the SmartArt graphic will construct the object based on the dataset entered by the user (in the case of Figure 1, this dataset consisted of three text strings: "Design", "Create", and "Test"). SmartArt graphic layouts use a specific set of algorithms that provide different layout options, including a linear flow algorithm, a cycle algorithm, and two algorithms that work together to create a hierarchy diagram. It also supports a composite algorithm that lets you determine exactly where and how to size and position various shapes, giving you the flexibility to create a wide range of graphics. In this article, I'll go through the basics of creating your own SmartArt graphic layout.

Figure 1a Sample Diagrams Created with SmartArt Graphics
Figure 1a Sample Diagrams Created with SmartArt Graphics

Figure 1b

Figure 1c

Figure 1d

Planning a Graphic Layout

The first step in creating a SmartArt graphic layout is deciding what the graphic should look like. Once that's settled, you can start to analyze the SmartArt graphic to figure out how to create it.

Let's say you want to create a graphic that looks like the one shown in Figure 2. It will take several steps to achieve this SmartArt graphic, and I'll walk through the details of each in this article. First, figure out the shapes you'll need to create the graphic-in this case, you'll need rounded rectangles for the blue shapes, rectangles for the white lines, and rectangles with no lines or fill for text areas.

Figure 2 Target Graphic Layout
Figure 2 Target Graphic Layout(Click the image for a smaller view)
Figure 2 Target Graphic Layout
Figure 2 Target Graphic Layout (Click the image for a larger view)

Next, look at how the shapes are arranged. In this case, the blue rectangles are in a horizontal line, starting at the left side of the drawing area; the white boxes are combined with the blue boxes to create a single composite shape; and there is white space between each of the composite shapes. The arrangement of shapes is determined by one of SmartArt's algorithms for linear flows, bending linear flows, cycles, hierarchies, and composite (or fixed position) layouts.

Now look at how the text is displayed in the shapes. All SmartArt graphics map back to an underlying data model that can be visualized as a hierarchical list. For example, the text shown in the graphic in Figure 2 is based on the list shown in Figure 3.

Figure 3 Model (Close Figure 3)

Design
Step One
Step Two
Step Three
Create
Step One
Step Two
Step Three
Test
Step One
Step Two
Step Three

In a SmartArt graphic, each shape can support multiple levels of text, which map back to this structure. In the sample in Figure 2, each composite shape contains two levels of text. The first level is sometimes called parent text or level-one items. The second level is sometimes called child text or level-two items. In general, child text has bullets in front to show that it's subordinate to the parent text.

The final step is to determine how you want it to display when it's abstracted away from a specific dataset. SmartArt graphic layouts provide a number of possibilities.

Static Graphic Layout If you always want to display the same number of shapes, you can create a static graphic layout. For our example, I can create a layout that always includes three shapes. The text from the users first three level-one items displays in the shapes, but any text added to subsequent level-one items won't display.

Semi-Dynamic Graphic Layout Using a semi-dynamic graphic layout, you can display only as many composite shapes as there are level-one items, up to a specified maximum number of shapes. For this example, I can create a graphic that includes from zero to three composite shapes. As users add subsequent lines of level-one text, new shapes are added. However, if they add more than three lines of level-one text, no new shapes are added.

Dynamic Graphic Layout A dynamic graphic layout isn't limited to a specific number of shapes. You can have as many composite shapes as there are lines of level-one text. As more shapes are added, the shapes get smaller as needed to fit inside the drawing area. For our sample graphic, a dynamic graphic layout is the best choice.

Layout Nodes and Algorithms

SmartArt graphic layouts are created in XML files that describe what shapes to create, how they map back to the data model, and which algorithms to use to lay out the shapes. (The XML files are part of an open document package, similar to the new file formats used for Word and PowerPoint.) One of these files, layout1.xml, provides the main layout definition.

The basic building block of a layout definition is the layout node. Each node in a layout has an associated algorithm that specifies either how to size and position its child layout nodes or how to size the text within its shape. Ultimately, all the nodes in a layout are nested together under a single layout node (see Figure 4).

Figure 4 Graphical Layout Definition
Figure 4 Graphical Layout Definition(Click the image for a smaller view)
Figure 4 Graphical Layout Definition
Figure 4 Graphical Layout Definition (Click the image for a larger view)

The layout nodes in the layout definition form a hierarchical layout tree. In reality, the tree will have a composite layout node and a space layout node for every parent text item in the user's data model. However, the definition of this layout node and its descendants is the same for each one, so the layout definition file only needs to specify this set of layout nodes once, with some additional information that controls how many times to create this pattern of layout nodes for the graphic. Figure 5 shows the basic structure of the XML for this example.

Figure 5 XML Structure Defining the Sample Layout Tree (Close Figure 5)
<?xml version="1.0" encoding="utf-8"?>
<layoutDef xmlns=
        "http://schemas.openxmlformats.org/drawingml/2006/diagram">
    <layoutNode name="diagram">
        <alg type="lin" />
        <presOf />
        <forEach axis="ch" ptType="node">
            <layoutNode name="composite">
                <alg type="composite" />
                <presOf />
                <layoutNode name="roundRect">
                    <alg type="sp" />
                    <presOf axis="self" />
                </layoutNode>
                <layoutNode name="parentText">
                    <alg type="tx" />
                    <presOf axis="self" />
                </layoutNode>
                <layoutNode name="whiteRect">
                    <alg type="sp" />
                    <presOf />
                </layoutNode>
                <layoutNode name="childText">
                    <alg type="tx" />
                    <presOf axis="des" ptType="node" />
                </layoutNode>
            </layoutNode>
            <forEach axis="followSib" ptType="sibTrans" cnt="1">
                <layoutNode name="space">
                    <alg type="sp" />
                    <presOf axis="self" />
                </layoutNode>
            </forEach>
        </forEach>
    </layoutNode>
</layoutDef>

Notice that in the XML, the layout nodes all have assigned names. These names are optional in this example, but may be required for reference in other parts of the definition. In addition, this naming can help you structure the XML and identify the various sections.

Layout Tree and Data Model Mapping

You'll notice that in addition to the layout node and algorithm tags, the XML sample also contains forEach and presOf tags. These indicate how to map the layout nodes to the user's data model.

The underlying data model in a SmartArt graphic is a collection of nodes and relationships. There are two kinds of relationships in SmartArt graphics: parent/child transitions and sibling transitions. Parent/child transitions construct the hierarchical relationship between nodes. Sibling transitions are relationships between adjacent nodes that have parent/child relationships to the same parent. Figure 6 illustrates these relationships.

Figure 6 Relationships in SmartArt Graphic (Close Figure 6)

Sample Bulleted List

o Parent One
    o Child One
    o Child Two

o Parent Two
    o Child One
Corresponding Data Model
o Root Element
    o Parent/child transition: Root to "Parent One"
    o Node: "Parent One"
        o Parent/child transition: "Parent One"
                  to "Child One"
        o Node: "Child One"
        o Sibling transition: "Child One"
                  to "Child Two"
        o Parent/child transition: "Parent One"
                  to "Child Two"
        o Node: "Child Two"
        o Sibling transition: "Child Two"
                  to "Child One"
    o Sibling transition: "Parent One" to "Parent Two"
    o Parent/child transition: Root to "Parent Two"
    o Node: "Parent Two"
        o Parent/child transition: "Parent Two" to "Child One"
        o Node: "Child One"
    o Sibling transition: "Parent Two" to "Parent One"

The forEach elements in the layout definition XML move through the data model to select a set of nodes, and the presOf (presentation of) elements map the layout nodes to specific items in the data model. The attributes elements are outlined in Figure 7.

Figure 7 forEach and presOf Attributes (Close Figure 7)

Attribute | Description
axis | This determines how to navigate through the data model, setting the context node as it moves. The context node always begins with the root element of the data model. Generally, the following axis values are used in SmartArt graphic layouts: self (returns the current context node), child (returns the immediate children of the current context node), des (returns all descendants of the current context node), desOrSelf (returns the current context node and all its descendants), and followingSib (returns the next item in the data model, regardless of whether it's a node or a transition).
ptType | This determines the type of data model item to select and works as a filter against the items returned by the axis value. ptType can have one of three values: node (text items, which display as bullet points in the SmartArt graphic text area), parChTrans (parent-child transitions), or sibTrans (sibling transitions).
st | For "start", this indicates which item in the returned set to select first. The default value for this attribute is 1, so it is often omitted.
cnt | For "count", this indicates how many items in the returned set to select. The default value for this attribute is 0, which selects all items.
hideLastTrans | This flag determines whether the last sibling transition in the returned set is filtered out of the set. The default value for this attribute is true, but in some cases, such as cycle diagrams, you may want to set it to false.

The XML for the sample graphic layout contains two forEach elements, as shown here:

<forEach axis="ch" ptType="node">
<forEach axis="followSib" ptType="sibTrans" cnt="1">

The first selects all nodes that are direct children of the root element, essentially selecting each level-one text item in the bulleted list. It then creates the layout nodes nested inside the forEach element for each selected data model item. As each item is acted on, the context item resets to the next node in the selected set.

The second forEach element selects the sibling transition item following the current context element. Because this forEach element is nested inside the preceding forEach element, it essentially selects the sibling transitions between each item in the bulleted list.

In the sample XML, each layoutNode has a presOf statement. For example:

<presOf />
<presOf axis="self" />
<presOf axis="des" ptType="node" />

An empty presOf element (<presOf />) indicates that the layoutNode does not map to anything in the data model. These tags are used for shapes without text, as well as for layout nodes without shapes that are used by algorithms to lay out the actual shapes.

A presOf with axis="self" associates the context item with the generated layout node. In the sample XML, this presOf puts the parent text into the parent text area shape.

A presOf with axis="des" and ptType="node" associates all of the descendants of the context item with the generated layout node. For the sample, this presOf puts all of the child text for a specific parent into the same text area shape.

Shape Properties

Now that the structure of the graphic is set up and the layout nodes are mapped to the data model, you can add information about the shapes being created. SmartArt graphic layouts can use any of the standard Office shapes.

Not all layout nodes need to display shapes; some are used to structure the graphic or provide additional space in the resulting graphic during layout. In the sample graphic, the layout nodes used for the linear flow and composite algorithms and for the white space between composite shapes do not display shapes. Each of these nodes is assigned an empty shape element, as shown in the XML in Figure 8.

Figure 8 Assigning an Empty Shape Element (Close Figure 8)
<layoutNode name="diagram">
    <alg type="lin" />
    <presOf />
    <shape />
    <forEach axis="ch" ptType="node">
        ...
    </forEach>
</layoutNode>

<layoutNode name="composite">
    <alg type="composite" />
    <presOf />
    <shape />
    ...
</layoutNode>

<layoutNode name="space">
    <alg type="sp" />
    <presOf axis="self" />
    <shape />
</layoutNode>

The empty shape element isn't required and is usually the default value (the actual default is determined by the algorithm associated with a layout node). However, including the empty tag explicitly indicates that you are not assigning a shape to the layout node.

Now we can start adding shapes for the actual graphic. As I described, each composite shape is composed of two visible rectangles: the rounded blue rectangle and the regular white rectangle. The shape is identified by assigning the shape ID as the type in the shape tag. For the rounded rectangle, this value is roundRect.

The rounded rectangle is one of the Office shapes that has handles that let you adjust aspects of the shape appearance. For this graphic, we don't want the shape to be as rounded as it is by default. So we use the XML to set the adjust points as well as the shape type.

Finally, to make sure the graphic assigns an appropriate look to the rectangle (for example, blue when the default theme is used in PowerPoint), we use the XML to assign a style label to the layout node. The default style label for shapes associated with data model nodes is node1. However, that style generally uses a different colored line and won't align well with the white rectangle. So for this example I chose to use the style label alignNode1:

<layoutNode name="roundRect" styleLbl="alignNode1" >
    <alg type="sp" />
    <presOf axis="self" />
    <shape type="roundRect">
        <adjLst>
            <adj idx="1" val="0.1" />
        </adjLst>
    </shape>
</layoutNode>

Now, you might be wondering how you know what values to use for the adjust handles and how you can determine the shape types without looking up the values in reference material. Here's a little trick. The new PowerPoint file format, like the new Word and Excel file formats, is based on XML. You can add the shape you want to a blank slide, set the adjust handles, and then save the file. By looking at the resulting XML (first rename the resulting PowerPoint file to have a .zip extension and then open the ZIP package to access the XML data), you can deduce the appropriate shape name and adjust handle values. But keep in mind that PowerPoint shapes and SmartArt graphic layouts use different measurement scales, so you'll need to adjust the value. For example, here is the XML from the PowerPoint file for the rounded rectangle used in the sample graphic:

<a:prstGeom prst="roundRect">
    <a:avLst>
        <a:gd name="adj" fmla="val 10000" />
    </a:avLst>
</a:prstGeom>

The white rectangle is much easier. For this shape, all that's required is the shape type-in this case, rect. For the style label, you need to ensure that the box remains contrasted from the rounded rectangle and also stays in front when 3D properties are set. The style label fgAcc1 (foreground accent 1) defines the appropriate look:

<layoutNode name="whiteRect" styleLbl="fgAcc1" >
    <alg type="sp" />
    <presOf />
    <shape type="rect" />
</layoutNode>

Finally, our graphic contains two text area shapes, which are also simple rectangles. Since you don't want these shapes to be visible to the user, you can use the hideGeom (hide geometry) attribute to prevent line and fill values from displaying. And because you want the text to look as if it's part of the blue rectangle, use the same style label for these shapes as you used for the rounded rectangle:

<layoutNode name="parentText" styleLbl="alignNode1" >
    <alg type="tx" />
    <presOf axis="self" />
    <shape type="rect" hideGeom="true" />
</layoutNode>

<layoutNode name="childText" styleLbl="alignNode1" >
    <alg type="tx"/>
    <presOf axis="des" ptType="node" />
    <shape type="rect" hideGeom="true" />
</layoutNode>

Constraints, Rules, and Text Properties

The algorithms assigned to layout nodes determine how the subsequent layout nodes and their shapes are arranged on the canvas. While the algorithms support default shape sizes and fallback behavior, you'll almost always want to control some aspects of the shape sizing and text font behavior through a layout definition.

Constraints allow you to specify an ideal (or starting point) size for each shape, as well as for each shape's font size and margin values. Rules allow you to specify how these constraint values can be altered within a range, if more space is needed for additional shapes or text.

For example, in a linear flow of rectangles, the desired shape size may be 2 inches wide by 1 inch high, with a text font size of 65 points. However, if you add 10 shapes to a standard page, each with a paragraph of text, nothing will fit. You could scale everything proportionally, but this may not provide the look you want. As one alternative, you can add rules that allow the shape height to change up to a value of 5 inches, and font size to shrink to 10 points, but no smaller. These constraints and rules would use the following XML:

<constrLst>
    <constr type="w" val="50" />
    <constr type="h" val="25" />
    <constr type="primFontSz" val="65" />
</constrLst>
<ruleLst>
    <rule type="h" val="125" />
    <rule type="primFontSz" val="10" />
</ruleLst>

When creating rules and constraints, values are specified in millimeters and font sizes are specified in points. Rules are applied sequentially, so in the XML just shown, shapes would first grow up to 125 millimeters, and then the font size would shrink. The rule values are not absolute, meaning that if the text fits at 14 points, the font size will stop shrinking.

Constraints and rules can also be specified as reference values. For example, if you prefer that the height of the rectangles start as half of their width and then grow up to two times their width, the XML would look like this:

<constrLst>
    <constr type="w" val="50" />
    <constr type="h" refType="w" fact="0.5" />
    <constr type="primFontSz" val="65" />
</constrLst>
<ruleLst>
    <rule type="h" fact="2" />
    <rule type="primFontSz" val="10" />
</ruleLst>

Once you've added the constraints and rules, the layout definition file XML is complete. Now let's look at the constraints and rules in terms of the example layout.

Diagram Layout Node Constraints First, you need to determine where to place the various constraints and rules. These can be specified on either the parent layout node or the layout node itself. The location depends on several factors.

  • Convenience-storing the font size information for all layout nodes in one location makes it easier to change them at a later point.
  • Reference-when a constraint or rule refers to another constraint or rule, both generally need to be defined in the same location.
  • Algorithm-some algorithms require constraints to be in specific places. For example, with the composite algorithm, the size and position of the child layout nodes must be specified at the level of the composite layout node.

In the sample layout, the composite layout node's width and height should be as big as the layout area. The root layout node always inherits the canvas dimensions by default, so the first constraints in the list should set the composite node width and height equal to the width and height of the root layout node.

Next, the space between the composite nodes should scale with the graphic so that as more composite nodes are added, the space doesn't take over the graphic. To accomplish this, the space width is set to be 10 percent of the composite node width.

Finally, the font size for all layout nodes is set to 65 points and includes an equality value to ensure that they remain the same size across the graphic. The code in Figure 9 shows the completed XML with the new elements.

Figure 9 Diagram Layout Node with Constraints (Close Figure 9)
<layoutNode name="diagram">
    <alg type="lin" />
    <presOf />
    <shape />
    <constrLst>
        <constr type="w" for="ch" forName="composite"
                refType="w" fact="1" />
        <constr type="h" for="ch" forName="composite"
                refType="h" fact="1" />
        <constr type="w" for="ch" forName="space"
                refType="w" refFor="ch" refForName="composite"
                fact=".1" />
        <constr op="equ" type="primFontSz" for="des"
                ptType="node" val="65" />
    </constrLst>
    <ruleLst />
    <forEach axis="ch" ptType="node">
        ...
    </forEach>
</layoutNode>

Composite Layout Node Constraints The composite algorithm is unique among the SmartArt algorithms in that it doesn't actually control the size and position of the shapes. Instead, the size and position values are specified as constraints, and the composite algorithm uses these values to lay out the shapes. For this reason, any layout node that is positioned by the composite algorithm needs to have its constraints defined in the constraint block of the composite layout node.

When specifying the size and position for shapes, you need to determine the values along both the horizontal axis and the vertical axis. Both start in the upper-left corner of the layout area, with a value of 0. These constraint values are often expressed as a percentage of the composite layout node's width and height.

Along the horizontal axis, the composite algorithm determines the left, center, and right positions, as well as the overall width. Similarly, along the vertical axis, the composite algorithm determines the top, middle, and bottom positions, and the overall height. Any two of these values along either axis must be specified; the rest can then be calculated.

One other consideration when specifying composite constraints is that the constraints must be specified in the same order as the nested layout nodes. Stacking order of shapes is also determined, by default, by the order of the layout nodes, though you can specify stacking order overrides if necessary to achieve the correct look. The constraints for the example graphic layout are shown in Figure 10.

Figure 10 Composite Layout Node Constraints (Close Figure 10)
<layoutNode name="composite">
    <alg type="composite" />
    <presOf />
    <shape />
    <constrLst>
        <!--constraints for roundRect -->
        <constr type="w" for="ch" forName="roundRect"
                refType="w" fact="1" />
        <constr type="h" for="ch" forName="roundRect"
                refType="h" fact="1" />
        <constr type="l" for="ch" forName="roundRect" val="0" />
        <constr type="t" for="ch" forName="roundRect" val="0" />

        <!--constraints for parentText -->
        <constr type="w" for="ch" forName="parentText"
                refType="w" fact="1" />
        <constr type="h" for="ch" forName="parentText"
                refType="h" fact=".175" />
        <constr type="l" for="ch" forName="parentText" val="0" />
        <constr type="t" for="ch" forName="parentText" val="0" />

        <!--constraints for whiteRect -->
        <constr type="w" for="ch" forName="whiteRect"
                refType="w" fact="1" />
        <constr type="h" for="ch" forName="whiteRect"
                refType="h" fact=".025" />
        <constr type="l" for="ch" forName="whiteRect" val="0" />
        <constr type="t" for="ch" forName="whiteRect"
                refType="h" fact=".175" />

        <!--constraints for childText -->
        <constr type="w" for="ch" forName="childText"
                refType="w" fact="1" />
        <constr type="h" for="ch" forName="childText"
                refType="h" fact=".8" />
        <constr type="l" for="ch" forName="childText" val="0" />
        <constr type="t" for="ch" forName="childText"
                refType="h" fact=".2" />
    </constrLst>
    <ruleLst />
    <layoutNode name="roundRect" styleLbl="node1" >
        ...
    </layoutNode>
    <layoutNode name="parentText" styleLbl="node1" >
        ...
    </layoutNode>
    <layoutNode name="whiteRect" styleLbl="fgAcc1" >
        ...
    </layoutNode>
    <layoutNode name="childText" styleLbl="node1" >
        ...
    </layoutNode>
</layoutNode>

Constraints on roundRect and whiteRect Layout Nodes The roundRect and whiteRect shapes do not contain text and do not have specific resizing behavior. Their width, height, and positions are specified in the composite node's constraint block, so no additional constraints or rules are needed. The empty constraint and rule list element tags are included here for completeness, but these are optional (see Figure 11).

Constraints on parentText and childText Layout Nodes The height, width, and position of the parentText shape are specified in the composite node's constraints, so the only additional constraints and rules for this layout node are those concerning text properties. For most text in SmartArt shapes, the font size is controlled by the primary font size (primFontSz) constraint. In our example, this constraint is specified at the root layout node to start at a value of 65 points.

For most graphics, 65 points will be too large. So you can add a rule that allows the font size to shrink to a minimum of 5 points:

<layoutNode name="parentText" styleLbl="node1" >
    <alg type="tx" />
    <presOf axis="self" />
    <shape type="rect" hideGeom="true" />
    <constrLst />
    <ruleLst>
        <rule type="primFontSz" val="5" />
    </ruleLst>
</layoutNode>

If the text still doesn't fit at 5 points, it will extend beyond outside of the shape.

The childText layout node also has constraints and rules related to font sizing. In addition, to get the text to align properly, text parameters are needed as elements nested inside the algorithm element, as shown in Figure 12.

Figure 12 Text Parameters Inside the Algorithm Element (Close Figure 12)
<layoutNode name="childText" styleLbl="node1" >
    <alg type="tx">
        <param type="stBulletLvl" val="1" />
        <param type="parTxLTRAlign" val="l" />
        <param type="parTxRTLAlign" val="r" />
        <param type="txAnchorVert" val="t" />
    </alg>
    <presOf axis="des" ptType="node" />
    <shape type="rect" hideGeom="true" />
    <constrLst>
        <constr type="secFontSz" refType="primFontSz" />
    </constrLst>
    <ruleLst>
        <rule type="primFontSz" val="5" />
    </ruleLst>
</layoutNode>

In this layout node, the graphic displays a bulleted list of text, instead of a single text item. We want the bullets to start with the first level of text that displays in this layout node, but the default behavior is to start bullets at the second level inside a shape. The stBulletLvl (start bullet level) parameter controls this behavior. For the graphic here, the value should be 1 instead of the default value of 2. Note that these are the only two values supported for this parameter.

Because this text is a bulleted list, it should be left aligned and top aligned. For the first level of text within a shape, the default is to center the text vertically and horizontally, so these behaviors are changed using the parameters parTxLTRAlign (parent text LTR align) and txAnchorVert (text anchor vertical), setting them to l (left) and t (top), respectively.

Of course, if this graphic is used with language settings that display text from right to left, the text should align to the right. There are two parent text alignment parameters that determine the value to use according to system settings. So for this graphic, you should also include parTxRTLAlign (parent text right-to-left align) and set this value to r (right).

All shapes also support a secondary font size (secFontSz). These values correspond to where bullets begin in a shape. If a line has a bullet, it uses the secondary font size. If it doesn't have a bullet, it uses the primary font size. By default, the secondary font size is 78 percent of the primary font size.

Each shape also has margin values, and these values are, by default, proportional to the primary font size. If a shape only has secondary text in it, it makes more sense to have the margins reference the secondary font size. For the sample graphic, it's not necessary to have the secondary text smaller than the primary text, so you simply set the two values equal, which also ensures that the margins pick up the appropriate values.

Space Layout Node Constraints The space algorithm is used either as a placeholder to indicate that no sizing is needed or to preserve some minimum amount of space between other layout nodes. For this layout node, there should be a horizontal space between the composite shapes; therefore, you set a width constraint for the layout node. However, this value was defined on the diagram layout node, so the constraint and rule lists are empty for the actual space layout node as they are for the layout nodes shown in Figure 11. Now let's delve into packaging the layout file.

Figure 11 Empty Constraint and Rule List Element (Close Figure 11)
<layoutNode name="roundRect" styleLbl="node1" >
    <alg type="sp" />
    <presOf axis="self" />
    <shape type="roundRect">
        <adjLst>
            <adj idx="1" val="0.1" />
        </adjLst>
    </shape>
    <constrLst />
    <ruleLst />
</layoutNode>

<layoutNode name="whiteRect" styleLbl="fgAcc1" >
    <alg type="sp" />
    <presOf />
    <shape type="rect" />
    <constrLst />
    <ruleLst />
</layoutNode>

Packaging the Layout Definition File

The XML I've discussed is part of an overall Open Document package. This is essentially a compressed ZIP file with the extension .glox and the following folder and file structure:

  • _rels
  • .rels
  • diagrams
  • layout1.xml
  • layout1header.xml
  • [Content_Types].xml

Let's take a closer look at what all these files and folders are.

The rels Folder and File The .rels file defines the relationships between the parts in the .glox file format. It is a text file that contains the following XML:

<?xml version="1.0" encoding="utf-8"?>
<Relationships xmlns=
        "http://schemas.openxmlformats.org/package/2006/relationships">
    <Relationship
        Type="http://schemas.openxmlformats.org/officeDocument/2006/
             relationships/diagramLayoutHeader"
        Target="/diagrams/layoutHeader1.xml" Id="rId1" />
    <Relationship
        Type="http://schemas.openxmlformats.org/officeDocument/2006/
             relationships/diagramLayout"
        Target="/diagrams/layout1.xml" Id="rId2" />
</Relationships>

The Diagrams Folder and Layout Definition Parts The diagrams folder contains the two XML layout definition files that comprise the SmartArt graphic layout. Layout1.xml is the main definition file, and contains the XML described in this article, which documents the layout and mapping for the graphic.

Layout1header.xml contains the header information for the layout definition, including the unique ID, title, and description:

<?xml version="1.0" encoding="utf-8"?>
<layoutDefHdr uniqueId="msdn/sampleGraphicLayout"
      xmlns="http://schemas.openxmlformats.org/drawingml/2006/diagram">
    <title val="MSDN Sample Graphic Layout" />
    <desc val=" " />
    <catLst>
        <cat type="list" pri="500" />
    </catLst>
</layoutDefHdr>

Each graphic layout requires a uniqueID. If another layout definition has the same uniqueID, the file will not load.

The title and description appear in the user interface. You can use this area to provide specific information about when and how to use the graphic layout.

The category and priority information determine where in the SmartArt layout gallery the graphic layout appears. SmartArt graphics supports the following category types: list, process, cycle, relationship, pyramid, matrix, and other. The built-in layout files begin with priority 1000 in each category and increment by 1000. You can use any positive integer as a priority.

The [Content_Types].xml File The content types file sets up the structure and namespace area for the entire package. It should be placed at the root level and contain the following XML:

<?xml version="1.0" encoding="utf-8"?>
<Types xmlns="http://schemas.openxmlformats.org/package/2006/
              content-types">
    <Default Extension="xml" ContentType="application/
        vnd.openxmlformats-officedocument.drawingml.
        diagramLayoutHeader+xml" />
    <Default Extension="rels" ContentType="application/
        vnd.openxmlformats-package.relationships+xml" />
    <Override PartName="/diagrams/layout1.xml"
        ContentType="application/vnd.openxmlformats-
            officedocument.drawingml.diagramLayout+xml" />
</Types>

Testing the Graphic Layout

Once your layout definition is packaged into a .glox file, it must be placed in the correct directory for it to be included in the SmartArt layout gallery. By default, SmartArt graphic layout files are stored in the local settings template directory, at the following location: %APPDATA%\Microsoft\templates\SmartArt Graphics.

You can also set a registry key to change the templates directory to another location. But keep in mind that the registry key applies to all Office templates, including the location of the normal.dotx file in Word. This registry key is stored under HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Common\General as UserTemplates as a string. As the value, specify a new path for the general templates folder and the SmartArt graphic feature will then look for a SmartArt graphics folder under the specified path.

After you've placed the graphic layout file in the templates directory, launch an application that supports SmartArt graphics and click the Insert SmartArt button. If there are no immediate errors, the graphic layout you created will be shown in the gallery under the category you specified.

If, by chance, you need to correct any errors in any of the XML files, you'll need to restart the Office application session for the modified XML file to be reloaded into the gallery.

Error Types and the Error Log

SmartArt graphics provide an XML log file that tracks errors and warnings associated with the layout definition files. You can use the log file to help determine what needs to be fixed in your graphic layout. Errors with layout files generally fall into three categories: non-unique uniqueID values, XML or schema errors, and layout validation errors.

When a uniqueID isn't actually unique, one of the layout files with the duplicate uniqueID will fail to load and an error message will display when the SmartArt layout gallery is opened. If this happens, an entry similar to the following will be added to the log file:

<entry>
    <time>2006-10-20T15:51:14.650</time>
    <sev>err</sev>
    <host>POWERPNT.EXE</host>
    <file>example.glox</file>
    <type>nonUniqueId</type>
    <desc>A user-defined or built-in definition with this
          uniqueId was already loaded.</desc>
    <context>example</context>
</entry>

XML formatting or schema validation errors also prevent the layout definition from loading properly. If the error occurs in the layoutHeader1.xml file, no part of the file will load. However, if the error occurs in the layout1.xml file, the file will load, but a red X will be displayed in the gallery. And if you attempt to select this layout from the gallery, an error message will be displayed. The following log entry shows a typical XML formatting error:

<entry>
    <time>2006-10-20T15:53:39.619</time>
    <sev>err</sev>
    <host>POWERPNT.EXE</host>
    <file>example.glox</file>
    <type>xmlError</type>
    <desc>No error detail available</desc>
    <context></context>
    <line>58</line>
    <col>35</col>
</entry>

The most valuable piece of this error is the indication of the line and column where the error occurs.

Layout validation errors behave like XML errors, but they only occur when the SmartArt graphic attempts to run the layout. The following log entry shows an error that occurs when two layout nodes have the same name attribute:

<entry>
    <time>2006-10-20T15:59:25.650</time>
    <sev>err</sev>
    <host>POWERPNT.EXE</host>
    <file>example.glox</file>
    <type>invalidName</type>
    <desc>The name attribute must be unique.</desc>
    <context>&lt;layoutNode name='sibTrans'/&gt;</context>
    <line>131</line>
    <col>17</col>
</entry>

SmartArt graphics reference three registry keys (which are stored under HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Common\SmartArt Graphics) for controlling the log file: LogFileSeverityLevel, LogFileMaxSize, and LogFileNumEntriesToRemove.

LogFileSeverityLevel is a DWORD that sets the severity level of errors or warnings to report. 0 reports only errors, while levels 1 through 4 report different levels of warnings, with 1 being the most severe. LogFileMaxSize is a DWORD that sets the maximum number of bytes that can be stored in the log file. If this value is reached, a set of entries is removed from the log to provide additional space. And LogFileNumEntriesToRemove is a DWORD that specifies the number of entries to remove when the maximum log file size is exceeded.

Valid File with Design Errors

Once the graphic layout file loads successfully, you need to check for design errors. Some common errors are:

  • Text that appears in wrong shape.
  • Text that doesn't appear at all.
  • Shape size or font size that's not consistent across shapes.
  • Shapes that aren't positioned properly.
  • Graphics that don't react as you would expect when shapes are added.

Correcting these problems is really a matter of trial and error. As you gain more experience with creating definition files, you'll find the errors easier to identify and correct.

When testing your graphic layout, you should check the way it looks with various SmartArt graphic styles and colors and across different themes, to make sure it looks good in most scenarios. But keep in mind that not every graphic will look good with every style and color.

Modifying Existing Graphic Layout Definitions

Creating a graphic layout from scratch provides a good way to understand the structure and syntax for SmartArt graphic layout files. However, you may simply want to modify an existing graphic layout to get the look you want.

When you add a SmartArt graphic to a document and save it, a copy of the graphic layout file is stored with the other parts of the document as well. If you look inside the package and navigate to the diagrams directory, you can find layout and layout header XML files corresponding to each SmartArt graphic layout in the document. You can remove this file from the package, rename it layout1.xml, modify the appropriate sections, and then place the file into a new .glox.

If you're modifying an existing file, you may notice additional tags and attributes not discussed in this article or perhaps that other tags and attributes are missing. In general, the new attributes are optional or are attributes for which I've assumed default values. The missing attributes are those that use default values.

What's Next?

I've covered the basics of SmartArt graphic layout extensibility, but I've barely touched upon the many complexities of creating SmartArt graphic layout files. If you want to explore further, I'd recommend starting with the schema and looking through existing graphic layout definitions to see what's possible.n

Janet Schorr has been a program manager at Microsoft for six years and is currently a lead program manager on the Office Publisher team. She has been on a variety of product teams and recently worked with the Office Graphics team on the SmartArt layout architecture.


Office Add-Ins: Develop Add-Ins For PowerPoint And Visio Using VSTO

Office Add-Ins

Develop Add-Ins For PowerPoint And Visio Using VSTO

Paul Stubbs
This article discusses:
  • Extensibility for Microsoft Office applications
  • PowerPoint object model
  • Visio object model
This article uses the following technologies:
Microsoft Office
Contents
PowerPoint 2007
PowerPoint 2007 Object Model
Create a Presentation
Add a Title Slide
Add a Bulleted Slide
Set the Presentation Theme
Create a Ribbon
Visio 2007
Visio 2007 Object Model
Create New Document
Adding Shapes
Connecting Shapes
Adding a Background Page
Set the Document Theme
Create a Commandbar
Summary

M
icrosoft Visual Studio 2005 Tools for the Microsoft Office system (VSTO) is a powerful set of tools and features that let developers extend and customize Microsoft® Office applications using Visual Basic® and Visual C#®. With the release of the 2007 Microsoft Office system comes a new version of VSTO. This new version, Microsoft Visual Studio® 2005 Tools for the 2007 Microsoft Office system (known as Visual Studio 2005 Tools for Office Second Edition, or VSTO 2005 SE), adds new capabilities for Word, Excel®, Outlook®, PowerPoint®, Visio®, and InfoPath®. Available as a free download it can be installed on top of Visual Studio 2005 Professional Edition or above and on top of VSTO 2005.

With VSTO 2005 SE, you customize 2007 Microsoft Office system features (including the Ribbon, custom task panes, and Outlook form regions) in your add-ins. The VSTO 2005 SE runtime also adds support for your existing VSTO 2005 applications in the 2007 Office system. And it can be used to create Office 2003 add-ins.

This new release is the first that has included support for PowerPoint and Visio. Let's look at the added support for these two applications, looking at how you can easily create great new apps.

PowerPoint 2007

Support for PowerPoint was one of the most requested additions to VSTO 2005 SE. There are many tasks, such as generating and updating presentations, that can be made easier using VSTO add-ins. A typical example of presentation generation is creating a new presentation by stitching together existing slides from a slide library stored on a server. Other common scenarios include creating new slides programmatically and updating existing slides with current data. I want to walk you through some of the common tasks associated with creating and updating slides. Note that this discussion pertains primarily to the 2007 Office system.

When you install VSTO 2005 SE, it adds two folders under the Office node in the Visual Studio New Project dialog (see Figure 1): one for creating add-ins that target Office 2003 and one for add-ins that target the 2007 Office system apps. Let's start creating a new PowerPoint add-in by selecting the PowerPoint Add-in template shown in the 2007 Add-ins folder.

Figure 1 VSTO Templates
Figure 1 VSTO Templates(Click the image for a smaller view)
Figure 1 VSTO Templates
Figure 1 VSTO Templates (Click the image for a larger view)

The PowerPoint project template creates a class called ThisAddIn. You will see that there are two events: ThisAddIn_Startup and ThisAddIn_Shutdown. ThisAddIn_Startup is called when the add-in is loaded. (This is a good place to put your initialization code.) ThisAddIn_Shutdown is called when the add-in is being unloaded. (This is where you should put your cleanup code.)

PowerPoint 2007 Object Model

PowerPoint has a rich object model you can use to create add-ins. And you do not need to understand the entire object model to get started.

As in all Office applications, the top-level object for PowerPoint is the Application object, which represents the entire PowerPoint application. The VSTO programming model gives you easy access to this Application object. Typing Me.Application returns a PowerPoint.Application type. Unlike shared add-ins, there is no need to cast the object to the correct Office Application type because the VSTO project template sets the correct type for the add-in you are creating. (For example, calling Me.Application in a Word add-in returns a Microsoft.Office.Interop.Word.Application object.)

The Application object contains a Presentations collection that, in turn, contains a Presentation object. The Presentation object, which represents the actual .ppt document, contains a Slides collection that holds the Slide objects (there's one Slide object for each slide in your presentation). And each slide contains a Shapes collection, which holds Shape objects. The Shape objects are the items that you see on your slides, such as the title, text frames, pictures, and charts (see Figure 2).

Figure 2 Shape Objects on a PowerPoint Slide
Figure 2 Shape Objects on a PowerPoint Slide(Click the image for a smaller view)
Figure 2 Shape Objects on a PowerPoint Slide
Figure 2 Shape Objects on a PowerPoint Slide (Click the image for a larger view)

Create a Presentation

You use a reference to the Application object to create a new Presentation and add it to the Presentations collection. First, create a variable to hold a reference to the new Presentation object that you will create. Then all you need to do is call the Add method on the Presentations collection:

   Dim presentation As PowerPoint.Presentation = _
       Me.Application.Presentations.Add()

You may be wondering where the PowerPoint object comes from. The VSTO project template automatically adds an Imports alias to the namespace Microsoft.Office.Interop.PowerPoint. You can see the other references and aliases that are created by looking at the Imported namespaces section on the References project properties page.

Add a Title Slide

The title slide, which displays the title and subtitle of the presentation, is usually the first slide in a presentation. Creating a new slide is similar to creating a new presentation. You add a slide to the slides collection of the presentation by calling the AddSlide method. This method takes two parameters: the slide index (used to specify where in the presentation the slide should be inserted) and a customLayout object.

You can create a customLayout object from the SlideMaster. The SlideMaster contains all of the objects that you want to appear on your slides. As you can see in this code, you can select the type of layout from the PpSlideLayout enumeration:

Dim customLayout As PowerPoint.CustomLayout = _
    presentation.SlideMaster.CustomLayouts.Item( _
        PowerPoint.PpSlideLayout.ppLayoutTitle)

You can now create a new slide using the AddSlide method. Use a slide index of 1 to add the slide as the first slide in the presentation, and pass the customLayout object that you created from the SlideMaster:

Dim slide as PowerPoint.Slide = _
    presentation.Slides.AddSlide(1, customLayout)

The title slide is now created and you can set the title and subtitle with a single line of code. Using the slide object that you just created you can get the Title property of the Shapes collection to return a Shape object that is the title object for the slide. The Shape object has a TextFrame property that returns a TextFrame object, which exposes a TextRange object. You set the Text property of the TextRange object, like so:

slide.Shapes.Title.TextFrame.TextRange.Text = "VSTO 2005 SE Rocks"

Setting the subtitle is very similar except that there is no subtitle shape property. In this case, I know that the subtitle is the second shape on the slide so I can reference it by the shape index:

slide.Shapes(2).TextFrame.TextRange.Text = _
    "AN OFFICE DEVELOPMENT STORY"

Add a Bulleted Slide

Next, you will want to add another slide that, typically, has a title at the top and a bulleted list of items as the main content. This is the most commonly used slide type. To do this, create a new slide as you did in the previous section, but choose a different CustomLayout type: ppLayoutText. I used 1 as the slide index for the title slide since I want the title slide to appear as the first slide. To insert slides in order (after the last existing slide), you should count the number of existing slides:

Dim customlayout As PowerPoint.CustomLayout = _
    presentation.SlideMaster.CustomLayouts.Item( _
        PowerPoint.PpSlideLayout.ppLayoutText)
Dim slide as PowerPoint.Slide = _
    presentation.Slides.AddSlide( _
        presentation.Slides.Count + 1, customlayout)

Set the Presentation Theme

A new feature in the 2007 Office system, Office themes, lets you apply predefined styles to your documents. A theme will globally change the look of your presentation, including the fonts, graphics, colors, and effects. Themes are supported by Word, Excel, PowerPoint, and Outlook. They are defined in a Theme file with the .thmx extension and are shared across all of the supported applications. The 2007Office release ships with a few default themes located in the Document Themes 12 directory under the Office installation path.

You set the theme by calling a new method of the presentation object: ApplyTheme. This method takes one parameter-the path to the .thmx theme file:

presentation.ApplyTemplate( _
    "C:\Program Files\Microsoft Office\Document Themes 12\Civic.thmx")

Create a Ribbon

The Ribbon is central to the new Office UI. Replacing the menus and command bars of previous versions of Office, the Ribbon organizes features and functions into context sensitive tabs. The tabs contain groupings of controls, such as buttons, galleries, and dropdown lists. For this project, you'll create a simple Ribbon with a single button that creates a new presentation when clicked. VSTO 2005 SE supports Ribbon extensions and makes it easy to add new custom Ribbon tabs and controls to your applications.

First, add a Ribbon support item to your project from the Project | Add New Item menu. This creates two files in your project: one for the Ribbon definition and one for the Ribbon codebehind. The Ribbon definition file is an XML file that describes the Ribbon. The Ribbon codebehind handles the loading of and the events of the Ribbon. Figure 3 shows a simple Ribbon definition XML file that adds a Ribbon tab called My Tab, which contains a single group called My Group. The group contains only one button, called My Button. The button has a callback specified for the onAction called OnClick.

Figure 3 Simple Ribbon Definition XML File (Close Figure 3)
<customUI xmlns="http://schemas.microsoft.com/office/2006/01/customui"
     onLoad="OnLoad">
  <ribbon>
    <tabs>
      <tab id="MyTab" label="My Tab">
        <group id="MyGroup" label="My Group">
          <button id="Button1" size="large" label="My Button"
                  screentip="My Button Screentip" onAction="OnClick"
                  imageMso="HappyFace" />
        </group>
      </tab>
    </tabs>
  </ribbon>
</customUI>

In the Ribbon definition that is automatically created, a callback is defined for the onLoad of the Ribbon and the onAction for the button. You must edit the methods generated in the Ribbon1 class to handle these callbacks. The Ribbon1 class was created when you added the Ribbon support item. The Ribbon1 class implements the IRibbonExtensibility interface that is passed to Office when requested.

The Office Ribbon uses a service request model, meaning Office calls the RequestService method and asks whether you have a class that implements the IRibbonExtensibility interface. After you pass back an instance of the Ribbon1 class, Office uses this instance for the callbacks.

The code to handle the RequestService method is already written when you added the Ribbon support item-simply uncomment the partial class code at the top of the Ribbon1 file. Then write the methods to handle the callbacks; you can find the callback signature documentation for all of the Ribbon callbacks on MSDN®. In the OnLoad callback, Office passes a reference to the Ribbon instance. You can save this reference for when you need to invalidate the Ribbon.

Invalidating the Ribbon is the only way to refresh it. When you invalidate the Ribbon, Office refires all of the callbacks for the Ribbon needed for rendering. You can also invalidate a single control for more granular handling:

Private ribbon As Office.IRibbonUI
Public Sub OnLoad(ByVal ribbonUI As Office.IRibbonUI)
    Me.ribbon = ribbonUI
End Sub

In the OnClick callback for the button, use the Globals class to get a reference to the ThisAddIn class. And call the CreateNewPresentation subroutine to create a new presentation and perform all steps:

Public Sub OnClick(ByVal control As Office.IRibbonControl)
    Globals.ThisAddIn.CreateNewPresentation()
End Sub

Running the solution by pressing F5 registers the add-in and starts PowerPoint. The new Ribbon then appears as the last Ribbon tab on the right, called My Tab.

For more information on developing with the Ribbon, see Eric Faller's article on the subject in this issue of MSDN Magazine.

Visio 2007

Visio has long supported custom development, with support for Visual Basic for Applications and the availability of its own SDK. VSTO 2005 SE extends this capability with the ability to create managed VSTO add-ins.

Visio solutions help you visualize your data. For example, you could diagram a timeline of your schedule using data from a database, or you could create an organization chart with live data from Active Directory overlaid with sales data for each employee. Let's look at some basic tasks in Visio that can lay the foundation for larger applications.

As with all VSTO add-ins, you begin by creating a new VSTO add-in project. Again, you can choose to target either Visio 2003 or Visio 2007; I will be using Visio 2007. VSTO creates an add-in with the ThisAddIn_Startup and ThisAddIn_Shutdown event handlers, just like when creating the PowerPoint add-in.

Visio 2007 Object Model

The object model in Visio, which is quite different from the other Office applications, can be challenging to understand. Fortunately, you don't need to understand the entire object model for this project-a few basic concepts will get you started.

As with all Office applications, the root of the object model is the Application object, which contains a collection of Document objects. The Document is the object that represents the drawing, and it contains a collection of Page objects. Each Page object, in turn, contains a collection of Shape objects. These Shape objects are the key to creating Visio applications.

Shapes are the objects that you drop on the page from stencils. Stencils (the collections of Visio shapes that you can add to your drawings) are contained in a .vss file and are visible in a container on the left side of your Visio drawing.

Create New Document

Create a new document by calling the Add method on the Document collection:

Dim newDocument As Visio.Document = Application.Documents.Add("")

This will return an instance of a Visio.Document object.

You may also want to set some other properties when you create the document. For instance, you can set the layout to portrait or landscape, as well as set the height and width of the document. This is common for Visio documents since they can be very large and are often printed on plotters or across many letter size pages.

The page has a special shape called a PageSheet, which is a collection of properties for the page. This is an important concept to understand because this is how all Visio properties are set. Properties are stored in a Cell object. Once you have a reference to the Cell, you can set the Formula, which is evaluated by Visio to obtain the final value:

Dim page As Visio.Shape = newDocument.Pages(1).PageSheet
' Set the layout.
page.Cells("PrintPageOrientation").Formula = _
    Visio.VisCellVals.visPPOPortrait
' Set the paper size.
page.Cells("PageWidth").Formula = "8.5 in"
page.Cells("PageHeight").Formula = "11.0 in"

Adding Shapes

Before adding shapes to the document, you need to open a stencil. In this case, you'll open the basic stencil. Pass the name of the stencil file and flags-for example, open read-only and docked to the application window:

Stencil = Application.Documents.OpenEx( "BASIC_M.VSS", _
    Visio.VisOpenSaveArgs.visOpenRO + _
    Visio.VisOpenSaveArgs.visOpenDocked)

Now call the Drop method of Page object to add a shape to the page, passing a Master object (which comes from the Masters collection of the stencil) and an X and Y position of where you want the shape placed:

Dim AddedShape As Visio.Shape = _
    Application.ActiveWindow.Page.Drop( _
        Stencil.Masters.ItemU("Square"), 2, 9)

You can then customize the shape, adding text, for example, that is displayed in the shape. Just as you did with the properties of the document, you set the properties of the shape by getting a reference to the Cell object that contains the property. The Text property, however, is so common that it was added directly to the Shape object, and therefore you don't need to use the Cell object to set the text:

AddedShape.Text = "Shape 1"

You will add shapes many times, so I highly recommend refactoring this code into a separate method:

Public Function AddShape(ByVal ShapeName As String, _
    ByVal X As Double, ByVal Y As Double) As Visio.Shape

    ' Add a new shape to the document from the stencil.
    Dim AddedShape As Visio.Shape = _
        Application.ActiveWindow.Page.Drop( _
            Stencil.Masters.ItemU(ShapeName), X, Y)

    Return AddedShape
End Function

Now it is easy to add multiple shapes. Here's a sample for adding a square and a triangle, and setting the text of each shape:

' Add a square.
shape1 = AddShape("Square", 2, 9)
shape1.Text = "Shape 1"
' Add a Triangle.
shape2 = AddShape("Triangle", 2, 6)
shape2.Text = "Shape 2"

For Visio drawings, 0,0 represents the bottom-left of the page. Therefore, in this sample, the square is placed two inches from the left and nine inches from the bottom.

Connecting Shapes

In Visio, you can visualize relationships and data flow by connecting shapes. The lines connecting shapes are actually shapes themselves. Thus, you add these connecting lines using the same pattern you use to add shapes:

Dim connector As Visio.Shape = AddShape("Dynamic Connector", 4, 4)
connector.Text = "Shape1 to Shape2"

To connect the triangle to the square (which is located above the triangle), first define two Cell objects to hold a reference to the properties of the connection points. You use ShapeConnectionPoint (the point on the shape) and ConnectorEndPoint (the point on the connection line):

Dim ShapeConnectionPoint As Visio.Cell
Dim ConnectorEndPoint As Visio.Cell

Shape2 is the triangle you created earlier and the property for the top point on the triangle is called Connections.X4. Connector is the dynamic connector shape and EndX is the property name for the bottom end of the line:

ShapeConnectionPoint = shape2.Cells("Connections.X4")
ConnectorEndPoint = connector.Cells("EndX")

Then you glue the cells together to connect the line to the shape:

ConnectorEndPoint.GlueTo(ShapeConnectionPoint)

The triangle is easy to connect since it has three connection points. The square has four connection points (one on each corner). You can see these points when you drop a shape onto the page-they appear as little blue Xs. You can only glue to connection points. Thus, if you want to connect to a different part of the shape (say, the bottom line of the square), you must add a new connection point. Figure 4 shows a helper method to assist with this.

Figure 4 AddConnectionPoint Helper Method (Close Figure 4)
Public Function AddConnectionPoint( _
        ByVal shape As Visio.Shape) As Visio.Cell

    Dim newConnectionPointIndex As Integer
    Dim newConnectionPoint As Visio.Row
    newConnectionPointIndex = shape.AddRow( _
        Visio.VisSectionIndices.visSectionConnectionPts, _
        Visio.VisRowIndices.visRowLast, _
        Visio.VisRowTags.visTagCnnctPt)
    'Get a reference to the new connection point.
    newConnectionPoint = shape.Section( _
        Visio.VisSectionIndices.visSectionConnectionPts) _
        .Row(newConnectionPointIndex)
    'Create the connection point in the middle on the bottom of the shape.
    With newConnectionPoint
        .Cell(Visio.tagVisCellIndices.visCnnctX).FormulaU = "Width*0.5"
        .Cell(Visio.tagVisCellIndices.visCnnctY).FormulaU = "Height*0"
        .Cell(Visio.tagVisCellIndices.visCnnctDirX).FormulaU = 0.0#
        .Cell(Visio.tagVisCellIndices.visCnnctDirY).FormulaU = 1.0#
        .Cell(Visio.tagVisCellIndices.visCnnctType).FormulaU = _
            Visio.tagVisCellVals.visCnnctTypeInward
    End With

    Return shape1.Cells("Connections.X" & newConnectionPointIndex + 1)
End Function

Now, to connect a straight line down from the square to the top of the triangle (as shown in Figure 5), you add a new connection point to the bottom center line of the square with the following:

ShapeConnectionPoint = AddConnectionPoint(shape1)
ConnectorEndPoint = connector.Cells("BeginX")
ConnectorEndPoint.GlueTo(ShapeConnectionPoint)

Figure 5 Creating a Document in Visio 2007
Figure 5 Creating a Document in Visio 2007(Click the image for a smaller view)
Figure 5 Creating a Document in Visio 2007
Figure 5 Creating a Document in Visio 2007 (Click the image for a larger view)

The AddConnectionPoint helper method adds a new connection point and returns the Cell object. You add a new connection point by adding a new property, or row, as Visio stores the properties internally as a list. Once you have added the row, you can set the formula values. So far, most of the formulas have consisted of simple values. In this case, the formula dynamically calculates the center of the bottom line with Width*0.5. Finally the method returns the Cell instance for the newly created connection point.

Adding a Background Page

In Visio, the background page lets you add text or shapes (a title and legend, for instance) that you want to appear behind multiple pages. The background page is a normal page that has the Background property set to 1 for true. You associate the background page with another page by setting the BackPage property:

Dim backgroundPage As Visio.Page = _
    Application.ActiveDocument.Pages.Add()
backgroundPage.Name = "TheBackground"

backgroundPage.Background = 1 'True

Application.ActiveDocument.Pages(1).BackPage = backgroundPage.Name

To add text to the background page, you create a shape object from a rectangle, set the font property of the shape by passing the font index number, and set the text of the Characters object to set the text that will appear in the shape. This is shown in Figure 6.

Figure 6 Adding Text to a Background Page (Close Figure 6)
' Create a textbox shape.
Dim textBox As Visio.Shape
textBox = Application.ActiveWindow.Page.DrawRectangle( _
    1.0, 10.0, 3.0, 10.5)
textBox.TextStyle = "Normal"
textBox.LineStyle = "Text Only"
textBox.FillStyle = "Text Only"
' Set the font.
fontIndex = Application.ActiveDocument.Fonts("Arial Black").ID
textBox.Cells("Char.Font").Formula = fontIndex
' Set the text.
Dim textBoxText As Visio.Characters
textBoxText = textBox.Characters
textBoxText.Begin = 0
textBoxText.End = 0
textBoxText.Text = "VSTO 2005 SE Rocks"

Set the Document Theme

You can set predefined themes in Visio just as you can in PowerPoint. In fact, you can set the same theme across most of the Office applications. One difference in Visio, though, is that you can specify the theme using the VisThemeColors enumeration. There are multiple ways to specify the value. You can, for example, specify the value as a string or as an integer. Another difference with Visio, is that you specify the theme colors and theme effects separately. From the ActivePage, you can control these with the Theme Colors and Theme Effects properties, respectively:

Application.ActivePage.ThemeColors = _
    Visio.VisThemeColors.visThemeColorsCivic

Application.ActivePage.ThemeEffects = _
    Visio.VisThemeEffects.visThemeEffectsPillow

Create a Commandbar

Obviously, you need a way to launch the subroutine that will create the drawing. Visio doesn't support Ribbons, so for this add-in you use a Commandbar. (This code has not changed from Visio 2003 and other Office 2003 applications.) Figure 7 shows the code used to create a Commandbar and add a button control with a smiley face icon. You can see this button in Figure 5. The final result is a square connected to a triangle with a theme applied, as shown in Figure 5.

Figure 7 Creating a Commandbar (Close Figure 7)
Dim commandBars As Office.CommandBars
Dim commandBar As Office.CommandBar

commandBars = Application.CommandBars
commandBar = commandBars.Add("Demo Toolbar", _
    Office.MsoBarPosition.msoBarTop, , True)
' Set the context when the toolbar is visible.
commandBar.Context = Visio.VisUIObjSets.visUIObjSetDrawing & "*"

' Add a button with a smiley face icon.
DemoButton = _
    commandBar.Controls.Add(Office.MsoControlType.msoControlButton)
DemoButton.Tag = "Demo Button"
DemoButton.FaceId = 2950
DemoButton.TooltipText = "Run Demo"

Summary

Available as a free add-on to Visual Studio Professional or above, VSTO 2005 SE lets you build applications for the six most common Office applications. It also lets you ensure that your existing VSTO 2005 applications continue working as you move to the 2007 Office system. Whether you are using Office 2003 or the 2007 Office system, VSTO 2005 SE provides an easy way to create great Office add-ins.n

Paul Stubbs is a Program Manager with the Visual Studio Tools for Office and Visual Studio Tools for Applications team. He coauthored VSTO for Mere Mortals with Kathleen McGrath (Addison-Wesley, 2006). He has spoken at TechEd and TechReady and participates in the developer community on the Microsoft forums. Read Paul's blog at blogs.msdn.com/pstubbs.


Data Joins The Team: Introducing Visual Studio 2005 Team Edition for Database Professionals

Data Joins The Team

Introducing Visual Studio 2005 Team Edition for Database Professionals

Brian A. Randell
This article discusses:
  • Source control and change management for databases
  • Working with new and existing databases
  • Unit testing stored procedures, functions, and triggers
  • Comparing databases
This article uses the following technologies:
Visual Studio 2005 Team System
Contents
Managing Database Versions
Getting Started
The Import Process
Editing Schema and Objects
Data Generation
Unit Testing
Comparing Databases
Build and Deployment
Conclusion

D
atabase development has gotten easier, thanks to Visual Studio® 2005 Team Edition for Database Professionals (or DB Pro edition, for short). If you need to create, manage, or build solutions that connect to SQL Server™, this tool is for you.

DB Pro edition provides a number of features to make managing databases easier. As a member of Visual Studio 2005 Team Suite, DB Pro edition works in conjunction with Team Foundation Server's source code control feature to manage database changes just as you manage your source code. DB Pro edition supports sandbox development of databases, and it includes tools for database comparison, data generation, unit testing, and database deployment. But what is most compelling about DB Pro edition is using all of its features together as part of a holistic, database development life cycle.

There are a few things that should be clarified before we dive into the article. First, the initial release of DB Pro edition will support SQL Server 2000 and SQL Server 2005 relational databases, but it will offer no Analysis Services support. Support for other Microsoft and non-Microsoft databases is planned for future versions via a provider-based model, but it is not available as of this writing. Second, using DB Pro edition in conjunction with Team Foundation Server enhances many of the DB Pro edition features; however, Team Foundation Server is not a prerequisite for using DB Pro edition.

Managing Database Versions

To build a successful database-driven application, it is essential that you know which copy of the database is the "true" version. The problem is that making changes and testing against a live system is fraught with peril. In fact, typically only the database administrator (DBA) has the power and authority to do such things. DB Pro edition offers a solution. The production database system maintains the true, live version of the data. Meanwhile, DB Pro edition maintains the true version of the schema via a project-based system that, ideally, should be under version control. DB Pro edition manages each database object as a set of .sql files; it also understands the dependencies and relationships between these objects.

This separation of data and schema, from a change management perspective, is a big deal. Now, you can have a true iterative development process for your database-backed projects.

In addition, as a Team System family member, DB Pro edition integrates with Team Foundation Server and other products in the Team System family. This allows for true team development and management of your database schema. Arguably the most compelling feature is version control. Additional features-such as work item tracking, code check-in policies, and integration with nightly builds-make the database a regular artifact that's tracked and managed. You no longer need to treat it as a specialized entity that needs its own, potentially costly, management and development procedures.

Getting Started

At its core, DB Pro edition is a set of project templates and tools that live inside Visual Studio 2005. You can use some of the tools without generating a project; however, the real power starts with a DB Pro edition project.

You will need at least two installations of SQL Server: one that will function as your production system, and another as your development system. Depending on the complexity of your database and the software built on top of it, you may choose to have additional installations. For a team environment, you should have a local private instance for each database professional. In addition, you should have at least one shared development instance for integration and regression testing. (Remember, as part of your DB Pro edition license, you also receive a license for SQL Server 2005 Developer Edition, which has all the features of the Enterprise Edition.)

Getting started on your local developer machine requires that you have DB Pro edition installed with a local copy of SQL Server 2005 Developer Edition, Enterprise Edition, or even Express Edition. DB Pro edition uses a SQL Server 2005 instance for design-time validation of both SQL Server 2000 and SQL Server 2005 projects. To control the instance of SQL Server used for validation, go to the Visual Studio 2005 Tools | Options | Database Tools | Design-time Validation Database option. If you're working with Team Foundation Server, you'll also want Team Explorer installed.

While DB Pro edition will let you create a database from scratch, you most likely have existing databases you need to manage. There are two ways to get your database's schema loaded into a project. The first is to import a SQL script file. This is useful, for example, if you've been managing your schema changes by storing copies of the schema in version control, or when you can't connect to your current version of the schema directly and do a live import due to security or other business requirements.

The second way to load your schema is to connect to a running SQL Server instance and import the schema live. You'd use this option if you've already created a new project or as part of project creation via the New Project wizard. If you work in an environment where you have a DBA, this second option offers a good way to integrate the DBA into the development process. In fact, in a perfect world, when you start working with DB Pro edition against an existing database, the DBA should create the project, load it into source code control, and then hand it off to development, but all the while, the DBA should remain present during the development life cycle.

There is one important distinction between importing via a script versus importing via a live database. You can only import from a live database once, whereas you can import via scripts as often as needed.

The Import Process

To start the import process, open the New Project dialog box in DB Pro edition. From the top-level node called Database Projects (after a bit of node expansion), the templates for SQL Server are available. You can choose either a blank project or to use a wizard.

Running the wizard makes it easy to get started. The wizard's questions are broken down into pages labeled Project Properties, Set Database Options, Import Database Schema, and Configure Build/Deploy.

Project Properties allows you to specify how you want your project organized on disk-either by object type or by schema. It's important that you think about this up front because there's no easy way to change this option once you have created the project and imported the schema. Schema-based organization can be useful, for example, when you need to restrict access to parts of the database for security reasons.

You use the Set Database Options page to define database settings, such as ANSI Nulls, Quoted Identifies, and more. On the Import Database Schema page, you choose whether to import a database schema from a live server. If you do choose to import at this point, you then specify whether to use the database's options or the project's options.

Finally, on the Configure Build/Deploy page, you adjust options that affect build scripts placement, the default target database connection, and so on. These options, like most of those defined by the wizard, are adjustable via the Project Properties.

Once you have completed the wizard and imported the production database schema, you have successfully created your offline workspace. This is a place where you can safely work offline on the database schema.

DB Pro edition provides two views of the project. There's a literal file view exposed through the Solution Explorer (see Figure 1), and a hierarchical schema view (organized by object type or by schema and then object type) viewable through the Schema View window. These two views provide flexibility in project organization and management. You can completely rearrange the file structure on disk using drag-and-drop functionality in the Solution Explorer, yet maintain easy access to objects by object and/or schema type.

Figure 1 Database in Solution Explorer
Figure 1 Database in Solution Explorer(Click the image for a smaller view)
Figure 1 Database in Solution Explorer
Figure 1 Database in Solution Explorer (Click the image for a larger view)

DB Pro edition shreds the database schema into atomic objects represented as .sql files in the Solution Explorer and thus the file system (not just top-level objects like tables, views, and stored procedures, but indexes, constraints, and the like). In addition, the project gets a set of scripts segmented into a set of folders labeled Pre-Deployment and Post-Deployment. You can use these scripts to handle T-SQL operations that don't represent schema objects. Additionally, you can create your own management and utility scripts and organize them as you see fit.

At this point, if Team Foundation Server is in use, you add the project to Source Code Control and check off work items. Microsoft is updating the guidance provided with Team Foundation Server to include roles and work streams specific to database development for both the MSF and MSF CMMI templates.

Editing Schema and Objects

With the database under Source Code Control, making a change to an existing object isn't so overwhelming. Simply open it, start editing (which, by default, checks it out), and save your changes. From there you can check in your changes and move forward. On the other hand, you could use the shelving feature to post your changes to the server for review by another team member, such as your DBA, before checking it in.

The offline workspace encourages team collaboration. You also mitigate the inherent risk of making changes to the database schema with the ability to roll back. In addition, Team Foundation Server check-in policies enable teams to control workflow, making sure, for example, that all changes are associated with a work item, that all check-ins have comments, and so on.

The offline workspace provides a way to manage the database schema. However, it does not provide a runtime environment to actually create objects, run queries, or change data. You do this work against a live instance of SQL Server-your sandbox. Once you have imported the production schema, the next step is to publish the schema to your local developer instance of SQL Server. This requires two simple steps. First, you specify the target connection and new database name on the Build Tab of the Project designer. This option can be set via the import wizard if you used it to create your database project. Then you right-click on the database project in the Solution Explorer and select the Deploy option. Once complete, you are ready to start working on your database.

Designing database objects is a matter of adding a new item from one of three locations: the Project menu, the context menu in the Solution Explorer, or the context menu in Schema View. (One feature that's not making this first release is interactive design tools.) You create all of your database objects by entering the appropriate T-SQL statements. For example, to create a new table, you select the Add Item command from the Project menu, which opens the Add New Item dialog. You then select the Tables and Views option from the Categories section followed by the Table item in Templates section. Finally, you provide a name for the new table. Be aware that when you import a database from a live server, the names used for the .sql files on disk will follow the form of objectname.objecttype.sql. Keep this in mind when working out your standards for file names in your database projects. However, to be clear, the name you choose for your file is orthogonal to the object name used inside the database. Once you've provided a name and closed the Add New Item dialog, a T-SQL Editor window opens, providing a basic template from which to start.

The T-SQL Editor included with DB Pro edition is a full-featured editor and execution window. It supports color-coded editing, syntax validation, client execution statistics, and multiple result formats. This first release, however, doesn't support IntelliSense®. As you create your table, you can validate syntax by pressing Ctrl+F5. DB Pro edition will prompt you to make a connection the first time you do this.

Take care when working on Data Definition Language (DDL) statements in the T-SQL Editor. The familiar F5 keystroke will execute the T-SQL and perform the commands listed against whatever database you've chosen for syntax validation. Once the table design is complete, you can commit it to Source Code Control or shelve it for review.

Remember that when working in T-SQL, the objects defined inside DB Pro edition are very granular. You create indexes, constraints, and more individually. This can be quite a change depending on how you are used to building database objects. Say, for example, you wanted to add auction support to the Northwind sample database. Creating an auction table that has an identity column for its primary key would require that you first create the auction table, followed by a primary key constraint. Adding a foreign key constraint would also be its own step and require its own .sql file.

While you might find yourself adding new objects to a database, you will also find yourself making changes to existing objects. Some changes aren't so bad, like adding additional error handling to a stored procedure. Other changes feel like a more serious redesign, and words like refactoring start running through your brain. Changing object names, a major refactoring pain point with databases, is addressed by a refactoring tool provided by DB Pro edition.

To rename an object, you first need to find it using the Schema View window. Next, you right-click on the object and select Refactor | Rename. The first dialog to appear (see Figure 2) lets you specify the new name, as well as indicate if you want to preview changes, whether you wish to have a log generated, and how you want to handle unqualified script and unit test references. Clicking OK to continue brings up the Preview Changes dialog (see Figure 3) if you accept the default options.

Figure 2 The Rename Dialog
Figure 2 The Rename Dialog(Click the image for a smaller view)
Figure 2 The Rename Dialog
Figure 2 The Rename Dialog (Click the image for a larger view)
Figure 3 The Preview Changes Dialog
Figure 3 The Preview Changes Dialog(Click the image for a smaller view)
Figure 3 The Preview Changes Dialog
Figure 3 The Preview Changes Dialog (Click the image for a larger view)

A tree view displays every item that needs updating. Not only will it look for changes inside your database project, but DB Pro edition will also examine your database unit tests, your data generation files, and additional supporting .sql files.

Expanding an item displays each change, clicking on an item displays the new T-SQL, and clicking on a specific change highlights the new object name. Assuming the changes look good, you click the Apply button. Remember, this will only apply changes to your local workspace copy of the schema. If you're using Source Code Control, you will need to commit the changes. You will also need to deploy an update script to have the changes show up in an existing database-preferably your sandbox copy.

Data Generation

There's a chicken and egg issue when it comes to database applications. There is a need for data, but security and legal constraints often prevent you from working with copies of production data. In addition, new tables are somewhat uninteresting without data, as are objects that work with those tables, such as stored procedures.

DB Pro edition provides rich data-generation tools to ease your development pain. A database project can contain any number of data generation plans. Each plan supports data generation for one or more tables in your schema. If you enable a table for data generation, you specify a data generation plan for each column. You can use the built-in generators or you can write your own custom data generators.

The data generation tools provided let you generate the same set of repeatable test data over time, assuming the data generators are deterministic. This gives a consistent baseline against which all your tests can run.

Out of the box, you'll find a number of different standard generators. When you define a data generation plan, DB Pro edition automatically assigns a default value generator based on the data type of the column. In addition, DB Pro edition is aware of columns that enforce constraints, such as foreign key columns.

You can use value generators, available for most of the built-in data types (such as strings, integers, GUIDS) to generate arbitrary data for a column. Each value generator supports a number of properties to control generation. For example, for string data types such as NVARCHAR, you can set properties like Minimum Length, Maximum Length, and Percentage Null. If your column requirements are more complex, you can use the Regular Expression generator for generating phone numbers, postal codes, and the like. You can use the Data Bound generator to load existing data from tables in the database or from another supporting database. With the ability to create custom generators using managed code, you should have no trouble creating really useful data generation plans.

Unit Testing

You probably use unit testing for your Visual Basic® or Visual C#® code. Most likely, however, you don't use this same powerful technique when testing stored procedures, functions, and triggers. DB Pro edition brings the notion of a repeatable mechanism for testing the aforementioned objects in an easy, automated fashion. It includes database unit tests that you use as part of a standard Team System test project. The unit tests themselves are T-SQL mapped on top of either Visual Basic or Visual C# class files, allowing for advanced customization.

You use unit testing in DB Pro edition to validate the contract between the database schema and the applications that consume said schema. Getting started is easy-the Northwind sample database has a number of stored procedures with which you can start practicing.

With the Northwind database already imported, you'll want to find the CustOrderHist stored procedure via the Schema View window. Right-clicking on the object provides an option to Create Unit Tests. Selecting this displays the dialog shown in Figure 4.

The dialog is broken down into three groups. You use the Current selection group to specify which item or items for which DB Pro edition should generate unit tests. You can select any number of stored procedures, functions, or triggers. (Note that the dialog box in Figure 4 doesn't show a functions node or triggers node because the Northwind database only has stored procedures.)

Figure 4 The Create Unit Tests Dialog
Figure 4 The Create Unit Tests Dialog(Click the image for a smaller view)
Figure 4 The Create Unit Tests Dialog
Figure 4 The Create Unit Tests Dialog (Click the image for a larger view)

In the Output project group, you define whether to create a new test project or add the new unit tests to an existing project. If you are creating a new project, you choose the default programming language (Visual Basic or Visual C#). Finally, you specify a new class name or select an existing class (assuming you have an existing test project selected).

The first time you start to create unit tests for a database and you encounter the Create Unit Tests dialog, you might be tempted to just select everything and click OK. Don't. Do you really want all of your testable objects in the same class? Do you even want all of them in the same test project? Just as you don't put all of your code in a single class inside a single class library, you probably don't want to put all of your tests in such a confining configuration. The fact that your database might use schemas provides a good factoring mechanism. Just think before you click OK.

For this example, select the CustOrdersOrders and CustOrdersDetails stored procedures. DB Pro edition automatically puts a check next to CustOrderHist. Each programming object you select will have a test script auto-generated for it (you will probably need to customize these). Choose a programming language, change the name of the project to NorthwindTests, change the name of the class to Customers, and click OK to generate the test project and unit tests. (Note that choosing a programming language does not mean you have to use it, as you can do everything you need to with T-SQL.) After you click OK, the Test Project Configuration dialog box appears (see Figure 5). Starting from the top, you set connection settings in the first section.

Figure 5 Test Project Configuration Dialog
Figure 5 Test Project Configuration Dialog(Click the image for a smaller view)
Figure 5 Test Project Configuration Dialog
Figure 5 Test Project Configuration Dialog (Click the image for a larger view)

The first item lets you specify the connection to the live database server to use when executing the unit tests. The second item lets you specify the connection to use when validating the unit tests.

Consider using two different connections. You set the first connection for execution to run using credentials of a typical user account that will execute the object you're testing. You set the second connection for validation to run using credentials of a user with elevated privileges. If you specify two connections, the testing runtime executes pre-test, post-test, initialize, and cleanup procedures using the validation connection-otherwise, it uses the execution connection. This is particularly useful since many validation operations, like cleaning up tables, require elevated privileges.

With the connections defined, you set deployment options in the next section of the dialog box. You can choose to deploy a new or updated database schema prior to execution of the unit test by specifying an existing DB Pro edition project. You also specify the deployment configuration to use. Finally, you can choose a data generation plan to run. The testing runtime executes the data generation plan after it has deployed the database (if chosen) but before it executes the tests.

If you do choose to generate data, you can have the testing runtime clear the tables before loading new data. If you're following along, just choose the same Northwind connection you used to create the database project for the execution connection, accept the defaults for the other settings, and click OK. The test project generation completes with the Database Unit Test designer open, showing generated T-SQL code to help you get started (see Figure 6).

Figure 6 Test Designer
Figure 6 Test Designer(Click the image for a smaller view)
Figure 6 Test Designer
Figure 6 Test Designer (Click the image for a larger view)

The designer supports a T-SQL script editor and a class editor for Visual Basic and Visual C#. You'll see that the window is broken into two sections: an editor and a conditions manager.

Each database unit test class can contain one or more database unit tests, and each unit test is composed of up to three scripts: a pre-test, a test, and a post-test. However, only the test script is required. In addition, each database unit test class supports common pre- and post-test scripts. At the top of the designer, you'll find a navigation bar where you can choose the active unit test and test script.

Your generated tests will have a default test condition of inconclusive. You manage test conditions using the lower portion of the designer; each test can have more than one test condition. The lack of any test conditions will cause the test to pass automatically when run. For this reason, the default use of an inconclusive test condition ensures that you edit the auto-generated script and define meaningful test conditions.

In the case of the sample dbo_CustOrderHistTest, modify the auto-generated T-SQL to provide a customer ID, such as ALFKI. Then select the default test condition and delete it by clicking the Delete button (just to the right of the Test Conditions combo box). If you expand the Test Conditions combo box, you'll see a list of built-in Test Conditions, including Execution Time, Not Empty ResultSet, and Row Count. In addition, you can write your own custom test conditions using managed code.

For the dbo_CustOrderHistTest unit test, add the Not Empty ResultSet test condition. Also add the Row Count test condition and change its Row Count property to 11. Note that you can rename each test condition and you can enable or disable test conditions as needed.

With the designer window active, you can run your tests by pressing F5. This saves your project and compiles it. The test engine then executes all of your unit tests. Figure 7 details the execution flow of a database unit test class.

Figure 7 Database Unit Test Execution
Figure 7 Database Unit Test Execution(Click the image for a smaller view)
Figure 7 Database Unit Test Execution
Figure 7 Database Unit Test Execution (Click the image for a larger view)

You can run an individual test by showing the Test View window, selecting a particular unit test, right-clicking on it, and selecting the Run Selection command. In addition, you can debug your tests. If you're using DB Pro edition with Team Foundation Server, you can publish your test results to the Team Foundation Server as well as have your tests run as part of your team builds. Also, if you're running a version of Visual Studio 2005 that includes the Team Test features, you can create any number of test lists to logically group tests into different test suites containing any type of valid test.

Comparing Databases

After you've done all this work on your database, it's time to move the changes into your test system. DB Pro edition lets you compare your project's schema with any running database. In addition, you can compare the data contained in tables and views in two databases using the data comparison tools. You can run either the schema or the data compare tools with or without a DB Pro edition project open.

You can use the schema compare tool to compare two schemas and, if you want, generate a DDL change script. You can immediately apply the script from within DB Pro edition or export it for later use. You access the schema compare tool via Data | Schema Compare | New Schema Comparison. The new Schema Comparison dialog box then opens, allowing you to specify the source and target schemas. You can compare a DB Pro edition project to a database, or compare one database to another.

Once you've defined the source and target, the comparison starts. The Schema Compare window, which has three sections, opens and displays the results. The top section provides a list of all the schema objects, grouped by type. The read-only Object Definitions section in the middle displays two text comparison windows showing the T-SQL for the currently selected object. The read-only Schema Update script window at the bottom displays the proposed update script.

You can filter the displayed objects based on whether they compare as different, missing, new, or equal. As you examine objects, you can either include or exclude them from the schema update script by changing an item's Update Action property. DB Pro edition sets the Update Action property to Skip for equal objects. All other objects are set to either Create, Update, or Drop and can be changed from the default Update Action to Skip. You can then easily filter out which objects you want modified from those you want skipped. All the while, DB Pro edition updates the proposed script in real time.

Once you're happy with the update script, you have a few choices. If you're feeling bold, you can write the changes to the target schema. This might work well in your lab, but probably won't fly with your DBA. Instead, you can either export the update script to a file or to DB Pro edition's built-in T-SQL editor. After a good review, you're ready to update the target database. You have lots of control over how schema comparisons are performed via Tools | Options | Database Tools | Schema Compare.

To perform a data comparison, first specify the source and target databases. Tables and views must have the same name, the same owner, and matching schemas. Tables must have the same key and index layout. You start the wizard by selecting the Data | Data Compare | New Data Comparison menu item. You can adjust various options related to the data comparison process by going to Tools | Options | Database Tools | Data Compare. Once you have completed a data comparison, you can use the generated script to perform data motion between the source and target databases.

Build and Deployment

You've seen that you can generate a T-SQL update script manually via the Schema Comparison tool. However, as part of the build process, DB Pro edition can generate a complete script for deploying your database project. This deployment script can do either a complete build or an incremental update. The build process can even consolidate all of your pre- and post-deployment scripts into one complete deployment script. You can deploy the script via the Build | Deploy Selection command right from within Visual Studio 2005.

Under project properties, you will find a number of options to control and adjust the build process. The Build tab contains the core settings, such as Target connection, Target database name, and Block incremental deployment if data loss might occur. You'll note there is also a Build Events tab that you can use to type pre- or post-build event commands. DB Pro edition uses MSBuild for its build process and supports integration with Team Build if you're using Team Foundation Server.

Conclusion

If you're a professional who works with SQL Server 2000 or SQL Server 2005, you will find Visual Studio 2005 Team Edition for Database Professionals to be an essential tool for building database-backed solutions. The product provides a rich feature set-especially for a first release-that will change the way you manage databases in a development environment.

Brian A. Randell is a senior consultant with MCW Technologies, LLC. Brian spends his time speaking, teaching, and writing about Microsoft technologies. Brian is the author of Pluralsight's Applied Team System course and a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


SQL Server 2005: Regular Expressions Make Pattern Matching And Data Extraction Easier

SQL Server 2005

Regular Expressions Make Pattern Matching And Data Extraction Easier

David Banister
This article discusses:
  • Efficient SQL querying using regular expressions
  • Support in SQL Server 2005 for regular expressions
  • Using .NET Regex classes from SQL Server
  • Effective uses for regular expressions in a database
This article uses the following technologies:
SQL Server 2005, .NET Framework
Code download available at: Regex2007_02.exe (154KB)
Contents
CLR User-Defined Functions
Pattern Matching
Data Extraction
Pattern Storage
Matches
Data Extraction within Matches
Conclusion

A
lthough T-SQL is extremely powerful for most data processing, it provides little support for text analysis or manipulation. Attempting to perform any sophisticated text analysis using the built-in string functions results in massively large functions and stored procedures that are difficult to debug and maintain. Is there a better way?

In fact, regular expressions provide a much more efficient and elegant solution. It is easy to see how they would be helpful for comparing text to identify records, but there is so much more they can do. I'll show you how to perform a variety of tasks, some simple and some amazing, that were considered impractical or impossible in SQL Server™ 2000, but are now possible in SQL Server 2005 due to its support for hosting the common language runtime (CLR).

Regular expressions are not new to SQL. Oracle introduced built-in regular expressions in 10g, and many open source database solutions use some kind of regular expressions library. Regular expressions could actually be used in earlier versions of SQL Server, but the process was inefficient.

Using the sp_OACreate stored procedure, any OLE automation object that implemented regular expressions could be used, but you had to create a COM object first, then make at least one IDispatch call, then destroy the object. For most purposes this was too inefficient and caused too many performance problems. The only alternative was to create an extended stored procedure. However, now there's SQLCLR, a CLR user-defined function (UDF) that lets you create an efficient and less error-prone set of functions using the Microsoft® .NET Framework.

CLR User-Defined Functions

CLR user-defined functions are simply static methods (shared functions in Visual Basic) defined within a .NET assembly. To use the SQLCLR objects, you must register the assembly with SQL Server using the new CREATE ASSEMBLY statement and then create each object pointing to its implementation within the assembly. For functions, the CREATE FUNCTION statement has been extended to support the creation of CLR user-defined functions. To make things easier, Visual Studio® 2005 takes care of all of the registration processes on your behalf when you use a SQL Server Project. This kind of project is different than most Visual Studio projects because when you attempt to debug (or start without debugging), the project is recompiled and the resulting assembly, as well as all of the SQLCLR objects defined within it, are then deployed to and registered with SQL Server. The IDE then runs the test script designated for the project. Breakpoints can be set in both the SQL script and in your .NET code, making debugging a simple process.

Adding a function is just like adding a new class to any other project type. Simply add a new item to the project and select User-Defined Function when prompted. The new method is added to the partial class that contains all of your functions. Your new method will also have a SqlFunction attribute applied to it. It is used by Visual Studio to create the SQL statements necessary to register the function. The IsDeterministic, IsPrecise, DataAccess, and SystemDataAccess fields on SqlFunction are also used by SQL Server for a variety of purposes.

Pattern Matching

Determining if a string matches a pattern is the simplest use of regular expressions and, as you see in Figure 1, it's, easy to do.

Figure 1 String Matching (Close Figure 1)
public static partial class UserDefinedFunctions
{
    public static readonly RegexOptions Options =
        RegexOptions.IgnorePatternWhitespace |
        RegexOptions.Singleline;

    [SqlFunction]
    public static SqlBoolean RegexMatch(
        SqlChars input, SqlString pattern)
    {
        Regex regex = new Regex( pattern.Value, Options );
        return regex.IsMatch( new string( input.Value ) );
    }
}

First I use the Options field to store the regular expression options for the functions. In this case, I've selected RegexOptions.SingleLine and RegexOptions.IgnorePatternWhitespace. The former specifies single-line mode, and the latter eliminates unescaped whitespace from the regular expression and enables comments marked with a pound sign. Another option you might want to use after careful thought and analysis is RegexOption.Compiled. As long as there aren't too many, you will see significant performance gain if you use Compiled on expressions that are heavily used. Expressions that are used over and over again should definitely be compiled. However, for regular expressions that are rarely used, do not use Compiled as it causes increased startup costs and memory overhead. As such, you might want to augment my general purpose RegexMatch function with an additional parameter that specifies whether you want the expression to be compiled; that way, you can decide on a case by case basis whether the additional overhead will be worth the resulting performance gains.

After specifying the RegexOptions to be used, I define the RegexMatch function, using the SqlChars data type instead of SqlString. The SqlString data type translates into nvarchar(4,000) while SqlChars translates into nvarchar(max). The new max size functionality allows strings to extend beyond the 8,000 byte limit of SQL Server 2000. Throughout this article, I use nvarchar(max) to be as generic as possible and to allow for the most flexibility. However, performance can be significantly better using nvarchar(4,000) if all the relevant strings contain less than 4,000 characters. You should examine your specific needs and code to them appropriately.

The remaining code in the method is simple. A Regex instance is created with the defined options and the provided pattern and the IsMatch method is then used to determine if the specified input matches the pattern. Now you need to add a simple query to the test script:

select dbo.RegexMatch( N'123-45-6789', N'^\d{3}-\d{2}-\d{4}$' )

The pattern in this statement is a simple test for a US Social Security Number. Set a breakpoint on the new query and then start debugging to step through the function. This function lets you do many different tests, but I'll show you some things most people don't consider. For example, it's very important to maintain consistent naming conventions within a database, and writing a query to validate that all of your stored procedures meet your organization's guidelines is difficult. The RegexMatch function makes this task much simpler. For example, the following query test performs this task:

select ROUTINE_NAME
from INFORMATION_SCHEMA.ROUTINES
where ROUTINE_TYPE = N'PROCEDURE'
    and dbo.RegexMatch( ROUTINE_NAME,
        N'^usp_(Insert|Update|Delete|Select)([A-Z][a-z]+)+$' ) = 0

This query tests that every stored procedure is prefixed with "usp_" followed by "Insert", "Update", "Delete", or "Select", followed by at least one entity name. It also verifies that each word in the entity begins with a capital letter. Compare those four lines with this oversimplified version using only built-in functions:

select ROUTINE_NAME
from INFORMATION_SCHEMA.ROUTINES
where ROUTINE_TYPE = N'PROCEDURE'
    and ( LEN( ROUTINE_NAME ) < 11
    or LEFT( ROUTINE_NAME, 4 ) <> N'usp_'
    or SUBSTRING( ROUTINE_NAME, 5, 6 ) not in
        ( N'Insert', N'Update', N'Delete', N'Select' ) )

Even though this is more code, this query is actually missing several features present in the regular expressions version. First, it is not case-sensitive, and using collations within the query to perform the tests would make it unruly. Second, it does not perform any tests on the actual entity name contained within the procedure name. The third problem is the four strings tested in the query are all six characters long, which allowed me to simplify the code by pulling out a single substring of six characters that I could then compare against each of the acceptable operations. This is not problematic specifically in this example, since all of the operation names are six characters long, but imagine a standard that specifies more complex verbs like "Get", "List", or "Find". Those verbs are easily handled by the RegexMatch function, because they are just additional alternatives in the list.

Validation is a very common use of regular expressions, for anything from a phone number to a ZIP code to a custom account number format. The CHECK constraint is perfect for this, as the following table definition shows.

CREATE TABLE [Account]
(
    [AccountNumber] nvarchar(20) CHECK (dbo.RegexMatch(
        [AccountNumber], '^[A-Z]{3,5}\d{5}-\d{3}$' ) = 1),
    [PhoneNumber] nchar(13) CHECK (dbo.RegexMatch(
        [PhoneNumber], '^\(\d{3}\)\d{3}-\d{4}$' ) = 1),
    [ZipCode] nvarchar(10) CHECK (dbo.RegexMatch(
        [ZipCode], '^\d{5}(\-\d{4})?$' ) = 1)
)

The AccountNumber column is validated against an arbitrary convention that begins with three to five letters followed by five numbers, then a dash, and finally three more numbers. Both the phone number and ZIP codes are validated against standard US phone number and ZIP code formats. The RegexMatch function provides many features to SQL Server, but the regular expressions implementation in .NET provides much more, as you'll see next.

Data Extraction

The grouping features of regular expressions can be used to extract data from a string. My RegexGroup function provides that functionality to T-SQL:

[SqlFunction]
public static SqlChars RegexGroup(
    SqlChars input, SqlString pattern, SqlString name )
{
    Regex regex = new Regex( pattern.Value, Options );
    Match match = regex.Match( new string( input.Value ) );
    return match.Success ?
        new SqlChars( match.Groups[name.Value].Value ) : SqlChars.Null;
}

This function creates a Regex object just as the RegexMatch function did. Instead of testing for a match, however, a Match object is created for the first match found in the input string. The Match object is used to retrieve the group that is specified. If no match is found within the input, a null value is returned. If you prefer to use numbered groups instead of named groups, this function will still work for you. Just pass an integer value to the function within your SQL code and it will be implicitly cast to an nvarchar and the appropriate group will be returned.

You can use the RegexGroup function within a SELECT list to extract specific pieces of information from some other piece of data. For example, if you had a column that stored a URL, you can now easily parse the URL to determine individual pieces. This query uses grouping to determine every distinct server stored in the Url column of the UrlTable table.

select distinct dbo.RegexGroup( [Url],
    N'https?://(?<server>([\w-]+\.)*[\w-]+)', N'server' )
from [UrlTable]

You can also use this function within computed columns. The following table definition divides e-mail addresses into the mailbox and the domain.

CREATE TABLE [Email]
(
    [Address] nvarchar(max),
    [Mailbox] as dbo.RegexGroup( [Address],
        N'(?<mailbox>[^@]*)@', N'mailbox' ),
    [Domain] as dbo.RegexGroup( [Address], N'@(?<domain>.*)', N'domain' )

The mailbox column will return the mailbox or username of the e-mail address. The domain column will return the domain of the e-mail address.

Pattern Storage

All of the patterns used by these functions are just strings, which means that any of them can be stored in a table within your database. Most databases that store international data have a table representing countries. By adding a few extra columns to that table, you could store country-specific validation patterns. That would allow the constraint applied to an address row to vary based on the country for that row.

In databases that store data on behalf of clients, there is typically already a table representing a client. That table can be used to store grouping patterns that let you describe the way raw client data is stored within the database, and this allows you to create computed columns to pull the data you actually need from the client data. For example, if each of your clients has unique schemes for account numbers and you only need specific pieces of that account number, you could easily create an expression that pulls the correct piece of information for each client.

Matches

Rather than determining if a string matches a pattern, it is sometimes desirable to extract every match. Previously this kind of extraction would require cursors iterating over sections of a string. Not only is that process slow, but the code is also difficult to understand and maintain. Regular expressions are a much better means of performing this operation. The problem at hand is how to return all of the required data within a SQL construct. Table-valued functions are the answer.

Table-valued functions are somewhat similar to the previous functions, but vary in two distinct ways. First, the attributes applied to the method must fully declare the structure of the table that is returned. Second, there are two methods involved. The first returns an enumerable object instead of the actual result of the function. The second method is passed the enumerated objects to populate the fields of each row. Each value retrieved via the enumerator should correspond with one row of the resultset. The ICollection interface in the .NET Framework implements IEnumerable which means that any collection can be returned by the first method. The Regex class contains a Matches method that returns a MatchCollection that you could use. The problem with the MatchCollection is that the entire string must be processed prior to the Matches method returning. SQL Server includes optimizations that depend on processing occurring as needed, so instead of returning the entire collection up front, I prefer to write my own enumerator that returns each match as requested. This decision really depends on how the function is used and should be heavily tested prior to optimizing the enumerator.

The code in Figure 2 shows the enumerator. The MatchNode class wraps an individual match in the string while tracking its position within the set of matches returned. The MatchIterator class is enumerable and handles the regular expression processing. It uses the new yield keyword to create the enumerator much more easily than previous versions of the framework. It will return each match detected within the input string as requested.

Figure 2 Custom Enumerable Object for Matches (Close Figure 2)
internal class MatchNode
{
    private int _index;
    public int Index { get{ return _index; } }

    private string _value;
    public string Value { get { return _value; } }

    public MatchNode( int index, string value )
    {
        _index = index;
        _value = value;
    }
}

internal class MatchIterator : IEnumerable
{
    private Regex _regex;
    private string _input;

    public MatchIterator( string input, string pattern )
    {
        _regex = new Regex( pattern, UserDefinedFunctions.Options );
        _input = input;
    }

    public IEnumerator GetEnumerator()
    {
        int index = 0;
        Match current = null;
        do
        {
            current = (current == null) ?
                _regex.Match( _input ) : current.NextMatch( );
            if (current.Success)
            {
                yield return new MatchNode( ++index, current.Value );
            }
        }
        while (current.Success);
    }
}

The code in Figure 3 defines the table-valued CLR UDF. The RegexMatches method returns a new MatchIterator. The SqlFunctionAttribute on the RegexMatches method also includes some additional properties. The TableDefinition property is set to the table definition of the function. The FillRowMethodName is set to the name of the method to call for each iteration of the enumerable object returned. In this case, that method is FillMatchRow.

Figure 3 Table-Valued CLR UDF for Matches (Close Figure 3)
[SqlFunction( FillRowMethodName = "FillMatchRow",
    TableDefinition = "[Index] int,[Text] nvarchar(max)" )]
public static IEnumerable RegexMatches(SqlChars input, SqlString pattern)
{
    return new MatchIterator( new string( input.Value ), pattern.Value );
}

[SuppressMessage( "Microsoft.Design", "CA1021:AvoidOutParameters" )]
public static void FillMatchRow( object data,
    out SqlInt32 index, out SqlChars text )
{
    MatchNode node = (MatchNode)data;
    index = new SqlInt32( node.Index );
    text = new SqlChars( node.Value.ToCharArray( ) );
}

For each iteration of the MatchIterator, a MatchNode is passed to the FillMatchRow method as its first argument. The remaining parameters of the FillMatchRow method must be declared as out parameters and must match the table definition defined in the first function. The FillMatchRow function simply uses the properties of the MatchNode to populate the field data.

With this function you can finally extract multiple pieces of data from a string with ease. To illustrate the use of the RegexMatches function, let's process a string to determine how many distinct words are contained within it using this query:

declare @text nvarchar(max), @pattern nvarchar(max)
select
    @text = N'Here are four words.',
    @pattern = '\w+'
select count(distinct [Text])
    from dbo.RegexMatches( @text, @pattern )

This example is rather straightforward. It shows some potential for using the function but by removing the distinct keyword, it returns the total word count of a string. There are many Web sites that limit text entry to what seems like an arbitrary length. With this kind of test combined with the new nvarchar(max) notation, it becomes possible to limit input to a word count instead. This kind of query can be used for various analytic processing needs, but the RegexMatches function can also be used for more common tasks. Unfortunately, this kind of query also represents an overzealous use of regular expressions. The splitting operation accomplished by the "\w+" expression in this case could be just as easily accomplished with the String.Split method, which would be much faster. Regular expressions are a very powerful tool, but do make sure when you use them that you're using them for a good reason: there might be simpler tools you can use for specific cases that would yield better performance.

I often see questions in the MSDN® forums about how to pass a list of values to a stored procedure. I have also seen various convoluted methods of parsing such a list into an actual list to determine the correlated records. The RegexMatches function provides a much cleaner approach.

declare @pattern nvarchar(max), @list nvarchar(max)
select @pattern = N'[^,]+', @list = N'2,4,6'

select d.* from [Data] d
inner join dbo.RegexMatches( @list, @pattern ) re
    on d.[ID] = re.[Text]

The pattern matches any group of characters not containing a comma. Given a table named Data with an integer column named ID, this query will return each record identified in the list. This becomes more useful when considering the implicit casting features within SQL Server. The same query can be used for integer, date/time, GUID, or floating-point data types. Other methods of processing a list of values would require multiple functions or stored procedures to be this flexible. This function can also be used for lists that are not comma-delimited. A list separated by spaces, semicolons, tabs, carriage returns, or any other identifiable character can be processed.

Data Extraction within Matches

Similar to returning matches, we can also extract data from each match. Attempting to do this using SQL is very difficult. Normally, this kind of task would be implemented within an application instead of the database, which causes problems because each application using the database would have to implement the required processing. In such scenarios, a reasonable approach might be to implement this functionality within stored procedures.

As with the RegexMatches implementation, I prefer to use a custom enumerable object to return group information. Grouping is only slightly more complicated because we also have to iterate over groups within each match. In Figure 4, the GroupNode class is just like the MatchNode class except that it also includes the name of group it represents. The GroupIterator class is similar to the MatchIterator class but includes an additional loop to return each group. Now that I have an enumerable object, I define a table-valued function just as I did with the RegexMatches function.

Figure 4 Custom Enumerable Object for Groups (Close Figure 4)
internal class GroupNode
{
    private int _index;
    public int Index { get { return _index; } }

    private string _name;
    public string Name { get { return _name; } }

    private string _value;
    public string Value { get { return _value; } }

    public GroupNode( int index, string group, string value )
    {
        _index = index;
        _name = group;
        _value = value;
    }
}

internal class GroupIterator : IEnumerable
{
    private Regex _regex;
    private string _input;

    public GroupIterator( string input, string pattern )
    {
        _regex = new Regex( pattern, UserDefinedFunctions.Options );
        _input = input;
    }

    public IEnumerator GetEnumerator()
    {
        int index = 0;
        Match current = null;
        string[] names = _regex.GetGroupNames();
        do
        {
            index++;
            current = (current == null) ?
                _regex.Match( _input ) : current.NextMatch( );
            if (current.Success)
            {
                foreach(string name in names)
                {
                    Group group = current.Groups[name];
                    if (group.Success)
                    {
                        yield return new GroupNode(
                            index, name, group.Value );
                    }
                }
            }
        }
        while(current.Success);
    }
}

In Figure 5, the RegexGroups function is defined like the RegexMatches function except that it returns an additional column of data containing the name of the group within the match. With this function we can now find multiple matches within a string and extract specific pieces of the information from within each match.

Figure 5 Table-Valued CLR UDF for Groups (Close Figure 5)
[SqlFunction( FillRowMethodName = "FillGroupRow", TableDefinition =
    "[Index] int,[Group] nvarchar(max),[Text] nvarchar(max)" )]
public static IEnumerable
    RegexGroups( SqlChars input, SqlString pattern )
{
    return new GroupIterator( new string( input.Value ), pattern.Value );
}

[SuppressMessage( "Microsoft.Design", "CA1021:AvoidOutParameters" )]
public static void FillGroupRow( object data,
    out SqlInt32 index, out SqlChars group, out SqlChars text )
{
    GroupNode node = (GroupNode)data;
    index = new SqlInt32( node.Index );
    group = new SqlChars( node.Name.ToCharArray( ) );
    text = new SqlChars( node.Value.ToCharArray( ) );
}

Importing data in various formats is a common task when dealing with databases. Importing files in a comma-delimited format is much more of a chore than it should be. Most developers create an application that processes each line, extracts the data, and then executes a stored procedure for each line. Although that process works, I'd like to propose another solution. What if you could pass the entire file to a stored procedure and let the stored procedure handle the entire process? This idea is usually considered too complicated to implement, but with the RegexGroups function you can actually perform this insert with a single query. For example, consider the following customer data.

2309478,Janet Leverling,J
2039748,Nancy Davolio,N
0798124,Andrew Fuller,M
4027392,Robert King,L

There are three different pieces of information you need from each line: the seven-digit customer number, the customer name, and the single character customer type. With the following expression you can extract all three pieces of information.

(?<CustomerNumber>\d{7}),(?<CustomerName>[^,]*),(?<CustomerType>[A-Z])\r?\n

The problem you now have is that the results returned by the RegexGroups function are not directly usable. Instead of using a cursor to iterate over the results, you can use the pivot functionality in SQL Server 2005. Putting all of this together into a stored procedure, you have everything you need. The stored procedure in Figure 6 accepts the text of an entire comma-delimited file containing up to 2GB of Unicode data. It processes the entire file and inserts each line of the file as a row into the Customer table. Any delimited text file could be processed the same way. With some small changes to the pattern, escape sequences could be added to support commas within strings.

Figure 6 Processing a Comma-Delimited File (Close Figure 6)
create proc ImportCustomers
(
    @file nvarchar(max)
)
as

declare @pattern nvarchar(max)

set @pattern = N'(?<CustomerNumber>\d{7}),
    (?<CustomerName>[^,]*),(?<CustomerType>[A-Z])\r?\n'

insert [Customer]
(
    [CustomerNumber],
    [CustomerName],
    [CustomerType]
)
select
    f.[CustomerNumber],
    f.[CustomerName],
    f.[CustomerType]
from dbo.RegExGroups( @file, @pattern ) regex
pivot
(
    max([Text])
    for [Group]
    in ( [CustomerNumber], [CustomerName], [CustomerType] )
) as f

Again, however, this procedure also demonstrates that there are multiple ways to do the same task, and sometimes regular expressions aren't always the best option. In this example, using a pivot is, in effect, undoing all the work that RegexGroups did to return the data in the special grouped format. One could insert the data directly into the table using a much simpler and faster TVF that just read each line, String.Split on commas, and returned each row.

Conclusion

Although these matching functions are very powerful, they are not complete. There are many possible options that determine the exact way matches are performed. If your database collation is case-insensitive, you may want the functions to perform their matches in a case-insensitive manner as well. The explicit capture option may be required to reduce some resultsets. The multi-line option can allow you to create more precise patterns for some tasks. You may even want to create a user-defined type to pass the exact required options to each of the functions, which would allow each execution of a function to use a different set of options.

You should also be aware that there are localization issues when processing text. For example, the .NET Framework Regex class is aware of many more characters than the Latin ones used in my examples, so care should be taken when developing patterns for databases that use international data.

And of course, as mentioned several times in the article, while regular expressions are immensely powerful, make sure you actually need that power. Some tasks can be performed faster and more simply with more basic tool sets.

For simplicity, the examples I provided lack validation and error handling, which should be included in any production system. Each of the inputs to the function should be validated and your requirements should determine the responses to null or empty string inputs. The Regex class can throw exceptions when the pattern can't be parsed or the options are invalid. Those exceptions should be handled gracefully.

Combining regular expressions with SQL provides many alternative means of processing data. Using these functions can reduce the amount of time required to add functionality to your database as well as make the system more maintainable. Any database could use regular expressions, and I recommend that you experiment with these functions to find new and even more creative uses.

David Banister is a Senior Software Developer for a top-4 accounting firm in Atlanta. He has been writing software for many years. In his spare time he enjoys reading ECMA language specifications, playing tennis, and helping local bands.


Editor's Note: Hello, Office

Editor's Note
Hello, Office
Joshua Trupin

Nothing’s more disappointing than clever design paired with poor implementation. Bluetooth headsets come to mind: they’re tiny, wireless, and cute. But, they don’t pair properly, they turn on when you want them off, or off when you want them on. You just never know when they’re going to work the way they’re supposed to.

Good design needs to be paired with a solid implementation. After months of testing Windows Vista and the 2007 Microsoft Office system, we’re seeing a lot of well-designed features that work solidly.

This month’s MSDN Magazine focuses on the new 2007 Microsoft Office system, which has a number of innovations that make tasks more convenient and/or consistent for both users and programmers. (When we think about who does development, it’s easy to forget that millions of people do some level of programming with Microsoft Office, even if it’s just the occasional macro.)

The most obvious change in the 2007 Microsoft Office system is the new Ribbon interface. In place of the familiar menuing system used by almost every Windows-based app of the past 20 years, Microsoft Office provides a UI that puts a lot more information and functionality right in front of the user, and a lot more underlying capability for the developer. A product with as large an installed base as the 2007 release doesn’t change things on a whim, so what, after all this time, prompts such a fundamental design change? In our lead article, Eric Faller discusses the whys and wherefores behind the Ribbon, goes over new and unfamiliar concepts, components, and terminology, and shows developers how to take advantage of associated APIs to create their own Ribbons for the 2007 Microsoft Office system.

Another unusually cool feature in the 2007 system is called SmartArt graphics. This set of intelligent graphics lets you add oomph to your Office documents. It’s a bit like Visio on steroids, but also lets you do a lot of things that you never could in Visio. Even someone who has no graphical arts training to speak of can make attractive diagrams to display, say, beloved pet beagle timelines. For this lovely inset, we just clicked the shape we wanted, added text, and clicked again to insert pictures.

The 2007 Office system comes with a gallery of templates and predefined shapes that can quickly be inserted into your documents. Even better, it’s extensible! In this issue, Janet Schorr discusses what SmartArt graphics are, how they work, and how you can create your own.

Another way to extend Microsoft Office is by creating custom add-ins for the 2007 Office system applications using Microsoft Visual Studio Tools for Office (VSTO). There’s a new release of VSTO and it marks the first time VSTO has included support for PowerPoint and Visio. Paul Stubbs looks at this support in his feature article and shows you how to extend these applications.

Finally, in yet another upgrade, Ted Pattison puts his Basic Instincts column to rest and turns his attention to Office Space, a new column devoted to developing with and for the 2007 Microsoft Office system. In this issue, he focuses on programmatically building Office Open XML files.

There’s a lot of treasure to be dug out of Microsoft Office, not just for users but for developers too. And unlike with Bluetooth, the features actually work reliably, and the 2007 Office system and Windows Vista pair up properly every time.

Before we sign off, we’d like to add a personal note. After 165 columns, Paul DiLascia has decided to take a break from the monthly grind after this issue. We wish Paul well, and hope to see him back in MSDN Magazine soon. —J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Shawn Burke, Claudio Caldato, Pablo Castro, Jonathan Caves, Shy Cohen, Matt Gibbs, Mark Hall, Steven Hemingray, Doug Mahugh, Matt Nunn, Doug Purdy, Stefan Schackow, Ayman Shoukry, Don Smith, Herb Sutter, Damien Watkins, and Matthew Yip.

Access, Active Directory, ActiveX, Excel, Expression, IntelliSense, Internet Explorer, InfoPath, Microsoft, MSDN, Outlook, PowerPoint, SmartArt, Virtual Earth, Visio, Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Win32, Windows CardSpace, Windows Live, Windows Media, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. Msdn magazine disclaims any liability whatsoever for any use of such code or other information. The recommendations and technical guidelines in MSDN Magazine are based on specific environments and configurations. These recommendations or guidelines may not apply to dissimilar configurations. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Technology LLC. CMP Technology LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Rich Web App UIs, Tools for Regular Expressions, and More

Toolbox
Rich Web App UIs, Tools for Regular Expressions, and More
Scott Mitchell

Contents
Create Rich Web App UIs
Tools for Regular Expressions
Create Screen Mockups
The Bookshelf

Create Rich Web App UIs

ASP.NET ships with a variety of built-in Web controls that provide the core user interface elements. However, if you need to employ a richer user interface or if you just need more visually stunning results, your best bet is to turn to a third-party vendor.

There are a variety of companies that supply rich ASP.NET user interfaces including enhanced grids, toolbars, tabstrips, charts, menus, and more, all wrapped into a single control suite package. One such product line is NetAdvantage for ASP.NET version 2006 Volume 2 by Infragistics. The NetAdvantage for ASP.NET library contains a suite of ASP.NET Web controls that you can use to work with data, collecting user input, navigation, charting, and grouping controls.

The Infragistics flagship product is UltraWebGrid, which provides a fully functional and customizable grid experience that is not available with the built-in ASP.NET DataGrid or GridView Web controls. For example, features like grouping, column resizing, fixed column headers, filtering, and column and row selection—which are not provided by the DataGrid or GridView by default—can be added to the UltraWebGrid merely by setting a few properties through the Designer. UltraWebGrid can also be configured to utilize AJAX techniques to load data on demand and perform updates to the grid's data.

NetAdvantage for ASP.NET
NetAdvantage for ASP.NET (Click the image for a smaller view)
NetAdvantage for ASP.NET
NetAdvantage for ASP.NET (Click the image for a larger view)

Also available in NetAdvantage for ASP.NET are tree controls, navigation controls (toolbars, tabstrips, menus, and so on), rich text editor and masked input controls, charting controls, calendar and additional date/time-related controls, enhanced button and panel controls, and even a spell checker control. All of these controls provide a rich design-time experience and require little to no code. And most of these controls offer built-in AJAX features, including load on demand, type-ahead features, row filtering, and paging.

Not only do the NetAdvantage for ASP.NET controls open a new world of user interface design, but they also look spectacular. All controls provide a bevy of properties through which their appearance can be customized. There are also numerous predefined themes that can be applied at the click of a button. If you need to take your ASP.NET UIs to the next level, or just want an aesthetic Web site with minimal effort, check out Infragistics NetAdvantage for ASP.NET.

Price: $795 per developer license.

www.infragistics.com

Tools for Regular Expressions

Regular expressions are strings that use a special syntax to find matches of a specified pattern in other strings and are commonly used for input validation and string processing. For example, to ensure that a user has entered a date using the format YYYY-MM-DD, the regular expression pattern ^\d{4}-\d{2}-\d{2}$ could be used (although this simple pattern only checks for the correct number of digits and hyphens, and therefore allows invalid dates like 1111-99-44).

Creating and testing regular expressions can be a trying experience. For developers new to regular expressions, the mishmash of text and special characters can quickly lead to incomprehensible gobbledygook. When creating a regular expression pattern in code using Visual Studio®, there's no tool support. Gone are color syntax highlighting and IntelliSense®, and there's no easy way to debug or test a regular expression pattern within Visual Studio.

The Regulator (Click the image for a smaller view)

The Regulator (Click the image for a larger view)

Thankfully, Roy Osherove (weblogs.ASP.NET/rosherove) has created a number of free tools that make up for the lack of regular expression support in Visual Studio. The most well-known of these tools is Regulator, a standalone regular expression editor that includes color syntax highlighting, IntelliSense-like hints, a code snippets window, and search access to RegExLib.com, an online library of common regular expression patterns. Included are input and output windows; you provide an input string to test against and see the matches found using the regular expression. And, at the click of a button, Regulator will automatically generate the Visual Basic® or C# code to implement the regular expression for pattern matching, string replacement, or string splitting.

Roy also provides a regular expression Regex Kit Visualizer for Visual Studio 2005. When it is installed, regular expressions can be dissected and explored during debug time in the Visual Studio 2005 IDE through an interface similar to that of Regulator.

Price: Free.

tools.osherove.com

Create Screen Mockups

When engineering a software system, it's vital that all stakeholders have a clear understanding of the system's purpose and functionality. Such information is commonly communicated through the use of prototypes. The Visual Studio Designer allows for creating functional prototypes quickly, but such prototypes are often overkill because the needed information usually can be illustrated with diagrams and simple screen mockups. Moreover, Visual Studio wasn't designed to be a prototyping tool and therefore lacks features such as arranging forms into use case scenarios, illustrating changes to the UI based on user action, and so on.

A simpler and more efficient approach is to use a program designed specifically for making screen mockups. MockupScreens version 3.09 by Igor Ješe makes creating wire frame screens and organizing them into scenarios as simple as pointing and clicking. A MockupScreens project is composed of a set of scenarios, with each scenario comprising a set of screens. Each screen is made up of UI elements, such as buttons, labels, dropdown lists, and textboxes. Creating screens is a breeze with the Screen Editor, whose functionality resembles the Visual Studio Designer. To add user interface elements to a screen, simply drag the UI element from the toolbox onto the screen itself, after which the element can be precisely positioned and configured via its properties.

Once a mockup has been created, it can be shared with stakeholders in a number of ways. For those onsite, Mockup Screens offers a slide-show view of the project, allowing the prototype to be demonstrated in a presentation setting. For those off-site, the Mockup Screens project can be exported to a Web page. Also, individual screens can be exported as a JPG or GIF image file.

MockupScreens (Click the image for a smaller view)

MockupScreens (Click the image for a larger view)

There are many ways to create prototypes of an application, from simply sketching it out on a piece of paper to building functional prototypes using Visual Studio. MockupScreens offers an expedient and easy way to create and share wire frame mockups organized into use case scenarios. If you need to convey UI ideas or design decisions to stakeholders in a clear and efficient manner, give MockupScreens a try.

Price: $79 for a single-user license.

www.mockupscreens.com

The Bookshelf

All programs, regardless of their size, scope, or complexity, should be thoroughly tested to ensure correct results and behavior for both expected and unexpected inputs. Many developers employ commercial or open source testing frameworks to aid and automate the testing process. While testing frameworks have their place, in his book .NET Test Automation Recipes: A Problem-Solution Approach (Apress, 2006), fellow MSDN® Magazine columnist Dr. James D. McCaffrey proposes that these testing frameworks be complemented with lightweight test harnesses. A lightweight test harness is a short (four "pages" of source code or less), simple, standalone program written to perform a test or set of tests against the target app. All testing frameworks have a learning curve and many force developers into adopting a particular style for creating their test cases. With lightweight software test automation, however, developers creating the test harnesses can utilize their current knowledge base and programming style.

(Click the image for a smaller view)

Building a lightweight test harness involves a number of steps that, in a formal testing framework, would be handled by the framework itself. This includes reading the input test data, performing the test, comparing the actual output with the expected output, and saving and summarizing these results. With modern programming frameworks like .NET, writing the code to perform these tasks is straightforward. The book contains code and explanation for over 100 typical tasks in creating lightweight test harnesses.

The book is broken down into three sections—Windows Application Testing, Web Application Testing, and Data Testing—with each section encompassing four chapters. The Windows Application Testing chapters present recipes for using Reflection to programmatically access a class's properties or invoke its methods, as well as code snippets for performing user interface actions, like clicking on a particular control or selecting an item from a menu. The Data Testing chapters look at how to test ADO.NET code and SQL stored procedures, along with recipes for parsing, validating, and comparing XML data.

Lightweight test harnesses are, by definition, simple and should not take significant time to develop. This time can be lessened by having the recipes discussed in Dr. McCaffrey's book at your fingertips.

Price: $49.99.

www.apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out: .NET Application Extensibility

CLR Inside Out
.NET Application Extensibility
Jack Gudenkauf and Jesse Kaplan

Code download available at: CLRInsideOut2007_02.exe (197KB)
Contents
Building an Extensible Host
Discovery
Activation, Isolation, Security, and Sandboxing
Lifetime Management
Add-In Development
Wrap-Up

The Microsoft .NET Framework has made it very easy to take components built by different developers and companies and integrate them into your own applications. But it's only easy if you know which components you're building against. If those components are not known at build time, which is typically the case for add-ins, then things become much more difficult. Developers often have questions when extending their applications. For example, should the add-ins be stored in a database or on disk? Should they reflect on the add-in for a known interface so as to acquire the type for activation? What is the best way to use AppDomain, AppDomainManager, and AppDomainSetup?

We will address these and other questions in two installments of CLR Inside Out by introducing the new System.AddIn namespace in the Base Class Library (BCL), available in the next release of Visual Studio® (code-named "Orcas"). You may download the libraries prior to the full release of Visual Studio "Orcas" via the Community Technology Preview. You can also find information in the common language runtime (CLR) Add-In team blog.

In this series of columns we will walk you through the development lifecycle of adding extensibility to an application and then versioning it over time. This first column will cover all the challenges developers face when they first add extensibility to their application and how our APIs make their lives easier. Next month, we will focus on the longer-term problems of keeping add-ins working even as the host changes: we will take the sample application built in this column, change it significantly for its next version, and walk you through the steps needed to keep the original add-ins working.

So let's start talking about extensibility challenges and the beauty of APIs. With the addition of the System.AddIn namespace and guidance on building a managed add-in model, the CLR provides a solution that solves problems common to the first version of an application and guidance for ensuring the future compatibility of your applications. See the "Extensibility Terminology" sidebar to better understand the terms we'll use.

Extensibility Terminology

Host An application that supports extensibility. A host may be a client or server (for example, Excel, Word, Paint.NET, SQL Server, and Exchange Server).

Extensibility A mechanism for manipulating host application objects or extending host functionality, sometimes referred to as automation. Generally made available via an object model published as part of a host's SDK.

Software Development Kit (SDK) A set of public interfaces (APIs) and associated documentation. Allows the host application to provide services to an add-in and vice versa. This SDK may be provided by different parties in different situations. For example, the host may provide the SDK to enable host-specific add-ins, an add-in may provide the SDK to let many different hosts use it (for example, Windows Media® Player), a third party may provide the SDK to let many different hosts, and add-ins communicate with each other.

Add-In (also called Add-On, Extension, Plug-In, Snap-In) A customization; an assembly loaded by a host. Also, something that automates a host application (such as a client). Something that provides additional functionality to a host (service). An add-in is custom code, usually written by a third party, that is dynamically loaded and activated, by a host application, based upon some context (such as host startup or document loading). The add-in extends the host application via the host's public API (for example, an object model or managed class library) which was made available to the add-in via an SDK.

Pipeline The machinery that enables the host and add-in to communicate over a version-resilient, secure protocol.

Building an Extensible Host

After a host has defined an object model that it wants to expose, there are three main phases in a host's interaction with its add-ins: discovery, activation, and lifetime management. Discovery involves finding the available add-ins and providing the host or user with enough information to select which add-ins to use. Activation involves loading and starting the selected add-ins in an appropriate isolated security sandbox. Lifetime management refers to keeping an add-in, and the objects it exchanges with its host, alive for as long as the application has references to it while giving the host the ability to shut the add-in down immediately when necessary. Unless specified otherwise, all of the code here is part of the Program.cs file in SampleExtensibleCalculator.csproj available for download from the MSDN®Magazine Web site.

Before we go into detail about using an add-in at run time, let's talk about the SDK and defining the object model that a host provides to its add-ins. Our add-in model supports a wide range of scenarios, but typically they fall into one of three categories.

In the first category, the add-in provides a service to the host application. For example, a mail server might host add-ins for virus scanning, spam filtering, or IP protection; a word processor might have need for a spell-check add-in; and a Web browser might host add-ins for handling certain file types. Our Calculator example illustrates this scenario. If you look at the add-in code found in SampleCalculatorAddIn.cs, you will see that the add-in is providing the service implementation of Add, Subtract, Multiply, and Divide to the host.

In the second category, the host offers its behavior to the add-in and lets the add-in define how it automates the host. In this scenario, the host is really providing a service to the add-in. Most Microsoft® Office add-ins fall under this category: upon initialization of the add-in, the Office application will pass its root object to the add-in, allowing it to automate the host.

Using this model, a host can let third parties extend it in ways the application never considered. We've seen Microsoft Excel® add-ins that automatically replace text symbols with real-time stock quotes, and Microsoft Word add-ins that add virtual links to phone numbers in documents that start up a Voice over IP (VoIP) program and call a phone number. Sometimes you see entire applications built as one large add-in to an existing application. There are several CRM (customer relationship management) applications whose entire client front-end is contained inside Microsoft Outlook®.

The final category represents add-ins that use the host mainly for screen real-estate rather than any host-specific functionality. Search tool bars are an example of this: the same toolbar may add itself to your browser, the windows taskbar, and your e-mail client while providing the same functionality regardless of its host.

In our sample application, our host provides an SDK to add-ins in order to use them as a service; in this case, it will use the add-in to implement simple calculator functions. In our model there are actually three, often very similar, definitions of the object model. We define these components in what we call a "pipeline": the host's view, the add-in's view, and the contracts that cross the isolation boundary. We'll discuss the pipeline in more detail in the future, but for now let's just look at the host's view.

The following code is part of the host view, rather than the core host code, so you can find it in CalculatorView.cs in hostCalculatorView.csproj:

public abstract class Calculator
{
    public abstract double Add(double a, double b);
    public abstract double Subtract(double a, double b);
    public abstract double Multiply(double a, double b);
    public abstract double Divide(double a, double b);
}

As you can see, the host's view is simply an abstract base class that defines the functionality it expects from its add-ins. The host will program against this view directly and never be aware there is an isolation boundary between it and its add-in or that the add-in may have a very different view of the host.

Discovery

The first step in using add-ins in your application is discovering them: the application needs a way to find the available add-ins and get enough information about them to decide which ones to load and activate. Depending on the application, the host will typically perform this step at application start-up, when a document is loaded, or upon a user request.

Some of the high-level discoverability requirements include the ability to find particular types of add-ins in a specified location, providing enough information to let the host or user decide which add-ins it wants to use without having to activate them first, and intelligently caching so that the list of available add-ins does not have to be recomputed, and make it possible for add-ins to update the cache at installation time. To achieve these ends, you can follow the steps we took. First, define where you want to look for add-ins. In this sample, we expect the AddIns and components to be installed in the current directory, but often applications will choose several different locations (for example, the user's application data directory).

String addInRoot = Environment.CurrentDirectory;

Next, update your add-in store so that it is aware of newly installed add-ins and pipeline components. Our AddInStore class is used in determining what can connect to what. This information makes up what we call the pipeline.

The store contains all the components that make up the pipeline between the host and its add-ins. The AddInStore may be updated at run time using the AddInStore.Update APIs and may also be updated at add-in installation time, using our command-line tool, in order to enhance the discovery startup performance. The following code checks to see if new AddIns have been installed. If no new add-ins or pipeline components have been installed since this was last called, this returns very quickly:

AddInStore.Update(addInRoot);

Once the cache has been updated, the host will want to find all the available add-ins that can be connected to the view (abstract base class) that the host built against. This code looks for Calculator AddIns in our root directory and stores the results:

Collection<AddInToken> tokens =
    AddInStore.FindAddIns(typeof(Calculator), addInRoot);

The AddInStore.FindAddIns method takes in the type of add-in you are looking for and the path where they are stored and returns a collection of AddInTokens. It is important to note that no add-in assemblies have been loaded at this point and no add-in code has been executed. Each token describes a unique add-in and four properties (Name, Version Number, Publisher, and Description) provided by the add-in to allow the application/user to make decisions about which add-ins to load. In our calculator scenario, we built a ChooseCalculator method to display the tokens to the user, allowing them to select which they want to use:

AddInToken calcToken = ChooseCalculator(tokens);

Activation, Isolation, Security, and Sandboxing

Once the application has decided which add-ins it wants to use, it then needs to activate them. The host needs, from its perspective, an instance that implements the requested abstract base class (that is, the view). In reality, however, the host usually needs more than that: it really wants the add-in isolated from the host and other add-ins, and it wants to be able to sandbox that add-in with a particular permission set.

An isolated add-in assembly may not directly access code or resources in another assembly. The .NET Framework uses AppDomains as the in-process unit of isolation. For those who are unfamiliar with AppDomains, you can think of them as lightweight processes. Typically the host will run its code in one AppDomain, usually the default AppDomain that is created when the runtime starts up, and will want each add-in to have its own AppDomain. By isolating add-ins from the host and from each other, you gain several important advantages:

  • Each add-in is in its own folder, which has its own application base and loads its own assemblies without conflicting with the host or other add-ins. This also enables add-ins to have their own configuration file which may be used for functionality specific to add-ins.
  • The host can catch failures in one add-in and shut it down (and possibly restart) without impacting the host or other add-ins in the host process.
  • The add-in and its dependencies can be unloaded (more on this in the Lifetime Management section of this column).
  • Add-ins can be sandboxed to a particular security permission set (more on this just ahead).

It is important to note, however, that when developing the host and the add-in itself you are programming directly against the abstract base classes that make up the host and add-in views and therefore you don't have to worry about the constraints of the isolation boundary. In this version, we support AppDomain level isolation and may add process level isolation in future versions. For more information on using AppDomains as a security boundary for hosting untrusted add-ins. For more on add-ins and AppDomains, see AppDomain's and Addin's.

The host needs to consider the security context with which to load and activate the add-in. A host application may be running in full trust but may wish to run add-ins with less permission. The preexisting security and AppDomain creation APIs are very powerful and let you tweak many different settings as to how the AppDomain is sandboxed. In most cases the host will want to grant the add-in either FullTrust, intranet, or Internet security settings and rely on the framework to decide what that means. Our APIs make it very easy to choose one of those three settings but we also give you the ability to fine-tune the setup parameters if needed. Putting it all together in one line of code, you get the following, which activates the selected AddInToken in a new AppDomain and sandboxes the assembly in the internet zone:

Calculator calculator =
    calcToken.Activate<Calculator>(AddInSecurityLevel.Internet);

This single line of code returns to the host an activated instance of the selected add-in. So what happens behind the scenes to make this happen?

The AddIn framework creates an AppDomain and loads the appropriate assemblies in the AppDomain. Next, we give the code running in that AppDomain the security permissions appropriate for the specified security level (in this case corresponding to the Internet permission set). Then we set the ApplicationBase to be the location of the actual add-in assembly, to allow it to find its dependencies and resources. We then set the config file for the AppDomain to be the one sitting next to the add-in DLL. Finally, we hook up required pipeline components that connect the host to the add-in.

There is another overload that lets you specify a custom permission set for the add-ins and yet another that lets you provide your own AppDomain. This will allow you to fine-tune the AppDomain and security settings for the add-in or even group add-ins that have the same lifetime, security context, and/or enable performance benefits of re-using an AppDomain.

Figure 1 provides an end-to-end hosting that uses what we've covered thus far.

Figure 1 Hosting Add-Ins from Discovery to Activation (Close Figure 1)
static void Main()
{
    // In this sample we expect the AddIns and components to
    // be installed in the current directory
    String addInRoot = Environment.CurrentDirectory;

    // Check to see if new AddIns have been installed
    AddInStore.Rebuild(addInRoot);

    // Look for Calculator AddIns in our root directory and
    // store the results
    Collection<AddInToken> tokens =
        AddInStore.FindAddIns(typeof(Calculator), addInRoot);

    // Ask the user which AddIn they would like to use
    AddInToken calcToken = ChooseCalculator(tokens);

    // Activate the selected AddInToken in a new AppDomain set sandboxed
    // in the internet zone
    Calculator calculator =
        calcToken.Activate<Calculator>(AddInSecurityLevel.Internet);

    // Run the read-eval-print loop
    RunCalculator(calculator);
}

Lifetime Management

An essential part of any add-in model is the method used for managing the lifetime of the objects within it. COM lifetime management was entirely based on reference counting, while CLR lifetime management is based on a garbage collector, and CLR remoting uses a sponsor/timeout approach. In our system, we need to go across AppDomains or process boundaries and in these scenarios, the CLR's garbage collector is no longer able to provide adequate support. Therefore, we need to provide a system of lifetime management that makes it completely transparent to the host and add-in developers, and makes it as easy as possible for the pipeline (object model) developers. We provide a solution for lifetime management that will be discussed in more detail in next month's column.

A host application may be a long-running service, have memory constraints, or need to be able to unload add-ins at run time for these and other reasons. The runtime itself only allows the unloading of assemblies when you unload the AppDomain they were loaded in: it doesn't have the traditional DllUnload. This is yet another way the AppDomain boundary adds value: it lets you reclaim system resources without having to cycle the process.

If the host wants to shut down an add-in immediately, it doesn't have to worry about keeping track of which add-in belongs in which AppDomain. Instead we provide a class called the AddInController that takes care of it for you. As long as you have a reference to the add-in you want to shut down, you can do it in one line of code:

// Retreive the AddInController for my add-in and then shut it down
AddInController.GetAddInController(calculator).Shutdown();

Aside from the host's ability to shut down its add-ins eagerly, our model enables the host and add-ins to act as if their lifetime management was completely controlled by a garbage collector: if they want something to be released, they just null out the references or dispose it. Behind the scenes we actually manage lifetime with a combination of a reference-counting system and the runtime's remoting services.

Add-In Development

From the developer's perspective, the mechanics of the add-in system are hidden. The host's object model interaction, and/or the means by which the add-in offers extensibility to the host, should be as transparent as possible. Writing an add-in using our model is very easy: we employ a custom attribute that defines the add-in assembly entry in its assembly and provides descriptive information about the add-in to the host. Once the attribute is applied, all the add-in has to do is implement the methods defined in its abstract base class.

Note that in the sample code for the add-in (found in SampleCalculatorAddIn.cs, as part of SampleCalculatorAddIn.csproj in the code download), the add-in developer does not see the Pipeline code. The mechanism for handling the remoting and lifetime management, for instance, is abstracted away (see Figure 2).

Figure 2 Simple Calculator Add-In (Close Figure 2)
namespace SampleCalculatorAddIn
{
    [AddIn("Sample Calculator AddIn", Version="1.0.0.0")]
    public class SampleCalculatorAddIn : Calculator
    {
        public override double Add(double a, double b)
        {
            return a + b;
        }
        public override double Subtract(double a, double b)
        {
            return a-b;
        }
        public override double Multiply(double a, double b)
        {
            return a * b;
        }
        public override double Divide(double a, double b)
        {
            return a / b;
        }
    }
}

As you can see, the add-in model contributes almost no overhead to the add-in developer. All the developer needs to describe the add-in to the host is to use the custom attribute and then implement the functionality the host required of it, in this case the simple arithmetic operations of a calculator.

Wrap-Up

Our goal in designing this model was to enable dynamic composition of version resilient, isolatable components. Our primary concerns were to ensure the add-in developer experience would not be complicated by the mechanics involved in meeting this goal and to make the host's job as easy as possible.

Next month we will discuss, in greater detail, the pipeline and utilization of the model to enable compatibility as your host and add-in version independently. In the meantime, please pay a visit to our team blog for more samples and to give us your feedback.

Send your questions and comments to clrinout@microsoft.com.

Jack Gudenkauf is a Software Architect on the CLR team at Microsoft. He is currently focused on engaging with ISV/VAR/SI customers to ascertain .NET architectural adoption issues. Jack has held positions including Developer Manager, MIS Director, software engineering, and consulting services. Previously, he ran his own consulting company, The Logical Choice.

Jesse Kaplan is the Program Manager for Extensibility and Add-Ins for the CLR team at Microsoft. His past responsibilities include compatibility and managed/native interoperability.


Data Points: Data Bound Applications with ADO.NET and Custom Objects

Data Points
Data Bound Applications with ADO.NET and Custom Objects
John Papa

Code download available at: DataPoints2007_02.exe (174KB)
Contents
Test-Driving the Application
Implementing Custom Entities and Interfaces
Generics and ADO.NET
Data Sources
Binding Controls
Binding Code
Saving Data
Using Predicates to Find an Entity
Wrapping It Up

The Windows Forms binding controls are vastly improved descendents of the data binding controls of the past. They get you going quickly and handle the redundant tasks associated with setting up forms, and you can customize and extend their behavior significantly. Data can travel in a variety of containers, including DataSets and custom class entities, and the Windows® Forms binding tools allow you to bind to all of these types of objects. If you don't want to use a DataSet, you can create custom entities to be used as the data store for your application, and you can use List<T> and other collection types to store a set of your custom entities. These types of custom entities can easily be bound using the BindingSource and the BindingNavigator. In this column I'll demonstrate how to bind a custom list of business entities using the binding tools present in the Microsoft® .NET Framework 2.0, and I'll do so by writing a fully functional data-driven Windows Forms application.

I'll begin by describing the application, in particular its use of the DataGridView and the BindingSource and BindingNavigator binding controls. Then I'll walk through the lower tiers and demonstrate how they are architected and how the data is retrieved, persisted, accessed, and sent back to the database. All code for the sample application is included in this issue's download file.

Test-Driving the Application

The application will let the user view, add, delete, update, find, and navigate through records. It loads Northwind order data into the DataGridView, as shown in Figure 1. When an order is selected, the TextBoxes, ComboBoxes and other controls on the right side of the form are filled in with the selected order's information. All the controls are bound to the same data source via a BindingSource control.

Figure 1 Viewing Northwind Orders in a DataGridView
Figure 1 Viewing Northwind Orders in a DataGridView(Click the image for a smaller view)
Figure 1 Viewing Northwind Orders in a DataGridView
Figure 1 Viewing Northwind Orders in a DataGridView (Click the image for a larger view)

The BindingNavigator control is the toolbar across the top of the form in Figure 1. It has standard navigation buttons for changing the order record the screen will display. The navigation buttons work in concert with the grid on the left which allows them to remain in sync with the current record. The toolbar also has buttons that execute event handlers for adding, deleting, and updating order information. Finally, the application lets you search for a specific order (note the binoculars).

The fields for the order record that are foreign key references can be displayed using ComboBox controls. For example, a ComboBox could be used to display a list of the salespeople (that is, employees). The salesperson for a specific order would be selected in the ComboBox. This is a better option than displaying the Employee ID, which likely has little significance to the user of the application. In Figure 1, notice that the employee name rather than the employee ID is displayed in a ComboBox. Customer name is also in a ComboBox.

Implementing Custom Entities and Interfaces

While the DataSet is a powerful tool in the data access arsenal, using custom classes to manage and represent that data model in an application is also powerful. The pros and cons of both are widely documented, and many people are planted firmly in either the DataSet camp or the custom class camp. In fact, both methods are viable in enterprise architectures. Furthermore, ADO.NET tools work with both DataSets and custom classes to create the entities to represent the data objects. The bottom line is you need some sort of data store to contain your data. In this application, I'll use custom entities.

The sample application includes two projects: one for the presentation of the data and another for the business logic and data access. When creating a custom entity in the lower layer, you have to create the properties for the entity. For example, the Customer class has a CustomerID property and a CompanyName property. Figure 2 shows the private fields and the public properties that represent the CustomerID and the CompanyName. While this code can become tedious to type, especially compared to using a DataSet, employing some of the many refactoring tools that generate properties on the fly, or even a code generation tool to generate the entire class, can make the creation of the class quite simple.

Figure 2 Customer Class Properties (Close Figure 2)
public event PropertyChangedEventHandler PropertyChanged;

private string _CustomerID;
public string CustomerID
{
    get { return _CustomerID; } set { _CustomerID = value; }
}

private string _CompanyName;
public string CompanyName
{
    get { return _CompanyName; }
    set
    {
        if (!value.Equals(_CompanyName))
        {
            _CompanyName = value;
            if (PropertyChanged != null)
            {
                PropertyChanged(this,
                    new PropertyChangedEventArgs("CompanyName"));
            }
        }
    }
}

Take a look at the code in Figure 2. The Customer class in this example implements the INotifyPropertyChanged interface that raises an event called PropertyChanged if the CompanyName property changes. Notice that the CompanyName property's set accessor checks to make sure the value has actually been changed before it sets the new value. If so, the PropertyChanged event is raised and the class can notify anyone listening about the change. In my application, the BindingSource will automatically update the controls on the form with the new value when notified of a change.

The application also contains three entity classes: Order, Customer, and Employee. All of them implement the INotifyPropertyChanged interface and contain Property accessors to handle the getting and setting of the values for the properties.

Generics and ADO.NET

Once the entities have been built, methods must be created to retrieve and save them. This application's entities implement a standard list of static methods including some or all of the following: GetEntity, GetEntityList, SaveEntity, and DeleteEntity.

You could create one class for the entities and a separate class to contain the data access methods. Generally, I only separate these classes if the architecture dictates that the entities are not tightly coupled with the methods that save and retrieve them. In this sample application, the methods are tightly coupled so I chose to put them in a single class and make the methods static.

Figure 3 shows the GetEntityList<Order> method which returns an Order entities list representing all orders in the Northwind database. (Of course, this could be filtered if we added a parameter for orders for a specific customer or a date range.) By using generics and returning a List<T> instead of an ArrayList, the code guarantees that any object contained in the List will be of type T. This also means you can access an entity's properties within in the List<T> without casting it to the entity's type, as you would have to do if the entity was stored in a non-generic list. For example, you could get the OrderID from the first order in the list using the following code:

List<Order> orderList = GetMyListOfOrders();
int orderID = orderList[0].OrderID;

Figure 3 Getting a List <Order> (Close Figure 3)
public static List<Order> GetEntityList()
{
    List<Order> OrderList = new List<Order>();

    using (SqlConnection cn = new SqlConnection(Common.ConnectionString))
    {
        string proc = "pr_Order_GetList";
        using (SqlCommand cmd = new SqlCommand(proc, cn))
        {
            cmd.CommandType = CommandType.StoredProcedure;
            cn.Open();
            SqlDataReader rdr = cmd.ExecuteReader(
                CommandBehavior.CloseConnection);
            while (rdr.Read())
            {
                Order order = FillOrder(rdr);
                OrderList.Add(order);
            }
            if (!rdr.IsClosed) rdr.Close();
        }
    }
    return OrderList;
}

If the entities were stored in a non-generic list, the object would have to be cast to the type:

ArrayList orderList = GetMyListOfOrders();
int orderID = ((Order)orderList[0])).OrderID;

I could have created and filled a DataSet or a DataTable with the orders. However, I chose to use a SqlDataReader because I prefer the quicker access to the data it provides. As shown in Figure 3, I grab the data via the SqlDataReader, for each row instantiating and filling an Order entity, and then adding that entity to a List<Order>, and repeating the process for each row in the SqlDataReader. I could have used a DataTable and iterated through the DataRows as well. The performance difference was negligible, but there's really no advantage in this case to incurring the extra overhead of a DataTable since I am simply iterating through the rows and filling my own custom entity list. The FillOrder method executes the following code, which creates the instance of the Order and sets its properties from the SqlDataReader:

Order order = new Order();
order.OrderID = Convert.ToInt32(rdr["OrderID"]);
order.CustomerID = rdr["CustomerID"].ToString();
order.EmployeeID = Convert.ToInt32(rdr["EmployeeID"]);
order.OrderDate = Convert.ToDateTime(rdr["OrderDate"]);
order.ShipVia = rdr["ShipVia"].ToString();
order.ShipName = rdr["ShipName"].ToString();
order.ShipAddress = rdr["ShipAddress"].ToString();
order.ShipCity = rdr["ShipCity"].ToString();
order.ShipCountry = rdr["ShipCountry"].ToString();
return order;

Notice that in Figure 3 I passed CommandBehavior.CloseConnection to the ExecuteReader method. This causes the SqlConnection object to close as soon as the SqlDataReader closes.

There are static methods in the entities to insert, update, and delete data, too. I created a public static method called SaveEntity, which accepts an entity and determines whether it is new or existing, and then calls the appropriate stored procedure to perform the action query. The static AddEntity (that is shown in Figure 4) for the Customer class accepts the Order entity and then maps the values from the entity to the corresponding parameters of the stored procedure.

Figure 4 Adding an Entity (Close Figure 4)
private static Order AddEntity(Order order)
{
    int orderID = 0;
    using (SqlConnection cn = new SqlConnection(Common.ConnectionString))
    {
        string proc = "pr_Order_Add";
        using (SqlCommand cmd = new SqlCommand(proc, cn))
        {
            cmd.CommandType = CommandType.StoredProcedure;
            cmd.Parameters.AddWithValue("@customerID", order.CustomerID);
            cmd.Parameters.AddWithValue("@employeeID", order.EmployeeID);
            cmd.Parameters.AddWithValue("@orderDate", order.OrderDate);
            cmd.Parameters.AddWithValue("@shipVia", order.ShipVia);
            cmd.Parameters.AddWithValue("@shipName",
                GetValue(order.ShipName));
            cmd.Parameters.AddWithValue("@shipAddress",
                GetValue(order.ShipAddress));
            cmd.Parameters.AddWithValue("@shipCity",
                GetValue(order.ShipCity));
            cmd.Parameters.AddWithValue("@shipCountry",
                GetValue(order.ShipCountry));
            cmd.Parameters.Add(new SqlParameter("@orderID",
                SqlDbType.Int));
            cmd.Parameters["@orderID"].Direction =
                ParameterDirection.Output;
            cn.Open();
            cmd.ExecuteNonQuery();
            orderID = Convert.ToInt32(cmd.Parameters["@orderID"].Value);
        }
        order = GetEntity(orderID);
    }
    return order;
}

ADO.NET and the custom entities are both vital pieces of this architecture. The retrieval methods use ADO.NET to get the data from the database; the custom entities are then filled and returned to the presentation layer. The SaveEntity and DeleteEntity methods accept custom entities and then pull values out of them to apply the changes against the database.

Data Sources

Now that I've shown how the custom entities are created, filled, and returned, let's walk through the presentation layer. Because I have Customer, Order, and Employee classes in my class library project, I can use them to create the bound controls on a form. The Data Sources window shows all data sources that are available to the project. In Figure 5 you can see that I've added the three entities created in the class library project to the Data Sources window of the UI project.

Figure 5 Data Source Window
Figure 5 Data Source Window

Data sources can be obtained from a Web service, an object, or a database. In this case, I added object data sources since I am using class entities. I added the data source by going through its wizard, which prompts you to select the namespace and the class you want to add as a data source. If you are referencing the class library project already, the Customer, Employee, and Order classes will be displayed.

Once the data sources are added, they are displayed in the Data Sources window. The Order data source in Figure 5 shows all of the Order class's public properties. Next to each property name is an icon representing the type of control that will be used to display each property's value. The OrderDate property shows a DateTimePicker control and the ShipAddress shows a TextBox control. These controls can be changed by clicking on the property name's text in the Data Sources window and selecting a different control from the dropdown list. For example, I changed the control for the OrderID property to a Label control because I wanted it to be read-only.

I do not want properties that are reference fields to other entities, such as the Order.CustomerID property, to be displayed directly. Instead I want a more descriptive value to be displayed for them. For example, I changed the control type for the CustomerID property to a ComboBox with the intention of filling the list with all customer data and showing the appropriate customer's CompanyName property. I changed both the CustomerID and the EmployeeID properties to be ComboBoxes. You can also specify that you do not want to display a property on the form by selecting the [None] option from the list instead of a control type.

After setting up my properties and the controls I want them to be displayed in, I click on the Order entity in the Data Sources window and select Details from the list. This allows me to drag the Order data source to a form and automatically generate both the display controls and the binding controls (BindingSource and BindingNavigator) on the form. This is how the controls on the right-hand side of the form in Figure 1 were created.

Binding Controls

BindingSource is a non-visible control that is the link between the Order entity and the controls on the form. The current Order entity in BindingSource will be displayed in the controls on the right-hand side of the screen where they can be viewed or edited. In this application, it will be bound to a List<Order> entity I'll pass to it from the Order class's static method GetEntityList.

The BindingNavigator control was also created by dragging the Order data source to the form. This control displays a toolbar at the top of the form that by default has a handful of buttons to handle adding, saving, and deleting records, as well as navigation buttons. The BindingNavigator is in sync with the BindingSource control so that when a record is repositioned in either control, the other control will automatically reflect that position change. For example, if the user clicks on the Next button in the BindingNavigator control, the BindingSource control will change its current entity to be the next entity, and in turn the controls to the right will display the current entity's property values.

The BindingNavigator's buttons are also controls so they have properties and events you can set, too. I added event handlers for the Save and Delete buttons on the toolbar so I could add validation and other logic before changed data was persisted to the database through the entities' static data access methods previously created.

While the BindingNavigator control handles record movement and provides an easy way to save data changes, you don't necessarily need to include it. Once the control is automatically created by dragging a data source to the form, you can delete it from the form with no ill effects. Of course, then you'll have to write your own code to implement movement throughout the list of orders and the saving of changes. Whether or not you should really depends on your UI requirements.

The CustomerID's ComboBox is currently bound to the CustomerID property of the Order data source (through the BindingSource control). I still need to fill the ComboBox with a list of customers and display the customer's CompanyName. When I created my class library project, I created a Customer entity and exposed a GetEntityList<Customer> method from it. I then added a data source for the Customer entity. All I need to do is drag the Customer data source over the CustomerID ComboBox and drop it. This will cause a second BindingSource control to be created. The ComboBox uses this new BindingSource to load the list of customers for display. So the selected value of ComboBox is still bound to the Order's BindingSource control and the list of customers and their CompanyName properties are displayed in the control. This process is repeated to fill the list for the Employee ComboBox, too. Since I dropped the data source on ComboBox, it was smart enough to know that I did not need another BindingNavigator control.

However, I wanted to give users a second way, besides BindingNavigator's toolbar, to navigate the order records. So I went back to the Data Sources window, clicked on the Order data source, changed the selection to DataGridView and dragged the Order data source onto the form. This created a DataGridView control on the form that contained a column representing each of the Order entity's properties. I then removed most of the columns so that only the key information is displayed in the grid. No additional binding controls are created because there is already a BindingSource control on this form that binds the Order data source. At this point the current order for all of the controls is exactly the same since they all are linked to the Order's BindingSource control.

Binding Code

At this point I've designed the controls but I haven't written any code for the form. The BindingSource controls for the customers, employees, and orders are all set up and ready to go, but I have yet to pass them their data. This can easily be done by first calling each entity's static method-GetEntityList-and retrieving a List<T> of the entities. The List<T> is then converted to a BindingList<T> and set as the data source for each respective BindingSource control.

Figure 6 shows how each of the three BindingSource controls are set to their lists. This is all the code that's required in the UI to allow the user to navigate and view the data. I call the SetupBindings method in the form's constructor so the data is retrieved, bound, and displayed when the form is first loaded. The user can then move through the records using the navigation buttons on the BindingNavigator toolbar or by selecting a row in the DataGridView control. However, we have yet to write the event handlers that will allow users to make changes.

Figure 6 Setting the BindingSource Control's Data (Close Figure 6)
private void SetupBindings()
{
    BindingList<Order> orderList =
        new BindingList<Order>(Order.GetEntityList());
    orderBindingSource.DataSource = orderList;

    BindingList<Customer> customerList =
        new BindingList<Customer>(Customer.GetEntityList());
    customerBindingSource.DataSource = customerList;

    BindingList<Employee> empList =
        new BindingList<Employee>(Employee.GetEntityList());
    employeeBindingSource.DataSource = empList;
}

Saving Data

I want the application to allow users to delete the currently selected and displayed order by clicking on the Delete button on the toolbar. First, I set the Delete button's DeleteItem property to "none" to force it to use custom code to perform the deletion. Next, I add the event handler that will perform the delete, which calls a private method named Delete, shown in Figure 7.

Figure 7 Deleting the Current Order (Close Figure 7)
private void Delete()
{
    Order order = orderBindingSource.Current as Order;
    int orderID = order.OrderID;
    DialogResult dlg = MessageBox.Show(
        string.Format("Are you sure you want to delete Order # {0}?",
        orderID.ToString()));
    if (dlg == System.Windows.Forms.DialogResult.OK)
    {
        Order.DeleteEntity(order);
        orderBindingSource.RemoveCurrent();
        MessageBox.Show(string.Format(
            "Order # {0} was deleted.", orderID.ToString()));
    }
}

The Delete method grabs the current order from the BindingSource's Current property and casts it to the entity type Order. It then asks the user if he is sure he wants to delete the order. If the user clicks OK, the Order entity is passed to the DeleteEntity static method in the class library project. Finally, the order is removed from the BindingSource by executing the BindingSource control's RemoveCurrent method.

The BindingNavigator's toolbar also has an Add button that adds a new row to the DataGridView and clears out the controls on the right side of the form. The user can then enter and select values for the new order and click the Save button on the BindingNavigator toolbar. I added an event handler to this Save button that passes the order entity to the Order entity's SaveEntity static method. The user can also edit the current order record and click the same Save button to pass the edited order entity to the Order entity's SaveEntity static method. The SaveEntity method handles both inserts and updates by checking the value of the OrderID property of the Order entity. Once the entity has been inserted or updated, the DataGridView is reloaded by grabbing the list of orders again and resetting the BindingSource's data source to keep the data fresh.

Using Predicates to Find an Entity

Creating a friendly user interface is important. Users may want to jump to a specific order if they know the Order ID. I handled this by adding a ToolStripTextBox control to the BindingNavigator where the user can enter an entire or partial OrderID to locate. I then added a new toolbar button and an event handler that initiates the search for the order record. Figure 1 shows these controls as well as the tooltip text where the user can find an order by Order ID.

Finding a DataRow inside a DataTable is easy with the Select method or the Find method. But just because I am using custom entities does not mean I have to give up features like this. In this case, I have to find an Order entity in the List<Order> that begins with the value that the user enters in the search control. The List<T> exposes a handful of methods to assist in locating one or more items in its list including the Find, FindAll, FindIndex, FindLast, and FindLastIndex methods. I will use the Find method, which accepts a Predicate<T> as its sole argument.

The Predicate<T> must be a Predicate<Order> since I have a List<Order>. The Predicate is a delegate that searches for an Order entity in the List<Order> that matches the criteria I defined (that it begins with the search value entered in the search field). Before creating the Predicate, I first created a class to assist with the search (shown in Figure 8). The OrderFilter class accepts the order to search for in its constructor. This class also has two methods for finding a specific entity. Each of the methods returns a Boolean value indicating whether there is a match. These methods are the basis of the delegate I will pass to the Predicate.

Figure 8 OrderFilter Class (Close Figure 8)
private class OrderFilter
{
    private int orderID = 0;

    public OrderFilter(int orderID)
    {
        this.orderID = orderID;
    }

    public bool MatchesOrderID(Order order)
    {
        return order.OrderID == orderID;
    }

    public bool BeginsWithOrderID(Order order)
    {
        return order.OrderID.ToString().StartsWith(orderID.ToString());
    }
}

The code that locates the order and repositions the BindingSource is shown in Figure 9. First, I grab a reference to the list of Order entities from the BindingSource to make the code easier to read. Then I create an instance of the OrderFilter class and initialize it with the order ID to search for. Next I create the Predicate and pass to it the OrderFilter class' BeginsWithOrderID method. Finally, I execute the List<Order>'s Find method and pass it the Predicate I just created. This will in turn iterate through the list of Order entities and pass them all to the OrderFilter.BeginsWithOrderID method. The first entity that returns true will be returned and then used to reposition the BindingSource to its index.

Figure 9 Locating the Order (Close Figure 9)
private void toolBtnFindOrderNumber_Click(object sender, EventArgs e)
{
    List<Order> orderList = new List<Order>(
        orderBindingSource.DataSource as BindingList<Order>);
    OrderFilter orderFilter = new OrderFilter(
        Convert.ToInt32(toolTxtFindOrderNumber.Text));
    Predicate<Order> filterByOrderID =
        new Predicate<Order>(orderFilter.BeginsWithOrderID);
    Order order = orderList.Find(filterByOrderID);
    if (order == null)
        MessageBox.Show("No matching Order found",
                         "Not Found", MessageBoxButtons.OK);
    else
    {
        int index = orderBindingSource.IndexOf(order);
        orderBindingSource.Position = index;
    }
}

Wrapping It Up

In this column I demonstrated the power of the .NET Windows Forms binding controls. The binding controls can be used with existing entities or DataSets in your architecture to create a bound form rather quickly. If you find these controls lacking any features, you can customize the form to get the functionality you need. Whether you prefer custom entities or DataSets, both tools can be implemented in a .NET enterprise application using data binding.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences such as VSLive or blogging at codebetter.com/blogs/john.papa.


Office Space: Building Office Open XML Files

Office Space
Building Office Open XML Files
Ted Pattison

Code download available at: OfficeSpace2007_02.exe (338KB)
Contents
Generating a .docx File
A Closer Look at Relationships
The Package Viewer Sample Application
New Developer Features in Word 2007
Separating Data from Presentation in Word 2007
Updating the XML Data Store Programmatically
Summary

Welcome to Office Space, a new column devoted to developing with and for the Microsoft® Office system. In this column, I will be exploring how you can extend and customize Microsoft Office system applications and file formats. With the 2007 Microsoft Office system, Microsoft has introduced new file formats for Word, Excel®, and PowerPoint® based on the Office Open XML File Format specification. Office documents stored in this new format are structured inside ZIP archives known as packages. Inside a package, the actual content is stored in components known as parts. Parts are typically stored as internal XML documents whose content is structured in accordance with published XML schemas.

My November 2006 Basic Instincts column introduced the fundamental concepts of the Office Open XML File Formats. I also discussed how to get up and running with the new packaging API that is part of the Microsoft .NET Framework 3.0. If you are new to working with the Office Open XML File Formats you should read my November column before continuing here because this column builds on the topics I discussed there.

Generating a .docx File

In the November Basic Instincts column, I also walked through writing the code required to generate your first .docx file. I have included a skeleton of the basic code required in order to review the basic steps (see Figure 1). First, you must create a new package file using the Package class exposed by the WindowsBase assembly that is part of the .NET Framework 3.0. Second, you must create one or more parts within the package and write into them whatever content these parts require. In the case of a simple .docx file, all that is required is that you create the part with a Uniform Resource Identifier (URI) of /word/document.xml and write WordprocessingML content into this part to create the classic "Hello World" Word document.

Figure 1 Generating a Basic .docx File (Close Figure 1)
Dim pack As Package = Nothing
pack = Package.Open("basic.docx", FileMode.Create, FileAccess.ReadWrite)

'*** create main document part (document.xml) ...
Dim uri As Uri = New Uri("/word/document.xml", UriKind.Relative)
Dim partContentType As String
partContentType = "application/vnd.openxmlformats" & _
                  "-officedocument.wordprocessingml.document.main+xml"
Dim part As PackagePart = pack.CreatePart(uri, partContentType)

'*** get stream for document.xml
Dim streamPart As New StreamWriter( _
    part.GetStream(FileMode.Create, FileAccess.Write))

Dim xmlPart As XmlDocument = New XmlDocument()
'*** code to generate 'Hello World' WordProcessingML omitted for clarity
xmlPart.Save(streamPart)

streamPart.Close()
pack.Flush()

'*** create the relationship part
Dim relationshipType As String
relationshipType = "http://schemas.openxmlformats.org" & _
                   "/officeDocument/2006/relationships/officeDocument"
pack.CreateRelationship(uri, TargetMode.Internal, _
                        relationshipType, "rId1")
pack.Flush()

'*** close package
pack.Close()

Remember that you must specify a content type whenever you create a new part within a package. The code in Figure 1 includes the content type required for the /word/document.xml part as the second parameter in the call to CreatePart. When you call this method and pass a content type, the packaging API takes care of adding whatever entry is required into the content type item named [Content_Types].xml that exists at the root of any package file.

The last thing I want to call your attention to in Figure 1 is the call to the CreateRelationship method. This is a critical line of code because it establishes a parent/child relationship between the package and the /word/document.xml part. As you will see in the next section, creating relationships is critical. If you fail to establish the proper relationships, Word will not be able to load and render the document.

A Closer Look at Relationships

The structure of a package in the Office Open XML File Formats is heavily dependent upon relationships. As mentioned, if you create parts, but fail to associate them to the package through relationships, then consumer applications (such as Word) will not be able to recognize them. That's because every part must have a relationship or a chain of relationships that associate it with its containing package.

There are two types of relationships. First, there are package relationships that define an association between a package and its top-level parts. Figure 2 shows that the typical top-level parts in a .docx file created with Microsoft Office Word 2007 are /docProps/app.xml, /docProps/core.xml and /word/document.xml. Second, there are part relationships that define a parent/child relationship between two parts within the same package. The part /word/document.xml typically has relationships to several different child parts such as /word/settings.xml and /word/styles.xml.

Figure 2 Package Contents
Figure 2 Package Contents

The Office Open XML File Format specification states that every part inside a package must be associated either directly or indirectly with the package itself. A part such as /word/document.xml is directly associated with the package through a package relationship. Another part such as /word/styles.xml is associated with the package indirectly because it is associated with the top-level part /word/document.xml that is, in turn, associated with the package.

A very important concept to understand is that a consumer application must be able to discover any part within a package by enumerating through its relationships. In fact, when you are writing your own applications that read packages created by other applications such as Word and Excel, you are also encouraged to discover the existing parts by enumerating relationships as well.

Let's look at an example. Imagine you want to write the code to open an existing .docx file so you can print the XML contents of the part /word/document.xml to the console window. You can take an easy approach and access this part using a hardcoded URI:

Dim pack As Package = Package.Open("c:\Data\Hello.docx", _
                                   FileMode.Open, FileAccess.Read)
Dim partUri As Uri = New Uri("/word/document.xml", UriKind.Relative)
Dim part As PackagePart = pack.GetPart(partUri)

'** now add code to program against part

This code gains access to the /word/document.xml part, which then makes it possible to open a stream and read its content. Now let's accomplish the same goal of getting the part a different way. This time I am going to enumerate through the package relationships looking for a specific relationship type. In particular, I will look for the part that is related to the package with the relationship type used for the main document part in a Word document:

Dim pack As Package = Package.Open("c:\Data\Hello.docx", _
    FileMode.Open, FileAccess.Read)

Dim relationshipType As String = _
    "http://schemas.openxmlformats.org" & _
    "/officeDocument/2006/relationships/officeDocument"

Dim rel As PackageRelationship, partUri As Uri

For Each rel In pack.GetRelationshipsByType(relationshipType)
    partUri = PackUriHelper.ResolvePartUri( _
        rel.SourceUri, rel.TargetUri)
    Dim part As PackagePart = pack.GetPart(partUri)
    '** now add code to program against part
Next

As you can see, this code calls the GetRelationshipsByType method on the Package object, passing the string name of the target relationship type. This technique allows you to enumerate through all the package relationships based on that type. In the case of the discovering parts with the officeDocument relationship type shown here, there should only be one part within the package.

After finding the correct PackageRelationship object, the code dynamically builds the URI to the target part using the PackUriHelper class provided by the .NET Framework packaging API. You can acquire the URI object you need by calling ResolvePartUri and passing the SourceUri and TargetUri of the current PackageRelationship object. Once you have dynamically built the URI for the target part, you can call the GetPart method.

Also keep in mind that a Package object is a disposable object and should be managed accordingly. Therefore, your code to access items inside the package should be structured within a Using statement, like this:

    Using pack As Package = _
    Package.Open("c:\Data\Hello.docx", _
    FileMode.Open, FileAccess.Read)
    '*** code to access package goes here
End Using

Now that you have seen the individual pieces, let's look at how everything fits together. Figure 3 shows code from the sample console application named DocumentReader that accompanies this month's column. This code has been written to open the package and discover the target part using a target relationship type. The code then builds a URI so it can open the target part and acquire stream-based access to its content. When you run this console application, it produces the output shown in Figure 4.

Figure 3 Discover Target Parts (Close Figure 3)
Dim relType As String
Dim rel As PackageRelationship
Dim partUri As Uri

'*** define target relationship type
relType = "http://schemas.openxmlformats.org" & _
    "/officeDocument/2006/relationships/officeDocument"

Using pack As Package = Package.Open("minimal.docx", _
    FileMode.Open, FileAccess.Read)

    For Each rel In pack.GetRelationshipsByType(relType)
        partUri = PackUriHelper.ResolvePartUri( _
            rel.SourceUri, rel.TargetUri)
        Dim part As PackagePart = pack.GetPart(partUri)
        Dim partStream As Stream, partReader As StreamReader
        partStream = part.GetStream(FileMode.Open, FileAccess.Read)
        partReader = New StreamReader(partStream)
        '*** print contents of part to console window
        Console.WriteLine(partReader.ReadToEnd)
        '*** close stream
        partStream.Close()
    Next

End Using
Figure 4 Output from Console Application
Figure 4 Output from Console Application(Click the image for a smaller view)
Figure 4 Output from Console Application
Figure 4 Output from Console Application (Click the image for a larger view)

The Package Viewer Sample Application

Now that you have seen how to discover and access parts within a package using its relationships, let's take this idea a little bit further. As I have mentioned, the specification for the Office Open XML File Format states that all parts within a package must be discoverable through relationships. Therefore, it's possible to write an application that inspects a package and displays all the parts inside of it.

The month's column is accompanied by a second sample application named Package Viewer, a Windows® Forms-based application that populates a tree view control with nodes that show a package and all of its parts nested within a hierarchy of relationships. Figure 5 shows the Package Viewer application in action. You will notice that the application also provides more information about the package and specific parts when you click on a node in the tree view. For each part, you can see its content type, its parent and the relationship type that associates it with its parent.

Figure 5 Package Viewer Inspects the Parts in a Package
Figure 5 Package Viewer Inspects the Parts in a Package(Click the image for a smaller view)
Figure 5 Package Viewer Inspects the Parts in a Package
Figure 5 Package Viewer Inspects the Parts in a Package (Click the image for a larger view)

Package Viewer allows the user to select a package file using the standard Open File dialog. Once the user selects a file, the application enumerates through all the package relationships to discover the top-level parts. Figure 6 shows the basic structure of the code, omitting some of the details of populating the Windows tree view control with TreeNode objects.

Figure 6 Populating the Tree View (Close Figure 6)
Dim rootNode As TreeNode
rootNode = New TreeNode(PackageName, 0, 0)

For Each rel As PackageRelationship _
    In CurrentPackage.GetRelationships()

    Dim PartUri As Uri = PackUriHelper.ResolvePartUri( _
        rel.SourceUri, rel.TargetUri)
    Dim part As PackagePart = CurrentPackage.GetPart(PartUri)
    Dim topLevelPartNode As TreeNode = New TreeNode( _
        part.Uri.OriginalString, 1, 1)

    '*** call helper method to discover child controls
    PopulateChildPartNode(part, topLevelPartNode)

    '*** add node to
    rootNode.Nodes.Add(topLevelPartNode)
Next

treePackageContents.Nodes.Add(rootNode)

Calling the GetRelationships method on the Package class allows the application to enumerate through all the top-level parts inside the package. If you look inside the For Each loop, you can see that each iteration for a specific top-level part creates a TreeNode object and also calls a helper method named PopulateChildPartNode to deal with the discovery of any related child part and to create TreeNode objects for them as well. Now let's examine the code inside the PopulateChildPartNode method (see Figure 7).

Figure 7 PopulateChildPartNode Method (Close Figure 7)
Private Sub PopulateChildPartNode( _
    ByVal part As PackagePart, ByVal partNode As TreeNode)

    For Each rel As PackageRelationship In part.GetRelationships()

        Dim ChildPartUri As Uri = _
            PackUriHelper.ResolvePartUri(rel.SourceUri, rel.TargetUri)
        Dim ChildPart As PackagePart = part.Package.GetPart(ChildPartUri)

        Dim ChildPartNode As TreeNode = _
            New TreeNode(rel.TargetUri.OriginalString, 1, 1)

    '** use recursion
        PopulateChildPartNode(ChildPart, childPartNode)

        partNode.Nodes.Add(ChildPartNode)

    Next

End Sub

You can see that each PackagePart object exposes a GetRelationships method. This makes it possible to enumerate through the child parts for each top-level part. However, you should keep in mind that the relationship hierarchy of parts can grow to an arbitrary number of levels with child parts having child parts having child parts, and so on. Therefore, the application is designed to use recursion by having the PopulateChildPartNode call itself. This makes it possible to write a single method to crawl as many levels as necessary to populate the tree view control with all parts that can been found within the current package.

If you examine the code for the Package Viewer application, you will see that it employs a user-defined structure and the Tag property of each TreeNode object to track various attributes for the entity that each node represents. For example, there's information to track whether each node represents a package, a part, or an external relationship. For nodes that represent parts, there is information to track the part's content type, parent, and the relationship type it has with its parent. When you click on a node in the tree, this information is used to populate the controls on the right-hand side of the application's main form.

Package Viewer also provides the functionality to display the contents of any XML-based parts within the package. It accomplishes this by reopening the package and acquiring stream-based access to the target part. Then the XML-based content of the part is written to a temporary file. Finally, the file is loaded into to the Windows Forms WebBrowser control, which displays the XML content with color coding and collapsible sections. Hopefully, you will find this sample application useful as you begin learning about the parts and the XML required to work with Office Open XML File Format documents.

Now you should have some idea of what's required to generate documents for Word, Excel, and PowerPoint using the Office Open XML File Formats. In theory, it's simple. All you have to do is create a new package file, add the required parts, and fill them up with XML content structured in accordance with the appropriate XML schemas. But while the theory is simple, getting up to speed on the details will take some time. It will also require you to read through the relevant sections of the file format specifications, which can be downloaded from OpenXmlDeveloper.org.

If you are going to work with Word documents, you must learn what type of parts go inside a package and how they must be structured in terms of content types and relationships. You also have to learn how to generate the WordprocessingML that goes inside each of these parts. If you want to work with Excel spreadsheets, the content types and relationships will be different. Instead of using WordprocessingML, you'll have to learn SpreadsheetML. It will take an investment on your part if you want to generate the XML required to create documents from scratch that contain things like tables, graphics, and formatting.

New Developer Features in Word 2007

I'll conclude this month's column by showing you a technique for creating professional-looking Word 2007 documents that will not require writing code to generate WordprocessingML. I'll start by introducing two new features of Word 2007 that can be used when working with documents stored in the new Office Open XML File formats. The first feature is the XML data store, which allows you to embed one or more user-defined XML documents as parts inside a .docx file. The second feature is content controls, which are user interface components defined inside the /word/document.xml part that support data entry and data binding.

If you want to experiment with using content controls, you should first go to the Word Options dialog in Word 2007 and, on the Popular tab, make sure the Show Developer tab in the Ribbon option is checked. On the Developer tab you will see the set of controls shown in Figure 8. All of these controls can be added to a Word document.

Figure 8 Developer Tab Controls
Figure 8 Developer Tab Controls

Note that you will only be able to add content controls to Word .docx files saved in the new format. They are not supported in .doc files because they cannot be defined using the older binary format. However, when working in the new format, content controls can be added to provide user input elements into a Word document. For example, you can construct a Word document that solicits the user for certain pieces of information to complete a business document, such as a calendar to select a date.

Keep in mind that content controls have two different modes: edit and display. Edit mode allows the user to do things such as type in text, pick a date from a date picker, or select an item from a dropdown list. Display mode is optimized for rendering and printing. In other words, the editing aspects of content controls are invisible to the user after they have finished being edited and they are no longer in edit mode.

Separating Data from Presentation in Word 2007

While the XML data store and content controls are two separate features of Word 2007 that can be used independently of one another, they really pack a punch when used together. For example, you can embed an XML file with the data for a customer or an invoice inside a Word document. Then you can bind content controls to data inside the XML file using XPath expressions. This effectively allows you to separate your data from the formatting instructions that tell Word how to display it.

Let's start with a simple example. You can follow along by examining the sample Word document named CustomerEntry.docx in the download accompanying this month's column. I encourage you to inspect this file with Package Viewer as this allows you to quickly see how all the pieces fit together. However, when actually building one of these documents yourself, you should change the extension of the .docx file to .zip so you can open the package file directly and manipulate its contents as I discussed in the November 2006 Basic Instincts column.

CustomerEntry.docx contains a simple user-defined XML document that contains customer data:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Customer xmlns="http://litware.com/2006/customer">
  <Name>Brian Cox</Name>
  <Address>2732 Baker Blvd</Address>
  <City>Eugene</City>
  <State>OR</State>
  <Zip>97403</Zip>
</Customer>

This XML document is embedded as a part inside the .docx file using a URI of /customXml/item1.xml. Note that you can name this part something other than item1.xml. However, I want to be consistent with the naming scheme that Word 2007 uses when it creates and renames parts in the XML data store.

Next, you need to create a way to identify the content within the customer XML document. You do this by defining a dataStoreItem with an identifying GUID. This is accomplished by creating a part named /customXml/itemProps1.xml and establishing a relationship between this part and its parent part, /customXml/item1.xml. Here's an example of what the contents of /customXml/itemProps1.xml should look like:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<ds:dataStoreItem ds:itemID="{D5CB27EE-AE18-48C7-B53F-E921F4653E70}"
      xmlns:ds="http://schemas.openxmlformats.org/officeDocument/2006/
      customXml">
  <ds:schemaRefs>
    <ds:schemaRef ds:uri="http://litware.com/2006/customer"/>
  </ds:schemaRefs>
</ds:dataStoreItem>

The next step is to create a part relationship between /word/document.xml and /customXml/item1.xml. The relationship should define the parent part as /word/document.xml and child part as /customXml/item1.xml. The relationship type should be established using the following string:

http://schemas.openxmlformats.org/officeDocument/2006/relationships/customXml

After you have gone through all these steps to set up a user-defined XML document as an identifiable dataStoreItem, you can access its data from /word/document.xml. For example, you can write Visual Basic® for Applications (VBA) code behind a macro-enabled Word 2007 document that retrieves a CustomXmlPart object from the ActiveDocument object and pulls the required customer data out of the embedded XML document. In the example here, I am going to create content controls in /word/document.xml and bind them to particular elements within the XML document with the customer data.

Unfortunately, Word 2007 provides no way through its user interface to bind content controls to elements within the XML data store. Therefore, you are required to make direct edits to the /word/document.xml using an XML editor such as Visual Studio® 2005. Here's an example of the WordprocessingML element you can add to the /word/document.xml part to create a binding to the Name element within the Customer element of the user-defined XML document:

<w:sdtPr>
  <w:id w:val="256125470"/>
  <w:dataBinding
    w:prefixMappings="xmlns:ns0='http://litware.com/2006/customer'"
    w:xpath="/ns0:Customer[1]/ns0:Name"
    w:storeItemID="{D5CB27EE-AE18-48C7-B53F-E921F4653E70}"/>
  <w:text/>
</w:sdtPr>

The dataBinding element contains a storeItemID attribute that references the GUID of the dataStoreItem that holds the customer data. The dataBinding element also contains an XPath attribute which defines an XPath expression to bind the content control to a specific element within the user-defined XML file. Once you have updated the /word/document.xml file to contain all the dataBinding elements you need to display the relevant data from the user-defined XML file, you can construct a Word document.

While it might take some time to become fluent with the technique of manually constructing Word documents that contains user-defined XML files and bound content controls, it is worth the effort. It provides an elegant solution to separating your data from the presentation of that data. Once you have created a minimal Word document with bound content controls, you can do the rest of your work for making the document look professional directly within Word. For example, you can add text and a logo just as would any other Word user. You can create new tables and sections and simply drag and drop the content controls where you would like them to appear.

It's also important to keep in mind that bound content controls can provide two-way synchronization with data in a user-defined XML file. While users can see whatever data you have added to the user-defined XML file, they can also update this data. When a user updates the data in a content control and saves the document, those changes are written back to the user-defined XML file. This makes it very easy to extract the updated data in an XML format that is very easy to use.

Updating the XML Data Store Programmatically

Now you have seen how to bind content controls to a user-defined XML document in the XML data store. The final step is writing the code that generates a new instance of that XML document with the data for a particular customer and to overwrite the XML document inside an existing Word document that has been set up with the proper bound content controls.

The download for this month's column includes one more sample application named Letter Generator, an ASP.NET 2.0 application that works with a Word document named LetterTemplate.docx. This Word document has an embedded XML document shown in Figure 9 that contains all the content required for a Litware Corporation employee to draft a letter to a customer.

Figure 9 User-Defined XML Document (Close Figure 9)
<?xml version="1.0" encoding="utf-8"?>
<LitwareLetter xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:xsd="http://www.w3.org/2001/XMLSchema"
               xmlns="http://litware.com/2006/letters">
  <Customer>
    <ContactFirstName>Yoshi</ContactFirstName>
    <ContactLastName>Latimer</ContactLastName>
    <Company>Hungry Coyote Import Store</Company>
    <Address> 516 Main St.</Address>
    <City>Elgin</City>
    <State>OR</State>
    <Zip>97827</Zip>
  </Customer>
  <Date>October 16, 2006</Date>
  <Body>Thanks for your recent purchase at Litware </Body>
  <Employee>
    <Name>Rob Verhoff</Name>
    <Title>Director of Accounting</Title>
  </Employee>
</LitwareLetter>

The Letter Generator application has been written to generate XML documents in this consistent form by pulling information data out of three different tables in an Access™ MDB file. I will let you use your imagination about how you could extend this sample to extract data from other data sources such as SQL Server™ or a custom Web service.

When a user retrieves the page named GenerateLetter.aspx, the user is presented with WebControls that allow for the selection of a customer, a letter type, and the employee who is intended to sign the letter. Once the user has chosen all the correct parameters for the letter to be drafted, the user can then click the button with the caption of Generate Letter to begin the server-side processing of the request.

The server-side code behind the Generate Letter button begins by loading an existing Word 2007 document named LetterTemplate.docx into a MemoryStream object and opening it up with the .NET packaging API. Next, the code creates a new XML document according to the parameters requested by the user. It then opens the /customXML/item1.xml part and overwrites its content with the newly created XML document. Finally, the code closes the package file and streams the resulting Word document back to the user using the same technique I discussed in the November 2006 Basic Instincts column. The end result is that the user is able to automate the process of drafting a customer letter.

Summary

This month's column has continued my examination of working with the Office Open File Formats. You should now understand how relationships and relationship types play a key role in defining the structure for a Word 2007 document. You have also seen a powerful new technique for creating Word 2007 documents that separates data from presentation using content controls bound to elements in a user-defined XML document. Hopefully, this has fueled your imagination and given you a starting point for creating applications and components that automate the process of creating standard business documents for users that rely on Word, Excel, and PowerPoint.

Send your questions and comments for Ted to mmoffice@microsoft.com.

Ted Pattison is an author, trainer, and SharePoint MVP who lives in Tampa, Florida. Ted is writing a book titled Inside Windows SharePoint Services 3.0 for Microsoft Press and delivers advanced SharePoint training to professional developers through his company, Ted Pattison Group.


Test Run: AJAX Test Automation

Test Run
AJAX Test Automation
Dr. James McCaffrey

Code download available at: TestRun2007_02.exe (574KB)
Contents
The Web Application under Test
The Test Automation
Inside asyncCall
Inside Delay
Building the Test Page
Extending the Test Harness
Wrapping Up

The popularity of Web applications that use AJAX (Asynchronous JavaScript And XML) technology has increased steadily over the past year. When written correctly, AJAX can yield significant improvements in performance and user experience compared with non-AJAX Web applications. However, because AJAX Web applications work asynchronously, traditional synchronous test automation techniques generally don't work. In this month's column, I present a technique that allows you to write lightweight test automation to verify the functionality of AJAX Web applications.

The best way for me to show you where this column is headed is with a couple of screenshots. Figure 1 shows a simplistic but representative ASP.NET Web application that uses AJAX technology. The application simulates a mapping tool such as Microsoft® Windows Live™ Local. When a user clicks on one of the directional button controls, the application fetches the appropriate map image from the Web server and displays the new map in the center area. What is not apparent from the screenshot is that the Web application uses Microsoft ASP.NET AJAX to send and retrieve map images asynchronously. Of course, your Web application will be much more complex, but the technique I'll show you easily generalizes to arbitrarily complex applications and works for any AJAX-enabled application, regardless of the implementation technology.

Manually testing the application in Figure 1 would be time consuming, inefficient, and tedious. A better testing approach is to write lightweight test automation as shown in Figure 2. The test harness is a simple HTML page that consists of two frames. The right-hand frame hosts the Web application without any modifications or instrumentation. The left-hand frame hosts a single HTML page that contains JavaScript code that manipulates the Web application in the other frame, using the Internet Explorer® Document Object Model (DOM).

Although the testing technique is specifically designed to handle applications that use asynchronous operations, the technique also works for Web applications that use the traditional synchronous HTTP request/response mechanism. In this column, I will briefly describe the Web application under test so you'll understand why traditional test automation techniques are ineffective on AJAX applications. Then, I'll explain in detail the test harness that generated the image in Figure 2, and I'll describe how you can modify and extend the technique presented here to meet your own needs. In the end, I think you'll find this technique a useful addition to your developer and tester skill sets.

Figure 1 AJAX Web Application under Test
Figure 1 AJAX Web Application under Test(Click the image for a smaller view)
Figure 1 AJAX Web Application under Test
Figure 1 AJAX Web Application under Test (Click the image for a larger view)

The Web Application under Test

I created the AJAX map application shown in Figure 1 using the ASP.NET AJAX code library, which greatly simplifies creating AJAX applications. Instead of using real map images, I simply used nine dummy images containing the digits 1 through 9, named 1.jpg through 9.jpg. (see Jeff Prosise's Wicked Code column in this issue for an example of using Microsoft TerraServer to get real map images.) My miniature world consists of a 3-by-3 grid with 1, 2, and 3 in the first row, 4, 5, and 6 in the second row, and 7, 8, and 9 in the third row. 5 is the center of the world and if a user clicked on the North control button from 5, the application should display 2. The logic for the North button is shown in Figure 3.

Figure 3 Code Behind the North Button (Close Figure 3)
protected void Button1_Click(object sender, EventArgs e)
{
  string imageUrl = Image1.ImageUrl.ToString();
  if (!(imageUrl == "~/1.JPG" || imageUrl == "~/2.JPG" ||
        imageUrl == "~/3.JPG"))
 {
    string oldNum = imageUrl.Substring(2, 1);
    int oldNumAsInt = int.Parse(oldNum);
    int newNumAsInt = oldNumAsInt - 3;
    string newNum = newNumAsInt.ToString();
    string newImageURL = imageUrl.Replace(oldNum, newNum);
    Image1.ImageUrl = newImageURL;
  }
}
Figure 2 Sample Test Run
Figure 2 Sample Test Run(Click the image for a smaller view)
Figure 2 Sample Test Run
Figure 2 Sample Test Run (Click the image for a larger view)

If the current map is in the top row (1, 2, 3) of the world, a click on North has no effect. Otherwise, I obtain the digit of the current map location, subtract 3, determine the corresponding new JPEG image, and update the Image control's ImageUrl property. The handlers for the South, East, and West button controls are similar.

Let me emphasize that I'm purposely not using good coding practices here in order to keep the application code short and easy to read. In a real Web application, your code logic would likely access and fetch data from a backend SQL or XML store and update the application state using that data. Because the technique in this column tests a Web application through the Internet Explorer DOM, it doesn't matter how the application state is determined-any user input causes the app state to change, which is reflected in the app's UI, which you can access through the Internet Explorer DOM.

A traditional, non-AJAX approach posts an HTTP request to the Web server, passing request information in a Form object or a query string. Without AJAX technology, the map application works, but has two drawbacks. First, because the HTTP request/response mechanism is synchronous, while the request is being processed by the Web server, the user loses most ability to interact with the Web application until the response is returned to the client. Second, in most cases the HTTP request causes an entire response page to be created, so when the response is received by Internet Explorer, the entire page is repainted. This leads to an annoying page flicker effect if the request/response is relatively quick-or worse, a page blank-out effect if the action is slower.

AJAX solves both of these problems. AJAX works by sending an XMLHTTP request instead of an HTTP request. XMLHTTP requests are asynchronous. So while the XMLHTTP request is being processed behind the scenes, the user can continue interacting with the Web application. And when the XMLHTTP response is received, the Internet Explorer DOM can be used to repaint only the area of the Web page that holds the new data, rather than having to repaint the entire page.

Instead of writing "raw" JavaScript to enable AJAX on my ASP.NET map application, I used the Microsoft ASP.NET AJAX framework. Using it is remarkably easy, as its installation provides a Visual Studio® AJAX Web Site template. Selecting this template adds the necessary assembly references to your Web application project. To enable AJAX functionality in my map application, I added the following tag to my source file:

<asp:ScriptManager ID="sm" runat="server" />

Then I wrapped my Image control (the control that I want to update after an asynchronous request-response) in an ASP.NET AJAX UpdatePanel control like this:

<asp:UpdatePanel ID="up1" runat="server">
  <ContentTemplate>
    <asp:Image ID="Image1" runat="server" ImageUrl="~/5.JPG"
               (other attributes omitted) />
  </ContentTemplate>
  <Triggers>
    <asp:AsyncPostbackTrigger ControlID="Button1" EventName="Click" />
    <asp:AsyncPostbackTrigger ControlID="Button2" EventName="Click" />
  </Triggers>
</asp:UpdatePanel>

And that's all there is to it. Very nice! The framework takes care of all the messy details such as creating the XMLHTTP object, listening for the asynchronous response, error handling, cross-browser differences, and so on.

Notice that I enable AJAX asynchronous request/responses only when there is a click event for the Button1 (North) and Button2 (South) controls, rather than enabling for all four directional controls. I did this on purpose in order to illustrate that the testing technique in this month's column works for both asynchronous and synchronous requests. And if you try the application yourself using the code download that accompanies this article, you'll see a significant performance difference between North and South requests (AJAX) and East and West requests (non-AJAX).

Most traditional test automation techniques just don't work with AJAX Web applications. The most fundamental way to test a Web application's functionality is to programmatically send an HTTP request that corresponds to the request generated by some user input to the Web server, fetch the HTTP response, and then examine the response in order to determine a pass/fail result. This approach doesn't work for AJAX applications because they use special XMLHTTP requests.

Another traditional approach is to use JavaScript to manipulate the Internet Explorer DOM to send a request to the Web server, wait until the onload event fires (signaling that the response has been received by the client and has been loaded), then use JavaScript and the Internet Explorer DOM to examine the new state of the Web page to determine a pass/fail result. The problem with this approach is that because AJAX works asynchronously, you can't use the onload event to tell when the response has been received by the client. But you can modify this approach to create lightweight test automation for AJAX applications. Instead of using the onload event to signal when a response has been received, you can write code that watches the application for some expected change in application state and then transfers control to a callback function.

The Test Automation

The AJAX application test harness system illustrated in the screenshot in Figure 2 consists of three files. Figure 4 shows a block diagram of the overall structure of the harness. The harness itself, TestHarness.aspx, is a simple Web page with two frames. The right-hand frame holds the AJAX Web app under test. The left-hand frame holds a test scenario page that has some minimal UI and all the JavaScript code to manipulate and examine the Web app under test. For consistency, I decided to make all three pages .aspx files even though I could have used basic HTML for the test harness and test scenario pages. Here is the complete source for the TestHarness.aspx page:

<html>
<head>
  <title>Test Harness for AJAX Web Apps</title>
</head>
  <frameset cols="45%,*">
    <frame src="http://localhost/AjaxTest/TestScenario001.aspx"
      name="leftFrame">
    <frame src="http://localhost/AjaxApplication/Default.aspx"
      name="rightFrame">
  </frameset>
</html>

Figure 4 Structure of the Test Harness
Figure 4 Structure of the Test Harness(Click the image for a smaller view)
Figure 4 Structure of the Test Harness
Figure 4 Structure of the Test Harness (Click the image for a larger view)

As you can see, the main test harness page is really just a container file. A test scenario run starts by calling a program-defined function asyncCall, which causes the application under test to send an asynchronous XMLHTTP request to the Web server and also calls a program-defined delay function. The delay function loops while the Web server processes the request and returns the asynchronous response, probing the state of the Web application every so often. After the response is received by the client and the app state is updated, the delay function finds the Web page state change, and a new asynchronous call can be made.

Inside asyncCall

The heart of my AJAX test automation technique is a pair of program-defined functions, asyncCall and delay, which work together. The asyncCall method is shown here:

function asyncCall(action, checkFunc, arg, callback, pollTime)
{
  numTries = 0;
  action();
  window.setTimeout("delay(" + checkFunc + ", " + "'" + arg +
    "'" + ", " + callback + ", " + pollTime + ")", pollTime);
}

I pass in five arguments. The first parameter, action, is a function pointer to some routine which uses the Internet Explorer DOM to fire an action that initiates an asynchronous XMLHTTP request. The second parameter, checkFunc, is a function pointer to a routine that returns true when the state of the Web application indicates that the asynchronous response has completed. The third parameter, arg, is an argument that is passed to the checkFunc function. The fourth parameter, callback, is a function pointer to the routine to invoke when the asynchronous response has completed. The last parameter, pollTime, specifies how long, in milliseconds, to wait between calls in the partner delay function.

Inside the asyncCall function, I first set a global numTries counter to zero. I use this variable to keep track of how many times I've entered the delay function so that I can exit after a specified number of attempts to find the state that indicates the asynchronous response has occurred. Next I call the action argument, which in turn causes the asynchronous request to fire. You should note that the parentheses appended to the function name is the syntax mechanism to call the function.

Now comes the real trick: I call the intrinsic window.setTimeout function. As you can see, setTimeout accepts two arguments. The first argument is a JavaScript statement to execute one time only. The second argument is a time, in milliseconds, to delay before executing the first argument. If you examine the code in asyncCall, you'll see that my first argument to setTimeout resolves to this:

delay(checkFunc, 'arg', callback, pollTime)

The second arg is just pollTime. In short, after a delay of pollTime milliseconds, I invoke the program-defined delay function.

A call to the asyncCall function looks something like this:

asyncCall(clickNorth, imgIsTwo, "2", clickSouth, 200);

This call can be interpreted as "invoke a function named clickNorth, then go into a delay loop, calling a function imgIsTwo with an argument of '2' every 200 milliseconds, and when imgIsTwo finally returns true, transfer control to a function named clickSouth."

The single-quote characters around arg in the definition of the asyncCall method are important. Without the single quotes in the asyncCall definition, the parameter would be passed by reference, as shown here:

asyncCall(doThis, findX, "X", doThat, 200);

A call like this would generate an error saying that variable X is not defined. With the single quotes, the parameter is passed by value, which is what you want in this situation.

There is an interesting alternative way to call the window.setTimeout function inside asyncCall. Normally you'd write something like this:

window.setTimeout("delay(" + checkFunc + ", " + "'" + arg +
    "'" + ", " + callback + ", " + pollTime + ")", pollTime);

Instead, you can use JavaScript's anonymous function feature and write this:

window.setTimeout(
    function(){delay(checkFunc, arg, callback, pollTime);},
    pollTime);

Because the quote characters and string concatenation are removed, this code is a bit cleaner than the non-anonymous alternative. It's also slightly more efficient because the non-anonymous alternative requires the browser to create a new script environment to process the script. However, without the single quote characters delimiting the arg argument to checkFunc, it is not obvious that arg is being passed by value (although a simple comment would clear up any possible confusion).

Inside Delay

Now here is my partner to asyncCall, the delay function:

function delay(checkFunc, arg, callback, pollTime)
{
  ++numTries;

  if (numTries > maxTries) finish();
  else if (checkFunc(arg)) callback();
  else window.setTimeout("delay(" + checkFunc + ", " + "'" + arg +
      "'" + ", " + callback + ")", pollTime);
}

The delay function accepts four arguments that correspond exactly to the last four arguments that were passed to the calling asyncCall function.

Inside delay, I first increment the global numTries counter. Recall that I'm using this to keep track of how many times I enter the delay function-or put another way, how many times I've checked the state of the Web application to see if the asynchronous response has completed. If this global counter is greater than a global maxTries constant, then either my Web application is timing out on the server side or the XMLHTTP response is incorrect, so I transfer control to a finish function where I can determine what happened.

If delay hasn't timed out, I call the checkFunc function to test whether the condition that indicates a correct response is true. If checkFunc returns true, then I can call the callback function to continue the test scenario execution. But if the checkFunc function returns false, I need to continue waiting, so I call the intrinsic setTimeout function again, which will wait for pollTime milliseconds and then call delay again.

My delay function is not quite recursive because it doesn't call itself directly, but it is self-referential because it calls itself indirectly via the setTimeout function. The net effect is a waiting loop that will either transfer control to a finish function (if the maximum number of times in the delay loop is exceeded) or a callback function (if some Web application state becomes true). This approach is one of those things that seems obvious in hindsight, but isn't so obvious before you see the solution. There are other approaches, of course, but this one has proven to be simple and effective in practice.

Just as in the asyncCall function, instead of calling the window.setTimeout function using a named function argument, you can use the anonymous function feature like this:

window.setTimeout(
    function(){delay(checkFunc, arg, callback);},
    pollTime);

Building the Test Page

With the asyncCall and delay functions taken care of, I can now construct my test scenario page. The overall structure of the test scenario is presented in Figure 5.

Figure 5 Test Scenario Structure (Close Figure 5)
<html>
<head>
  <script type="text/javascript">
    var maxTries = 10;
    var numTries;
    var polling = 200; // milliseconds

    function asyncCall(action, checkFunc, arg, callback, pollTime)
    {
      // code here
    }

    function delay(checkFunc, arg, callback, pollTime)
    {
      // code here
    }

    // other functions go here
  </script>
</head>
<body bgcolor="#22EE66">
  <h3>This is the test scenario script page</h3>
  <p>Actions:</p><p><textarea id="comments" rows="16"
                      cols="34"></textarea></p>
  <input type="button" value="Run Test" onclick="runTest();" />
</body>
</html>

The test scenario page <body> section consists of just a title, a textarea with ID "comments" to display comments, and a button control to launch the automation. The <head> section contains all of my JavaScript code. I declare and initialize a global maxTries constant to specify the maximum number of times I'm willing to enter the delay function, checking for some condition that tells me that the asynchronous response has completed. Global variable numTries keeps track of how many times I've entered the delay function. Global constant polling sets the delay time between successive calls to my delay function.

Depending on the complexity of your AJAX Web application under test, you may have to modify the values of maxTries and polling. Here, 10 times through with a delay of 200 milliseconds between calls is just 2 seconds, which may not be long enough for your Web server to process the XMLHTTP request and return the response to the client.

The test scenario begins by calling a runTest function:

function runTest()
{
  try
  {
    logRemark("Test Scenario #001");
    logRemark("Starting test run\n");
    step1(); // starting at 5, go N to 2
  }
  catch(ex) { logRemark("Fatal error: " + ex); }
  }
}

All I do is display a couple of messages in the <textarea> on the harness body with a program-defined logRemark function, and then transfer control to a step1 function. I wrap my code in a simple try/catch block so that any exceptions that are thrown during the test run will bubble up and be caught. My logRemark function is very simple:

function logRemark(comment)
{
  var currComment = document.all["comments"].value;
  var newComment = currComment + "\n" + comment;
  document.all["comments"].value = newComment;
}

I retrieve the current contents of the <textarea> element with ID "comments," append a newline character and the new comment text to the current contents, and replace the <textarea> contents with the updated comment set. This isn't an efficient approach, but in lightweight test automation, simplicity considerations often outweigh efficiency concerns.

The step1 function begins the manipulation of my AJAX Web application under test:

function step1()
{
  logRemark("Clicking North, waiting for '2'");
  asyncCall(clickNorth, checkImageSrc, "2", step2, polling);
}

After logging a comment, I call the asyncCall function, which is the key to my test automation. The call here means: call function clickNorth and then delay loop until checkImgSrc("2") returns true, pausing 200 milliseconds between checks, and then transfer control to function step2. If the delay loop is called more than 10 (maxTries) times, control will be transferred to a finish function.

The clickNorth function is very simple:

function clickNorth()
{
  var btnNorth = parent.rightFrame.document.all["Button1"];
  if (!btnNorth) throw "Did not find btnNorth";
  btnNorth.click();
}

I use the Internet Explorer DOM to obtain a reference to the Button1 control and then invoke its click method. Notice that because my automation is in one frame and my AJAX application is in another frame, in order to access controls on the app from my test script.I have to use the parent keyword to go "up" one level and then use the frame ID of the app container. I like to use the document.all collection to get references to Web page controls, but some of my colleagues prefer to use the getElementById method like this:

var btnNorth = parent.rightFrame.document.getElementById("Button1");

The checkImageSrc function tells the automation when the Image1 control on the AJAX app under test has been updated by an asynchronous response:

function checkImageSrc(target)
{
  try
  {
    var s = parent.rightFrame.document.all["Image1"].src;
    return s.indexOf(target) >= 0;
  }
    // traps case where Image1 is not yet loaded
    // logRemark("Error in checkImageSrc(): " + ex);
  catch(ex) { return false; }
}

Basically, I just check the src property of the Image1 control to see if the property contains some target string. For example, the call checkImageSrc("2") will return true if the image src property contains "2" as in "~/2.jpg" which is the file name I used for map 2. When you write your own AJAX automation you will have to create appropriate check functions. For example, suppose your Web app updates a TextBox control with ID "TextBox1". One possible check function is:

function checkTextBoxValue(target)
{
  try
  {
    var s = parent.rightFrame.document.all["TextBox1"].value;
    return s.indexOf(target) >= 0;
  }
    // traps case where TextBox1 is not yet loaded
    // logRemark("Error in checkTextBoxValue(): " + ex);
  catch(ex) { return false; }
}

Notice the try/catch block in my checkImageSrc function. Unlike most situations where try/catch is used to trap an unexpected error condition, here I'm using try/catch as part of my normal function logic. The idea is that during the time the Web page is being repainted, you cannot get a reference to the Image1 control and any attempt to do so will throw an exception. This doesn't necessarily mean the Web application has an incorrect state, it just means the application has an incomplete state, so I want to catch the exception and return false. You can observe this behavior by un-commenting the call to logRemark and making the polling time very short (say, 10 milliseconds).

By the way, using try/catch as part of normal function logic is usually considered bad coding style, but in this case, the simplicity gained makes using try/catch for normal logic flow acceptable, in my opinion.

My test scenario script defines functions step2, step3, step4, and step5 much like the step1 function. The step2 function calls function clickEast, waits until a "3" appears in the map display area, and then calls step3. The step3 function calls function clickSouth and waits until a "6" appears, and so on. The step5 function calls a brief intermediate finish function step6:

function step6()
{
  finish();
}

Here I simply transfer control to the real finish function (see Figure 6). Recall that the finish function also handles situations where the maximum number of allowed calls to the delay function has been exceeded.

Figure 6 Finish Function (Close Figure 6)
function finish()
{
  if (numTries > maxTries)
  {
    logRemark("\nnumTries has exceeded maxTries");
    logRemark("\n*FAIL*");
  }
  else
  {
    logRemark("Checking final state");
    var is = parent.rightFrame.document.all["Image1"].src;
    if (is.indexOf("8") >= 0) logRemark("\n*Pass*");
    else logRemark("\n*FAIL*");
  }
  logRemark("\nAll done");
}

First I deal with situations where global counter numTries has exceeded global constant maxTries. This can happen in one of two ways: either my AJAX app's logic is correct and the app just took too long to update page state or the app's logic is bad and the check function was not able to find the correct state. Here I arbitrarily choose the simple solution of failing the test scenario.

The second logic branch in finish checks the final state of the application to determine a pass/fail result. In this case, I am checking to see whether the final map image is "8" as it should be if I start at "5" and then click the North, East, South, South, and West controls in succession.

Extending the Test Harness

The test automation system I've presented here is very lightweight and is designed specifically for you to modify. As usual for a magazine article, I've removed most error checking to keep the main ideas clear. You, however, will want to liberally add error traps-test automation is inherently tricky and errors are the rule rather than the exception.

Notice that the test harness is not fully automated: you have to manually click on the Run Test button to launch the automation. There are several ways you can fully automate the harness. One simple way is to add an onload handler to the test harness frame:

<frame src="http://localhost/AjaxApplication/Default.aspx"
  name="rightFrame"
  onload="leftFrame.launch();" >

In this case, the launch function in the test scenario code is defined as follows:

var started = false;
function launch()
{
  if (!started) runTest();
}

In addition, the runTest function described in this column is edited to begin with this statement:

started = true;

When the right-hand frame containing the AJAX Web application under test is loaded, the launch function is called. The first time launch executes, global variable started is false and control is transferred to the runTest function, which sets variable started to true and then executes the automation. On subsequent page loads of the Web app, which occur for regular HTTP request/response actions, the launch function is called, but because global started is true, control is not transferred to runTest again. With a fully automated version of the test harness, you can create multiple harness pages and scenario pages and then execute several scenarios by placing statements like the following into a BAT file and then using the Windows Task Scheduler to launch the automation:

iexplore http://localhost/AjaxTest/TestHarness01.aspx
iexplore http://localhost/AjaxTest/TestHarness02.aspx

Notice that a weakness of my lightweight system is that you cannot conveniently parameterize the system. Although you can easily place the common JavaScript code from the test scenario pages inside a single separate .js file, there's no easy way to coordinate executing multiple test scenarios from a single test harness page. This is not to say you can't do so; I'm just saying it's not a completely trivial task.

Another extension to the AJAX test automation is automating the process of saving test scenario results. The system as presented does not log results, but there are several ways you can do this. One approach is to place a Form element into the test scenario page, put a text field that holds the test scenario result inside the form, and then POST the result to a server, like this:

<form name="resultForm" method="Post" action="saveResults.aspx">
  <p>Result: <input type="text" name="result"></p>
  <p><input type="submit" name="saver" value="Save Results"></p>
</form>
You can supply a pass/fail value to the result field inside the finish function along the lines of this code:
if (is.indexOf("8") >= 0)
{
  logRemark("\n*Pass*");
  theForm.result.value = "Pass";
}
else
{
  logRemark("\n*FAIL*");
  theForm.result.value = "Fail";
}

Now you can manually save the scenario result by clicking on the Save Results button, or you can automate saving by placing the following in your scenario code:

document.all["theForm"].submit();

Wrapping Up

The test harness I've presented here is just one of many ways to test AJAX Web applications. After reading this column you should have no trouble using both my lightweight approach and more sophisticated harnesses to test your AJAX Web applications. With the release of the ASP.NET AJAX framework, the growth of AJAX Web applications is likely to accelerate. The ability to write lightweight test automation should become increasingly important, making it a valuable addition to your personal skill set.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006). James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com Thanks to Doug Walter, Shawn Burke, and Scott Guthrie (all at Microsoft), whose comments significantly improved this column.


Cutting Edge: Perspectives on ASP.NET AJAX

Cutting Edge
Perspectives on ASP.NET AJAX
Dino Esposito

Contents
Ways to AJAX in ASP.NET
The Straight Way to ASP.NET AJAX
UpdatePanel and the Interceptor Pattern
Taking Control of Out-of-Band Calls
Page Methods vs. Web Service Methods
My Way to ASP.NET AJAX

Unless you've spent the past 12 months disconnected from the Net-perhaps vacationing on a remote tropical island or participating in a reality game show-you should know a few things about AJAX. But I'll do a quick refresher just in case.

First, there's the acronym-it stands for Asynchronous JavaScript and XML. Then there is its purpose: a true breakthrough, AJAX makes possible solutions that would be otherwise impossible or just impractical to obtain. And finally, you should understand what AJAX is made up of. It's not a technology, per se. Instead, it is a blanket term to indicate rich browser applications built using powerful combinations of client-side Web technologies that have been around for years, such as JavaScript, cascading style sheets (CSS), DHTML, and XMLHttpRequest.

A large share of the browsers currently available support a common set of features-and these features allow Web developers to build applications that are very interactive, able to refresh quickly, and capable of doing a good deal of work on the client. Leveraging theses capabilities, AJAX applications provide rich user experiences that are similar to using desktop applications. Furthermore, AJAX makes it easier than ever to create "mash-up" applications. A mash-up is simply a Web application that combines information from multiple sources (for example, bringing together Microsoft® Virtual Earth™ with property sales information to create a helpful real estate tool).

I actually think that AJAX is so big, so important, that it will be forgotten in the next year or so. What? This is because I think AJAX will be absorbed by everybody in the industry, becoming a standard part of everyday life that we just don't think about-like today you don't feel the need to emphasize the use of XML or HTTP.

Unveiled at the Microsoft Professional Developers Conference in September 2005, Microsoft ASP.NET AJAX (formerly referred to as "Atlas") adds many new capabilities to ASP.NET 2.0 (see ajax.asp.net), all of which are geared towards making it easy to add AJAX-based functionality to your Web sites. In this month's column, I examine ASP.NET AJAX, looking closely at some of its key features. This column assumes you have at least some degree of familiarity with the basic concepts and tools. For a good introduction, see the article "Atlas at Last: ASP.NET Atlas Powers the AJAX-Style Sites You've Been Waiting For" by Matt Gibbs in the July 2006 issue of MSDN®Magazine. Also, see the sidebar "Still Synchronous for Now" for some additional information.

Ways to AJAX in ASP.NET

What new application functions can you build with AJAX? More specifically, what functions can you build using asynchronous JavaScript calls and the browser's object model? There are three main technologies that enable AJAX functionality: remote method calls, client-side data binding, and visual effects. Remote method calls sounds a bit too vague, so I'll narrow this to partial page postbacks and out-of-band requests.

How do you turn your new or existing ASP.NET apps into AJAX applications? It has been reported that more than 100 frameworks for AJAX development exist today-they support a variety of platforms and utilize a number of programming approaches. In general, I tend to group AJAX frameworks into three main categories based on their capabilities: callback frameworks, UI frameworks, and full frameworks.

A callback framework consists of a simple set of client and server libraries. It doesn't let you do much more than invoke a piece of server-side code from the client, moving input and output parameters in a serialized format. A typical UI framework is the evolution of an existing professional control library that provides advanced grid, charting, and tree controls. These support asynchronous postbacks and injecting JavaScript code on the client for automatic page refreshes. Finally, a full framework provides a rich programming model that includes controls and application services, preferably on both the client and the server. Microsoft ASP.NET AJAX falls into this third category.

If you already use a commercial suite of controls, upgrading to the next AJAX-powered version will likely be a seamless experience. If you develop applications entirely in-house and take care of the presentation layer as well as the back-end layers, then a full framework is the most viable option. Finally, a simple framework is useful only to add quick callback capabilities to existing ASP.NET applications, for both version 1.1 and 2.0.

ASP.NET 2.0 already provides a built-in script callback API, which I've written about in this column several times, for instance "Cutting Edge: Script Callbacks in ASP.NET" and "Cutting Edge: Custom Script Callbacks in ASP.NET"). And in the companion code for this column, I've included a demo of an extremely simple AJAX framework for remote method calls. It uses just 50 lines of code, part in JavaScript and part in managed code. Obviously, it can't compete with ASP.NET AJAX, but it can be used as the core code to add basic AJAX capabilities to ASP.NET 1.1 applications.

The Straight Way to ASP.NET AJAX

ASP.NET AJAX is composed of two distinct, though not mutually exclusive, APIs: client and server. You can build AJAX functionalities using direct client-side programming, traditional server-side programming, or any combination of the two. Any AJAX-based page inherently requires some client-side JavaScript code to address the browser's document object model (DOM) and any application-specific extension. It isn't necessary, however, to leave the writing of such a script code up to the ASP.NET programmer. A framework, in fact, could generate made-to-measure script code as the output of a server-side control. This form of indirect page updating is by far the simplest way to add AJAX capabilities to new and existing ASP.NET 2.0 pages. In ASP.NET AJAX, page updates can be governed by a piece of client code automatically injected by a server control-the UpdatePanel control.

The UpdatePanel control represents the nerve center of the server-centric programming model of ASP.NET AJAX. It lets you execute server-side code and return updated markup to the client browser. You may wonder how this differs from classic postbacks. The difference is in how the postback is implemented-instead of a full page refresh, the UpdatePanel control manages to send an out-of-band request for fresh markup and then update the DOM tree when the response is ready.

The client-centric programming model of ASP.NET is centered around the ability of placing calls to remote endpoints (mostly, but not exclusively, ASP.NET Web services and Windows® Communication Foundation services). When initiated straight from the client browser, a call to a remote endpoint requires a JavaScript proxy and a pinch of JavaScript code. Finally, client-side data binding can be viewed as an extension to the classic JavaScript runtime and DOM. In a purely client-side programming style, you connect to a remote endpoint, download data, and bind it to a DOM subtree. The structure of the template remains on the client, along with some state information, and only raw data is moved from the server to the client.

There are three main approaches to programming in ASP.NET AJAX. Figure 1 outlines these tools and tells you when they are most appropriate to use.

Figure 1 Approaches to Programming in ASP.NET AJAX (Close Figure 1)

Approach | Description | Use
UpdatePanel | Contains a group of server controls and monitors their postbacks and rendering process. Any requests for postback that originate from the contained controls are hijacked and served via script. Changes to the page are applied using DHTML. | Used when your team lacks significant JavaScript skills or you simply prefer to minimize the exposure to client-side programming. The use of UpdatePanel is also recommended when you need to protect any sensitive business logic in your app.
Remote method calls | Blanket term for asynchronous calls to page methods and local and external Web services. Calls move input arguments and receive return values via JSON streams in a type-independent manner. Strongly typed JavaScript code is required to trigger the call and apply changes back on the client. | Used when you need to work with a smarter client that triggers and controls remote operations in an asynchronous manner. This approach requires JavaScript coding to process return values and update any portion of the current DOM that is affected by results.
Client-side data?binding* | Two client-side controls written entirely in AJAX JavaScript to implement template-based binding. They are ListView, for multiple-records view, and ItemView, for single-record view. These two controls are combined together with client data source and filtering components. | Used when you need a more modern and cross-browser version of DHTML applications. With client-side data binding, your final pages include a thick layer of JavaScript and/or XML Script and virtually no managed code except for the Script Manager control.

UpdatePanel and the Interceptor Pattern

The UpdatePanel control heralds a programming style where page contents update without complete page refreshes and without a need for client-side programming. In ASP.NET AJAX lingo, this is referred to as partial rendering. The UpdatePanel control is the most visible component involved in enabling this capability. However, the real brains behind the internal implementation of partial rendering is the ScriptManager control. Still, the UpdatePanel control is the main tool developers use when writing pages that are partially updatable.

A Web page refresh can be obtained in either of two ways-manually clicking on a Submit button or programmatically calling into the submit method of the DOM form object. In both cases, the browser freezes the current page and sends a new request to the back-end Web server. The incoming response then overrides the existing contents.

The purpose of the UpdatePanel control is simply to hook up this process. It intercepts form submissions, captures any information being sent, and sends the same information through an out-of-band call based on XMLHttpRequest. The UpdatePanel control also adds some extra information to indicate which selection of controls are to be processed and rendered on the server-partial page rendering.

The behavior that results out of the UpdatePanel control is fully described by the interceptor pattern. The pattern refers to any external component that observes an object call and injects its own code between the caller and the receiver. In this way, the external component takes full control of the operation, controls how the task is accomplished, and returns a valid response to the caller. Figure 2 illustrates the implementation of this pattern in ASP.NET AJAX.

Figure 2 UpdatePanel Control and the Interceptor Pattern
Figure 2 UpdatePanel Control and the Interceptor Pattern(Click the image for a smaller view)
Figure 2 UpdatePanel Control and the Interceptor Pattern
Figure 2 UpdatePanel Control and the Interceptor Pattern (Click the image for a larger view)

The script code that is injected into the client page registers a handler for the form's onsubmit DOM event. This new handler simply replaces the classic request with one that goes through XMLHttpRequest and passes extra information. In particular, it adds the name of the UpdatePanel control responsible for the postback to the existing list of input fields. The modified contents for a page with a pageable grid look like this:

ScriptManager1=UpdatePanel1|GridView1&
__EVENTTARGET=GridView1&
__EVENTARGUMENT=Page%241&
__VIEWSTATE=...&
__VIEWSTATEENCRYPTED=&
__EVENTVALIDATION=...

The ScriptManager pseudo parameter refers to the ID of the UpdatePanel control that is responsible for the postback. Only the controls associated with the specified UpdatePanel control will be refreshed and their modified markup sent back to the browser, along with updated viewstate information.

Partial rendering is a common feature in various AJAX frameworks, although the internal implementations vary. Most of the differences have to do with the approach used to associate updatable controls with the request. In ASP.NET AJAX, updatable controls are grouped in an outermost container control (the UpdatePanel). Everything inside the container is rendered out to markup-this represents the delta of the currently displayed page. One alternative approach is to require that individual controls register programmatically with the framework for rendering over AJAX-style postbacks. This is more or less what happens in ASP.NET 2.0 with controls that support control state.

A more granular, more flexible approach than grouping controls in a container is to have each individual control decide how it renders AJAX-style postbacks. However, you'll also need additional codebehind instructions or ad hoc controls should be used. While good for control libraries and other such products, this approach is far too intrusive for a Web framework such as ASP.NET.

The UpdatePanel control is a class that's derived from System.Web.UI.Panel. A UI-free container of child ASP.NET controls, UpdatePanel can contain any ASP.NET server controls and doesn't require you to use any special AJAX-enabled controls. By using UpdatePanel, AJAX programming is done entirely on the server using the same application model and programming style as classic ASP.NET. Developers don't need to learn about JavaScript or the client-side Microsoft AJAX Library. Likewise, they have no exposure to XML Script or the browser's DOM.

Transforming a traditional ASP.NET page into an AJAX page is a two-step operation. First, you add a ScriptManager control to the page. You then wrap any group of controls that you want to refresh partially and asynchronously with an UpdatePanel control. The UpdatePanel control supports nesting scenarios and can be created dynamically, and an ASP.NET page can contain as many UpdatePanel controls as you want.

The actual update is triggered in two possible ways: when child controls post back and when specific external events occur. Through triggers and other properties you can make each UpdatePanel control refresh its contents conditionally.

With the UpdatePanel control, you can migrate ASP.NET pages to AJAX piecemeal and do so very quickly. The same ASP.NET page, in fact, can fire both AJAX-style and classic postbacks. For this reason, UpdatePanel must move the view state of the page with each and every request. In particular, the view state is sent to the server, updated to reflect changes on selected controls, and then downloaded to the client. The same happens to event validation data. View state and event validation data cannot be updated on the client and for this reason must be sent back to the server, checked for tampering, and properly updated. Having up-to-date copies of the view state and event validation data available on the client is key to maintaining the page in a consistent state while allowing successful postbacks, in both AJAX and classic styles.

Ultimately, UpdatePanel is the easiest approach to learn for ASP.NET AJAX, it involves the least possible upgrade and migration work, and it is fully interoperable between classic and AJAX pages. When you use updatable panels, pages refresh more quickly. In fact, these updates are sometimes too fast for users to note the changes. For this reason, a busy panel that tells what's going on may not be enough. A busy panel like the UpdateProgress control is designed for long tasks-not to give users feedback on displayed changes. A better tool for giving users feedback on such changes is the UpdatePanelAnimation extender, which is defined in the latest CTP of the AJAX Control Toolkit. You can see a sample of this in action. The extender animates the area covered by UpdatePanel during the postback. In particular, you can specify animations for the updating and updated events on the extender. Note, though, that you have to install the AJAX Control Toolkit separately, as it is not incorporated in the ASP.NET AJAX Extensions download.

Taking Control of Out-of-Band Calls

If you're comfortable with JavaScript and want to try making direct server calls from the client, you have two main options: using Web services and calling into static page methods.

By registering an ASMX Web service with the script manager, you instruct the ASP.NET AJAX infrastructure to generate a JavaScript proxy class and inject it into the client page. You can take a look at the proxy class by typing the URL of the Web service into the address bar (local machine) followed by the /js suffix. Here's an example of a JavaScript proxy for a Web service with two public methods. The code has been simplified for reading:

Type.registerNamespace('Samples.MyWebService');
Samples.MyWebService = new function()
{
    this.appPath = "http://YourServer/AjaxDemo/";
    var cm = Sys.Net.ServiceMethod.createProxyMethod;
    cm(this, "LookupCustomer", "id");
    cm(this, "LookupAllCustomers");
}

This code makes it possible for you to call the Web service from a script tag. It uses the classic Proxy pattern, and the overall mechanism is exactly the same as when you're using Web services on the server in classic ASP.NET or in Windows applications.

There are some caveats, though. What kind of server-side code would you call from a client? Or, put another way, what kind of operations do you expect to be invoked from the client on Web services? Web services that are invoked from AJAX pages expose some business logic to the client pages. A public endpoint, a Web service can be invoked from the Internet and there's no built-in security layer to protect your application logic. This vulnerability is nothing new.

ASP.NET AJAX gives developers a chance to expose some application logic through Web services. Once published on the Internet, this logic (not protected by any security layer) is available to any callers. In general, you should only expose data and logic that is considered safe for Internet use. There might be no real problem if you write and invoke a Web service that returns a products catalog. However, exposing the business logic layer (BLL), including, for example, the credit card verification process, is not something you should do.

ASP.NET AJAX leads you to use Web services to expose BLL to client pages for direct use. You should think of AJAX Web services as a sort of non-critical, user interface layer BLL. Any truly sensitive pieces of BLL should be invoked using classic postbacks or UpdatePanel controls.

To link Web services directly to an ASP.NET AJAX page, they must be local Web services hosted in the same Web application as the caller page. There are two main reasons for this requirement. First, the Web service must be backed by an ASP.NET AJAX application that knows how to handle the /js suffix appended to each request. You could, however, use another ASP.NET AJAX application to host the Web service. In this case, your client calls might clash with the stricter security settings in recent browsers that prevent cross-site calls.

Page Methods vs. Web Service Methods

Another possible approach to binding server code to ASP.NET AJAX pages is to invoke page methods. A callable page method is a public static (or Shared in Visual Basic® .NET) method defined in the codebehind class and decorated with the same WebMethod attribute used for Web service methods. At present, this is limited to ASPX pages-both inline and codebehind code-but might be extended in the future to user controls and custom controls.

I should note that the ASMX Web services and page methods require different settings. In particular, you must register Web services with the script manager and you must install a script HTTP module to run page methods. Figure 3 shows what's needed in the web.config file.

Figure 3 Changes in Web.config for ASP.NET AJAX (Close Figure 3)
<microsoft.web>
   <scripting>
      <webServices>
          ...
      </webServices>
   </scripting>
</microsoft.web>

<system.web>
    <httpHandlers>
    <remove verb="*" path="*.asmx"/>
        <add verb="*" path="*.asmx"
             type="Microsoft.Web.Script.Services.ScriptHandlerFactory"
             validate="false"/>
    </httpHandlers>
    <httpModules>
        <add type="Microsoft.Web.UI.ScriptModule" name="ScriptModule"/>
    </httpModules>
</system.web>

To register a Web service and force the generation of the proxy you need the following:

<asp:ScriptManager ID="scriptManager1" runat="server">
  <Services>
     <asp:ServiceReference Path="~/WebServices/MyDataService.asmx" />
  </Services>
</asp:ScriptManager>

You'll also need additional attributes to decorate the Web service class. In particular, you need the ScriptService attribute to enable the generation of the JavaScript proxy class and GenerateScriptType for each custom type you plan to use, except for generic types. Figure 4 shows an example.

Figure 4 ASP.NET AJAX Web Service Methods (Close Figure 4)
namespace Samples
{
    [WebService(Namespace = "http://samples.ajax/")]
    [ScriptService]
    [GenerateScriptType(typeof(Customer))]
    [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
    public class MyDataService : System.Web.Services.WebService
    {
        public MyDataService() {}

        [WebMethod]
        public Customer LookupCustomer(string id)
        {
            return CustomerManager.Load(id);
        }

        [WebMethod]
        public CustomerCollection LookupAllCustomers()
        {
            return CustomerManager.LoadAll();
        }
}

The GenerateScriptType must be applied for the Customer type, but not for the CustomerCollection if this type is defined to derive from the generic Collection<T> class, for which ASP.NET AJAX has built-in support:

public class CustomerCollection : Collection<Customer> {}

The HTTP module required for page methods hooks up the default HTTP handler for ASPX requests and redirects the rendering stage to a custom piece of code that executes the specified method and serializes return values.

Both the page methods approach and the Web service methods approach help you access the back-end BLL. Like publicly exposed Web service methods, page methods can easily be automated from the address bar of a browser-it only requires some elementary JavaScript Object Notation (JSON) skills. Thus, you should be just as careful with these page methods as with the WebMethods in terms of what BLL logic you expose.

In the end, whether you use page methods or Web service methods is really just a matter of preference. In both cases, you should call into a façade layer that then routes to the proper BLL function, as shown in Figure 5. Web service methods can be called from any page, provided you register the endpoint to the service in each page. Page methods can only be called from the page that defines them. Currently, my personal preference is to use page methods.

Figure 5 Comparison of Page and Web Service Methods
Figure 5 Comparison of Page and Web Service Methods(Click the image for a smaller view)
Figure 5 Comparison of Page and Web Service Methods
Figure 5 Comparison of Page and Web Service Methods (Click the image for a larger view)

Both ASP.NET AJAX Web service methods and page methods make extensive use of JSON to pass object data around. A lightweight data-interchange format, JSON is used to serialize objects to strings. Because of its inherent simplicity, JSON is good for both people and machines. More people can write and read JSON strings than can write and read SOAP packets. And JSON is very easy for machines to parse and process.

You might expect page methods to offer better performance than Web services. After all, to resolve Web service calls, the ASP.NET runtime has to parse SOAP packets. This, however, isn't exactly true. ASP.NET AJAX installs a tailor-made HTTP handler (see Figure 3) that intercepts all ASMX requests. Requests with a /js suffix are processed differently, working directly with the JSON payload and Web service method. As a result, no SOAP is involved whatsoever and the body of the request simply contains the JSON stream of input arguments. For non-AJAX requests, the new HTTP handler just delegates the call back to the original ASP.NET handler that understands SOAP.

My Way to ASP.NET AJAX

All things considered, I think UpdatePanel is the best approach for the majority of development teams. It doesn't preclude classic ASP.NET and allows you to modify existing pages at your convenience. Also, it's unobtrusive and doesn't require you to learn many new things before starting. In addition, UpdatePanel gives the same level of protection for your BLL as classic Web pages and it fully supports asynchronous ASP.NET pages running lengthy tasks.

A final piece of advice: avoid mixing various AJAX platforms. In terms of JavaScript built-in object extensions, there might be collisions between ASP.NET AJAX and other frameworks. More importantly, there's no guarantee that a combination of products that works today will still work in the future. Any new version of any framework can introduce new collisions.

Still Synchronous for Now

AJAX functionalities are often associated with asynchronous techniques. Consequently, it is often assumed that ASP.NET AJAX does programming asynchronously by default. This is certainly true from a client perspective—the task starts on the server and the current page remains available for further interaction.

However, ASP.NET AJAX doesn't support ASP.NET 2.0 asynchronous pages, nor does it implement remote calls asynchronously (meaning via asynchronous HTTP handlers). The HTTP handler that serves AJAX Web service requests works synchronously. The same is true for the HTTP module that serves page methods, regardless of whether the ASP.NET page is marked for asynchronous processing. Most of these architectural issues will likely be worked out in a future version of ASP.NET, currently code-named "Orcas." For now, however, the best way out is to implement AJAX through the UpdatePanel control.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or weblogs.asp.net/despos.


Service Station: The Service Factory for WCF

Service Station
The Service Factory for WCF
Aaron Skonnard

Contents
The Service Factory Development Process
Getting Started
Data Contracts
Message Contracts
Fault Contracts
Service Contracts and Implementations
Exposing Service Endpoints
WCF Security Guidance Package
Customizing Security Choices
Advanced Customizations
Wrapping It Up

In my last column, I introduced you to the concept of software factories and gave a tour of the new Web Service Software Factory (also known as Service Factory) produced by the patterns & practices (p&p) team at Microsoft. As I explained, a software factory is a collection of guidance assets that help speed up the process of building software using patterns and best practices. Before you read on, if you haven't already, you should check out my December 2006 column so you're familiar with the material.

This month I'll continue exploring software factories with a look at the Web Service Software Factory for Windows® Communication Foundation. Along the way I'll show you how to customize the guidance package, which provides a suite of code recipes that integrate with Visual Studio® 2005 and automate common development tasks.

There are two flavors of Service Factory guidance packages that you can use: one that generates ASP.NET Web services (ASMX) code and another that generates Windows Communication Foundation code. The ASMX version has already shipped on MSDN® while the Windows Communication Foundation version is still under development, as of this writing. Nevertheless, a community drop of the Windows Communication Foundation guidance package is currently available on GotDotNet (see practices.gotdotnet.com/svcfactory).

Last time, I looked at the ASMX guidance package but didn't have time to explore the Windows Communication Foundation version so I'll do that here. Be aware that the final details of the Windows Communication Foundation guidance package may change en route to the final release.

The Service Factory Development Process

First, I want to briefly describe the Service Factory development process. The p&p team has adopted a transparent and agile development approach. They ship regular drops to their online community and ask for feedback. But the best feature is how they actually use the community feedback to improve the next version of the product. I've seen first-hand how much influence the developer community can have on the outcome of the p&p deliverables. This adds significant value to developers who want to get involved and ultimately improves the quality of the deliverables, which is a benefit to all.

The other interesting aspect of the development process is how they rely on a reference implementation to drive their work. As illustrated by Figure 1, developing a realistic reference implementation can not only verify the applicability of the recommended practices under consideration but also help discover new practices that would have otherwise been missed. The final set of recommended practices ends up influencing everything that ships with Service Factory including the written patterns, how-to topics, and, of course, the guidance packages.

Figure 1 Software Factory Development Process
Figure 1 Software Factory Development Process(Click the image for a smaller view)
Figure 1 Software Factory Development Process
Figure 1 Software Factory Development Process (Click the image for a larger view)

Getting Started

Once you've downloaded and installed the Windows Communication Foundation guidance package, you'll see it show up in the Visual Studio 2005 New Project dialog. You start using the Windows Communication Foundation guidance package by selecting the Windows Communication Foundation Service template and entering a name for your new solution. Then you press OK to launch the recipe that generates the initial solution structure. You'll notice that the initial solution structure is very similar to the one used in the ASMX guidance package.

The most significant differences are found in the service layer. The service layer contains four projects: DataContracts, FaultContracts, ServiceContracts, and ServiceImplementation. This provides a logical structure for organizing the corresponding Windows Communication Foundation artifacts you'll be building. The main change is the project for fault contracts, which is a new concept introduced by Windows Communication Foundation. Another change is that the service host is now placed in the Tests folder.

Remember, if this initial solution structure doesn't fit your exact needs and you'd like to make changes, you can do so by unlocking and reorganizing the solution. Once you've organized the solution to your liking, you can reassign project responsibilities, which essentially tells Service Factory which recipes to make available on each project. I discussed how to do this in my last column.

Now let's dig into some of the changes within the service layer. Implementing a Windows Communication Foundation service layer consists of several steps. First, you define the data contracts that you'll use within messages. Then you model the message, fault, and service contracts that will make up the operations. And finally you implement the service contract and expose the service endpoints.

The Windows Communication Foundation guidance package provides numerous recipes for accomplishing each of these tasks. Within Visual Studio, you can open the Guidance Navigator window (View | Other Windows) and see the various Windows Communication Foundation-specific recipes provided within the Windows Communication Foundation guidance package (see Figure 2).

Figure 2 Guidance Navigator
Figure 2 Guidance Navigator

Data Contracts

The Windows Communication Foundation guidance package provides a few ways to generate data contracts. In cases where you already have XML schema definitions in place, you can generate data contract types from the XML schema definition (XSD) itself. The recipe is similar to the ASMX recipe called Create XML Serializable type from Schema, only it generates [DataContract] types when possible. The DataContractSerializer in Windows Communication Foundation only supports a subset of the XML schema language so if the schema uses something outside of that subset (like an XML attribute, for example), the recipe will fall back to generating XmlSerializer-compatible types.

When designing new data contracts from scratch, you can run the Create Data Contract recipe, which launches a wizard. The wizard asks you to specify the name of the class as well as the name of the data contract (the XML name) and the namespace. Then you specify the members of the data contract; notice you can specify various [DataContract] details such as Order and IsRequired (see Figure 3). Once you press Finish, a class like the one shown in Figure 4 will be generated.

Figure 4 WinePreference Class (Close Figure 4)
[DataContractAttribute(
 Namespace = "http://CohoWinery.DataContracts/2006/09",
 Name = "WinePreference")]
public class WinePreference
{
    private string country;
    private string region;
    private string type;

    [DataMemberAttribute(Name = "Country", Order = 1)]
    public string Country
    {
        get { return this.country; }
        set { this.country = value; }
    }
    ... // remaining properties omitted
}
Figure 3 Create Data Contract Recipe
Figure 3 Create Data Contract Recipe(Click the image for a smaller view)
Figure 3 Create Data Contract Recipe
Figure 3 Create Data Contract Recipe (Click the image for a larger view)

You can also turn existing class definitions into data contracts by using the Decorate type as a DataContract recipe. This recipe automates the process of applying the various [DataContract] attributes on an existing class. There's also a recipe named Validate DataContract that analyzes a selected class to see if it's capable of being serialized by DataContractSerializer.

Message Contracts

Before you can start defining your service contracts, you also need to define the message contracts you'll use within each operation. The ASMX guidance package referred to this task as creating message types. In ASMX, a message type is simply an XmlSerializer type that represents the entire request or response message. If you used such a type with the ASMX unwrapped parameter style, the type itself defines the entire structure of the XML used in the SOAP body.

In Windows Communication Foundation, message types are first-class citizens. You define them with a different set of attributes ([MessageContract], [MessageHeader], and [MessageBodyHeader]). A message contract allows you to model the structure of the entire SOAP envelope. You can specify which fields go in the SOAP header and which go in the body. Message contracts also allow you to specify whether the message should be unwrapped (similar to the ASMX parameter style). Otherwise, Windows Communication Foundation wraps all messages within an element representing the operation name.

Windows Communication Foundation allows you to define operations using either data contract or message contract types. So a common question is when you should use one approach versus the other. The general guidance from p&p is that you should use data contracts anytime you're dealing with a reusable type (say, a type that needs to be used in five different messages). And you should use message contracts when you're defining types that represent request/response messages that won't be reused. Using a message contract also gives you the flexibility to introduce custom header processing down the road so that may also become a deciding factor during design.

If you want to define your message types in terms of data contracts, you can simply continue to use the data contract recipe. If you want to define Windows Communication Foundation message contracts, there is a recipe specifically for that. The message contract recipe walks you through the process of designing a message contract out of existing data contracts, and it generates class definitions with the [MessageContract] attributes.

Fault Contracts

If you plan to return custom SOAP faults to clients, you need to define the types you intend to return within the fault element. In Windows Communication Foundation, you define these fault types using data contract definitions. The data contract models the structure of the XML that will be serialized into the fault detail element. Then you annotate the operation contract with [FaultContract] specifying the detail type. You can apply [FaultContract] to an operation multiple times if it returns multiple fault types.

The [FaultContract] annotations give Windows Communication Foundation enough information to include the appropriate fault information in the service metadata-that is, the XSD and Web Services Description Language (WSDL) definitions, which lets clients generate the necessary client-side code to process the custom faults. This makes it possible for Windows Communication Foundation clients to catch user-defined exceptions using FaultException<T>.

The Windows Communication Foundation guidance package provides a recipe for generating custom faults. When you launch this recipe, it asks you to name the new fault type and indicate whether you want to use the DefaultFaultContract base class (see Figure 5).

Figure 5 DefaultFaultContract Base Class (Close Figure 5)
[DataContractAttribute(
 Namespace="http://FaultContracts/Default/DefaultFaultContract",
 Name="DefaultFaultContract")]
public class DefaultFaultContract
{
    int errorId;
    string errorMessage;
    Guid correlationId;

    public DefaultFaultContract() : this(-1, String.Empty, Guid.Empty)
    {
    }

    public DefaultFaultContract(
        int errorId, string errorMessage, Guid correlationId)
    {
        this.errorId = errorId;
        this.errorMessage = errorMessage;
        this.correlationId = correlationId;
    }

    [DataMemberAttribute(IsRequired = true, Name = "ErrorId", Order = 1)]
    public int ErrorId
    {
        get { return errorId; }
        set { errorId = value; }
    }

    [DataMemberAttribute(IsRequired = true, Name = "ErrorMessage",
        Order = 2)]
    public string ErrorMessage
    {
        get { return errorMessage; }
        set { errorMessage = value; }
    }

    [DataMemberAttribute(IsRequired = true,
        Name = "CorrelationId", Order = 3)]
    public Guid CorrelationId
    {
        get { return correlationId; }
        set { correlationId = value; }
    }
}

The default bass class provides some common error-related fields that you might want to take advantage of in your exception handling scheme. You can also define additional members in the custom fault type. The recipe generates a new data contract type for you, like the one in Figure 6.

Figure 6 New Data Contract (Close Figure 6)
[DataContractAttribute(
   Namespace = "http://CohoWinery.FaultContracts/2006/09",
   Name = "RegistrationError")]
public class RegistrationError : DefaultFaultContract
{

    private string registrationDetails;

    [DataMemberAttribute(Name = "RegistrationDetails", Order = 1)]
    public string RegistrationDetails
    {
        get { return this.registrationDetails; }
        set { this.registrationDetails = value; }
    }
}

Once you have all of your data, message, and fault contracts in place, you can begin defining your service contracts.

Service Contracts and Implementations

As with data contracts, there are a few ways to define service contracts using the Windows Communication Foundation guidance package. When you already have a WSDL definition that you need to implement, you can run the recipe for generating the service contract from WSDL. This facilitates WSDL-driven scenarios in which collaboration around XSD and WSDL is absolutely necessary.

You can also define new service contracts from scratch by running the Create Service Contract recipe available within the Windows Communication Foundation guidance package. This recipe asks you for the .NET interface name as well as the service contract name (the WSDL name) and the namespace. It also asks you if you want to require sessions on the service contract.

Figure 7 Creating a Service Contract
Figure 7 Creating a Service Contract(Click the image for a smaller view)
Figure 7 Creating a Service Contract
Figure 7 Creating a Service Contract (Click the image for a larger view)

Now you can define the operations (see Figure 7). For each operation, you can specify the action, the request and response types (which can be either data or message contract types), the fault types, whether it's one-way, and additional details relating to the session contract when Windows Communication Foundation sessions are being used. Once you've finished defining the operations, the recipe generates a Windows Communication Foundation service contract definition like this:

[ServiceContractAttribute(
    Namespace = "http://CohoWinery.ServiceContracts/2006/09",
    Name = "ClubMembershipService")]
[ServiceKnownTypeAttribute(typeof(RegistrationError))]
public interface IClubMembershipService
{
    [OperationContractAttribute(Action = "RegisterNewMember")]
    [FaultContractAttribute(typeof(RegistrationError))]
    void RegisterNewMember(RegisterNewMember request);
    ...
}

You can implement this service contract by running the Implement Service Interface recipe. This recipe generates a class that implements the .NET interface representing the service contract and stubs out each method, as shown here:

public class ClubMembershipService : IClubMembershipService
{
    #region IClubMembershipService Members

    public void RegisterNewMember(RegisterNewMember request)
    {
        throw new Exception(
          "The method or operation is not implemented.");
    }

    #endregion
}

If you select XmlSerialzer-compatible types for the request and response types, the recipe will be smart enough to apply the [XmlSerializerFormat] attribute to that particular operation contract. This contract, in turn, instructs the Windows Communication Foundation dispatcher to use XmlSerializer at run time.

Then it's your job to finish implementing each service operation. This is where you can take advantage of the Create Service Contract Translator recipe to generate code that translates between the message types and the business entity types, which is, in fact, the primary role of the service implementation in the Service Factory architecture.

Exposing Service Endpoints

Finally, in order to test the service implementation, you must host it and expose at least one service endpoint. You can accomplish this by using the Expose Service recipe, which is made available to you on the Host project. When you run this recipe, simply specify the name of the service implementation class, choose a binding type, and press Finish. Currently, the Windows Communication Foundation guidance package only provides guidance for the main HTTP bindings (basicHttpBinding and WSHttpBinding). The recipe generates an .svc file, which references the service implementation class, as shown here:

<%@ServiceHost language="c#" Debug="true"
 Service="CohoWinery.ServiceImplementation.ClubMembershipService" %>

It also generates the <system.serviceModel> configuration shown in Figure 8 within the host's web.config file. You'll notice that the recipe enables WSDL metadata retrieval and a WS-Metadata Exchange (MEX) endpoint by default. Hence, at this point you can browse to the .svc endpoint and view the generated metadata. Then you can also run the Add Service Reference recipe within the client test application. This recipe queries the service using MEX and asks you to choose an endpoint. Then it generates the client proxy code and configuration needed to call the selected endpoint.

Figure 8 Generated ServiceModel Configuration (Web.config) (Close Figure 8)
<system.serviceModel>
  <behaviors>
   <serviceBehaviors>
    <behavior name=
      "CohoWinery.ServiceImplementation.ClubMembershipService_Behavior">
     <serviceDebug includeExceptionDetailInFaults="false" />
     <serviceMetadata httpGetEnabled="true" />
    </behavior>
   </serviceBehaviors>
  </behaviors>
  <bindings>
   <basicHttpBinding>
    <binding name="basicHttpBinding_
       CohoWinery.ServiceContracts.IClubMembershipService">
     <security mode="None" />
    </binding>
   </basicHttpBinding>
  </bindings>
  <services>
   <service behaviorConfiguration=
    "CohoWinery.ServiceImplementation.ClubMembershipService_Behavior"
    name="CohoWinery.ServiceImplementation.ClubMembershipService">
    <endpoint
       binding="basicHttpBinding" bindingConfiguration=
        "basicHttpBinding_
         CohoWinery.ServiceContracts.IClubMembershipService"
       contract="CohoWinery.ServiceContracts.IClubMembershipService" />
    <endpoint address="mex" binding="mexHttpBinding"
       contract="IMetadataExchange" />
   </service>
  </services>
 </system.serviceModel>

With Windows Communication Foundation you have a tremendous number of options available to you for the binding configuration, behaviors, and hosting environments. Security is one aspect of that configuration that requires extra attention.

WCF Security Guidance Package

Web service security has a large surface area. WS-Security and the related specifications offer numerous possibilities for securing services with different types of security tokens; this provides for great flexibility but also demands more security-related guidance and automation.

As a result, the Windows Communication Foundation service factory comes with another guidance package for Windows Communication Foundation security. You can enable this guidance package by opening the Guidance Package Manager within Visual Studio and selecting Enable / Disable Packages. Once enabled, you can browse the security-related guidance using the Guidance Navigator (see Figure 9).You'll also find a new context menu named Service Factory - Security when you right-click on the Host project.

Figure 9 Recipes for WCF Security
Figure 9 Recipes for WCF Security(Click the image for a smaller view)
Figure 9 Recipes for WCF Security
Figure 9 Recipes for WCF Security (Click the image for a larger view)

You'll notice there are several recipes available to you that automate the most common security patterns and practices including mutual certificate authentication, Kerberos, and direct authentication using either Windows accounts, SQL Server™, or Active Directory® Application Mode (ADAM) as the identity store. They also provide a secure form of anonymous authentication where messages are protected with a server certificate, but no client authentication is required.

Let's take a look at a common scenario: direct authentication with SQL Server. When you run the recipe, the first thing you'll see is the wizard. The first screen of the wizard describes the security pattern this particular recipe implements and provides a link back to the p&p Web site, which describes the pattern in more detail. The rest of the wizard automates the process of specifying the Windows Communication Foundation endpoint, binding, and behavior details, along with the server certificate to use, the level of message protection required, and the SQL Server role provider details. When the wizard completes, it generates the appropriate Windows Communication Foundation configuration in the Host project's web.config file and it's ready to go. The other security recipes are just as easy to use.

Once you've secured the service, the next time you run the Add Service Reference recipe within the client application, the security settings will be imported into the client configuration and the recipe may ask you for any additional details such as credentials.

This type of security automation is extremely valuable. It allows architects and developers to know that they're following proven security patterns and practices. You can surely expect more advances around security as the Windows Communication Foundation guidance package continues to evolve.

Customizing Security Choices

Although the security recipes do provide valuable automation, developers must still choose the right security recipe in the first place, and once they do, they still have to make quite a few choices while walking through a particular recipe. As an architect, you may want to constrain the set of choices made available to your developers in order to further simplify the process and to enforce your own security standards.

The Windows Communication Foundation security guidance package makes this possible by providing a recipe for modifying the available security settings. If you right-click on the solution, select Service Factory - Security, and then Modify security settings, you'll get a wizard that lets you reduce the set of security scenarios made available to users. For example, if you only want your developers to be able to use Kerberos or X.509 credentials, simply deselect the other options. You can also constrain the message protection level options. If you only want your developers to use the Sign and Encrypt option, for example, uncheck everything else.

Once you're done, the wizard updates your solution configuration and from that point on, only the security options you specified will be made available by the guidance package. For example, if you return to the Guidance Navigator or the Service Factory context menu, only the Kerberos and X.509 recipes will appear in the list. Furthermore, when you walk through one of the available recipes, only the Sign and Encrypt option will appear in the message protection options.

Security choices become very important when you need to ensure interoperability across toolkits. Hence, in order to further simplify common interop scenarios, the Windows Communication Foundation security guidance package provides another wizard that constrains security choices with a specific interoperability scenario in mind. You can launch this wizard by selecting Modify interop settings from the solution context menu.

If your system needs to ensure that all of your Windows Communication Foundation services can be called by Web Services Enhancements (WSE) 3.0 clients without fail, you'd only want your developers to see the Windows Communication Foundation security options that will make that possible. You can accomplish this by selecting the WSE 3.0 | Windows Communication Foundation option, and you can further constrain the security choices around the individual specifications that both toolkits support. Then, next time you run a security recipe, only the specified choices will be made available. Eventually, more interoperability scenarios will be supported (even scenarios like "a Windows Communication Foundation service consumed by a BEA WebLogic Portal 9.2 Client").

Both of these wizards generate a file named SecurityConfiguration.xml that is associated with the solution. This file tells the Windows Communication Foundation security guidance package what options to make available. This file can be redeployed as part of a custom guidance package to ensure everyone sees the same options.

Advanced Customizations

In my opinion, the crown jewel of the Microsoft software factory initiative is the ability to create custom guidance packages for your own organizations. Being able to leverage the common recipes produced by p&p along with your own domain-specific recipes makes for a compelling combination. Automating as many of your common tasks as possible will ultimately result in improved productivity and better quality, consistency, and predictability throughout your organization.

As you've seen in these columns, the ASMX and Windows Communication Foundation guidance packages allow for some simple customizations, such as modifying the solution structure and reassigning project responsibilities, as well as modifying the security/interop settings made available by the package. But for anything more advanced, you'll need to become familiar with the Guidance Automation Toolkit (GAT) and learn how to build and deploy your own guidance packages.

There are several articles and other resources available on the Web that will help you become more familiar with the GAT. However, if you have the Service Factory for ASMX installed, you'll find a detailed document that describes the step-by-step process for creating a new guidance package. It's called "How to: Create a New Guidance Package that Includes your Configuration Settings." If you're interested in this level of customization, be sure to start there.

Wrapping It Up

The Service Factory for Windows Communication Foundation brings valuable guidance assets to the emerging landscape of the .NET Framework 3.0 where guidance is sorely needed. The Windows Communication Foundation guidance will help you more quickly develop Windows Communication Foundation services that conform to accepted security patterns. Look for the official release around the time that the .NET Framework 3.0 ships.

The final version is likely to provide some new features that aren't available as of this writing, such as guidance around contract design (data, message, and service), contract versioning, message validation, deployment, handling large messages, and even patterns around incorporating workflows into your Windows Communication Foundation service implementations. They're also working on a feature that will validate the security settings for a particular service configuration (similar to the WSE Policy Advisor). By the time this column goes to press, many of these features may already be in place.

If you have feedback on what you've seen or would like to provide input on future features, get involved by joining the Service Factory Community on GotDotNet. For more information see the "Additional References" sidebar.

Additional References
  • Microsoft patterns & practices Developer Center
  • Web Service Software Factory on MSDN
  • Web Service Software Factory Community
  • Introduction to the Guidance Automation Toolkit

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a cofounder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight’s Applied Web Services 2.0, Applied BizTalk Server 2006, and Introducing Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


Foundations: Build a Queued WCF Response Service

Foundations
Build a Queued WCF Response Service
Juval Lowy

Code download available at: Foundations2007_02.exe (188KB)
Contents
Queued Calls
Response Service
Designing a Response Service Contract
Using Message Headers
The ResponseContext Class
Client-Side Programming
Service-Side Programming
Response Service Programming
Conclusion

Windows Communication Foundation (WCF) enables disconnected work between clients and services. The client posts messages to a queue and the service processes them later. Such interaction permits a different programming model from the default request/response pattern, enabling possibilities such as load leveling, improved availability, and compensating work, to name just a few. This column starts with a brief introduction to Windows® Communication Foundation queued calls and then presents an interesting problem-how to get results out of a queued call-and the solution via some cool Windows Communication Foundation programming techniques and a few helper classes I wrote for this purpose.

Queued Calls

Windows Communication Foundation provides support for queued calls using NetMsmqBinding. Instead of transporting a message over TCP or HTTP, Windows Communication Foundation transports the message over Microsoft® Message Queuing (MSMQ). Instead of sending the Windows Communication Foundation message to a live service, the client posts the message to an MSMQ queue. All the client sees and interacts with is the queue, not a service endpoint. As a result, the calls are inherently asynchronous and disconnected. The calls will execute when the service processes the messages at some time in the future.

Note that there is no direct mapping of Windows Communication Foundation messages to MSMQ messages. A single MSMQ message can contain multiple Windows Communication Foundation messages or just a single one according to the contract session mode. With a required session mode, multiple Windows Communication Foundation calls can be in one MSMQ message, and with the allowed or not allowed session modes (used by per-call and singleton services), each Windows Communication Foundation call will be in a separate MSMQ message.

As with every Windows Communication Foundation service, the client interacts with a proxy, as shown in Figure 1. Since the proxy is configured to use the MSMQ binding, it does not send the Windows Communication Foundation message to any particular service. Instead, it converts the calls to MSMQ messages and posts them to the queue specified in the endpoint's address.

Figure 1 WCF Queued Calls Architecture
Figure 1 WCF Queued Calls Architecture(Click the image for a smaller view)
Figure 1 WCF Queued Calls Architecture
Figure 1 WCF Queued Calls Architecture (Click the image for a larger view)

On the service side, when a service host with a queued endpoint is launched, the host installs a queue listener. The queue listener detects that there is a message in the queue, de-queues the message, then creates the host side's chain of interceptors ending with a dispatcher. The dispatcher calls the service instance as usual. If multiple messages are posted to the queue, the listener can create new instances as the messages come off the queue, thus ending with asynchronous, disconnected, and concurrent calls.

If the host is offline, messages will simply be pending in the queue. The next time the host is up, the messages will be forwarded to the service.

A potentially disconnected call made against a queue cannot possibly return any values because no service logic is invoked at the time the message is dispatched to the queue. In addition, the call may be dispatched to the service and processed after the client application has shut down, when there is no client available to process the returned values. Similarly, the call can't return to the client any service-side exceptions, and there may not be a client around to catch and handle the exception. Since the client cannot be blocked by invoking the operation-or rather, the client is only blocked for the briefest moment it takes to queue up the message-queued calls are inherently asynchronous from the client's perspective. These are the classic characteristics of one-way calls. Consequently, any contract exposed by an endpoint that uses the NetMsmqBinding can only have one-way operations. Windows Communication Foundation verifies this at the service and proxy load time:

//Only one-way calls on queued contracts
[ServiceContract]
interface IMyContract
{
   [OperationContract(IsOneWay = true)]
   void MyMethod();
}

Since the interaction with MSMQ is encapsulated in the binding, there is nothing in the service or client invocation code pertaining to the fact that the call is queued. The service and client code look like any other Windows Communication Foundation client and service code, as shown in Figure 2.

Figure 2 Implementing and Consuming a Queued Service (Close Figure 2)
Service Side
[ServiceContract]
interface IMyContract
{
   [OperationContract(IsOneWay = true)]
   void MyMethod();
}
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyService : IMyContract
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod() {...}
}
Client Side
MyContractClient proxy = new MyContractClient();
proxy.MyMethod();
proxy.Close();

When defining an endpoint for a queued service, the endpoint address must contain the queue's name and the type of the queue (public or private):

<endpoint
     address = "net.msmq://localhost/private/
         MyServiceQueue"
   binding = "netMsmqBinding"
   ...
/>

Finally, MSMQ is a Windows Communication Foundation transactional resource manager. If the queue is transactional, a message posted by the client will be rolled back if the client's transaction aborts. On the service side, reading the message from the queue kick-starts a new transaction. If the service participates in that transaction and the service aborts the transaction (perhaps as a result of an exception), then the message will roll back into the queue for another retry. Windows Communication Foundation offers elaborate failure detection and poison message handling support that is beyond the scope of this column.

Response Service

The programming model of queued calls described so far is one-sided: the client posts a one-way message to a queue, and the service processes that message. This model is sufficient when the queued operations are one-way calls by design. However, the queued service may need to report back to its client on the result of the invocation, returned results, and even errors. By default, however, this is impossible. Windows Communication Foundation equates queued calls with one-way calls, which inherently forbid any such response. In addition, queued services (and their clients) are potentially disconnected. If the client posts a queued call to a disconnected service, when the service finally gets the messages and processes them there may not be a client to return the values to because the client may be long gone. The solution is to have the service report back to a client-provided queued service. I call such a service a response service. Figure 3 shows the architecture of such a solution.

Figure 3 Response Service
Figure 3 Response Service(Click the image for a smaller view)
Figure 3 Response Service
Figure 3 Response Service (Click the image for a larger view)

The response service is just another queued service in the system. It may be disconnected from the client as well and hosted in a separate process or a separate machine, or it can share the client's process. If the response service shares the client's process, when the client is launched the response service will start processing the queued responses. Having the response service in a separate process (or even on a separate machine) from the client helps to further decouple the lifetime of the response service from the clients that use it.

Designing a Response Service Contract

As with the use of any Windows Communication Foundation service, the client and the service need to agree beforehand on the response contract and what it will be used for, such as returned values and error information or just returned values. Note that you can also split the response service into two services and have one for results and another for errors. For example, consider the ICalculator contract implemented by the queued MyCalculator service, as shown here:

   [ServiceContract]
   interface ICalculator
   {
      [OperationContract(IsOneWay = true)]
      void Add(int number1,int number2);
      ... //More operations
   }
   [ServiceBehavior(InstanceContextMode =
      InstanceContextMode.PerCall)]
   class MyCalculator : ICalculator {...}

The MyCalculator service is required to respond to its client with the result of the calculation and to report any errors. The result of the calculation is an integer, and the error is in the form of the Windows Communication Foundation ExceptionDetail data contract. For the response service, you could define the ICalculatorResponse contract like this:

[ServiceContract]
interface ICalculatorResponse
{
   [OperationContract(IsOneWay = true)]
   void OnAddCompleted(int result,ExceptionDetail error);
}

The response service supporting ICalculatorResponse needs to examine the returned error information, notify the client app, the user, or the application administrator on the method completion, and make the results available to the interested parties. Here is a simple response service that supports ICalculatorResponse:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyCalculatorResponse : ICalculatorResponse
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void OnAddCompleted(int result,ExceptionDetail error)
   {
      MessageBox.Show("result =  " + result,"MyCalculatorResponse");
      if(error != null)
      {
         //Handle error
      }
   }
}

There are two immediate problems with the implementation of both MyCalculator and MyCalculatorResponse. The first is that the same response service could be used to handle the response (or completion) of multiple calls on multiple queued services, and yet MyCalculatorResponse (and more importantly the clients it serves) has no way of distinguishing among responses. The solution is to have the client that issued the original queued call tag the call by associating it with some unique ID. The queued service MyCalculator needs to pass that ID to MyCalculatorResponse so that it could apply some custom logic regarding that ID.

The second problem is the queued service discovering the address of the response service. Unlike duplex callbacks, there is no built-in support in Windows Communication Foundation for passing the response service reference to the service. Placing that address in the service host config file (in a client section) is not a good idea because the same queued service could be called by multiple clients, each with its own dedicated response service and address.

One possible solution is to explicitly pass both the client-managed ID and the desired response service address as parameters to every operation on the queued service contract:

[ServiceContract]
interface ICalculator
{
   [OperationContract(IsOneWay = true)]
   void Add(int number1,int number2,
            string responseAddress,string methodID);
}

In much the same way, the queued service could explicitly pass the method ID to the response service as a parameter to every operation on the queued response contract:

[ServiceContract]
interface ICalculatorResponse
{
   [OperationContract(IsOneWay = true)]
   void OnAddCompleted(int result,ExceptionDetail error,
                       string methodID);
}

Using Message Headers

While passing the address and the ID as explicit parameters would work, it does distort the original contract, and introduces plumbing-level parameters along business-level parameters in the same operation. A better solution is to have the client store the response address and operation ID in the outgoing message headers of the call. Using the message headers this way is a general-purpose technique for passing out-of-band information to the service, information that is otherwise not present in the service contract.

The operation context offers collections of incoming and outgoing headers, available with the IncomingMessageHeaders and OutgoingMessageHeaders properties:

public sealed class OperationContext : ...
{
   public MessageHeaders IncomingMessageHeaders {get;}
   public MessageHeaders OutgoingMessageHeaders {get;}
   ... //More members
}

Each collection is of the type MessageHeaders, which represents a collection of MessageHeader objects:

public sealed class MessageHeaders : IEnumerable<...>, ...
{
   public void Add(MessageHeader header);
   public T GetHeader<T>(int index);
   public T GetHeader<T>(string name,string ns);
   ... //More members
}

The class MessageHeader is not intended for application developers to interact with directly. Instead, use the MessageHeader<T> class which provides type safety and easy conversion from a common language runtime (CLR) type to a message header:

public abstract class MessageHeader : ...
{...}

public class MessageHeader<T>
{
   public MessageHeader();
   public MessageHeader(T content);
   public T Content {get;set;}
   public MessageHeader GetUntypedHeader(string name,string ns);
   ... //More members
}

As the type parameter for MessageHeader<T> you can use any serializable or data contract type. You can construct a MessageHeader<T> around a CLR type, and then use the GetUntypedHeader method to convert it to a MessageHeader and store it in the outgoing headers. GetUntypedHeader requires you to provide the generic type parameter name and namespace. The name and namespace will be used to look up the header from the header collection. You perform the lookup via the GetHeader<T> method of MessageHeaders. Calling GetHeader<T> obtains the value of the type parameter of the MessageHeader<T> used.

The ResponseContext Class

Since the client needs to pass both the address and the method ID in the message headers, a single primitive type parameter will not do. Instead, use my ResponseContext class, which is shown in Figure 4 with some of the error-handling code omitted.

Figure 4 ResponseContext Class (Close Figure 4)
Service Side
[ServiceContract]
interface IMyContract
{
   [OperationContract(IsOneWay = true)]
   void MyMethod();
}
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyService : IMyContract
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod() {...}
}
Client Side
MyContractClient proxy = new MyContractClient();
proxy.MyMethod();
proxy.Close();

ResponseContext provides a place to store both the response address and the ID. In addition, if the client would like to use a separate response service for faults, ResponseContext also provides a field for the fault response service address. (This is not shown in Figure 4-see the download for this issue for the complete code.)

The client is responsible for constructing an instance of ResponseContext with a unique ID. While the client can supply that ID as a construction parameter, the client could also use the constructor of ResponseContext, which takes just the response address, and have that constructor generate a GUID for the ID. The important property of ResponseContext is the static Current property. It completely encapsulates the interaction with the message headers. By accessing the Current property you have a nice programming model: the get accessor reads from the incoming message headers the instance of ResponseContext, and the set accessor stores in the outgoing headers an instance of a ResponseContext.

Client-Side Programming

The client can provide an ID per method call by using a different instance of ResponseContext for each call. The client needs to store an instance of ResponseContext in the outgoing message headers. In addition, the client must do so in a new operation context; it can't use its existing operation context. This requirement is true for service and non-service clients alike. Windows Communication Foundation enables a client to adopt a new operation context for the current thread with the OperationContextScope class defined as follows:

public sealed class OperationContextScope : IDisposable
{
   public OperationContextScope(IContextChannel channel);
   public OperationContextScope(OperationContext context);
   public void Dispose();
}

OperationContextScope is a general technique for spinning a new context when the one you have is inadequate. The constructor of OperationContextScope replaces the current thread's operation context with the new context. Calling Dispose on the OperationContextScope instance restores the old context (even if it was null). If you do not call Dispose, that may damage other objects on the same thread that expect the previous context. As a result, OperationContextScope is designed to be used inside a using statement and to provide only a scope of code with a new operation context, even in the face of exceptions.

When constructing a new OperationContextScope instance, you provide its constructor with the inner channel of the proxy used for the call. The client needs to create a new OperationContextScope, and inside the scope assign to ResponseContext.Current. Figure 5 shows these steps.

Figure 5 Client-Side Programming with a Response Service (Close Figure 5)
string methodId = GenerateMethodId(); // use helper method
string responseQueue =
   "net.msmq://localhost/private/MyCalculatorResponseQueue";

CalculatorClient proxy = new CalculatorClient();

using(OperationContextScope contextScope =
   new OperationContextScope(proxy.InnerChannel))
{
   ResponseContext.Current = new ResponseContext(
      responseQueue,methodId);
   proxy.Add(2,3);
}

proxy.Close();

To streamline and automate the client's work, you need a proxy base class that encapsulates the response service setup steps of Figure 5. Unlike with duplex callback, Windows Communication Foundation does not provide such a proxy class, so you have to hand-craft one. To ease that task, I wrote the ResponseClientBase<T> defined in Figure 6.

Figure 6 ResponseClientBase<T> (Close Figure 6)
public class ResponseClientBase<T> : ClientBase<T> where T : class
{
   public readonly string ResponseAddress;

   public ResponseClientBase(string responseAddress)
   {
      ResponseAddress = responseAddress;
   }

   public ResponseClientBase(string responseAddress,string endpointName)
      : base(endpointName)
   {
      ResponseAddress = responseAddress;
   }

   public ResponseClientBase(string responseAddress,
      NetMsmqBinding binding,EndpointAddress remoteAddress) :
         base(binding,remoteAddress)
   {
      ResponseAddress = responseAddress;
   }

   protected string Enqueue(string operation,params object[] args)
   {
      using(OperationContextScope contextScope =
         new OperationContextScope(InnerChannel))
      {
         string methodId = GenerateMethodId();

         ResponseContext.Current =
            new ResponseContext(ResponseAddress,methodId);

         Type contract = typeof(T);
         MethodInfo methodInfo = contract.GetMethod(operation);
         methodInfo.Invoke(Channel,args);

         return responseContext.MethodId;
      }
   }

   protected virtual string GenerateMethodId()
   {
      return Guid.NewGuid().ToString();
   }
}

To use ResponseClientBase<T>, derive a concrete class out of it, and provide the queued contract type for the type parameter. Unlike the practice for a normal proxy, do not have the subclass also derive from the contract. Instead, provide a similar set of methods that all return a string for the method ID, not void. (This is why you can't derive from the contract, because the operations on the contract do not return anything-they are all one-way). For example, Figure 7 shows the matching response-service-aware proxy using this queued service contract:

[ServiceContract]
interface ICalculator
{
   [OperationContract(IsOneWay = true)]
   void Add(int number1,int number2);
   ... //More operations
}

Figure 7 Deriving from ResponseClientBase<T> (Close Figure 7)
class CalculatorClient : ResponseClientBase<ICalculator>
{
   public CalculatorClient(string responseAddress) :
      base(responseAddress) {}

   public CalculatorClient(string responseAddress,string endpointName) :
      base(responseAddress,endpointName) {}

   public CalculatorClient(string responseAddress,
      NetMsmqBinding binding,EndpointAddress remoteAddress) :
      base(responseAddress,binding,remoteAddress) {}

   ... //More constructors

   public string Add(int number1,int number2)
   {
      return Enqueue("Add",number1,number2);
   }
}

Using ResponseClientBase<T>, the code shown in Figure 5 is reduced to this:

string responseAddress =
   "net.msmq://localhost/private/MyCalculatorResponseQueue";
CalculatorClient proxy = new CalculatorClient(responseAddress);
string methodId = proxy.Add(2,3);
proxy.Close();

The virtual method GenerateMethodId of ResponseClientBase<T> uses a GUID for a method ID. Your subclass of ResponseClientBase<T> can override it and provide any other unique string, such as an incremented integer.

The constructors of ResponseClientBase<T> accept the response address and the regular proxy parameters such as endpoint name, address and binding. The constructors store the response address in a read-only public field. ResponseClientBase<T> derives from a regular ClientBase<T>, and so all constructors delegate to their respective base constructor.

The heart of ResponseClientBase<T> is the Enqueue method. Enqueue accepts the name of the operation to invoke (actually queue a message to) and the operation parameters. Enqueue creates a new operation context scope, generates a new method ID, and stores the ID and the response address in a ResponseContext. It assigns the ResponseContext to the outgoing message headers by setting ResponseContext.Current and then uses reflection to invoke the provided operation name. Due to the use of reflection and late binding, ResponseClientBase<T> does not support contract hierarchy or overloaded operations. For those, you need manual coding such as in Figure 5.

Service-Side Programming

The queued service accesses its incoming message headers collection via ResponseContext.Current and reads from it the response address and the method ID. The service needs the address so that it can construct the proxy to the response service, and the ID so that it can provide that ID to the response service. The service usually has no direct use for the ID.

The service can use the same technique as the client to pass the method ID to the response service: instead of explicit parameters, use the outgoing message headers to pass the ID out-of-band to the response service. Much like the client, the service must use a new operation context via OperationContextScope to be able to modify the outgoing headers collection. The service can programmatically construct a proxy to the response service and provide the proxy with the response address and an instance of NetMsmqBinding. The service can even read the binding settings from the config file. Figure 8 shows these steps.

Figure 8 Server-Side Programming with a Response Service (Close Figure 8)
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyCalculator : ICalculator
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void Add(int number1,int number2)
   {
      int result = 0;
      ExceptionDetail error = null;
      try
      {
         result = number1 + number2;
      }
      catch(Exception exception) //Don't re-throw
      {
         error = new ExceptionDetail(exception);
      }
      finally
      {
         ResponseContext responseContext = ResponseContext.Current;

         EndpointAddress responseAddress =
            new EndpointAddress(responseContext.ResponseAddress);

         NetMsmqBinding binding = new NetMsmqBinding();
         CalculatorResponseClient proxy =
            new CalculatorResponseClient(binding,responseAddress);

         using(OperationContextScope scope =
            new OperationContextScope(proxy.InnerChannel))
         {
            ResponseContext.Current = responseContext;
            proxy.OnAddCompleted(result,error);
         }

         proxy.Close();
      }
   }
}

The service catches all exceptions thrown by the business logic operation and wraps each exception with an ExceptionDetail object. The service does not re-throw the exception because that would also cancel the response as it will abort the transaction that queues the response message to the response service queue.

In the finally statement, regardless of exceptions, the service responds. It constructs a proxy to the response service using the address from the response context and a new instance of NetMsmqBinding. The service uses the proxy's inner channel to seed a new OperationContextScope. Inside the new scope the service adds to the new context's outgoing headers the same response context it received by setting Response.Current and calls the response service proxy, in effect queuing the response. The service then disposes of the context scope and closes the proxy.

To streamline and automate the work required by the service to extract the response parameters from the headers and to respond, I created the ResponseScope<T> class:

public class ResponseScope<T> : IDisposable where T : class
{
   public readonly T Response;

   public ResponseScope();
   public ResponseScope(string bindingConfiguration);
   public ResponseScope(NetMsmqBinding binding);

   public void Dispose();
}

ResponseScope<T> is a disposable object-it installs a new operation context, and when it is disposed of, the scope restores the old operation context. To automate that even in the face of exceptions, ResponseScope<T> should be used in a using statement.

ResponseScope<T> takes a type parameter representing the response contract, and it offers the Response a read-only public field of the same type. Response is a proxy to the response service and the clients of ResponseScope<T> use Response to call operations on the response service. There is no need to dispose of Response because ResponseScope<T> does that in Dispose. ResponseScope<T> will instantiate Response based on the default endpoint in the config file or using the binding information (the config section name or an actual binding instance) provided to the constructor. Figure 9 shows the implementation of ResponseScope<T>.

Figure 9 Implementing ResponseScope<T> (Close Figure 9)
public class ResponseScope<T> : IDisposable where T : class
{
   private OperationContextScope m_Scope;
   public  readonly T Response;

   public ResponseScope() : this(new NetMsmqBinding()) {}

   public ResponseScope(string bindingConfiguration) :
      this(new NetMsmqBinding(bindingConfiguration) {}

   public ResponseScope(NetMsmqBinding binding)
   {
      ResponseContext responseContext = ResponseContext.Current;
      EndpointAddress address =
         new EndpointAddress(responseContext.ResponseAddress);

      ChannelFactory<T> factory = new ChannelFactory<T>(binding, address);
      Response = factory.CreateChannel();

      //Switching context now
      m_Scope = new OperationContextScope(Response as IContextChannel);
      ResponseContext.Current = responseContext;
   }

   public void Dispose()
   {
      IDisposable disposable = Response as IDisposable;
      disposable.Dispose();
      m_Scope.Dispose();
   }
}

Using ResponseScope<T> the finally statement of Figure 8 is reduced to this:

finally
{
   using(ResponseScope<ICalculatorResponse> scope =
      new ResponseScope<ICalculatorResponse>())
   {
      scope.Response.OnAddCompleted(result,error);
   }
}

The constructor of ResponseScope<T> uses ResponseContext.Current to extract the incoming response context. It then uses a channel factory to initialize Response with a proxy to the response service. The trick in implementing ResponseScope<T> is not using OperationContextScope inside a using statement. By constructing a new OperationContextScope, ResponseScope<T> establishes a new operation context. ResponseScope<T> encapsulates an OperationContextScope and saves the newly created OperationContextScope. The old context will be restored once ResponseScope<T> is disposed of in a using statement. ResponseScope<T> then uses ResponseContext.Current to add to the new operation context's outgoing headers the response context and returns.

Response Service Programming

The response service accesses its incoming message headers collection, reading from it the method ID and responding accordingly. Figure 10 demonstrates a possible implementation of such a response service.

Figure 10 Implementing a Response Service (Close Figure 10)
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyCalculatorResponse : ICalculatorResponse
{
   [OperationBehavior(TransactionScopeRequired = true)]
   public void OnAddCompleted(int result,ExceptionDetail error)
   {
      string methodID = ResponseContext.Current.MethodId;

      if(error == null)
      {
         //Notify client, fire event, and so on
      }
      else
      {
         //Handle error
      }
   }
}

The response service accesses its operation context incoming headers via ResponseContext.Current and, just like the queued service, it extracts the method ID and handles the completion or the error.

Conclusion

Queued services are instrumental in many applications. If being able to report completion, errors, and results to clients is essential to your application, the helper classes in this column augment the basic offering of Windows Communication Foundation and provide the missing support in a line or two of code.

The Windows Communication Foundation techniques shown in this column, such as interacting with message headers, replacing the operation context, writing custom proxy base classes and custom contexts are useful in many other cases, especially with custom frameworks.

Send your questions and comments for Juval to mmnet30@microsoft.com.

Juval Lowy is a software architect with IDesign providing WCF training and WCF architecture consulting. His recent book is Programming WCF Services (O'Reilly, 2007). He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at www.idesign.net.


Netting C++: Mapping Native C++ to the Common Type System

Netting C++
Mapping Native C++ to the Common Type System
Stanley B. Lippman

Contents
Keyword Surprises
Inheritance
Copy Semantics
New Language Lessons
Next Time

In this column, I want to pick up where I left off in the December 2006 installment and begin translating the Text Query Language (TQL) Query class hierarchy from C++ to the .NET Common Type System (CTS). In this instance, I'll use C++/CLI, but you could use C# too. The primary benefit of C++/CLI is that the app can be incrementally translated, maintaining a source-level mix of native and managed C++. To get started, let's look at the native Query base class implementation as presented in Figure 1.

Figure 1 Native Abstract Query Base Class (Close Figure 1)
#include <vector>
#include <iostream>
#include <set>
#include <string>
#include <utility>

using namespace std;

typedef pair< short, short > location;

class Query {
public:
   virtual ~Query(){ delete _solution; }
   Query& operator=( const Query& );

   // the virtual set of operations ...
   virtual void eval() = 0;
   virtual Query *clone() const = 0;
   virtual ostream& print( ostream &os ) const = 0;
   virtual bool add_op( Query* ) { return true; }
   virtual void display_solution( ostream &os = cout, int tabcnt = 0 );

   // factored query-wide operations
   void  handle_tab( ostream &os, int tabcnt ) const;
   void display_partial_solution();

   const set< short > *solution();
   const vector< location > *locations() const { return &_loc; }

   // support for recognizing embedded parentheses within the query
   // as in alice && ( fiery || magical )
   void lparen( short lp ) { _lparen = lp; }
   void rparen( short rp ) { _rparen = rp; }

   short lparen() { return _lparen; }
   short rparen() { return _rparen; }

   void print_lparen( short cnt, ostream& os ) const;
   void print_rparen( short cnt, ostream& os ) const;

protected:

   Query() : _lparen( 0 ), _rparen( 0 ), _solution( 0 ) {}
   Query( const Query &rhs )
      : _loc( rhs._loc ), _solution( 0 ),
        _lparen( rhs._lparen ), _rparen( rhs._rparen )
      {}

   Query( const vector<location> &loc )
      : _loc( loc ), _solution( 0 ),
        _lparen( 0 ), _rparen( 0 )
      {}

    set<short>* _vec2set( const vector<location>* );
    short  _lparen;
    short  _rparen;

    vector<location> _loc;
    set<short> *_solution;
};

The Query class implementation represents a hybrid abstract class design: abstract because it provides a pure virtual interface that each derived class instance must override, and hybrid because it also provides a concrete implementation of the infrastructure shared among all the derived types. The benefit of a hybrid design is twofold: first, it simplifies the delivery of specialized derived class types because it allows their designers to focus on the specialized algorithms where their expertise lies without requiring the repetitive implementation of non-domain infrastructure in which they may have little interest or understanding. Second, it allows non-virtual and therefore potentially inline implementations of the shared functionality. While this efficiency gain can prove significant in native mode, the gain in .NET is less certain because of platform security issues.

The CTS can elegantly capture the hybrid nature of the design in a way that native C++ cannot-the set of pure virtual functions is captured in an IQuery interface class and the concrete shared infrastructure is factored into an abstract Query reference base class, the outline of which is presented in Figure 2.

Figure 2 Query Reference Base Class (Close Figure 2)
// pure virtual interface goes into IQuery
interface class IQuery {
    void eval();
    void display_solution();
    // ...
};

// shared implementation goes into Query
public ref class Query abstract : IQuery {

    short lparen;
    short rparen;

    list<location>^ _loc;
    set<short>^_     solution;

public:
    // ...
};

// AndQuery inherits Query and implements IQuery
public ref class AndQuery : Query { ... };

Recall you must specify the reference classes as public if you want them to be accessible outside the assembly. Interfaces are public by default, and so the public keyword, while permitted, is redundant. (C++/CLI has a lot of optional redundancy like this.)

Keyword Surprises

Two surprises for the native C++ programmer in Figure 2 are the keyword abstract in our declaration of the Query class, which is not present in ISO-C++, and the absence of the public keyword before the Query base class specification in the declaration of AndQuery. Let's look briefly at each.

Bjarne Stroustrup introduced support for a language-level notion of an abstract class in the late beta phase of what was then called cfront Release 2.0 back around 1988. This product straddled multiple project groups within Bell Laboratories with different management priorities. The instantiation of an abstract class generates a compile-time error, preventing a potentially disastrous run-time failure. While everyone agreed this was a good thing, not everyone agreed that it was a good thing to add a new keyword this late in the release cycle. As I recall, Bjarne was judged to have the freedom to add support for an abstract base class, being the language inventor, but he did not enjoy the freedom to add a product keyword when doing so was judged potentially invasive to the successful release of that product. To move forward, Bjarne invented the rule that a class with an associated pure virtual function is to be treated as an abstract class. Interestingly enough, this design aspect of C++ has a remarkably long and unbroken history of being criticized.

For C++/CLI, it was decided that we'd add the keyword abstract, and it was, in my opinion, an elegant design decision to place it after the class name, as it appears in the Query base class in Figure 2. The abstract keyword was also extended to be used as an alternative syntax for declaring a pure virtual function. (Of course, this is analogous to the use of abstract in C#.) Both forms of pure virtual function declaration are considered equivalent within C++/CLI, and over time I suspect the form you choose to code in will give away your programming platform origins, be it native or managed. An additional keyword that can be used to explicitly constrain either inheritance of a class or the ability to override a virtual function is the sealed keyword. It would be very cool if in the next iteration of ISO-C++, the committee backpatched the language with these extensions.

Inheritance

In the CTS, only public inheritance is supported. This makes sense given its restriction to single inheritance only. The primary design use of private inheritance is the mix-in model of multiple inheritance. In this design model, an implementation is privately inherited while the public inheritance represents the interface of the abstraction. For example, a fixed-length queue in the visionary Booch Components back in the 1990s was, if I recall correctly, implemented as follows:

class FixedLengthQueue :
      public Queue, private Vector { ... };

The dynamic-length queue, on the other hand, maintained the same public interface, but had a private backing store of a list rather than a vector, with the gain in flexibility at the expense of space and time. This kind of design lost favor in both the inheritance community and within C++ with the success of the Standard Template Library (STL).

In the CTS, interfaces remove the need for multiple inheritance and, with that, the need for private inheritance. In the earlier Managed Extensions to C++ that was released with Visual Studio® .NET back in 2001 (Visual C++® 7.0), the public keyword was required even though public inheritance is the only possible relationship between the base and derived class. If you left it off, it was a compiler error. It was felt that this was necessary back then because there was concern that native C++ programmers moving to .NET would otherwise become confused. (The absence of the public keyword in native C++, of course, indicates a private inheritance relationship.) In C++/CLI, the public keyword is optional. The idea now basically is that there are plenty of more substantial issues that the native C++ programmer may find confusing, so let's choose our battles carefully. In fact, this is a sort of set-up to my next topic, which is the difference in copy semantics between native C++ and .NET, and how that impacts your designs. In particular, if you look again at Figure 1, you'll see that the issue of copy construction and the use of the virtual clone pattern within the native Query implementation needs to be addressed.

Copy Semantics

Before considering how to address copy construction and the virtual clone pattern, let's make sure you know why it is necessary to do so within a native implementation.

In native C++, a frequent design pitfall manifests in the following way. You first declare a pointer member within the class that addresses some memory allocated on the heap, either within the class constructor or through a factory pattern. You then define a destructor that deletes the memory associated with that pointer member, but you forget or fail to realize you need to provide a copy constructor, and so the combination of a pointer member and destructor makes the program technically undefined in its behavior. Statistically, this is not a good thing.

The job of the copy constructor (and the copy assignment operator, of course) is to effect a deep copy of the object addressed by the pointer member (of course, this can scale to any number and any depth of pointer members), to create a new object on the native heap, and to initialize that object with a value-by-value copy of the original. This is necessary (in lieu of a more complex reference-counting scheme) because the default behavior of native pointers within native C++ is a shallow copy; that is, the target pointer is assigned to address the same object in memory as addressed by the source pointer. Without the deep copy (or some form of reference counting), the aliasing of a single object across multiple class objects is disastrous.

This problem is compounded when the pointer addresses the base class of an inheritance hierarchy, such as Query. In this case, the copy constructor cannot be correctly invoked because the pointer is polymorphic and requires a virtual method that is resolved at run time. A constructor, however, cannot be virtual because it is invoked on raw memory prior to the creation of the type you want to select on.

These are the kinds of problems that pioneers in a language either celebrate through feats of cleverness, or find most foul. At this point within the history of C++, the solution is widely familiar. The accepted design pattern is to provide a virtual clone function, as in Query, that is overridden within each derived class. The implementation of the clone method is trivial; it's the pattern that is powerful:

//native code clone pattern ...
class AndQuery : public Query {
public:
    virtual Query *clone() const
          { return new AndQuery( *this ); }

    // ...
};

New Language Lessons

As a first attempt to translate the solution into a new language, I am often tempted to mechanically map the original implementation to the new language, and if the obvious mapping is unavailable, my instinct is often to criticize the new language. In this case, I might be tempted to map this deep copy mechanism to .NET. After all, C++/CLI offers a copy constructor (and copy assignment operator) and the .NET Framework offers the IClone interface. But doing so would be wrong.

The first problem is that the C++/CLI copy constructor, like the C++/CLI destructor, may or may not get invoked. It depends largely on how the reference class object is declared, and the class designer cannot impose that on the programmer. For example, if I provide a copy constructor for the AndQuery class, and someone writes this

AndQuery ^aq1 = gcnew AndQuery;
AndQuery ^aq2 = aq1;

the copy constructor for AndQuery is not invoked by the second declaration statement; rather, aq2 and aq1 both reference the same AndQuery object allocated in the first declaration statement. That's nonintutive to a lot of programmers. When we couldn't guarantee the invocation of the default constructor in a value class, we decided it was safer to remove the default constructor for a value class supported within the Managed Extensions. (I don't personally see how this case differs in terms of potential trouble.)

Microsoft itself fell into the same sort of "well, it was a good idea but it just didn't pan out" challenge with the IClone interface. In the .NET Framework 2.0 version of the common language runtime (CLR), the use of IClone has been deprecated. If you look at the generic collection types, you'll see that they do not provide an IClone implementation although the non-generic collection types they mirror do. The problem in this case is an ambiguity as to just how deep a copy IClone guarantees.

In both cases, if you can't depend on consistent semantics, it seems prudent to avoid the use of the mechanism altogether. So, where does that leave things? Actually, just fine, if you step back and think about it. That's one of the rewarding side-effects of learning another language; you realize how often you have stopped thinking in a habitually familiar language.

I needed to introduce clone because a copy constructor cannot be declared virtual. I needed to introduce a copy constructor because the class destructor deallocated the heap object, and so I had to avoid inadvertent aliasing of that object across class instances. I needed to introduce a destructor because the native heap requires that I explicitly manage heap memory.

Hopefully, a lightbulb should go off right now, you know, like in the cartoons. In .NET, the managed heap is garbage-collected. That means you don't need the destructor. But then that also means you don't need to manage a deep copy. And it means you don't need the clone pattern either. The whole complex machinery, in fact, can be discarded. It proves to be an artifact of the programming language, not of the program.

Next Time

The next snarly concerns for reengineering from native C++ to C++/CLI are the general issue of templates, and the specific one of the STL. Look again at Figure 1; you'll see that the Query class implementation makes considerable use of the STL as well as the pair template, neither of which are available in a managed form (yet). I'll explore how to cope in their absence in the next column.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman currently works for Perpetual Entertainment, a company specializing in the development of massively multiplayer online games, and the infrastructural technology to develop and support them. Stan is best known for working on the original C++ cfront compiler with its creator, Bjarne Stroustrup. His favorite all-time job was Software Technical Director on the Firebird segment of Fantasia 2000. He worked with the Visual C++ team at Microsoft in the invention of C++/CLI.


C++ at Work: Rationales, Highlights, and a Farewell

C++ At Work
Rationales, Highlights, and a Farewell
Paul DiLascia

Contents
The Rationale
Ditching Managed Extensions
Future Fun
The Evolution of Programming
The Ultimate Goal
Farewell, Though Not Goodbye

This month I'm going to deviate from my regular Q&A format to tell you about a really great document I discovered online. A few weeks ago, someone wrote me asking why he couldn't declare a const function in C++/CLI:

// reference class
ref class A {
   void f() const; // NOT!
};

To which I replied: you just can't, them's the rules. The Common Language Infrastructure (CLI) is designed to support many languages-like Visual Basic®, Java, and even COBOL-that don't even know what const means. The CLI doesn't know what const member functions are, so you can't use them.

After I fired off my reply, I vaguely remembered something buried way down in the depths of my memory blanks, something about const, something to do with compiler hints that other languages are free to ignore. I searched my old columns and discovered that I answered a question about const back in July 2004. In fact, C++/CLI does let you declare const data members and parameters-but not const member functions. Figure 1 shows a little program with a reference class that has a const static data member. If you compile this program, then use ILDASM to disassemble it, you'll see a line that looks like this:

field public static int32
    modopt([mscorlib]System.Runtime.CompilerServices.IsConst)
        g_private = int32(0x00000001)

Figure 1 const.cpp (Close Figure 1)
////////////////////////////////////////////////////////////////
// To compile type:
//    cl /clr const.cpp
//
#include <stdio.h>

ref class A {
   int m_val;
   // const data member allowed, will generate modopt
   static const int g_private = 1;
public:
   // public const member could be modified by Visual Basic or other
   // programs that don't know const -- so use literal instead.
   literal int g_public = 1;
   A(int i) { m_val = i; }
   void print();  // const;             // NO!--const fn not allowed
};

void A::print()
{
   printf("Val is %d\n",m_val);
}

int main()
{
   A a(17);
   a.print();
}

Modopt (optional modifier) is an MSIL declarator that says to CLI consumers: if you understand what this thing is, great; if not, you can safely ignore it. Conversely, modreq (required modifier) says: if you don't understand this thing, you can't use this function. Volatile is an example of a modreq. Since a volatile reference can be changed at any time by the OS/hardware (or even another thread), a CLI consumer better know what volatile means if it wants to use a volatile object. But const is optional. Note, however, that whereas the Managed Extensions would convert a C++ const object into a CLI literal, C++/CLI no longer does this. So watch out-if you declare a public const data member, clients written in a language like Visual Basic could change its value. If you don't want CLI clients to be able to change the value, you should declare your object literal, as in Figure 1. But what about member functions-why are const member functions not allowed?

The Rationale

Well, in the course of researching the answer, I stumbled upon a really nice paper called "A Design Rationale for C++/CLI" by Herb Sutter from Microsoft. Herb was one of the architects of C++/CLI. You can get the document, which I refer to hereafter as "the Rationale," from Herb's blog (the URL is a long ugly string-just search the Web for "C++/CLI Rationale" and you'll find it). As the title suggests, the Rationale explains why most everything in C++/CLI is the way it is. It answers everything from "Why extend C++ in the first place?" to my own query about const functions. I consider the Rationale required reading for anyone who wants to understand the reasoning behind C++/CLI. Since so much of the Rationale bears repeating, I thought I'd give you some highlights.

Let's start with the most important question, why extend C++ in the first place? The simple and obvious answer is: to make C++ a first-class CLI citizen. The Microsoft® .NET Framework is the future of Windows® development. Heck, even COBOL supports the CLI. So C++/CLI is designed to ensure C++'s continued success.

But why muck with C++? Why extend the language with yucky new concepts like ^ and % and new keywords like ref, value, and property? Well, there's no other way. The Rationale quotes none other than Bjarne Stroustrup on this: "Classes can represent almost all the concepts we need. Only if the library route is genuinely infeasible should the language extension route be followed."

For plain C++ code, targeting the CLI is like writing a compiler for a different processor: no problem. But CLI introduces new concepts that require special code generation and simply can't be expressed in C++. For example, properties require special metadata. You can't implement properties with a library or template. The Rationale discusses some of the alternative syntaxes considered for properties (and other CLI features) and why they were rejected.

Ditching Managed Extensions

The Rationale also explains why Microsoft decided to ditch the Managed Extensions. The Managed Extensions' use of * for both managed and native objects was a clever and valiant attempt to unify C++ and CLI without changing the language, but it obscures the important differences between reference and native objects. The two kinds of pointers look the same, but behave differently. For example, destruction semantics, copy constructors, and virtual calls within constructors/destructors all behave differently depending on what kind of object the pointer really points to. Bad! As the Rationale says, "It is important not only to hide unnecessary differences, but also to expose essential differences," and again cites Bjarne: "I try to make significant operations highly visible." Native and managed classes are fundamentally different beasts, so better to highlight the difference than gloss over it. Various mechanisms were considered and rejected, and the C++/CLI team finally settled on gcnew, ^ (handle), and % (tracking reference). Separating managed and native classes like this has unanticipated side-benefits. For example, using a separate gcnew operator to allocate managed objects leaves open the possibility that one day native classes could be allocated from the managed heap-and vice versa!

Did you ever wonder why the Managed Extensions have all those ugly underbar keywords like __gc and __value? Because the Managed Extensions religiously follow the C++ standard, which says, "If thou really must introduce new keywords, thou shalt name them beginning with double underscore!" But guess what? When Microsoft introduced __gc, __value and the rest, the Redmondtonians received "unexpectedly strong" complaints from programmers. Yeah! Programmers of the world, unite! You have nothing to lose but your underbars. Underbars make your code look icky, like some kind of assembly language program or something. So C++/CLI has ref and value, sans underscores. This meant adding new keywords to C++, but so what? As Bjarne says, "My experience is that people are addicted to keywords for introducing concepts to the point where a concept that doesn't have its own keyword is surprisingly hard to teach. This effect is more important and deep-rooted than people's vocally expressed dislike for new keywords." (True, true. I love that Bjarne is describing the psychology of programming.) So C++/CLI ditched the underbars. By making them positional keywords instead of reserved keywords, they can't conflict with programs that may already use these words as variable or function names.

Curious readers might wonder: how come gc changed to ref in the process of losing its underbars? As the Rationale explains, the important thing about managed classes isn't that they live on the managed (garbage collected) heap. The important thing is their reference semantics. Handles (^) act like references, not pointers. See? Once you read the Rationale, everything makes sense.

If exposing essential differences is important, so is hiding superficial ones. For example, all the operator overloading "just works" the way any C++ programmer would expect, whether the object is native, ref, or value. For another example where C++/CLI papers over a difference, consider this snippet:

// ref class R as local variable
void f()
{
   R r; // ref class on stack???
   r.DoSomething();
   ...
}

Here r looks like a stack object, but even your Aunt Mopsie knows managed classes can't be physically allocated on the stack; they must be allocated from the managed heap. So? The compiler can make this snippet work as expected by generating the equivalent of this:

// how the compiler sees it.
void f()
{
   R^ r = gcnew R; // allocate on gc heap
   try
   {
      r->DoSomething();
      ...
   }
   finally
   {
      delete r;
   }
}

The object doesn't live on the physical stack, but who cares? The important thing is that the local variable syntax behaves the way any C++ programmer would expect. In particular, r's destructor is invoked before leaving f. And speaking of destructors, the same reasoning explains why C++/CLI restored deterministic destruction: so destruction follows the same semantics every C++ programmer has come to know and love. Yippee! Nondeterministic destruction was one of the worst nightmares of the Managed Extensions. C++/CLI maps the destructor to Dispose and introduces a special ! syntax for finalizers. The memory used by reference objects still isn't reclaimed until the garbage collector gets around to it, but that's no big deal. C++ programmers don't care so much that the memory be reclaimed the moment an object is destroyed; what really matters is that the destructor run when expected. C++ programmers frequently use the construct/destruct pattern to initialize/release non-memory resources like file handles, database locks, and so on. With C++/CLI, the familiar construct/destruct pattern works like you'd expect, for reference as well as native classes. The Redmondtonians deserve credit for owning up to the problems with the Managed Extensions-and for fixing them.

Future Fun

Among the more intriguing sections in the Rationale is one called "Future Unifications." It gives some tantalizing hints about where C++/CLI might go in the future. For example, you can't currently derive native classes from managed ones or vice versa. But you could achieve the same effect by implementing a workaround: include the "base" class as a data member, then write passthrough wrappers that do nothing but invoke the contained instance. Well, that sounds pretty formulaic, so why couldn't the compiler do it? The compiler could represent mixed objects as a two-part object with one CLI object containing all the CLI parts and one C++ object containing all the C++ parts.

And here Sutter reports an interesting anecdote: when he first showed this mixed-class idea to Bjarne Stroustrup, Bjarne went to the bookshelf "and opened a book to show where, despite criticism, he had always insisted that C++ must not require objects to be laid out contiguously in a single memory block." No one at the time saw any benefit to noncontiguous objects-but then no one could ever have anticipated .NET and the CLI. Bjarne's insistence on leaving the noncontiguous door open makes mixed objects possible. Don't be surprised if you see them in a future version of C++/CLI. And the moral is: when you're designing a new language or complex program that you expect will live forever and grow in unexpected ways, don't make unnecessary assumptions just because it makes your life easy!

The Rationale offers another interesting historical tidbit: the Redmondtonians' original internal name for C++/CLI was MC^2. As in M(anaged)C(++), with a tip of the hat (^) to Albert Einstein. But, as Sutter says, that was just "too cute." I agree. Do you really think anyone would embrace C++/CLI if it were called MC^2? In deciding to call it C++/CLI, the architects again followed Bjarne's footsteps. "I picked C++ because it was short, had nice interpretations, and wasn't of the form 'adjective C'," Bjarne says. C++/CLI suggests that C++ comes first, and also deliberately avoids the form "adjective C++."

The Rationale justifies other C++/CLI extensions like property, gcnew, generic, and const-and concludes with a useful FAQ section at the end. For details, download the Rationale and read it yourself. And speaking of const, what about my original question-Why does C++/CLI allow const data, but not const functions? The short answer is that CLI doesn't let you put a modopt/modreq directly on a function (though the CLI does actually have a way to encode this information in the metadata, it's just not tested). At least, not yet, as the Rationale is careful to say, implying that this feature may one day be added.

The Evolution of Programming

C++/CLI makes C++ a first-class CLI citizen. Moreover, if you read the Rationale, you'll realize it does so with minimum perturbation of C++. And C++ is still the best language for systems programming, because it provides more direct access to the CLI than any other language, and because you can still drop down to C if you want to call the old Win32® API-which will remain with us for a while longer, I'm sure.

To understand how important C++/CLI is and what it represents, you have to consider where we are in the evolution of programming. Let me give you a brief and idiosyncratic whirlwind review. In the old days, programmers used toggle switches to write their programs. Paper tape was an improvement over that, but each computer had its own "machine" language. As computers proliferated, programmers had to write their programs all over again for each new machine. Well, that was pretty tiresome, so programmers invented high-level languages like FORTRAN, BASIC, and C, that used something called a "compiler" to translate the high-level language into machine instructions for each machine. Figure 2 illustrates this. Now you could write your program once and compile it for different machines. Cool! C became the language of choice for systems programming because it was the "lowest-level high-level language," meaning it introduced the least baggage between itself and the machine. Most operating systems in use today are written in C, perhaps with a few performance-critical or hardware-interfacing sections coded in assembly language.

Many years later, C++ improved on C by making it object-oriented and, well, more fun. To quote Bjarne again: "C++ was designed so that the author and his friends would not have to program in assembler, C, or various modern high-level languages. Its main purpose is to make writing good programs easier and more pleasant for the individual programmer." C++ was great, but high-level languages didn't talk to each other very well. If you wrote some code in C++, you couldn't use it in BASIC or vice versa, at least not without great difficulty. Each language operated inside its own private world. That was fine for standalone apps, but as apps grew more complex, and the environment more distributed, the need to share code became more pressing. Ever since the first subroutine, programmers have sought the ultimate goal of totally encapsulated reusable components: little building blocks of software that programmers can assemble to create applications. And why should all the pieces have to be written in the same language?

Various solutions to the interoperable components problem evolved over the years. From the beginning, languages came with libraries (think C runtime and printf). In the Windows world, DLLs provided delay loading (the Dynamic in DLL) to conserve memory. DLLs also afforded interoperability because languages like Visual Basic and COBOL could call DLLs by introducing import statements that directed their compilers to emit the proper C-linkage calls into the DLL. But the linkage between apps and DLLs is too tight, too brittle, and too apt to break. Each app has to know the names and signatures of each entry in the DLL. Also, calling the other way (from DLL to app) is cumbersome, as you have to pass function pointers as callbacks. So programmers invented VBX, which became OCX, which became COM. Happily, COM is language-neutral: it has "type libraries" so languages don't have to know functions at link time; they can interrogate the typelib at run time. COM is pretty cool, but notoriously difficult to program.(Ironically, COM is most difficult to program in C++, the language in which it was conceived and implemented!) COM has other problems, too: it's too low-level and doesn't address things like security or memory management.

The Ultimate Goal

Now it's 2007 and we have the .NET Framework and its standardized subset, CLI. CLI solves the reusability problem in a completely different way, by inserting a new layer of abstraction between programming languages and the machine. Instead of generating machine instructions, compilers now generate MSIL code, which the CLI virtual machine/just-in-time (JIT) compiler then compiles to machine code on the fly. The virtual machine (VES, or Virtual Execution System, for those who love acronyms) provides a level of abstraction above the machine. Virtual machines aren't new. In fact, they've been around quite a while. Languages like Pascal and ZIL (the Zork Implementation Language, used internally to write games at Infocom, where I was once employed), work by compiling high-level programs into P-code (or Z-code), which is then interpreted by a virtual machine. But the CLI provides a common virtual machine (the C in CLI) all languages can use. The CLI supports fundamental concepts like classes, properties, inheritance, reflection, and so on. The VES/VM provides things like memory management and security, so programmers don't have to fret buffer overruns or other bugs that might open the door to malicious viruses. As the .NET Framework and the CLI have become more popular, high-level languages like Visual Basic, Fortran, COBOL, and C# have all morphed to more and more resemble each other, and to C++, because they must all support the underlying CLI concepts-classes, inheritance, member functions, and the rest. Each language still retains its distinct feel, so programmers don't have to retool themselves to use the .NET Framework; they only have to learn a few new concepts.

So now programmers can write classes in whatever language they choose and other programmers can consume those classes in whatever language they choose. Anyone can program any component in any language, and all the components work together seamlessly with very little programming effort. They all benefit from security, garbage collection, and other infrastructure features (the I in CLI). And when the Redmondtonians add new CLI features tomorrow, all languages will benefit from those, too. With Windows Vista™ and the .NET Framework 3.0 (which has something like 10,000 new classes associated with new technologies like Windows Presentation Foundation, Windows Communication Foundation, Windows Workflow Foundation, and Windows CardSpace™), Windows itself is being rewritten as CLI classes. It seems the goal of reusable, language-independent interoperable components has finally arrived. This represents a huge paradigm shift, and yet one which, amazingly, has proceeded in a relatively gradual, evolutionary manner. You should rejoice! Programming really is getting easier.

If you step back to contemplate all that, how could C++ not participate in this Brave New World? That's where C++/CLI comes in! Without C++/CLI, C++ would be in the lonely position of being the only modern programming language you can't use to program Windows! C++ might otherwise die a slow death or at least become severely marginalized. C++/CLI ensures that won't happen. It guarantees that programmers (like me) who love C++ can go on using it for generations to come. Figure 2 illustrates the paradigm shift to a virtual instead of physical machine and where "C++" fits in with and without "/CLI".

Figure 2 How Programming Evolved
Figure 2 How Programming Evolved(Click the image for a smaller view)
Figure 2 How Programming Evolved
Figure 2 How Programming Evolved (Click the image for a larger view)

Farewell, Though Not Goodbye

And now a special message. This is the last time Yours Truly will pen (type?) his name to C++ At Work. My column is being retired. I like to think it's in honor of my longevity-you know, the way teams retire the numbers of great athletes? My departure does not reflect any lack of commitment to C++ on the part of Microsoft or MSDN®Magazine. Rather, it reflects the simple fact that after 164 columns (this is my 165th) spanning almost 14 years, Mr. DiLascia is getting a little tired. And here an interesting anecdote: Josh Trupin, the Executive Editor, recently called me the "Cal Ripken of MSDN Magazine." Well, I'm not a baseball fan (I like football better) but I know who Cal Ripken is. I was curious, though: how many games did Cal actually play? I searched the Web to find the answer: 2,632. I also found a bio that said something like, "most experts think Cal would've been a better player if he'd taken a break." Well, I took that as a message. But never fear, it doesn't mean I won't write for MSDN Magazine ever again.

Before I go, I want to thank all my loyal readers who've sent their questions, kept me honest by reporting errors and omissions, and inflated my ego by sending kind words of praise that I'll always treasure. Some readers even helped write my columns for me by testing solutions before I shipped them. I also want to thank all the great folks at MSDN Magazine I've worked with, past and present, in no particular order: Steve, Josh, Joanne, Eric, Etya, Terry, Laura, Joan (both of them), Nancy, Val, and Milo. Did I leave anyone out? These people have generously supported me over the years and given me wide latitude to write mostly whatever I wanted. I especially want to thank Gretchen Bilson. She left several years ago, but she's the one who hired me way back in 1993, when MSDN Magazine was still Microsoft Systems Journal. Thanks, Gretchen!

Well, that's all folks. B4N. That's for now, not forever. In the immortal Austrian monotone of Arnold Schwarzenegger: I'll be back. As always, happy programming!

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.

Gak Pile: Say What?

Now that I've dispensed with formalities, I thought I'd end on a humorous note by sharing with you some of the most, er, unusual questions I've received over the years. I've been collecting these in a folder named Gak, as in: "Gak!" I've tried to group them in categories—for example, I always get a steady dose of language questions.

Where can I download C++?

Everybody speaks so highly of C++, why is it? What is the exact kind of stuff you can do in C++ and you can't on Visual Basic or Delphi? I've worked in both these, even in Visual C++®, and most everything I ever needed I've found in all of these languages. Is it just because most old senior programmers are used to it, it sounds fancy?

Gosh, I hope the "old senior programmers" doesn't mean me. Pointers are often a source of confusion, as these next questions attest:

I still could not understand pointers (var, object, functional) kindly explain me or send me some study materials.

I'm new to C++ and I was wondering what Pointers are and what they are used for. Many books I've seen described this in a very complex manner. I know that pointers point to a specific memory location but I don't really know why you need them. I always thought that the computer keeps track of each location automatically. Please reply soon!

(I forwarded this last question to my fellow writer pals Matt Pietrek and John Robbins for advice. Matt offered this suggestion: "Tell him he's right. Pointers are an old, obsolete concept. There's no need to learn about them now." John wrote, "I thought a Pointer was a kind of dog. You mean there's more?")

Moving along, I always get a compiler question or two.

Dir [sic] Paul, During compile time. How Compile link MFC library? Sometimes DLL compile. Because Hooking. So far, I'm not compile this source. Please. How to compile or send source. Have a Nice Day.

But more often I get Windows questions:

I came upon your keyboard/mouse detector program, it was an astounding experience for me. Yet, I have a question for you, as of now your program works if you have the "time counting panel" focused, is there a way to make it so that the program still does its job when it isn't on focus? I tried to figure this out but I am coming to a deadend so I email to ask, thank you for your time.

The thing is, I'm not sure what keyboard/mouse detector program he's referring to. I can't recall ever writing one. You'd think I'd remember, if it was astounding. Oh well.

Every now and then I get a math question:

I'm working on a program that uses a certain sequence of operations to "code" a letter, and later "decode" the letter. I'm having extreme difficulty in the decode part, where I have to do the reverse of the % operation. I know what the remainder is, and I know what you percent by, but I can't seem to find out what the starting number would be. For example, if the remainder of the operation was 22, and the operation would be (the number)%63, what would the number be? If you can help me, that would be great.

Hm. I can understand why this chap might have "extreme difficulty" decoding his letters, since there are an infinite number of solutions for n%a=b, where a and b are given and n is unknown. Alas, I suspect he didn't get an A in the class. Speaking of classes, I get a steady dose of questions that clearly fall into the "can you do my homework for me?" category.

I want a short program to: Enter 20 numbers and arranged [sic] them in assending [sic] order.

Could I get some help with these two programs? I need them tomorrow before noon. Please help me!

Write a program that will request a users beginning balance, request withdrawals debited against the account until the users enters a -1 or user's account balance goes below zero. Print the user's beginning balance, ending balance, total number of checks written, total amount of checks written and the check average.

Write a payroll program and print a users biweekly net pay. Ensure that the user provides their hours worked and pay rate. Also accommodations should be made for overtime pay. Assume that all pay a total of 18 percent of their gross pay towards deductions. Print the users pay, net pay and annual salary.

(There's a #3 too, but I'll spare you.) I often get questions that include long pages of source code, which I usually delete without reading. I sometimes get questions about topics that have nothing to do with my column—like Visual Basic, Microsoft Access™, or how to program Excel®. I even get questions from ordinary users, like "My machine crashed—what should I do?" or "How come to shut down my computer I have to press the Start button?"—which, when you think about it, is a very sensible question! Why indeed? (Raymond Chen answers this question.) Then there are the ones that defy categorization.

Dear Paul, Can you kindly guide to me to some statistical site which shows the average number of lines of Visual C++ code that an average developer can write in a day? Thanks for your help.

Dear Mr DiLascia, How can I be like you?

Dear Sir: I buy the MSDN to so much, look you in MSDN for MFC, feel so good, so ask you a question about "What can want to work for Microsoft." Very like the programming for Visual C++. I have dream then, is want to work for Microsoft, this is for skill, for value, for dream. Today, I can use MFC write the multimedia programming. In school, I can say I am goodhand. But, my English is so-so. I not girlfriend to five year for programming. Because programming is so fun. Maybe you will think, this good for nothing. I can understand. But hope you can reply letter to me. Tell me, What can want to work for Microsoft, and what can I do. Thank you very so much.

What I love is how this fellow's enthusiastic-geek personality shines through, despite "so-so" English (which, I assure you, is way better than my Mandarin!). I usually try to respond to most questions, even if all I can say is "Sorry, I don't know." Many readers are surprised when I actually write back.

Paul, I was not sure that a famous author like you would so readily and promptly help a stranger. You are not only a very knowledgeable professional, but also a very nice person. Thank you.

Well gosh—thank YOU, Mike! Finally, there's this gem:

May I know what is psychic window in windows programming. I don't know the spelling actually. My friend asked me this. Plz help me.

I'm afraid I don't know the answer to that one. Perhaps he should consult Uri Geller. If anyone out there knows what a psychic window is, please drop me a line—via cosmic telepathy.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: Let My People Code

{End Bracket}
Let My People Code
Charles Petzold

In the 1920s, Hollywood was in trouble. Plagued by scandals and a general perception that the movies had gone too far in portraying sex and violence, there was a distinct danger that the government might step in. The studios responded by creating the Hays Office to police themselves and clean up the silver screen. The Hays Office established rules involving everything from proper language to appropriate displays of affection, and became the main arm of non-governmental censorship of the movies.

Movie writers and directors responded in the mid-1930s by finding creative ways around these restrictions. One such workaround was a new genre called the screwball comedy, which featured romance, slapstick, and wisecracks. There was little danger of lewd kissing in the screwball comedy-the romantic leads were often at odds for much of the movie. Even better, the dialogue was so fast and so hip that the censors couldn't even hear it. The result: classics such as It Happened One Night, Twentieth Century, and Bringing Up Baby-movies that would never have been conceived if writers and directors were free to do whatever their filthy minds desired.

Creativity often blossoms not when there is total freedom, but when severe restrictions limit the creator. Engineers as well as artists are familiar with this concept. When an oxygen tank exploded in Apollo 13 on its way to the moon, great creativity and ingenuity were required to make fixes. The creativity arose directly from the limited materials at hand.

I was reminded of this paradox recently while enjoying some of the most challenging and amusing programming I've ever done-and I wasn't even using a proper programming language! Lately, I've been getting my programming kicks with XAML, the Extensible Application Markup Language that forms a significant part of the Microsoft® Windows® Presentation Foundation.

XAML can access powerful classes in the Windows Presentation Foundation for performing layout and displaying graphics and animations. XAML can even be classified as a declarative programming language. But compared to more familiar programming languages, XAML is missing many basic programming features: There are no loops in XAML, no conditionals, and no way even to add or multiply numbers.

And yet, the more I restricted myself to XAML-only problem-solving-a state of existence I thought of as "living on Planet XAML"-the more creative I got. I realized that I could define composite graphics transforms in XAML that would essentially multiply matrices, thereby providing all the multiplication and addition I needed. And I found that I could simulate arrays in XAML by using a ListBox to hold multiple elements and then index those elements with data bindings.

My proudest achievement came with a XAML clock application. I wanted circular tick marks around the circumference of the clock. Customarily a clock has 12 large tick marks and 48 small tick marks, but without a for loop these tick marks would normally require 60 separate XAML elements. Alas, I just couldn't bring myself to create that much repetitive markup. I moaned over this problem for days until the breakthrough came. I could create precisely the tick marks I wanted by drawing two circles using dotted lines. Not only did it work, but I realized that I had done the job with just two graphics objects rather than the sixty most other clock applications required.

Writing XAML is fun. Writing XAML is challenging. Writing XAML is mind-expanding. But in some people's minds, writing XAML is an aberration. XAML is not supposed to be hand-written. As one Microsoft blogger put it, "XAML is for Tools," and over the months prior to its release, some XAML features have been removed because they were strictly for the benefits of humans rather than tools. Of course, XAML-generating tools such as Visual Studio® and Microsoft Expression® Interactive Designer may stimulate our aesthetic creativity, but they do nothing for our coding creativity.

Will future programmers even bother to learn XAML syntax? Or will they treat it as just "some weird XML stuff" that Visual Studio generates to save the layout of buttons and comboboxes?

Interactive designers and code generators certainly have their place in the modern programmer's arsenal. I'm sure they save lots of time. But let's not forget who we are. We are programmers. We are skilled in the art of writing powerful code. We get our kicks from using minimum code for maximum effect. We can make languages like XAML do things it was never intended to do. We can make it sing.

Charles Petzold is a contributing editor to MSDN Magazine and the author of Applications = Code + Markup: A Guide to the Microsoft Windows Presentation Foundation (Microsoft Press, 2006).