 MSDN Magazine December 2005

MSDN Magazine December 2005

Power to the Pen: The Pen is Mightier with GDI+ and the Tablet PC Real-Time Stylus

Charles Petzold Can't Commit?: Volatile Resource Managers in .NET Bring Transactions to the Common Type

Here Juval Lowy discusses the benefits you'll get when you implement transactions in your non-database applications, with a little help from volatile resource managers in the .NET Framework 2.0.

Juval Lowy UI on the Fly: Use the .NET Framework to Generate and Execute Custom Controls at Run Time

Creating UI controls on the fly can be accomplished via run-time code generation. And there are lots of reasons to do so. Generating these controls once and then reusing them as needed is more efficient than generating the controls each time. Read on.

Morgan Skinner Got Directory Services?: New Ways to Manage Active Directory using the .NET Framework 2.0

System.DirectoryServices is a managed code layer on top of Active Directory Service Interfaces, and you can employ it to better manage Active Directory from your code. Here Ethan Wilansky helps you get started.

Ethan Wilansky Ten Essential Tools: Visual Studio Add-Ins Every Developer Should Download Now

James Avery does it again with his popular list of developer tools. This time he covers the best Visual Studio add-ins available today that you can download for free.

James Avery Editor's Note: The Dark Side of Being a Guru

Back in the September 2005 issue of MSDN Magazine, we offered advice to our readers on how to become a guru. Perhaps "guru" is too strong a word, but it conveys the essence of our thoughts on the subject.

New Stuff: Resources for Your Developer Toolbox

Databeacon Smart Client software allows users to perform data analysis, turning relational data from any data source into online analytical processing (OLAP) cubes that can be explored and manipulated using one of three Databeacon viewers.

Marnie Hutcheson Advanced Basics: What's My IP Address?

If you're like me, you regularly do tech-support for family, friends, and neighbors. You can't go to a party without hearing the familiar refrain: "I've just got a quick question. " It's always something—their Internet connections get dropped, they've got a virus, they can't install some piece of hardware, or some file has gone missing.

Ken Getz Test Run: Software Testing Paradoxes

Paradoxes are fun. In this month's column I show you three interesting cases that can occur when you are performing software testing. They're fundamentally mathematical in nature, and they can be a useful addition to your troubleshooting arsenal.

James McCaffrey Cutting Edge: Flexible Custom Data Views

ASP. NET 1. x introduced some powerful and useful data-bound controls. However, none were designed specifically to manage the view of a single record. When you build master/detail views, you need to display the contents of a single record.

Dino Esposito Service Station: An XML Guru's Guide to BizTalk, Part 2

In my last column, I provided a brief introduction to BizTalk® Server 2004 for XML developers (see Service Station: An XML Guru's Guide to BizTalk Server 2004, Part I). I covered the product evolution, core architecture, and several aspects of the underlying messaging layer, all of which have helped make BizTalk Server 2004 the powerful integration technology it is today.

Aaron Skonnard .NET Matters: BigInteger, GetFiles, and More

Stephen Toub Pure C++: Reflecting on Generic Types

A funny thing happened to templates on their way to the common language runtime (CLR)—they lost their {type} identity. This is analogous to what happens with macros under native programs. Just as the C/C++ compilers have no awareness of macro preprocessor expansions, the CLR has no awareness of template instantiations.

Stanley B. Lippman C++ at Work: Layered Windows, Blending Images

Paul DiLascia { End Bracket }: Trustworthy Software

The Authenticode dialog box that users see when they download software from the Internet, asking them if they trust the publisher and want to install the software is, in my opinion, a useless and annoying waste of time that provides no safety whatsoever.

David S. Platt


Power to the Pen: The Pen is Mightier with GDI+ and the Tablet PC Real-Time Stylus

Power to the Pen

The Pen is Mightier with GDI+ and the Tablet PC Real-Time Stylus

Charles Petzold
This article discusses:
  • InkOverlay versus the Real-Time Stylus
  • Implementing Real-Time Stylus plug-ins
  • Saving strokes and rendering with GDI+
  • Disappearing ink, packet transformations, and other fun activities
This article uses the following technologies:
Tablet PC, .NET Framework, C#
Code download available at: TabletPC.exe (170KB)
Contents
What's Wrong with InkOverlay?
Implementing the Interface
Packets of Data
Converting the Points
Sync and Async
The Uniclass Real-Time Stylus
The Other Methods
Saving Strokes and Ink
The Pen Problem
Rendering with Windows Forms Pens
Stylus Pressure
Disappearing Ink
Chaining the Plug-Ins
The Stylus is a Method Call!

M
any programming interfaces are the result of a compromise between the polar ideals of simplicity and versatility. Some, however, break the interface in two and pursue both goals independently. They keep the simple programming interface for many common programming tasks, but also provide a versatile interface for more unusual needs.

Such a separation between the simple and versatile characterizes the application programming interface for the Tablet PC. Much of the Tablet PC API is remarkably easy. With just a few lines of code you can attach an InkOverlay object to any Windows® Forms control and start doing some of the stuff Paul Yao demonstrated in his article "Add Support for Digital Ink to Your Windows Application" in the December 2004 issue of MSDN®Magazine.

Yet, as a result of this simplicity, InkOverlay has some severe and inescapable limitations. If InkOverlay doesn't quite meet your needs, you may want to make the leap to the other Tablet PC programming interface—the far more versatile Real-Time Stylus. That's what this article is all about.

What's Wrong with InkOverlay?

Generally, you attach an InkOverlay object to a control (or form) to perform two vital functions. First, InkOverlay renders stylus input on the screen using a pen described by a DrawingAttributes object. Second, the stylus input is accumulated in an Ink object as a collection of Stroke objects. (A single stroke occurs when the user touches the stylus to the screen, moves it, and then lifts it.) Ink objects can be saved to files in various formats, copied to the clipboard, or passed to a recognizer for conversion to text or gestures.

This is great stuff, but there are several limitations. Let's start with the pen that InkOverlay uses to render the strokes of the stylus. This is an object of type DrawingAttributes, and a cursory examination of the DrawingAttributes properties reveals that this is not the Pen object that Windows Forms programmers have known and loved for the past several years. The Windows Forms Pen class in the System.Drawing namespace (and part of a graphics interface sometimes called GDI+) lets you base pens on bitmapped images and gradient brushes. But the pen used by InkOverlay to render stylus input is the crushingly boring two-decades-old Windows API pen.

If you want the strokes of the stylus to be rendered with a System.Drawing.Pen object, and perhaps to base this pen on a bitmapped image or gradient brush, you need to use the Real-Time Stylus. Or you may want to restrict the rendered strokes of the stylus to a subset of the control. With InkOverlay, you can set a rectangular clipping area, but if you want to clip to any arbitrary region, again you need to use the Real-Time Stylus.

The InkOverlay conveniently draws the strokes as the user is moving the stylus, but your program may need to react to stylus movement in ways other than simple line and curve drawing. You can prevent InkOverlay from drawing entirely, but for efficient substitutions of the InkOverlay rendering, Real-Time Stylus is the better tool.

The InkOverlay object gives you a certain amount of information about what's going on with the stylus. The NewPackets event can tell you when the stylus is being dragged across the control, and the Stroke event notifies you when the stylus is lifted and the stroke has been completed. However, if you require more information about stylus activity, the InkOverlay can't help and you'll need to turn to the Real-Time Stylus.

The Real-Time Stylus provides notifications of stylus activity in a form similar to mouse events. However, instead of using MouseDown, MouseUp, and MouseMove events, you implement methods named StylusDown, StylusUp, and Packets. How you handle that stylus input—how you draw it to the screen and how you save it—is entirely up to you. There is no default rendering and nobody saves the data for you.

Implementing the Interface

While most of the Tablet PC classes are located in the Microsoft.Ink namespace, the Real-Time Stylus classes are relegated to the namespaces Microsoft.StylusInput and Microsoft.StylusInput.PluginData. Of primary importance are two interfaces defined in Microsoft.StylusInput. As you'll recall, a Microsoft ®.NET Framework interface looks a lot like a class in that it contains methods and properties, but these methods and properties have no bodies. A class "implements" the interface by providing code for all the methods and properties that are defined in the interface.

The two Real-Time Stylus interfaces are named IStylusSyncPlugin and IStylusAsyncPlugin, and they both define the same 15 methods and one property. To use the Real-Time Stylus in any nontrivial way, you must define a class that implements one (or both) of these interfaces, which means that you must create a class that provides code for these 15 methods and one property. That seems like an onerous task—it took me some time to accept the fact that I was actually being required to create such a class—until you realize that most of the methods can simply have empty bodies in the form of matching curly brackets.

A class that implements one or both of these interfaces is called a plug-in and, in this article, all my classes that implement these interfaces will end in the word Plugin. Figure 1 shows a simple plug-in class called SimpleRendererPlugin.

Figure 1 A Simple Real-Time Stylus Plug-In (Close Figure 1)
using System;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Windows.Forms;
using Microsoft.StylusInput;
using Microsoft.StylusInput.PluginData;

class SimpleRendererPlugin : IStylusAsyncPlugin, IStylusSyncPlugin
{
    Control ctrl;
    Point ptLast;

    // Constructor
    public SimpleRendererPlugin(Control ctrl)
    {
        this.ctrl = ctrl;
    }

    // Required property
    public DataInterestMask DataInterest
    {
        get
        {
            return DataInterestMask.StylusDown |
                   DataInterestMask.Packets |
                   DataInterestMask.StylusUp;
        }
    }

    // Implemented methods

    public void StylusDown(RealTimeStylus sender, StylusDownData data)
    {
        ptLast = new Point(data[0], data[1]);
    }

    public void Packets(RealTimeStylus sender, PacketsData data)
    {
        using (Graphics grfx = ctrl.CreateGraphics())
        {
            grfx.PageUnit = GraphicsUnit.Millimeter;
            grfx.PageScale = 0.01f;
            grfx.SmoothingMode = SmoothingMode.AntiAlias;

            for (int i = 0; i < data.Count; i += data.PacketPropertyCount)
            {
                Point pt = new Point(data[i], data[i + 1]);
                grfx.DrawLine(SystemPens.ControlText, ptLast, pt);
                ptLast = pt;
            }
        }
    }

    public void StylusUp(RealTimeStylus sender,  StylusUpData data)
    {
        using (Graphics grfx = ctrl.CreateGraphics())
        {
            grfx.PageUnit = GraphicsUnit.Millimeter;
            grfx.PageScale = 0.01f;
            grfx.SmoothingMode = SmoothingMode.AntiAlias;
            grfx.DrawLine(SystemPens.ControlText, ptLast,
                new Point(data[0], data[1]));
        }
    }

    // Methods with empty bodies

    public void CustomStylusDataAdded(RealTimeStylus sender,
        CustomStylusData data) {}
    public void Error(RealTimeStylus sender, ErrorData data) {}
    public void InAirPackets(RealTimeStylus sender,
        InAirPacketsData data) {}
    public void RealTimeStylusDisabled(RealTimeStylus sender,
        RealTimeStylusDisabledData data) {}
    public void RealTimeStylusEnabled(RealTimeStylus sender,
        RealTimeStylusEnabledData data) {}
    public void StylusButtonDown(RealTimeStylus sender,
        StylusButtonDownData data) {}
    public void StylusButtonUp(RealTimeStylus sender,
        StylusButtonUpData data) {}
    public void StylusInRange(RealTimeStylus sender,
        StylusInRangeData data) {}
    public void StylusOutOfRange(RealTimeStylus sender,
        StylusOutOfRangeData data) {}
    public void SystemGesture(RealTimeStylus sender,
        SystemGestureData data) {}
    public void TabletAdded(RealTimeStylus sender,
        TabletAddedData data) {}
    public void TabletRemoved(RealTimeStylus sender,
        TabletRemovedData data) {}
}

This plug-in includes code for the DataInterest property and the three most commonly implemented methods in these interfaces: StylusDown, Packets, and StylusUp. The other 12 methods are defined at the bottom of the class with empty bodies. In addition, I've given SimpleRenderPlugin a constructor. An explicit constructor is not required in plug-ins but is often useful.

Implementing the DataInterest property is essential: the property must return the members of the DataInterestMask enumeration corresponding to the methods of the interface that you're interested in; otherwise, the methods will never be called. The three methods that SimpleRenderPlugin provides are called when the stylus touches the screen, when it is moved, and when it is released.

In the downloadable source code for this article, available from the MSDN Magazine Web site, the SimpleRendererDemo project includes SimpleRenderPlugin.cs and SimpleRendererDemo.cs. All source code was developed under Visual Studio® .NET 2003 with the .NET Framework 1.1 and the Tablet PC SDK 1.7 installed. Projects were created using the Empty Project option and assigned references to the System, System.Drawing, System.Windows.Forms, and Microsoft.Ink assemblies, the last of which shows up in the Add Reference dialog box as "Microsoft Tablet PC API." It is possible to install the Tablet PC SDK 1.7 on a machine without a tablet, in which case the mouse mimics the stylus.

The SimpleRendererDemo class inherits from Form and includes the following code in its constructor:

RealTimeStylus rts = new RealTimeStylus(this);
rts.SyncPluginCollection.Add(new SimpleRendererPlugin(this));
rts.Enabled = true;
The code first creates an object of type RealTimeStylus. Except in a very special case, the RealTimeStylus object must be associated with a particular control and only receives stylus input directed to that control. In this case, the RealTimeStylus is associated with the application form.

The second statement creates an object of type SimpleRendererPlugin (again with the form object as an argument to the constructor) and adds that object to the SyncPluginCollection property of RealTimeStylus. As the name implies, this SyncPluginCollection is a collection of objects that implement the IStylusSyncPlugin interface. Finally, the RealTimeStylus object is enabled and you're ready to get started. You can now use the Real-Time Stylus to draw on the form's client area.

Packets of Data

The StylusDown, Packets, and StylusUp methods in SimpleRendererPlugin are called when the stylus is touched to the screen over the form's client area, when it moves while touching the screen, and when it's lifted. The calls to Packets (and StylusUp) continue even if the stylus moves beyond the form's client area.

These three methods have arguments of type StylusDownData, PacketsData, and StylusUpData, respectively, but these three classes all derive from StylusDataBase and they are basically the same. (The name of the StylusDataBase class is a bit awkward—the name does not refer to a "database" for the stylus, but instead a base class for stylus data.) StylusDataBase implements an indexer of type int, so you can essentially treat the object as an array of integers. StylusDataBase also has a GetData method that returns this array of integers directly.

These integers are grouped into "packets." Each packet contains the X and Y coordinates of the stylus and some other general information, primarily stylus pressure. The StylusDown and StylusUp methods are accompanied by just one packet. The Packets method, as the name implies, can be accompanied by more than one. The total number of integers in the array is available from the Count property of StylusDataBase. The number of integers per packet is provided by PacketPropertyCount. The number of packets is therefore Count divided by PacketPropertyCount. The for loop in the Packets method in SimpleRendererPlugin shows one way to access these packets.

The first two integers of each packet are always the stylus X and Y coordinates relative to the upper-left corner of the control. Units are hundredths of a millimeter, a coordinate system that in the dark ages of the Win32® API was referred to as HIMETRIC.

For StylusDown, the SimpleRendererPlugin class simply saves the coordinates of the stylus as the ptLast field. The Packets method does the drawing. It first obtains a Graphics object for the control (that's why the class wanted a Control object in its constructor), then sets the PageUnit and PageScale properties so that the page units of the Graphics object agree with the tablet data coordinate system, and turns on antialiasing. For each point in the packet, a line is drawn using SystemPens.ControlText, which is a predefined pen based on the default foreground color of controls. The last point is again saved in ptLast.

The StylusUp method is similar to Packets except that only a final point is involved, and it doesn't need to be saved. Though I generally implement StylusUp out of a sense of symmetry, it can usually be ignored without anybody noticing.

Converting the Points

The screen resolution under Windows is generally set at 96 or 120 pixels per inch. The resolution of the Tablet PC packet data is 2540 units per inch—approximately 20 to 25 times finer. When I first started working with the Real-Time Stylus, I was concerned that many Packets calls would come through where subsequent points would correspond to the same pixel. Code that called DrawLine when there was essentially no line to draw seemed wasteful.

For that reason, in my early Real-Time Stylus coding, I often converted the packets to pixels and then checked whether the point was truly different from the last point before calling DrawLine. Recently, however, I've decided to let the Graphics object do this conversion for me. I am no longer very concerned about the potential problem here. I have examined some of the Packets calls and have determined that there really are not a lot of packets falling between the pixels.

Later in this article, I'll show you a program using a pen created from a bitmapped image to demonstrate how to convert the packets to pixels. The motivation there, however, resulted from an image-scaling problem rather than a superfluous-packet problem.

Sync and Async

The Real-Time Stylus can deliver packet data to your program either synchronously or asynchronously. The terms are relative to the tablet software rather than to your application, which at first may cause you some confusion. If you choose the asynchronous option, the calls to StylusDown, Packets, and StylusUp occur in the application user interface thread. With the synchronous option, these calls occur in the tablet thread.

How do you choose? First, the two interfaces defined in the Microsoft.StylusInput namespace are named IStylusSyncPlugin and IStylusAsyncPlugin. Because these interfaces implement the same 15 methods and one property, it's just as easy to derive a plug-in from both interfaces rather than just one. The plug-ins shown in this article derive from both interfaces to make it easier for me (and you) to experiment with them. However, some plug-ins can be defined to derive from only one interface to strongly suggest they should be used in a synchronous or asynchronous manner.

The RealTimeStylus class contains two properties named SyncPluginCollection and AsyncPluginCollection. These both work in the same way. Both properties expose collections that include Add and Remove methods. Both collections can accept multiple plug-ins, but a particular plug-in can only be a part of one collection.

The tablet system component delivers data packets to the RealTimeStylus object in the tablet thread. On that same thread, the RealTimeStylus then delivers the packets to each of the plug-ins in the synchronous collection in sequence. Once the synchronous plug-in objects are finished processing, the data is put into an internal queue and control is returned to the system. The application thread picks up the data from the queue and delivers the packets to all the plug-ins in the asynchronous collection in sequence.

As a general rule, you should use synchronous plug-ins only when an immediate response is required, such as during rendering. Other plug-ins are better off in the asynchronous collection. However, a little experimentation reveals that the synchronous collection is more forgiving of sluggish code. With fast code, each Packets call generally delivers only one packet. If slow code in the Packets method can't quite keep up at that pace, then multiple packets will be consolidated into single calls. This consolidation of packets does not occur with plug-ins in the asynchronous collection. (Then again, this consolidation only helps if your code has a per-Packets call problem rather than a per-packet problem.)

Note, too, that the calls to synchronous plug-ins are blocking calls—they block the tablet thread until the calls return from the plug-in objects. This directly affects the responsiveness of the stylus. Thus, a plug-in's design must be carefully thought through before making it synchronous, especially when using synchronization objects (such as semaphores and mutexes) in the plug-in's implementation.

The Microsoft.StylusInput namespace includes two predefined plug-ins. DynamicRenderer (which I'll make use of later in this article) is defined as implementing only the IStylusSyncPlugin interface, suggesting that the important job of rendering works best in response to user input. The GestureRecognizer plug-in implements both interfaces so it can be used in a more versatile manner.

The Uniclass Real-Time Stylus

When experimenting with the Real-Time Stylus, sometimes it's convenient to have a single class in a single source code file that does double duty as both the application form and the plug-in. This is fairly easy to accomplish because any class can inherit from another class (Form, for instance) and any number of interfaces:

class MyAppForm : Form, IStylusSyncPlugin, IStylusAsyncPlugin
{
    ...
}

To implement the interfaces, of course, the class must include the 15 methods and one property defined in the interfaces. The code in the constructor to create RealTimeStylus adds the form object to the plug-in collection simply by referring to the object as this:

RealTimeStylus rts = new RealTimeStylus(this);
rts.SyncPluginCollection.Add(this);
rts.Enabled = true;
Because the plug-in methods are part of the form class, they can simply make calls to CreateGraphics, for example, without prefacing it with a control object.

The Other Methods

Although StylusUp, Packets, and StylusDown are undoubtedly the three most popular methods of the Real-Time Stylus interfaces, some of the other twelve may also be useful when developing some applications.

TabletAdded and TabletRemoved signal when tablet hardware is added or removed from the system. (External tablets generally have USB interfaces.) RealTimeStylusEnabled and RealTimeStylusDisabled signal changes in the Enabled property of the RealTimeStylus object. StylusButtonDown and StylusButtonUp refer to the push button on the side of the stylus that is sometimes used to alter the meanings of strokes.

The tablet can detect the presence of the stylus even if it's not touching the screen. A call to the StylusInRange method indicates that the stylus has come close enough to the tablet to be detected. StylusOutOfRange signals when the stylus has been moved a sufficient distance away from the tablet. These calls occur based on the distance of the stylus from the tablet, regardless of the distance between the stylus and the control.

If the stylus enters the control's air space, then the plug-in will receive a profusion of calls to InAirPackets. These calls stop when the stylus is moved away from the control, and the cessation is also signaled by a call to StylusOutOfRange, even if the stylus is still close to the tablet.

Some movements of the stylus (either in the air or touching the tablet) result in calls to SystemGesture, and a member of the SystemGesture enumeration indicates the gesture. The most common in-air gestures are HoverEnter and HoverLeave. The most common gestures that involve touching the stylus to the tablet are Tap and DoubleTap.

An application can place custom data in the Real-Time Stylus queue by calling the AddCustomStylusDataToQueue method of RealTimeStylus. Plug-ins are notified of the presence of this data by a call to CustomStylusDataAdded. Finally, a plug-in can choose to receive a call to the Error method that signals when an exception is thrown by one of the plug-ins.

Saving Strokes and Ink

The SimpleRendererDemo program has a severe problem. If you draw on the window using the stylus and then, for example, minimize and restore the window, the stylus output will be gone. Nobody is saving the strokes. This is one of the things that makes the Real-Time Stylus so scary. The built-in amenities are gone. When you use the Real-Time Stylus, this is one of the responsibilities you have to assume.

How should strokes be saved? If you were to use InkOverlay as a model, you would store all the strokes in an instance of the Ink class. So, let's try to write a simple plug-in that also saves strokes in an object of type Ink. The SimpleInkSaverPlugin class is shown in Figure 2. The class creates an object of type Ink that is saved as a private field and exposed through a public property. The StylusDown method creates a new ArrayList object that it uses to store the first point. The Packets method stores subsequent points of the stroke in the ArrayList object. When the stroke is completed, the StylusUp method obtains the array of Point objects from ArrayList, and creates a Stroke object that is added to the Ink.

Figure 2 Plug-In That Saves Strokes (Close Figure 2)
using System;
using System.Collections;
using System.Drawing;
using Microsoft.Ink;
using Microsoft.StylusInput;
using Microsoft.StylusInput.PluginData;

class SimpleInkSaverPlugin : IStylusAsyncPlugin, IStylusSyncPlugin
{
    Ink ink = new Ink();
    ArrayList arrlstStroke;

    // Property
    public Ink Ink { get { return ink; } }

    // Required property
    public DataInterestMask DataInterest
    {
        get
        {
            return DataInterestMask.StylusDown |
                   DataInterestMask.Packets |
                   DataInterestMask.StylusUp;
        }
    }

    // Implemented methods

    public void StylusDown(RealTimeStylus sender, StylusDownData data)
    {
        arrlstStroke = new ArrayList();
        arrlstStroke.Add(new Point(data[0], data[1]));
    }

    public void Packets(RealTimeStylus sender, PacketsData data)
    {
        for (int i = 0; i < data.Count; i += data.PacketPropertyCount)
            arrlstStroke.Add(new Point(data[i], data[i + 1]));
    }

    public void StylusUp(RealTimeStylus sender,  StylusUpData data)
    {
        arrlstStroke.Add(new Point(data[0], data[1]));
        Point[] apt = (Point[]) arrlstStroke.ToArray(typeof(Point));
        ink.CreateStroke(apt);
    }

    // Methods with empty bodies
    ...
}

The SimpleInkSaverDemo project included with the downloadable code includes SimpleInkSaverPlugin.cs, a link to SimpleRendererPlugin.cs, and SimpleInkSaverDemo.cs, which has a class that derives from Form and creates a button, label, and panel on the surface of its client area. It creates a RealTimeStylus object based on the Panel object named pnl:

RealTimeStylus rts = new RealTimeStylus(pnl);
rts.SyncPluginCollection.Add(new SimpleRendererPlugin(pnl));
inkplugin = new SimpleInkSaverPlugin();
rts.AsyncPluginCollection.Add(inkplugin);
rts.Enabled = true;
Notice that the renderer plug-in is added to the synchronous plug-in collection while the ink-saver plug-in (saved as a field named inkplugin) is added to the asynchronous plug-in collection because it doesn't have to respond as quickly as the renderer.

The SimpleInkSaverDemo class makes use of the Ink object in two ways. First, an event handler for the panel's Paint event accesses the Ink object to repaint its surface:

foreach (Stroke stk in inkplugin.Ink.Strokes)
    grfx.DrawLines(SystemPens.ControlText, stk.GetPoints());
Secondly, clicking the button on the form causes the program to pass the Ink object to a recognizer that converts it into text to be displayed in the label:
RecognizerContext recoContext = new RecognizerContext();
recoContext.Strokes = inkplugin.Ink.Strokes;
RecognitionStatus recoStatus;
RecognitionResult recoResult = recoContext.Recognize(out recoStatus);
lblOutput.Text = recoResult.TopString;
Figure 3 Ink Recognition Without InkOverlay
Figure 3 Ink Recognition Without InkOverlay
Figure 3 shows some handwriting converted to text by the program. Yes, there can be understandable anxiety when first approaching the Real-Time Stylus because it doesn't implement Ink collection or recognition. I hope this example makes it clear that it's not very hard to add simple implementations of these features.

The Pen Problem

The real problem involved in creating Ink from a Real-Time Stylus application is the pen. When you're using InkOverlay, each Stroke object stored with the Ink has its own DrawingAttributes object that indicates (among other things) the color and width of the pen used to draw the stroke. If the strokes are to be properly redrawn on the screen from the Ink object, the DrawingAttributes object must be set to the same values used when originally rendering the stroke, and they must be taken into account in the repainting logic.

However, one big reason to use the Real-Time Stylus is to render with a GDI+ Pen object. Certainly the color and width of the GDI+ Pen could be stored in a DrawingAttributes object, but what if the pen is based on a hatch brush? One approach might be to save the Pen characteristics in the ExtendedProperties property of DrawingAttributes.

The SimpleInkSaverDemo program attempts to solve this problem by using SystemPens.ControlText both in the renderer plug-in and when redrawing the panel. However, that's certainly not a good general solution, and it can't be used in the other programs discussed in this article. Ultimately, the solution to the problem of reconciling Ink objects and Pen objects will need to be approached on an application basis: just how fancy is your Pen and how much detail about it do you need stored with each Stroke object?

Meanwhile, after knowing that it's possible to accumulate Ink and pass it to a recognizer, you can relax and have some fun. In the remainder of this article I'm going to focus on interesting approaches to rendering rather than saving and recognizing Ink.

Rendering with Windows Forms Pens

The GdiPlusRendererPlugin class included with this article's downloadable code is a renderer that uses a GDI+ Pen object settable through a public property of the plug-in. The plug-in creates a default pen in its constructor based on the foreground color of the control (or form) it's drawing on, and with a width of 53. This may seem like a strange number, but keep in mind that the pen width is in world coordinates, and when used with a Graphics object set for hundredths of millimeters, it's about half a millimeter. It's the same width you'll find in default DrawingAttributes objects and it was chosen because it is equivalent to 1.5 points. Most importantly, it looks about right.

When the plug-in receives a call to StylusDown, it saves the stylus point (as usual) but also clones the pen to use in subsequent drawing, as shown here:

ptLast = new Point(data[0], data[1]);
pnStroke = (Pen) Pen.Clone();
This code ensures that the same pen is used for the entire stroke. The GdiPlusRendererPlugin class also includes a public property named Clip of type Region. The RenderData method (called from both Packets and StylusUp) uses this property to set a clipping region for the Graphics object:
if (Clip != null) grfx.Clip = Clip;

Figure 4 Flat Line Caps
Figure 4 Flat Line Caps

The GdiPlusRendererDemo project includes GdiPlusRendererPlugin.cs and GdiPlusRendererDemo.cs. The GdiPlusRendererDemo class inherits from Form and creates the RealTimeStylus object, as usual. But it also creates a modeless dialog box containing a PropertyGrid control that lets you change the Pen properties interactively.

You can change the pen color, of course, and you'll want to try changing the pen width, as well. Keep in mind that the width is in hundredths of millimeters. A value of 635 is equivalent to one quarter inch. At widths of that size, you'll see something very disturbing happen, as shown in Figure 4.

The problem here is that GDI+ pens begin and end abruptly at the geometric start point and end point of the line. As you draw with the stylus, each piece of the line becomes just a sliver. Fixing this problem is fairly easy: change the StartCap and EndCap properties of the Pen to something other than the default LineCap.Flat. A value of LineCap.Round results in a filled half-circle added onto the ends and works well regardless of how sharp you draw corners, as Figure 5 shows.

Figure 5 Rounded Line Caps
Figure 5 Rounded Line Caps

One of the powerful aspects of GDI+ pens is that they can be based on brushes. These can be solid brushes (SolidBrush), gradient brushes (objects of type LinearGradientBrush or PathGradientBrush), hatch brushes (HatchBrush), or brushes based on bitmaps or metafiles (TextureBrush). When you draw with the latter type, it's as if the pen is cutting a swath through a mask to reveal the brush-covered surface underneath. For example, you might create a gradient brush large enough to cover the client area of a form, with red at the right of the form and blue on the left. If you then base a pen on that brush, lines you draw at the right of the form will be red, and lines at the left will be blue. Lines in the middle will be magenta, and horizontal lines across the form will show the gradient.

The HatchBrushDemo class inherits from GdiPlusRendererDemo and creates a one-half inch-wide pen based on HatchStyle.HorizontalBrick. The resulting pen is set to the Pen property of the GdiPlusRendererPlugin object:

HatchBrush brsh = new HatchBrush(HatchStyle.HorizontalBrick,
                                 Color.White, Color.Black);
gdiplus.Pen = new Pen(brsh, 1270);
gdiplus.Pen.StartCap =
gdiplus.Pen.EndCap = LineCap.Round;
This program also demonstrates the Clip property of GdiPlusRendererPlugin. Some simple code creates a five-pointed star in an array of five PointF structures named aptf. From these points, a closed path is created:
GraphicsPath path = new GraphicsPath();
path.AddLines(aptf);
path.CloseFigure();
When converted to a region, that path becomes the clipping region:
gdiplus.Clip = new Region(path);

The stylus in this program draws bricks clipped to the interior of the five-pointed star, as shown in Figure 6. In a similar spirit, the ImagePenDemo program (which consists of the classes ImagePenDemo and ImagePenPlugin) creates a TextureBrush based on a bitmap or metafile and uses that brush to create the pen.

Figure 6 Stylus Output with Hatch Pen and Clipping
Figure 6 Stylus Output with Hatch Pen and Clipping

Windows Forms is usually quite good about using embedded resolution information in bitmaps to display images at their proper size. But that doesn't seem to be the case for texture brushes. When using a Pen based on a TextureBrush and drawing with page units of hundredths of a millimeter, the bitmap image shrinks in size. This problem can be fixed by setting the Transform property of the TextureBrush object to scale up the size of the image, but I decided it might be useful showing an example where drawing is done in units of pixels.

The ImagePenPlugin class has two public properties named Image and WidthInPoints (my preferred device-independent units). It creates a pen during the StylusDown method:

TextureBrush tb = new TextureBrush(Image, WrapMode.Tile);
float fPenWidth = grfx.DpiX * WidthInPoints / 72;
pn = new Pen(tb, fPenWidth);
pn.StartCap = pn.EndCap = LineCap.Round;
The RenderData method (called from both Packets and StylusUp) converts packet points to pixels and draws a line based on this pen, as shown in the following:
Point pt = new Point((int) (grfx.DpiX * data[i + 0] / 2540 + 0.5),
                     (int) (grfx.DpiY * data[i + 1] / 2540 + 0.5));
if (pt != ptLast)
{
    grfx.DrawLine(pn, ptLast, pt);
    ptLast = pt;
}
The ImagePenDemo class creates the RealTimeStylus object and adds the ImagePenPlugin object to its SyncPluginCollection. The class also creates a modeless dialog box with a PropertyGrid control for the ImagePenPlugin class, letting the user set the Image and WidthInPoints properties directly. (Using the PropertyGrid to set a property of type Image is an easy way to get a File Open dialog box up on the screen.) Once an image is selected and then loaded, each stroke of the stylus reveals more of the image, as you can see in Figure 7.

Figure 7 Stylus Output Using ImagePen
Figure 7 Stylus Output Using ImagePen

Stylus Pressure

The programs shown so far in this article have used only the first two integers of each packet. The third integer, if it exists, always indicates stylus pressure ranging from 0 to 255. The DrawingAttributes object used with InkOverlay and Ink includes a property named IgnorePressure that is normally set to false. By default, the pressure of the stylus on the screen causes the width of the rendered line to vary between 50 percent and 150 percent of its normal width.

Mimicking this behavior is fairly easy, but it's more fun to take it a step further and accentuate the effect. The StylusPressureWidthDemo program (consisting of StylusPressureColorPlugin.cs and StylusPressureWidthDemo.cs) contains the following code that creates a new pen for each packet:

float fWidth = 53;

if (data.PacketPropertyCount > 2)
    fWidth *= 1 + (data[i + 2] - 128f) / 16;

Pen pn = new Pen(ctrl.ForeColor, fWidth);
To reproduce the normal variation associated with stylus pressure, change the denominator in the fWidth calculation to 255.

It's also possible to reflect pressure width in other ways. The StylusPressureColorDemo program (consisting of StylusPressureColorPlugin.cs and StylusPressureColorDemo.cs) varies the pen color from blue to red depending on the stylus pressure:

Color clr = ctrl.ForeColor;

if (data.PacketPropertyCount > 2)
    clr = Color.FromArgb(data[i+2], 0, 255-data[i+2]);

Pen pn = new Pen(clr, 53);
Another possibility for use with a white background is to vary the pen color from white to black based on the pressure:
int iShade = 255 – data[i + 2];
clr = Color.FromArgb(iShade, iShade, iShade);

Disappearing Ink

Entirely invisible ink is a snap, even if you're using InkOverlay. Simply set the RasterOperation property of the DrawingAttributes to the enumeration value RasterOperation.NoOperation. Somewhat more of a challenge is to produce ink that draws normally but fades out over time.

I originally wrote the DisappearingInk program simply for my own amusement, but somebody pointed out to me that this would be a great feature in a note-taking program, particularly when you're working in public and want to hide what you're writing from roving eyes. You get the immediate feedback of ink from the stylus, but then it fades out and can't be seen.

The DisappearingInk program begins with a class named TimeStampedPoint. This class, shown in Figure 8, simply combines a Point structure and a DateTime structure. The read-only property Age indicates the age of the object in milliseconds.

Figure 8 A Point That Reveals Its Age (Close Figure 8)
using System;
using System.Drawing;

class TimeStampedPoint
{
    Point pt;
    DateTime dt;

    public TimeStampedPoint(Point pt)
    {
        this.pt = pt;
        dt = DateTime.Now;
    }

    public Point Point
    {
        get { return pt; }
    }

    public long Age
    {
        get { return (long) (DateTime.Now - dt).TotalMilliseconds; }
    }
}

The DisappearingInkPlugin class maintains two ArrayList objects: alPoints and alStrokes. The first ArrayList is a collection of TimeStampedPoint objects that make up the current stroke. The StylusDown method creates a new alPoints object to signal the beginning of a new stroke. The method also adds that alPoints object to alStrokes—the ArrayList that stores all the strokes. The Packets and StylusUp calls draw the packets normally, but they also add additional TimeStampedPoint objects to alPoints.

The constructor of DisappearingInkPlugin starts a timer. The Tick event handler steps through all the points of all the strokes, and redraws the lines using a color calculated based on the Age property of each point.

Doing strange things with the Real-Time Stylus can become an obsession. On example: my quest to display drop-shadow on stylus input is documented on my Web site at In Search of the Real-Time Drop Shadow.

Chaining the Plug-Ins

As you saw in the SimpleInkSaver program, it's possible to have a chain of multiple plug-ins. This is sometimes useful in dividing the processing of stylus input, perhaps between rendering and saving. But it's also possible for plug-ins earlier in the chain to alter the data seen by plug-ins later in the chain.

If a plug-in wants to change the packet data but retain the same number of data packets, it can simply change all or some of the integer values stored in the data object passed to the StylusDown, Packets, or StylusUp methods. If the plug-in needs to change the number of packets, it calls the SetData method defined by StylusDataBase with the new array of integers. If the SetData method is called with a null argument, then the event is canceled for all subsequent plug-ins.

The remaining two example programs in this article use DynamicRenderer for rendering the strokes. DynamicRenderer is a class included in the Microsoft.StylusInput namespace that implements IStylusSyncPlugin and renders stylus input to the screen based on a DrawingAttributes object.

The DynamicRenderer constructor requires a Control object, and the Enabled property must be set to true. If you also set the EnableDataCache property to true, then DynamicRenderer retains all the strokes and can repaint the control with a call to its Refresh method. However, DynamicRenderer does not expose this collection of strokes so it can't be used to fabricate an Ink object. It doesn't make much sense to use DynamicRenderer by itself; it's more useful in conjunction with other plug-ins. Although the next two programs use the DynamicRenderer constructor for drawing the strokes, they use custom plug-ins for altering the packet data before it gets to DynamicRenderer.

The RotateAroundStart project includes the RotateAroundStartPlugin class (most of which is shown in Figure 9) and the RotateAroundStart class. The plug-in has a constructor that requires an angle (in degrees) as an argument. This is saved in a field. In the StylusDown method, a matrix transform is created that defines a rotation around the point at which the stylus touched the screen:

xform = new Matrix();
xform.RotateAt(fAngle, new Point(data[0], data[1]));
This transform is saved as a field. Both the Packets and StylusUp methods call the TransformPoints method. This method begins by transferring all the packet points to a Point array. It then uses the Matrix object to rotate the points, and stores the transformed points back to the StylusDataBase object.

Figure 9 Transforming Packet (Close Figure 9)
using System;
using System.Drawing;
using System.Drawing.Drawing2D;
using System.Drawing.Imaging;
using System.Windows.Forms;
using Microsoft.StylusInput;
using Microsoft.StylusInput.PluginData;

class RotateAroundStartPlugin : IStylusSyncPlugin, IStylusAsyncPlugin
{
    float fAngle;
    Matrix xform;

    // Constructor
    public RotateAroundStartPlugin(float fAngle)
    {
        this.fAngle = fAngle;
    }

    // Required property
    public DataInterestMask DataInterest
    {
        get
        {
            return
                DataInterestMask.StylusDown |
                DataInterestMask.Packets |
                DataInterestMask.StylusUp;
        }
    }

    // Implemented methods.
    public void StylusDown(RealTimeStylus sender,  StylusDownData data)
    {
        // Create matrix for rotation around point.
        xform = new Matrix();
        xform.RotateAt(fAngle, new Point(data[0], data[1]));
    }

    public void Packets(RealTimeStylus sender,  PacketsData data)
    {
        TransformPoints(data);
    }

    public void StylusUp(RealTimeStylus sender,  StylusUpData data)
    {
        TransformPoints(data);
    }

    void TransformPoints(StylusDataBase data)
    {
        Point[] apt = new Point[data.Count / data.PacketPropertyCount];

        // Get the points.
        for (int i = 0; i < data.Count; i += data.PacketPropertyCount)
            apt[i] = new Point(data[i], data[i + 1]);

        // Transform the points.
        xform.TransformPoints(apt);

        // Set the points.
        for (int i = 0; i < data.Count; i += data.PacketPropertyCount)
        {
            data[i] = apt[i].X;
            data[i + 1] = apt[i].Y;
        }
    }

    // Methods with empty bodies
    ...
}

Now let's see how this class is used. RotateAroundStart inherits from Form and defines iAngle as 30 degrees. After creating a RealTimeStylus object, the program adds to the SyncPluginCollection no fewer than 12 instances of RotateAroundStartPlugIn and 12 instances of DynamicRenderer in an alternating pattern:

for (int i = 0; i < 360 / iAngle; i++)
{
 rts.SyncPluginCollection.Add(new
RotateAroundStartPlugin(iAngle));
    DynamicRenderer dynaren = new DynamicRenderer(this);
    dynaren.Enabled = true;
    rts.SyncPluginCollection.Add(dynaren);
}
Notice that the argument to the RotateAroundStartPlugin constructor is always the same iAngle value. The first instance rotates the packet points 30 degrees, the second instance 30 more degrees, and so forth. The result is a pattern centered around the initial point. Figure 10 shows one of many possible results. But watch out—by installing 24 plug-ins in a chain, RotateAroundStart is certainly pushing the envelope of the capabilities of the Real-Time Stylus. If you draw very quickly, some packets will be dropped and it won't be quite as symmetrical.

Figure 10 Twelve Transforms and Renderings
Figure 10 Twelve Transforms and Renderings

RotateAroundStart overrides the OnDoubleClick method of the form and calls Invalidate to erase the client area. In such a fun program, that seemed more useful than saving and restoring the strokes.

The Whirligig program installs just one instance of WhirligigPlugin and one instance of DynamicRenderer, but it replaces the entire data array during the Packets and StylusUp calls to display a spiral, as shown in Figure 11.

Figure 11 Whirligig Spirals Around Stylus Tip
Figure 11 Whirligig Spirals Around Stylus Tip

For each movement of the stylus, Packets and StylusUp both call a method named Generate that calculates points constituting an arc at a constant distance from the stylus point. The mathematics was a little tricky, and filling up the array of integers to pass to SetData also required a bit of thought.

The PacketPropertyCount is read-only and cannot be changed. The Count property is also read-only, but it will reflect the new number of integers specified in the SetData call. The number of integers in the array must be PacketPropertyCount times the number of new packets.

WhirligigPlugIn defines the ArrayList object named arrlst to store the new points. The Count property of the array list is the number of Point structures it stores, and hence also the number of packets. The array of integers must have PacketPropertyCount items for each Point in the ArrayList:

int[] arr = new int[data.PacketPropertyCount * arrlst.Count];

The Point coordinates are stored in the first two elements of each packet as shown here:

for (int i = 0, j = 0; i < arr.Length; i += data.PacketPropertyCount) {
        Point pt = (Point) arrlst[j++];
        arr[i] = pt.X;
        arr[i + 1] = pt.Y;
The remainder of the packet must also be set. I simply used the additional integers from the first original packet:
    for (int k = 2;         k < data.PacketPropertyCount; k++)
        arr[i + k] = data[k];
    }
In most cases, PacketPropertyCount will be 3 and that third integer will be the stylus pressure. DynamicRenderer responds to that pressure, so it's important to pass it through.

Finally, calling SetData replaces the original array of points in the data object with the following array:

data.SetData(arr);
This new array of points is the data that the DynamicRenderer will see and render.

The Whirligig class sets the EnableDataCache property of the DynamicRenderer to true to save all the strokes. These are, of course, the modified strokes created by WhirligigPlugin. A call to Refresh causes DynamicRenderer to redraw these strokes, and a good place to call Refresh is in the control's OnPaint method. For that reason, Whirligig overrides the OnPaint method of the form.

The documentation of Refresh reads "When calling the DynamicRenderer object's Refresh method with a Paint event handler, set the DynamicRenderer object's ClipRectangle property to the PaintEventArgs object's ClipRectangle property." But it doesn't really mean that literally. The documentation really means to say that you might temporarily set the ClipRectangle property of the DynamicRenderer object to save some processing time. If you don't restore the ClipRectangle of DynamicRenderer, then subsequent stylus input will be restricted to that rectangle. Figure 12 shows how the OnPaint method of Whirligig sets the ClipRectangle of the DynamicRenderer object to refresh the form's client area but then restores it.

Figure 12 Refreshing the Client Area Using DynamicRenderer (Close Figure 12)
using System;
using System.Drawing;
using System.Windows.Forms;
using Microsoft.StylusInput;

class Whirligig : Form
{
    DynamicRenderer dynaren;

    public static void Main()
    {
        Application.Run(new Whirligig());
    }

    public Whirligig()
    {
        Text = "Whirligig";

        RealTimeStylus rts = new RealTimeStylus(this);
        rts.SyncPluginCollection.Add(new WhirligigPlugin());

        dynaren = new DynamicRenderer(this);
        dynaren.Enabled = true;
        dynaren.EnableDataCache = true;

        rts.SyncPluginCollection.Add(dynaren);
        rts.Enabled = true;
    }

    protected override void OnPaint(PaintEventArgs args)
    {
        Rectangle rectSave = dynaren.ClipRectangle;
        dynaren.ClipRectangle = args.ClipRectangle;
        dynaren.Refresh();
        dynaren.ClipRectangle = rectSave;
    }
}

The Stylus is a Method Call!

The stylus of the Tablet PC looks like a pen, so it's natural to render the strokes of the stylus with lines that resemble ink or pencil or a felt-tip marker. And it's helpful that InkOverlay renders stylus input in a format that is familiar and comforting to users. But the stylus is not a real pen or a pencil or a felt-tip marker. Ultimately, the stylus is a series of method calls, and your program can respond to these method calls in whatever way you see fit. Breaking free of the restraints of InkOverlay into the realm of the Real-Time Stylus is the necessary first step.

Charles Petzold, a long-time contributing editor to MSDN Magazine and its predecessor, MSJ, is the author of four books on .NET Framework programming, and a forthcoming book on the Windows Presentation Foundation (all from Microsoft Press). His Web site is www.charlespetzold.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Can't Commit?: Volatile Resource Managers in .NET Bring Transactions to the Common Type

Can't Commit?

Volatile Resource Managers in .NET Bring Transactions to the Common Type

Juval Lowy
This article discusses:
  • Overview of transactional processing
  • Transactions in the .NET Framework
  • Volatile Resource Managers
  • Transactional classes and collections
This article uses the following technologies:
.NET Framework 2.0, Visual C# 2.0
Code download available at: Transactions.exe (148KB)
Contents
Transaction Problem Space
Transaction Properties: ACID
Transactions in .NET Framework 2.0
Volatile Resource Managers
Transaction-Based Locks
Serialization and Cloning
Transactional<T> Walkthrough
Transactional Collections
Conclusion

T
ransactional programming has traditionally been restricted to database-centric applications. Other types of applications did not see the same benefits from this superior programming model. The Microsoft® .NET Framework 2.0 introduces rudimentary support for volatile resource managers, enabling you to enlist transaction support against memory-based resources, such as class member variables. I will begin this article by briefly discussing the problem space that transactions address and the motivation for using transactions, as well as defining some basic terms such as "ACID." I will then introduce the concept of a resource manager, and explain how and to what extent volatile resource managers are supported in the .NET Framework 2.0.

Next, I'll present the required building blocks for properly utilizing volatile resources, and explain how these building blocks can be used to wrap existing types, such as the collections from System.Collections.Generic. Beyond making extensive use of Visual C# ®2.0, the article walks through some advanced .NET Framework 2.0 programming techniques, including type constraining, type cloning, transaction events, and resource enlistment.

Transaction Problem Space

Proper error handling and recovery are the most vulnerable points of many applications. Once an application fails to perform a particular operation, it should recover and restore the system to a consistent state—usually the state the system was in before the operation that caused the error took place. Typically, any operation that can fail is comprised of multiple smaller steps. Some of those steps can fail after others have succeeded.

The problem with recovery is the sheer number of partial success and partial failure permutations that you have to code against. Trying to handcraft recovery code in a decent-size application is often a futile attempt, resulting in fragile code that is susceptible to any change in the application execution or the business use case, incurring both productivity and performance penalties. In addition, because recovery is all about restoring the system to a consistent state (typically the state before the failed operation), you will need to undo all the steps of the operation that have succeeded. But how do you undo a step such as deleting a row from a table, a node from a linked list, or an item for a collection? Not only that, but what if before the operation failed, some other party accessed your applications and acted upon the system state that you are going to roll back during the recovery? That other party is now acting on inconsistent information and, by definition, is in error. Moreover, your operation may be just a step in some other, much wider operation that spans multiple components from multiple vendors on multiple machines. How would you recover the system as a whole in such a case?

Figure 1 Ensuring Consistency Using Transactions
Figure 1 Ensuring Consistency Using Transactions

The best (and perhaps only) way of maintaining system consistency and dealing properly with the error recovery challenge is to use transactions. A transaction is a set of potentiality complex operations in which the failure of any single operation causes the entire set to fail as one atomic operation. As illustrated in Figure 1, while the transaction is in progress, the system is allowed to be in a temporarily inconsistent state. But once the transaction is complete, the system is guaranteed to be in a consistent state: either a new consistent state (State B in the diagram) or the original consistent state the system was at before the transaction started (State A in the diagram). If the transaction executes successfully, and manages to transfer the system from consistent State A to consistent State B, it is called a committed transaction. If the transaction encounters any error during execution and rolls back all the steps that have already succeeded, it is called an aborted transaction. If the transaction fails to either commit or abort, it is called an in-doubt transaction— these usually require administrator or user assistance to resolve.

Transactional programming requires working with a resource such as a database or a message queue that is capable of participating in a transaction, and able to commit or roll back the changes made during the transaction. Such resources have been around in one form or another for decades. Typically, you inform a resource that you want to perform transactional work against it. This is called enlisting the resource in the transaction. You then perform work against the resource, and if no error occurs, you ask the resource to commit the changes made to its state. If you encounter any error, you ask it to roll back the changes. During a transaction, it is vital that you do not access any non-transactional resources (such as the non-transactional file system), because changes made to those resources will not roll back if the transaction is aborted.

Since systems typically contain in-memory state, objects must also manage their state proactively if it's accessed during a transaction. One way of doing this is to store all the state in a transactional resource, such as a database, and have changes to the state commit or roll back as part of the transaction. Consequently, even though transactions offer a superior, robust, productivity-oriented programming model, they are utilized today only by applications that actually do use resources such as transactional databases. Other types of applications generally provide lip service to recovery functionality by manually crafting solutions for the limited set of error cases that the developers know how to deal with, and as a result they suffer the productivity and quality consequences.

Transaction Properties: ACID

The resource's transactions must abide by four core properties: atomic, consistent, isolated, and durable. These are known collectively as the ACID properties.

Atomic means that when a transaction completes, all the changes it made to the resource state must be made as if they were all one indivisible operation. The changes made to the resource are made as if everything else in the universe stops, the changes are made, and then everything resumes. A transaction should not leave tasks to do in the background once it is finished, as this would violate atomicity. Every operation resulting from the transaction must be included in the transaction itself.

Because transactions are atomic, a client application becomes a lot easier to develop. The client does not have to manage partial failure of its requests or have complex recovery logic. The client knows whether the transaction either succeeds or fails as a whole. In case of failure, the client can choose to issue a new request (start a new transaction), or it can choose to do something else, such as alert the user. The important thing is that the client does not have to recover the system.

Consistent means the transaction must leave the system in a logical state. Note that consistency is different from atomicity. Even if all the changes are committed as one atomic operation, the transaction is required to guarantee that all those changes make sense within the context of the system. Usually, it is up to the developer to ensure the semantics of the operations are consistent. All the transaction is required to do is to transfer the system from one consistent state to another.

Isolated means no other entity (transactional or not) is able to see the intermediate state of the resource during the transaction, because it may be inconsistent (in fact, even if the resource's state is consistent, the transaction could still abort and the changes could be rolled back). Isolation is crucial to overall system consistency. Suppose transaction A allows transaction B access to its intermediate state. Transaction A then aborts, and transaction B decides to commit. The problem is that transaction B based its execution on system state that was rolled back and, therefore, transaction B is left unknowingly inconsistent.

Managing isolation is not trivial. The resources participating in a transaction must lock the data accessed by the transaction from all other parties, and must unlock access to that data when the transaction commits or aborts. In theory, various degrees of transaction isolation are possible. In general, the more isolated the transaction is, the more consistent its results will be.

Transactions in the .NET Framework 2.0 by default use the highest degree of isolation, called serialized. This means the results obtained from a set of concurrent transactions are identical to the results obtained by running each transaction serially. To achieve serialization, all the resources a transaction touches are locked from any other transaction. If other transactions try to access those resources, they are blocked and cannot continue executing until the original transaction commits or aborts.

Durable means the results of a successful transaction are persisted in the system. At any moment, the application could crash, and the memory it was using could be erased. If the changes to the system state were in-memory changes, they would be lost, and the system would be in an inconsistent state. However, even if you store the information in the file system, while it can withstand an application crash, the changes will be lost if the disk crashes. And while you can compensate for disk crashes using redundant drives, it will be of no use in case of a fire in the server room. To handle that, you might utilize multiple mirror sites for your resources, putting them in places with no earthquakes or floods.

As you can see, durability is really a range of options. How resilient to such catastrophes the resource should be is an open question that depends on the nature and sensitivity of the data, your budget, available time and available system administration staff, and so on. If durability is a range that actually means various degrees of persistence, then you could also consider one of the extremes on the spectrum: volatile, in-memory resources. The advantage of volatile resources is that they offer better performance than durable resources, and more importantly, as you will see in this article, they allow you to approximate much better conventional programming models, while using transaction support for error recovery.

Transactions in .NET Framework 2.0

The .NET Framework 2.0 automates the act of enlisting and managing a transaction against transactional resources. The System.Transactions namespace offers a common infrastructure for transaction classes, common behaviors, and helper classes. The .NET Framework defines a resource manager as a resource that can automatically enlist in a transaction managed by System.Transactions. The resource has to detect that it is being accessed by a transaction and enlist in that transaction. The System.Transactions namespace supports a concept called an ambient transaction. This is the transaction your code executes in (and it is stored in thread-local storage). To obtain a reference to the ambient transaction, you call the static Current property of the Transaction class:

Transaction ambientTransaction = Transaction.Current;

If there is no ambient transaction, Current will return null. Figure 2 outlines the most relevant portions of the Transaction class. To perform transactional work against a resource, use the TransactionScope class, defined as follows:

public class TransactionScope : IDisposable
{
   public void Complete();
   public void Dispose();
   public TransactionScope();
   ... // Additional constructors
}
As the name implies, the TransactionScope class is used to scope a code section with a transaction:
using(TransactionScope scope = new TransactionScope())
{
   ... // Perform transactional work here

   // No errors - commit transaction
   scope.Complete();
}
In its constructor, the TransactionScope object creates a transaction object and assigns it as the ambient transaction by setting the static Current property of the Transaction class. TransactionScope is a disposable object—the transaction will end once the Dispose method is called (the end of the using block).

Figure 2 Transaction Class (Close Figure 2)
[Serializable]
public class Transaction : IDisposable, ISerializable
{
   public event TransactionCompletedEventHandler TransactionCompleted;

   public Enlistment EnlistDurable(Guid resourceManagerIdentifier,
      IEnlistmentNotification enlistmentNotification,
      EnlistmentOptions enlistmentOptions);

   public Enlistment EnlistDurable(Guid resourceManagerIdentifier,
      ISinglePhaseNotification singlePhaseNotification,
      EnlistmentOptions enlistmentOptions);

   public Enlistment EnlistVolatile(
       IEnlistmentNotification enlistmentNotification,
       EnlistmentOptions enlistmentOptions);

   public Enlistment EnlistVolatile(
       ISinglePhaseNotification singlePhaseNotification,
       EnlistmentOptions enlistmentOptions);

   public void Rollback();

   public static Transaction Current{ get; set; }

   ... // Additional members
}

The TransactionScope object has no way of knowing whether the transaction should commit or abort. To address this, every TransactionScope object has a consistency bit, which is set to false by default. You can set the consistency bit to true by calling the Complete method (typically, as the last line in the scope, so that any exceptions thrown up to that point will skip the call). If the transaction ends and the consistency bit is set to false, the transaction will abort. For example, the following scope object will roll back its transaction, because the consistency bit is never changed from its default value:

using(TransactionScope scope = new TransactionScope())
{
}
On the other hand, if you do call Complete and the transaction ends with the consistency bit set to true, the transaction will try to commit.

System.Transactions is nice to use because you rarely need to interact with the transaction object directly or explicitly enlist resources, nor must you care if you are part of a local or a distributed transaction. By the virtue of being a System.Transactions transactional resource manager, when it accesses the underlying durable resource, it will use Transaction.Current to obtain a reference to the ambient transaction. It will then call one of the EnlistDurable methods, passing an implementation of IEnlistmentNotification, as shown in Figure 3.

Figure 3 IEnlistmentNotification Interface (Close Figure 3)
public interface IEnlistmentNotification
{
   void Commit(Enlistment enlistment);
   void InDoubt(Enlistment enlistment);
   void Prepare(PreparingEnlistment preparingEnlistment);
   void Rollback(Enlistment enlistment);
}

public class Enlistment
{
   public void Done();
}

public class PreparingEnlistment : Enlistment
{
   public void ForceRollback();
   public void Prepared();
   ... // Additional members
}

The System.Transactions transaction manager uses IEnlistmentNotification to tell the resource manager the outcome of the transaction, asking it to commit or abort. In addition, IEnlistmentNotification is used by System.Transactions to manage the two-phase commit protocol. If all of the participating objects in the transaction vote to commit it, the transaction manager calls the Prepare method on each resource:

void Prepare(PreparingEnlistment preparingEnlistment);

The Prepare essentially asks each resource the following question: if you were asked to commit the changes, would you? This is the first phase of the two-phase commit protocol. If the resource wants to abort the transaction, it calls the ForceRollback method of the supplied PreparingEnlistment object. If the resource is capable of committing the changes, it calls the Prepared method.

In the second phase of the protocol, if all of the resource managers have voted to commit the transaction, the transaction manager calls the Commit method of the IEnlistmentNotification interface. If any one of the resource managers have voted to abort the transaction, then the transaction manager calls the Rollback method. This is how System.Transactions is able to maintain both atomicity and consistency across multiple resources.

Since the two-phase commit protocol is overkill when just a single resource is involved, System.Transactions also offers the ISinglePhaseNotification interface:

public class SinglePhaseEnlistment : Enlistment
{
   public void Aborted();
   public void Committed();
   ... // Additional methods
}

public interface ISinglePhaseNotification : IEnlistmentNotification
{
   void SinglePhaseCommit(SinglePhaseEnlistment singlePhaseEnlistment);
}
The resource manager can choose to implement ISinglePhaseNotification, and enlist using one of the enlisting methods of Transaction that take an ISinglePhaseNotification. In such a case, if that resource is the single resource in the transaction, System.Transactions will only call the SinglePhaseCommit method and avoid conducting the two-phase commit protocol. It is then up to the resource to tell the transaction manager the outcome of the commit attempt, using the SinglePhaseEnlistment parameter.

Volatile Resource Managers

System.Transactions also enables volatile resource managers, which store their state in memory. While their state is not durable, volatile resource managers do often cater to a much wider set of applications than durable resources. If your class member variables (or method local variables) are volatile resources, you can access them from within the transaction, thus significantly simplifying transactional programming.

As far as System.Transactions is concerned, all that a volatile resource needs to do is implement IEnlistmentNotification, detect that it is being accessed by a transaction, and auto-enlist in the transaction by calling the VolatileEnlist method of the transaction object.

There are a number of hurdles in trying to implement IEnlistmentNotification. Ideally, I would like all my data types to be transactional. I like a transactional integer, transactional string, transactional customer object, and so on:

public class TransactionalInteger : IEnlistmentNotification
{...}
public class TransactionalString  : IEnlistmentNotification
{...}
public class TransactionalCustomer : IEnlistmentNotification
{...}
But done this way, I would have to implement IEnlistmentNotification by each type I want to enlist in a transaction. It is therefore much better to use generics and provide a generic implementation of IEnlistmentNotification. To accommodate that, I wrote a Transactional<T> class:
public class Transactional<T> : IEnlistmentNotification
{
   public Transactional(T value);
   public Transactional();
   public T Value { get; set; }
   public static implicit operator T(Transactional<T> transactional);
   ... // More members
}
Transactional<T> is a full-blown volatile resource manager that automatically enlists in an ambient transaction and commits or rolls back any changes that are made to its state according to the transaction's outcome.

You can use the Value property of Transactional<T> to access the underlying type and call any method or operator it supports, as shown in Figure 4. The values of the class member variable m_Number and the local variable city rolled back to their state before the transaction.

Figure 4 Using Transactional<T> (Close Figure 4)
public class MyClass
{
   Transactional<int> m_Number = new Transactional<int>(3);

   public void MyMethod()
   {
      Transactional<string> city = new Transactional<string>("New York");

      using(TransactionScope scope = new TransactionScope())
      {
         city.Value = "London";
         m_Number.Value = 4;
         m_Number.Value++;
         Debug.Assert(m_Number.Value == 5);

         //No call to scope.Complete(), transaction will abort
      }

      Debug.Assert(m_Number.Value == 3);
      Debug.Assert(city.Value == "New York");

      Debug.Assert(m_Number == 3);//Uses the == operator
      int number = m_Number;//Uses the casting operator
   }
}

There are a few challenging issues when implementing Transactional<T> or any other volatile resource manager. The first problem is isolation. The volatile resource must adhere to the isolation property of ACID, lock the underlying resource it manages, and prevent access by multiple transactions. However, .NET only offers thread-based locks, which prevent access only by concurrent threads, not concurrent transactions. The second problem is state management. The resource manager needs to enable the transaction to modify the state of the resource and yet must be able to roll back any such changes if the transaction is aborted.

Transaction-Based Locks

Because the .NET Framework 2.0 does not offer a transaction-based lock, I wrote one myself. I called it TransactionalLock, and gave it a simple interface:

public class TransactionalLock
{
   public void Lock();
   public void Unlock();
   public bool Locked { get; }
}
TransactionalLock provides exclusive locking, thus supporting serializable-level isolation. Only one transaction at a time is allowed to own the lock. You acquire the lock by calling the Lock method, and you release it by calling the Unlock method. If another transaction tries to acquire the lock, it is blocked. Once the lock is acquired, if the same transaction calls Lock multiple times it has no further effect. If there are multiple pending transactions trying to acquire the lock then they are all blocked, they are placed in a queue and allowed ownership of the lock in order. If a transaction owns the lock, TransactionalLock will block even non-transactional calls and place those callers in the queue as well. If the transaction is completed (usually aborted or timed out) while it is waiting to acquire the lock, that transaction is unblocked.

It is important to note that TransactionalLock provides transaction-based synchronized access, not thread-based access. The .NET Framework 2.0 allows multiple threads to execute in the scope of the same transaction, so multiple threads can own the lock as long as they are in the same transaction. In addition, if one thread unlocks TransactionalLock, it unlocks it on behalf of all other threads in the transaction, regardless of how many times Lock was called. This is by design, to facilitate automatic unlocking based on transaction completion. If a multithreaded application interacts directly with a TransactionalLock, then the threads need to coordinate among themselves when it is permissible to call Unlock.

Figure 5 shows the implementation of TransactionalLock with some of the code removed for conciseness. Figure 6 and Figure 7 are UML activity diagrams depicting the Lock and Unlock methods, respectively.

Figure 5 TransactionalLock Class (Close Figure 5)
public class TransactionalLock
{
   LinkedList<KeyValuePair<Transaction,ManualResetEvent>>
      m_PendingTransactions =
        new LinkedList<KeyValuePair<Transaction,ManualResetEvent>>();

   Transaction m_OwningTransaction;

   // Property provides thread-safe access to m_OwningTransaction
   Transaction OwningTransaction { get { ... } set { ... } }

   public bool Locked { get { return OwningTransaction != null; } }

   public void Lock() { Lock(Transaction.Current); }

   void Lock(Transaction transaction)
   {
      Monitor.Enter(this);
      if(OwningTransaction == null)
      {
         //Acquire the transaction lock
         if(transaction != null) OwningTransaction = transaction;
         Monitor.Exit(this);
         return;
      }
      else //Some transaction owns the lock
      {

         //We're done if it's the same one as the method parameter
         if(OwningTransaction == transaction)
         {
            Monitor.Exit(this);
            return;
         }
         //Otherwise, need to acquire the transaction lock
         else
         {
            ManualResetEvent manualEvent = new ManualResetEvent(false);

            KeyValuePair<Transaction,ManualResetEvent> pair =
               new KeyValuePair<Transaction,ManualResetEvent>(
                  transaction,manualEvent);

            m_PendingTransactions.AddLast(pair);

            if(transaction != null)
            {
               transaction.TransactionCompleted += delegate
               {
                  lock(this)
                  {
                     //Pair may have already been removed if unlocked
                     m_PendingTransactions.Remove(pair);
                  }
                  lock(manualEvent)
                  {
                      if(!manualEvent.SafeWaitHandle.IsClosed)
                      {
                          manualEvent.Set();
                      }
                  }
               };
            }
            Monitor.Exit(this);

            //Block the transaction or the calling thread
            manualEvent.WaitOne();
            lock(manualEvent) manualEvent.Close();
         }
      }
   }

   public void Unlock()
   {
      Debug.Assert(Locked);
      lock(this)
      {
         OwningTransaction = null;

         LinkedListNode<KeyValuePair<Transaction,ManualResetEvent>>
            node = null;

         if(m_PendingTransactions.Count > 0)
         {
            node = m_PendingTransactions.First;
            m_PendingTransactions.RemoveFirst();
         }

         if(node != null)
         {
            Transaction transaction = node.Value.Key;
            ManualResetEvent manualEvent = node.Value.Value;
            Lock(transaction);
            lock(manualEvent)
            {
               if(!manualEvent.SafeWaitHandle.IsClosed)
               {
                  manualEvent.Set();
               }
            }
         }
      }
   }
}
Figure 6 Operation of the Lock Method
Figure 6 Operation of the Lock Method

Let's walk through the Lock method first. TransactionalLock has the m_OwningTransaction member variable of the type Transaction, and the OwningTransaction property provides thread-safe access to it. As long as OwningTransaction is null, the lock is considered unlocked. When the Lock method is called, if OwningTransaction is null, TransactionalLock simply sets OwningTransaction to the current transaction.

Figure 7 Unlock
Figure 7 Unlock

If OwningTransaction is not null, TransactionalLock checks whether the incoming transaction (obtained by calling Transaction.Current) is the same as the owning transaction and, if so, the lock does nothing. If, on the other hand, the incoming transaction is different from the owning transaction, TransactionalLock adds that transaction to a private queue it maintains, m_PendingTransactions, which is merely a generic linked list that stores pairs of key and value. The key for each pair is the transaction that tried to acquire the already-owned lock, and the value is a manual reset event. The idea is that the calling transaction will be blocked waiting for that event to be signaled, and the Unlock method will signal it. The problem now is that the blocked transaction can be aborted (perhaps by another thread in the transaction), or it can simply time out and then abort. However, if the calling transaction thread is blocked, how would it know it was aborted?

Fortunately, the Transaction class provides the TransactionCompleted event (see Figure 2) that you can subscribe to. That event is raised on a thread managed by the transaction manager. The Lock method uses an anonymous method to subscribe to that event. If the transaction is still in the queue, the anonymous method removes the transaction from the queue and signals the manual reset event. The Unlock method is even simpler: first it resets the lock by setting OwningTransaction to null. Unlock then removes from the head of the m_PendingTransactions queue the next pair of transaction and matching. It acquires the lock on behalf of the transaction that was at the head of the queue and then unblocks it by signaling the manual reset event.

Serialization and Cloning

The second problem in implementing Transactional<T> is to support rolling back the state of the underlying resource in case a transaction is aborted. In addition, Transactional<T> must also be able to commit the changes as one atomic operation. I've chosen to make a temporary copy of the state of the resource and let the transaction work on that temporary copy. If the transaction aborts, Transactional<T> simply discards the temporary copy. If the transaction commits, Transactional<T> uses the temporary copy as the new state of the resource.

The question then is how do you make a copy of the state of a resource? When dealing with reference types, merely using the assignment operator is not good enough because it only copies the reference. Constraining the type to support ICloneable is not sufficient because there is no telling whether the returned clone is a deep clone or a shallow copy of the reference (it also is not type safe, as IClonable.Clone returns type object). The best solution is to use serialization, because serialization makes a deep copy of the object and all its members. You can serialize an object to a stream and deserialize a deep copy of the object from that stream. This sequence is encapsulated by the static generic Clone method of the ResourceManager static helper class, shown in Figure 8 with some of the code removed for conciseness. Clone takes a parameter called source of the type parameter T, uses the binary formatter to serialize and deserialize it to and from a memory stream, and returns the cloned object.

Figure 8 ResourceManager Helper Class (Close Figure 8)
public static class ResourceManager
{
   public static T Clone<T>(T source)
   {
      IFormatter formatter = new BinaryFormatter();
      using(Stream stream = new MemoryStream())
      {
         formatter.Serialize(stream, source);
         stream.Seek(0, SeekOrigin.Begin);
         return (T)formatter.Deserialize(stream);
      }
   }

   public static void ConstrainType(Type type)
   {
      if(!type.IsSerializable)
      {
         string message = "The type " + type + " is not serializable";
         throw new InvalidOperationException(message);
      }
   }
}

Choosing serialization as the cloning mechanism has an important consequence: Transactional<T> only works with serializable types. Sadly, C# 2.0 does not support constraining a type parameter to be a serializable type. To compensate, Transactional<T> will check the type parameter T in its static constructor using the ConstrainType method of ResourceManager (see Figure 9). The static constructor will be called before anything else is done with Transactional<T>, and it is a common technique for enforcing constraints that have no support at compile time. ConstrainType verifies that a given type is serializable by accessing the IsSerializable property of the type.

Figure 9 Implementing Transactional<T> (Close Figure 9)
public class Transactional<T> : IEnlistmentNotification
{
   T m_Value;
   T m_TemporaryValue;
   Transaction m_CurrentTransaction;
   TransactionalLock m_Lock;

   public Transactional(T value)
   {
      m_Lock = new TransactionalLock();
      m_Value = value;
   }

   public Transactional() : this(default(T)) {}

   static Transactional()
   {
      ResourceManager.ConstrainType(typeof(T));
   }

   void IEnlistmentNotification.Commit(Enlistment enlistment)
   {
      IDisposable disposable = m_Value as IDisposable;
      if(disposable != null) disposable.Dispose();

      m_Value = m_TemporaryValue;
      m_CurrentTransaction = null;
      m_TemporaryValue= default(T);
      m_Lock.Unlock();
      enlistment.Done();
   }

   void IEnlistmentNotification.InDoubt(Enlistment enlistment)
   {
      // Bad for a volatile resource, but not
      // much that can be done about it
      m_Lock.Unlock();
      enlistment.Done();
   }

   void IEnlistmentNotification.Prepare(
       PreparingEnlistment preparingEnlistment)
   {
      preparingEnlistment.Prepared();
   }

   void IEnlistmentNotification.Rollback(Enlistment enlistment)
   {
      m_CurrentTransaction = null;

      IDisposable disposable = m_TemporaryValue as IDisposable;
      if(disposable != null) disposable.Dispose();

      m_TemporaryValue = default(T);
      m_Lock.Unlock();
      enlistment.Done();
   }

   void Enlist(T t)
   {
      Debug.Assert(m_CurrentTransaction == null);
      m_CurrentTransaction = Transaction.Current;
      m_CurrentTransaction.EnlistVolatile(this,EnlistmentOptions.None);
      m_TemporaryValue = ResourceManager.Clone(t);
   }

   void SetValue(T t)
   {
      m_Lock.Lock();
      if(m_CurrentTransaction == null)
      {
         if(Transaction.Current == null)
         {
            m_Value = t;
            return;
         }
         else
         {
            Enlist(t);
            return;
         }
      }
      else m_TemporaryValue = t;
   }

   T GetValue()
   {
      m_Lock.Lock();
      if(m_CurrentTransaction == null)
      {
         if(Transaction.Current == null) return m_Value;
         else Enlist(m_Value);
      }
      return m_TemporaryValue;
   }

   public T Value
   {
      get { return GetValue(); }
      set { SetValue(value); }
   }
}

Transactional<T> Walkthrough

With the key elements of transactional locking and resource state cloning in place, implementing Transactional<T> is possible. Figure 9 shows the implementation with some of the code removed for clarity.

Transactional<T> has two member variables of type T. m_Value represents the actual consistent state, and m_TemporaryValue is the temporary copy given to a transaction to work on. Transactional<T> also has the m_Lock member of the type TransactionalLock, used to isolate m_Value and allow only serialized access to it. Finally, Transactional<T> maintains a reference to the current transaction in the m_CurrentTransaction member variable. If m_CurrentTransaction is not null, then Transactional<T> is considered to be enlisted in a transaction.

The Value property delegates to the GetValue and SetValue methods. Figure 10 illustrates the operation of the SetValue method. The first thing SetValue does is lock the transactional lock m_Lock. If the volatile resource is owned by another transaction, then this will block SetValue until its turn comes to access the resource. Once the lock is acquired, SetValue checks whether it is already enlisted in a transaction. If m_CurrentTransaction is null and the call has no transaction, SetValue sets the actual value m_Value, since no transaction support is required. If, on the other hand, Transactional<T> is not enlisted, but the call has a transaction, SetValue enlists in the transaction by calling the Enlist helper method. GetValue functions in a similar manner, except it reads rather than writes to the resource.

Figure 10 Operation of SetValue
Figure 10 Operation of SetValue

The Enlist method sets m_CurrentTransaction to the incoming current transaction and calls the EnlistVolatile method of the current transaction, thus hooking itself up to the transaction manager of System.Transactions. Finally, Enlist uses ResourceManager.Clone to make a deep copy of the resource and then assigns that to m_TemporaryValue. Note that EnlistVolatile accepts an implementation of IEnlistmentNotification, which Transactional<T> implements explicitly. IEnlistmentNotification is used by the transaction manager to ask Transactional<T> to vote during the two-phase commit protocol and to notify Transactional<T> about the outcome of the transactions (instructing it to commit or abort). Since Transactional<T> will always commit if asked to do so, Prepare simply calls the Prepared method of the preparing element.

Implementing IEnlistmentNotification.Commit and IEnlistmentNotification.Rollback is now straightforward. Commit disposes of m_Value (if it offers an IDisposable), copies the reference to the temporary value in m_TemporaryValue to the actual value m_Value, thus committing the transaction, and unlocks the lock, allowing another transaction to set or get the value of the volatile resource. IEnlistmentNotification.Rollback simply disposes of m_TemporaryValue and then sets m_Value back to its default value, thus discarding the changes, and unlocks the lock.

Transactional Collections

The generic type parameter T for Transactional<T> can be any serializable type. However, you are most likely to use it with two main categories of data structures or resources. The first category includes primitive types like integers and strings or custom objects like a customer or an order. Because there is an unlimited number of such types, what Transactional<T> offers is good enough. Whatever the type may be, as long as it is serializable, you can treat it as a volatile resource manager and access it through Value.

The second category is a collection of individual items such as arrays, linked lists, queues, and so on. You can specify such a collection as the type parameter and access it through the Value property, as shown in the following:

Transactional<int[]> numbers = new Transactional<int[]>(new int[3]);
numbers.Value[0] = 1;
numbers.Value[1] = 2;
numbers.Value[2] = 3;

using(TransactionScope scope = new TransactionScope())
{
   numbers.Value[0] = 11;
   numbers.Value[1] = 22;
   numbers.Value[2] = 33;
   scope.Complete();
}
Debug.Assert(numbers.Value[2] == 33);

However, such use of the type parameter leads to somewhat cumbersome programming. I would like to use a transactional array as if it were a normal array and a transactional linked list as if it were a normal linked list, without the need to always dereference it through Value. Since there are only a handful of such useful collections, I defined all the collections in System.Collections.Generic as transactional collections. As an example, TransactionalQueue is shown in Figure 11.

Figure 11 Transactional Collections (Close Figure 11)
public abstract class TransactionalCollection<C,T> :
   Transactional<C>, IEnumerable<T> where C : IEnumerable<T>
{
   public TransactionalCollection(C collection)
   {
      Value = collection;
   }

   IEnumerator<T> IEnumerable<T>.GetEnumerator()
   {
      return Value.GetEnumerator();
   }

   IEnumerator IEnumerable.GetEnumerator()
   {
      return ((IEnumerable<T>)this).GetEnumerator();
   }
}

public class TransactionalQueue<T> : TransactionalCollection<Queue<T>,T>,
                                     ICollection
{
   public TransactionalQueue(int capacity) :
      base(new Queue<T>(capacity)) {}

   public void Enqueue(T item) { Value.Enqueue(item); }

   public T Dequeue() { return Value.Dequeue(); }
}
...//additional collections

These collections are completely polymorphic with the collections available in System.Collections.Generic. They offer the same methods, and they implement implicitly or explicitly the same interfaces as their respective non-transactional cousins. As a result, they are used in the same way. Here's an example:

TransactionalArray<int> numbers = new TransactionalArray<int>(3);
numbers[0] = 1;
numbers[1] = 2;
numbers[2] = 3;

using(TransactionScope scope = new TransactionScope())
{
   numbers[0] = 11;
   numbers[1] = 22;
   numbers[2] = 33;
}
Debug.Assert(numbers[2] == 3);

Figure 12 shows an example of using TransactionalQueue as a message queue. Any message added to the queue is rejected if the enqueuing transaction is aborted. Any message removed from the queue is added back if the dequeuing transaction is aborted. This enables programming models similar to System.Messaging and Windows Communication Foundation MSMQ binding, where you have automatic canceling (enqueuing is rolled back), guaranteed delivery, and an auto-retry mechanism (processing the message fails, so the dequeuing rolls back).

Figure 12 TransactionalQueue as a Message Queue (Close Figure 12)
[Serializable]
class MyMessage { ... }

TransactionalQueue<MyMessage> messageQueue =
   new TransactionalQueue<MyMessage>();
messageQueue.Enqueue(new MyMessage());
using(TransactionScope scope = new TransactionScope())
{
   messageQueue.Enqueue(new MyMessage());
   messageQueue.Enqueue(new MyMessage());
   messageQueue.Enqueue(new MyMessage());
   Debug.Assert(messageQueue.Count == 4);
}
Debug.Assert(messageQueue.Count == 1);

To automate parts of the implementation (all collections need to support IEnumerable<T>), the transactional collections derive from the abstract class TransactionalCol-lection<C,T>, which, in turn, derives from Transactional<C>, passing C as the type parameter to it. C is constrained to support IEnumerable<T>, and Value is assigned a value of the type C in the constructor. As a result, TransactionalCollection<C,T> can delegate to Value the implementation of IEnumerable<T>. In each of the concrete transactional collections, implementing the various methods and properties is done by delegating the implementation to the methods and properties now exposed by Value. For each concrete transactional collection, Value exposes the methods it requires because each concrete transactional collection specifies the collection it wraps as the type parameter C when deriving from TransactionalCollection<C,T>:

public class TransactionalList<T> : TransactionalCollection<List<T>,T>,
                                    IList<T>,ICollection<T>
{...}
You can also use TransactionalCollection<C,T> as the base class for your own custom transactional collections.

Conclusion

I believe that transactional programming, from durable to volatile resources will be the predominant programming model of the future. Volatile resource managers allow you to unleash the fundamental benefits of transactions to everyday objects and common data structures—from integers to dictionaries. Adding this transactional functionality requires minimal changes to the conventional programming model. By eliminating the need to handcraft error handling and recovery, you gain productivity, quality, and faster time to market. This also lowers the cost of long-term maintenance and conforming to new requirements and scenarios.

Juval Lowy is a software architect providing .NET architecture consultation and advanced training. He is the Microsoft Regional Director for the Silicon Valley. His latest book is Programming .NET Components, 2nd Edition (O'Reilly, 2005). Contact Juval at www.idesign.net.


UI on the Fly: Use the .NET Framework to Generate and Execute Custom Controls at Run Time

UI on the Fly

Use the .NET Framework to Generate and Execute Custom Controls at Run Time

Morgan Skinner
This article discusses:
  • Run-time code generation
  • Generating classes using Reflection.Emit
  • Generating classes using System.CodeDom
  • Creating Windows Forms and ASP.NET controls at run time
This article uses the following technologies:
Windows Forms, ASP.NET, .NET Framework, C#
Code download available at: CodeGeneration.exe (146KB)
Contents
Code Generation Basics
Generating Classes Using Reflection.Emit
Generating Classes Using System.CodeDom
Storing Generated Assemblies
Securing Generated Assemblies
Windows Forms Example
Using Reflection.Emit to Generate a Windows Forms Control
Emitting the Assembly
Defining the Type
Defining the Constructor
Defining the InitializeComponent Method
Using System.CodeDom to Generate a Windows.Forms Control
Defining and Compiling the Assembly
Defining the Type with CodeDOM
Defining the Constructor
Defining InitializeComponent
An ASP.NET Example
Data Binding
Simplifying the Code
Conclusion

O
ne of the often overlooked features of the Microsoft® .NET Framework is its ability to generate, compile, and execute custom code at run time. This is done, for example, during serialization of XML data, and in the use of regular expressions, where the expression evaluation function is emitted at run time.

This article describes another area in which run-time code generation can be used: the creation of UI controls. Generating these controls once and then reusing them as necessary is much more efficient than generating the controls each time a form or page is requested.

This is applicable to any application that has user-configurable fields (for instance, where the end user can select data items to be displayed on screen). It is common to define custom forms using XML. This is then parsed at run time to construct the UI dynamically when the page is loaded. However, this parsing frequently happens each time the form is displayed, or in a server scenario, for each user, thereby creating unnecessary load on the application.

In this article, I will provide detailed examples of how to use run-time code generation to construct, load, and execute controls at run time. The examples I will describe can be applied equally well to the .NET Framework 1.x and to the .NET Framework 2.0. There are some significant additions to the Reflection namespaces in the 2.0 release of the Framework, but these changes do not negate or impair the solutions outlined here in any way.

Code Generation Basics

Many applications provide a user interface (UI) that can be customized—possibly by adding extra fields or altering the order and position of existing fields. This is frequently accomplished in one of two ways:

  • The user makes changes by editing the user interface inside Visual Studio®.
  • The app generates controls at run time based on some form of configuration data, commonly stored in an XML file.

Neither of these solutions is ideal. What's really needed is a combination of these two methods that offers the performance of the hand-crafted solution and the flexibility of the application generating the controls at run time.

There are two supported methods for constructing controls at run time: Reflection.Emit and System.CodeDom. The former, which requires intimate knowledge of Intermediate Language (IL), generates managed assemblies using explicitly specified IL instructions. The latter uses an object model of the code to emit source code that can then be compiled into IL and executed. Both of these methods are covered in this article.

There is actually another method, a variant of the CodeDOM approach, which is to construct source code manually, meaning to write the .cs or .vb file yourself, rather than using the object model to do so, and then compile it at run time. While this is possible, I recommend using one of the previously mentioned mechanisms rather than a hand-crafted solution.

Generating Classes Using Reflection.Emit

The Reflection.Emit namespace permits you to generate assemblies that are entirely transient; they can be generated in memory and executed without ever being persisted to disk. The option to write to disk is available, however, if necessary. (I discuss this a bit later in the article.)

To generate a class using Reflection.Emit, the following steps are required:

  • Define a dynamic assembly.
  • Create a module within that assembly.
  • Define a type in that module, deriving the type from the appropriate base class and interface(s):Create the methods on that type, obtain an ILGenerator for each method, emit IL opcodes as appropriate, and persist the type.
  • Optionally save the assembly for future use.

Constructing classes using Reflection.Emit is a complex approach, and for any given source code operation (such as calling a method) you will typically have to generate several lines of IL code. I'll get back to that later in the article.

Generating Classes Using System.CodeDom

The System.CodeDom namespace provides a language-agnostic way to define types. You construct an in-memory model of the code, and then use a code generator to emit the source code for that model. As long as you have an appropriate code generator, you can emit code in any language (the .NET Framework redistributable includes code generators for Visual Basic®, C#, and JScript®; if Visual Studio is installed, generators for C++ and J# will also be available).

To generate a class using the CodeDOM:

  • Create a new CodeCompileUnit.
  • Add namespaces to the CodeCompileUnit.
  • Add any import statements as appropriate.
  • Add a CodeTypeDeclaration for the class being constructed.
  • Add CodeExpressions to each method.
  • Obtain a code compiler and compile the CodeCompileUnit.
One of the benefits of this method is that you use higher-level concepts than when working with Reflection.Emit.

Once your classes have been defined, you need to construct instances of them at run time. The most common method is to use the Activator class to load the type, as shown in the following code snippet:

public Control LoadControl ( string typeName )
{
    return LoadControl(Type.GetType(typeName));
}

public Control LoadControl ( Type controlType )
{
    return Activator.CreateInstance(controlType) as Control;
}

Here I call the static GetType member of the Type class. GetType returns the .NET type information for the type named in its argument. The Activator class is then used to construct an instance of that type. Note that if the specified type lives in an assembly other than mscorlib or the currently executing assembly, the type name must be qualified with the assembly's name. When stored as a string, the names of managed types can be either partially qualified with the namespace of the type, or fully qualified with both the namespace of the type and with the name of the assembly in which the type is stored. Here's an example of an assembly-qualified type name:

TestControls.TestControl, TestAssembly

This type name corresponds to a type as shown in the following code, compiled into an assembly called TestAssembly:

using System.Web.UI;

namespace TestControls
{
  public class TestControl : Control
  {
    ...
  }
}

When generating controls, it would be customary to store this type name in a persistent storage medium, such as a SQL Server database. When your form or page is rendered, you would load the type using Activator.CreateInstance and then display that object to the user.

Storing Generated Assemblies

Assemblies created using Reflection.Emit or System.CodeDom can be entirely transient or can be generated on disk for future use. Transient assemblies exist purely for the lifespan of the application domain in which they're created. Once this application domain is unloaded (which for the default application domain occurs when the application is unloaded), these assemblies are unloaded from memory and no longer exist. Since there is a cost to generating these assemblies at run time, I recommend that you consider persisting generated assemblies to the file system and looking for them there first. If an assembly can't be found on the file system at run time, it can then be regenerated. Of course, if you choose this approach, you'll need to ensure that the definition for the controls and the generated assemblies remain in sync. For a commercial application, I suggest that you provide your users with an administrative tool that can run through all the controls that need to be generated and store these in a single assembly that is held on disk or within SQL Server™.

I also recommend limiting the number of dynamically generated assemblies to the smallest practical number—preferably just one. This does mean that the entire set of controls will need to be generated each time a change is made in any control. However, as this is an administrative activity that is unlikely to occur too frequently, this should not pose a performance problem.

Securing Generated Assemblies

One other important area to understand is the security issues associated with generated assemblies. This is of paramount importance, as generated assemblies pose a unique threat to an application—namely, the application in question expects to dynamically load some types, and these types, in theory, could execute any arbitrary code.

The preferred way to secure generated code is to assign a limited set of permissions to all generated code. This can be accomplished in .NET by specifying a code group within the .NET configuration tool, mscorcfg.msc.

A custom code group can be defined with a specific membership condition and set of permissions. As an example, you might want to assign all dynamically generated code a very limited set of permissions, such as Execute (which permits the assembly to execute, but severely restricts what that code can do). A code group is applied to an assembly by means of a membe rship condition—in effect this is used to define inclusion into the group.

There are several types of membership conditions. Probably the most useful in this example would be the URL permission, which can be used to define membership based on a particular directory on disk (using, for example, "file://E:/Code Generation/bin/Debug/*" as the URL). The permission set defined within the tool will apply to any generated assemblies that are stored within the specified folder.

Generating controls once and loading them at run time has performance benefits over, for example, using an XML representation of a form and evaluating it at run time. Loading an XML representation typically involves instantiating and loading an XmlDocument, a relatively slow operation compared to instantiating a typical control, even with Activator.CreateInstance.

Windows Forms Example

This section provides an example that shows how a Windows® Forms control can be generated at run time using both Reflection.Emit and System.CodeDom. The control that is generated will present a simple user interface comprised of just a static textbox, as shown in Figure 1.

Figure 1 Control Generation Sample App
Figure 1 Control Generation Sample App

The generated control is the portion that appears within the red outline. Although this is a simple example, it does provide all of the necessary concepts that can be extended to deliver a more complete implementation.

What we're after is a control whose source code mimics that shown in Figure 2. This code demonstrates several concepts that can be used no matter what amount of code you are emitting:

  • The class is derived from a base class.
  • The class contains a private field.
  • The class contains a public default constructor.
  • The class contains a private method.
  • The class shows how to use properties.
  • The class shows how to call methods.
  • The class shows how to construct new object instances.
This is by no means an exhaustive list of all the capabilities that could be emitted, but it's a good starting point.

Figure 2 Prototype UserControl (Close Figure 2)
using System;
using System.Drawing;
using System.Windows.Forms;

public class TestControl : UserControl
{
    private Label _label ;

    public TestControl ( ) { this.InitializeComponent ( ) ; }

    private void InitializeComponent ( )
    {
        _label = new Label();
        this.SuspendLayout();
        _label.Anchor = (AnchorStyles.Top | (AnchorStyles.Left |
                         AnchorStyles.Right));
        _label.BorderStyle = BorderStyle.Fixed3D;
        _label.Location = new Point(8, 8);
        _label.Name = "_label";
        _label.Size = new Size(312, 23);
        _label.TabIndex = 0;
        _label.Text = "***User defined message goes here***";
        this.Controls.Add(_label);
        this.Name = "MyControl";
        this.Size = new Size(328, 100);
        this.ResumeLayout(false);
    }
}

Using Reflection.Emit to Generate a Windows Forms Control

The Reflection.Emit namespace contains a small number of classes, as it's a very low-level API and most of the hard work has to be performed by your own code. I'll present the code in sections and describe each in detail.

Working with Reflection.Emit is hard—the only real way to ensure that the correct code is generated is to write some test code, compile it, and then examine the generated IL (using tools like ildasm or Lutz Roeder's .NET Reflector). This is the method I used in order to generate the following IL. That's not to say that you could not generate your own IL without reverse engineering some other code, but I doubt there are many developers who consider IL to be their primary language, so skills in this area are few and far between (if you are looking to become an expert in IL, a great reference is Serge Lidin's book Inside Microsoft .NET IL Assembler, Microsoft Press®, 2002). For this discussion, I've split the code into four logical sections: emitting the Assembly, defining the type, defining the constructor, and defining the InitializeComponent method.

Emitting the Assembly

The code in Figure 3 constructs a dynamic assembly, adds a module to that assembly, and creates a unique name for the assembly. First, an assembly name is constructed. In this example, I'm using a GUID to ensure a unique enough name for the assembly.

Figure 3 Dynamic Assembly Builder (Close Figure 3)
AssemblyName assemName = new AssemblyName ( ) ;
assemName.Name = string.Format(
    "MyControl{0}", Guid.NewGuid().ToString("N") ) ;

// Create the assembly builder object
AssemblyBuilder assemblyBuilder =
    AppDomain.CurrentDomain.DefineDynamicAssembly(
        assemName, AssemblyBuilderAccess.RunAndSave ) ;

// Then create the module builder
ModuleBuilder moduleBuilder = assemblyBuilder.DefineDynamicModule(
    "MyModule", "MyAssembly.dll" ) ;

...

assemblyBuilder.Save ( "MyAssembly.dll" ) ;

It's then necessary to call the static DefineDynamicAssembly method of the AppDomain class. There are a number of overloaded versions of the DefineDynamicAssembly function, but here I opted to call the simplest override. Next, a module builder object is created by defining the module name and file name. After the code in the following sections, the assembly is finally created using the AssemblyBuilder.Save method.

Defining the Type

To create a type using Reflection.Emit, you need to call one of the DefineType methods on the ModuleBuilder class. There are various methods that permit you to define the class, its base class, and any interfaces it explicitly implements. The following segment of code shows how to define a class derived from System.Windows.Forms.UserControl:

Type baseClass = typeof ( System.Windows.Forms.UserControl ) ;
TypeBuilder typeBuilder = moduleBuilder.DefineType(
    "MyControls.MyControl",
    TypeAttributes.Public | TypeAttributes.Class |
    TypeAttributes.BeforeFieldInit, baseClass ) ;

DefineType allows you to specify type attributes for the generated type. In this case, the new type is a class; it has public visibility, and static members can be called without forcing the runtime to initialize the class.

I then construct a private field which will hold the label control. This follows a similar pattern as the code just shown, in that the name, type, and attributes are specified in a single call. The following code snippet shows how it looks:

// Create a private field for the label
FieldBuilder labelField = typeBuilder.DefineField(
    "_label", typeof( System.Windows.Forms.Label ) ,
    FieldAttributes.Private ) ;

Defining the Constructor

In order to add anything to a type, you need to call one of the Define* methods of the TypeBuilder class (this object was created earlier). There are a number of these methods for defining constructors, events, fields, methods, and other constructs.

In Figure 4 I'm using the DefineConstructor method to create a public constructor with no parameters. I supply appropriate method attributes to the DefineConstructor method, and define the parameter array to be empty in this instance. The IL generated from the constructor code is shown here:

.method public hidebysig specialname rtspecialname instance void .ctor()
cil managed
{
   .maxstack 3
   L_0000: ldarg.0
   L_0001: call instance void System.Windows.Forms.UserControl::.ctor()
   L_0006: ldarg.0
   L_0007: call instance void
           MyControls.MyControl::InitializeComponent()
   L_000c: ret
}
Note that there is a nearly 1:1 correspondence between the generated IL and the commands that emitted that IL in the C# code.

Figure 4 ConstructorBuilder (Close Figure 4)
// Then create the constructor which will call InitializeComponent
ConstructorBuilder constructor = typeBuilder.DefineConstructor(
    MethodAttributes.Public | MethodAttributes.HideBySig ,
    CallingConventions.Standard , new System.Type[] { } ) ;

// Get an IL generator so I can write some code
ILGenerator constructorIL = constructor.GetILGenerator ( ) ;

// Call the base class constructor and then InitializeComponent
constructorIL.BeginScope ( ) ;
constructorIL.Emit ( OpCodes.Ldarg_0 ) ;
constructorIL.Emit ( OpCodes.Call, baseClass.GetConstructor (
    new Type[] {} ) ) ;
constructorIL.Emit ( OpCodes.Ldarg_0 ) ;
constructorIL.Emit ( OpCodes.Call, initComponentBuilder ) ;
constructorIL.Emit ( OpCodes.Ret ) ;
constructorIL.EndScope ( ) ;

Once you have a ConstructorBuilder object, you can retrieve an ILGenerator from it. This is the object that permits you to emit IL into the assembly.

The BeginScope and EndScope methods are not strictly necessary here, and basically equate to { and } in C#. Within this scope I emit IL opcodes.

The CLR is a stack-based architecture, so the first opcode (Ldarg_0) is pushing an argument (in this case, the this pointer for the current object) onto the evaluation stack. The second instruction is a call to the base class constructor. This pops the this pointer off the stack and calls the constructor, so the next opcode loads the this pointer onto the stack ready for the next call.

The penultimate opcode is a call to the InitializeComponent method, and the final opcode is a return from the constructor.

Defining the InitializeComponent Method

This method is much more involved than the others, as it has to do the following:

  • Create the new label control.
  • Call SuspendLayout.
  • Set all properties of the label control.
  • Add the label control to the user control's Control collection.
  • Set the initial size of the user control.
  • Call ResumeLayout.
These tasks are outlined in Figure 5.The amount of code required to generate the control is fairly significant, and easy to get wrong. You can, however, break this into separate functions for readability.

Figure 5 Generating InitializeComponent (Close Figure 5)
// Now generate the InitializeComponent method
MethodBuilder initComponentBuilder = typeBuilder.DefineMethod(
    "InitializeComponent" , MethodAttributes.Private |
    MethodAttributes.HideBySig, CallingConventions.Standard,
    typeof ( void ) , new System.Type[] { } ) ;

// Get an IL generator for the InitializeComponent method
// and write the method body.
ILGenerator initIL = initComponentBuilder.GetILGenerator ( ) ;
initIL.BeginScope ( ) ;

// Construct the label
initIL.Emit ( OpCodes.Ldarg_0 ) ;
initIL.Emit ( OpCodes.Newobj , typeof(
    System.Windows.Forms.Label).GetConstructor( new Type[] { } ) ) ;
initIL.Emit ( OpCodes.Stfld , labelField ) ;

// Then call Control.SuspendLayout
initIL.Emit ( OpCodes.Ldarg_0 ) ;
initIL.Emit ( OpCodes.Call , baseClass.GetMethod ( "SuspendLayout" ) ) ;

// Anchor the control to the top left and right
initIL.Emit ( OpCodes.Ldarg_0 ) ;
initIL.Emit ( OpCodes.Ldfld, labelField ) ;
initIL.Emit ( OpCodes.Ldc_I4_S,
    (int) (AnchorStyles.Left | AnchorStyles.Top | AnchorStyles.Right ) ) ;
initIL.Emit ( OpCodes.Callvirt ,
   labelClass.GetProperty(
      "Anchor",typeof(AnchorStyles)).GetSetMethod ( ) ) ;

... // Set the other properties of the label control

// Call Control.ResumeLayout
initIL.Emit ( OpCodes.Ldarg_0 ) ;
initIL.Emit ( OpCodes.Ldc_I4_0 ) ;
initIL.Emit ( OpCodes.Call , baseClass.GetMethod ( "ResumeLayout" ,
    new Type[] { typeof ( bool ) } ) ) ;

// And emit the end of the function
initIL.Emit ( OpCodes.Ret ) ;
initIL.EndScope ( ) ;

Most of the sections of code in Figure 5 do something similar—they load the this pointer onto the stack with the Ldarg_0 opcode, and then load parameters onto the stack. The parameters are then used in a call to a property setter or, in some cases, to a method call. As an example, the code that sets the size of the control is outlined here:

initIL.Emit ( OpCodes.Ldarg_0 ) ;
initIL.Emit ( OpCodes.Ldc_I4, 328 ) ;
initIL.Emit ( OpCodes.Ldc_I4, 200 ) ;
initIL.Emit ( OpCodes.Newobj ,
    typeof ( Size ).GetConstructor(
        new Type[] { typeof ( int ) , typeof ( int ) } ) ) ;
initIL.Emit ( OpCodes.Callvirt , controlClass.GetProperty("Size",
    typeof(Size)).GetSetMethod ( ) ) ;

I load the this pointer onto the stack followed by two integers, 328 and 200. I then make a call to the Size constructor, which constructs a size object using these two integers. This size object is returned at the top of the stack. I then obtain the control's Size property and retrieve the setter for this property, which I call with the arguments currently on the stack. The net effect is that I generate the following call:

this.Size = new Size ( 328, 200 ) ;

Due to the inherent complexity of this solution, I would not suggest that it be used widely, if at all. The following solution provides much of the flexibility of IL generation without as steep a learning curve for the developers writing the code. There are some esoteric things that can be done in IL that you can't achieve in other CLR-targetting languages, but these are unlikely to be necessary in many applications.

Using System.CodeDom to Generate a Windows.Forms Control

The CodeDOM namespace provides an abstracted code model that you can program against. This model is built up in memory using fairly simple expressions and can then be converted into source code using one of the available .NET code generators. Microsoft currently provides code generators for C#, Visual Basic, C++, JScript, J#, and MSIL. Generators are also available from third parties for other languages.

I've broken the code down into the same sections as used in the Reflection.Emit example so direct comparisons of each step can be made. Note that the two methods of code generation are not interchangeable—you cannot mix CodeDOM with emitted code in the same assembly.

Defining and Compiling the Assembly

The first step for working with the CodeDOM is to reference the System.CodeDom namespace. Then you need to create a CodeCompileUnit, which will form the basis of the assembly. Here, you can add namespaces, add import statements to these namespaces, and define types.

The following code shows how to create a particular assembly. I've created a CodeCompileUnit, added the MyControls namespace to it, and then defined import statements for all the classes used in the code. The namespace is then added to the code compile unit:

// Create a code compile unit and a namespace
CodeCompileUnit ccu = new CodeCompileUnit ( ) ;
CodeNamespace ns = new CodeNamespace ( "MyControls" ) ;

// Add some imports statements to the namespace
ns.Imports.Add ( new CodeNamespaceImport ( "System" ) ) ;
ns.Imports.Add ( new CodeNamespaceImport ( "System.Drawing" ) ) ;
ns.Imports.Add ( new CodeNamespaceImport ( "System.Windows.Forms" ) ) ;

// Add the namespace to the code compile unit
ccu.Namespaces.Add ( ns ) ;

Once you have a completed type definition, you can use the following code to use the C# code provider to compile the definition into an assembly. I've included the compilation step here just to complete this section. Logically, however, this would occur after creating the types in the assembly, which is described later in the article.

CodeDomProvider provider = new Microsoft.CSharp.CSharpCodeProvider ( ) ;
ICodeCompiler compiler = provider.CreateGenerator ( ) as ICodeCompiler ;
CompilerParameters cp = new CompilerParameters ( new string[] {
    "System.dll", "System.Windows.Forms.dll", "System.Drawing.dll" } ) ;
cp.GenerateInMemory = true;
cp.OutputAssembly = "AutoGenerated";
CompilerResults results = compiler.CompileAssemblyFromDom ( cp, ccu ) ;
Here, a CodeDomProvider is created for C#, and the call to CreateGenerator returns an ICodeCompiler interface. I then define the libraries that should be referenced from this assembly. After setting some parameters on this CompilerParameters class, I call the CompileAssemblyFromDom method. Based on the settings I specify in this code, an assembly is generated in memory with the name AutoGenerated.

The CompilerResults class contains an assembly property that is set on successful compilation of the assembly. You can then load types from this assembly or do whatever is appropriate.

Defining the Type with CodeDOM

Type creation using the CodeDOM is very easy. The following code shows how to create a class that derives from UserControl:

CodeTypeDeclaration ctd = new CodeTypeDeclaration ( "MyControl" ) ;
ctd.BaseTypes.Add ( typeof ( System.Windows.Forms.UserControl ) ) ;
ns.Types.Add ( ctd ) ;
The CodeTypeDeclaration contains a collection called BaseTypes, which contains the class that the type will derive from. It can also contain interfaces that the type should implement. Note that even though this is a collection of arbitrary types, you cannot use two or more classes here as the CLR only supports single inheritance. If you try to use more than one base class, an error will be thrown.

Defining the Constructor

When you have a CodeTypeDeclaration, all other code is added to this declaration. The following snippet uses the ctd variable created when defining the constructor:

CodeConstructor constructor = new CodeConstructor ( ) ;
constructor.Statements.Add ( new CodeMethodInvokeExpression (
    new CodeThisReferenceExpression ( ) ,
        "InitializeComponent", emptyParams ) ) ;
constructor.Attributes = MemberAttributes.Public ;
ctd.Members.Add ( constructor ) ;

Constructors (and other types derived from CodeMemberMethod) include a CodeStatementCollection property called Statements. To this, you add one or more statements in the form of CodeStatement objects (if you add a CodeExpression, it will be automatically wrapped in a CodeExpressionStatement object). In this example, I have added a CodeMethodInvokeExpression that calls this.InitializeComponent and passes no parameters. I then specify that the constructor is publicly accessible and finally add the constructor to the type definition.

Defining InitializeComponent

The InitializeComponent method, which is shown in Figure 6, is obviously more complex than the constructor, as there are numerous statements that need to be created. Though somewhat lengthy, this code is simpler to write than the IL equivalent. Let's dissect some of this code so you can see how it works. First look at the SuspendLayout call:

this.SuspendLayout ( ) ;
This expression is generated by using CodeMethodInvokeExpression, and passing a reference to the object that the method is called on. This is followed by the method name and any parameters that need to be passed to that method. CodeMethodInvokeExpression is called as follows:
CodeMethodInvokeExpression ( targetObject, methodName, parameters )
In this instance, the targetObject is this, the methodName is SuspendLayout, and there are no parameters, so it is defined in code as follows (emptyParams is an empty array of type CodeExpression):
initializeComponent.Statements.Add(
  new CodeMethodInvokeExpression(
  new CodeThisReferenceExpression(), "SuspendLayout",
  emptyParams));

Figure 6 InitializeComponent with CodeDom (Close Figure 6)
// Now add the InitializeComponent method
CodeMemberMethod initializeComponent = new CodeMemberMethod ( ) ;
initializeComponent.Name = "InitializeComponent" ;
initializeComponent.ReturnType = new CodeTypeReference ( typeof(void) );

initializeComponent.Statements.Add( new CodeAssignStatement (
    new CodeVariableReferenceExpression ( "_label" ),
    new CodeObjectCreateExpression (
        typeof ( System.Windows.Forms.Label ), Type.EmptyTypes ) ) ;

// Add the SuspendLayout() call
initializeComponent.Statements.Add( new CodeMethodInvokeExpression (
    new CodeThisReferenceExpression ( ) , "SuspendLayout", emptyParams));

// Setup the Anchor property of the label
CodeBinaryOperatorExpression leftAndRight =
    new CodeBinaryOperatorExpression (
        new CodeFieldReferenceExpression (
            new CodeTypeReferenceExpression (
                typeof ( System.Windows.Forms.AnchorStyles ) ),
            "Left" ),
        CodeBinaryOperatorType.BitwiseOr ,
        new CodeFieldReferenceExpression (
            new CodeTypeReferenceExpression (
                typeof ( System.Windows.Forms.AnchorStyles ) ),
            "Right" ) ) ;
CodeBinaryOperatorExpression topToo = new CodeBinaryOperatorExpression (
    new CodeFieldReferenceExpression (
        new CodeTypeReferenceExpression (
            typeof ( System.Windows.Forms.AnchorStyles ) ),
        "Top" ) ,
    CodeBinaryOperatorType.BitwiseOr, leftAndRight ) ;

initializeComponent.Statements.Add (
    new CodeAssignStatement (
        new CodePropertyReferenceExpression (
            new CodeVariableReferenceExpression ( "_label" ), "Anchor" ),
    topToo ) ) ;

... // Set the other properties of the label control

// Add the ResumeLayout ( false ) call
initializeComponent.Statements.Add (
    new CodeMethodInvokeExpression (
        new CodeThisReferenceExpression ( ) , "ResumeLayout",
        new CodeExpression [] {
            new CodePrimitiveExpression ( false ) } ) ) ;

// And finally add initializeComponent to the members for the class
ctd.Members.Add ( initializeComponent ) ;

The next example shows how to set a property to a particular value. In this instance, the code emitted is:

_label.TabIndex = 0 ;
Here you need to use CodeAssignmentStatement, which takes two parameters: a left-hand side, or lhs (the thing you are assigning to), and a right-hand side, or rhs (the object being assigned). In this instance, the lhs is a CodePropertyReferenceExpression and the rhs is a CodePrimitiveExpression:
initializeComponent.Statements.Add(
    new CodeAssignStatement(
        // This forms the left hand side and is
        // _label.TabIndex
        new CodePropertyReferenceExpression(
            new CodeVariableReferenceExpression ( "_label" ),
            "TabIndex"),
        // This is the right hand side
        new CodePrimitiveExpression ( 0 ) ) ) ;

CodePropertyReferenceExpression itself takes two parameters: the object that the property is defined upon and the property name. Here I've used CodeVariableReferenceExpression, which references the _label object, and specified that the property of interest is TabIndex.

I've surrounded this with a CodeAssignStatement call, which assigns the value 0 to this property. CodePrimitiveExpression can be used for certain built-in types, such as string and integer.

The enumerated value example demonstrates how to use an enumerated value in the code. Enumerated values are stored as fields on the enumerated type. Therefore, in order to get the value of a given field (such as the AnchorStyles.Left value in the example), it is necessary to use CodeFieldReferenceExpression.

For more complex enumerated values where the value is a combination of two or more fields, it is necessary to combine these using CodeBinaryOperatorExpression. This can be used, for example, to combine field values together, as is common when defining the AnchorStyle property. These are generally combined using a logical OR and emitted to code as field1 | field2.

Within the constructed code, there are several places where objects are created and assigned to appropriate properties. One example is shown here:

_label.Location = new System.Drawing.Point(8, 8);
The code that generates this is shown here:
initializeComponent.Statements.Add(
    new CodeAssignStatement(
      new CodePropertyReferenceExpression( // Equates to this._label
          new CodeVariableReferenceExpression ( "_label" ), "Location"),
          new CodeObjectCreateExpression(  // Eguates to new Point (8,8)
              typeof ( System.Drawing.Point ) ,
              new CodeExpression[]{
                  new CodePrimitiveExpression ( 8 ) ,
                  new CodePrimitiveExpression ( 8 ) } ) ) );

I use the customary CodePropertyReferenceExpression to generate the left-hand side of the statement. CodeObjectCreateExpression is passed the type of the object to be created, and then a collection of the parameters is passed to the constructor. Similar code can be found when setting the size of the label or the initial size of the entire control.

An ASP.NET Example

Now let's look at how to construct an ASP.NET control at run time. Since the code is fairly similar to that in the Windows Forms example, I have omitted it. If you would like to view this code, it is available in the download that accompanies this article. In this example I have created a server control (derived from WebControl) that renders a string that is entered by the user. When you run the site, you can enter some text, as shown in Figure 7, and then generate either an IL or a CodeDOM example.

Figure 7 Generating ASP.NET Controls
Figure 7 Generating ASP.NET Controls

After clicking one of the buttons, the code generates a server control and loads it up inside a panel. The resulting display is shown in Figure 8. While this isn't the most useful control, it does illustrate the principle that you can create controls at run time that are used on ASP.NET pages.

Figure 8 Generated Control Placed in a Web Page
Figure 8 Generated Control Placed in a Web Page

With ASP.NET, there are two types of controls: server controls and user controls. A server control resides within an assembly and is commonly created by subclassing WebControl. They typically have excellent design-time behavior for the end user.

A user control, on the other hand, consists of an .ascx file with an associated codebehind file. The .ascx defines the layout of the control and the codebehind defines the behavior. Generating a server control for this example is far easier than generating a user control, so this is the option I have chosen for this article.

The main difference between this and the Windows Forms example is how the control is rendered. With Windows Forms custom controls, each constituent control is added within the InitializeComponent method. With ASP.NET controls, you have a couple options for rendering the HTML output. The first is to generate all of the content for the container control by overriding the Render method and explicitly writing HTML to the output stream. The second is to utilize controls that are added within the CreateChildControls method.

In this example, I am using the first method—an override of the Render method. This example is simple, but provides a basis for understanding how to define controls at run time. The example could be extended to add extra properties and rendering logic, if desired.The control is loaded at runtime using the following code:

Control ctrl = Activator.CreateInstance ( t ) as Control ;
t.GetProperty("Text").SetValue ( ctrl, caption, new object[] { } ) ;
this.controlPlaceholder.Controls.Add ( ctrl ) ;
The type t represents the generated control, and I've used Activator.CreateInstance to instantiate the control. The next line sets the value of the Text property, showing an example of run-time binding (which I'll expand upon in a moment). The final piece adds the newly created control to a placeholder control on the page.

For more details about creating custom server controls, I suggest the book Developing Microsoft ASP.NET Server Controls and Components by Nikhil Kothari and Vandana Datye (Microsoft Press, 2002). In my opinion, this book offers excellent coverage of all aspects of control creation and is a must-have if you ever define your own server controls.

Note that I have not persisted the generated controls to disk in these examples. I chose not to do this for two reasons. The first, and most obvious, reason is to prove that you can generate controls without persisting them to disk; they can be entirely transient and thus will be deleted once the application is restarted. The second, and less obvious, is that by permitting the application to generate controls on disk, you are opening up a potential security flaw in the application.

If the application were to save the generated assemblies to disk, the ASP.NET process would have to have permission to save DLL files to disk. These files would then be loaded up again to display data to the user. If an intruder were to somehow gain access to writing files to this directory, he could potentially inject code into your application with serious consequences.

If your application is modified to permit generation of controls, I suggest you make that part of an administrative toolset that is not running within a Web site—it should, for instance, be a Windows Forms application. This would read any control definitions from your database, generate the controls, and store these to disk or within a SQL Server database. The ASP.NET site would then simply load these generated controls at run time.

Data Binding

When these generated controls are used on a page, you need some way to bind data from the back-end database objects to the controls. You could use generated code (which is the recommended approach), or you could use reflection as a means of binding data to the UI. In this section, I'll describe both methods. In these examples, I am assuming there is a 1:1 correspondence between properties on the underlying business object and controls on the user interface.

Since the main purpose of this article is to provide strategies for making a high-performance configurable solution, emitting code to perform the data binding is an obvious step to take. You could quite easily generate the data binding code while generating the control (again, assuming there is a direct 1:1 relationship between the UI control and the underlying business object). The code that should be generated is essentially looping through each property and setting the textual value of control to the value of that property, as shown in this code snippet:

control.Text = businessObject.Text;
The appropriate way to emit this code for ASP.NET controls is as an override to the standard DataBind method, defined on the Control class.

Reflection can be used to retrieve data from your business logic component and display this data in the control. When using Visual Studio, a control defines properties as available for data binding by adding the Bindable attribute to the property definition:

[Bindable(true)]
public string Text { get; set; }

So, it would be possible to retrieve all the bindable properties of a control using code like that shown in Figure 9. The first line requests a collection of all properties defined on type t that are public instance properties. BindingFlags.DeclaredOnly ensures that only properties of the direct type are used—without this, all properties in the inheritance hierarchy for the type will be returned.

Figure 9 Retrieving the Bindable Properties of a Control (Close Figure 9)
ArrayList bindableProperties = new ArrayList ( );

foreach ( PropertyInfo pi in t.GetProperties ( BindingFlags.Instance |
    BindingFlags.Public | BindingFlags.DeclaredOnly) )
{
    // Check if this has the Bindable(true) attribute...
    object[] attributes = pi.GetCustomAttributes (
        typeof ( BindableAttribute ) , true ) ;

    if ( attributes.Length > 0 )
    {
        BindableAttribute bindable = attributes[0] as BindableAttribute ;

        if ( bindable.Bindable ) bindableProperties.Add ( pi ) ;
    }
}

This code then loops through these properties, looking for any that include the Bindable attribute. For each of these, it specifically checks whether the attribute has been defined as Bindable(true), as it's possible that the developer has used Bindable(false).

With this collection of bindable properties, it is possible to write code, like that shown in Figure 10, to set values of the control based on values in the business object.

Figure 10 Setting Values for Bindable Properties (Close Figure 10)
private void SetProperties ( ArrayList bindableProperties, object control,
    object businessObject )
{
    Type bot = businessObject.GetType ( ) ;
    object[] emptyIndex = new object[] { } ;

    foreach ( PropertyInfo bindable in bindableProperties )
    {
        PropertyInfo bop = bot.GetProperty (
            bindable.Name, bindable.PropertyType ) ;
        if ( null != bop )
        {
            object value = bop.GetValue ( businessObject, emptyIndex ) ;
            bindable.SetValue ( control, value , emptyIndex ) ;
        }
    }
}

This function loops through all bindable properties on the control and, for each, checks to see if a property with the same name and return type exists on the business object. If so, the property on the control is set to the value of the property on the business object. Similar code could be used to read data that the user enters into the UI, then could send the data back to the business object.

Simplifying the Code

The code emitted by this sample at present is fairly detailed. To simplify matters, you can construct a base class (derived from WebControl for ASP.NET or Control for Windows Forms) and include functions on the control to do some of the trickier aspects of control generation. As an example, you can define methods that construct controls using a set of functions, such as the following:

private Label CreateLabelControl ( string ID, string caption )
{
  Label newLabel = new Label ( ) ;
  newLabel.ID = ID;
  newLabel.Text = caption ;
  this.Controls.Add ( newLabel ) ;
}

You could create base class functions to create all types of controls. Then the emitted code would simply have to generate the instance variables for the controls and call these base class functions to do the control creation.

Conclusion

This article has illustrated how you can use two styles of runtime code generation—Reflection.Emit and System.CodeDom—to generate controls for Windows Forms and ASP.NET applications. I've given examples, and described some of the facilities available for these two differing approaches. Now I urge you to try employing these ideas in your own applications.

If you currently generate any of your interface controls using XML definitions, I hope the section on relative performance of these methods has given you the impetus to at least consider changing to a CodeDOM or Reflection.Emit approach. The performance enhancement you'll gain by creating your controls up front will be well worth the investment.

Morgan Skinner works for Microsoft in the UK, where he is an Application Development Consultant specializing in C#, Windows Forms, and ASP.NET. He's been working with .NET since its inception. Check out the Premier Support for Developers (PSfD) team blog.


Got Directory Services?: New Ways to Manage Active Directory using the .NET Framework 2.0

Got Directory Services?

New Ways to Manage Active Directory using the .NET Framework 2.0

Ethan Wilansky
This article discusses:
  • Enhancements in the System.DirectoryServices namespace
  • Using the InvokeSet and InvokeGet methods to manage ADSI properties
  • Reading and writing Active Directory security descriptors
  • Using extended search capabilities in Active Directory
This article uses the following technologies:
.NET Framework 2.0, Active Directory
Code download available at: DirectoryServices.exe (183KB)
Contents
Welcome InvokeGet and InvokeSet
Active Directory Security Descriptors
Finding Your Way with DirectorySearcher
Tombstone Search
ExtendedDN Search
Attribute Scoped Query
Virtual List View Search
What Else?

A
s more and more large enterprises use Active Directory®, Microsoft has been investing time and effort to make it simpler to install, deploy, and manage. Devs will see enhancements to the System.DirectoryServices namespace in the NET Framework 2.0. In addition, Microsoft has added two new namespaces under System.DirectoryServices:ActiveDirectory , which provides Active Directory management, and Protocols, which lets you work with the Lightweight Directory Access Protocol (LDAP) 3.0 and Directory Services Markup Language (DSML) 2.0 standards.

Just as in the .NET Framework 1.x, System.DirectoryServices is a managed code layer on top of Active Directory Service Interfaces (ADSI). Note that only some of the classes in the ActiveDirectory namespace and none of the Protocols namespace rely on the ADSI layer. When you needed to get to an ADSI interface that wasn't available from managed code, the simplest approach was to use COM interop. In the Microsoft® .NET Framework 2.0, the need to use COM interop is significantly reduced. This article explores a number of these improvements in managed code, from calling ADSI properties, to interacting with security descriptors, and finally through using some powerful extended search capabilities.

Welcome InvokeGet and InvokeSet

Reading and writing attributes of directory objects is a common management activity in directories such as Active Directory. To read and write attributes with the .NET Framework, you first bind to the directory object using the DirectoryEntry class (part of the System.DirectoryServices namespace) and then use the PropertyValueCollection class to read or write values. After writing a value, you commit the change to the directory by calling the CommitChanges method of the DirectoryEntry class. Figure 1 shows a typical example of a read and write operation on a single-valued attribute.

Figure 1 Reading and Writing an Attribute (Close Figure 1)
using System;
using System.DirectoryServices;

// Bind to the directory and set the path to an object
DirectoryEntry de = new DirectoryEntry();
de.Path = "LDAP://CN=User1,OU=TechWriters,DC=Fabrikam,DC=Com";

// Display the value of the displayName attribute
Console.WriteLine("The displayName attribute is: {0}",
    de.Properties["displayName"].Value);

// Set a value on the displayName attribute and commit the
// change to Active Directory
de.Properties["displayName"].Value = "User One";
de.CommitChanges();

In both reading and writing attributes, System.DirectoryServices is relying on the ADSI IADs core interface. The beauty of relying on IADs is the consistency it affords from attribute to attribute. In both cases, "displayName" is the lDAPDisplayName of the attribute to read or write. Every attribute in Active Directory has an associated lDAPDisplayName.

Reading and writing multivalued attributes is almost as easy. After binding to a directory object, as shown in Figure 1, use a foreach loop to iterate through the contents of a multivalued attribute. To write a multivalued attribute, use the PropertyValueCollection Add or Insert methods. To add multiple values, use the AddRange method. And, as you guessed, to delete values use the Remove method. This snippet demonstrates how to use the AddRange method to write two values to the otherTelephone multivalued attribute:

... //bind to a directory object as shown in the previous code snippet

//Add more than one additional value using the AddRange method
de.Properties["otherTelephone"].AddRange(
    new object[] { "<phone number 1>", "<phone number 2>" });
de.CommitChanges();

When it comes to managing directory attributes, the details are most important. There are three characteristics of an attribute that complicate their reading and writing:

  • An attribute's syntax
  • Operations the system lets you perform on that attribute
  • The meaning of the data in the attribute

See the Platform SDK Active Directory schema topics for more information about these characteristics. In addition, the .NET Framework 2.0 includes the new System.DirectoryServices.ActiveDirectory namespace that you can use to interrogate attributeSchema objects. This namespace can help you with the first two attribute characteristics. The third characteristic, the meaning of the data in the attribute, might require that you read the SDK documentation to understand the data.

The Active Directory schema contains the following major categories of data types: simple, string, time, and object reference. Within each of these major categories, Active Directory defines multiple attribute syntaxes (28 in all), such as Boolean and integer simple data types, NTSecurityDescriptor and OctetString string data types, and GeneralizedTime and UTCTime date data types. You can get a full list of the data types from the Platform SDK in the topic Syntaxes for Active Directory Attributes or by reviewing the contents of the ADSTypeEnum enumeration. There are a variety of syntax choices of which you need to be cognizant before writing code to read attributes.

Consider the following code that reads and displays the value of the userAccountControl attribute:

Console.WriteLine("userAccountControl: {0}",
  de.Properties["userAccountControl"].Value);
The userAccountControl attribute is an integer data type whose return value will be something like 544. Unless you know a little bit more about the attribute, this decimal value is meaningless. It turns out that the userAccountControl attribute is a bit mask for various settings associated with a user account. You set flags (defined in the ADSI ADS_USER_FLAG enumeration) by changing a bit value in userAccountControl. For example, you can check whether a user account is disabled by evaluating the second bit in the userAccountControl attribute (ADS_UF_ACCOUNTDISABLE flag), as the following code demonstrates:
using System;
using System.DirectoryServices;
using ActiveDs;  //this is for getting to the AD user flag enumeration
...
bool isDisabled;
isDisabled = ((int)de.Properties["userAccountControl"].Value &
   (int)ADS_USER_FLAG.ADS_UF_ACCOUNTDISABLE) != 0;
Console.WriteLine("Account disabled: {0}", isDisabled);

An example of setting the bits in userAccountControl by using the PropertyValueCollection.Value property is included in the code download for this article. (Before compiling and running the code, make sure that you read the "About the Code" sidebar.)

Note that the previous code snippet uses the ADSI ActiveDs type library. This ADSI type library exposes, among other things, the ADS_USER_FLAG enumeration. While you can set a constant to the value of ADS_UF_ACCOUNTDISABLE, it's more elegant to take advantage of this and other enumerations in ActiveDs if an enumeration isn't already exposed through managed code.

About the Code

The accompanying code download includes a sample app that covers the features explored in the article. I wrote it as a console application to keep the code as straightforward as possible. After you compile the application and run it, a series of switches will appear. You can call any of these switches to run the samples. Here are important caveats before compiling and running this application:

  1. The code is not meant to be a production-ready utility. There is very little error-handling in the code.
  2. You need to change the path values in the program.cs file to valid paths in your implementation of Active Directory.
  3. Your computer needs to be a member of the domain where the code will run.
  4. You need to run this on Active Directory, not ADAM.
  5. You need to be logged on as administrator or equivalent.
  6. Make sure the .NET Framework 2.0 is installed on the computer where you will compile and run the code.
  7. Run the InvokeSet switch before running the InvokeGet switch.

So far, all of the examples have shown reasonably simple uses of the PropertyValueCollection class for read and write operations. However, this is just scratching the surface. There are some attributes that cannot be read nor set using the underlying ADSI IADs core interface alone. An excellent example of this is the user account lockout status. The ADSI documentation suggests that Active Directory stores lockout status in the fifth bit of the userAccountControl attribute. This likely applies to the lockout status of SAMAccounts, but not Active Directory accounts. Instead, you have to read the value of the lockoutTime attribute to determine if and when an account was locked. The lockoutTime attribute stores a Universal Time Coordinate (UTC) value in 100-nanosecond increments representing exactly when the account was locked after a start date of 1/1/1601. The attribute syntax for lockoutTime is LargeInteger. The PropertyValueCollection class does not provide a method for converting the large integer into a readable value. Instead, you have to help the PropertyValueCollection class by using COM interop to get to the ADSI IADs LargeInteger interface. Figure 2 shows code that performs this task.

Figure 2 Reading the lockoutTime Attribute (Close Figure 2)
//COM Interop for accessing the IADsLargeInteger interface
using ActiveDs;
using System.DirectoryServices;

public static void GetLockoutTime(DirectoryEntry de)
{
    // Change this to a valid ADSPath in your implementation of
    // Active Directory.
    de.Path = "LDAP://CN=user1,OU=TechWriters,DC=Fabrikam,DC=Com";

    if (de.Properties.Contains("lockoutTime"))
    {
        IADsLargeInteger int64Value =
            (IADsLargeInteger)de.Properties["lockoutTime"].Value;

        // This is the standard formula for converting the large integer
        // (it's highpart and lowpart) into a readable value.
        long largeInt =
            (long)((uint) int64Value.LowPart +
            (((long) int64Value.HighPart) << 32 ));
        if (largeInt != 0)
        {
            Console.WriteLine("Lockout Status {0}", largeInt);

             //pad the date with the starting date of 1\1\1601
             DateTime dtmPad = new DateTime(1601, 1, 1);

             DateTime dtm = new DateTime(largeInt);
             Console.WriteLine("DC reported time account locked is {0}",
                dtm.AddTicks(dtmPad.Ticks).ToLocalTime().ToString());
        }
        else Console.WriteLine("lockoutTime is set to: {0}", largeInt);
    }
    else Console.WriteLine("lockoutTime is not set");
}

The purpose of this code, however, is not to show you how to read the lockoutTime attribute or to show you how to read attributes stored as large integers. Instead, it's a preamble to my proposal that sometimes sticking with the IADs core interface isn't worth the heavy lifting.

Thankfully, there's another option: using ADSI properties to read values instead of Active Directory attributes. For example, you can determine whether an account is locked by using the IsAccountLocked property of the IADsUser persistent interface. To read this property using the .NET Framework 1.1, you have to use the Type.InvokeMember method and reflection, as shown in the following code snippet:

using System.DirectoryServices;
using System.Reflection;
...
object ads = de.NativeObject;
Type type = ads.GetType();
bool isLocked = (bool)type.InvokeMember
    ("IsAccountLocked", BindingFlags.GetProperty,
    null, ads, null);
Console.WriteLine("The IsAccountLocked property is: {0}",
    isLocked);
This is a fair amount of work to retrieve a single property and, thankfully, the .NET Framework 2.0 makes this very easy by exposing the InvokeGet and InvokeSet methods for reading and writing properties of ADSI objects. In contrast to the code in Figure 2 and the snippet just shown, here's what it takes to read and display the IsAccountLocked property using InvokeGet:
using System.DirectoryServices;
...
Console.WriteLine("Account locked: {0}",
    de.InvokeGet("IsAccountLocked"));

Writing a property is just as easy. InvokeSet takes two parameters, the first being the property name and the second being the value you want to set. Active Directory restricts you to reading the underlying lockoutTime attribute and setting it to 0 or unlocked. Therefore, InvokeSet takes one parameter—in this case, the Boolean false:

de.InvokeSet("IsAccountLocked", false);
de.CommitChanges();

In general, I don't recommend using ADSI properties because they do not provide access to all attributes associated with an Active Directory object, and their names often diverge from the underlying lDAPDisplayName of the attribute they modify. In some cases this name divergence is necessary because a single Active Directory attribute might store values for multiple settings in Active Directory even though an ADSI property represents only one of those settings. The userAccountControl attribute and the IsAccountDisabled property provide a perfect example of that divergence. However, for attributes that are difficult to read and write any other way, InvokeGet and InvokeSet are welcome additions to the DirectoryEntry class. Additional examples appear in this article's code download. For example, the code download demonstrates using InvokeGet and InvokeSet respectively for reading and writing terminal services user profile settings contained in the userParameters attribute and reading and writing multivalued properties.

Active Directory Security Descriptors

In the .NET Framework 1.x, you modified Active Directory security descriptors using COM interop for accessing three of the four ADSI Security interfaces (IADsSecurityDescriptor, IADsAccessControlList, and IADsAccessControlEntry). Probably one of the most significant improvements in System.DirectoryServices moving from the .NET Framework version 1.x to version 2.0 is the ability you now have to modify Active Directory and Active Directory Application Mode (ADAM) security descriptors from managed code.

If you aren't already painfully familiar with security descriptors, I suggest you take a look at Keith Brown's Security Briefs column in the January 2005 issue of MSDN®Magazine and Mark Novak's "Safety in Windows" article in the November 2004 issue of MSDN Magazine. Although these two articles are not primers on security descriptors, they do provide insights into the basis for a lot of the improvements in Active Directory security descriptor management.

Figure 3 shows the structure of an Active Directory security descriptor and the primary .NET Framework 2.0 namespaces and classes that you use to interact with it.

Figure 3 Security Descriptor Structure
Figure 3 Security Descriptor Structure

The Active Directory ntSecurityDescriptor attribute is a complex structure. Working with the attribute from managed code requires a variety of classes in the System.DirectoryServices, System.Security.Principal, and System.Security.AccessControl namespaces. This multiple namespace involvement is the result of work Microsoft undertook in building the managed Access Control List (ACL) classes. These classes provide a unified approach for modifying security descriptors. While all this class involvement might make you think that interacting with the ntSecurityDescriptor attribute is difficult, it's not so bad.

Prior to working with the security descriptor, you must bind to an Active Directory object, as I showed in my first code snippet. After binding to an object, you retrieve the security descriptor by using the new System.DirectoryServices ObjectSecurity property. But not so fast! Before you call ObjectSecurity, you need to decide if reading from or writing to the System Access Control List (SACL) is important to you. The SACL contains Access Control Entries (ACEs) for applying auditing permissions to an Active Directory object.

If reading the SACL is important, you employ the services of the DirectoryEntryConfguration class. This class, new in the .NET Framework 2.0, calls the ADSI IADsObjectOptions interface for accessing provider-specific options. Currently IADsObjectOptions supports Active Directory and, in this case, you use it to retrieve the SACL from the security descriptor. The following code demonstrates how you get the SACL and then get the ntSecurityDescriptor prior to performing Active Directory security descriptor management operations:

//include the SACL of the security descriptor
de.Options.SecurityMasks = SecurityMasks.Owner | SecurityMasks.Group |
    SecurityMasks.Dacl | SecurityMasks.Sacl;

// Get the Active Directory security descriptor by using the
// ObjectSecurity property
ActiveDirectorySecurity sd = de.ObjectSecurity;

Figure 4 shows how you can read properties of the security descriptor. After binding to the Active Directory object and retrieving its security descriptor, use the GetGroup and GetOwner methods to get the primary group and security descriptor owner. The return value in both cases is a SecurityIdentifier that you can easily display as a Windows NT® user account using the translate method of the SecurityIdentifier class, as the code demonstrates.

Figure 4 Reading the Security Descriptor (Close Figure 4)
using System;
using System.DirectoryServices;
using System.Security.Principal;

public static void ADSReadACLsExp(
    DirectoryEntry de, ActiveDirectorySecurity sd)
{
    Console.WriteLine("\r\nACLs of DE");

    //The return value of GetGroup is a SID
    string grpSID = sd.GetGroup((typeof(SecurityIdentifier))).Value;
    Console.WriteLine("Group SID: {0}", grpSID);

    //Similar operation to get the SD owner. Uses GetOwner
    string ownerSID = sd.GetOwner((typeof(SecurityIdentifier))).Value;
    Console.WriteLine("Owner SID: {0}", ownerSID);

    /* GetGroup and GetOwner return the SID in Security Descriptor
     * Definition Language (SDDL) format. An efficient way to return the
     * account name is to use the Translate method to convert the
     * SecurityIdentifier to an NTAccount, then call the value property
     * of the NTAccount class to retrieve the account name. */
    NTAccount groupAcctName =
        (NTAccount)sd.GetGroup(typeof(SecurityIdentifier))
            .Translate(typeof(NTAccount));
    Console.WriteLine("Primary Group Account name:\t{0}",
        groupAcctName.Value);

    NTAccount ownerAcctName =
        (NTAccount)sd.GetOwner(typeof(SecurityIdentifier))
         .Translate(typeof(NTAccount));
    Console.WriteLine("Owner Account name:\t\t{0}", ownerAcctName.Value);

    //get the AccessRules for this Active Directory object's SID
    //(boolean values)

    //Windows Server 2003 correctly orders ALL ACEs when a modified
    //security descriptor is written back to the property cache
    Console.WriteLine("ACEs in DACL are properly ordered:\t{0}",
           sd.AreAccessRulesCanonical);

    //this means whether permissions are inherited or not.
    //if it's true, then inheritance is disabled.
    Console.WriteLine("ACE inheritance in DACL is disabled:\t{0}",
        sd.AreAccessRulesProtected);

    Console.WriteLine("ACEs in SACL are properly ordered:\t{0}",
        sd.AreAuditRulesCanonical);

    Console.WriteLine("ACE inheritance in SACL is disabled:\t{0}",
        sd.AreAuditRulesProtected);
}

The remaining properties appearing in Figure 4 provide information on whether the security descriptor contains correctly ordered ACEs and whether the ACEs are inherited. Windows Server™ 2003 correctly orders the ACEs while in Windows 2000 Server you must write back ACEs from the local property cache to the security descriptor in the proper order. Therefore, it's worthwhile to check the AreAccessRulesCanonical and AreAuditRulesCanonical properties when managing and reading security descriptors on Windows 2000 Server.

An Active Directory security descriptor contains two ACLs: the Discretionary Access Control List (DACL) and the SACL. These lists contain ACEs, as Figure 3 shows. In the case of the DACL, they are access permissions, and in the case of the SACL, they are audit permissions. Managing these two types of ACEs is essentially the same in managed code with some small differences.

The solution that you can download with this article includes examples for reading and writing ACEs in an ACL. Rather than spending too much time on the entire code for reading and writing ACEs, I'll point out several important differences between reading ACEs in a DACL versus a SACL.

There are two methods for accessing the ACEs in a security descriptor, GetAccessRules for ACEs in a DACL and GetAuditRules for ACEs in an SACL. These methods allow you to specify whether you want to get only explicitly assigned ACEs, only inherited ACEs, or both. The following code demonstrates how you get both explicitly assigned and inherited ACEs from a security descriptor named sd:

sd.GetAccessRules(true, true, typeof(SecurityIdentifier))

The GetAccessRules method returns a collection of authorization rules (AuthorizationRuleCollection) while the GetAuditRules method returns a collection of audit rules (AuditRuleCollection). In either case, begin by verifying that the Count property of the collection is greater than 0. If so, call the GetType method to determine whether the ACEs returned are either ActiveDirectoryAccessRule or ActiveDirectoryAuditRule types. With that information, you can then iterate the collection and call applicable properties of the ACE. For instance, for an ActiveDirectoryAccessRule, you call the AccessControlType property to determine if the ACE is an Allow or Deny type while if the ACE is an ActiveDirectoryAuditRule, you call the AuditFlags property to determine if the ACE is a Success or Failure type.

Finding Your Way with DirectorySearcher

System.DirectoryServices search capabilities in the .NET Framework 2.0 provide powerful new features that are not present in the .NET Framework 1.0. In the following section I will explore these four new search capabilities:

  • Directory Synchronization (DirSync) search
  • Tombstone search
  • Attribute Scoped Query (ASQ) search
  • Virtual List View (VLV) search

Directory Synchronization search is an interesting feature that is new to System.DirectoryServices in the .NET Framework 2.0. This feature provides an effective approach for tracking changes occurring in an Active Directory domain, schema, or configuration partition. DirSync is a managed code layer on top of the ADSI DirSync control. The control is an LDAP server extension designed to specifically search for changes in Active Directory.

A DirSync operation is a two-step process. First, you capture a snapshot of a partition and then store it as a blob data (a cookie). Next, you read the cookie back and compare it to the same partition at a later time. You should save a cookie in whatever data store you will use to perform comparison or synchronization operations with Active Directory (for example, in SQL Server).

To keep this example straightforward, Figure 5 demonstrates how to capture a snapshot of the partition to the file system. Like all search operations from managed code, you use the DirectorySearcher class to create an object for configuring and executing the search. The DirectorySearcher requires a directory entry object. In the case of a DirSync search, the directory entry object must begin at the root of a partition. For example, to begin a search at the root of the fabrikam.com domain, the path property of the directory entry object is:

de.Path = "LDAP://DC=Fabrikam,DC=Com";

Figure 5 Taking a Snapshot for a Later DirSync Search (Close Figure 5)
public static void DirSyncSnapShot(DirectoryEntry de)
{
    //Create a DirectorySearcher object and specify
    //the root of the domain for the search
    using(DirectorySearcher srch = new DirectorySearcher(de))
    {
        //Optionally, limit the search using a filter
        srch.Filter = "ou=tech*";

        //Set the search to the entire subtree
        srch.SearchScope = SearchScope.Subtree;

        //Create a DirectorySynchronization object to begin
        //a DirSync operation - take a snapshot of the directory
        srch.DirectorySynchronization = new DirectorySynchronization();

        Console.WriteLine(
            "The OUs within scope for this sync. search are:");

        using(SearchResultCollection results = srch.FindAll())
        {
            foreach (SearchResult res in results)
            {
                Console.WriteLine("\t{0}", res.Path);
            }
        }

        //Create a file and return a FileStream object in preparation
        //for storing data in the file. If the file exists, it will be
        //overwritten.
        BinaryFormatter bFormat = new BinaryFormatter();
        using(FileStream fsStream =
            new FileStream("ADSync.data", FileMode.Create))
        {
           //Serialize the data to the steam. To get the data for
           //the cookie, call the GetDirectorySynchronizationCookie
           //method.
           bFormat.Serialize(fsStream, srch.DirectorySynchronization.
               GetDirectorySynchronizationCookie());
        }
    }
}

To create the DirectorySearcher object, provide the DirectorySearcher constructor with the directory entry object representing the root of a partition. Following the creation of the DirectorySearcher, you can set properties on the object. For example, Figure 5 shows how to set the Filter property to limit the search to a set of organizational units (OUs) and how to set the search scope to Subtree to search everything below the starting point of the search.

Next, create a DirectorySynchronization object from the DirectorySearcher to create a cookie to hold the search results. After iterating the SearchResult collection, you need to store the DirSync cookie somewhere. In Figure 5, I use a FileStream object and a BinaryFormatter object to create a file stream, and then serialize the data to the stream in binary format. The GetDirectorySynchronizationCookie method of the DirectorySynchronization object is responsible for providing the data to the stream.

Following the snapshot operation and after a change or two in directory objects within scope of the original capture operation, you run a DirSync search again to detect changes. Figure 6 shows the code for comparing the Active Directory partition to the stored cookie on the file system. The method returns a Boolean value so that you can inform the user if no change was detected. This code snippet shows an example of calling code for the DirSyncChanges class:

if (!DirSynch.DirSyncChanges(de))
    Console.WriteLine("No changes detected");

Figure 6 Performing a DirSync Search to Detect Changes (Close Figure 6)
public static bool DirSyncChanges(DirectoryEntry de)
{
    bool changesDetected = false;

    //Open the data source containing the snapshot.
    BinaryFormatter bFormat = new BinaryFormatter();
    byte[] cookie;
    using(FileStream fsStream =
        new FileStream("ADSync.data", FileMode.Open))
    {
        //Deserialize the data in the file stream
        cookie = (byte[])bFormat.Deserialize(fsStream);
    }

    //Initialize a new instance of the DirectorySynchronization class and
    //pass it the cookie that you deserialized using the BinaryFormatter
    using (DirectorySynchronization syncData =
        new DirectorySynchronization(cookie))
    {
        //Create a DirectorySearcher object
        DirectorySearcher srch = new DirectorySearcher(de);

        //Set the DirectorySearcher object DirectorySynchronization
        //property equal to the cookie you retrieved from a data store.
        srch.DirectorySynchronization = syncData;

        //Example option to specify that the replicate changes right is
        //unnecessary, but limit results to objects and attributes to
        //which the caller has rights to read
        syncData.Option = DirectorySynchronizationOptions.ObjectSecurity;

        //Iterate results
        using(SearchResultCollection results = srch.FindAll())
        foreach (SearchResult res in results)
        {
            //Entering loop means a change was detected
            changesDetected = true;

            ResultPropertyCollection deltaAttribs;
            deltaAttribs = res.Properties;
            Console.WriteLine(
                "adsPath of created or changed object: {0}\n",
            res.Path);

            Console.WriteLine("New or changed attributes\n");

            foreach (string deltaAttrib in deltaAttribs.PropertyNames)
            {
                switch (deltaAttrib.ToLower())
                {
                    case "objectguid":
                    case "adspath":
                    case "instancetype":
                    case "distinguishedname":
                        break;
                    default:
                        Console.Write(deltaAttrib);
                        foreach (object attribVals in
                            deltaAttribs[deltaAttrib])
                        {
                            Console.WriteLine(": {0}", attribVals);
                        }
                        break;
                }

            }
        }

        Console.WriteLine("\n{0}\n", new string('-', 10));
    }
    return changesDetected;
}

To start the comparison operation, load the cookie (the prior snapshot of the directory partition) from the data store. In Figure 5, the code uses the file system as the data store and it creates a file named ADSync.data. Therefore, as shown in Figure 6, create a FileStream object to open the cookie stored in the ADSync.data file. Next, create a BinaryFormatter object and deserialize the data in the file stream. With the deserialized binary data in memory stored in a variable named cookie, call the DirectorySynchronization constructor and this time pass in the cookie as the first parameter of the constructor to create a DirectorySynchronization object named syncData. The DirectorySynchronization class also contains an Option property to control exactly what data is returned in a DirSync search. The ObjectSecurity object appearing in Figure 6 relaxes the requirement that the caller running this search must have the replicate change right to see detected changes. However, it also limits search results to objects and attributes to which the caller has permission to read.

Now that you have a DirectorySynchronization object representing the original snapshot, create a DirectorySearcher object and provide it with the directory entry object representing the partition to compare to the cookie. Be sure that the root partition you provide as a parameter to the DirectorySearcher constructor is the same partition used for cookie creation. Next, set the DirectorySynchronization property of the DirectorySearcher object equal to syncData (the DirectorySynchronization object representing the original snapshot of the partition) and iterate the collection of search results. If there isn't anything to iterate, the changesDetected variable initialized in the beginning of the class code does not get set to true and the class returns false to indicate that no changes were detected.

Inside of the foreach loop, you might notice a little more is happening than a typical iteration of a SearchResultCollection object. The directory synchronization results will always contain the ADsPath and the objectGUID, instanceType and distinguishedName attributes of an object. This information is useful in a directory synchronization operation, but might not be as interesting to you for simply displaying the detected changes in a directory partition. Therefore, after returning the ADsPath of the new or changed object, the code skips repeating the ADsPath and doesn't list the three attributes returned by default for any new or changed object.

The code in Figure 6 does not do anything special to deal with complex attributes. For example, suppose you modify the security descriptor of an object. When you run the code, it will tell you that the ntSecurityDescriptor has changed, but it won't tell you exactly how it changed. As I mentioned earlier, you must accommodate the special characteristics of attributes, such as attribute syntax, in order to read or write to them in your code. Using various examples included with this article's code download should provide you with a starting point for getting to the details of changes occurring in complex attributes.

Tombstone Search

While DirSync search in managed code is a very useful feature for auditing new object creation and changes to existing objects, there are times when you might also want to determine when or if objects were deleted from a directory. This is where the new Tombstone search feature in the .NET Framework 2.0 comes in quite handy. Tombstone search enables the DirectorySearcher to return deleted objects that remain in the directory. Deleted objects are not typically returned by the DirectorySearcher. The default retention policy for deleted directory objects in Active Directory is 60 days. If the retention period for an object hasn't expired, you can use the Tombstone search feature in managed code to read information about the deleted object, as shown in Figure 7.

Figure 7 Performing a Tombstone Search (Close Figure 7)
public static void deletedObjs(DirectoryEntry de)
{
    de.AuthenticationType = AuthenticationTypes.FastBind |
        AuthenticationTypes.Secure;

    using(DirectorySearcher srch = new DirectorySearcher(de))
    {
        //Return only deleted objects
        srch.Filter = "isDeleted=TRUE";

        //Search OneLevel if you know your starting point.
        srch.SearchScope = SearchScope.OneLevel;

        string[] attribs = new string[] {
            "lastknownparent", "whencreated", "whenchanged"};
        srch.PropertiesToLoad.AddRange(attribs);

        // Instruct the DirectorySearcher to return deleted objects
        srch.Tombstone = true;

        Console.WriteLine("Objects in the deleted objects container");
        Console.WriteLine("New Relative Distinguished Name (RDN) is:");
        string ADsPathFormat =
            @"<original RDN name>\OADEL:<previous objectGUID string>," +
                "<dn of current location>";
        Console.WriteLine(new string('_', ADsPathFormat.Length) +
            Environment.NewLine);
        using(SearchResultsCollection results = srch.FindAll())
        {
            foreach (SearchResult res in results)
            {
                if (res.Path.ToLower() == de.Path.ToLower())
                {
                    Console.WriteLine("Deleted Objects Container:");
                    Console.WriteLine("ADsPath: {0}\n", res.Path);
                }
                else
                {
                    Console.WriteLine("ADsPath: {0}", res.Path);
                    Console.WriteLine("LastKnownParent:\t{0}",
                        res.Properties["lastknownparent"][0]);
                    DateTime dt = DateTime.Parse(
                        res.Properties["whencreated"][0].ToString());
                    Console.WriteLine("WhenCreated (local):\t{0}\n",
                        dt.ToLocalTime());
                }
            }
        }
    }
}

To begin a Tombstone search, use the DirectorySearcher class to create an object for configuring and executing the search. The directory entry object for a Tombstone search can be the domain partition. However, it's more efficient to start the search in the Deleted Objects container. This is the container where all deleted objects (orphans) are stored for the default retention period. For example, the following directory entry path specifies this special container in the fabrikam.com domain:

de.Path = "LDAP://cn=Deleted Objects,DC=Fabrikam,DC=Com";

Before you attempt to iterate the SearchResultCollection object, the code must specify the FastBind option of the System.DirectoryServices.AuthenticationTypes enumeration. This instructs ADSI not to verify that the object actually exists in Active Directory before executing the query. FastBind serves other purposes as well, but for the goal of performing a Tombstone search, the other capabilities are not essential to running the code. Setting the Filter property of the DirectorySearcher to "isDeleted=TRUE" causes the DirectorySearcher to return only deleted items. Setting the SearchScope property of the DirectorySearcher to OneLevel is appropriate in this case because Active Directory stores all deleted objects in the Deleted Objects container. You'll see in Figure 7 that I'm only interested in a handful of returned attributes about each object. I specify these attributes in which I'm interested by using the PropertiesToLoad.AddRange method. The next part of the code is essential to performing a search for deleted objects. That is, setting the Tombstone property of the DirectorySearcher object to true. If you don't do this, the DirectorySearcher will not return deleted objects in the search results.

After configuring the DirectorySearcher for a Tombstone search, the code retrieves the SearchResultCollection using the FindAll method. In the ensuing iteration of the search results, I do massage the data a little so that the code returns the whenCreated attribute in local time.

If you run the code, you will notice a somewhat odd ADsPath. The ADsPath format of a deleted object is in this form:

LDAP://<original RDN name>\OADEL:<previous objectGUID string>,
<DN of current location>
Active Directory stores the deleted object with its original relative distinguished name (RDN) and its original GUID followed by the fully qualified DN of its current location. For example, the following results show the ADsPath for a group named group1. The output appears on lines one and two:
ADsPath: LDAP://CN=group1\0ADEL:939a7dd5-02be-4616-abeb-59751b315f0f,
    CN=Deleted Objects,DC=FABRIKAM,DC=COM
LastKnownParent:        OU=HR,DC=FABRIKAM,DC=COM
WhenCreated (local):    7/21/2005 11:06:23 AM
Notice the other attributes that I've chosen to return in the code. The LastKnownParent attribute indicates the last place where the object existed prior to being deleted. The WhenCreated value indicates when the object was created in the Deleted Objects folder, which indicates the approximate time that the object was deleted from Active Directory.

ExtendedDN Search

The next search capability I'll explore is ExtendedDN search. In most cases anyone who works with System.DirectoryServices is used to looking at an object's distinguishedName attribute in distinguished name format. For example, the DN of User1 in the HR OU of the fabrikam.com domain is:

CN=User1,OU=HR,DC=fabrikam,DC=com

This common format is useful, but there are times you might also want to read the SID of a security principal or the GUID of any object in Active Directory. Granted, you can bind to an object in the directory and use the new InvokeGet method to easily get the GUID property of an object from the IAD's core interface or use the SecurityIdentifier class to translate an SID into an account name. However, neither of these approaches use the high-performance search facility built into ADSI and extended to System.DirectoryServices. This is where ExtendedDN search becomes useful. Using the ExtendedDN enumeration, you can specify any of these DN formats: None, Standard, or HexString. None is the default distinguished name format. Standard returns a GUID, SID, and DN in string format. HexString returns the GUID and SID in hex format. Here's an example of a User1 account in Standard format:

<GUID=9564482a-4822-4da4-bd38-ab6a740dc2d8>;<SID=S-1-5-21-2422933499-
3002364838-2613214872-1173>;CN=User1,OU=TechWriters,DC=FABRIKAM,DC=COM
And here is the same account in HexFormat:
<GUID=2a4864952248a44dbd38ab6a740dc2d8>;<SID=010500000000000515000000f
b076b90a673f4b2987ec29b95040000>;CN=User1,OU=TechWriters,DC=FABRIKAM,D
C=COM

The code download includes examples that demonstrate how you perform an ExtendedDN search against objects in an OU and objects in the special Deleted Objects container. The essential element for performing an ExtendedDN search is specifying the ExtendedDN property of the DirectorySearcher and providing a value equal to one of the three values in the ExtendedDN enumeration. The code in Figure 8 shows how you can use the String.Split method to create a string array and display the GUID and SID as separate string values.

Figure 8 Splitting Contents of an ExtendedDN Search Result (Close Figure 8)
...
using(SearchResultCollection results = srch.FindAll())
{
    foreach (SearchResult res in results)
    {
        Console.WriteLine("{0}\n",
            res.Properties["distinguishedName"][0].ToString());

        Console.WriteLine("\nADsPath: {0}", res.Path);
        String extDn =
            res.Properties["distinguishedName"][0].ToString();
        String[] dnParts = extDn.Split(new char[] { ';' });
        if (dnParts.Length == 1)
        {
            Console.WriteLine(extDn);
        }
        else
        {
            Console.WriteLine(dnParts[0]);
            Console.WriteLine(dnParts[1]);
            Console.WriteLine(new string('-', 30));
        }
    }
}

Attribute Scoped Query

Another valuable feature added to the DirectorySearcher in the .NET Framework 2.0 is the ASQ. In a typical search, you set the search scope to somewhere in a partition, such as the domain partition or an OU. ASQ brings more granularity to search by allowing you to set an attribute as the starting point for a search operation. This is particularly useful for large multivalued attributes, such as the member attribute of an Active Directory group object. Here's a simple search example using ASQ to read the members attribute of a group:

public static void ForMembers(DirectoryEntry de)
{
    using(DirectorySearcher srch = new DirectorySearcher(de))
    {
        srch.SearchScope = SearchScope.Base;
        srch.AttributeScopeQuery = "member";
        srch.PropertiesToLoad.Add("cn");

        using(SearchResultCollection results = srch.FindAll())
        {
            foreach (SearchResult res in results)
            {
                Console.WriteLine(res.Properties["cn"][0]);
            }
        }
    }
}

The DirectoryEntry for an ASQ search is an Active Directory leaf object, such as a user account or group. For example, the code for DirectoryEntry.Path property for the ForMembers code snippet could be the following:

de.Path = "LDAP://CN=administrators,cn=Builtin,DC=Fabrikam,DC=Com";
Notice that I scoped the search to SearchScope.Base. This is critical because a scope of OneLevel or Subtree is meaningless when scoping a query to an attribute. If you specify anything other than Base, your code will throw an exception. To scope a query to an attribute, you set the AttributeScopeQuery to the name of the attribute.

Virtual List View Search

The last new search feature in the .NET Framework 2.0 I'll discuss here is the Virtual List View search. The purpose of a VLV search is to retrieve a small subset of a large data set returned by ADSI. This is a convenience feature so that users get good performance when searching for a lot of information in a large directory. VLV avoids overburdening a user's local cache with an enormous data set. For a little more information on this feature, see Keith Brown's Security Briefs column mentioned earlier. In addition, I've included a working example of this capability in the download.

What Else?

As this article demonstrates, there are quite a few significant enhancements in the System.DirectoryServices namespace. Other enhancements, such as the SecurityMask search capability, are worth exploring. SecurityMask provides a high-performance approach to returning security descriptor information about objects specified in your search.

Beyond System.DirectoryServices, take a close look at the System.DirectoryServices.ActiveDirectory and System.DirectoryServices.Protocols namespaces. System.DirectoryServices.ActiveDirectory improves access to Active Directory and ADAM with classes designed for how system administrators manage Active Directory. For example, you can use classes in this namespace to manage replication, trust relationships, and the schema. The System.DirectoryServices.Protocols namespace provides classes for working with the LDAP 3.0 and DSML 2.0 standards. As you can see, Microsoft has been hard at work enhancing access to directories and, as expected, significantly improving access to Active Directory and ADAM from managed code.

Ethan Wilansky is a Directory Services MVP and a chief technologist for EDS in its Next Generation Technologies and Architectures practice. He has authored or coauthored more than a dozen books for Microsoft, has just completed coauthoring the Microsoft Shell (MSH) language reference, and is a contributing editor for Windows IT Pro.


Ten Essential Tools: Visual Studio Add-Ins Every Developer Should Download Now

Ten Essential Tools

Visual Studio Add-Ins Every Developer Should Download Now

James Avery
This article discusses:
  • Tools for unit testing
  • Tools for code and documentation management and generation
  • Tools for modifying the development environment
  • Tools for working with data
This article uses the following technologies:
Visual Studio
Contents
TestDriven.NET
GhostDoc
Smart Paster
CodeKeep
PInvoke.NET
VSWindowManager PowerToy
WSContractFirst
VSMouseBindings
CopySourceAsHTML
Cache Visualizer
Wrapping It Up

add-ins as opposed to standalone tools. Visual Studio provides a rich extensibility model that developers at Microsoft and in the community have taken advantage of to provide a host of quality add-ins. Some add-ins contribute significant how-did-I-live-without-this functionality, while others just help you automate that small redundant task you constantly find yourself performing.

10 Must-Have Add-Ins
TestDriven.NET
GhostDoc
Paster
CodeKeep
PInvoke.NET
VSWindowManager PowerToy
WSContractFirst
VSMouseBindings
CopySourceAsHTML
Cache Visualizer

In this article, I introduce you to some of the best Visual Studio add-ins available today that can be downloaded for free. I walk through using each of the add-ins, but because I am covering so many I only have room to introduce you to the basic functionality. Each of these add-ins works with Visual Studio .NET 2003 and most of them already have versions available for Visual Studio 2005. If a Visual Studio 2005 version is not available as of the time of this writing, it should be shortly.

TestDriven.NET

Test-driven development is the practice of writing unit tests before you write code, and then writing the code to make those tests pass. By writing tests before you write code, you identify the exact behavior your code should exhibit and, as a bonus, at the end you have 100 percent test coverage, which makes extensive refactoring possible.

NUnit gives you the ability to write unit tests using a simple syntax and then execute those tests one by one or altogether against your app. If you are using Visual Studio Team System, you have unit testing functionality built into the Visual Studio IDE. Before the Visual Studio Team System, there was TestDriven.NET, an add-in that integrates NUnit directly into the Visual Studio IDE. Using a non-Team System version of Visual Studio 2005 or Visual Studio .NET 2003, is, in my opinion, still the best solution available.

TestDriven.NET adds unit testing functionality directly to the Visual Studio IDE. Instead of writing a unit test, switching over to the NUnit GUI tool, running the test, then switching back to the IDE to code, and so on, you can do it all right in the IDE.

Figure 1 New Testing Options from TestDriven.NET
Figure 1 New Testing Options from TestDriven.NET

After installing TestDriven.NET you will find a number of new menu items on the right-click context menu as shown in Figure 1. You can right-click directly on a unit test and run it. The results will be displayed in the output window as shown in Figure 2.

Figure 2 Output of a Unit Test
Figure 2 Output of a Unit Test

While executing unit tests in the IDE is invaluable by itself, perhaps the best feature is that you can also quickly launch into the debugger by right-clicking on a test and selecting Test With | Debugger. This will launch the debugger and then execute your unit tests, hitting any breakpoints you have set in those tests.

In fact, it doesn't even have to be a unit test for TestDriven.NET to execute it. You could just as easily test any public method that returns void. This means that if you are testing an old app and need to walk through some code, you can write a quick test and execute it right away.

TestDriven.NET is an essential add-in if you work with unit tests or practice test-driven development. (If you don't already, you should seriously consider it.) TestDriven.NET was written by Jamie Cansdale and can be downloaded from www.testdriven.net.

GhostDoc

XML comments are invaluable tools when documenting your application. Using XML comments, you can mark up your code and then, using a tool like nDoc, you can generate help files or MSDN-like Web documentation based on those comments. The only problem with XML documentation is the time it takes to write it you often end up writing similar statements over and over again. The goal of GhostDoc is to automate the tedious parts of writing XML comments by looking at the name of your class or method, as well as any parameters, and making an educated guess as to how the documentation should appear based on recommended naming conventions. This is not a replacement for writing thorough documentation of your business rules and providing examples, but it will automate the mindless part of your documentation generation.

For instance consider the method shown here:

private void SavePerson(Person person)
{

}
After installing GhostDoc, you can right-click on the method declaration and choose Document this. The following comments will then be added to your document:
/// <summary>
/// Saves the person.
/// </summary>
/// <param name="person">Person.</param>
private void SavePerson(Person person)
{

}
As you can see, GhostDoc has automatically generated a summary based on how the method was named and has also populated the parameter comments. Don't stop hereyou should add additional comments stating where the person is being saved to or perhaps give an example of creating and saving a person. Here is my comment after adding some additional information by hand:
/// <summary>
/// Saves a person using the configured persistence provider.
/// </summary>
/// <param name="person">The Person to be saved</param>
private void SavePerson(Person person)
{

}

Adding these extra comments is much easier since the basic, redundant portion is automatically generated by GhostDoc. GhostDoc also includes options that allow you to modify existing rules and add additional rules that determine what kind of comments should be generated.

GhostDoc was written by Roland Weigelt and can be downloaded from www.roland-weigelt.de/ghostdoc.

Smart Paster

Strings play a large role in most applications, whether they are comments being used to describe the behavior of the system, messages being sent to the user, or SQL statements that will be executed. One of the frustrating parts of working with strings is that they never seem to paste correctly into the IDE. When you are pasting comments, the strings might be too long or not aligned correctly, leaving you to spend time inserting line breaks, comment characters, and tabbing. When working with strings that will actually be concatenated, you have to do even more work, usually separating the parts of the string and inserting concatenation symbols or using a string builder.

The Smart Paster add-in helps to limit some of this by providing a number of commands on the right-click menu that let you paste a string from the clipboard into Visual Studio using a certain format. After installing Smart Paster, you will see the new paste options available on the right-click menu (see Figure 3).

Figure 3 String Pasting Options from Smart Paster
Figure 3 String Pasting Options from Smart Paster

For instance, you might have the following string detailing some of your business logic:

To update a person record, a user must be a member of the customer
service group or the manager group. After the person has been updated, a
letter needs to be generated to notify the customer of the information
change.
You can copy and paste this into Visual Studio using the Paste As | Comment option, and you would get the following:
//To update a person record a user must be a member of the customer
//service group or the manager group. After the person has been updated
//a letter needs to be generated to notify the customer of the
//information change.

The correct comment characters and carriage returns are automatically inserted (you can configure at what length to insert a character return). If you were inserting this text without the help of Smart Paster, it would paste as one long line, forcing you to manually add all the line breaks and comment characters. As another example, let's say you have the following error message that you need to insert values into at run time:

You do not have the correct permissions to perform <insert action>.
You must be a member of the <insert group> to perform this action.
Using the Paste As | StringBuilder command, you can insert this string as a StringBuilder into Visual Studio. The results would look like this:
StringBuilder stringBuilder = new StringBuilder(134);
stringBuilder.AppendFormat(
    @"You do not have the correct permissions to ");
stringBuilder.AppendFormat(
    @"perform <insert action>. You must be a member of ");
stringBuilder.AppendFormat(
    @"the <insert group> to perform this action.");
It would then simply be a matter of modifying the code to replace the variables sections of the string:
StringBuilder stringBuilder = new StringBuilder(134);
stringBuilder.AppendFormat(
    @"You do not have the correct permissions to ");
stringBuilder.AppendFormat(
    @"perform {0}. You must be a member of ", action);
stringBuilder.AppendFormat(
    @"the {0} to perform this action.", group);
Smart Paster is a time-saving add-in that eliminates a lot of the busy work associated with working with strings in Visual Studio. It was written by Alex Papadimoulis and is available from weblogs.asp.net/alex_papadimoulis/category/5341.aspx.

CodeKeep

Throughout the process of software development, it is common to reuse small snippets of code. Perhaps you reuse an example of how to get an enum value from a string or a starting point on how to implement a certain pattern in your language of choice.

Visual Studio offers some built-in functionality for working with code snippets, but it assumes a couple of things. First, it assumes that you are going to store all of your snippets on your local machine, so if you switch machines or move jobs you have to remember to pack up your snippets and take them with you. Second, these snippets can only be viewed by you. There is no built-in mechanism for sharing snippets between users, groups, or the general public.

This is where CodeKeep comes to the rescue. CodeKeep is a Web application that provides a place for people to create and share snippets of code in any language. The true usefulness of CodeKeep is its Visual Studio add-in, which allows you to search quickly through the CodeKeep database, as well as submit your own snippets.

After installing CodeKeep, you can search the existing code snippets by selecting Tools | CodeKeep | Search, and then using the search screen shown in Figure 4.

Figure 4 Searching Code Snippets with CodeKeep
Figure 4 Searching Code Snippets with CodeKeep

From this screen you can view your own snippets or search all of the snippets that have been submitted to CodeKeep. When searching for snippets, you see all of the snippets that other users have submitted and marked as public (you can also mark code as private if you want to hide your bad practices). If you find the snippet you are looking for, you can view its details and then quickly copy it to the clipboard to insert into your code.

You can also quickly and easily add your own code snippets to CodeKeep by selecting the code you want to save, right-clicking, and then selecting Send to CodeKeep.This will open a new screen that allows you to wrap some metadata around your snippet, including comments, what language it is written in, and whether it should be private or public for all to see.

Whenever you write a piece of code and you can imagine needing to use it in the future, simply take a moment to submit it; this way, you won't have to worry about managing your snippets or rewriting them in the future. Since CodeKeep stores all of your snippets on the server, they are centralized so you don't have to worry about moving your code from system to system or job to job.

CodeKeep was written by Arcware's Dave Donaldson and is available from www.codekeep.net.

PInvoke.NET

P/Invoke is the process used to access native Win32 API calls within the .NET Framework. One of the hard parts of using P/Invoke is determining the method signature you need to use; this can often be an exercise in trial and error. Sending incorrect data types or values to an unmanaged API can often result in memory leaks or other unexpected results.

PInvoke.NET is a wiki that can be used to document the correct P/Invoke signatures to be used when calling unmanaged Win32 APIs. A wiki is a collaborative Web site that anyone can edit, which means there are thousands of signatures, examples, and notes about using P/Invoke. Since the wiki can be edited by anyone, you can contribute as well as make use of the information there.

While the wiki and the information stored there are extremely valuable, what makes them most valuable is the PInvoke.NET Visual Studio add-in. Once you have downloaded and installed the PInvoke.NET add-in, you will be able to search for signatures as well as submit new content from inside Visual Studio. Simply right-click on your code file and you will see two new context items: Insert PInvoke Signatures and Contribute PInvoke Signatures and Types.

Figure 5 Using PInvoke.NET
Figure 5 Using PInvoke.NET

When you choose Insert PInvoke Signatures, you'll see the dialog box shown in Figure 5. Using this simple dialog box, you can search for the function you want to call. Optionally, you can include the module that this function is a part of. Now, a crucial part of all major applications is the ability to make the computer Beep. So I will search for the Beep function and see what shows up. The results can be seen in Figure 6.

Figure 6 Finding the Beep Function in PInvoke.NET
Figure 6 Finding the Beep Function in PInvoke.NET

The results display a summary of what the method does (in this case, "Generates simple tones on the speaker"). You can also see code signatures for both C# and Visual Basic .NET. The wiki suggests alternative managed APIs, letting you know that there is a new method System.Console.Beep in the .NET Framework 2.0.

There is also a link at the bottom of the dialog box that will take you to the corresponding page on the wiki for the Beep method. In this case, that page includes documentation on the various parameters that can be used with this method as well as some code examples on how to use it.

After selecting the signature you want to insert, click the Insert button and it will be placed into your code document. In this example, the following code would be automatically created for you:

[DllImport("kernel32.dll", SetLastError=true)]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool Beep(
    uint dwFreq, uint dwDuration);
You then simply need to write a call to this method and your computer will be beeping in no time.

The PInvoke.NET wiki and Visual Studio add-in take away a lot of the pain and research time sometimes involved when working with the Win32 API from managed code. The wiki can be accessed at www.pinvoke.net, and the add-in can be downloaded from the Helpful Tools link found in the bottom-left corner of the site.

VSWindowManager PowerToy

The Visual Studio IDE includes a huge number of different Windows, all of which are useful at different times. If you are like me, you have different window layouts that you like to use at various points in your dev work. When I am writing HTML, I like to hide the toolbox and the task list window. When I am designing forms, I want to display the toolbox and the task list. When I am writing code, I like to hide all the windows except for the task list. Having to constantly open, close, and move windows based on what I am doing can be both frustrating and time consuming.

Visual Studio includes the concept of window layouts. You may have noticed that when you start debugging, the windows will automatically go back to the layout they were in the last time you were debugging. This is because Visual Studio includes a normal and a debugging window layout. Wouldn't it be nice if there were additional layouts you could use for when you are coding versus designing? Well, that is exactly what VSWindowManager PowerToy does.

After installing VSWindowManager PowerToy, you will see some new options in the Window menu as shown in Figure 7.

Figure 7 VSWindowManager Layout Commands
Figure 7 VSWindowManager Layout Commands

The Save Window Layout As menu provides commands that let you save the current layout of your windows. To start using this power toy, set up your windows the way you like them for design and then navigate to the Windows | Save Window Layout As | My Design Layout command. This will save the current layout. Do the same for your favorite coding layout (selecting My Coding Layout), and then for up to three different custom layouts.

VSWindowManager will automatically switch between the design and coding layouts depending on whether you are currently viewing a designer or a code file. You can also use the commands on the Apply Window Layout menu to choose from your currently saved layouts. When you select one of the layouts you have saved, VSWindowManager will automatically hide, show, and rearrange windows so they are in the exact same layout as before.

VSWindowManager PowerToy is very simple, but can save you a lot of time and frustration. VSWindowManager is available from workspaces.gotdotnet.com/vswindowmanager.

WSContractFirst

Visual Studio makes creating Web services deceptively easy You simply create an .asmx file, add some code, and you are ready to go. ASP.NET can then create a Web Services Description Language (WSDL) file used to describe behavior and message patterns for your Web service.

There are a couple problems with letting ASP.NET generate this file for you. The main issue is that you are no longer in control of the contract you are creating for your Web service. This is where contract-first development comes to the rescue. Contract-first development, also called contract-driven development, is the practice of writing the contract (the WSDL file) for your Web service before you actually write the Web service itself. By writing your own WSDL file, you have complete control over how your Web service will be seen and used, including the interface and data structures that are exposed.

Writing a WSDL document is not a lot of fun. It's kind of like writing a legal contract, but using lots of XML. This is where the WSContractFirst add-in comes into play. WSContractFirst makes it easier to write your WSDL file, and will generate client-side and server-side code for you, based on that WSDL file. You get the best of both worlds: control over your contract and the rapid development you are used to from Visual Studio style services development.

The first step to using WSContractFirst is to create your XML schema files. These files will define the message or messages that will be used with your Web services. Visual Studio provides an easy-to-use GUI interface to define your schemasthis is particularly helpful since this is one of the key steps of the Web service development process. Once you have defined your schemas you simply need to right-click on one of them and choose Create WSDL Interface Description. This will launch the Generate WSDL Wizard, the first step of which is shown in Figure 8.

Figure 8 Building a WSDL File with WSContractFirst
Figure 8 Building a WSDL File with WSContractFirst

Step 1 collects the basics about your service including its name, namespace, and documentation. Step 2 allows you to specify the .xsd files you want to include in your service. The schema you selected to launch this wizard is included by default. Step 3 allows you to specify the operations of your service. You can name the operation as well as specify whether it is a one-way or request/response operation. Step 4 gives you the opportunity to enter the details for the operations and messages. Step 5 allows you to specify whether a <service> element should be created and whether or not to launch the code generation dialog automatically when this wizard is done. Step 6 lets you specify alternative .xsd paths. Once the wizard is complete, your new WSDL file is added to your project.

Now that you have your WSDL file there are a couple more things WSContractFirst, can do for you. To launch the code generation portion of WSContractFirst, you simply need to right-click on your WSDL file and select Generate Web Service Code. This will launch the code generation dialog box shown in Figure 9.

Figure 9 WSContractFirst Code Generation Options
Figure 9 WSContractFirst Code Generation Options

You can choose to generate a client-side proxy or a service-side stub, as well as configure some other options about the code and what features it should include. Using these code generation features helps speed up development tremendously.

If you are developing Web services using Visual Studio you should definitely look into WSContractFirst and contract-first development. WSContractFirst was written by Thinktecture's Christian Weyer and can be downloaded from www.thinktecture.com/Resources/Software/WSContractFirst/default.html.

VSMouseBindings

Your mouse probably has five buttons, so why are you only using three of them? The VSMouseBindings power toy provides an easy to use interface that lets you assign each of your mouse buttons to a Visual Studio command.

VSMouseBindings makes extensive use of the command pattern. You can bind mouse buttoms to various commands, such as open a new file, copy the selected text to the clipboard, or just about anything else you can do in Visual Studio. After installing VSMouseBindings you will see a new section in the Options dialog box called VsMouseBindings. The interface can be seen in Figure 10.

Figure 10 VSMouseBindings Options for Visual Studio
Figure 10 VSMouseBindings Options for Visual Studio

As you can see, you can select a command for each of the main buttons. You probably shouldn't mess around with the left and right mouse buttons, though, as their normal functionality is pretty useful. The back and forward buttons, however, are begging to be assigned to different commands. If you enjoy having functionality similar to a browser's back and forward buttons, then you can set the buttons to the Navigate.Backward and Navigate.Forward commands inVisual Studio.

The Use this mouse shortcut in menu lets you set the scope of your settings. This means you can configure different settings when you are in the HTML designer as opposed to when you are working in the source editor.

VSMouseBindings is available from www.gotdotnet.com/team/ide/#original.

CopySourceAsHTML

Code is exponentially more readable when certain parts of that code are differentiated from the rest by using a different color text. Reading code in Visual Studio is generally much easier than trying to read code in an editor like Notepad.

Chances are you may have your own blog by now, or at least have spent some time reading them. Normally, when you try to post a cool code snippet to your blog it ends up being plain old text, which isn't the easiest thing to read. This is where the CopySourceAsHTML add-in comes in to play. This add-in allows you to copy code as HTML, meaning you can easily post it to your blog or Web site and retain the coloring applied through Visual Studio.

After installing the CopySourceAsHTML add-in, simply select the code you want to copy and then select the Copy Source as HTML command from the right-click menu. After selecting this option you will see the dialog box shown in Figure 11.

Figure 11 Options for CopySourceAsHTML
Figure 11 Options for CopySourceAsHTML

From here you can choose what kind of HTML view you want to create. This can include line numbers, specific tab sizes, and many other settings. After clicking OK, the HTML is saved to the clipboard. For instance, suppose you were starting with the following code snippet inside Visual Studio:

private long Add(int d, int d2)
{
    return (long) d + d2;
}
Figure 12 HTML Formatted Code
Figure 12 HTML Formatted Code
After you select Copy As HTML and configure the HTML to include line numbers, this code will look like Figure 12 in the browser. Anything that makes it easier to share and understand code benefits all of us as it means more people will go to the trouble of sharing knowledge and learning from each other.

CopySourceAsHTML was written by Colin Coller and can be downloaded from www.jtleigh.com/CopySourceAsHtml.

Cache Visualizer

Visual Studio 2005 includes a new debugging feature called visualizers, which can be used to create a human-readable view of data for use during the debugging process. Visual Studio 2005 includes a number of debugger visualizers by default, most notably the DataSet visualizer, which provides a tabular interface to view and edit the data inside a DataSet. While the default visualizers are very valuable, perhaps the best part of this new interface is that it is completely extensible. With just a little bit of work you can write your own visualizers to make debugging that much easier.

While a lot of users will write visualizers for their own custom complex types, some developers are already posting visualizers for parts of the Framework. I am going to look at one of the community-built visualizers that is already available and how it can be used to make debugging much easier.

The ASP.NET Cache represents a collection of objects that are being stored for later use. Each object has some settings wrapped around it, such as how long it will be cached for or any cache dependencies. There is no easy way while debugging to get an idea of what is in the cache, how long it will be there, or what it is watching. Brett Johnson saw that gap and wrote Cache Visualizer to examine the ASP.NET cache.

Once you have downloaded and installed the visualizer you will see a new icon appear next to the cache object in your debug windows, as shown in Figure 13.

Figure 13 Selecting Cache Visualizer While Debugging
Figure 13 Selecting Cache Visualizer While Debugging

When you click on the magnifying glass to use the Cache Visualizer a dialog box appears that includes information about all of the objects currently stored in the ASP. NET cache, as you can see in Figure 14.

Figure 14 Cache Visualizer Shows Objects in the ASP.NET Cache
Figure 14 Cache Visualizer Shows Objects in the ASP.NET Cache

Under Public Cache Entries, you can see the entries that I have added to the cache. The entries under Private Cache Entries are ones added by ASP.NET. Note that you can see the expiration information as well as the file dependency for the cache entry.

The Cache Visualizer is a great tool when you are working with ASP.NET. It is also representative of some of the great community-developed visualizers we will see in the future. You can get Cache Visualizer from Brett's blog at blog.bretts.net.

Wrapping It Up

While this article has been dedicated to freely available add-ins, there are also a host of add-ins that can be purchased for a reasonable price. I encourage you to check out these other options, as in some cases they can add some tremendous functionality to the IDE. This article has been a quick tour of some of the best freely available add-ins for Visual Studio. Each of these add-ins may only do a small thing, but together they help to increase your productivity and enable you to write better code.

James Avery runs his own .NET consulting practice, Infozerk Inc. He has written a number of books and articles, his most recent book is Visual Studio Hacks (O'Reilly, 2005). You can e-mail him at javery@infozerk.com and read his blog.


Editor's Note: The Dark Side of Being a Guru

Editor's Note
The Dark Side of Being a Guru

Back in the September 2005 issue of MSDN Magazine, we offered advice to our readers on how to become a guru. Perhaps "guru" is too strong a word, but it conveys the essence of our thoughts on the subject. Development is a unique profession, as it's performed primarily as an individual activity, yet to really be successful you almost always have to work with and contribute to a team effort.

This spirit—individualist yet supportive—is what has always driven the programming community we all belong to. We don't have to be told that we should create ways to share information; it's in our DNA. Of course, so many developers are considered experts by their peers, and they have come to that position through a variety of paths.

In talking to several of these experts, it became clear that with prominence and fame, even moderate fame, comes responsibility. One developer whose name will remain unmentioned (but whose photo is on this page) worked for a midsized software company a decade ago. He walked into a meeting with a software architect, ready to explain how the company could use the newly documented OLE controls technology to improve their product. Instead of the nodding acceptance and adoration he was looking for, he got an earful about how the technology would never work because we've all seen how it can take ten seconds to open an embedded Excel file from a Word doc. Repeated pleas of "but the controls are in-process, not out-of-process" were ignored as the spec was tossed back in a blizzard of letter-size copy paper.

Having advanced knowledge of a particular topic definitely has drawbacks. We asked some of our authors what it's been like advising readers and interacting with members of the dev community, and most of them had good and bad experiences (see the right column).

Maybe you're a guru at work. Maybe you're asked to perform miracles. In fact, we're sure you are. Drop us a line with your stories, and maybe we'll talk about them in a future issue.

—J.T.

"A downside of being an expert in testing is receiving e-mail messages along the lines of, 'mee name is Qzwhtrty and my program it did not working what is wronmg you fix quick thank you very much please.' At the other end of the spectrum are messages like, 'Your supercilious use of the term 'lexicographic' rather than the more canonically correct 'lexicographical' is in counter-juxtaposition to the fifth law of thermodynamics as expostulated by the ECMA799 section 3.1.8.2a.'"
—James McCaffrey, columnist

"I receive a ton of e-mail saying, 'I have a software problem. Please help soon, as I have urgent business to conduct."
—Dino Esposito, columnist

"The thing that makes me most uncomfortable is when people approach me at a show with a scrap piece of paper for me to sign. What am I supposed to write? Should I do like in the movies and ask them what they want it to say? The pressure. I usually serialize some well-formed XML onto their paper, which seems most appropriate."
—Aaron Skonnard, columnist

"I need to check, double-check, and triple-check something before stating it in public, because I need to really understand what I'm saying."
—Christophe Nasarre, contributor

"At PDC two years ago, Jeff Richter and I were walking out of the convention center. A couple of guys came running up, and one of them asked Richter to sign his book. When the guy handed him the book, it was MY book. Richter didn't miss a beat. He opened the cover, signed "Jeff Prosise," and handed it back. People confuse us a lot."
—Jeff Prosise, columnist

"You want to help everyone who asks for aid, answer all of their questions, write that crucial bit of code for which they're searching, and be the hero who saves the day (and their job), but there comes a point when you realize that sleep is sometimes necessary."
—Stephen Toub, columnist and technical editor par excellence

"What—me well-known?"
—Paul DiLascia, columnist

Thanks to the following Microsoft technical experts for their help with this issue: Manoj Biswas, Shawn Burke, Mike Clark, Kit George, Eliot Graff, Andy Harjanto, Brian Johnson, Brian Keller, Ronald Laeremans, Steven Lees, Bradley Millington, Craig Skibo, Dave Templin, Weiqing Tu, Maura Van Der Linden, and Scott Woodgate.

Active Directory, BizTalk DirectX, Excel, InfoPath, JScript, Microsoft, Microsoft Press MSDN, Outlook, Visio Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Win32, Windows NT, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Marnie Hutcheson

Contents
Analysis and Reporting Solutions
View, Print, and Markup Formats
Localize Your Apps
Create Help Files
The Bookshelf

Analysis and Reporting Solutions

Databeacon Smart Client software allows users to perform data analysis, turning relational data from any data source into online analytical processing (OLAP) cubes that can be explored and manipulated using one of three Databeacon viewers.

The simplest viewer is Databeacon Web Reporter. Next is Databeacon Player, which lets you not only view reports but also explore the data cube more completely. You can isolate a particular group of data, drill down, view DataSets, modify filters, and even save your own flavor of a report. Last and most interesting is Databeacon Insight, which gives you full access to all the report creation and presentation capabilities. Insight has two main windows. The first lets you do analysis, set up data, perform drill downs, and so on. The presentation pane shows you the report that your users will see.

Databeacon Insight and Databeacon Player give you the chance to share your insights with others through static reports (in either table or chart format) or by exporting your data to Microsoft® Excel®. All Databeacon Smart Client viewers provide Microsoft Office integration that lets you print, copy and paste, and e-mail your reports.

You design and build multidimensional OLAP cubes using Databeacon Publisher, a simple and intuitive GUI application that gives you access to the Client OLAP Reporting Environment (CORE). Databeacon Insight is then used to create reports that can be viewed using any of the three Databeacon viewers.

Databeacon Smart Client gives you the three viewers plus Publisher. All three viewers take advantage of .NET-based smart client technology so they can be embedded in a Web page. They install and run automatically as long as the Microsoft .NET Framework is running on an appropriately configured client machine.

Price: $899 per single license.

www.databeacon.com

View, Print, and Markup Formats

Brava! Desktop v1.2, by Informative Graphics, is a multiformat graphics viewer that lets you view, print, and markup formats such as TIFF, CALS, PDF, CGM and HPGL plot files, and native formats from Microsoft Office applications, Visio®, AutoCAD, MicroStation, and SolidWorks.

Brava Desktop also includes the company's own Content Sealed Format (CSF), which gives you a new alternative for content distribution that has several advantages over PDF, which can now be edited, converted, or changed using third-party software. Secure CSF files saved by Brava Desktop or published by IGC publishing products can be viewed with the free Brava Reader, which also views PDF and TIFF files.

Brava Desktop doesn't just let you scroll through page images and CAD drawings; you can also manipulate a document's built-in attributes. For example, in a multilayer document, you select the layers you want to view, and use the measuring tool to pick dimensions off the drawing (even with a scanned document).

Other cool features new in version 1.2 include Dynamic Compare which can overlay two images and highlight the differences and the ability to flip between two drawings or image revisions to compare differences. It also does redaction, which lets you block confidential areas, words, and phrases from being viewed, printed, and searched on in your documents. You can save any document, image, or CAD drawing format type to PDF. It even saves your markup as Adobe annotation elements.

Price: $345 per single license.

www.bravaviewer.com

Localize Your Apps

Lingobit Localizer version 4.0 supports more than 180 different languages, including various European, Asian, and right-to-left languages.

The Translate Expert helps you to ensure that your application is ready for localization. For example, it examines source code and resource separation, and it will check to see if your code crashes when strings are translated. The Validation Expert recognizes several types of common mistakes like inconsistent format strings, duplicate hotkeys, missing periods, and so on. The Statistics Expert helps you track your progress and pinpoint potential problem areas.

The Professional version includes these Expert features and is probably all you will need for small localization projects. The Enterprise version targets large projects with numerous target languages. It leverages the Experts extensively and includes several additional tools. Both versions come with the Lingobit Localizer Translator and aim to help you "never translate the same string twice."

Price: $895 per single license

(Professional Edition).

www.lingobit.com

Create Help Files

Dr. Explain, by Cognitive Force, presents an automated approach to creating very comprehensive help files quickly. Dr. Explain captures windows from a live Windows®-based application and creates screenshots as you would expect, but then it analyzes the window content, enumerates the screens, and attaches numbered callouts to all the significant elements.

All captured windows are stored in a single project file. You can group them by modules, and when something changes, you can update individual windows, add new screenshots, and update control references. You don't need to recreate the whole sequence of windows.

With Dr. Explain, you can customize the structure of the menus, screen layout, colors, and callout details. Then you just export the project to HTML. You get instant online help complete with index page, menus, hyperlinked callouts, and professional screens all formatted and ready to upload to your server.

Price: $79 per single license.

www.drexplain.com

The Bookshelf

ASP.NET 2.0 Upgrader's Guide: C# Edition (Murach), by Doug Lowe and Joel Murach, focuses not only on what's new in ASP.NET 2.0 (that's the easy part) but, more importantly, it shows you how and when you can use these new features in the real world.

This book is laid out in an unusual but helpful way. The left hand page presents an explanation and the right hand page offers you guidelines, examples, screen shots, and how-to notes. This makes it very easy to find what you are looking for,understand it, and decide whether its right for your application.

Each chapter deals with a key feature in ASP.NET 2.0 like the SQL Server™ and XML data sources; the GridView, DetailView, and FormView controls; login and site navigation controls; and tools for configuring and deploying new apps.

Price: $39.50, 526 pages.

www.murach.com

The Windows Presentation Foundation of the upcoming Windows Vista™ operating system is the graphics subsystem for building vector-based user interfaces. This new approach to UI creation and presentation will let you take advantage of the graphics capabilities built into the latest video cards, and simplifies UI creation with XML support.

In Programming Windows Presentation Foundation: Building Windows XP/Vista User Interfaces (O'Reilly), authors Chris Sells and Ian Griffiths provide a comprehensive look at all the new technologies inside the product. These include an engine that supports 3D graphics and animation, a new XML-based markup language called XAML, and a radical new model for customizing controls—the text boxes, buttons, and listboxes that constitute the building blocks of a user interface. By page two of the book, you will have written your first Windows Presentation Foundation app, and by the end of the first chapter you will be familiar with much of the new framework.

The result is a platform that unifies several services—including 2D and 3D drawing and imaging, document-based rendering, and audio and video services—that developers can use to build rich UIs for both standalone applications and applications blended with a Web site within the browser.

The book introduces you to the powerful set of tools for managing the visual layout of your apps, describes Windows Presentation Foundation features for connecting the UI to underlying data, and shows you how to use styles and control templates to customize UI appearance. There are also chapters on using drawing tools, writing custom controls, and using the framework's animation facilities.

Sells and Griffiths provide plenty of C# code and examples of the new eXtensible Application Markup Language (XAML) for declaring the structure of a Windows Presentation Foundation user interface. The book also includes insightful discussions on the powerful new programming styles, along with a comparison of the features that support interoperability with Windows Forms and other Windows-based legacy applications.

Price: $39.95, 430 pages.

www.oreilly.com

Visual Studio Tools for Office: Using C# with Excel, Word, Outlook, and InfoPath (Addison Wesley Professional), by Eric Carter (who also happens to have created the software) and Eric Lippert, gives readers the first look inside programming with this new toolset. Using Visual Studio® Tools for Office, you can program Microsoft Excel 2003, Word 2003, Outlook® 2003, and InfoPath® 2003 to create your own custom Office-based solutions. Data/view separation, design-time views of Excel and Word documents inside Visual Studio, and rich support for Windows Forms controls in a document are just some of the powerful features the authors explain.

You'll come away with an understanding of the architecture of Microsoft Office programming and the object models behind it, Office automation and add-ins, and the code behind a document. For more advanced coverage, the authors present server data scenarios and explore .NET code security and Visual Studio Tools for Office deployment. They also show the reader how to develop COM add-ins for Word and Excel, and create Outlook add-ins as well.

Price: $49.99, 1008 pages.

www.aw-bc.com

Send your New Stuff to newstuff@microsoft.com.

All prices were confirmed at press time and are subject to change.

Marnie Hutcheson is president of Internet Development Associates (Ideva), a firm in Ocala, Florida, that specializes in Internet and intranet Web application design and development. She has published technical papers and books on various computing topics. You can reach her at marnie@ideva.com.


Advanced Basics: What's My IP Address?

Advanced Basics
What's My IP Address?
Ken Getz

Code download available at: AdvancedBasics0512.exe (154KB)
Contents
Dealing with the Issues
Retrieving the IP Address
Creating the User Interface
Handling Data Binding

If you're like me, you regularly do tech-support for family, friends, and neighbors. You can't go to a party without hearing the familiar refrain: "I've just got a quick question." It's always something—their Internet connections get dropped, they've got a virus, they can't install some piece of hardware, or some file has gone missing.

Maybe you support your brother or your grandmother. In my case, I spend a good deal of time helping out my father, who is astoundingly agile with his computer considering he didn't pick it up until he was 65. When he has trouble, it would be convenient if I could use Remote Desktop Connection to connect into his computer and see what's going on. Of course, connecting to his computer would require that I know his IP address. That would be simple if he wasn't using Network Address Translation (NAT) behind a firewall, and if he didn't get a dynamic IP address from his ISP. I could have him run a command prompt, get his IP address, and read it to me, but that address would be a useless internal address. I could have him browse to one of the many sites that reports the external (outward-facing) IP address, such as www.WhatIsMyIP.com, and have him read me the address. But attempts to perform this exercise have been frustrating, resulting in misplaced digits, missing dots, and incorrect addresses.

It would be best if he could click a button and have his IP address e-mailed to me. That's the goal of this column. (Of course, one could also use Remote Assistance to make the process easier, but that wouldn't make for a very interesting column, now would it)? Follow along as I create a Windows®-based application that retrieves the IP address of a remote machine and e-mails it to the appropriate mailbox. Along the way, you'll learn a few handy bits of information about ASP.NET Server variables. If you're already using ASP.NET, you should be comfortable with this information, but I'll bet anyone moving from Visual Basic® 6.0 to Visual Basic .NET isn't familiar with the information that's available to a Web site.

Dealing with the Issues

To begin, I created the Web service and got it running, and so can you. Your best bet is to follow the instructions here to create your own, so that you can control its behavior and availability.

Next, I created the Windows-based application that calls the Web service, and added functionality to retrieve the IP address and e-mail it to a specified address. Of course, making it into a usable application required a little more effort. As with many weekend software projects, getting the application working is only about 40 percent of the problem (an arbitrary value, if there ever was one). The fit and finish requires the bulk of the effort. To be honest, I didn't expend much energy on error handling and reporting; that wasn't the point. If something goes wrong, the application simply doesn't do what it's supposed to do. Figure 1 shows the application in action.

Figure 1 Call the Service, Display the IP, and Send E-Mail
Figure 1 Call the Service, Display the IP, and Send E-Mail

Although the application supplies default values for the sending and receiving e-mail address, SMTP server, and message fields, a user might want to supply different values. Of course, it would be great if the application remembered the user-supplied values on subsequent invocations. In order to preserve these values, the sample application takes advantage of two new features in Visual Studio® 2005: the My.Settings class and object data binding. I'll describe how the sample uses both features later in the column.

Retrieving the IP Address

Getting the plumbing working on this project is simple: each time you make a Web request, the HTTP headers passed from the client to the Web server include information about the request, including your public Web address. That allows the Web server to respond back to the correct address. The challenge is to find a way to retrieve that header information, and return it to a client application. The answer, of course, is to create a Web service that returns the IP address of the request. This task, in fact, is amazingly simple.

The HttpRequest instance that ASP.NET makes available to you from a Web page or a Web service makes it possible to retrieve all the information you need. From within a Web page—actually within a class that inherits from System.Web.UI.Page—you can refer to Me.Request for access to this instance. In a Web service that inherits from System.Web.Services.WebService, you can use Me.Context.Request to get at the same instance. And regardless of the type of server-side code you're writing, be it a page or a service or an arbitrary handler, you can always go directly to the HttpContext.Current.Request. The HttpRequest.ServerVariables property exposes a name/value collection that contains one item for each of the well-known server variables provided by IIS and ASP.NET. Within this collection, you can retrieve the address of the requesting page, and can easily create a method that returns the value on demand.

The HttpRequest.ServerVariables collection contains all sorts of information about the current request, the client browser, the server, and more. In order to see what's in the collection, the simplest solution is to create a Web page that displays the information for you (there are many such sites available on the Web—search on "ASP Server Variables" and you'll find some). To investigate this useful collection, create a new Web application, using either Visual Studio .NET 2003 or Visual Studio 2005. In the default page's code, simply add the following procedure:

Private Sub ListServerVariables()
    For Each key As String In Request.ServerVariables.Keys
        Response.Write(String.Format("<b>{0}</b> = {1}<br>", _
            key, Request.ServerVariables(key)))
    Next
End Sub

Finally, add code to the Page_Load event handler to call the ListServerVariables procedure. Although you can test out the page on your local computer, the information displayed by the page will be more interesting if you can deploy the page to an external server—otherwise, you're only looking at information about your computer, as opposed to information about your network and the server itself. (You're welcome to try the public version I've posted at ListServerVariables, but again I can't guarantee the availability of the site.) Figure 2 shows output from running the sample code; your results will be different, of course.

Figure 2 Using the HttpRequest.ServerVariables Collection
Figure 2 Using the HttpRequest.ServerVariables Collection

My goal was to create a programmatic entity that can return the IP address of the caller, and that, in turn, requires creating a Web service. To do that, you can create a new Web service project in either Visual Studio .NET 2003 or Visual Studio 2005. I named my project WhatIsMyIP, and I named the Web service (the .asmx page) AddressInfo. Within the Web service code, add the procedure shown in Figure 3.

Figure 3 Get Address (Close Figure 3)
<WebMethod()> _
Public Function GetAddress() As String
  ' Send exceptions back to the caller.

  Dim req As HttpRequest = HttpContext.Current.Request

  ' See if the address has been forwarded through a proxy server:
  Dim addr As String = _
      req.ServerVariables("HTTP_X_FORWARDED_FOR")
  ' If not, then get the real remote address:
  If addr = String.Empty Then
      addr = req.ServerVariables("REMOTE_ADDR")
  End If

  ' Return the address:
  Return addr
End Function

Although retrieving just the REMOTE_ADDR server variable should be enough, I found resources online that suggested that code like this should also check the HTTP_X_FORWARDED_FOR variable; if the request comes through a proxy server that translates the address, it's this variable that contains the correct address. If you request a server variable that doesn't exist, the ServerVariables property returns an empty string. Therefore, even though this property doesn't appear in my tests, attempting to retrieve its value doesn't cause trouble.

You'll need to deploy the Web service to a public site (outside your network) in order to test the service. To try it out, create a Windows-based application, set a Web reference to the Web service (in the code samples, I've renamed this reference to GetIP), and call the GetAddress method, like this:

' In Visual Studio .NET 2003 or Visual Studio 2005
Dim ip As New GetIP.AddressInfo
MessageBox.Show(ip.GetAddress())

' Or in Visual Studio 2005
MessageBox.Show(My.WebServices.AddressInfo.GetAddress())
If you've set up all the plumbing correctly, the application displays your current external IP address, and the information provides the basis for the remainder of this column.

Creating the User Interface

As you've seen, calling the Web service to retrieve the IP address requires no real effort at all. In order to provide my father with a simple-to-use front end, I fired up Visual Studio 2005 and created a new Windows application. (I used Visual Studio 2005 so I could take advantage of the new My.Settings and object binding features—these additions make it easy to retrieve, display, and store user settings without having to worry about where and how to store them.) I modified the default form so that it looked like Figure 1, and added a Web reference to the Web service I discussed previously. Rather than walk you through the creation of this application, I'll focus on the finished solution instead.

Clicking the Retrieve button runs the code in Figure 4, which displays the resulting IP address in the label on the form, and enables the group box containing e-mail information.

Figure 4 Retrieve Button (Close Figure 4)
Private Sub retrieveButton_Click( _
    ByVal sender As System.Object, _
    ByVal e As System.EventArgs) _
       Handles retrieveButton.Click

    Dim results As String = "Unable to retrieve your IP address."

    Try
        Me.Cursor = Cursors.WaitCursor
        address = My.WebServices.AddressInfo.GetAddress()

        If Not String.IsNullOrEmpty(address) Then
            ' You got a result. Set up the display:
            results = "Your IP address is: " & address
            emailGroupBox.Enabled = True
        End If
    Catch ex As Exception
        ' Don't complain.
    Finally
        Me.Cursor = Cursors.Default
    End Try

    ' Display the returned address:
    addressLabel.Text = results
End Sub

When it comes time to send the e-mail with your IP address, the application uses the values in the To and From textboxes to address the e-mail. The body of the message contains your IP address, along with the custom information contained within the Message textbox. In order to send the e-mail, you must supply the name or IP address of the SMTP server. These values are required, and the Send button can't be enabled until all three are provided and the e-mail addresses are in the correct format.

The application includes the procedure in Figure 5, which handles the TextChanged event for each of the required textboxes. This procedure verifies that the necessary values have been supplied and that the e-mail addresses are in the correct format. The code only enables the Send button when all the conditions have been met.

Figure 5 Handling TextChanged (Close Figure 5)
Private Sub HandleTextChanged(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) _
        Handles toTextBox.TextChanged, fromTextBox.TextChanged, _
        smtpTextBox.TextChanged

    ' Verify that both from and to are real email addresses:
    Dim rx As New Regex( _
        "\w+([-+.]\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*")
    sendButton.Enabled = rx.IsMatch(toTextBox.Text) And _
        rx.IsMatch(fromTextBox.Text) And _
        (smtpTextBox.Text <> String.Empty)

    ' Indicate with a label if the required field hasn't
    ' been supplied. You could make this user interface a bit
    ' more informative. What if the user supplies an invalid
    ' email address?
    toRequiredLabel.Visible = (toTextBox.Text = String.Empty)
    fromRequiredLabel.Visible = (fromTextBox.Text = String.Empty)
    smtpRequiredLabel.Visible = (smtpTextBox.Text = String.Empty)
End Sub

The form's Load event handler disables the Send button and the e-mail information group box:

sendButton.Enabled = False
emailGroupBox.Enabled = False

Sending the e-mail doesn't require much code, and the procedure in Figure 6 takes advantage of the new System.Net.Mail namespace. Except for storing and retrieving user settings, there's no more code in the sample.

Figure 6 Sending the Mail (Close Figure 6)
Private Sub sendButton_Click( _
    ByVal sender As System.Object, ByVal e As System.EventArgs) _
        Handles sendButton.Click

    Try
        Dim smtp As New System.Net.Mail.SmtpClient(smtpTextBox.Text)
        smtp.Send(fromTextBox.Text, toTextBox.Text, _
            "IP Address Information", _
            "My IP address is: " & address & Environment.NewLine & _
            messageTextBox.Text)
        MessageBox.Show("Mail sent!", Me.Text)
    Catch ex As Exception
        MessageBox.Show("Unable to send email: " & ex.Message)
    End Try
End Sub

Handling Data Binding

Using the new features provided by Visual Basic 2005, retrieving, binding, and storing changes to user values takes only a few lines of code, as you'll see. In order to store the four text values, I created four settings in my project. To try this yourself, select the Project | Your Project Properties menu item to display the project properties, and then select the Settings tab. Figure 7 shows the settings for the sample project. Note that each setting has its Scope set to User (as opposed to Application). Without this change, the application can't modify the settings on a per-user basis. Most of the settings for this application also contain a default value, so that the form can prompt the user for input.

Figure 7 Set Up User Settings (Close Figure 7)

Once you've created the settings and closed the designer, Visual Studio creates a class that exposes these settings to your application as properties of the My.Settings class. (To see the class, select the Project | Show All Files menu item, and expand the My Project | Settings.settings node to find the Settings.designer.vb class.)

With the settings class created, the next step is to bind the properties of the My.Settings class to controls on the form. Although the sample application has done this already, you can try it yourself. First, create the settings as shown in Figure 7. Next, create a form containing four textboxes, as shown in Figure 1. To perform the data binding visually, start by selecting the Data | Add New Data Source menu item. In the Choose a Data Source type page, select Object, and then click Next.

On the page where you can select the object you want to bind to, expand the project node until you find the Settings node. Click Finish to accept the Settings object as your data source. Once you're finished, you'll see the Data Sources window including the Settings class (and your settings) as an available data source (see Figure 8). An instance of the System.Windows.Forms.BindingSource class named SettingsBindingSource will also be added to the tray area of your form. You'll interact with this object when you want to control the data binding in your code.

Figure 8 New Data Source
Figure 8 New Data Source

To hook up the data binding for the existing form, you can take advantage of a feature the Visual Basic team calls "connect-the-dots data binding." That is, you can drag fields from the Data Sources window and drop them onto controls that exist on your form—doing this sets up the appropriate data binding for each control. This action has already been taken on the sample form, but on your form, you can click and drag each field from the Data Sources window to the matching textbox. Once you've done this, select one of the textboxes and examine the DataBindings property in the control's properties. You should see information indicating that the control has been bound to the correct property of the Settings class.

All that's left, then, is to add the two lines of code that hook up the data binding and save the values when you close the form. In the form's Load event handler, you can add the following code:

' Set up the data binding.
SettingsBindingSource.DataSource = My.Settings

After you've done that, in the form's Closed event handler, add the following line of code:

My.Settings.Save()

That's all it takes! Now, when you load the form, you'll see the default values for the settings and when you close the form, your code will save the settings. Finally, when you reopen the form, your saved settings should appear in the form's controls.

Without data binding, you would have been required to store the values somewhere on your own (I originally wrote this example using isolated storage, serializing values from a class I had created) and display and retrieve the values on the form. I'm sure you'll agree that between the My.Settings class and object data binding, the process is a lot simpler.

In fact, Visual Basic 2005 stores user settings much as it might handle isolated storage. Figure 9 shows the location, on my test computer, of the settings for the sample project. Clearly, Visual Basic is doing as good a job as you could possibly do in hiding this information in a safe place. Remember, just as with isolated storage, the goal here is simplicity, not security. Although another user cannot see the current user's data, the user can always browse through the file system and view the information in the user.config file. Don't use this technique to store information you don't want the current user to see!

Figure 9 Visual Basic 2005 Stores Saved Settings (Close Figure 9)

After deploying the Microsoft® .NET Framework 2.0, creating the Web service was really simple and the client application needed almost no code.

Though my intended purpose was to show you how to create an application that can retrieve and e-mail the outward-facing IP address for any internal computer, it was also a good opportunity to look at the My.Settings class and the simple techniques required to perform object data binding. You'd be surprised how much code I removed when moving this application from Visual Basic .NET 2003 (using my own class, serialized into isolated storage) to Visual Basic 2005, using the available new features. Clearly, it's worth rethinking existing development techniques as you adopt Visual Basic 2005.

Send your questions and comments to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies, and a courseware author for AppDev. He is coauthor of ASP.NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2002), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Test Run: Software Testing Paradoxes

Test Run
Software Testing Paradoxes
James McCaffrey

Code download available at: TestRun0512.exe (120KB)
Contents
Simpson's Paradox
Braess's Paradox
Parrondo's Paradox

Paradoxes are fun. In this month's column I show you three interesting cases that can occur when you are performing software testing. They're fundamentally mathematical in nature, and they can be a useful addition to your troubleshooting arsenal.

In the first part of this column, I explain Simpson's Paradox. This is the situation when Software System A is worse in every area of comparison than Software System B, yet System A can still be shown to be the better system. In the second part, I demonstrate Braess's Paradox, which is when adding a perfectly good load-balancing server actually reduces network performance in a most surprising way. I finish up with a look at Parrondo's Paradox, where two completely independent but faulty systems can, incredibly, produce a correct system.

Simpson's Paradox occurs quite frequently. Although the chances of you actually encountering Braess's or Parrondo's Paradoxes are slim, I think you'll find reading about them quite entertaining.

Simpson's Paradox

Imagine you have two different software system prototypes: Prototype A and Prototype B. You want to evaluate which prototype is better based on user feedback. Suppose that for ease-of-use, Prototype A received an excellent evaluation by 50 out of 200 people, while Prototype B received excellent from 15 out of 100 people (see Figure 1). From this data, Prototype A is clearly superior to Prototype B in terms of ease-of-use, with Prototype A getting 25 percent excellent and Prototype B getting just 15 percent.

Figure 1 Comparing Prototype A and Prototype B (Close Figure 1)

Ease of Use Data | Prototype A | Prototype B
Number excellent scores | 50 | 15
Total users | 200 | 100
Score | 25% | 15%
Security Data | Prototype A | Prototype B
Number excellent scores | 85 | 300
Total users | 100 | 400
Score | 85% | 75%

Now, suppose that for security features, Prototype A receives a rating of excellent from 85 out of 100 people, while Prototype B receives excellent from 300 out of 400 people. From this data, you can see that Prototype A (with 85 percent excellent) is again clearly superior to Prototype B (with just 75 percent excellent). So, which prototype do you conclude is better? Before you answer, let's combine the data from Figure 1 into a single table as shown in Figure 2.

Figure 2 Combined Data for Security and Ease-Of-Use (Close Figure 2)

Combined Data | Prototype A | Prototype B
Number excellent scores | 135 | 315
Total users | 300 | 500
Score | 45% | 63%

If you combine the number of excellent evaluations, you see that Prototype A receives 135 out of 300, which is just 45 percent excellent. Meanwhile, Prototype B dominates with 63 percent excellent, based on its 315 out of 500. So Prototype B is superior, right? Or is Prototype A superior? And if it is, how can you explain away the combined data?

This is an example of Simpson's Paradox and it is a real phenomenon. There is no trick involved. The correct answer is that Prototype A is the better system; you should not combine the data together into a single table, as I've done in Figure 2. A brief, informal description of Simpson's Paradox is: a situation in which two or more sets of data lead to one conclusion when evaluated individually, but lead to an opposite conclusion when the sets are combined. If you look back at the data in Figure 1, you'll notice that the total number of evaluations for each prototype is different in each column. This is a necessary condition for Simpson's Paradox.

The preceding example is a bit artificial, but it illustrates Simpson's Paradox quite clearly. The important point is that Simpson's Paradox does actually happen, even with real-world data, and it is an occurrence you should be on the alert for. The paradox can easily sneak by you, especially if you are only presented with summary data and don't get a chance to see the original, uncombined data. Here is an example where the numbers are made up, but the scenario is based on an actual incident. The actual example appeared in "Sex Bias in Graduate Admissions: Data from Berkeley" by P.J. Bickel, E.A. Hammel, and J.W. O'Connell (1975).

Consider the graduate school admission data for a university, shown in Figure 3. The data shows that 4000 of 9000 males (44.4 percent) who applied to the university were admitted for graduate study, but only 1500 of 4500 females (33.3 percent) were admitted. Is this a case of some sort of gender inequity? Not necessarily. The data in Figure 3 represents combined admissions for four departments in the university. Take a look at Figure 4, which represents the data for each individual department. With these figures, you find that women were admitted at a higher rate than men in every case! Clearly the uncombined data in Figure 4 presents a better picture of admissions than the combined data in Figure 3. Even though this data is significantly simplified from the real incident, you can see that Simpson's Paradox can be quite difficult to spot.

Figure 4 Uncombined Graduate School Admissions Data (Close Figure 4)

 | Men |  |  | Women
Department | Admitted | Rejected | Acceptance Rate | Admitted | Rejected | Acceptance Rate
A | 2000 | 2400 | 45% | 400 | 450 | 47%
B | 1200 | 1000 | 55% | 100 | 80 | 56%
C | 700 | 900 | 44% | 600 | 730 | 45%
D | 100 | 700 | 13% | 400 | 1740 | 19%
Total | 4000 | 5000 |  | 1500 | 3000 |
Figure 3 Combined Graduate School Admissions Data (Close Figure 3)

 | Admitted | Rejected | Acceptance Rate
Men | 4000 | 5000 | 44.4%
Women | 1500 | 3000 | 33.3%

Simpson's Paradox is not a true paradox in the mathematical sense. A mathematical paradox produces a logically inconsistent result, whereas examples of Simpson's Paradox are unexpected and surprising, but fully explainable. There is a lot of information available on the Web about Simpson's Paradox if you want to investigate further. The moral is that when you are looking at test result data, you should first question whether the data is aggregated from other sources. If it is, you need to look at the original uncombined data. Additionally, if you are generating a test result data report, be very careful about combining the data in an attempt to simplify your presentation. When people say that "statistics can be made to say whatever you want," this is exactly the sort of situation they're referring to.

Braess's Paradox

Now let's take a look at Braess's Paradox. My example uses a traditional scenario—automobile traffic—but I also describe how the paradox maps more broadly to software testing and other situations. The original work for Braess's Paradox is "Über ein Paradox der Verkerhsplannung," by Dietrich Braess (1968). My example, however, is based upon several examples I found that were, in turn, based on examples attributed to Mark Wainwright. I was unable personally to locate Wainwright's original work.

Braess's Paradox is quite complex so I present a simplified, discrete approximation rather than the full problem. Suppose there are four towns—Town A, Town B, Town C, and Town D—as shown in Figure 5. Each road connecting any two of the towns has an associated cost given by the equation shown next to the road in the figure. The cost is a function of the number of cars (n) on the road. You can imagine that the cost represents the time it takes to travel the road, or the amount of gasoline required, or some other factor that you'd like to minimize. Now suppose that each morning there are six cars that leave Town A, one at a time, and all are headed to Town D. Car 1 leaves and has a completely open road. The car has two routes to choose from: A-B-D or A-C-D. The cost of A-B-D is [4(1) + 1] + [1 + 16] = 22. Because of the symmetry in the map, the cost of route A-C-D is also 22. Let's assume that Car 1 selects route A-B-D and starts on its way.

Figure 5 Road Network: Braess's Paradox
Figure 5 Road Network: Braess's Paradox

Now Car 2 prepares to leave. He sees Car 1 on route A-B-D and knows that route A-B-D will now cost [4(2) + 1] + [2 + 16] = 27, so he selects route A-C-D with a cost of 22. Car 3 sees that there is one car on each route and selects route A-B-D with a cost of 27. Car 4 then selects route A-C-D, which has a cost of 27. Car 5 sees the four cars evenly distributed and he decides to take route A-B-D bringing its cost to [4(3) + 1] + [3 + 16] = 32. Finally, Car 6 selects route A-C-D, which also costs 32. All six cars are now en route from Town A to Town D. Because there are three cars on each path and the two paths are symmetric, the cost for each car will be 32.

Here's where Braess's Paradox comes into play. What do you suppose will be the effect of adding a new, efficient shortcut road between Town B and Town C? Common sense says that adding more road capacity should decrease the cost to our drivers. But since this phenomenon is called "Braess's Paradox" and not "Braess's Common Sense," you can guess that's not what happens.

Suppose that the map in Figure 5 is modified with a highly efficient shortcut between Town B and Town C, with a cost function equal to a constant 1. On the first morning with the new shortcut road in place, Car 1 prepares to leave Town A. He has four reasonable routes, each with an associated cost:

A-B-D      cost = [4(1) + 1] + [1 + 16]       = 22
A-C-D      cost = [1 + 16] + [4(1) + 1]       = 22
A-B-C-D    cost = [4(1) + 1] + 1 + [4(1) + 1] = 11
A-C-B-D    cost = [1 + 16] + 1 + [1 + 16]     = 35

This looks promising. Car 1 selects route A-B-C-D and uses the new shortcut to reduce his travel cost significantly, at least for the moment. Car 2 prepares to leave. He sees Car 1 has selected route A-B-C-D and analyzes his potential costs:

A-B-D      cost = [4(2) + 1] + [1 + 16]       = 26
A-C-D      cost = [1 + 16] + [4(2) + 1]       = 26
A-B-C-D    cost = [4(2) + 1] + 1 + [4(2) + 1] = 19
A-C-B-D    cost = [1 + 16] + 1 + [1 + 16]     = 35

After some quick math, Car 2 also elects route A-B-C-D. Even though Car 1 is on that route already, the efficiency of the shortcut still makes that route the best choice.

Now Car 3 prepares to leave and considers his options:

A-B-D      cost = [4(3) + 1] + [1 + 16]       = 30
A-C-D      cost = [1 + 16] + [4(3) + 1]       = 30
A-B-C-D    cost = [4(3) + 1] + 1 + [4(3) + 1] = 27
A-C-B-D    cost = [1 + 16] + 1 + [1 + 16]     = 35

Again, the shortcut route A-B-C-D remains the best choice. Car 4 now prepares to leave Town A, observes the choices of the first three drivers and determines his costs:

A-B-D      cost = [4(4) + 1] + [1 + 16]       = 34
A-C-D      cost = [1 + 16] + [4(4) + 1]       = 34
A-B-C-D    cost = [4(4) + 1] + 1 + [4(4) + 1] = 35
A-C-B-D    cost = [1 + 16] + 1 + [1 + 16]     = 35

The traffic on the shortcut now makes route A-B-C-D more costly than routes A-B-D and A-C-D. Let's assume Car 4 selects route A-B-D. (If Car 4 takes route A-C-D, the details are slightly different but the overall result is the same).

Car 5 now prepares to leave and analyzes the options:

A-B-D      cost = [4(5) + 1] + [2 + 16]       = 39
A-C-D      cost = [1 + 16] + [4(4) + 1]       = 34
A-B-C-D    cost = [4(5) + 1] + 1 + [4(4) + 1] = 39
A-C-B-D    cost = [1 + 16] + 1 + [2 + 16]     = 36

So Car 5 decides to take route A-C-D. The last car, Car 6, now prepares to leave, observes the decisions made by the first five cars and comes up with this analysis:

A-B-D      cost = [4(5) + 1] + [2 + 16]       = 39
A-C-D      cost = [2 + 16] + [4(5) + 1]       = 39
A-B-C-D    cost = [4(5) + 1] + 1 + [4(5) + 1] = 43
A-C-B-D    cost = [2 + 16] + 1 + [2 + 16]     = 37

Car 6 chooses route A-C-B-D because it has the lowest cost. With all six cars on the road, you can determine each car's cost:

Car 1     route A-B-C-D, cost = [4(4) + 1] + 1 + [4(4) + 1] = 35
Car 2     route A-B-C-D, cost = [4(4) + 1] + 1 + [4(4) + 1] = 35
Car 3     route A-B-C-D, cost = [4(4) + 1] + 1 + [4(4) + 1] = 35
Car 4     route A-B-D,   cost = [4(4) + 1] + [1 + 16]       = 34
Car 5     route A-C-D,   cost = [2 + 16] + [4(4) + 1]       = 35
Car 6     route A-C-B-D, cost = [2 + 16] + 1 + [1 + 16]     = 36

Recall that without the shortcut road, the cost to each car was 32. By adding additional road capacity we have actually increased the cost to every driver!

This example provides a feel for the actual Braess's Paradox. Because of its obvious relation to data packets traveling on a network system, Braess's Paradox has been investigated heavily by researchers. You can informally summarize the paradox: it is sometimes possible to increase network congestion by increasing the number of routes between nodes. From a software testing point of view, Braess's Paradox can arise when running network performance tests. To be honest, the chances of you running into Braess's Paradox are very slim. But the phenomenon does exist and the moral is that you should not assume that adding capacity to a network will necessarily improve performance. If your add capacity and do not see the performance improvement you were expecting, Braess's Paradox is one of the things to investigate.

Parrondo's Paradox

Essentially, Parrondo's Paradox states that two losing gambling games, let's call them Game A and Game B, can be set up so that when they are played one after the other, they become winning games. There are many ways to construct examples of Parrondo's Paradox, the simplest is to use three biased coins.

Game A is a simple coin-flipping game. You flip a coin and if the coin comes up heads, you win $1. If the coin comes up tails, you lose $1. Coin 1 is slightly biased so that it comes up heads with probability of p1 = 0.495 (and therefore tails has a probability of 1-p1 = 0.505). If you play this game over and over, you will steadily lose money over time. Game B is slightly more complicated, with two coins. The first coin in this game, Coin 2, has a probability of winning (heads) p2 = 0.095. This is a very bad coin. Coin 3, the second coin, has a probability of winning p3 = 0.745. This is a good coin.

You start out with some fixed amount of money, say $300. To play Game B you have two steps. First, you check to see if the amount of money you have is a multiple of 3 ($300, $297, $303, or such). If your current capital is a multiple of three, you flip Coin 2 (the very bad coin) and either win $1 or lose $1. If your current capital is not evenly divisible by 3, you flip Coin 3 (the good coin) and win $1 or lose $1. Although it is not entirely obvious, if you play Game B over and over, you will certainly lose money over time.

At this point, you have two losing games. What do you suppose happens if we play Game A and then Game B either in a random pattern or a fixed pattern like ABB, ABB, ABB, and so on? Astonishingly, when played together, the losing games will actually win over time! The code accompanying this column gives a very lightweight simulation written in C#. Figure 6 shows a screen shot of the last part of a sample run of the program.

Figure 6 Parrondo's Paradox in Action
Figure 6 Parrondo's Paradox in Action

The game was created by the Spanish physicist Juan M. R. Parrondo. In 1999, G. P. Harmer and D. Abbott published "Losing Strategies Can Win by Parrondo's Paradox, "naming the paradox after Parrondo. Since then, there has been quite a bit of research on this game and its variants, including the effects of changing the various probabilities of the three coins and changing the pattern which determines whether to play Game A or Game B.

Realistically, the chances of you encountering Parrondo's Paradox are remote. In one far-fetched scenario, you start with some fixed amount of process priority (corresponding to capital in Parrondo's Paradox), you have some subroutine that increases or decreases the process priority based on CPU usage (corresponding to Game A), and you have a second subroutine that increases or decreases process priority contingent upon the current priority (corresponding to Game B). The two subroutines are designed to probabilistically reduce process priority separately over time. However, when the two subroutines are called over and over, they actually increase process priority. Admittedly, this is a real stretch, but Parrondo's Paradox is just too interesting to be left out of this column!

Send your questions and comments for James testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Cutting Edge: Flexible Custom Data Views

Cutting Edge
Flexible Custom Data Views
Dino Esposito

Code download available at: CuttingEdge0512.exe (129KB)
Contents
Problems with Real World Data
Keeping Data Synched
Managing Foreign Key Lists
Validation Through Validators
Validation Through Events
The Best Approach?
Command Bar Manipulations
Conclusion

ASP.NET 1.x introduced some powerful and useful data-bound controls. However, none were designed specifically to manage the view of a single record. When you build master/detail views, you need to display the contents of a single record. Then when the user selects a master record from a list or a grid, you'll typically want the application to drill down to show all the available fields. You can use the DataGrid and other controls to display a single record, but this approach is just a workaround. For a true record-view control in ASP.NET 1.x, you have to buy one or roll your own.

Aware of this problem, Microsoft has added some new controls to ASP.NET 2.0 that enable single-record views. Specifically, you can use the DetailsView and FormView controls with the DataGrid and GridView to achieve this functionality. Functionally speaking, FormView and DetailsView are nearly identical. They differ mostly in the layout of the generated output. FormView allows full control of the layout, letting you define a custom form that is filled with controls to display and edit field values. DetailsView, on the other hand, is restricted to a tabular layout with two columns: header and value. The value cell, however, can be customized with templates.

These features provide important functionality that was previously lacking. And while they are still somewhat limited, there are workarounds—aren't there always? In this column, I delve into the programming interface of the DetailsView and FormView controls to explain the changes that are necessary to make these controls more versatile.

Problems with Real World Data

Figure 1 shows a sample master/detail ASP.NET page that uses DetailsView and GridView (for the actual code, see the code download). Note that the app provides only textboxes to edit data regardless of the type of data being edited. While that's okay for some sorts of data, it's not ideal for all. What if, for instance, the data was a date?

Figure 1 GridView and DetailsView in Action
Figure 1 GridView and DetailsView in Action

Or what about the country field? The country name is going to be stored in a database that you may want to search, so consistency in the way a country name is expressed makes a difference. In this case, a dropdown list would be a better choice. It keeps the data consistent.

The page in Figure 1 is composed using the GridView and DetailsView controls. Bound to distinct data source controls, the GridView and DetailsView controls rely on the underlying data source controls to select and update the displayed records. Once the user has edited a record and clicks the Update button, the DetailsView control executes the associated update command and refreshes the view. So far so good.

In the same page, the GridView control displays a preview of the same record. However, the corresponding row is not updated to reflect changes. This isn't a surprise since there is no link between the GridView and DetailsView controls. When the user clicks to update the DetailsView, the page posts back. As a result, all page controls, including the GridView, are rebuilt. Typically, when the page is loading, the GridView, as well as other data-bound and composite controls, rebuild themselves from the view state. If the view state is not available (for example, if it's disabled), the GridView will access the bound data source.

The data source of the control is not stored in the view state, but its visual representation is. The visual representation of a data source is the part of the data source that is displayed in the grid rows. Grid row objects serialize all their contents to the view state, including the data item they embed.

After the page is loaded and all controls are initialized, the page can process the postback event. In this case, the postback event is the update of the detail record followed by the refresh of the detail view. As a result, the DetailsView control and its master grid are now out of sync. The GridView won't automatically reflect changes because its state is restored before the actual update is made.

You'll see a lot of activity if you look at SQL Profiler, tracking all database commands executed when the Update button is clicked to save changes on a DetailsView. The first command executed is "SELECT * FROM Customers", which fills out the GridView. Following that, there's an UPDATE command that updates the database. Finally, there's a SELECT statement, which reads back the modified record to update the DetailsView. (Note that this is an example where the GridView has ViewState disabled and the bound data source controls have caching disabled.)

Keeping Data Synched

The solution to data that's out of sync is straightforward—simply force the GridView to refresh its data source after the detail record has been updated. Fortunately, the DetailsView control fires a timely ItemUpdated event that can be used to keep master and detail views in sync, as done with the following event handler:

Sub OnItemUpdated(sender As Object, e As DetailsViewUpdatedEventArgs)
  GridView1.DataBind()
End Sub
However, when the GridView's data source control has caching enabled, this may not work. The DataBind method correctly triggers the binding process but, rather than querying the database, the process actually reloads the old data from the cache. On data source controls, caching is disabled by default and can be enabled by setting the EnableCaching property to true. While caching is a best practice, it is only supported when DataSet or DataTable objects are used to return data (if you use ObjectDataSource with custom collections, the data source control infrastructure doesn't provide caching capabilities. Caching is still possible, but it must be coded within the custom object model bound to ObjectDataSource).

When caching is enabled, it defaults to a time-based expiration policy. The default duration is set to 0, specifying an infinite time period. Unless you specify a nonzero duration (in seconds), the cached data will never expire and you'll have no way to refresh the view other than by binding to another data source.

There are two alternatives to consider here. The first is to set up a database cache dependency mechanism through the SqlCacheDependency property of the data source control. While smart and powerful, this approach requires changes to the physical database, adding triggers, stored procedures, and even a new table. Yup, this is just the kind of stuff that makes a database administrator jump for joy. Nonetheless, a database cache invalidation mechanism relieves the burden of having to do any further updates of cached data to keep views in sync.

The second approach is to use the CacheKeyDependency property of the data source controls, as shown here:

Sub Page_Load(sender As Object, e As EventArgs)
  If Not IsPostBack Then
    Cache(SqlDataSource1.CacheKeyDependency) = someData
  End If
End Sub

Sub OnItemUpdated(sender As Object, e As DetailsViewUpdatedEventArgs)
   Cache.Remove(SqlDataSource1.CacheKeyDependency)
   GridView1.DataBind()
End Sub

This property indicates a user-defined key dependency that is linked to all data cache entries created by the data source control. All cache entries expire when the key expires. What's important is that by removing or modifying the key specified through the CacheKeyDependency property, you can programmatically invalidate the data cached by a data source control and keep views in sync.

Managing Foreign Key Lists

The DetailsView control doesn't support templates for entirely customizing the structure of the output. When editing the contents of a record, you either go with the standard layout of the user interface—a vertical list of header/value pairs—or use another control, such as the FormView or a custom control.

While the overall layout of DetailsView cannot be modified, you can change the way in which a particular field is displayed within the standard layout. For example, you can use a Calendar control to render a date field. To do this, you employ the TemplateField class in much the same way you do for certain columns of grid controls. By using a TemplateField class to render a data field, you are free to use any layout you like for view, edit, and insert operations, as I've done here:

<asp:TemplateField HeaderText="Country">
   <ItemTemplate>
      <asp:literal runat="server" Text='<%# Eval("country") %>' />
   </ItemTemplate>
   <EditItemTemplate>
      <asp:dropdownlist ID="DropDownList1" runat="server"
          DataValueField="country" DataTextField="country"
          DatasourceID="CountriesDataSource"
          SelectedValue='<%# Bind("country") %>' />
   </EditItemTemplate>
</asp:TemplateField>

In this code, the field Country is rendered through a literal control in view mode, and turns to a data-bound dropdown list control in edit mode. The dropdown list is bound to a data source control which provides the list of countries, as done with this snippet of code:

<asp:SqlDataSource ID="CountriesDataSource" runat="server"
     ConnectionString='<%$ ConnectionStrings:LocalNWind %>'
     SelectCommand="SELECT DISTINCT country FROM customers">
</asp:SqlDataSource>

In the templates, note the use of two distinct operators for binding: Eval and Bind. Eval is an operator you use in data-binding expressions to access a public property on the bound data item. Functionally speaking, it is nearly equivalent to the DataBinder.Eval method you would have used in ASP.NET 1.x in the same situation. As used in the preceding code, Eval is an ASP.NET 2.0-only feature and will generate a compile error if used in an ASP.NET 1.x application. Furthermore, it will throw an exception if used in an ASP.NET 2.0 page outside of a template.

The Bind operator is like Eval, except that it adds the ability to write data back to the original data source field. In the previous code snippet, notice that at display time the Bind operator sets the SelectedValue property of the DropDownList control with the value of the country field from the data source. So, how are values passed back to the control for updating the data source?

When the page is compiled on the first hit, the EditItemTemplate property of the DetailsView control is set to an instance of a special internal template builder class. In addition to the methods of the ITemplate interface—an essential requirement for being bound to a template property—this class exposes a public method named ExtractValues. The body of the method consists of a simple call to a delegate object that looks like this:

Delegate Function ExtractTemplateValuesMethod( _
    ctl As Control) As IOrderedDictionary

The delegate object is passed to the template builder's constructor. The page parser is responsible for generating the code that sets the EditItemTemplate property of the DetailsView control and for emitting the source of the function bound to the delegate. This function simply populates an ordered collection with all the values of the properties bound to the Bind operator within the template. The code in Figure 2 gives you an idea of how the extractor method works in the template. There will be just one such method in each template class that handles all controls in the template bound to the Bind operator.

Figure 2 The Extractor Method (Close Figure 2)
Function ExtractValues_DetailsView1(c As Control) As IOrderedDictionary
    Dim table As New OrderedDictionary

    Dim DropDownList1 As DropDownList
    DropDownList1 = CType(c.FindControl("DropDownList1"), DropDownList1)
    If Not (table("country") Is Nothing) Then
        table("country") = DropDownList1.SelectedValue
    End If

    ... ' Repeat for each control in the template using Bind

    Return table
End Function
Figure 3 Details View Control
Figure 3 Details View Control

As shown in Figure 3, the ordered collection is accessed by the DetailsView control through the ExtractValues public method on the class that represents EditItemTemplate. When an update is performed, the DetailsView control retrieves the current values from bound input controls and uses them to compose the parameter list of the insert or edit command to run against the data source. The argument passed to Bind must match the name of a parameter in the command. What I have demonstrated for the DetailsView control works in the same way for templates used inside a FormView or a GridView control.

Bear in mind that for calendars, in particular, the property to bind for correct data exchange is SelectedDate. You need to use Eval with VisibleDate, instead, to ensure that the currently set date is displayed in the control's user interface.

Validation Through Validators

Any real-world user interface will require validation. Since version 1.0, ASP.NET has provided a number of validation controls—for required fields, regular expressions, and specific ranges of values. In ASP.NET 2.0, you can group some validators and associate them with a button so that only those controls will be used to validate the contents of the page being posted.

There are two ways in which you can use validators to perform a preliminary test on the input values a user is entering via a DetailsView control. In the first approach, you use templates for each displayed field, and add the validator controls directly in the template. This method is easy and effective, but requires you to write a bit of boilerplate code. Figure 4 shows how to expand a BoundField element to an editable TemplateField that uses validators.

Figure 4 Expand a BoundField Element (Close Figure 4)
<asp:TemplateField HeaderText="ID">
  <ItemTemplate>
     <asp:Literal runat="server" Text='<%# Eval("CustomerID") %>' />
  </ItemTemplate>
  <EditItemTemplate>
     <asp:TextBox ID="CustomerID_TextBox" runat="server"
          Text='<%# Bind("CustomerID") %>' />
     <asp:RequiredFieldValidator runat="server"
          ControlToValidate="CustomerID_TextBox" Text="*" />
   </EditItemTemplate>
   <InsertItemTemplate>
      <asp:TextBox ID="CustomerID_TextBox" runat="server"
           Text='<%# Bind("CustomerID") %>' />
    </InsertItemTemplate>
</asp:TemplateField>

You remove the DataField attribute, which is not supported by a template field, and pass its value to any Eval and Bind operators you insert. The InsertItemTemplate in Figure 4 is used when the user attempts to add a new record to the table. In the code, I have only a RequiredFieldValidator control; you are free to add any combination of validators anywhere it makes sense for your app.

Note that a Literal control is preferable to Label because it is more lightweight. But, as a result, the Literal control cannot be styled. To use the Eval operator—a required feature here—you need to employ a server control, which eliminates the possibility of using an even lighter HTML <span> tag to render the value of the field, at least in view mode.

In addition to requiring more code, a template field is also heavier than a simpler BoundField element. If you don't feel comfortable with templated fields, you can try the following approach instead. You use an external piece of code (a separate and reusable component) that walks its way through the DetailsView object model and injects validators wherever needed. All fields will remain BoundField controls, and you can validate the input at will. Figure 5 shows the code needed to add to the page's code file (this is basically a handler for the ItemCreated event and a helper routine).

Figure 5 Adding Validators Programmatically (Close Figure 5)
Public Partial Class MasterDetail1
        Inherits System.Web.UI.Page

    Private Sub Page_Load(ByVal sender As Object, ByVal e As EventArgs)

    End Sub

    Protected Sub DetailsView1_ItemCreated( _
            ByVal sender As Object, ByVal e As EventArgs)
        If (DetailsView1.CurrentMode = DetailsViewMode.ReadOnly) Then
            Return
        End If

        ' Let's assume we know which fields we want to modify and
        ' their EXACT location (reasonable assumptions most of the time)
        AddRequiredFieldValidator(1, "Company name required")
        AddRequiredFieldValidator(2, "Contact name required")
    End Sub

    Private Sub AddRequiredFieldValidator( _
            ByVal rowIndex As Integer, ByVal msg As String)

        Const DataCellIndex As Integer = 1

        Dim row As DetailsViewRow = DetailsView1.Rows(rowIndex)
        Dim cell As DataControlFieldCell =
            CType(row.Cells(DataCellIndex), DataControlFieldCell)
        Dim req As New RequiredFieldValidator
        req.Text = String.Format("<span title='{0}'>*</span>", msg)

        Dim ctlID As String = cell.Controls(0).UniqueID
        Dim pos As Integer = ctlID.LastIndexOf("$")
        If (pos > 0) Then
            req.ControlToValidate = ctlID.Substring((pos + 1))
            cell.Controls.Add(req)
        End If
    End Sub
End Class

The ItemCreated event of a DetailsView control doesn't contain any additional information; it's merely a notification that the record has been created. This event occurs once per rendering of the control; it won't fire on rendering of each field.

To inject a new control in the standard layout, you must first get a reference to the row to extend. Each row corresponds to a record field. The Rows array contains a DetailsViewRow element for each bound field. This class inherits from TableRow and contains only two cells by design–the first cell is the header, the second is the value. The second cell is where you add a validator. Here, you'll find just one control: the textbox. The ID of this control is prefixed by the ID of the DetailsView, separated by a $. For example:

DetailsView1$CustomerID

To successfully bind the validator to the textbox, you need only the suffix, or textbox ID, as specified in the ASP.NET source. When the validator is all set, simply add the control to the Controls collection of the cell.

Validation Through Events

If none of these approaches work for you, I have another suggestion you can try for validating the data that users enter through a DetailsView control. Stop injecting validator controls into the layout! After all, the DetailsView control fires three pairs of pre/post events—Updating/Updated, Inserting/Inserted, and Deleting/Deleted. More importantly, all pre-action events (Updating, for instance) support a writable Boolean property—named Cancel—through which you can cancel the ongoing event. This means you can hook up the pre-action event for the operation (typically, insert or update), take a look at the data being posted, and cancel the event if something appears to be wrong or invalid.

To begin, add a handler for the ItemUpdating event. The event delegate has the prototype shown here:

Public Delegate Sub DetailsViewUpdatedEventHandler( _
    ByVal sender As Object, ByVal e As DetailsViewUpdatedEventArgs)
And the DetailsViewUpdateEventArgs class has the following structure, inherent from CancelEventArgs:
Public Class class DetailsViewUpdateEventArgs
        Inherits CancelEventArgs
    Public ReadOnly Property CommandArgument As Object
    Public ReadOnly Property Keys As IOrderedDictionary
    Public ReadOnly Property NewValues As IOrderedDictionary
    Public ReadOnly Property OldValues As IorderedDictionary
End Class

Two of the ordered collections in the listing are of particular interest here: OldValues and NewValues. The former contains the values that were originally displayed to the user in the update form. The latter contains the values the user posted for changes. You can do a number of things with this information, the most interesting of which can occur if you're writing a derived control. In this case, for example, you can compare old and new values and skip the update if no new value has really been specified. Doing the same from within a page that contains a DetailsView doesn't work because you have no way to stop the update if not canceling the event. Canceling the event, however, leaves the form in edit mode and, for consistency, should be used only if you detect an error in the posted data.

Intercepting the ItemUpdating event from within a page is useful for validation purposes. Here's a possible body for the handler that checks for a particular prohibited substring in the address:

Sub ItemUpdating(sender As Object, e As DetailsViewUpdateEventArgs)
   Dim address As String = CType(e.NewValues("Address"), string)
   If address Is Nothing OrElse address.IndexOf("Via") >= 0 Then
      e.Cancel = True
      Return
   End If
End Sub

Here, you retrieve values from the NewValues collection using either 0-based indexes or strings. In this case, you need to use strings that exactly match the values assigned to DataKeyField attributes. It's important to note that case matters. The address variable in the preceding snippet will be null if, say, you have the following markup in the .aspx page, where the data field name is lowercase:

<asp:BoundField DataField="address" HeaderText="Address" />

I'm not sure how you will want to deal with this situation in the handler. What if the value to check is null for some reason? Would you rather default to unfiltered updates or entirely cancel the event? I, personally, would rather take the second, more aggressive route.

Is this solution good enough? Well, it doesn't force you to use templated fields, nor does it require you to add quirky code to some event handler. And still you're able to detect invalid data and cancel the update. However, users have no clue about what went wrong. Again, the DetailsView control is not designed to help you much with this. To clarify what went wrong, you need to modify the control's layout and display a tooltip with error information. Here's code to do this:

If address.IndexOf("Via") >= 0 Then
   Dim index As Integer = IndexOf(e.NewValues, "Address")
   Dim row As DetailsViewRow = DetailsView1.Rows(index)
   Dim msg As String = "<span title='{0}' style='color:red'>*</span>"
   msg = string.Format(msg, "The address is invalid. Please edit.")
   row.Cells(1).Controls.Add(New LiteralControl(msg))
   e.Cancel = True
   Return
End If

The OrderedDictionary class doesn't have a public IndexOf method, so you have to write one. There is one implemented in the sample code that is available for download. Don't be too worried when you find that my code simply loops through the collection—this is exactly what the private IndexOf method on the OrderedDictionary class does. The code retrieves the index of the row where an error was detected, gets its second cell, and then adds a Literal control with an asterisk and a more explanatory tooltip.

The Best Approach?

So what's the best approach for validating data entered in a DetailsView control? All in all, I believe that using templated fields is the most flexible solution. Validators check input data already on the client, which in some cases saves a round-trip action. This is more an optimization than a real performance gain. As for security, you might still want to perform server-side validation. With validators, you likely do a smaller number of round-trips. Events, however, are fired only after a postback is made. But if you opt for events, bear in mind that you should add extra visual elements to the DetailsView to inform the user of what's happening.

Command Bar Manipulations

To wrap things up, let's focus on the command bar—the bottom row where the DetailsView places predefined buttons to execute basic actions. There are two enhancements I'll show you how to make to these buttons. One is to add a tooltip to explain what's going to happen if the button is clicked. The other is to add a bit of JavaScript code to ask for confirmation before deleting or updating.

DetailsView and FormView controls support delete operations and delegate the execution to the underlying data source control. If the data source control is configured to execute the delete operation, all works fine. Otherwise, an exception is thrown.

The DetailsView generates command buttons automatically and doesn't expose them directly to page code. The bit of code that asks for confirmation is shown here:

Sub ItemCreated(sender As Object, e As EventArgs)
   If DetailsView1.FooterRow IsNot Nothing Then
      Dim commandRowIndex As Integer = DetailsView1.Rows.Count - 1
      Dim commandRow As DetailsViewRow = DetailsView1.Rows(commandRowIndex)
      Dim cell As DataControlFieldCell = _
         CType(commandRow.Controls(0), DataControlFieldCell)
      For Each ctl As Control In cell.Controls
         If TypeOf ctl Is LinkButton
            Dim link As LinkButton = CType(ctl, LinkButton)
            ...
         End If
      Next
   End If
End Sub

The ItemCreated event doesn't provide any information about the row being created. However, you can assume that the footer row is always created and is always the last row to be created. If the FooterRow object is not null, you can conclude that all rows have been created, including the command bar. The command bar is the first row after the data rows and is stored in the Rows collection. It's the last element in the collection.

The command bar is a table row (type is DetailsViewRow) and contains a cell (type is DataControlFieldCell). The cell contains as many link buttons (of type DataControlLinkButton) as there are commands. Delete, Edit, New, Update, Insert, and Cancel are the command names used. They are useful for identifying the right button in code like this:

If link.CommandName = "Delete" Then
    Link.ToolTip = "Click here to delete"
    link.OnClientClick = "return confirm(msg);"
End If

Edit and New are used to enter edit or insert mode. Update and Insert refer to the buttons used to save changes in those modes.

Conclusion

If you are working with ASP.NET 1.x and need a record-view control, you'll need to purchase one or roll your own. I actually wrote one for the ASP.NET Developer Center that mimics the behavior of the ASP.NET 2.0 DetailsView control, which you can download at A DetailsView Control for ASP.NET 1.x.

ASP.NET 2.0 offers features that help address this issue. In this column, I have illustrated enhancements that will let your users edit records more effectively. If you plan to use the DetailsView control extensively in your applications, you should build all these features into a new derived control. By the way, everything stated here for the DetailsView also applies to FormView, with the exception of injection of controls and code. Being fully templated, the FormView lets you do the same directly through its templates.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog.


Service Station: An XML Guru's Guide to BizTalk, Part 2

Service Station
An XML Guru's Guide to BizTalk, Part 2
Aaron Skonnard

Contents
The Journey of a Message
Using Ports to Connect the Dots
Port Internals
Messages
Adapters
Pipelines
Maps
Where Are We?

In my last column, I provided a brief introduction to BizTalk® Server 2004 for XML developers (see Service Station: An XML Guru's Guide to BizTalk Server 2004, Part I). I covered the product evolution, core architecture, and several aspects of the underlying messaging layer, all of which have helped make BizTalk Server 2004 the powerful integration technology it is today. I wrapped up after introducing the various extensibility points offered by the messaging layer. These extensibility points are where most of the valuable XML nuggets reside. This month I'll discuss these concepts in more detail. Note that everything I'm discussing is also relevant to the new BizTalk Server 2006.

The Journey of a Message

As I discussed in Part 1 of this column, a message enters BizTalk Server 2004 through a receive port. The job of a receive port is to process the incoming bytes and then to publish the resulting message to the MessageBox.

The MessageBox is, in many ways, the heart of BizTalk Server 2004. All messages passing through the system are persisted to the MessageBox for processing. In addition to holding transient messages, the MessageBox holds subscription information that directs what happens when messages arrive. You create subscriptions either explicitly through BizTalk Explorer in Visual Studio® .NET (that is, via a send port filter) or implicitly through the orchestration designer (via Receive shapes). Hence, there are two primary types of message subscribers: send ports and orchestrations.

When a message hits the MessageBox, the subscriptions are evaluated to determine which ones match, and each matching subscription is activated. When an incoming message matches a send port subscriber, the message is simply sent out through the given send port. When an incoming message matches an orchestration subscriber, the orchestration is either activated (when it's not already running) or in the case of a running orchestration, the message is simply routed back to the existing orchestration instance (such as when an orchestration is waiting for a response message to return).

To sum it up, messages enter the system through receive ports and leave the system through send ports, potentially passing through orchestration instances along the way. Let's look at a simple scenario in which BizTalk Server 2004 can be used to connect the dots between a few disparate applications.

Using Ports to Connect the Dots

Assume that you need to connect a legacy application with a few new apps. The legacy system was designed to periodically spit out a batch file containing orders that the new applications need to process. One of the new applications was designed to receive orders through a particular FTP site while the other application was designed to receive them through a Microsoft® Message Queue (MSMQ). There are many ways you could write the code to connect these systems. But to do it right, you'll have to deal with multicasting, a variety of protocols, possible network problems, retry attempts, backup plans, and differences in message formats, so it isn't as simple as it first sounds.

Luckily, this is exactly the type of integration scenario BizTalk Server 2004 was designed to solve. And you can accomplish the integration without writing a single line of code. Just configure a few ports and you're finished. You're basically putting BizTalk Server 2004 between the legacy application and the new apps in what integration specialists typically refer to as a "hub and spoke" configuration. BizTalk Server 2004 becomes the hub and the various disparate applications are the spokes. In such a configuration, only the hub needs to understand all of the protocols used by the various spokes. Then any one spoke can talk to any other by relaying the message through the hub.

In order to accomplish this kind of protocol translation with BizTalk Server 2004, you first create a new receive port. You can do this in Visual Studio .NET by selecting Add Receive Port in BizTalk Explorer. In this case you need a one-way port (which I'll call OrderBatchReceivePort).

Once the receive port is created, you need to create a receive location and configure it to pick up the order batch file produced by the legacy application. The dialog for creating a receive location is shown in Figure 1. You'll give the receive location a name, choose a transport type, and specify an address. For this example you need to use the FILE transport type. If you click the button next to the Address field, you'll see a dialog box for entering address details. The configuration I've used tells the receive location to listen for all .xml files in the \\devserver\OrderBatchDrop network share, as I've entered that path in both the Receive folder and Public address fields. It also tells BizTalk to retry every five minutes for a total of five tries if the network share is unavailable because I entered the value 5 into Retry Count and Retry Interval boxes.

Figure 1 Configuring a New Receive Location
Figure 1 Configuring a New Receive Location

At this point you also need to choose a receive pipeline that will be used to process the incoming messages before they're published to the MessageBox. In this case I'll choose the built-in PassThruReceive pipeline, which doesn't do any special processing. It just passes the message through to the MessageBox.

Once you've created a receive location, it will show up in BizTalk Explorer. Then you need to right-click on the receive location and select Enable to have it begin listening for incoming files. Once you do that, your new receive location will begin picking up all .xml files written to the \\devserver\OrderBatchDrop share and publishing them to the MessageBox. However, if you do it at this point, an error will be written to the event log stating that no subscriptions were found for the incoming message.

Now you need to configure the send ports—they will be the subscribers in this case. This can be done by selecting Add Send Port in BizTalk Explorer and configuring it much like you did for the receive port. One of the main differences between send and receive ports is that send ports don't contain send locations. You configure the transport details right on the send port itself. For this example you'll need two one-way send ports. One needs to be configured for FTP and the other for the BizTalk Message Queuing Adapter (MSMQT), the BizTalk Server implementation of the MSMQ wire protocol. Figure 2 serves as an example of the configuration settings.

Figure 2 Configuring a New Send Port (MSMQ)
Figure 2 Configuring a New Send Port (MSMQ)

Note that the addresses are configured for each send port. One is an FTP address (not shown) while the other is specific to MSMQ (as shown in Figure 2). The FTP address includes a file name of %MessageID%.xml, which tells BizTalk to generate a new file name using a unique message identifier when sending the file to the FTP location. With send ports, you also specify retry information as well as a secondary transport to use when the retry attempts fail. And you must specify a send pipeline (which I'll discuss later), which will be used to process the message after it's read from the MessageBox, but before it's transmitted to the send port address. I'll use the PassThruTransmit pipeline for now.

Now you need to define subscription information for both of these send ports, so BizTalk knows that they need to be used whenever messages come in through the OrderBatchReceivePort. You do this by defining "filters" on the send port, as shown in Figure 3. The filter expression (BTS.ReceivePortName = = OrderBatchReceivePort) states that if the message entered the system via a receive port named "OrderBatchReceivePort," you have a match. You need to use this same filter expression on both send ports so they are both used for each message that hits the receive location. This is a situation in which you could also use a send port group so you only have to specify the filter expression once. When you're finished configuring the send ports, right-click on them in BizTalk Explorer and select Start. Doing so writes the send port subscription information to the MessageBox and enables them for use.

Figure 3 Configuring a Filter Expression (Subscription)
Figure 3 Configuring a Filter Expression (Subscription)

With this configuration in place, the legacy application can now drop arbitrarily named XML files into the \\devserver\OrderBatchDrop network share, at which point BizTalk Server 2004 will read the file from the network share and publish the contents to the MessageBox. When the message hits the MessageBox, the subscriptions are evaluated and, in this case, both send ports are activated. This causes the message to be sent to the FTP site and MSMQ queue, which are the integration points for the new applications, as illustrated in Figure 4.

Figure 4 BizTalk Integration Scenario (Close Figure 4)

Notice that you did not need an orchestration to solve this integration scenario, although you might choose to use one when you need more sophisticated error handling logic beyond what's provided by ports. Also notice that I have not performed any special message processing at this point. Since I used the "PassThru" pipelines, the messages simply pass through the system unchanged. It's likely you'll need to perform message transformations during the integration process. I've essentially used ports to connect the dots between these disparate applications at a protocol level. It's interesting to note that this type of integration would not have been possible using Web services alone, at least not without modifying the legacy application to some degree. This integration is accomplished by sending messages through a hub capable of speaking a variety of different dialects.

Port Internals

There is actually much more to a port than first meets the eye. As I walked through the integration scenario, I largely ignored message processing needs and focused on connecting the dots across different protocols. Message processing is a big part of what happens within a port, enabling you to inject code to process or to reshape messages as they pass through the system. This occurs right after a message is received and before it's published to the MessageBox. And, it also occurs after a message is read from the MessageBox but before it's transmitted.

A port primarily consists of an adapter, a pipeline, and a set of maps. Each piece focuses on a different aspect of message processing. Adapters deal with transport details and reading bytes. Pipelines are responsible for performing message processing in stages—they deal with things like message encoding/decoding, message context, validation, and security. A map is simply a transform that defines how to move data from one message schema into another. I'll discuss all three in more detail shortly.

When receiving messages through a receive location, the adapter does its job first, followed by the receive pipeline, and then finally the maps are applied. The resulting message is then published to the MessageBox. The same process is used when sending messages through a send port, except in reverse. The maps are applied first, followed by the send pipeline, and then the adapter transmits the resulting bytes using the specified transport.

On the receive side, you configure the adapter and pipeline on the receive location, but you configure the set of maps on the receive port itself (this allows you to have multiple receive locations that all use the same set of maps). However, on the send side, you configure all three on the send port (you can't configure maps on a send port group,as you might imagine).

As a developer, these three areas are your primary extensibility points for message processing.

Messages

Before diving into adapters, pipelines, and ports, it's important to first understand what messages are in BizTalk Server 2004 and how they are dealt with. When an external message enters the system through a receive location, BizTalk constructs a message that is used internally from that point on. The message is actually considered a multipart message, where each part can be of a different type (XML, flat file, serialized .NET Framework class, and so on). In the most common case, a message contains a single part of XML.

Once messages are constructed, they become immutable. This means that other components cannot modify the message directly. They can, however, construct a new message and populate it with data from another message, which is very common.

Messages also have a context associated with them while in the system. This is a very important concept in BizTalk Server 2004 as it's your only interface to the message during processing. You can think of the message context as the view of the message available to you at run time. For example, only the message context is available to you when writing filter expressions or orchestration logic—you don't have access to the entire message at those points. Some message context properties are related to transport details, others to BizTalk concepts (such as receive port names and IDs), and others come from the content of the message. You control what goes in the message context by promoting properties in your schemas during design.

Messages are the focal point in BizTalk. Adapters, pipelines, and maps all deal with them directly as do orchestrations.

Adapters

A BizTalk adapter is a piece of .NET code that implements a particular transport protocol and knows how to integrate with the rest of the BizTalk infrastructure. Adapters are used to both send and receive messages. A send adapter's primary job is to pump the bytes it receives through the transport. A receive adapter's primary job is to read the incoming bytes from the transport and to construct a BizTalk message. It's also responsible for creating the initial message context and populating it with some basic information, including transport and receive location details. After the message is constructed, it's handed off to the receive pipeline.

In the integration scenario presented earlier, I used the FILE adapter to receive the order batch file from a network share and the FTP and MSMQ adapters to send the message to different locations. BizTalk Server 2004 ships with several adapters in the box including EDI, FILE, FTP, HTTP, MSMQT, SMTP, SOAP, and SQL. These will show up in the Transport Type list in the various configurations screens.

These built-in adapters provide for some of the most common communication cases, but the BizTalk adapter framework allows for new adapters to be introduced at any point in time. Microsoft has shipped several downloadable adapters since the release of BizTalk Server 2004 (see Microsoft BizTalk Server 2004 Adapters). This list includes adapters for Web Services Enhancements (WSE) 2.0, MSMQ, WebSphere MQ v2.0, mySAP Business Suite, and even a community drop of a Windows Communication Foundation Beta 1 adapter.

But what's most impressive is how the BizTalk adapter framework has fostered a healthy third-party community. Check out the long list of available partner adapters, which covers almost everything available today, at BizTalk Partner Adapters 2004. It's primarily this healthy collection of adapters that makes BizTalk so compelling from an integration perspective.

And if that isn't enough for your needs, you can always write a custom adapter that can be plugged into the system. This is typically left to the integration vendors as it's not for the faint of heart. Adapters deal with low-level protocols and communication details, which is not where most developers want to spend their time. Most just choose an existing adapter and configure it to fit their particular needs. Pipelines are the main extensibility point for message processing.

Pipelines

I've mentioned the BizTalk pipelines already and you are probably wondering what they are, so let's take a look. A BizTalk pipeline is a .NET-based component that performs pre- and post-processing on a BizTalk message. There are two types of pipelines: receive pipelines and send pipelines. You can think of a pipeline as a skeleton for common processing stages that occur when sending or receiving messages. Receive pipelines have four stages: decode, disassemble, validate, and resolve party. Send pipelines have three stages: pre-assemble, assemble, and encode. You can add components to perform message processing during any of these stages.

You design pipelines in Visual Studio .NET using the BizTalk pipeline designer. The designer allows you to drag pipeline components from the toolbox into the different stages of interest (see Figure 5). BizTalk Server 2004 ships with several default pipeline components, which cover some of the most common message processing tasks. These will show up in the pipeline designer toolbox. You can also write custom pipeline components that can be used here.

Figure 5 XML Validation Pipeline
Figure 5 XML Validation Pipeline

There are a few default pipelines that are already preconfigured and ready to use when you install BizTalk Server 2004. There are two receive pipelines, XMLReceive and PassThruReceive, and two send pipelines, XMLTransmit and PassThruTransmit. The PassThru pipelines don't do anything; in other words, they don't contain any pipeline components.

The default XMLReceive pipeline uses the XML Disassembler component (as well as the Party Resolution component). The XML Disassembler component looks for the XML Schema Definition (XSD) corresponding to the incoming message and then uses it to parse the message and to build the message context according to the promotions found in the schema. (A common error when using BizTalk Server 2004 is to use this component without deploying the XSD schemas it needs to function.)

This pipeline does not perform XSD validation by default. However, if you want to enable XSD validation, you can create a new receive pipeline and drag the XML Disassembler component to the Disassemble stage, the Party Resolution component to the ResolveParty stage, and the XML Validator component to the validate stage (see Figure 5). Your new receive pipeline will be equivalent to XMLReceive but with XSD validation enabled. I love how easy it is to incorporate XSD validation into your message processing without writing any code.

The default XMLTransmit pipeline only contains a single component, the XML Assembler, which focuses primarily on transferring properties from the message context back into the document and serializing it for transmission.

In my integration scenario, it's likely that the legacy application doesn't produce XML, but rather some proprietary flat file format. In cases like these, you can design pipelines that use the Flat File Assembler/Disassembler components. With these components, you specify the XSD schema that contains the information about how to map the flat file format to an internal XML format, and the components make it happen (in either direction). You'll design the specific flat file translation information into your XSD schemas using the BizTalk Schema Editor.

The receive pipeline is where the message context is built and populated. So if you ever need to get special stuff in the message context (that isn't automatically provided elsewhere), this is where you want to do it. You'll need to write a pipeline component that promotes the information into the context during the Disassemble stage. Writing custom pipeline components is probably the most common place you'll inject .NET code within the messaging layer. And the beauty of this model is that once you've developed pipeline components, all your developers can take advantage of them in a declarative manner using the pipeline designer.

Maps

After a message comes out of a receive pipeline, you can apply maps to it. A map is essentially an XSLT transformation that reshapes the message into a different XML format. You design maps using the BizTalk Mapper, which is integrated with Visual Studio .NET (see Figure 6). The BizTalk Mapper is a wonderful tool that takes the pain out of XSLT. It allows you to open the source and destination XSD schemas and then you typically just drag lines between them in order to specify how the data should map over.

Figure 6 Map for Transforming Between Order Schemas
Figure 6 Map for Transforming Between Order Schemas

This technique works well in simple scenarios where you mostly have a 1-to-1 mapping. But in more complex scenarios, you'll need to utilize functoids in your maps. A functoid is a special component that executes some code during the XSLT transformation (the red squares in Figure 6).

BizTalk Server 2004 ships with numerous functoids that you'll find in the BizTalk Mapper toolbox. There is also a special Scripting functoid that allows you to embed .NET code or XSLT templates directly into the map. It also allows you to call out to external .NET assemblies if you need some help from imperative code along the way. And if that isn't enough to satisfy your extensibility cravings, you can also write custom functoids that can be deployed and used by all the developers on your team when designing new maps (similar to pipeline components).

Since BizTalk maps are simply XSLT, you can author them using the XSLT tool of your choice and then bring them into your BizTalk solution. However, the BizTalk mapping tools are some of the most user-friendly tools I've used, which makes it hard to find a reason to look elsewhere.

In general, the mapping layer makes it simple to address tough integration issues at the message level. Going back to my integration scenario, it's likely that each of the new applications require a different XML dialect. You can solve this by using a receive pipeline to disassemble the flat file into an internal XML message format. Then you can design two maps, one for moving between the internal schema and each schema required by the new applications. Then you simply apply those maps to the send ports.

Maps are also commonly used within receive ports to normalize different external document formats that you need to support (from different customers or application versions, for instance) into a single internal message format that you've built your business process around. Maps can also be used within orchestrations to move between different message formats as needed by the business process.

The BizTalk mapping layer is one of the product's crown jewels, and combined with the adapter framework, it becomes very compelling. I've met several customers who use BizTalk Server 2004 strictly for these powerful XML messaging components alone.

Where Are We?

You can use BizTalk Server 2004 in a variety of configurations to connect your systems without much code. It's truly fit to become your infrastructure duct tape once you understand how it works. It can provide a quick fix to tough integration problems at both the protocol and message level. If you added yet another new application to the integration scenario I presented here, and this one was built using WSE 2.0 with various WS-Security features, you could easily integrate the legacy application with the fancy new WSE 2.0 endpoint through yet another send port. The connectivity possibilities are endless.

In this two-part series on BizTalk Server 2004, I've discussed the overall architecture and core messaging components that bring XML to life in some pragmatic and flexible ways. The BizTalk Server 2004 messaging layer provides powerful connectivity and integration benefits when building connected systems in the real world where legacy applications abound. But it's the ability to deal with legacy as well as new Web service infrastructure that makes BizTalk Server 2004 a compelling XML platform to build on long term.

If you'd like some resources to explore, check out Understanding BizTalk Server 2004, BizTalk Server 2004: A Messaging Engine Overview, and my BizTalk Wiki.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, a premier Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services, Applied BizTalk Server 2004, and Introducing Indigo courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


.NET Matters: BigInteger, GetFiles, and More

.NET Matters
BigInteger, GetFiles, and More
Stephen Toub

Code download available at: NETMatters0512.exe (123KB)

Q I want to do some work with large numbers, larger than the sizes allowed by UInt64. Does the Microsoft® .NET Framework have any support for this?

A The base class libraries (BCL) don't expose publicly any big number libraries, although some of the cryptography-related classes do make use of an internal big number implementation. However, all is not lost. While you can search the Web to find a plethora of implementations in C#, C++, and a variety of other languages, it might not be necessary. If you don't mind taking a dependency on the J# libraries, you already have a big number implementation at your disposal. In fact, you have two.

The J# run-time library, vjslib.dll, is available as a redistributable component, just like the .NET Framework. You can download it from Visual J# Downloads (it's also installed as a prerequisite by Visual Studio®). In the same manner that a C# or C++ application can make use of Microsoft.VisualBasic.dll (the Visual Basic run-time library), C#, Visual Basic®, and C++ applications can use the J# run-time library and the numerous interesting classes it exposes.

Some folks use the J# Zip libraries to meet their compression requirements (see Zip Your Data: Using the Zip Classes in the J# Class Libraries to Compress Files and Data with C#), but beyond that there are some very interesting gems hidden in the library. For your needs, I suggest you see the java.math.BigInteger class (a BigDecimal class is also available). Here's an example of using BigInteger, showing that it can be used with values larger than the largest UInt64:

BigInteger i = new BigInteger(ulong.MaxValue.ToString());
BigInteger iSquared = i.multiply(i);
Console.WriteLine("i:\t" + i);
Console.WriteLine("i^2:\t" + iSquared);

This outputs:

i:      18446744073709551615
i^2:    340282366920938463426481119284349108225

BigInteger exposes a large number of operations. This includes the standard operations, such as addition, subtraction, multiplication, division, and modulation, but it also exposes functionality such as the ability to find the GCD of two BigIntegers, primality testing, bit set testing, and conversion to other data types. All in all, it's a very useful class.

Of course, an answer on this subject wouldn't be complete without a strict warning. Most of the time I hear people asking for such primitives, it's because they want to roll their own cryptographic operations. Don't do it. You'll quite likely be opening yourself up to a world of hurt. Instead, use the classes in the System.Security.Cryptography namespace. If they don't meet your needs (which they should most of the time), consider using the unmanaged Crypto API. If that doesn't meet your needs, look into implementations from trusted vendors. Don't roll your own.

While you're considering exploring vjslib.dll, look around a bit. There are a lot of interesting classes available there that might help you in your projects.

Q I have an attribute that I use on classes to help with my development process, but I don't want this information exposed to the public. As such, I've been wrapping all applications of the attribute to classes with compilation directives such that they're only applied in debug builds:

#if DEBUG
[MySecretAttribute("private data")]
#endif
public class MyClass { ... }
Is this the best way?

A In the .NET Framework 1.x, yes, you're not going to find anything much better. However, the .NET Framework 2.0 adds support for applying conditional attributes to attributes. In other words, you can tag your custom attribute with ConditionalAttribute ("DEBUG"), and any applications of your attribute will only be compiled into builds with the DEBUG flag set. To exemplify this, I've compiled the code in Figure 1 into a debug build as well as into a release build. Figure 2 shows the results of running ILDASM on each of the resulting executables. The top screenshot is the debug build and, as you can see, the attribute was compiled into the build. The bottom screenshot is the release build and, as desired, it lacks the attribute. This is a very welcome change to the compiler.

Figure 1 Using Conditional Compilation with Attributes (Close Figure 1)
using System;
using System.Diagnostics;

[MySecret("private data")]
class Program
{
    static void Main() { }
}

[AttributeUsage(AttributeTargets.Class)]
[Conditional("DEBUG")]
class MySecretAttribute : Attribute
{
    public MySecretAttribute(string value)
    {
        _value = value;
    }

    private string _value;

    public string Value { get { return _value; } }
}
Figure 2 ILDASM of Debug and Release Builds
Figure 2 ILDASM of Debug and Release Builds

Q I'm working on a utility and, as part of that utility, I need to display the contents of a file in a nice fashion, ideally in hexadecimal. What are my options?

A That's a very generic request, but what you want to do is certainly possible. First, if you just want to get the contents of the file as a string of hexadecimal numbers, and if you don't care much about the formatting, you can simply use the System.BitConverter class:

byte [] data = ...;
string result = BitConverter.ToString(data);

If you need more control, you can format the string however you like by manually iterating over the data:

byte [] data = ...;
StringBuilder text = new StringBuilder(data.Length*2);
foreach(byte b in data) text.AppendFormat("{0:X2}", b);
string result = text.ToString();

However, I'll take this opportunity to discuss two things that are slightly more complex. The first is a class that has gone largely unnoticed in the .NET Framework but which I think deserves more recognition, and the second is the new debugger visualizations feature available in Visual Studio 2005, one of the most useful additions to the new version.

Did you know that System.Design.dll contains a Windows® Forms control ready-made for viewing the contents of files? System.ComponentModel.Design.ByteViewer has been available in the .NET Framework since 1.x, but I've rarely seen anyone use it. The control has a SetBytes method that accepts a byte array to be rendered. It also has a SetFile method that accepts the full path to a file as a string and loads the contents of the file (of course, this is just as easy to do in the .NET Framework 2.0 using the new File.ReadAllBytes method).

To show off its usefulness, I've created a debugger visualizer for visualizing sequences of bytes. Debugger visualizers are a feature of the Visual Studio debugger interface, displaying an object in a meaningful way that is appropriate to the object's data type. For example, Visual Studio comes with visualizers that allow you to view managed strings as HTML, XML, or plaintext. Writing visualizers entails writing a piece of code to execute in the context of the debugger, and possibly a piece of code to run within the context of the application. For my purposes, I only need to create the debugger-side code.

This visualizer can be used for Memory Streams and FileInfo objects (byte arrays would be a natural addition, but unfortunately Visual Studio 2005 doesn't support debugger visualizers for array types). For the Memory Stream, the visualizer displays the contained bytes in the ByteViewer by passing the retrieved byte array to the viewer's SetBytes method. For FileInfo objects, it passes the FileInfo's FullName property to the SetFile method.

Figure 3 shows my implementation. To implement a simple visualizer like this one, you need to create a class that derives from DialogDebuggerVisualizer, a class contained in Microsoft.VisualStudio.DebuggerVisualizers.dll, which is installed with Visual Studio 2005. The derived class overrides the Show method, which accepts two parameters.

Figure 3 ByteArrayVisualizer (Close Figure 3)
using System;
using System.IO;
using System.Windows.Forms;
using System.ComponentModel.Design;
using Microsoft.VisualStudio.DebuggerVisualizers;

[assembly: System.Diagnostics.DebuggerVisualizer(
    typeof(ByteArrayVisualizer),
    typeof(VisualizerObjectSource), Target = typeof(FileInfo),
    Description = ByteArrayVisualizer.Description)]
[assembly: System.Diagnostics.DebuggerVisualizer(
    typeof(ByteArrayVisualizer),
    typeof(VisualizerObjectSource), Target = typeof(MemoryStream),
    Description = ByteArrayVisualizer.Description)]

public class ByteArrayVisualizer : DialogDebuggerVisualizer
{
    public const string Description = "Byte Array Visualizer";

    protected override void Show(
        IDialogVisualizerService windowService,
        IVisualizerObjectProvider objectProvider)
    {
        object data = objectProvider.GetObject();

        using (Form f = new Form())
        using (ByteViewer viewer = new ByteViewer())
        {
            if (data is FileInfo)
                viewer.SetFile(((FileInfo)data).FullName);
            else if (data is MemoryStream)
                viewer.SetBytes(((MemoryStream)data).ToArray());
            else return;

            viewer.SetDisplayMode(DisplayMode.Hexdump);
            viewer.Dock = DockStyle.Fill;

            f.FormBorderStyle = FormBorderStyle.SizableToolWindow;
            f.Text = "Byte Array Viewer";
            f.ClientSize = viewer.Size;
            f.Controls.Add(viewer);

            windowService.ShowDialog(f);
        }
    }
}
Figure 4 Selecting a Visualizer in Visual Studio
Figure 4 Selecting a Visualizer in Visual Studio

The first parameter is of type IDialogVisualizerService; its ShowDialog method can be used to display a CommonDialog or a Control in the debugger. The second parameter is of type IVisualizerObjectProvider, and can be used to retrieve the data for the object to be visualized. Once the Show method has been implemented to display the necessary information, the only code left to write is the assembly attributes that inform Visual Studio how and where this visualizer can be used. As my visualizer can be used with two different types, I have two DebuggerVisualizerAttribute instances at the assembly level, informing the debugger that ByteArrayVisualizer should be usable for the two corresponding types. After compiling the code, copy the compiled assembly to your My Documents\Visual Studio\Visualizers directory, and it'll be available the next time you run Visual Studio 2005 (see Figure 4 and Figure 5).

Figure 5 Byte Array Visualizer in Action
Figure 5 Byte Array Visualizer in Action

Q I'd like to use DirectoryInfo.GetFiles to enumerate all of the files in a hierarchy of folders, but I have a few concerns. First, GetFiles only appears to enumerate the files in the specified folder itself and not those within subfolders. Second, I only need to do a little work for each of the files, but in some of my directories I have thousands of files, and it seems like a waste to load them all into an array before processing any. Suggestions?

A The .NET Framework 2.0 already addresses your first concern. A new overload has been added to Directory and DirectoryInfo that accepts a value from the new SearchOption enumeration. SearchOption exposes two values: TopDirectoryOnly and AllDirectories. The 1.x behavior you're used to is equivalent to specifying SearchOption.TopDirectoryOnly; specifying AllDirectories causes GetFiles to traverse into subdirectories as well. So, for example, to find all of the files on your C drive, you could write code like:

FileInfo [] files = Directory.GetFiles(
   "C:\\", "*.*", SearchOptions.AllDirectories);

Of course, as you point out, this will end up finding all files on your disk and creating an array out of all of them before it returns any to you. Moreover, if you don't have access to some of those subdirectories, the whole operation could result in an exception—that one forbidden directory preventing you from retrieving a list of any of them.

I'll show you how to overcome some of these obstacles, but before continuing, make sure these really are problems for you. Redeveloping Directory.GetFiles might be a fun learning project, but the code is actually quite difficult to get right, given the myriad of security issues you need to be concerned about.

Directory.GetFiles relies on the Win32® functions FindFirstFile, FindNextFile, and FindClose, all of which are exposed from kernel32.dll. FindFirstFile searches a directory for the first file whose name matches the specified pattern. It returns a handle that allows for further files to be enumerated by the FindNextFile function, and that handle can be closed using FindClose. My implementation also uses these functions. In order to ensure that the handle returned from FindFirstFile is properly released, I wrap it in a SafeHandle (for more information on safe handles, see my article on reliability in the October 2005 issue of MSDN®Magazine, available at High Availability: Keep Your Code Running with the Reliability Features of the .NET Framework). My SafeFindHandle is shown in Figure 6, and is used in the FindFirstFile and FindNextFile signatures as follows:

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
static extern SafeFindHandle FindFirstFile(string lpFileName,
    [In, Out, MarshalAs(UnmanagedType.LPStruct)]
    WIN32_FIND_DATA lpFindFileData);

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
static extern bool FindNextFile(SafeFindHandle hndFindFile,
    [In, Out, MarshalAs(UnmanagedType.LPStruct)]
    WIN32_FIND_DATA lpFindFileData);

Figure 6 SafeHandle for Use with FindClose (Close Figure 6)
sealed class SafeFindHandle : SafeHandleZeroOrMinusOneIsInvalid
{
    [SecurityPermission(SecurityAction.LinkDemand, UnmanagedCode=true)]
    private SafeFindHandle() : base(true) { }

    protected override bool ReleaseHandle()
    {
        return FindClose(this.handle);
    }

    [DllImport("kernel32.dll")]
    [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
    private static extern bool FindClose(IntPtr handle);
}

In order to allow intermediate results to be returned, I've implemented my GetFiles as a C# iterator, as shown in Figure 7. This allows me to yield results as I come across them, with the C# compiler implementing the necessary state machine rather than forcing me to do it manually. To begin, a stack of DirectoryInfo objects is created, and the initial directory is pushed onto it. In this fashion, I maintain an explicit stack of directories to be examined, including subdirectories, rather than using recursion, which can add an extra expense if the file hierarchy is particularly deep (for more information, see Recursive iterator performance, part 3). While there are still directories to be examined, I pull the top one off the stack and enumerate all of the files it contains.

Figure 7 Iterator Using FindFirstFile and FindNextFile (Close Figure 7)
public static IEnumerable<FileInfo> GetFiles(
    DirectoryInfo dir, string pattern, SearchOption searchOption)
{
    if (dir == null) throw new ArgumentNullException("dir");
    if (pattern == null) throw new ArgumentNullException("pattern");

    WIN32_FIND_DATA findData = new WIN32_FIND_DATA();
    Stack<DirectoryInfo> directories = new Stack<DirectoryInfo>();
    directories.Push(dir);

    ErrorModes origErrorMode = SetErrorMode(ErrorModes.FailCriticalErrors);
    try
    {
        while (directories.Count > 0)
        {
            dir = directories.Pop();
            string dirPath = dir.FullName.Trim();
            if (dirPath.Length == 0) continue;
            char lastChar = dirPath[dirPath.Length - 1];
            if (lastChar != Path.DirectorySeparatorChar &&
                lastChar != Path.AltDirectorySeparatorChar)
            {
                dirPath += Path.DirectorySeparatorChar;
            }

            SafeFindHandle handle = FindFirstFile(
                dirPath + pattern, findData);
            if (handle.IsInvalid)
            {
                Errors error = (Errors)Marshal.GetLastWin32Error();
                if (error == Errors.AccessDenied ||
                    error == Errors.FileNotFound) continue;
                else throw new Win32Exception((int)error);
            }
            else
            {
                try
                {
                    do
                    {
                        if ((findData.dwFileAttributes &
                                FileAttributes.Directory) == 0)
                        {
                            yield return new FileInfo(
                                dirPath + findData.cFileName);
                        }
                    }
                    while (FindNextFile(handle, findData));
                    Errors error = (Errors) Marshal.GetLastWin32Error();
                    if (error != Errors.NoMoreFiles)
                        throw new Win32Exception((int)error);
                }
                finally { handle.Dispose(); }
            }

            if (searchOption == SearchOption.AllDirectories)
            {
                foreach (DirectoryInfo childDir in dir.GetDirectories())
                {
                    if ((File.GetAttributes(childDir.FullName) &
                        FileAttributes.ReparsePoint) == 0)
                    {
                        directories.Push(childDir);
                    }
                }
            }
        }
    }
    finally { SetErrorMode(origErrorMode); }
}

First, I ensure that the directory name ends with a path separator character (typically a '\'). I then pass this path along with the search pattern to the FindFirstFile function, getting back a SafeFindHandle. If the handle is invalid, then I examine the failure that caused this result. If FindFirstFile failed because it was denied access to the directory, I simply ignore this directory and continue to examine any others I might have come across. If it failed for any other reason, I abort the operation with an exception.

If the handle is valid, then FindFirstFile will have filled the WIN32_FIND_DATA structure I passed to it. After validating that the file found was not in fact a directory, I wrap the path in a FileInfo and yield it to the caller of this iterator. Then, while FindNextFile returns true (meaning that it continues to find files), I continually yield the relevant FileInfo objects.

When FindNextFile eventually returns false, I verify that it stopped because it ran out of files, and not because of some other error. At this point, the SafeFindHandle is no longer needed and can be disposed (if an exception were to occur that prevented the finally block containing the call to Dispose from running, the SafeFindHandle's critical finalizer would ensure it was eventually cleaned up after).

Now, if the user asked for all the directories' files to be enumerated rather than just the top-level directory's files, the DirectoryInfo's GetDirectories method would be used to retrieve a list of all of the directories (this could have been done explicitly using FindFirstFile and FindNextFile, as was done for files), each of which is added to the stack of directories to be examined. The only ones that aren't added are reparse points. Reparse points are collections of user-defined data and are used to implement NTFS links. These can cause cycles to occur in directory structures, so to avoid stumbling across one and ending up in an infinite operation, I've ignored them completely.

You'll notice I'm also making use of the SetErrorMode function exported from kernel32.dll. This is to prevent the system from directly notifying the user of the application when a serious I/O related error has occurred, such as when the code attempts to enumerate the files on a drive that is unavailable. For more information, see my .NET Matters column from the January 2005 issue of MSDN Magazine.

With that in place, my implementation is complete. To take advantage of it, I can simply use it in a foreach loop:

foreach (FileInfo fi in FileSearcher.GetFiles(
    new DirectoryInfo(@"C:\"), "*.*", SearchOption.AllDirectories))
{
    Console.WriteLine(fi.FullName);
}

This implementation does have some advantages. First and foremost, it allows you access to each file path as it is discovered, rather than forcing you to wait until all files have been found. Second, it doesn't bail when a directory is encountered that can't be enumerated, although an exception might be the behavior you desire in that situation. Third, you can modify this implementation to account for your particular needs; for example, by using FindFirstFileEx instead of FindFirstFile if you need more control over filtering, or by specifying an additional search pattern that should be used to determine which subdirectories to examine (rather than examining all of them).

However, there are disadvantages to this approach, and I strongly urge you to consider using Directory.GetFiles or DirectoryInfo.GetFiles if they work for your application's needs. First, the classes in System.IO have been optimized and tested for years by millions of applications using the .NET Framework. Second, I make no attempt to incorporate code access security (CAS) support into my implementation as well as the Framework has. When you use Directory.GetFiles or DirectoryInfo.GetFiles, your application only needs the relevant FileIOPermissions for the operation. My implementation requires UnmanagedCode SecurityPermission, a much more privileged permission that is rarely granted to anything but full-trust applications. I could assert that permission and then demand the appropriate file permissions, but in order to do that I would need to perform a significant amount of validation on the supplied directory and search paths in order to ensure that the user is not circumventing CAS. The classes in System.IO do this, and they do it very well.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


Pure C++: Reflecting on Generic Types

Pure C++
Reflecting on Generic Types
Stanley B. Lippman

Contents
Type Information in .NET
IsA/HasA Queries
Get Retrievals

A funny thing happened to templates on their way to the common language runtime (CLR)—they lost their {type} identity. This is analogous to what happens with macros under native programs. Just as the C/C++ compilers have no awareness of macro preprocessor expansions, the CLR has no awareness of template instantiations. In both cases, the expansions are baked into the data stream being processed, and all unique {type} identity is lost. (This has some design ramifications that I will address in a subsequent column when I look at STL/CLR, the Standard Template Library rearchitected for Visual C++® 2005.)

Generics, on the other hand, are directly supported by the runtime. Extensions to the Common Intermediate Language (CIL) explicitly support the specification of a generic type, as well as its type parameters, constraints, and so on. And an extension to the Reflection type facilities of the Base Class Libraries of the System namespace allow full reflection capabilities on generic types and object instances. Moreover, the runtime automatically handles the instantiation of generic instances in an optimal way. In this column, I provide an introduction to reflection and support for generic reflection in the .NET Framework 2.0.

Type Information in .NET

Most compiler writers, at some point in their careers, come to the realization that a good part of each day is spent figuring out how to be smarter about translating a program, building up a rich in-core program representation to support those smarts, and finally tossing all that information away when finished. Under .NET, however, this in-core program representation is persisted in the form of a standardized metadata specification, and is available both to the runtime (which has full access to the CIL and the generated metadata) and to you at execution time through run time discovery, known as reflection.

For example, when you write the following:

enum class test { fail, succeed, untested, rerun };
the underlying System::Enum class provides static methods through which you can access the enumerator names, their associated values, and so on. In order to print out the enumerator names by ascending order of the corresponding integral value, you can write this:
for each( String ^s in Enum::GetNames(
   status::typeid ))
Console::WriteLine( s );

It is worth taking a moment to think about what is happening here with the invocation of Enum::GetNames, rather than just jotting down how it's done. What you want to retrieve is the name of the enumerations of your enum status. It would not make practical sense to associate those names with an instance of status, which only needs to hold an integral value. Rather, the names and values of the enumerators are invariant and pertain to all instances of the enum you have defined. This represents a form of metadata. You only need to store it once, and the class through which this metadata is stored is System::Type—the System::Type that is associated with your enum status.

There are a number of ways of accessing the Type object associated with a type. T::typeid is the C++/CLI extension of the native RTTI (C++ Run-Time Type Information) typeid operator that returns the Type object associated with the specified type T—in this case, the status enum. This use of metadata through run-time access of Type objects is called, as I mentioned before, reflection.

If you have an object of some type, its associated Type is retrieved through the GetType instance method. For example, here is a general method to retrieve an object's associated Type and display both its type name and fully qualified type name:

void DisplayType( Object^ o )
{
    if ( !o ) return;

    Type^ t = o->GetType();
    Console::WriteLine( "Type is {0} : {1}",
                         t->Name, t->FullName );
}

Figure 1 shows code that passes various types to the DisplayType method, including a class generic type. This generates the output shown in Figure 2. (Note that I've cleaned up the line-spill a bit and annotated the results with comments. Also remember that the output may not be exactly as will be seen with the final product release.) Notice that the fully qualified name for a Type displays the scope operator as dot (.), the intermediate language (IL) scope operator, rather than as the C++ double-colon (::). Also take note of the full system information available for the generic type.

Figure 2 Results from DisplayType (Close Figure 2)

// Int32 i1 = 1024; DisplayType( i1 );
Type is Int32 : System.Int32
// String^ s1 = "Foo"; DisplayType( s1 );
Type is String : System.String
// array<bool>^ bits; DisplayType( bits );
Type is Boolean[] : System.Boolean[]
// DisplayType( gcnew Container<int> );
Type is Container`1 : templates.Container`1[[System.Int32, mscorlib,
Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]
// DisplayType( gcnew Container<String^> );
Type is Container`1 : templates.Container`1[[System.String, mscorlib,
Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]
Figure 1 Passing Code to DisplayType (Close Figure 1)
namespace foobar
{
    generic <class T> ref class Container{};
}

using namespace forbar;
int main()
{
    Int32 i1 = 1024;
    String^ s1 = "Foo";
    array<bool>^ bits = gcnew array<bool>( i1 );

    DisplayType( i1 );
    DisplayType( s1 );
    DisplayType( bits );

    DisplayType( gcnew Container<int> );
    DisplayType( gcnew Container<String^> );
}

If you do not have an object through which to grab the associated Type object, you can explicitly retrieve it using the typeid operator as follows:

Type^ ts = String::typeid;
Type^ ti = Int32::typeid;
Of course, if you have not specified a using statement for the relevant namespace, you will need to qualify the name of a type located in that namespace. For example:
Type^ tsb = System::Text::StringBuilder::typeid;

If you are unsure if the actual type exists, then you cannot actually use the explicit typeid operator. Rather, you can use the static GetType method of the Type class by passing it a string literal of the type name. You might do this, for example, to read a file or collection of type names that may or may not exist within your application (see Figure 3).

Figure 3 Finding Types with Type::GetType (Close Figure 3)
void DisplayType( String^ typeName )
{
    Type^ t = Type::GetType( typeName );

    // if null, couldn't find it...
    if ( ! t ) return;

    Console::WriteLine( "Type is {0} : {1}",
                         t->Name, t->FullName );
}

void display( array<String^>^ types )
{
    for each ( String^ ts in types )
        DisplayType( ts );
}

List<Type^>^
fetchTypes( array<String^> ^args)
{
    List<Type^> ^actualTypes = gcnew List<Type^>;

    for each ( String^ to in args )
          if ( Type^ tt = Type::GetType(ts) )
               actualTypes->Add( tt );

    return actualTypes;
}

There are a number of possible forms your literal type name can take. If you provide an unqualified name, such as the following, the type is found only if it is located at global scope:

// Unqualified name: not found if we intend System::Math ...
DisplayType( "Math" );

When you specify a fully qualified name, you must use the IL scope operator rather than that of C++. For example:

// A fully qualified name, but we can't use ::
DisplayType( "System.Math" );

Note that you cannot use this form of retrieval for a class generic or class template type. This is because the template or generic definition does not represent an actual type but only a pattern for generating instances based on actual type arguments.

A third form of string literal to retrieve a type name is to both fully qualify the name and specify the assembly within which you believe it will be found. The assembly is separated from the type by a comma, as in the following:

// A fully qualified name, and assembly
DisplayType( "System.Math, mscorlib" );

If you have an Assembly object, you can either retrieve a Type array of all the types defined within that assembly, or query for a particular type, as shown here:

array<Type^>^ types = a1->GetTypes();
Type^ query = a1->GetType( "Query" );

What if you have a Type, and want to discover all the other types defined within its assembly? Or, given an object, you want to discover which assembly its type definition is defined in? Or what if you want to know a half dozen other things about the type: is it an array, is it a class, is it a generic type, and so on? You can query and retrieve almost all of this information from a Type object.

The Type class is actually something magical; it serves as the key that unlocks the runtime reflection facility. For example, Figure 4 is a simple run-time utility that either returns the name of the type's base class, if it has one, or returns nullptr.

Figure 4 Get Base Class (Close Figure 4)
ref class TypeUtilities
{
    Assembly^ m_a;
    ...

public:

    String^ baseClass( String ^typeName )
    {
        Type^ query = m_a->GetType( typeName );
        if ( query && query->IsClass )
        {
            if ( Type^ base = query->BaseType )
                 return base->Name;
        }

        return nullptr;
    }
}

A Type supports a number of queries, such as whether the type is a class (IsClass). It returns additional type information, such as the Type of its base class, or if it should have one (BaseType). It also provides general properties of the type, such as its name (Name,FullName). I've categorized the services provided by Type into the general designations, which I'll discuss next.

IsA/HasA Queries

The IsA/HasA queries tell you whether the type is this thing or that. I used IsClass already. Other IsA properties include IsAbstract, IsArray, IsNested, IsPublic, IsSealed, IsSerializable, IsValueType, and so on. The HasA properties include HasGenericArguments, which returns true if the type has type arguments, and HasElementType, which returns true if the type holds or refers to another type—that is, if it is an array, a pointer, or is passed by reference. Figure 5 shows how you might discover a generic type at run time.

Figure 5 Discovering a Generic Type at Run Time (Close Figure 5)
void DisplayGeneric( Type^ t )
{
    if ( !t ) return;
    String^ name = t->Name;
    String^ display1 = "{0} is {1} a generic type definition!";
    String^ display2 = "{0} is {1} a generic instance!";

    if ( ! t->IsGenericTypeDefinition )
         Console::WriteLine( display1, name, " not" );
    else Console::WriteLine( display1, name,"" );

    if ( ! t->HasGenericArguments )
         Console::WriteLine( display2, name, " not" );
    else Console::WriteLine( display2, name, "" );
}

IsGenericTypeDefinition Apart from being impossible to type correctly, this returns true if the Type represents a generic definition. It returns false on an instance of the generic type, as well as on any non-generic type, including that of a template. For example, Figure 6 shows three types passed in and their associated output.

Figure 6 Using DisplayGeneric (Close Figure 6)
//    class: not a generic type definition
String^ bs = "not generic";
DisplayGeneric( bs->GetType() );

String is not a generic type definition!

// class generic: not a generic type definition
Container<String^> ^cs = gcnew Container<String^>;
DisplayGeneric( cs->GetType() );

Container`1 is not a generic type definition!

// class generic definition: ok!
Type^ gd = cs->GetType()->GetGenericTypeDefinition();
DisplayGeneric( gd );

Container`1 is a generic type definition!

HasGenericArguments This returns true if the Type represents either a generic definition or an instance of a generic type. For example, here is the output from running the three same types against this HasA property:

String^ is not a generic instance!
Container`1 is a generic instance!
Container`1 is a generic instance!

Because HasGenericArguments returns true on a generic definition and a generic instance, if you just want to handle a generic instance, you need some sort of conditional test, like this one:

if ( ! t->IsGenericTypeDefinition &&
       t->HasGenericArguments )
     // ok, a generic instance ...

GetGenericTypeDefinition represents a second category of service: retrieving a special aspect of the type.

Get Retrievals

These retrieve some specific type or collection of types that are possibly associated with this Type object. GetGenericTypeDefinition is one such method. To illustrate some of these retrieval methods (highlighted in red type) that are associated with generic types, let's augment DisplayGeneric to probe more deeply (see Figure 7).

Figure 7 Retrieval Methods (Close Figure 7)
void DisplayGeneric( Type^ t )
{
    // initial portion remains unchanged...

    if ( ! t->IsGenericTypeDefinition )
         Console::WriteLine( display1, name, " not" );
    else
    {  // let's probe the generic type definition ...

       // does it have type parameters ???
       array<Type^> ^parameters = t->GetGenericArguments();
       if ( parameters ){
          int count = parameters->Length;
          Console::WriteLine( "{0} has {1} parameters.",
                              name, count );

          for ( int ix = 0; ix < count; ++ix ){
              Type^ t = parameters[ ix ];
              Console::WriteLine( "\n\t{0} at position {1}",
                         t->Name, t-> GenericParameterPosition );

              array<Type^> ^constraints =
                    t-> GetGenericParameterConstraints();
              if ( constraints ){
                 int count = constraints->Length;
                 Console::Write("\t{0} has {1} constraint(s): ",
                            t->Name, count );
                 for ( int ix = 0; ix < count; ++ix ){
                      Type ^tt = constraints[ix];
                      Console::Write( " {0}", tt->Name );
                  }
                  Console::WriteLine();
              }
              Console::WriteLine();
           } // for each parameter
        } // if parameters
    } // else if generic definition
} // end of DisplayGeneric()...

GetGenericArguments returns an array of Type objects representing each of the generic type parameters. Each parameter element has an associated position (beginning at 0)—this is the return value of GenericParameterPosition. GetGenericParameterConstraints returns associated constraint clauses, if any, as an array of Type objects. Not being clear about any constraints on the use of this method, I mistakenly applied it to the generic definition itself rather than to the individual parameters. This results in a run-time exception:

System.InvalidOperationException: Method may only be called on a Type
for which Type.IsGenericParameter is true.

In any case, I changed the types to pass into this revised method. In the first case, I added three constraints to our class generic, as shown in the following:

generic <class T>
    where T : IComparable<T>,
              System::Collections::IEnumerable, ICloneable
ref class Container{};
When I invoke the method like this
Container<String^> ^cs = gcnew Container<String^>;
Type^ gd = cs->GetType()->GetGenericTypeDefinition();
DisplayGeneric( gd );
it generates the following output:
Container`1 is a generic type definition!
Container`1 has 1 parameters.

        T at position 0
        T has 3 constraint(s):  IComparable`1 IEnumerable ICloneable

The second class generic is an instance of the SortedDictionary container located in the System::Collections::Generic namespace. Here is what the call looks like:

SortedDictionary<String^, String^>^ gds =
      gcnew SortedDictionary<String^, String^>;
Type^ gdst = gds->GetType()->GetGenericTypeDefinition();
DisplayGeneric( gdst );
And here is the somewhat surprising output (I had expected a larger, more specific constraint set):
SortedDictionary`2 is a generic type definition!
SortedDictionary`2 has 2 parameters.

        K at position 0
        K has 1 constraint(s) :  Object

        V at position 1
        V has 1 constraint(s) :  Object

As I mentioned at the outset of this column, the next (and final) column on parameterized types under Visual C++ 2005 will look at the design interoperability of templates and generics using our STL/CLR library as a model. Until then, may your programs execute without bugs.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman is an Architect with the Visual C++ team at Microsoft. He began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. In between, he worked in feature animation at Disney and DreamWorks, was a Distinguished Consultant with JPL, and was a Software Technical Director on Fantasia 2000.


C++ at Work: Layered Windows, Blending Images

C++ at Work
Layered Windows, Blending Images
Paul DiLascia

Code download available at: CatWork0512.exe (493KB)

Q Is there a way that I can create a window that's 50 percent transparent and passes all mouse events through to the desktop or another app window beneath mine?

Scott Stringham

A Why yes, and it's fairly easy. All you have to do is create a "layered window." I wrote a little program called lwtest (layered window test) that shows how. You can download it from the MSDN®Magazine Web site. To create a layered window, you need the extended style WS_EX_LAYERED, and to make the window transparent to mouse clicks, you need WS_EX_TRANSPARENT. You can set both styles, after the window has been created, in your OnCreate handler. In MFC it looks like this:

int CMainFrame::OnCreate(...)
{
   ...
   ModifyStyleEx(0, WS_EX_LAYERED|WS_EX_TRANSPARENT);
}

ModifyStyle and ModifyStyleEx are special MFC CWnd methods that do what their names suggest. If you're writing in C, you'd call GetWindowLong(GWL_EXSTYLE) to get the extended style, then tweak it and call SetWindowLong(GWL_EXSTYLE) to set it. This is, in effect, what ModifyStyle(Ex) does. Or, of course, you can use the styles when you create your window.

Once you've created a layered window, you can call SetLayeredWindowAttributes to set the transparency. One of the layered window attributes you can set is LWA_ALPHA, which is the transparency. The transparency is measured from zero (totally transparent) to 255 (opaque). For 50 percent transparency you'd call SetLayeredWindowAttributes, like so:

// in CMainFrame::OnCreate
SetLayeredWindowAttributes(0, 255 * 0.50, LWA_ALPHA);

I've used the multiplication to highlight the general formula; you could just use 128 since that's half of 255 (rounded). You can also use a specific color as the transparent color. In that case, you'd use LWA_COLORKEY as the attribute, and specify a COLORREF in the first parameter. Windows® makes all pixels that have that color transparent. Note that the preceding snippet assumes you're calling from within a CWnd-derived object. In C, you'd use ::SetLayeredWindowAttributes, which takes the HWND as an additional first parameter.

You can use layered windows to do animations and other transition effects; for additional details, see the section entitled "Layered Windows" in the documentation.

Q I'm writing a slide show program that shows a sequence of JPEG images. I used your CPicture class from the March 2002 column to draw the images (see C++ Q&A: Do You Have a License for that GIF? PreSubclassWindow, EOF in MFC, and More). That part works fine. But now I'd like to add a feature that fades from one image to the next. I can do this in a Web page using transition effects. I'm wondering if there's any way to access this from code? Or is there some other way to fade one image into another?

Bob Kline

A Indeed you can access the Internet Explorer transition effects using COM. These effects—which include fade, wipe, box in, box out, bars, and all the others—are supported in DirectX®. The details are beyond the scope of this column, so I can only refer you to the documentation, which is in a section called "Using Transforms in C++" under Internet Development. You'll need some familiarity with COM as well as some basic knowledge of DirectX concepts like surfaces and transforms (DXSurface and DXTransform).

But if all you want is a simple blend from one image to the next, I can show you how to do it with the GDI+ function AlphaBlend, which the Redmondtonians were friendly enough to wrap in MFC as CDC::AlphaBlend. The alpha in AlphaBlend is a graphics term. Typically, bitmaps use 3 bytes to specify a pixel: 1 byte each for red, green, and blue color values. Since a 32-bit DWORD has 4 bytes, that leaves one extra byte by my count (4-3=1).The extra byte is often used as the "alpha channel," used to specify the pixel's transparency. The alpha value is used to merge pixels according to the following formula:

[R,G,B]blended = ?[R,G,B]image + (1-?? [R,G,B]background

When alpha is zero, you get the background (image is fully transparent); when alpha is 1, you get the image (fully opaque). In practice, using an 8-bit byte, the alpha value ranges from 0-255 instead of 0-1, where zero is transparent and 255 is opaque. You can use per-pixel alpha values, but most applications don't need it; most blending applications apply a constant alpha value to a whole object such as an image. For example, you might want a particular image to appear 25 percent transparent.

The AlphaBlend function is like the old familiar functions BitBlt and StretchBlt, but it does blending. Pronounced "blit," the term derives from the ancient PDP-10 BLT (block transfer) instruction used to transfer a large block of memory from one location to another. Figure 1 shows the AlphaBlend function in detail. The parameters are straightforward, but using AlphaBlend to do blending can be a chore because you can't just call it once, you have to call it repeatedly to gradually blend one image to the next, using a timer and different alpha values ranging from 0 to 255.

Figure 1 AlphaBlend Function and Parameters (Close Figure 1)
// MFC wraps the GDI+ function ::AlphaBlend, which has the
// destination HDC as the first parameter.

// From afxwin.h:
class CDC : public CObject
{
   BOOL AlphaBlend(int xDest, // destination: x
      int yDest,              // destination: y
      int nDestWidth,         // destination: width
      int nDestHeight,        // destination: height
      CDC* pSrcDC,            // source device context
      int xSrc,               // source: x
      int ySrc,               // source: y
      int nSrcWidth,          // source: width
      int nSrcHeight,         // source: height
     BLENDFUNCTION blend);    // see below
   ...
};

// From wingdi.h:
typedef struct _BLENDFUNCTION {
   BYTE BlendOp;              // op code: must be AC_SRC_OVER
   BYTE BlendFlags;           // flags must be zero
   BYTE SourceConstantAlpha;  // alpha value, 0 to 255
   BYTE AlphaFormat;          // AC_SRC_ALPHA for per-pixel alpha; else,
                              // 0
} BLENDFUNCTION,*PBLENDFUNCTION;

To show how AlphaBlend works in practice, I wrote a program called BlendView based on my image viewer from the March 2002 column. BlendView lets you view image files (BMP, JPG, GIF, or any other format supported by GDI+), but when you open a new image, the view gradually blends the new image with the old, as the sequence in Figure 2 shows.

Figure 2 Original Image
Figure 2 Original Image
Figure 2 Blended Image
Figure 2 Blended Image
Figure 2 Final Image
Figure 2 Final Image

To blend one image with another, you need both images. Duh. But when the user opens a new document, the first thing MFC does is destroy the old one. So you need to make some provision to save the old image somewhere before MFC loads the new one. Since the blending effect is conceptually part of the view (it falls under the category of painting), I want the view to handle this. That is, I want the view to save the old image. But how does the view know when to save it? You have to tell it, of course. Fortunately, CDocument has a method you can use to notify your view(s) whenever something important happens.It's CDocument::UpdateAllViews. Here's the code:

// in Doc.cpp:
BOOL CPictureDoc::OnOpenDocument(LPCTSTR lpszPathName)
{
   UpdateAllViews(NULL, PREOPENDOC, this);
   return m_pict.Load(lpszPathName);
}
PREOPENDOC is my own enum code defined in doc.h. When you call UpdateAllViews, you pass your own "hint code" (a 32-bit integer) along with a pointer to a "hint object," which can be any CObject-derived MFC class. In this case I pass the document itself. Note that I call UpdateAllViews just before loading the new image, while the old image is still valid. The view handles the notification by saving the image:
void CPictureView::OnUpdate(CView* pSender,
   LPARAM lHint, CObject* pHint)
{
   if (lHint==CPictureDoc::PREOPENDOC) {
      SaveDocImage((CPictureDoc*)pHint);
   }
}

The same OnUpdate function handles all document notifications, so you have to check which notification code was sent. In general, the idea behind the hint code and hint object is that the document provides information in the form of a hint that tells the view what part of the screen it needs to update. For CPictureView, if the hint code is PREOPENDOC, CPictureView calls a helper function SaveDocImage to save the current image. Figure 3 shows the code for SaveDocImage. It creates a bitmap and memory device context (DC) then renders the image into the memory device context, effectively copying it for blending after the document destroys the original image.

Figure 3 SaveDocImage (Close Figure 3)
#include "StdAfx.h"
#include "View.h"
#include "resource.h"

// total duration to perform alpha blend.
const BLEND_DURATION = 3 * CLOCKS_PER_SEC; // 3 seconds

////////////////////////////////////////////////////////////////
// CPictureView
//
IMPLEMENT_DYNCREATE(CPictureView, CScrollView)
BEGIN_MESSAGE_MAP(CPictureView, CScrollView)
   ...
   ON_COMMAND(ID_VIEW_ROTATE, OnViewRotate)
END_MESSAGE_MAP()

CPictureView::CPictureView()
{
   m_iHowScale = ID_VIEW_TOFIT;         // default scale mode: to fit
   m_iStartTime = 0;                    // no blending
   m_szBlend = CSize(0,0);
}

CPictureView::~CPictureView()
{
   DeleteTempBitmap();
}

//////////////////
// OnInitialUpdate - called by Framework when view is first created or
// the user opens a new document. This is the place to perform one-time
// per document initialization.
//
void CPictureView::OnInitialUpdate()
{
   CScrollView::OnInitialUpdate();      // default calls OnUpdate
   SetScrollSizes();                    // set scroll sizes from image
}

//////////////////
// OnUpdate - called by the Framework when you invoke
// CDocument::UpdateAllViews. UpdateAllViews can pass a hint code and hint
// object. CPictureDoc sends hint code CPictureDoc::PREOPENDOC just before
// it's about to open a document. This gives the view a chance to save the
// image (for blending) before the image file is closed.
//
void CPictureView::OnUpdate(CView* pSender, LPARAM lHint, CObject* pHint)
{
   if (lHint==CPictureDoc::PREOPENDOC) {
      SaveDocImage((CPictureDoc*)pHint); // save document image
      m_iStartTime = clock();            // note start time
   }
}

//////////////////
// Helper to save the current document image. Called just before
// opening new document. Create memory bitmap and render old
// image into it.
//
void CPictureView::SaveDocImage(CPictureDoc* pDoc)
{
   ASSERT(pDoc);
   CPicture* ppic = pDoc->GetPicture();
   ASSERT(ppic);

   DeleteDocImage();                    // delete old doc image

   CRect rc;
   GetImageRect(rc);
   CSize sz = rc.Size();

   CClientDC dc(this);                  // my client DC
   m_dcOld.CreateCompatibleDC(&dc);     // create memory DC
   m_bmOld.CreateCompatibleBitmap(&dc, sz.cx, sz.cy); // create bitmap
   m_dcOld.SelectObject(&m_bmOld);      // select it into DC
   ppic->Render(&m_dcOld,rc);           // draw picture on bitmap
   m_szOld = sz;                        // and save size
}

//////////////////
// Delete (old) document image.
//
void CPictureView::DeleteDocImage()
{
   if (HaveDocImage()) {
      m_dcOld.SelectObject((CBitmap*)NULL); // de-select bitmap from DC
      m_bmOld.DeleteObject();               // delete bitmap
      m_dcOld.DeleteDC();                   // delete DC
   }
}

//////////////////
// Draw the picture -- call CPicture to do it.
//
void CPictureView::OnDraw(CDC* pDC)
{
   CPictureDoc* pDoc = GetDocument();
   ASSERT_VALID(pDoc);
   CPicture* ppic = pDoc->GetPicture();
   ASSERT(ppic);

   CRect rc;
   GetImageRect(rc);

   if (m_iStartTime) {
      CSize sz = rc.Size();

      // if window size changed, create new blend bitmap.
      if (m_szBlend != sz)
         CreateTempBitmap(pDC, sz);
      ASSERT(m_dcBlend.m_hDC);

      // render current (NEW) image into temporary dc.
      ppic->Render(&m_dcBlend,rc);

      // Compute blend amount: goes from 255 to zero (backwards) because I
      // am starting with the new image (instead of old). Blend amount is
      // whatever fraction of time remains to do the blend.
      BLENDFUNCTION bf;
      memset(&bf,0,sizeof(bf));
      bf.BlendOp = AC_SRC_OVER;
      int alpha = ((clock() - m_iStartTime) * 255) / BLEND_DURATION;
      alpha = max(255-alpha,0);         // alpha = 1-alpha (backwards)
      bf.SourceConstantAlpha = alpha;

      // blend old image on top of new, in temporary DC
      m_dcBlend.AlphaBlend(0,0,sz.cx,sz.cy,
         &m_dcOld, 0,0,m_szOld.cx,m_szOld.cy, bf);

      // copy scratch bits to screen
      pDC->BitBlt(0, 0, sz.cx, sz.cy, &m_dcBlend, 0, 0, SRCCOPY);

      if (alpha>0) {                    // not done yet?
         Invalidate(FALSE);             //   do it again!
      } else {                          // otherwise:
         StopBlending();                //   stop
      }

   } else if (*ppic) {
      ppic->Render(pDC,rc);             // done blending: render as normal
   }
}

//////////////////
// Create temporary bitmap and memory DC used for bliting.
// Recreated when the window size changes.
//
void CPictureView::CreateTempBitmap(CDC* pDC, CSize sz)
{
   DeleteTempBitmap();
   m_dcBlend.CreateCompatibleDC(pDC);
   m_bmBlend.CreateCompatibleBitmap(pDC, sz.cx, sz.cy);
   m_dcBlend.SelectObject(&m_bmBlend);
   m_szBlend = sz;
}

/////////////////
// Delete temporary bitmap and memory DC.
//
void CPictureView::DeleteTempBitmap()
{
   if (m_dcBlend.m_hDC) {
      m_dcBlend.SelectObject((HBITMAP)NULL);
      m_dcBlend.DeleteDC();
      m_bmBlend.DeleteObject();
   }
}

//////////////////
// Stop blending: delete old doc image and reset start time.
//
void CPictureView::StopBlending()
{
   DeleteDocImage();       // delete old doc image
   m_iStartTime = 0;       // don't blend.
}

Now, when the user opens a new document, the document notifies the view, and the OnUpdate handler saves the image in a bitmap. How to do the blending? This requires calling AlphaBlend repeatedly to blend the new image with the old. The most obvious way to do it is to set a timer. Say you want the blend to take three seconds. To perform the blend in 100 steps, you'd set a timer for 3000/100 = 30 ms. The problem is that it takes a relatively significant amount of time to actually do the AlphaBlend. Moreover, the amount of time it takes depends on the size of the image. Larger images take longer to blend. If you use the timer approach, you'll end up with a slide show that blends smaller images more quickly and larger images more slowly—probably not what you want.

The way to keep the blend-time constant is to fix the total duration—say, 3,000 ms—and then calculate the alpha value based on how much time has actually elapsed. For example, suppose the first iteration happens at time t+20 ms. Then you'd use an alpha value of 20*255/3000 = 1 (rounded down to the nearest integer). Then you immediately do another blend using whatever alpha value you calculate based on what time it is now. If half the time has elapsed, you'll end up with an alpha of .5 (128 as a byte). By calculating the alpha value from the actual elapsed time, you guarantee that the blend always finishes on time. The drawback is that larger images won't blend as smoothly because they'll end up with fewer iterations, as more time is consumed in AlphaBlend. Got it?

Implementing all this is more or less straightforward. Figure 3 and Figure 4 show the details. When CPictureView::OnUpdate gets the PREOPENDOC notification, after saving the old image, it sets a data member m_iStartTime to the current clock time. The clock time is the number of "clock ticks" since the process started. There are CLOCKS_PER_SEC (usually 1,000) ticks per second. When OnUpdate returns, control passes back to the document and MFC, which calls your view's OnInitialUpdate function, which calls OnUpdate, which invalidates the window. Eventually, Windows sends your view a WM_PAINT message, which MFC handles by calling the view's virtual OnDraw method. This is MFC 101: in a view, you do the painting in OnDraw, not OnPaint. CPictureView handles painting like so:

void CPictureView::OnDraw(CDC* pDC)
{
   CPicture* ppic = // get current picture
   if (m_iStartTime) {
      // do blend
   } else {
      // render as normal
      ppic->Render(pDC,rc);
   }
}

Figure 4 CPictureView (Close Figure 4)
#include "Doc.h"

/////////////////
// Picture view is a typical scroll view.
// Modified to implement alpha blend when user opens a new image.
//
class CPictureView : public CScrollView {
protected:
   // stuff used for blending:
   CBitmap  m_bmOld;          // old bitmap..
   CDC      m_dcOld;          // ..device context
   CSize    m_szOld;          // ..and image size
   int      m_iStartTime;     // clock time for start of blending

   // scratch DC in which to blend
   CDC      m_dcBlend; // device context
   CBitmap  m_bmBlend; // bitmap
   CSize    m_szBlend; // size
   ...
   // helpers
   BOOL HaveDocImage() { return m_dcOld.m_hDC!=NULL; }
   void SaveDocImage(CPictureDoc* pDoc);
   void DeleteDocImage();
   void CreateTempBitmap(CDC* pDC, CSize sz);
   void DeleteTempBitmap();
   void StopBlending();

   virtual void OnUpdate(CView* pSender, LPARAM lHint, CObject* pHint);
   virtual void OnDraw(CDC* pDC);
   virtual void OnInitialUpdate();

   // command/message handlers
   afx_msg LRESULT OnDoBlend(WPARAM wp, LPARAM lp);

   ...
};

I've omitted the details of blending to highlight how CPictureView uses m_iStartTime as a flag to determine whether or not to blend. Here are the basic steps required to perform a blend.

  1. Create a memory DC.
  2. Draw bitmap A into the memory DC.
  3. Calculate alpha value for blending.
  4. AlphaBlend bitmap B into the memory DC.
  5. Copy the result to the screen (BitBlt).

It's important to do the blending off-screen in a memory DC and then copy to the screen; otherwise the user will see the intermediate images flash by. Since AlphaBlend requires a device context, not a CPicture object, it's more convenient to draw the new image first (so I can call CPicture::Render) and then blend the old one on top of it. So the alpha value I use is the inverse (1-alpha) of the alpha value I'd use if I were starting with the old image. In other words, instead of starting with the old image and blending more and more of the new one on top, I start with the new image and blend less and less of the old one on top. Pretty clever, eh? The net effect is the same. Here are the critical lines that calculate the alpha value for AlphaBlend:

int alpha = ((clock() - m_iStartTime) * 255) / BLEND_DURATION;
alpha = max(255-alpha,0);

If, after blending, the calculated alpha value is greater than zero, more blending needs to be done. So OnDraw calls Invalidate(FALSE) to invalidate the window without erasing the background. Windows sends another WM_PAINT message—but not until the current message finishes processing. By doing it this way (in effect posting the WM_PAINT), there's no blocking. The user can still use the app during blending. You can verify this by resizing the window during blending. CPictureView keeps on blending, using the new window size.

If the calculated alpha value is zero, blending is complete. Time to stop. In that case, OnDraw calls a helper function, StopBlending, which deletes the old image and sets m_iStartTime to zero as a cue for OnDraw to stop blending. Now when the view needs painting, OnDraw paints the normal way, by calling CPicture::Render to render the new image directly, without blending.

If you're using the Active Template Library (ATL) CImage class to hold your image (instead of CPicture, which I implemented long ago before CImage was available), you can use CImage::AlphaBlend, which comes in a few overloaded flavors, to do the blending.

If you're using the Microsoft® .NET Framework, you can do alpha blending with one of the Graphics.DrawImage overloads that takes an ImageAttributes structure. One of the methods in ImageAttributes is SetColorMatrix. The color matrix is a 5×5 matrix that defines the color mapping for red, green, blue, and alpha plus a fifth w channel that must be 1 on the diagonal and zero elsewhere (for math geeks, the fifth channel is used to perform nonlinear transformations). To perform a 50 percent blend, you'd use the identity matrix (all values 0, except 1's on the diagonal) and then set the alpha value (ColorMatrix.Matrix33) to .5f and use this to draw.

Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site.


{ End Bracket }: Trustworthy Software

{ End Bracket }
Trustworthy Software
David S. Platt

The Authenticode dialog box that users see when they download software from the Internet, asking them if they trust the publisher and want to install the software is, in my opinion, a useless and annoying waste of time that provides no safety whatsoever. If you know which vendor you want to trust, this mechanism ensures that it comes from them. But that's not the main problem users encounter on the Internet. Instead, it's usually, "I never heard of you until I googled for [whatever]; now I'm reading your site and wouldn't mind trying your code, but I have no good way of determining whether to trust you or not." This box doesn't answer that question or even help in any meaningful way.

How can a user possibly know enough to answer that question? He can't, he doesn't, and he never will. Depending on the user to do it properly, and washing our hands of responsibility because he gets it wrong, are not acceptable strategies in today's software environment.

How can we tell a user whether some app he's never seen before is trustworthy or not? Mature industries have solved this problem by delegating the testing to a knowledgeable third party whom customers trust. For example, when you buy food labeled "organic," how do you know that it actually is? Unless you're an expert in food production and spend essentially all of your life examining food, you cannot reliably make such a determination. Instead, you look for the certification seal from an agency you trust to have the specialized knowledge and perform the necessary inspections; for example, Oregon Tilth. This architecture scales up very well. Similar third-party certification programs exist for many qualities that customers value but can't trust a manufacturer's claim for—wood from sustainably managed forests, fair trade coffee, food prepared according to a consumer's religious beliefs. The software industry needs to do the same thing.

Software could be certified either by concentrated or distributed testing. The former requires a high-powered testing laboratory, conceptually similar to Underwriters Laboratories, which certifies electrical appliances in the U.S. It would publish a code of best practices, such as using a language that forbids buffer overruns and requires an acceptable privacy policy. It would then test applications for compliance with these standards and affix its digital seal to those that meet them. The lab would charge for this service, and the vendor would add the cost to the price of the application. Probably the mere fact of having to spend thousands on compliance testing would scare off most bad guys. Perhaps, to gain customer acceptance, the testing agency would offer some sort of insurance policy that would pay if the application turned out to be untrustworthy.

The concentrated model would leave smaller vendors out in the cold because they wouldn't be able to pay the fees. The alternative would be a distributed model, similar to the way buyers rate sellers, and vice versa, on eBay. Customers who install an app could vote on whether it is trustworthy or not, and potential downloaders could examine the results of that vote when making their purchase decision. It wouldn't help the very first users, and it wouldn't be infallible. But it would be pretty good, it would be easy to implement, and it would work for small vendors and even shareware.

The system would have to be structured so that a bad guy couldn't rig it. Perhaps voting would be allowed only by numbered e-mail ballots sent to paid and registered customers. Maybe some public interest software foundation would run it, and maybe eBay or Amazon would donate the software as a sponsor opportunity. Since I am writing a book for users entitled Why Software Sucks, I propose the "Doesn't Suck Seal" for applications that pass. It'll feature a lollipop in a circle with a slash through it. There's no reason that the concentrated and distributed certification mechanisms couldn't coexist.

It's not enough to put these certifications on a Web site that users can look at, since no one ever does that. The final piece of this puzzle is to make the operating system pay attention to these third-party certifications, seamlessly downloading and installing applications that have them, and blocking those that don't. This has to be the job of the OS manufacturers.

Once we have these certification agencies in place, it's easy to have them sign applications and to have Windows check that the signature is valid. That part of the certificate mechanism has always worked properly. We just didn't know who to trust before, and now we will.

David S. Platt teaches programming for the .NET Framework at Harvard University. He was selected by Microsoft as one of their Software Legends. He is the author of nine programming books, most recently The Microsoft Platform Ahead (Microsoft Press, 2004). Reach him at www.rollthunder.com.