 MSDN Magazine October 2003

MSDN Magazine October 2003

Tablet PC: Achieve the Illusion of Handwriting on Paper When Using the Managed INK API

Creating the illusion of a pen writing on paper is no easy software task. Fortunately, the .NET Framework hosts Tablet PC extensions, which lets you create ink-aware applications for the Tablet PC. This API allows applications to draw strokes on the screen and perform a variety of tasks including document markup, storage, and transmission.This article shows you how to handle a couple of inking events as used in the InkClipboard sample. Later, it discusses how to avoid common pitfalls including too frequent redrawing, which causes the ink flow to lag behind the pen movements, diminishing the illusion of ink on paper.

Carlos C. Tapang OLAP: Build an OLAP Reporting App in ASP.NET Using SQL Server 2000 Analysis Services and Office XP

Many organizations analyze their business-critical data using Online Analytical Processing (OLAP) technology. OLAP-based data mining provides a way to query multidimensional data sets and drill down into the data to find patterns. ASP.NET and the Microsoft Office Web Components (OWC) enable Web-based OLAP reporting. The OWC controls include PivotTable and Chart components that can be embedded in a Web page and scripted by programmers. In this article, the authors build a Web-based OLAP reporting app using ASP.NET, OWC, and SQL Server 2000 Analysis Services to illustrate the process.

Jeffrey Hasan and Kenneth Tu Blogging: Design Your Own Weblog Application from Scratch Using ASP.NET, JavaScript, and OLE DB

The ASP.NET advanced templated controls, such as the DataList and DataGrid, are perfect for many data representation situations. However, when you need the flexibility to render a variety of layouts, the Repeater control is what you need. In this article the author builds a full-featured blog application to illustrate the use of the Repeater and DataList controls that render nested data in a master-detail relationship. He then discusses how to override the default implementations of these controls by adding some client-side JavaScript code that makes the blog more responsive and enhances its usability.

Marco Bellinaso Plug-Ins: Let Users Add Functionality to Your .NET Applications with Macros and Plug-Ins

Most user applications benefit from the ability to be extended by other developers. It's often easier and more efficient to extend an existing application that users are already familiar with and trained on than it is to develop one from scratch. Thus, extensibility makes your application more attractive. You can build extensibility into your application by supporting features like plug-ins or macros. This is easily accomplished using the .NET Framework even if the core application isn't a .NET Framework app. In this article, the author describes extensibility features of the .NET Framework including late binding and reflection and how to use them, along with plug-in security considerations.

Jason Clark Editor's Note: Best Practices

Whew, what a month. As we worked on this super-fabulous issue, we've simultaneously been packing everything in the office for the move to our fancy new digs a few blocks away in the heart of Midtown Manhattan.

New Stuff: Resources for Your Developer Toolbox

Excel Software has announced the availability of QuickHelp for Windows, a development tool for creating and deploying application help to Windows® 95 through Windows XP, Mac OS 9, Mac OS X, and virtually all Linux distributions.

Theresa W. Carey Web Q&A: Schema From a DataSet, Exporting SQL Data to Excel, and More

Edited by Nancy Michell Data Points: Exploring the ADO.NET DataRow

John Papa Cutting Edge: Nested Grids for Hierarchical Data

Dino Esposito The XML Files: OPENXML, XSLT Keys, Select versus Match, XPath, and More

Aaron Skonnard Advanced Basics: Enterprise Services, SQL Script Editing

Ken Spencer Basic Instincts: Deploying Assemblies

Ted Pattison .NET Column: More on Generics in the CLR

Jason Clark C++ Q&A: Getting a Menu Handle, Declaring GetParam, and Filtering File Names

Paul DiLascia Talking To…: Eric Brown Discusses SQL Server—Past, Present, and Yukon


Tablet PC: Achieve the Illusion of Handwriting on Paper When Using the Managed INK API

Tablet PC

Achieve the Illusion of Handwriting on Paper When Using the Managed INK API

Carlos C. Tapang
This article assumes you're familiar with .NET, the CLR, and C#
Level of Difficulty 1 2 3
Code download available at: TabletPC.exe (186KB)
SUMMARY

Creating the illusion of a pen writing on paper is no easy software task. Fortunately, the .NET Framework hosts Tablet PC extensions, which lets you create ink-aware applications for the Tablet PC. This API allows applications to draw strokes on the screen and perform a variety of tasks including document markup, storage, and transmission.

This article shows you how to handle a couple of inking events as used in the InkClipboard sample. Later, it discusses how to avoid common pitfalls including too frequent redrawing, which causes the ink flow to lag behind the pen movements, diminishing the illusion of ink on paper.

Contents
Let Some Ink Flow
The InkCollector Object
A Couple of Inking Events
Slow Lasso in the Ink Clipboard SDK Sample
Ink Lag
Delegating DrawLasso
Conclusion

T
he Tablet PC is proving to be a fun, exciting platform to write applications for. And creating Tablet-enabled apps is easy thanks to the introduction of a managed SDK that integrates with the Microsoft® .NET Framework. When installed, the SDK provides a number of APIs that encapsulate a lot of the functionality you'll need in a Tablet PC app.

In this article you'll have an opportunity to see how these APIs, specifically the INK API and its managed Ink object, the InkCollector, enhance Tablet PC development. You'll get a chance to use this object in a sample application—namely, the Ink Clipboard program in the Microsoft Tablet PC Platform SDK version 1.1—for a real hands-on experience. You'll also learn to avoid potential pitfalls so that ink flows smoothly and your users experience the simulation of ink on paper.

Let Some Ink Flow

Just as a good understanding of how Win32® messages work helps you write solid apps for Windows®, a good understanding of inking will help you write solid apps for the Tablet PC.

One of the more important things to remember about inking is that the tablet digitizer that measures pen movement is very accurate. Microsoft recommends that OEMs use digitizer grids that have 1,000 lines per inch. Every slight movement of the tip of the pen is detected, even when the tip is not quite touching the digitizer but is hovering over it. As the pen touches the digitizer, data other than grid position may be collected. For example, for pressure-sensitive pens, pressure data is also collected. This means that as long as the pen tip touches the digitizer, even if the pen is not moving, data may be streamed from tablet hardware, collected by the pen driver, and eventually forwarded to applications through the ink objects. This occurs roughly 133 times per second, which is the recommended sampling rate. This data rate is not particularly high compared to current tablet processor speeds, but as you will see later in this article, it has important implications for pen UI design.

The illusion of pen writing on paper is maintained at great computing cost—a cost that doesn't even include recognizing what is written, which is still too expensive to be done on the fly. As the pen tip moves over the digitizer grid, the Tablet PC fires events that exercise both user and system-level code on the input side. On the output side, computations are even more complicated because the screen image has to be accessed for every event. This level of computing would have overwhelmed even the fastest processors a couple of years ago. Luckily for the growing group of Tablet PC aficionados, processors designed for mobile systems can now handle the load. For the application programmer, one tip to remember is to reduce the amount of redrawing or repaints that occur. It is easy to end up repainting more than necessary, which results in "ink lag," an effect that kills the illusion of pen-on-paper. Ink lag occurs when either the ink "flows" while the pen tip is not moving or it trails far behind the tip when the pen is moved fast.

Ink lag is particularly noticeable for tablet orientations other than primary landscape. This is because more computations occur per graphics access in primary portrait, secondary portrait, and secondary landscape modes than in the original, primary landscape mode. The Tablet PC is built on legacy PC graphics hardware, which is designed to work in only one orientation. Other orientations are implemented by applying a transform to the main display buffer at the driver level, increasing the lag (see the sidebar "Tablet PC Display Orientation").

The InkCollector Object

The Tablet PC API makes it possible to write ink-aware apps. This API introduces the concept of ink that allows applications to not only draw strokes on the screen, but also to perform a number of operations on the ink, such as marking up a document, storing it, or transmitting it via the Internet.

The Tablet PC API is designed around pen, ink collection, ink data, and recognition objects. The API is available in both managed and unmanaged forms. The InkCollector object is responsible for collecting ink as the pen writes, as its name implies. Referenced in the InkCollector object is the Ink object, which, in turn, contains Stroke objects.

Ink data is stored in the Ink object, which is referenced in the InkCollector as the Strokes collection property. Each Stroke object in the Strokes collection represents the stream of data collected as the pen moves, from the instant it touches the surface until it is lifted. Figure 1 shows the structure of InkCollector.

Figure 1 InkCollector
Figure 1 InkCollector

The InkCollector is the base set of pen and gesture input functionality. There are other objects in the INK API. The InkOverlay is a superset of the InkCollector object, and it adds selection and erasing support. The InkPicture is a superset of the InkOverlay object, which adds native support for annotating an image with ink. InkOverlay and InkCollector both need to be attached to a control-based object, while InkPicture is an actual control/component which can be embedded in a form (it has its own GUI). InkPicture is useful for annotating graphic images. InkOverlay is useful for annotating documents whose native data is not ink, for example an Adobe Acrobat document or a PowerPoint® slide.

Sometimes the best way to become familiar with an object such as InkCollector is to dive right in and play with it, so let's walk through some code. If you don't have it already, you can get the Tablet PC Platform SDK version 1.5.

After installing the Tablet PC Platform SDK, in C# under Visual Studio® .NET, add a reference to the "Microsoft Tablet PC API" (Microsoft.Ink.dll), and add a using directive at the beginning of the main C# source file to use the Microsoft.Ink package. Derive your main class from the System.Windows.Forms.Form class, then declare a private InkCollector object in this class:

private InkCollector myInkCollector;

To instantiate an InkCollector, you need a window with which to associate it. Simply use the handle to the form window:

myInkCollector = new InkCollector(this.Handle);
InkCollector has several constructors, and the one I just used accepts a window handle as a parameter.

This InkCollector is initially disabled, so it won't be collecting ink by default. To enable it just set the Enabled property to true:

myInkCollector.Enabled = true;
With just these few statements, you already have an inking application that displays ink as you draw it.

A Couple of Inking Events

Each stroke that's made by the user generates a stroke event. If your application requires that you handle this event (to start the recognizer, for example), you need to write an InkCollectorStrokeEventHandler. To connect your stroke event handler, add the handler delegate to the stroke event as follows:

myInkCollector.Stroke += new
InkCollectorStrokeEventHandler(myInkCollector_Stroke);
In the previous statement, InkCollectorStrokeEventHandler is the constructor for the stroke event delegate. Your event handler is myInkCollector_Stroke, which is provided as a parameter to the delegate constructor.

While a stroke is being drawn, a much more frequently occurring event is generated: NewPackets. This event corresponds to a packet or packets of ink data available for the application to process. Think of each packet as simply a data structure whose fields consist of pen coordinates, pressure, and so on. To connect your NewPackets event handler, add a handler delegate to the NewPackets event:

myInkCollector.NewPackets += new
  InkCollectorNewPacketsEventHandler(myInkCollector_NewPackets);
This statement creates a delegate to the handler you provide, myInkCollector_NewPackets, and adds that delegate to the NewPackets event.

Even if the pen is not moving, as long as it is touching the digitizer, NewPackets events occur at least 133 times per second. Your NewPackets event handler, therefore, has at most only 7.5 milliseconds to process each NewPackets event, which can arrive with one or more packets. At current processor speeds, this is not particularly difficult, but you'll still experience ink lag if your event handler is slow. As it turns out, the Ink Clipboard sample demonstrates how you can run into the ink lag problem.

Figure 2 (which comes from the SDK documentation) shows the inking events that can be triggered by different threads. It is important to know which thread executes which handler so that you can properly determine what that handler can do. For example, the NewPackets handler cannot make System.Windows.Forms.Form method calls because this handler is not executed by the UI thread, but rather by the ink thread. In the table in Figure 2, you can see that the NewPackets event is supposed to fire from the Ink thread, but in the InkClipboard sample it appears as if the NewPackets event is triggered from the UI thread. According to my Tablet PC sources, this depends on whether the main UI thread is declared as single-threaded apartment (STA) or multithreaded apartment (MTA). In STA programs like InkClipboard, the NewPackets event is triggered by the UI thread.

Figure 2 InkCollector Events (Close Figure 2)

Event | Fired From
CursorButtonDown | Ink thread
CursorButtonUp | Ink thread
CursorDown | Ink thread
CursorInRange | Ink thread
CursorOutOfRange | Ink thread
DoubleClick | UI thread (same as mouse)
Gesture | Ink thread
MouseDown | UI thread (same as mouse)
MouseMove | UI thread (same as mouse)
MouseUp | UI thread (same as mouse)
MouseWheel | UI thread (same as mouse)
NewInAirPackets | Ink thread
NewPackets | Ink thread (used in InkClipboard sample)
Stroke | Ink thread (used in InkClipboard sample)
SystemGesture | Ink thread
TabletAdded | Ink thread
TabletRemoved | Ink thread

Slow Lasso in the Ink Clipboard SDK Sample

The ink clipboard sample uses the InkCollector object to collect ink and allows the user to select ink strokes for copying to the clipboard. To do so, the user draws a selection lasso around the strokes.Tablet PC Display Orientation

Current display chips are designed to stream pixels out of memory in only one manner. At the hardware level, pixels are read from video memory from lower to higher addresses. In raster video, for which current display chips are optimized, the electron beam that draws the pixels starts from the upper-left corner of the monitor. It scans the surface of the monitor from left to right on every row. The rows are drawn from top to bottom. The resulting design naturally assigns bytes in the lowest video memory location to the upper-left pixel. The following bytes are then assigned to pixels in the topmost row, moving from left to right. The following rows of pixels are assigned in the same manner, so that the last, lower-right pixel is assigned to bytes in the highest memory location. There are as many memory allocation schemes as there are color schemes per pixel, but these schemes all boil down to one or more bytes assigned per pixel, with the lowest byte locations assigned to the upper-left pixel, and the highest byte locations assigned to the lower-right pixel.

Display hardware itself, therefore, is limited to one orientation: primary landscape. The other three orientations that the Tablet PC is capable of are all implemented at the display driver level. Bytes in every display buffer or bitmap are arranged in the same manner I just described (from upper-left down to lower-right). The trick is to transform in-memory bitmaps or video buffers before being moved, or sent via DMA, to video memory. This allows for the three additional orientations, but requires that display drivers maintain a transform buffer. Pixels are transform-copied to this buffer from user memory prior to being sent to video display memory. While the driver is doing this transform operation, the graphics I/O system also has to block all user-initiated repaints in order to prevent corruption of the transform buffer. The upshot of all this is that repaints of the screen for orientations other than landscape can be much slower.

The sample implements a lasso by disabling the automatic rendering of ink in the InkCollector, then doing its own rendering when collecting lasso points. (By the way, the InkOverlay object implements a lasso, so an alternative implementation of the concept in this sample would be to use InkOverlay. If I did that, however, I wouldn't have been able to show you how to handle NewPackets and Stroke events.) The sample illustrates the use of handlers quite well for both the Stroke and NewPackets events. It renders the lasso as NewPackets events arrive. For each event, it draws a number of evenly spaced dots, together with a line that changes as the lasso emerges in the form of dots around the selected strokes. The line completes the loop and, for each event, is redrawn from the first dot to the last dot drawn. Figure 3 shows a lasso being drawn around a selection.

Figure 3 Creating a Lasso
Figure 3 Creating a Lasso

The sample has a slight aesthetic problem. In portrait orientation, the lasso drawing is very slow, causing the lasso dots to lag behind the pen tip. For Tablet PCs capable of secondary portrait and secondary landscape orientations, the problem is evident in the following three orientations: primary portrait, secondary portrait, and secondary landscape. This problem does not occur in primary landscape orientation.

There is no way around the root cause of the problem, which is that display drivers for the Tablet PC have to perform extra buffering for the three new display orientations (see the sidebar "Tablet PC Display Orientation"). Nevertheless, there must be something in the implementation of the lasso that can be improved, for the lag only occurs when a large lasso is drawn quickly. Further examination of the problem reveals four relevant characteristics.

First, it's not only when the pen is moved quickly that the rendering can't keep up. Rendering also slows to a crawl if the pen tip is moved quickly and then suddenly stopped without being lifted off the digitizer. This ink lag is caused by "ink bleed," meaning that the pen tip is collecting ink even when it's not moving.

Second, if the drawing of the connector line is commented out the problem disappears completely, including all three other characteristics. Disabling the drawing of the lasso dots has no effect. Although it would appear as though drawing the connector line is computationally expensive, it's actually a contention problem that slows the NewPackets event handler, as you will see shortly.

Third, if the lasso is drawn fast enough, the lasso dots trail behind the pen tip, and the system fails to collect some ink, as evidenced by the resulting lasso being angular, not smooth (see the bottom part of the lasso in Figure 3). This ink lag occurs because the NewPackets event handler is too slow, as you'll see later.

Finally, the connector line is not being drawn consistently. Sometimes it is not being erased from a previous position, so more than one line appears.

Ink Lag

The myInkCollector_NewPackets routine embodies the NewPackets event handler of the sample. It basically collects stroke points in a collection called lassoPoints, then calls another method, DrawLasso, to draw that part of the lasso, which was collected as a result of a single event. It also draws the line that connects the first point of the lasso stroke to the last. Figure 4 is a code snippet from the myInkCollector_NewPackets event handler that draws the lasso points and the connector line.

Figure 4 Draw Lasso (Close Figure 4)
DrawLassoConnector(0,newPacketIndex-1);

for (int i = newPacketIndex; i < lassoPoints.Count; i++)
{
        . . . // some lines elided here
        dx = ((Point)lassoPoints[i]).X - lastDrawnLassoDot.X;
        dy = ((Point)lassoPoints[i]).Y - lastDrawnLassoDot.Y;

        lineLength = Math.Sqrt(dx*dx + dy*dy);

        // dots are drawn DotSpacing apart
        if (DotSpacing <= lineLength)
        {
            . . . // painting of lasso dot(s) elided
        }
}
    •••
DrawLassoConnector(0,lassoPoints.Count-1);

DrawLassoConnector, the routine that draws the connecting line, is called twice. The first call erases the previous line, and the second call draws the next incarnation of the line. The first subtle problem with this code is that these two calls occur for every NewPackets event regardless of whether a dot needs to be drawn. The second problem with this code is that the first DrawLassoConnector call does not necessarily remove the previous line drawn by the second call. The second parameter to DrawLassoConnector is an index into the lassoPoints array that determines one of two line endpoints to draw or delete. When deleting the previous line, this index should equal the index for that previous line, and a closer examination of the DrawLasso routine reveals that this is not necessarily the case.

The connector line does not need to be redrawn for every event, and it is being redrawn unnecessarily. Also, as I just mentioned, the second parameter in the first call to DrawLassoConnecter is incorrect. These would explain some of the previously mentioned characteristics of the problem. NewPackets events can occur even when the pen tip is not moving, as long as it is touching the digitizer, so DrawLassoConnector is continually being called with the same start and end points. This, coupled with the fact that the display driver blocks all repaints when it is doing the orientation transformation, exacerbates the resulting ink lag. DrawLassoConnector is trying to draw and redraw the same line about 133 times per second, while the display driver is blocking these redraws almost as often because it is continually calculating transformations. This is all happening within the myInkCollector_NewPackets handler, so the high-priority ink thread that generated the NewPackets event is being blocked longer than necessary. Even though the high-priority ink thread is blocked, the CPU is not overwhelmed because it is able to continue collecting ink at the driver level.

The event handler needs to call DrawLassoConnector only when a new lasso dot is drawn. To fix the ink lag I added a condition to drawing the lasso. I use the first DrawLassoConnector only when a previous line has to be erased, and I also have to ensure that it is indeed erasing a previously drawn line, not drawing a new one.

I introduced a new Boolean private member variable, m_bNewLassoDot. This variable is initialized to false before the lasso stroke is started. (It can be set to false as part of the Stroke event handling.) It is set to true only when a new lasso dot is drawn, so that if this event does not generate new lasso dots, the second call to DrawLassoConnector does not occur and no new lasso line is drawn. It is set to false right after the first call to DrawLassoConnector, thereby ensuring that the first call only occurs if there is a line that needs to be erased. This bool member variable or flag can alternatively be set in the routine DrawLassoDots because you need to draw the connector line only when new lasso dots are drawn. Another private member variable, m_iPreviousEndPointIndex, ensures that the first call to DrawLassoConnector deletes the previously drawn line and does not draw a new line.

The modifications I described take care of the problems relating to the fast-moving pen and the drawing of the connector line. The lasso is able to catch up when the pen tip suddenly stops moving, and no connector lines remain when the lasso completes. However, there are some issues with the third characteristic problem, in which a lasso is drawn quickly and the dots trail behind the pen tip (with the system failing to collect ink). When the pen tip is moved at the right speed, the resulting lasso is still angular, and the lasso dots trail behind the pen tip.

The event handler myInkCollector_NewPackets needs to be faster. To this end, I reduced the number of times DrawLassoConnector is called (see Figure 5). This code prevents the connector line from being drawn if the call to DrawLasso would only produce one lasso dot. Doing so reduces the number of lines drawn per sample, reducing the amount of time the ink thread spends executing DrawLasso. Figure 6 shows the revised code for DrawLasso.

Figure 6 Revised Draw Lasso (Close Figure 6)
/// <summary>
/// Draws the selection lasso.
///
/// Note that this method does NOT draw a point for every piece
/// of packet data.  Instead, it draws evenly spaced dots that
/// fall along the lasso path.
/// </summary>
/// <param name="g">The graphics device</param>
/// <param name="newPacketIndex">The index where the new packets begin
/// </param>
public void DrawLasso(int newPacketIndex, int lastNew)
{
    // bail out if lassoPoints array is empty
    if (lassoPoints.Count == 0)
        return;

    // create new client surface descriptor every time
    Graphics g = CreateGraphics();

    // The distance between the current point and the next point
    int     dx, dy;

    // The distance and angle of the line connecting the current
    // point and the next point
    double  lineLength;
    double  angle;

    // The horizontal and vertical spacing between the dots
    double  segSpaceX, segSpaceY;

    // The total number of dots to draw
    int     totalSegments;

    // Undraw the line that connected the first point to the
    // last point before the latest packets were added.
    if (m_bNewLassoDot)
        DrawLassoConnector(0, m_iPreviousEndPointIndex);
    m_bNewLassoDot = false;

    // For each lasso point, draw the corresponding lasso dots.
    // Note that this method does NOT draw a point for every piece
    // of packet data.  Instead, it draws evenly spaced dots that
    // fall along the lasso path.
    for (int i = newPacketIndex; i < lastNew; i++)
    {
        //  Always draw the first packet...
        if (0 == i)
        {
            // Update the last drawn lasso dot to this packet
            lastDrawnLassoDot.X = ((Point)lassoPoints[i]).X;
            lastDrawnLassoDot.Y = ((Point)lassoPoints[i]).Y;

            // use a helper method to draw the lasso dot
            DrawSelectionInkDot(g, lastDrawnLassoDot.X,
                lastDrawnLassoDot.Y);
        }
        else
        {
            // Compute the x and y distance between this point and the last
            // drawn lasso dot
            dx = ((Point)lassoPoints[i]).X - lastDrawnLassoDot.X;
            dy = ((Point)lassoPoints[i]).Y - lastDrawnLassoDot.Y;

            // Compute length of the line between this point and the last
            // drawn lasso dot
            lineLength = Math.Sqrt(dx*dx + dy*dy);

            // If the spacing between this packet and the last drawn lasso
            // dot is greater than the desired DotSpacing, draw as many
            // evenly spaced lasso points as needed to fill the distance...
            if (DotSpacing <= lineLength)
            {
               // Got at least a dot
               m_bNewLassoDot = true;

               // Calculate the angle of the line connecting this point
               // to the last drawn lasso point.
               angle = Math.Atan2(dy, dx);

               // Calculate the x and y components of the dot spacing
               segSpaceX = DotSpacing * Math.Cos(angle);
               segSpaceY = DotSpacing * Math.Sin(angle);

               // Calculate the number of dots we will need to draw.
               totalSegments = (int)(lineLength/ (double)DotSpacing);

               // draw the lasso dots...
               for (int j = 0; j < totalSegments; j++)
               {
               // The coordinates of the lasso dot are determined by the
               // previous lasso dot plus the x and y components of the
               // dot spacing (computed above).
               lastDrawnLassoDot.X = (int)(lastDrawnLassoDot.X + segSpaceX);
               lastDrawnLassoDot.Y = (int)(lastDrawnLassoDot.Y + segSpaceY);

               // use a helper method to draw the lasso dot
               DrawSelectionInkDot(g, lastDrawnLassoDot.X,
               lastDrawnLassoDot.Y);

               }
           }
       }
    }

    // draw the line connecting the first point to the last point
    if (m_bNewLassoDot)
    {
        // Draw the connector line only if there are several dots
        int currIndex = lastNew - 1;
        if ((currIndex - m_iPreviousEndPointIndex) > 1)
        {
            m_iPreviousEndPointIndex = currIndex;
            DrawLassoConnector(0, m_iPreviousEndPointIndex);
        }
        else
        {
            m_bNewLassoDot = false;
           }
      }
}
Figure 5 Draw the Connector if (m_bNewLassoDot) (Close Figure 5)
{
    // Draw the connector line only if there are several dots
    int currIndex = lastNew - 1;
    if ((currIndex - m_iPreviousEndPointIndex) > 1)
    {
        m_iPreviousEndPointIndex = currIndex;
        DrawLassoConnector(0, m_iPreviousEndPointIndex);
    }
    else
    {
        m_bNewLassoDot = false;
    }
}

The code in Figure 6 removes the Graphics parameter and instead declares and creates a Graphics instance within the function. This instance is created whenever the function is called, but it is not explicitly disposed, leaving it to the garbage collector to decide when to dispose this instance. This is not good programming practice because I never reuse any of the instances. To explicitly dispose the Graphics instance, I enclose all lines of code that use the instance in a using construct as follows:

using (Graphics g = CreateGraphics())
{
    // Put code that uses g here
}   // g is disposed here

Delegating DrawLasso

To improve the lasso further, I propose moving the routine DrawLasso to the UI thread. (This should help if NewPackets is not already triggered in the UI thread.) In order to accomplish this, I can declare a delegate type, LassoDelegate, and introduce a member variable, m_dAppendLasso, of type LassoDelegate. This will free the event handler of its lasso rendering duty and thereby shorten the execution time of the high-priority ink thread. The myInkCollector_NewPackets routine now simply puts lasso points in the lassoPoints array, then calls the delegate using BeginInvoke. BeginInvoke is a method in the Form control that allows the routine referred to by the delegate (in this case, DrawLasso) to execute in the UI thread. An implementation of this fix is available for download at the link at the top of this article.

Figure 7 Nice Smooth Lasso
Figure 7 Nice Smooth Lasso

As you saw, the slow lasso problem requires two fixes: getting rid of ink lag and allowing for better concurrency. With the second fix, the lasso is rendered beautifully, regardless of whether the pen tip is moved quickly around the client area (see Figure 7). Even the most discriminating user should be happy with the results.

Conclusion

As you can see, a lot of the ink functionality has been encapsulated for you in the INK API, making development of a simple handwriting collection application a snap. Examining some of the problems encountered when programming with this API gives you a good deal of insight into how the Tablet PC works its magic. With all its benefits and its rising popularity, I'm sure this introduction to Tablet PC programming will come in handy soon.

For related articles see:
Multithreaded Windows Forms Control Sample
Basic Instincts: Programming Events of the Framework Class Libraries

For background information see:
Building Tablet PC Applications by Philip Su and Rob Jarrett (Microsoft Press, 2002)

Carlos C. Tapang was on contract to the Tablet PC team at Microsoft when he wrote this article. He can be reached at ctapang@centerus.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


OLAP: Build an OLAP Reporting App in ASP.NET Using SQL Server 2000 Analysis Services and Office XP

OLAP

Build an OLAP Reporting App in ASP.NET Using SQL Server 2000 Analysis Services and Office XP

Jeffrey Hasan and Kenneth Tu
This article assumes you're familiar with ASP.NET and SQL Server
Level of Difficulty 1 2 3
Code download available at: OLAP.exe (373KB)
SUMMARY

Many organizations analyze their business-critical data using Online Analytical Processing (OLAP) technology. OLAP-based data mining provides a way to query multidimensional data sets and drill down into the data to find patterns. ASP.NET and the Microsoft Office Web Components (OWC) enable Web-based OLAP reporting. The OWC controls include PivotTable and Chart components that can be embedded in a Web page and scripted by programmers. In this article, the authors build a Web-based OLAP reporting app using ASP.NET, OWC, and SQL Server 2000 Analysis Services to illustrate the process.

Contents
Important Terms and Concepts
Web-based OLAP Reporting App Architecture
Office Web Components
Configuring the OLAP Source for HTTP Access
OLAP Security Access
Building the Application
Connect the Control to the Data Source
Dynamically Generate a Custom PivotTable Report
Save and Retrieve Custom Reports
Using OWC Events
Binding a Chart to the PivotTable Control
Creating Custom Groupings
Client Requirements for HTTP Use
Conclusion

A
growing number of companies are using online analytical processing (OLAP) and data mining technology to analyze their complex data sets. OLAP is specifically designed for analyzing multidimensional data sets, where patterns and relationships between the data are not immediately obvious. Microsoft® SQL Server™ 2000 Analysis Services is a popular choice for OLAP processing and data mining because it is tightly integrated with SQL Server 2000 and continues to integrate even more tightly with every new release of SQL Server.

As OLAP technology becomes more popular, developers are increasingly being asked to create custom reporting solutions that rely on OLAP-based multidimensional data sources. This is a complicated endeavor since OLAP-based reporting, or data mining, demands flexibility and a high level of interactivity. Data mining, by definition, is not static. Rather, the user must have the ability to construct any view of the data within the constraints of how the underlying data cube is structured. Compare this for a moment with traditional queries on relational database management systems (RDBMSs), which are constrained by tightly defined relationships between the member tables. RDBMS queries can certainly get creative, but only to a point. With OLAP-based data structures, you have much more flexibility in what you can query. Traditional static, canned reports add little value in helping you view and understand OLAP-based data.

Software developers face a four-fold challenge when they are asked to create an OLAP-based custom reporting solution. First, the solution must be able to connect with, and query, a multidimensional data source. Second, it must allow the user to drill down into the data with flexibility and ease. Third, it must let the user save a custom drill-down report because you cannot predict what data view the user finds important. Fourth, the solution must provide a readable interface that includes charts since reporting is more effective when it is visual.

Allow me to add an optional fifth requirement: the OLAP-based reporting solution should be Web based; that is, it should be implemented as a server-based Web application and be accessible from a Web browser (either over the Internet or on an intranet). With five stringent requirements, what's a software developer to do?

As it turns out, Microsoft provides a set of components called Office Web Components (OWC) that help you build sophisticated and user-friendly custom OLAP-based reporting solutions for the Web. The OWC components are a collection of COM controls for publishing spreadsheets, pivot tables, and charts to the Web. They provide the same rich, interactive experience for the user as does Microsoft Excel. The OWC components also expose a rich programmatic interface, which provides a high degree of functionality and flexibility around which you can build a custom solution. Figure 1 shows the OWC PivotTable® component, which allows you to connect to an OLAP data source and run queries against it using a convenient drag and drop interface.

Figure 1 OWC PivotTable Component
Figure 1 OWC PivotTable Component

The purpose of this article is to address the needs of developers who are using ASP.NET and Office Web Components to develop a custom OLAP-based reporting solution for the Web. We will present an application architecture that incorporates XML-based Web Services, and we'll show you how to build it and configure your OLAP data source for access over HTTP. In addition, we will describe how to deploy the solution to your users. Most Web applications are fully installed on a Web server and require no interaction with the client during the deployment process. However, OLAP and OWC introduce special requirements for the client who wants to use your custom solution.

Important Terms and Concepts

OLAP technology is replete with specialized terms that can be a barrier to understanding its real benefits. This article assumes you have general familiarity with OLAP and specific familiarity with SQL Server 2000 Analysis Services. Even so, before proceeding we should clarify the important terms and concepts both in this article and within the wider world of OLAP and data mining.

First, it is important to distinguish between OLAP and data mining technologies. An OLAP-based data source, or repository, may be subdivided into one or more cubes, which are multidimensional data structures that may be queried using MDX. A cube contains a set of attributes called dimensions, which roughly correspond to database fields, except that they also contain a hierarchical collection of levels. For example, an annual time dimension may be subdivided into levels of quarters, months, and weeks. A cube also contains a collection of measures, which are the actual data values and are typically numeric. For example, a retail cube will allow you to view unit sales (measure) according to store location (dimension), and time of year (dimension).

SQL Server 2000 Analysis Services enables you to perform OLAP processing, courtesy of a Microsoft Management Console (MMC) snap-in called the Analysis Manager. This is simply a pivot table interface that is integrated into Analysis Services. OLAP-based data is usually viewed using a pivot table, which allows you to drag and drop multiple dimensions and measures into a spreadsheet-like layout. The PivotTable control translates the drag and drop operations into MDX queries, which it relays to the OLAP data source and, in turn, receives a resultset.

Analysis Manager installs a PivotTable service on the database server, which includes an OLE DB provider that allows connections to OLAP data sources. The provider is named Microsoft OLE DB Provider for OLAP Services 8.0. Without it, no connection to the OLAP data source would be possible. The connection string for this provider includes a DataSource attribute for the repository name and a DataMember attribute for the name of the cube that you are connecting to. The PivotTable service must also be installed on the client machine; otherwise, the client's PivotTable control will be unable to communicate with the OLAP data source.

Web-based OLAP Reporting App Architecture

Figure 2 describes the high-level architecture of our Web-based OLAP reporting application. The reporting application architecture includes three pathways, which are numbered to match the sequence of the application's workflow. They combine to provide the following feature set:

  • Connection between a client-side PivotTable control and an OLAP data source over HTTP, through an XML Web Service.
  • Dynamic querying of the OLAP data source directly from the client-side PivotTable control (without any intermediate components).
  • The ability to save and retrieve custom PivotTable reports to a standard relational database structure, through an XML Web Service.

Figure 2 OLAP Architecture
Figure 2 OLAP Architecture

XML is the central player in this application architecture. The OWC PivotTable component natively serializes its contents to XML. Not only can you write out the data as XML, but you can also load the PivotTable component with XML data, as long as it conforms to the XML schema that is defined for Excel. This property allows ASP.NET developers to create XML-based Web Services that communicate bidirectionally with an OWC PivotTable control. One Web method can generate formatted XML for populating the PivotTable control (Pathway 1), while another can accept the serialized output from a PivotTable control and save it to a persistent medium such as a database or a file (Pathway 3).

Once a connection has been made between the PivotTable control and the OLAP data source, then the user is free to start assembling a custom view of the data using the PivotTable control's drag and drop UI. Every time the control is updated, it dynamically creates a multidimensional expression (MDX) query and sends the request directly to the OLAP data source through the HTTP gateway (Pathway 2). This communication occurs independently of the Web server; the Web page talks directly to the database. In this architecture, the role of the Web server is to establish the initial connection between the PivotTable control and the OLAP data source, and to enable saving and retrieving custom PivotTable reports.

Office Web Components

OWC version 10 provides out-of-the-box functionality that is not only powerful, but is also well known to the user. Your custom solution can thereby gain quicker acceptance from users who are already familiar and comfortable with Excel. Figure 3 summarizes the OWC components (version 10) that ship with Office XP. Note that Office 2000 shipped with a prior version of OWC (version 9), which provides much more limited programmatic interfaces and user interfaces. Much of our discussion of OWC version 10 also applies to OWC version 9, although not everything that we discuss will be supported in the earlier version.

Figure 3 Office Web Components (version 10) (Close Figure 3)

Component | Description
PivotTable | Enables users to connect to any OLAP data source that supports the Microsoft OLE DB Provider for OLAP Services 8.0 and higher. (Also connects to Excel spreadsheets and SQL Server and Access relational databases). The PivotTable control allows users to analyze data by pivoting, grouping, filtering, and sorting.
Spreadsheet | Provides a spreadsheet user interface, including a recalculation engine and an extensive function library.
Chart | Graphically displays data from a bound DataSource, PivotTable, or Spreadsheet control. The Chart component will automatically refresh when it is bound to a PivotTable control and the user repivots the data.
DataSource | Manages communication with back-end database servers. The PivotTable and Spreadsheet components can actually connect to a data source independently without specifically using a DataSource control. This is because their XML data can contain an OLE DB connection string directly.

ASP.NET developers have few resources to turn to for incorporating OWC components into a Web project. Most of the online code samples assume that you are interacting with the OWC components using a client-side scripting language, such as VBScript. While this approach is useful, it ignores a more powerful approach that ASP.NET makes possible. Namely, you can create a server-side instance of an OWC component, make a connection, set properties, and then use this instance to generate XML data for a separate, client-side instance of the PivotTable component. This programming logic is contained inside the OLAP Data Access Objects portion of the architecture diagram (see Figure 2), along Pathway 1. The XML can then be delivered to the client-side component using an XML-based Web Service, also shown in Pathway 1.

Figure 4 summarizes the OWC components that we will work with in this article—namely, the PivotTable and Chart components. The Spreadsheet and DataSource controls are not covered here because they do not factor into the architecture that we're discussing. Figure 5 and Figure 6 summarize the programmatic interfaces of the PivotTable and Chart components, respectively.

Figure 6 Chart Component's Programmatic Interface (Close Figure 6)

Member | Type | Description
DataSource | Property | Defines the data source for the chart control. When set to another control (PivotTable or Spreadsheet, for example), this effectively binds the chart control to the other control.
Type | Property | Defines the chart type as defined by the ChartChartTypeEnum enumeration. The default is a bar chart.
Figure 5 PivotTable Component's Programmatic Interface (Close Figure 5)

Member | Type | Description
ActiveView | Property | Represents the layout of the active PivotTable. This property returns a PivotView object.
ColumnAxis | Property | Represents the fields in the column axis. Returns a PivotAxis object.
ConnectionString | Property | Sets the ADO connection string to an Analysis Services server. The DataSource parameter within the ConnectionString property determines the type of connection protocol the OWC component will utilize.
DataAxis | Property | Represents the scale in the data axis. Returns a PivotAxis object.
DataMember | Property | Sets the data member name the control will request from Analysis Services. This is the same as the cube name.
IncludedMembers | Property | Defines the members to be included in a PivotField. The property accepts as input either a single member or an object array of members.
IsIncluded | Property | Sets the field to be included and active in the PivotFieldSet.
RowAxis | Property | Represents the fields in the row axis. Returns a PivotAxis object.
XMLData | Property | Sets or returns the XML data for the current report PivotTable control. The details about the report (formatting and data) are contained within the XML data. It also includes details of the OLAP connection.
AddCustomGroupField | Method | Adds a custom group field to a specific PivotFieldSet.
AddCustomGroupMember | Method | Adds a custom group member to a specific PivotFieldSet.
InsertFieldSet | Method | Inserts a field set into the row or column axis.
CommandExecute | Event | Fires after a command has been executed. The ChartCommandIdEnum and PivotCommandId constants contain lists of the supported commands for each OWC component.
Query | Event | Fires whenever a query is activiated by the PivotTable.
Figure 4 PivotTable and Chart-related Objects (Close Figure 4)

Member | Description
PivotTable | Enables users to connect to any OLAP data source that supports the OLE DB Provider for OLAP Services 8.0 and higher. (Also connects to Excel spreadsheets and SQL Server and Access relational databases). The PivotTable control allows users to analyze data by pivoting, grouping, filtering, and sorting.
PivotView | Represents a specific view of the PivotTable. Used to manipulate the rows, columns, scales, and formatting of a PivotTable view.
PivotDataAxis | Contains methods and properties related to the data axis.
PivotResultColumnAxis | Contains methods and properties related to the column axis.
PivotResultRowAxis | Contains methods and properties related to the row axis.
PivotFieldSet | A set of fields defined in the cube.
PivotField | A specific field defined in the cube.
ChartSpace | Enables users to connect to any OLAP data source that supports the OLE DB Provider for OLAP Services 8.0 and higher. (Also connects to Excel spreadsheets and SQL Server and Access relational databases). The ChartSpace control allows users to graphically display data and to bind the control to an existing PivotTable or Spreadsheet.
ChCharts | A collection of ChChart objects.
ChChart | Represents one chart in the ChartSpace. A ChartSpace can contain up to 64 charts.

The PivotTable component has a complicated interface, but the most important aspect to remember is the XMLData property. This is an XML representation of the complete PivotTable contents, including the connection information to the OLAP data source. The XMLData also stores the dimensions and measures currently in view and the formatting that has been applied. The XMLData property may be set and retrieved, and it is an important means of customizing a PivotTable component dynamically.

Configuring the OLAP Source for HTTP Access

Before we can begin building the Web-based OLAP reporting application, we must configure the Analysis Services data source for access by Microsoft Internet Information Services (IIS). The IIS Web site must be set up on the same server where Analysis Services is installed, and it must contain a single file called msolap.asp. This file provides the bridge between requests coming over an HTTP port and the Analysis Services data source. The file is located under the path \Program Files\Microsoft Analysis Services\bin. We recommend that you configure the data source for access over a secure connection (HTTPS) using a Secure Sockets Layer (SSL) certificate. This step requires that you purchase an SSL certificate for the Analysis Services server. You can also use a nonsecure (HTTP) connection to access the data source, but we don't recommend this approach.

Once IIS is installed on the Analysis Services server, you can configure a new Web site for HTTPS access. First create a new Web site directory outside of the default Web root directory (\Inetpub\wwwroot\). We recommend that you stop the default Web site for security purposes. In addition, you should close port 80 on your firewall and open port 443 for HTTPS access once the SSL certificate is installed.

Copy the msolap.asp file from the \Program Files\Microsoft Analysis Services\bin directory into the new Web site directory, and use IIS Manager to configure the new Web site directory as a Web Application and add a Host Header Name for the Web site (for example, olap.mycompany.com).

Purchase and install the SSL certificate on the server. The Web site must already be accessible over port 80 during the purchase verification process. Once the certificate is installed, you can close port 80 and open port 443 on your firewall. Use IIS Manager to set Directory Security so that Anonymous Access is disabled and Basic Authentication is enabled. Finally, reset IIS to apply the changes.

Once HTTPS access has been set up, you must update the OLE DB connection string that points to the OLAP data source. The format of the connection string is as follows:

Provider=MSOLAP;user id=Domain\Username;password=Password;Data
Source=https://ServerName;Initial Catalog=OLAPDataSourceName
For more details, consult the Knowledge Base article 279489, "How to Connect to Analysis Server 2000 By Using HTTP Connection".

OLAP Security Access

Security credentials are essential once your OLAP data source becomes accessible over HTTPS because your data source is now potentially available to anyone who can browse your Web site. For example, a remote user can open Excel and connect a PivotTable to your OLAP data source simply by providing the URL for the Analysis Server and the name of the Initial Catalog.

Analysis Services implements cube security roles based on Windows NT® user accounts. Basically, you create one or more local Windows NT user accounts on the server and assign them to a SQL Server database role. You then assign the database role to a cube role and assign it to the specific cube. The cube role essentially inherits the Windows NT user ID and password credentials. You can then add the user ID and password credentials to the OLAP data source connection string.

Note that one OLAP data source may contain multiple cubes. You can connect to a specific cube by setting the OWC PivotTable control's <DataMember> XML tag value to a valid cube name (explained in the next section). This cube must be assigned to a role that corresponds to the credentials in the OLAP data source connection string.

Building the Application

Now it is time to build this application using ASP.NET and OWC 10. The application's user interface is contained within a single Web Form called owc10.aspx, shown in Figure 7. The interface contains an embedded client-side OWC PivotTable control as well as a set of buttons for invoking connections to an OLAP data source and for saving and retrieving custom reports. This functionality is directly provided by a set of Web methods in the XML Web Service found in wsOLAP.asmx. The user interface hooks into these Web methods using a set of client-side JavaScript functions in the olap.js script file. These JavaScript functions delegate most requests to the Web Service methods by making client-side calls to the methods. This is accomplished using a DHTML behavior file called webservice.htc.

Figure 7 Web-based OLAP Reporting App
Figure 7 Web-based OLAP Reporting App

The sample app, called OLAPReport, is available for download from the link at the top of this article as a Visual Studio® .NET 1.1 project. It uses the Foodmart 2000 data warehouse as its OLAP data source and the Sales cube as its data member. Foodmart 2000 installs with SQL Server 2000 Analysis Services.

Once you have downloaded and installed the project files, there are a few steps you need to follow for the compiled project to work correctly. First, you will need to modify the OLE DB connection string, which the application uses to reference the OLAP data source. The connection string is stored as a custom application setting in the Web.config file:

<appSettings>
    <add key="OLAPConnectionString" value="Provider=MSOLAP.2;
        Data Source=https://ServerName;Initial Catalog=Foodmart 2000>
</appSettings>
Notice that the connection string contains no security credentials (such as a User ID and password). These credentials are only required if you have implemented security roles in your Analysis Server. We encourage you to add security roles, as described earlier. Once you have them in place, simply add the credentials to the end of the connection string.

Remember that you will need to configure your Analysis Server for HTTP access in order for this connection string to work. These steps were also described earlier. The project is initially set to use http://localhost/olap as its data source, so you will need to create a virtual directory named olap under the default Web site and copy the msolap.asp file to that directory. As a shortcut, you can simply replace the Data Source value with your machine name, omitting the leading https://. This will allow a standard connection to the OLAP data source, rather than an HTTP connection. This works for demonstration purposes, but remember that if you don't enable HTTP access, you cannot expose your Analysis Server to users who are outside your network domain.

Finally, before proceeding you may need to set a reference to the OWC10 COM component from the OLAPReport project. (The setup project should take care of this for you, but it is useful to know the steps in any case). To add the reference, open Solution Explorer, then right-click the project icon and select Add Reference from the popup menu. Once the dialog opens, switch to the COM tab, and locate the component named Microsoft Office XP Web Components (OWC10.DLL). Click the Select button, and then click OK. Visual Studio .NET will then generate a .NET wrapper for the component and will add a project reference to OWC10.

Connect the Control to the Data Source

Now let's implement each of the application's three features in turn. First, let's discuss Pathway 1—how to set a connection between a client-side OWC control and the OLAP data source. This function is initiated using the Set Connection button on the user interface (see Figure 7). This button calls the JavaScript function initializePivotTable and passes in the name of the data cube to connect with—in this case, "Sales."

The JavaScript function in turn makes an asynchronous call to the InitializePivotTableXML Web method (enabled by the DHTML behavior called webservice.htc, which we introduced earlier). The Web method connects to the OLAP data source and then generates the PivotTable's XMLData. The Web method's output is handled by a JavaScript function called onInitializePivotTableResult. Figure 8 shows the code for the JavaScript functions, while Figure 9 shows the code for the Web method. The code in these two figures is simple and, best of all, it is reusable. Quite simply, these functions allow us to generate custom XMLData on the Web server for delivery to the client-side PivotTable control.

Figure 9 Generate XMLData for a PivotTable Control (Close Figure 9)
<WebMethod()> Public Function InitializePivotTableXML(ByVal _
    strDataMember As String) As String

    Dim m_XML As String
    Dim strOLAPConn As String = _
      ConfigurationSettings.AppSettings("OLAPConnectionString")

    Try
        Dim objPT As PivotTableClass = New PivotTableClass
        objPT.ConnectionString = strOLAPConn
        objPT.DataMember = strDataMember
        m_XML = objPT.XMLData
        objPT = Nothing
    Catch err As Exception
        m_XML = "<err>" & err.Source & " - " & err.Message & _
                "</err>"
    Finally
    End Try
    Return (m_XML)
End Function
Figure 8 Connecting to an OLAP Data Source (Close Figure 8)
function initializePivotTable(strDataMember) {
    // This function calls the InitializePivotTableXML() Web
    // method
    var iCallID = service.svcOLAP.callService
        (onInitializePivotTableResult,
         'InitializePivotTableXML',
         strDataMember);
}

function onInitializePivotTableResult(result) {
    // This function handles the InitializePivotTableXML()
    // Web method result
    text = result.value; // result string
    // Evaluate return result
    if (!result.error) {
        // Assign the XML to the PivotList XMLData value
        frm1.PivotTable1.XMLData = text;
    }
    else {
        alert("Unhandled error - " + result.errorDetail.code +
              " " + result.errorDetail.string);
    }
}

In the process of establishing a connection, we are effectively initializing the OWC control with a blank report for the Sales data cube. The empty report populates the Field List with all the available fields and measures for the data cube. The user can then create a custom report by dragging and dropping fields into the Row and Column fields, and Measures into the Totals fields in the PivotTable. Every time that the user drops a field onto the PivotTable, the control sends a query directly to the data source, which in turn sends back more query results. This exchange is conducted independently of the Web Service, which is only required for establishing the initial connection between the control and the data source. This exchange is represented by Pathway 2 in Figure 2.

Dynamically Generate a Custom PivotTable Report

Now let's dig deeper into the PivotTable object model by generating a custom report programmatically. The custom report we want to build allows the user to select two cities and to compare the Store Sales for the Drinks Product Family. We use the same approach as Pathway 1, which means we call a Web method to generate XMLData that describes the report. We then assign the XMLData from the Web method to the embedded control's XMLData property. The interface provides two dropdown lists of city names and a button called Load Custom Report (as shown in Figure 7). This button fires off a JavaScript function called LoadCustomReport, which accepts as input the two city names from the Web Form. This function, in turn, calls a Web method called LoadCustomPivotTableReport, which programmatically assembles the XMLData for the custom report. Finally a JavaScript handler function receives the XMLData from the Web method and then assigns it to the embedded PivotControl's XMLData property. Figure 10 shows the code for the Web method.

Figure 10 LoadCustomPivotTableReport Web Method (Close Figure 10)
<WebMethod()> Public Function LoadCustomPivotTableReport(ByVal _
    strCity1 As String, ByVal strCity2 As String) As String
    Dim m_XML As String
    Dim strOLAPConn As String = _
      ConfigurationSettings.AppSettings("OLAPConnectionString")
    Dim objPT As PivotTableClass = New PivotTableClass
    Dim objPTView As PivotView
    Dim fldCity, fldName, fldProdFamily As PivotField
    Dim fSetCustomers, fSetProduct As PivotFieldSet

    Try
       objPT.ConnectionString = strOLAPConn
       objPT.DataMember = "Sales"
       objPT.AllowFiltering = False
       objPTView = objPT.ActiveView
       objPTView.TitleBar.Caption = "City Comparison of Drink
       Sales"

         ' Define the column elements
objPTView.ColumnAxis.InsertFieldSet(objPTView.FieldSets("Time"))
objPTView.ColumnAxis.FieldSets("Time").Fields("Year").Expanded = True

    ' Define the row elements
    fSetCustomers = objPTView.FieldSets("Customers")
    objPTView.RowAxis.InsertFieldSet(fSetCustomers)
    fSetCustomers.Fields("Country").IsIncluded = False
    fSetCustomers.Fields("State Province").IsIncluded = False
    fSetCustomers.Fields("Name").IsIncluded = False

    ' Define the members of the row elements
    fldCity = fSetCustomers.Fields("City")
    fldCity.IncludedMembers = New Object() {strCity1, strCity2}

    ' Exclude all other field row members in the fieldset
    fSetProduct = objPTView.FieldSets("Product")
    objPTView.RowAxis.InsertFieldSet(fSetProduct)
    fSetProduct.Fields("Product Department").IsIncluded = False
    fSetProduct.Fields("Product Category").IsIncluded = False
    fSetProduct.Fields("Product Subcategory").IsIncluded =False
    fSetProduct.Fields("Brand Name").IsIncluded = False
    fSetProduct.Fields("Product Name").IsIncluded = False

    fldProdFamily = fSetProduct.Fields("Product Family")
    fldProdFamily.IncludedMembers = "Drink"

    ' Define the measures
objPTView.DataAxis.InsertTotal(objPTView.Totals("Store Sales"))
    objPTView.DataAxis.Totals("Store Sales").NumberFormat = _
    "Currency"
    ' Return the XML data to the client side script
    m_XML = objPT.XMLData

    objPT = Nothing
    Catch err As Exception
  m_XML = "<err>" & err.Source & " - " & err.Message & "</err>"
    Finally

    End Try

    Return (m_XML)
End Function

The LoadCustomPivotTableReport method uses many of the PivotTable properties that are described in Figure 5. Also keep in mind the following tips:

  • Use the InsertFieldSet method to insert the appropriate PivotFieldSet into the row or column axis.
  • Use the IncludedMembers property to define the distinct PivotField values to include in the table. The IncludedMembers property requires a Variant array, which in the .NET Framework equates to an Object array. Any other array type will generate an error.
  • Use the IsIncluded property to enable or disable a particular PivotField in the PivotFieldSet.
  • Use the InsertTotal method to insert the appropriate measure into the totals grid.
  • The NumberFormat property defines the measure's format. The property accepts either a format code (such as, "0.###") or a predefined format such as "Currency" or "Percent".

Save and Retrieve Custom Reports

Now that we have gone through the effort of creating this custom report, we do not want to repeat this process every time we need to generate a new one. Thankfully, the data required to duplicate this report is contained in the XMLData property. Thus, saving a report is simply a matter of saving the contents of the XMLData property to a persistent storage medium, such as a relational database. Loading a report is simply a matter of retrieving the stored XMLData and rebinding it to the PivotTable. This feature is shown as Pathway 3 in Figure 2.

The application's Web Service includes two methods for saving and retrieving the XMLData of custom reports. In this example, we will actually write the XMLData to a text file called OLAPReport1.xml. The interface provides two buttons: Save Report and Load Saved Report. The Save Report button calls a JavaScript function that then sends the PivotTable's XMLData to the following Web method:

function SaveReport() {
    // Purpose:  Call Web Service method to save the report
    var strReportXMLData = frm1.PivotTable1.XMLData;
    var iCallID = service.svcOLAP.callService(onSaveReportResult,
        'SaveReport', strReportXMLData, 'OLAPReport1.xml');
}

The Web method in turn writes the XMLData to a file. The Load Saved Report button calls a JavaScript function that does the opposite, as shown in Figure 11. This function calls a Web method that opens the text file, retrieves the XMLData, and then passes it back for binding to the PivotTable control.

Figure 11 Load the XMLData for a Custom Report (Close Figure 11)
function LoadSavedReport() {
    // Purpose:  Call Web Service method to load the saved
    // report
    var iCallID = service.svcOLAP.callService(onLoadSavedReportResult,
        'LoadSavedReport', 'OLAPReport1.xml');
    }

function onLoadSavedReportResult(result) {
// Purpose: This function handles the
// wsOLAP.onLoadSavedReportResult() Web Service result
var text = result.value; // result string
// Evaluate return result
if (!result.error) {
// Assign the XML to the PivotList XMLData value
        frm1.PivotTable1.XMLData = text;
}
    }

The file approach is convenient for demonstration purposes, but a production system should use a relational database for storing the XMLData for custom reports. This approach provides better concurrency management and also allows you to store reports in a more organized way.

Using OWC Events

The OWC 10 components provide several useful events that you can capture in client-side event handlers. These may be written either in VBScript or in JavaScript. In VBScript you use basic event notation, as shown here:

Sub <ObjectName>_<EventName>
In JavaScript you must use for-event syntax:
<script language="javascript" event="EventName" for="ObjectName">

The sample application demonstrates how to capture the Query and CommandExecute events. The event-handling code is shown in Figure 12. When the Query event is fired, the application appends some event logging to the text area field on the left side of the Web page. When the CommandExecute event is fired, we check to see if it is a refresh command. If so, the application appends the date and time to the event logging text area.

Figure 12 JavaScript and VBScript Event Handler (Close Figure 12)
<script language="javascript" event="Query" for="PivotTable1">
{
    var sLog = document.Form1.Text1.value + "";
    document.Form1.Text1.value = "Query Event Fired. " + sLog;
}
</script>

<script language="vbscript">
    Sub PivotTable1_CommandExecute(Command, Succeeded)
        Dim ptConstants
        Set ptConstants = document.Form1.PivotTable1.Constants

        ' Check to see if the PivotTable list has been
        ' refreshed.
        If Command = ptConstants.plCommandRefresh Then
            ' Write the current data and time to the text box.
            document.Form1.Text1.value = vbCrLf & _
              "PivotTable Last Refreshed on " & Date & " at " _
                & Time & vbCrLf & document.Form1.Text1.value
        End If
    End Sub
</script>

Binding a Chart to the PivotTable Control

In many cases, a graphical representation of the data is very valuable to the user. Fortunately, OWC provides a ChartSpace control which provides exactly that type of functionality. To improve upon our custom report, we are going to generate a bar chart to represent the data. Although this might seem like a monumental task, the ChartSpace control will help us greatly in this effort. One of its useful features is that it can bind another control, such as the PivotTable control, as its primary data source. It will then display the current PivotTable view in a graphical format. The end effect is that changes in the PivotTable view will be automatically reflected in the ChartSpace control.

In our case, once we have generated the custom PivotTable report, we simply have to assign the PivotTable control to the ChartSpace control's DataSource property:

// Bind the Chart control to the PivotTable control
frm1.ChartSpace1.Clear();
frm1.ChartSpace1.DataSource = frm1.PivotTable1;

// Set the chart type to 3DBar.
frm1.ChartSpace1.Charts(0).Type = 50;
We need to call the Clear method before assigning the DataSource property in order to remove the instance of the previous chart. The ChartSpace control provides many customizable properties, including the Type property, which accepts a ChartChartTypeEnum enumeration value to define the chart layout.

Creating Custom Groupings

Finally, let's look at an important new feature of OWC 10—the ability to create custom groupings. Our report currently groups the store sales measure based on the Time field set. In this case, Time is grouped into Year/Quarter/Month. Suppose we want to group our custom report using a grouping other than the defined fields in the data cube. We can do this by using the custom grouping features in OWC 10. A button labeled Apply Custom Grouping calls a JavaScript function which in turn calls the ApplyCustomGrouping Web method. The input parameter is the XMLData for the current loaded report. The Web method will then load a PivotTable object and use the AddCustomGroupField method to define the custom group, in this case a half-year time grouping. The code is shown in Figure 13.

Figure 13 Creating Custom Groups (Close Figure 13)
<WebMethod()> Public Function ApplyCustomGrouping(ByVal _
    strReportXMLData As String) As String

    Dim m_xml As String
    Dim objPT As PivotTableClass = New PivotTableClass
    Dim objPTView As PivotView
    Dim fsTime As PivotFieldSet
    Dim fsHalfYear As PivotField

Try
    objPT.XMLData = strReportXMLData
    objPTView = objPT.ActiveView

    ' Set a variable to the Time field set.
    fsTime = objPTView.FieldSets("Time")

    ' Add a custom group field named "Group1" to the Time field
    ' set.
    fsHalfYear = fsTime.AddCustomGroupField("CustomGroup1", _
        "CustomGroup1", "Quarter")

    ' Add a custom field set member. This member includes all
    ' "Q1" and "Q2" members under 1997.
    fsHalfYear.AddCustomGroupMember _
        (fsTime.Member.ChildMembers("1997").Name, _
        New Object() {"Q1", "Q2"}, "1stHalf")

    ' Add another custom fieldset member to include all "Q3"
    ' and "Q4" members under 1997.
    fsHalfYear.AddCustomGroupMember _
        (fsTime.Member.ChildMembers("1997").Name, _
        New Object() {"Q3", "Q4"}, "2ndHalf")

    ' Collapse the fieldset at the custom member level
    fsHalfYear.Expanded = False

    m_xml = objPT.XMLData
    objPT = Nothing

Catch err As Exception
    m_xml = "<err>" & err.Source & " - " & err.Message & _
            "</err>"
Finally

End Try

Return (m_xml)
End Function

Please note that we have added one custom group ("CustomGroup1") with two members ("1stHalf" and "2ndHalf") to encompass the year. The custom members are added using the AddCustomGroupMember method. When you exercise this function on the Web Form, the ChartSpace control also incorporates the custom grouping into the bar chart—yet another benefit of using the integrated components found in Office Web Components 10.

There is one significant issue with the custom group support. The XMLData does not currently save the proper data to persist the names of the custom groupings. If you were to save the XMLData (as we did in the Save Report functionality earlier), and then load the PivotTable with the saved XMLData, the caption for the custom group ("1stHalf", "2ndHalf") would appear as GUIDs. Nonetheless, the integration of the OWC 10 components greatly improves upon the previous version and makes it a very useful tool for Web-based OLAP reporting.

Client Requirements for HTTP Use

Now that we have built our Web-based OLAP reporting application, there are four configuration steps that every Web client must implement in order to start using the application. First, every Web client must have a local licensed copy of Office XP installed on their machine. As an alternative, they must have access to a licensed copy of Office XP on a server to which they have network access. The OWC components come with Office XP and are subject to the same licensing requirements.

Second, every Web client must be using Internet Explorer 5.0 or higher. There are several reasons for this. The Web application uses the Web Service DHTML behavior, which requires Internet Explorer 5.0 or higher. In addition, the client-side OWC components are Microsoft COM components, which do not natively run in non-Microsoft browsers. (We are aware that some browsers provide plug-ins for running client-side COM controls, but we have not tested this ourselves with OWC).

The third configuration step is that every Web client must have the correct OLE DB provider installed: specifically, Microsoft OLE DB Provider for OLAP Services 8.0 or higher. This provider is installed with the redistributable PivotTable Service, which includes a set of DLLs and the required version of Microsoft Data Access Components (MDAC). You can install the PivotTable Service from the SQL Server 2000 CD-ROM under \Msolap\Install\Pts. You will need to choose one of the two installation files provided: PtsLite.exe or PtsFull.exe. The latter installs the PivotTable Service and MDAC, while the former installs the PivotTable Service only. If you are unsure which version you need, simply install PtsFull.exe. You will almost always need the PivotTable Service on each Web client machine because only Microsoft Analysis Services installs the service automatically. You may use the Excel PivotTable wizard to determine if the machine the Web client is running on already has the PivotTable Service installed.

The fourth configuration step is that every Web client must add the application's Web domain to their list of trusted sites in Internet Explorer. The application resides on a different domain than the client machine, so the OWC PivotTable component must cross a domain boundary every time it queries the OLAP data source. For this reason, the application's Web domain must be included as one of the client's trusted sites. To ensure this is reflected, open Internet Explorer and click on the Tools | Options menu. Select the Security Tab, click the Trusted sites icon, and then click the Sites button.

Alternatively, you can provide your clients with a registry file that will automatically update their list of trusted sites. Here's a sample registry key update:

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet
Settings\ZoneMap\Domains\mycompany.com] "https"=dword:00000002
Note that the registry file must be run once for every user on the client machine. This is because it updates the HKEY_CURRENT_USER registry hive. For example, if two domain users share the same machine then they will each be required to run the registry file for their respective accounts.

Conclusion

Web-based OLAP reporting solutions present a fascinating technical challenge because they involve several sophisticated technologies, including SQL Server 2000 Analysis Services, Office Web Components, XML, and ASP.NET. The architecture that we presented here is designed to utilize all of these technologies to the best of their capabilities. Other architectures are certainly possible, especially as current technologies evolve in the future.

For related articles see:
Creating a PivotTable List Control
Understanding the Chart Control
Analysis Services
Office

Jeffrey Hasan is a technical architect and software developer who specializes in developing enterprise applications using Microsoft technologies. He has coauthored numerous books and articles. E-mail Jeff at jeff@asptechnology.net.

Kenneth Tuis a software developer specializing in Microsoft technologies with extensive experience developing both Windows and Web-based enterprise applications. He and Jeffrey coauthored Performance Tuning and Optimizing ASP.NET Applications (APress, 2003). E-mail Ken at ken@asptechnology.net.


Blogging: Design Your Own Weblog Application from Scratch Using ASP.NET, JavaScript, and OLE DB

Blogging

Design Your Own Weblog Application from Scratch Using ASP.NET, JavaScript, and OLE DB

Marco Bellinaso
This article assumes you're familiar with Visual Basic .NET and JavaScript
Level of Difficulty 1 2 3
Code download available at: Blogging.exe (151KB)
SUMMARY

The ASP.NET advanced templated controls, such as the DataList and DataGrid, are perfect for many data representation situations. However, when you need the flexibility to render a variety of layouts, the Repeater control is what you need. In this article the author builds a full-featured blog application to illustrate the use of the Repeater and DataList controls that render nested data in a master-detail relationship. He then discusses how to override the default implementations of these controls by adding some client-side JavaScript code that makes the blog more responsive and enhances its usability.

Contents
The Database Design
The Business Layer
Showing Messages and Child Comments
Selecting an Interval and Loading the Blog
Popup Calendar
Posting Comments
Administering the Blog
Validation for Multiple Virtual Forms
Conclusion

I
t seems like everyone wants a blog these days—I know I did. But I couldn't find any pre-built ASP.NET blog code with the features I wanted, so I built my own. The great thing about building your own blogging application is that you get lots of practice with ASP.NET server controls, such as the Repeater, DataList, and Calendar. A blog application may seem like a simple exercise at first, but in fact it requires you to implement many features you'd need in a typical reporting application, such as building and rendering master-detail relationships or editing and deleting records, hiding or showing content and controls based on the logged-in user, and managing input validation for multiple virtual forms on the same page. This article presents the design and implementation details of a Web log and explains techniques that can be easily applied to a variety of ASP.NET projects, whether they're built for business or simply for fun.

Before you start coding, you should decide what kind of blog you want to build, what features it will have, and how the data store will be designed. There are a number of features an effective blog will include. The blog's messages should be shown from the newest to the oldest. Multiple messages could be posted the same day, and they should be visually grouped together in a table or a box, but still recognizable as separate posts ordered chronologically. Also, the user should be able to select a time interval for the entries she wants to read. This is important because you don't want to retrieve old content that the user has already seen.

The user should be able to comment on any message, and the posted comments should be shown directly under their parent message so it's easy to follow the thread. In addition, the blog's owner should be able to post, edit, and delete messages and comments, while the user should only be able to read messages and post comments. To allow or prohibit posting and editing based on the user, some controls will need to be either shown or hidden, and will require some form of authentication.

The Database Design

Next, you must decide how you want to store the messages and comments. In this project, I used a SQL Server™ database, but my sample code also includes a Microsoft® Access database, in case that's your data store of choice. The database contains just two tables: one for the messages and the other for the comments. The messages table stores unique IDs, an optional title or headline, the message text, and the date and time that the message is posted. The comments table stores unique IDs, the ID of the message the comment refers to, the author's name, the author's e-mail address, the comment text, and the date and time the message is posted. Figure 1 shows the design of the tables.

Figure 1 Parent/Child Tables
Figure 1 Parent/Child Tables

As you see, there is a one-to-many relationship between the two tables, linked with the MessageID field, which also enforces referential integrity with cascading updates and deletes. Note that the table names start with the prefix "Blog_". I always use a prefix for my SQL Server tables because they will be grouped together in Enterprise Manager when listed alphabetically. In addition, Web hosting plans usually offer only one SQL Server database that you have to share among all the Web modules you use. If you didn't use a table prefix, you may have a module that already uses a table named Messages, and solving such a name clashing problem is not immediate at deployment time.

Another important detail to remember is that no field should be nullable, even if the user leaves that field blank. Rather than permitting null values, you should set empty fields to the empty string to prevent unhandled exceptions. This isn't a problem if you use Web Forms for data input because the controls return an empty string if there is no content.

The Business Layer

In a typical database-centric application, you have data, business, and presentation layers. The data layer might consist of a set of stored procedures, while the business layer would be a set of classes (optionally compiled in their own separate assembly) that wrap the stored procedures to ensure data integrity, perform validation, and enforce other business rules. However, in this project I decided not to use stored procedures; I hardcoded the SQL queries and commands directly into the main assembly so it would be easier to use the blog with Access. I didn't separate the classes for the data and business layers either, but since there are just two tables and only simple business rules, I created a single class that takes care of validating the input and executing the proper SQL statements. All the code will be in the same assembly, together with the presentation layer, because I don't expect to update the business class separately from the program that uses it, and I don't expect to distribute the business class with other types of applications or among many clients. Thus, it's OK to compile it with the main ASP.NET application (called WebLogger). The business class is named Blog and is located under a namespace called "Business" so that it won't clash with the codebehind classes. Here's how the method that adds a new message is implemented:

Public Sub InsertMessage(ByVal title As String, ByVal message As String)
    Dim cmd As New OleDbCommand("INSERT INTO Blog_Messages (Title, _
        Message) VALUES (?, ?)")
    cmd.Parameters.Add("Title", OleDbType.VarChar).Value = title
    cmd.Parameters.Add("Message", OleDbType.LongVarChar).Value = _
        message.Replace(m_brChar, "<br>")
    ExecuteCommand(cmd)
End Sub

The method takes in all the values it needs to add a new record. There is no need to pass in the message ID since it's auto-incremented in the database table and the AddedDate uses the current date as the default value. The method defines a SQL INSERT command and executes it with the ExecuteCommand helper method. Note that the value for the Message parameter is the message text passed in as input, whereas the new-line characters are replaced with the HTML <br> tag. The administrator is allowed to use HTML formatting in his or her posts, but since new lines are so frequent, they are automatically converted to HTML to spare all that typing. As you may have already guessed, the ExecuteCommand helper method simply executes the command object that it takes as input:

Private Sub ExecuteCommand(ByVal cmd As OleDbCommand)
    cmd.Connection = m_Connection
    Try
        m_Connection.Open()
        cmd.ExecuteNonQuery()
    Finally
        m_Connection.Close()
    End Try
End Sub

The ExecuteCommand method sets the command's connection, which is created in the class's constructor method using a connection string retrieved from a custom key defined in the appSettings section of web.config. Then the connection is opened, the command is executed in a Try block, and the connection is closed in the corresponding Finally block. The connection is closed even if the ExecuteNonQuery method throws an exception. There is no Catch block, though, because I don't have to perform any business-specific operation, such as rolling back a transaction or logging an error. I just want to catch and handle the exception directly from the calling page, for example, to show a user-friendly error message. The other Insertxxx, Updatexxx, and Deletexxx methods work similarly, so I won't cover them all. It's valuable, though, to take a look at the InsertComment method because, in addition to adding a new record in the Blog_Comments table, it also sends a notification e-mail containing the comment text and some of the message it refers to the blog's owner.

With this feature in place, the blog owner doesn't have to continually load the blog and check for new e-mails. Of course, if the blog is very popular and gets a lot of comments, you might get too many e-mails. Therefore, the application's web.config should have a custom key that lets the administrator decide whether she wants to be notified by e-mail of any new comment. Yet another custom key is then required to store the destination e-mail address. These two settings are named Blog_SendNotifications and Blog_AdminEmail, respectively, and they are declared within the appSettings section of web.config (together with the custom key for the connection string) as follows:

<add key="Blog_SendNotifications" value="1" />
<add key="Blog_AdminEmail" value="mbellinaso@vb2themax.com" />
The Blog_SendNotifications value of 1 means that the notification feature is active and any other value or the absence of the key means that it is inactive.

After inserting the new comment and checking whether the comment notification is turned on you have to build the e-mail's body. In addition to the comment text, you also want to include the title, date, and time the parent message was posted so there is a clear connection between the original message and the comments. Retrieving the data of the parent message is performed by the code in Figure 2, with a command that retrieves the proper fields from the single message record in question.

Figure 2 Retrieving Parent Message Data (Close Figure 2)
Dim sendNotifications As String = ConfigurationSettings.AppSettings( _
    "Blog_SendNotifications")
If Not sendNotifications Is Nothing AndAlso sendNotifications = "1" Then
    Dim msgTitle, msgDate As String
    ' read a few details of the parent message
    cmd.CommandText = "SELECT Title, AddedDate FROM Blog_Messages" & _
        "WHERE MessageID = " & messageID
    m_Connection.Open()
    Dim reader As OleDbDataReader = cmd.ExecuteReader( _
        CommandBehavior.CloseConnection)
    If reader.Read Then
        msgTitle = reader("Title")
        msgDate = CType(reader("AddedDate"), Date).ToString("hh:mm tt" & _
           "on dddd, MMMM dd")
        reader.Close()
    End If
•••

Once you have all the required data, you can build and send the e-mail message. The e-mail's text is built through a call to the String.Format method, which takes in a template string with numeric placeholders (of the form {n}) and the values that replace the various placeholders in the template:

' build the msg's content
Dim msg As String = String.Format( _
   "{0} (email: {1}) has just posted a comment the message ""{2}"" " & _
   "of your BLOG that you posted at {3}. Here's the comment:{4}{4}{5}", _
   author, email, msgTitle, msgDate, Environment.NewLine, comment)
' send the mail
System.Web.Mail.SmtpMail.Send("WebLogger", _
    ConfigurationSettings.AppSettings("Blog_AdminEmail"), _
    "New comment notification", msg)
End If

The most important element of the business class is the GetData method, which retrieves the messages posted in the specified interval, along with their respective comments. Here you have to choose whether you want to use a DataReader or a DataAdapter plus a DataSet to retrieve and read the data. The DataReader is often the best option in Web applications, especially when you don't need a locally editable and cacheable copy of the data. In this particular case, however, I chose the DataSet because it allows you to store multiple tables and create parent-child relationships between them. That makes it easy to retrieve all the parent and child records in a single shot and navigate from parent records to their child data without executing separate queries, as would be required if I used the DataReader approach. This method is simpler for the programmer and saves database resources and network traffic because fewer SQL statements are sent to the database.

The ability to create relationships between tables is another cool feature because you can add calculated columns to the tables which evaluate expressions with references to other tables in the relationship. I created a relationship between the Blog_Messages and Blog_Comments tables against the MessageID field, just as I did for the physical database in Figure 1. I wanted the DataTable with the data from Blog_Messages to have a calculated column that returns the number of child comments for that message record. If I used a DataReader, this would require separate queries or at least a subquery (something that is not possible with Access). With a DataSet, this is performed with the disconnected copy of the data without asking the database to count the child records.

Let's see how the method works. It takes two dates as input, uses a DataAdapter to retrieve the messages posted in that interval, and then stores them in a DataSet table named Messages, like so:

Public Function GetData(ByVal fromDate As Date, ByVal toDate As Date) _
    As DataSet
    Dim ds As New DataSet()
        interval
    Dim da As New OleDbDataAdapter("SELECT * FROM Blog_Messages WHERE" & _
        "AddedDate BETWEEN ? AND ?" & _
        "ORDER BY AddedDate DESC", m_Connection)
    da.SelectCommand.Parameters.Add("FromDate", OleDbType.Date).Value = _
        fromDate
    da.SelectCommand.Parameters.Add("ToDate", OleDbType.Date).Value = _
        toDate.AddDays(1)
    m_Connection.Open()
    da.Fill(ds, "Messages")
•••

Note that you must add one day to the end date, so that the messages posted the day of the passed-in end date are included in the resultset. If you want to get only the comments of the returned messages, then in the SELECT statement you use an IN filter which contains a comma-delimited list of all the message IDs of the records returned by the query I just showed (see Figure 3).

Figure 3 Filtering Comments (Close Figure 3)
If ds.Tables("Messages").Rows.Count > 0 Then
    Dim inFilter As String = ""
    Dim dr As DataRow
    For Each dr In ds.Tables("Messages").Rows
        If inFilter = "" Then
            inFilter = dr("MessageID")
        Else
            inFilter &= ", " & dr("MessageID")
        End If
    Next
    ' build the SELECT for the Comments table
    da.SelectCommand.CommandText = "SELECT * FROM Blog_Comments" & _
        "WHERE MessageID IN (" & _
        "inFilter & ") ORDER BY AddedDate DESC"
Else
    da.SelectCommand.CommandText = "SELECT * FROM Blog_Comments" & _
        "WHERE MessageID = -1"
End If
da.Fill(ds, "Comments")
m_Connection.Close()

Then you create the relationship I just described between the two tables, like this:

ds.Relations.Add(New DataRelation("MsgComments", _
    ds.Tables("Messages").Columns("MessageID"), _
    ds.Tables("Comments").Columns("MessageID")))
Next you add a calculated column to the Messages table that returns the number of child comments:
ds.Tables("Messages").Columns.Add("CommentsCount", _
    GetType(Integer), "Count(Child(MsgComments).CommentID)")
The function Child(MsgComments) returns all the child data rows of the specified relationship, and the Count function works the same way it does in SQL.

One last detail to note is that before returning the filled DataSet and terminating the function, if the Comments table is empty, this calculated column will evaluate the expression as NULL, not 0, which will cause problems later when you show this value in the ASP.NET page or use it in other expressions. To solve the problem, you can add a mock comment that doesn't refer to any message:

If ds.Tables("Comments").Rows.Count = 0 Then
        Dim dr As DataRow = ds.Tables("Comments").NewRow()
        dr("CommentID") = -1
        dr("Author") = "none"
        dr("Email") = "none"
        dr("Comment") = "none"
        dr("AddedDate") = Date.Today
        ds.Tables("Comments").Rows.Add(dr)
        dr.AcceptChanges()
    End If

    Return ds
End Function
This way, the expression will evaluate to 0 if a message does not have any child comments.

Showing Messages and Child Comments

Now that the business layer is complete, the next step is to write the presentation layer where you can best appreciate the power of ASP.NET. Most of the work is performed on a single page, Default.aspx, which will render the messages and comments and allow the authenticated administrator to moderate the comments and edit her own messages (see the bottom pane of Figure 4).

Figure 4 The Blog in Administrator Mode
Figure 4 The Blog in Administrator Mode

Default.aspx shows a list of messages grouped by day; each message can have zero or more comments. Initially, the comments are hidden, but when the user clicks on the View link, an inner list of comments is dynamically expanded or collapsed. Multiple messages can have their lists of comments expanded at the same time, like a TreeView control with a single level of sub-nodes. Developing this page presents some challenges including how to sort the messages by day and group them in separate HTML tables, and how to show or hide the inner lists of child records.

Let's start with the first issue. Assume that you want to show five messages on the page, the first three of which were posted the same day and the other two on another day. Also assume that you want to keep each message initially wrapped in its own HTML table. If you want to group the first three and the remaining two messages in two separate tables, you should remove the table's opening and closing tags from the messages in the middle of the table (the messages that are not the first or the last of that day), remove the closing tag from the first message's table, and the opening tag from the last message's table. Because I want to use a templated data-bound control to render this view, and since I must have full control over all the produced HTML, I chose to use the Repeater control. The Repeater control has no predefined output or layout beyond what you specify in its templates. The code in Figure 5 is a partial definition of the Repeater's template.

Figure 5 Repeater Template (Close Figure 5)
<asp:repeater id="Blog" Runat="server">
    <HeaderTemplate><table width="100%" border="1"><tr><td>
    </HeaderTemplate>
    <ItemTemplate>
        <asp:Literal Runat="server" ID="DayBox">
            </td></tr></table><table width="100%" border="1"><tr><td>
        </asp:Literal>
        <asp:Panel Runat="server" ID="DayTitle">
            <h3><%# DataBinder.Eval(Container.DataItem, _
                "AddedDate", "{0:dddd, MMMM dd}") %></h3>
        </asp:Panel>
        <!-- here is the rest of the template that shows the data... -->
    </ItemTemplate>
    <FooterTemplate></td></tr></table></FooterTemplate>
</asp:repeater>

The HeaderTemplate includes the table's opening tags, and the FooterTemplate closes the table. The tags that open and close the various tables that group the messages by date are defined in a Literal control within the ItemTemplate section. By toggling this Literal control's visibility, you decide whether to output those tags, and thus when to close the current table and open a new one. The question is when to hide the Literal so that the current table is kept open and the messages grouped and when to show it so that the table is closed and a new table is opened.

The answer is simple: the messages are grouped together while the date of the message being processed is the same as that of the previous message. When the date changes, the Literal control is shown and the new group is started. The other question is when and where the control's content should be visible. You have to decide this while the data item (the blog's message record) is being processed and bound to the Repeater's template in the Repeater's ItemDataBound event. Here you can read all the values from the current data item and compare them with the previous message's data, which has been stored in a static variable so that the value is preserved between method calls. You get a reference to the template's Literal control and can set its visibility accordingly. Here's the code:

Private Sub Blog_ItemDataBound(...) Handles Blog.ItemDataBound
    Static prevDayDate As Date

    If e.Item.ItemType <> ListItemType.Item AndAlso _
        e.Item.ItemType <> ListItemType.AlternatingItem Then Return

    Dim dayDate As Date = e.Item.DataItem("AddedDate")
    Dim isNewDay As Boolean = (dayDate.ToShortDateString() <> _
        prevDayDate.ToShortDateString())
    prevDayDate = dayDate
    CType(e.Item.FindControl("DayTitle"), Panel).Visible = isNewDay
    CType(e.Item.FindControl("DayBox"), Literal).Visible = ( _
        isNewDay AndAlso e.Item.ItemIndex > 0)
End Sub

As you can see, I set the Visible property for a Panel control (also declared in the same template) that displays the date for the day of the current table. This panel is shown if the date of the current message differs from the date of the previous message, or it shows the default date if this is the first message being bound. The Literal's visibility has one more constraint for being set to True: the message being bound must not be the first one because in that case the day's table is opened with the tags declared in the Repeater's HeaderTemplate section. To dynamically collapse and expand the list of comments without a postback to the server and a page reprocessing, place the comments within a <DIV> tag whose display style can be set to "none" or to an empty string to hide or show it, respectively. The DIV is declared as follows, assigning it an ID based on the data-item being bound:

<div style="display:'none'; margin-left:2.0em; margin-top:.8em; "
    ID='<%# "div" & Container.DataItem("MessageID") %>'>
    <!-- put here the Comments DataList... -->
</div>

I assigned the ID this way so that it is unique for each DIV (remember that there is one for each message). To expand or collapse the DIV, I used a hyperlink that calls custom JavaScript code that takes input from the DIV's ID:

<asp:HyperLink Runat="server"
    Visible='<%# Container.DataItem("CommentsCount") > 0 %>'
    NavigateUrl='<%# "javascript:ToggleDivState(div" & _
       Container.DataItem("MessageID") & ");" %>'>
    View
</asp:HyperLink>

Also note that the Visible property is bound to an expression that returns True only if the message has comments to show (if the value in its calculated CommentsCount column is greater than 0). The ToggleDivState just inverts the value of the DIV's display style to make it alternately visible or hidden:

function ToggleDivState(ctrl)
{
    div = eval(ctrl);
    if (div.style.display == "none")
        div.style.display = "";
    else
        div.style.display = "none";
}

Now let's look at the comments feature. A DataList will do the job this time because its table layout is exactly what I want. Typically, the DataSource property of a template control, or any other data-bound list control, is assigned programmatically in the codebehind (or in a server-side script section). In this case, however, I don't have a direct reference to the DataList because it is created dynamically by the parent Repeater. Like most properties, though, it can have a data binding expression that sets its value at run time. If you were using the DataReader approach to retrieve the data, the DataSource property could be bound to a custom method that took in the message's ID and returned the child comments in a DataTable, a DataReader, or any other data type that implements the IEnumerable interface. In this case it is not necessary because all the data you need has been already stored in the same DataSet that contains the messages. Since you created a relationship between the table with the message and the table with the comments, you can easily retrieve an array with the child comments using the GetChildRows method of the current data item's DataRow. The expression is declared like so:

<asp:DataList Runat="server" DataSource=
'<%# Container.DataItem.Row.GetChildRows("MsgComments") %>'>

Complete the DataList's ItemTemplate with binding expressions to show the author name and e-mail address, the message text, and its date, and the code that outputs the blog's content is complete. Figure 6 shows the finished code for the output module.

Figure 6 Repeater and Nested Collapsible Comments (Close Figure 6)
<asp:repeater id="Blog" Runat="server">
    <HeaderTemplate>
        <table width="100%" border="1" cellpadding="0" cellspacing="0">
        <tr><td>
    </HeaderTemplate>
    <ItemTemplate>
        <asp:Literal Runat="server" ID="DayBox">
            </td></tr></table><div align="right"><a href="#TopPage">Top
            </a></div><br>
            <table width="100%" border="1" cellpadding="0"
             cellspacing="0"><tr><td>
        </asp:Literal>
        <asp:Panel Runat="server" ID="DayTitle">
            <h3><%# DataBinder.Eval(Container.DataItem, "AddedDate",
                 "{0:dddd, MMMM dd}") %></h3>
        </asp:Panel>

        <p align="justify">
            <b><%# Container.DataItem("Title") %></b>
            @ <%# DataBinder.Eval(Container.DataItem, "AddedDate",
               "{0:hh:mm tt}") %>
            <br>
            <%# Container.DataItem("Message") %>
            <br>
            -- <%# Container.DataItem("CommentsCount") %>
            comments:
            <asp:HyperLink Runat="server" Visible='
                <%# Container.DataItem("CommentsCount") > 0 %>'
                NavigateUrl='<%# "javascript:ToggleDivState(div" &
                Container.DataItem("MessageID") & ");" %>'>
                View</asp:HyperLink>
            - <a href='<%# "javascript:ShowCommentBox(" &
              Container.DataItem("MessageID") & ");" %>'>
                Post your own comment</a>
            <div style="display:'none'; margin-left:2.0em;
                margin-top:.8em; "
                ID='<%# "div" & Container.DataItem("MessageID") %>'>
                <asp:DataList Runat="server" Width="500px"
                    ItemStyle-BackColor="whitesmoke"
                    AlternatingItemStyle-BackColor="white"
                    DataSource='<%# Container.DataItem.Row.GetChildRows
                    ("MsgComments") %>'
                    OnDeleteCommand="Comments_DeleteCommand"
                    OnItemCreated="Comments_ItemCreated"
                    OnEditCommand="Comments_EditCommand">
                    <ItemTemplate>
                        <p align="justify">
                            <u>Posted by <a class="comment"
                               href='<%# "mailto:" &
                               Container.DataItem("Email") %>'>
                            <%# Container.DataItem("Author") %></a></b>
                            @ <%# CType(Container.DataItem("AddedDate"),
                               Date).ToString("hh:mm tt, MMMM dd") %>
                            </u>
                            <br>
                            <%# Container.DataItem("Comment") %>
                        </p>
                    </ItemTemplate>
                    <SeparatorTemplate><br></SeparatorTemplate>
                </asp:DataList>
            </div>
        </p>
    </ItemTemplate>
    <FooterTemplate>
        </td></tr></table><div align="right"><a href="#TopPage">Top</a>
        </div><br>
    </FooterTemplate>
</asp:repeater>

Selecting an Interval and Loading the Blog

Thus far I've covered the definition of the page content in the ASPX file, but I haven't yet reviewed any code to actually load the blog's content. To let the user select an interval, I used the Calendar control with the SelectionMode property set to DayWeekMonth, so that the user can select a single day, a week, or the entire month. It is a good idea to provide textboxes for a user's desired start and end dates in case he wants to select the last two weeks or the last 45 days, for example. Figure 7 shows the new controls added to the page with an entire week selected in the Calendar.

Figure 7 Intervals
Figure 7 Intervals

When the page isn't loaded because of a postback, you must select a default interval, such as the last week. However, for frequently updated blogs, it may be better to load the data for fewer days or for the entire last month for rarely updated blogs. As with the comment notification feature, the best option is to leave the choice to the blog administrator using a custom key in the web.config file that would allow them to specify the number of days for the default interval. The following code shows how the custom key is read from the file, parsed into an Integer, and used to calculate an interval for the last n days, as well as how the interval is highlighted in the calendar:

Private Sub Page_Load(...) Handles MyBase.Load
    If Not IsPostBack Then
        Dim defPeriod As Integer = Integer.Parse( _
            ConfigurationSettings.AppSettings("Blog_DefaultPeriod"))
        Dim fromDate = Date.Today.Subtract(New TimeSpan(defPeriod -_
            1,0,0,0))
        BlogCalendar.SelectedDates.SelectRange(fromDate, Date.Today)
        BindData()
    End If
End Sub

The start date is calculated by subtracting n-1 days from today's date. The call to BindData loads the data for the selected interval and binds it to the Repeater and its inner controls. This method calls the GetData method of the previously developed Blog business class and passes in the start and end dates of the calendar's selected date, which are read from its SelectedDates collection:

Private Sub BindData()
    Dim ds As DataSet = m_BlogManager.GetData( _
        BlogCalendar.SelectedDates(0), _
        BlogCalendar.SelectedDates(BlogCalendar.SelectedDates.Count - 1))
    Blog.DataSource = ds.Tables("Messages").DefaultView
    Blog.DataBind()
End Sub

If a single day is selected, the SelectedDates will have only one item and the start and end dates will be the same. When the user clicks the calendar, the page is posted back, the new interval is automatically selected, and you handle the calendar's SelectionChanged event to call the BindData again for the new interval. Finally, you must handle the Load button's Click event to select the specified custom interval in the calendar and load the blog's data. In this event procedure I parse the content of the two input controls and get two dates. Although I will eventually add validators to ensure that the date format is correct before the form is submitted, the parsing could throw an exception if the date format is not valid. In that case, I take today's date (see Figure 8).

Figure 8 Loading the Dates (Close Figure 8)
Private Sub LoadBlog_Click(...) Handles LoadBlog.Click
    Dim fromDate, toDate As Date
    Try
        fromDate = Date.Parse(IntervalFrom.Text)
    Catch
        fromDate = Date.Today
    End Try
    ' do the same for the toDate...
    ' if toDate is before fromDate, set toDate = fromDate
    If toDate < fromDate Then toDate = fromDate
    BlogCalendar.SelectedDates.SelectRange(fromDate, toDate)
    BlogCalendar.VisibleDate = toDate
    BindData()
End Sub

Note the use of the calendar's VisibleDate to ensure that the end date is visible in the calendar. This is necessary because if the user selects a period two months in the past, the selection would not be visible in the calendar. Instead, the calendar would show the current month, which wouldn't be very clear.

Popup Calendar

As it stands now, if the user wants to select an interval different from a single day, week, or month, they must manually type in the start and end date in the two textboxes and refer to an external calendar. In addition, they might type the date in an invalid format. For these reasons, it is a good practice to offer a popup calendar. When a date is clicked, the calendar should close and the date should appear in the main window's textbox control. This feature can be easily reproduced in ASP.NET using the Calendar control and a bit of client-side JavaScript.

First let's take care of the parent window's ASPX code. I added an image link that opens the popup window by calling the following JavaScript function:

<a href="http://javascript:PopupPicker('IntervalFrom', 200, 200);" >
    <img src="images/calendar.gif" border="0" >
</a>

The JavaScript procedure expects to receive the name of the textbox control that will be filled with the selected date, along with the width and height of the popup calendar window that will be opened. Here's the JavaScript code that goes in the <script> section already defined at the top of the page:

function PopupPicker(ctl,w,h)
{
    var PopupWindow=null;
    settings='width='+ w + ',height='+ h;
    PopupWindow=window.open('DatePicker.aspx?Ctl=' +
        ctl,'DatePicker',settings);
    PopupWindow.focus();
}
This procedure uses window.open to open the popup window with a specified, unresizable dimension, with no scrollbar, menu, toolbar, or status bar. The first parameter is the URL of the page to load into the new window, and in the code I just showed it loads a DatePicker.aspx page with a ctl parameter in the query string, whose value is passed in as input to the PopupPicker procedure.

Now that I'm finished in the main page, I have to write the DatePicker.aspx page that renders the calendar. This page has a Calendar control with its Width and Height properties set to 100 percent, so that it covers the whole page. The other important thing to note in the ASPX file is the client-side JavaScript procedure that takes a string as input and uses it as the value for the parent form's input control and whose name is passed on the query string. Finally, it closes the popup form itself. The JavaScript code is in Figure 9.

Figure 9 Setting the Date (Close Figure 9)
<script language="Javascript">
function SetDate(dateValue)
{
    // retrieve from the querystring the value of the ctl param,
    // that is, the name of the input control on the parent form
    // that the user wants to set with the clicked date
    ctl = window.location.search.substr(5);
    // set the value of that control with the passed date
    thisForm =
      window.opener.document.forms[0].elements[ctl].value = dateValue;
    // close this popup
    self.close();
}
</script>

When the user clicks a link in the Calendar control, instead of the normal processing that submits the form and selects the clicked day, I want the custom JavaScript procedure to be invoked. By default, all the links rendered by the Calendar control generate a postback to the server. What I want instead is for them to point to the custom SetDate procedure. Changing the default output for the table cells that contain the day links is pretty easy thanks to the Calendar's DayRender event, which is raised every time a day is rendered and which provides a reference to the table cell being created. The following code snippet replaces the default cell's content with my own hyperlink control, which has the same text but points to the JavaScript procedure:

Private Sub DatePicker_DayRender(...) Handles DatePicker.DayRender
    Dim hl As New HyperLink()
    hl.Text = CType(e.Cell.Controls(0), LiteralControl).Text
    hl.NavigateUrl = "javascript:SetDate('" & _
        e.Day.Date.ToShortDateString() & "');"
    e.Cell.Controls.Clear()
    e.Cell.Controls.Add(hl)
End Sub

The value passed to the JavaScript procedure is the date of the clicked day in short format (typically mm/dd/yy). This value will be used for the input control on the parent form. Figure 10 shows the resulting popup window.

Figure 10 Popup Calendar
Figure 10 Popup Calendar

As you can see, the ASP.NET server controls can be highly customized. Another time when you may want to use the DayRender event this way is when you need to redirect to another page with the date passed in the query string, instead of redirecting to the second page from the server after a postback with the date. To do this, just replace the line where you set the hyperlink's NavigateUrl property with something similar to the following:

hl.NavigateUrl = "SecondPage.aspx?Date=" & e.Day.Date.ToShortDateString()

Posting Comments

In Figure 4 you saw that below each message there is a "Post your own comment" link. When the user clicks it, he is presented with a box with input controls to post a comment. You can guess how this works because I used the same technique to build the collapsible lists of comments. The Comments box with its input controls is declared within a DIV whose display style is initially set to "none" so that it's not visible. When the link is clicked, the style is changed, and the page is scrolled to the bottom to make it visible. I define a single comment box at the bottom of the page, not one for each message, to avoid sending unnecessary HTML code that would slow down the page. Figure 11 shows the comment box and the required input controls.

Figure 11 Posting a Comment
Figure 11 Posting a Comment

How do you specify the message to which the comment is being posted? A good solution is to store the parent message's ID in a hidden ASP.NET textbox when the "Post your own comment" link is clicked. Later, when the Post button is pressed, this value can be retrieved from the codebehind. Note that you can't use the Visible property to make the control hidden, because when Visible is set to False the control is hidden and the HTML code isn't sent to the client at all. You must use the same display style used for the DIV. The DIV and the textbox are declared as follows:

<div id="CommentBox" style="DISPLAY: none">
   <a name="CommentBoxAnchor"></a>
   <asp:textbox id="ParentMessageID" style="DISPLAY: none"
    runat="server" />
Notice that I also used an anchor which will be used to ensure that the comment box is actually visible if the page is long and the message the user wants to comment on is at the top of the page. The link is declared like this:
<a href='<%# "javascript:ShowCommentBox(" &
    Container.DataItem("MessageID") & ");" %>'>Post your own comment</a>
The ShowCommentBox JavaScript routine takes in the ID of the message to comment on and uses it as the value for the hidden textbox control just declared:
function ShowCommentBox(msgID)
{
    document.forms[0].ParentMessageID.value = msgID;
    ShowCommentBox2();
}

The code that actually makes the comment box visible and scrolls down the page is in a separate routine (the ShowCommentBox2 procedure). I will call this procedure again when I want to show the comment box without setting the value attribute of the hidden textbox control:

function ShowCommentBox2()
{
    CommentBox.style.display = "";
    window.location.href = '#CommentBoxAnchor';
}
All I have left to do is to handle the click on the Post button to call the InsertComment of the Business.Blog instance and rebind the updated data to the Repeater:
Private Sub PostComment_Click(...) Handles PostComment.Click
    m_BlogManager.InsertComment(Integer.Parse(ParentMessageID.Text), _
        Author.Text, Email.Text, Comment.Text)
    BindData()

    ' reset the value of the input controls
    ParentMessageID.Text = ""
    ' reset the other visible textboxes...
End Sub

Administering the Blog

The code for the user tasks is almost complete now. Users can read the messages for the selected interval and post comments. The blog owner, on the other hand, has to add, insert, and delete messages and comments by acting directly on the data store. To allow this access, the next major step is to develop a login page and modify the blog's main page so that when the administrator is logged in, the page shows additional controls for performing administrative operations. The login page is composed of textboxes for the user name and password, a checkbox for the "persistent login" option, and a submit button. Since you'll have only a single administrator and don't need role-based security, storing the credentials in the web.config file is enough. Figure 12 shows the codebehind class. If the specified credentials are valid, it authenticates the user and redirects to the Default.aspx page.

Figure 12 Codebehind Class for the Login Page (Close Figure 12)
Imports System.Web.Security

Public Class Login
  Inherits System.Web.UI.Page

  ' Web Form Designer Generated Code here...

    Protected UserName As System.Web.UI.WebControls.TextBox
    Protected Password As System.Web.UI.WebControls.TextBox
    Protected Persistent As System.Web.UI.WebControls.CheckBox
    Protected WithEvents LoginUser As System.Web.UI.WebControls.Button
    Protected InvalidLogin As System.Web.UI.WebControls.Label

    Private Sub Page_Load(ByVal sender As System.Object, ByVal e As _
        System.EventArgs) Handles MyBase.Load
        If Not Page.IsPostBack Then
            ' if the querystring contains a "Action=logout" param, logout
            ' and delete the cookie
            If Request.Params("Action") = "logout" Then
                FormsAuthentication.SignOut()
            End If
        End If
    End Sub

    Protected Sub LoginUser_Click(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles LoginUser.Click
        ' check username and password
        If FormsAuthentication.Authenticate(UserName.Text, Password.Text)
        Then
            ' if ok, save the cookie
            FormsAuthentication.SetAuthCookie(UserName.Text, _
                Persistent.Checked)
            ' redirect to Default.aspx
            Response.Redirect("Default.aspx", True)
        Else
            ' if wrong credentials, show the error message
            InvalidLogin.Visible = True
        End If
    End Sub
End Class

In the Default.aspx page, I'll add the editing controls that will be visible only if the user is authenticated. At the top of the page I declare a panel with a "logout" link that points to Login.aspx with the "action=logout" parameter in the query string, and a table with the textboxes to specify the message's title and content. This panel is shown or hidden in Page_Load, as follows:

MessageBox.Visible = User.Identity.IsAuthenticated
When the administrator fills the textboxes and clicks the Post button, the server-side Click event is raised, and in its event handler I use the InsertMessage function to add the new message to the database and call BindData to load it in the Repeater.

Now it's time to add the editing functionality. You can do this by adding a LinkButton control to the Repeater's ItemTemplate:

<asp:LinkButton CausesValidation="False" runat="server" Text="Edit"
 CommandName="Edit" CommandArgument='<%# Container.DataItem("MessageID")
       %>'
 Visible='<%# User.Identity.IsAuthenticated %>'/>

The link is visible only if the user is authenticated (it works as it did for the MessageBox panel, but here you show or hide it by means of a data binding expression in the ASPX file). The CommandArgument property contains the ID of the message to edit, but you also must specify the CommandName as "Edit" because you have another LinkButton for deleting the message and you want to make sure that it's clear which of the two buttons was clicked. The code for the Repeater's ItemCommand event handler is shown in Figure 13.

Figure 13 Handling Blog_ItemCommand (Close Figure 13)
Private Sub Blog_ItemCommand(...) Handles Blog.ItemCommand
    Dim msgID = Integer.Parse(e.CommandArgument)
    If e.CommandName = "Edit" Then
        Dim msgTitle, msgText As String
        m_BlogManager.GetMessageData(msgID, msgTitle, msgText)
        MessageID.Text = msgID
        Title.Text = msgTitle
        Message.Text = msgText
    Else
        m_BlogManager.DeleteMessage(msgID)
        BindData()
    End If
End Sub

The code first retrieves the message's ID passed with the clicked button's CommandArgument property. Then the administrator decides whether to delete or edit the message according to the button's CommandName. When it equals "Edit," the current data for the specified message is retrieved and used to fill the MessageBox's textboxes so that the administrator sees the current text and can edit it. When the Post button is clicked, if the MessageID textbox is empty, it means the administrator is posting a new message; otherwise the textbox contains the ID of the message being edited. Here's part of the Click event handler:

Private Sub PostMessage_Click(...) Handles PostMessage.Click
    If Not User.Identity.IsAuthenticated Then _
        Response.Redirect("Login.aspx", True)
        If MessageID.Text.Trim().Length > 0 Then
            m_BlogManager.UpdateMessage(Integer.Parse(MessageID.Text), _
                Title.Text, Message.Text)
        Else
            m_BlogManager.InsertMessage(Title.Text, Message.Text)
        End If
        ' reset the textboxes to an empty string, and call BindData...
    End If
End Sub

The Add New and Edit functionalities are now complete (Figure 4 shows how the page looks in administrator mode). There is one more thing that's worth adding to the Delete functionality. As it stands now, if the administrator mistakenly clicks the Delete link, the message is deleted immediately since a confirmation was not requested. Adding a confirmation popup dialog is simple—it just needs some JavaScript in response to the hyperlink's onClick event. This is done by adding an entry in the control's Attributes collection when the link is created—that is, when the Repeater's item is created. I just need to handle the Repeater's ItemCreated event for both the odd and even items, get a reference to the Delete LinkButton, and add the JavaScript confirmation popup:

Private Sub Blog_ItemCreated(...) Handles Blog.ItemCreated
    If e.Item.ItemType <> ListItemType.AlternatingItem AndAlso _
    e.Item.ItemType <> ListItemType.Item Then Exit Sub
    Dim lnkDelete As LinkButton = CType( _
        e.Item.FindControl("DeleteMessage"), LinkButton)
    lnkDelete.Attributes.Add("onclick", _
        "return confirm('Are you sure you want to delete this" & _
        "message?');")
End Sub
If the administrator clicks Cancel, the JavaScript returns false, the page is not submitted, and the message is not deleted.

The editing and deleting of comments are implemented in the same way, so I won't explain everything in detail again. However, you can find the complete implementation in the code download for this article. There are just a couple of details worth explaining here. Every time I had to handle a Repeater's event, I used the Visual Basic® .NET Handle keyword, which enables you to associate a method with the event of a control instance declared with WithEvents. The same cannot be done with the inner DataList for the comments because it's dynamically created at run time and there isn't a WithEvents control variable for it. You can specify the event handlers directly in the control's declaration though, as shown here:

<asp:DataList Runat="server" OnDeleteCommand="Comments_DeleteCommand"
    OnItemCreated="Comments_ItemCreated"
    OnEditCommand="Comments_EditCommand" ...>

The other small detail is that when the administrator clicks the Edit link for a comment, the comment box must be shown and the page scrolled to the bottom to ensure it is visible. I did this previously for the "Post your own comment" link, but in that case the JavaScript routine that does this is directly associated with the link, and there is no round-trip to the server. Here the page is first posted back to pre-fill the edit textboxes with the comment, and the JavaScript routine is called when the page is resent to the client browser. To do this, I send some JavaScript to the client that just calls the ShowCommentBox2 routine written before, like this:

Sub Comments_EditCommand(...)
    ' fill the textboxes with the current data for the clicked comment
    •••
    Dim script As String = _
        "<script language=""JavaScript"">ShowCommentBox2();</script>"
    Me.RegisterStartupScript("ShowEditCommentBox", script)
End Sub
The RegisterStartupScript emits the specified block of JavaScript just before the page's server-side <form> tag is closed, ensuring that the CommentBox has already been created (otherwise you could get a wrong reference error when the CommentBox container is not found).

Validation for Multiple Virtual Forms

Where there are textboxes or other input controls, it is often a good idea to add validator controls to ensure that a value is supplied and that it's in the proper format and range. In this application, you must perform different validations according to the action that the user wants to take. If the user presses the Post button to submit a comment, you must ensure that he has provided his name and the comment text. If the Load Blog button is clicked, you must check that the format of the start and end dates is valid. I haven't added the validators yet because there is one more problem that I need to resolve.

I have three "virtual forms" with input controls: the comment box, the new message box, and the interval selection box. In an ASP.NET page you can have only one server-side form. This means that all the input controls, validators, and submit buttons are in the same form. Once the user has correctly filled the textboxes for the interval and clicked the submit button, the textbox validators validate the input. The validators for the comment and message boxes will prevent the form from being posted back when their textboxes don't have a value or the value is not formed correctly.

To solve the problem, I replaced the standard ASP.NET buttons with a custom control developed by James M. Venglarik, II for his MSDN® Magazine article "Selectively Enable Form Validation When Using ASP.NET Web Controls". This control creates buttons that disable a specified list of validators using some client-side JavaScript, and thus makes it possible to have buttons that, before submitting the page, validate some input controls but not others. Once the control is referenced by the page, here's how the Load Blog button is declared:

<nfvc:NoFormValButton ID="LoadBlog" Runat=Server Text="Load"
    NoFormValList="RequireAuthor,RequireComment,ValidateEmailFormat" />

The NoFormValList property specifies a comma-separated list of validators to disable when it's clicked, which in this case are all the validators for the textboxes in the comment and message boxes.

Conclusion

The application built in this article is now fully functional. You can upload it to your own server and write your blog, or you can see it live at http://www.bytecommerce.com/blog. The DataSet's relationship capabilities and the ability to create nested DataLists and DataGrids make it very easy to render master-detail reports, which is really what you're doing in this blog application. The flexibility of the template controls means you can create almost any layout. You can group multiple data items in tables and customize the behavior of the default implementation, as you saw when adding the confirmation popup to the Delete buttons. The controls combined with a bit of client-side JavaScript, to script their behavior and glue them together, makes for a complete and feature-rich ASP.NET reporting application.

For related articles see:
Cutting Edge Archive
ASP.NET: Selectively Enable Form Validation When Using ASP.NET Web Controls

For background information see:
ASP.NET Security: An Introductory Guide to Building and Deploying More Secure Sites with ASP.NET and IIS
ASP.NET Security: An Introductory Guide to Building and Deploying More Secure Sites with ASP.NET and IIS, Part 2
.NET Data Access Architecture Guide
http://weblogs.asp.net

Marco Bellinasois technical editor of VB-2-The-Max (http://www.vb2themax.com) and works as a software developer and trainer for Code Architects Srl, an Italian company that specializes in the .NET Framework. He is a coauthor of several books including ASP.NET Website Programming (C# and Visual Basic .NET editions, by Wrox Press, 2003).


Plug-Ins: Let Users Add Functionality to Your .NET Applications with Macros and Plug-Ins

Plug-Ins

Let Users Add Functionality to Your .NET Applications with Macros and Plug-Ins

Jason Clark
This article assumes you're familiar with .NET and C#
Level of Difficulty 1 2 3
Code download available at: Plug-Ins.exe (135KB)
SUMMARY

Most user applications benefit from the ability to be extended by other developers. It's often easier and more efficient to extend an existing application that users are already familiar with and trained on than it is to develop one from scratch. Thus, extensibility makes your application more attractive. You can build extensibility into your application by supporting features like plug-ins or macros. This is easily accomplished using the .NET Framework even if the core application isn't a .NET Framework app. In this article, the author describes extensibility features of the .NET Framework including late binding and reflection and how to use them, along with plug-in security considerations.

Contents
The Ideal Extensible Application
.NET Framework Extensibility Features
Getting Started with Extensibility
Loading: Binding to Code at Run Time
Discovery: Finding Code at Run Time
Activation: Create Instances and Call Methods
Secure Extensibility
Adjusting Security Policy
Extensible Application Design
Versioning
Robustness
Security Considerations
Unloadable Plug-in Assemblies
Managed Plug-ins in Unmanaged Apps
Wrapping It Up

I
magine the perfect text editor. It launches in under two seconds, supports context coloring and automatic indenting for popular programming languages, supports Multiple Document Interface (MDI) as well as that cool tabbed document arrangement that has become so popular. The problem with conceiving the perfect text editor is that perfect is in the eye of the beholder. These features may be my definition of a perfect text editor, but others will certainly have different criteria. Perhaps the most important feature the perfect text editor could have would be rich support for extensibility so that any developer can extend the application with features that they want.

An extensible text editor might support the creation of custom toolbars, menus, macros, and even custom document types. It would allow me to write plug-ins that hook into the editing process so that features like autocomplete, spell-checking, and other niceties could be added. Finally, the perfect text editor would let me write my plug-ins in any of a number of languages (my personal preference is C#).

To be honest, I want every application that I use to be extensible in this way. If I can customize my favorite applications by writing a little code here and there, that is great. And even if I don't, I know that somebody else will and I can then benefit from their extension by downloading it from the Internet. Thus begins my campaign to get all developers to write extensible applications.

The Ideal Extensible Application

Lots of applications can be modified using pluggable code. In fact, the entire Microsoft® Office suite of applications is so extensively customizable that folks write complete custom applications using Office as the platform. And yet, even with all of this customizability, I have yet to write my first plug-in for Microsoft Word (an application that I use nearly every day).

The reason for this is simple. Microsoft Office, with all of its features, simply doesn't meet all of my criteria, which include:

  • Simplicity. I want to be able to manipulate my pluggable application with very simple software tools with which I am already familiar.
  • Access. I want my plug-ins to have access to some subset of the objects and functionality built into the application. This access should be as natural as if it were part of my programming language of choice.
  • Programming language. I want to use my programming language of choice, period.
  • Power. I want a rich API in addition to my access to the Document Object Model (DOM) of the application.
  • Security. I want to be able to download plug-ins that other people have written and made available over the Internet. I want to execute potentially malicious or buggy components without concern for the safety of my system.

This short list is a tall order. In fact, up until the release of the Microsoft .NET Framework, these criteria were simply too strict to be met by the typical application. But now I can show you how to use the .NET Framework to add all of these extensibility features to your managed and unmanaged applications.

.NET Framework Extensibility Features

Extensibility is built on late binding, which is the ability to discover and execute code at run time rather than at compile time (the more typical case). Over the years, many innovations have contributed to late binding, including DLLs and the COM API. The .NET Framework brings the simplicity of late binding to a whole new level. Let's take a look at a very simple code example to get a better understanding.

The code in Figure 1 shows how easy it is to use reflection to perform late binding on a managed object. If you build the code in Figure 1 into LateBinder.exe and run it, you can pass it the name of any assembly (such as one built from the code in Figure 2) as its command-line argument. LateBinder.exe will reflect over the assembly and create instances of the Form-derived classes in the assembly and make them MDI children of itself. Reflection in the .NET Framework makes late binding incredibly simple.

Figure 2 CircleForm.cs (Close Figure 2)
using System;
using System.Drawing;
using System.Windows.Forms;

// Form class that draws a circle in its client area
public class Circle:Form{
   protected override void OnPaint(PaintEventArgs args){
      Int32 diameter =
         Math.Min(ClientRectangle.Width,
            ClientRectangle.Height);
      args.Graphics.FillEllipse(Brushes.Blue,
         (ClientRectangle.Width-diameter)/2,
         (ClientRectangle.Height-diameter)/2, diameter,
            diameter);
   }
}
Figure 1 Late Binding (Close Figure 1)
using System;
using System.IO;
using System.Reflection;
using System.Windows.Forms;
using System.Drawing;

class App{
   public static void Main(String[] args){
      // Load the assembly whose name was passed on the command-line
      Assembly a = Assembly.Load(
         Path.GetFileNameWithoutExtension(args[0]));

      // Pass the assembly object to a constructed LateBinderForm
      Application.Run(new LateBinderForm(a));
   }
}

// Creates late-bound form instances for MDI children
class LateBinderForm:Form{
   public LateBinderForm(Assembly a){
      // Make it MDI
      IsMdiContainer = true;

      // Get the types in the assembly
      Type[] types = a.GetTypes();
      // Iterate and find types derived from Form Instantiate them
      foreach(Type t in types){
         if(t.BaseType == typeof(Form)){
            Form f = (Form)Activator.CreateInstance(t);
            f.MdiParent = this; // Set as MDI children
            f.Show();
         }
      }
   }
}

Reflection is one of the essential tools of the .NET Framework that promotes the development of extensible applications. It is one of four features I'll mention here that you can use to make your applications extensible.

The common type system After using the .NET Framework for a while you may begin to take the Common Type System (CTS) for granted. However, it is really one of the reasons that extensibility is so easy on this platform. The CTS defines a subset of object-oriented features that all managed languages must adhere to, such as rules for derivation, namespaces, object types, interfaces, and primitive types. The CTS sets the ground rules for code run by the common language runtime (CLR).

Reflection Reflection is the ability to discover, at run time, information such as the types implemented by an assembly or the methods defined by a type. Reflection is possible because all managed code is self describing through a data structure embedded in the assembly called metadata.

Fusion The .NET Framework uses Fusion to load assemblies into a managed process (AppDomain). Fusion facilitates such advanced features as strong naming and simplified search rules for DLLs.

Code access security Code access security (CAS) is a feature of the .NET Framework that facilitates the execution of partially trusted code. In a nutshell, you can use features of the Microsoft .NET Framework in order to limit what your late-bound code can access so that you don't have to worry about a plug-in undermining the user's system.

These four features of the .NET Framework make extensibility a reality. However, as cool as these features are, an article on each might not bring the extensibility story to light. Instead, it is best to approach the topic from a task-oriented point of view.

Getting Started with Extensibility

Regardless of what your application does, if it is extensible then it must perform three basic tasks: discovering, loading, and activating extensions. Discovery is the process of finding plug-ins or other code that your application will bind to at run time. Loading is the process of bringing the code (packaged as an assembly) into your process or AppDomain so that types defined by the assembly can be activated and consumed. Activation is the process of creating instances of late-bound objects and calling methods on them.

Surrounding each of these three phases are a number of .NET technologies and application design considerations. Although the technology is in place, the .NET Framework does not define one particular way to implement extensibility; there are quite a lot of options open to you.

Loading: Binding to Code at Run Time

Logically speaking, extensible applications discover code before loading it. But reflection must load code to discover things about it, so in practice the discovery process may require code to be loaded ahead of time. Let's see what I mean.

Reflection can be used to implement late binding. Most of the reflection classes can be found in the System.Reflection namespace. The three most important classes are System.AppDomain, System.Type, and System.Reflection.Assembly. I'll look at System.Type later on. To understand the AppDomain and Assembly classes I'll provide a brief tour of a managed process.

The CLR runs managed code in a Win32® process and the granularity of it is finer than what you would find in unmanaged applications. For example, a single managed process can contain multiple AppDomains which you can think of as a sort of sub-process or lightweight application container.

Assemblies are the managed version of DLL and EXEs and they contain reusable object types such as class library types as well as your application code. Additionally, any extensions or plug-ins to your application would also live in an assembly (most likely a DLL). Assemblies are loaded into one or more AppDomains in your managed process.

Each managed process has at least one default AppDomain, and also contains certain shared resources such as the managed heap, the managed thread pool, and the execution engine itself. In addition to these logical components, a managed process can create any number of additional AppDomains. See Figure 3 for a representation of a managed process containing two AppDomains. AppDomains will become critically important when I cover the topic of plug-in discovery later.

Figure 3 Managed Process
Figure 3 Managed Process

Now, back to the AppDomain and Assembly types. You can use the static method Assembly.Load to load an assembly into your current AppDomain. The Assembly.Load method returns a reference to the Assembly object. This method binds code to your app at run time by loading the assembly in which the code is housed.

Assembly.Load loads an assembly by name (without extension) from your AppBaseDir directory, the directory from which your AppDomain loads privately deployed assemblies. By default Assembly.Load looks in the directory from which your EXE was loaded for regular executables. Assembly.Load follows the same assembly binding rules that the CLR uses when loading assemblies that are early bound.

You can adjust the AppBaseDir of your AppDomain by obtaining a reference to an AppDomain object and calling the AppendPrivatePath and ClearPrivatePath instance methods on AppDomain. If you use Assembly.Load to load your plug-in assemblies, you are likely to want to manipulate the AppBaseDir of your AppDomain. This is because it is useful to maintain your plug-ins in a subdirectory of your main application directory. I will discuss more reasons for this shortly.

The Assembly class also implements a method called LoadFrom which, unlike Load, takes the full name of your assembly file including the path and extension. LoadFrom simply loads the file that you point it to, rather than following binding and discovery rules for finding the assembly. This can be useful for loading assemblies used as plug-ins. Be aware, however, that LoadFrom affects performance adversely compared with Load and the lack of version-binding intelligence is a negative for almost all scenarios other than plug-in scenarios.

Once you have obtained a reference to an Assembly object you can discover the types contained within the assembly, as well as create instances and call methods on these types. I'll look at these operations in the next two sections.

Discovery: Finding Code at Run Time

When compiling an application you explicitly or implicitly tell the compiler about code that your application binds to and uses at run time. This code comes in the form of widely reusable class library types such as Object, FileStream, and ArrayList, as well as application-specific types that are packaged in helper assemblies. The compiler stores references to the assemblies that implement these types in your assembly's manifest, and the CLR references the manifest at run time in order to load all the necessary assemblies. This is the typical binding process for managed code.

As I explained previously, late binding also works with code in the form of assemblies; however, the compiler is not directly involved in the binding process. Instead, the code must discover at run time which assemblies it wants to load. This discovery process can be implemented in a variety of ways. As a matter of fact, the .NET Framework does not specify a single discovery method, although it does give you significant tools for implementing your own technique.

Two discovery mechanisms are worthy of mention. The first and simplest (from the software point of view) is to maintain an XML file that documents the code that your application should bind to at run time. The second is a more advanced reflection-based approach that can make for a more useable application. Let's look at the XML approach first.

The XML approach only requires that your code parse the XML file and use the information there to begin loading assemblies. The following example shows a potential XML format:

<PluginAssembly name="MyPlugin.dll">
   <Type name="SomeType"/>
   <Type name="AnotherType"/>
</PluginAssembly>
<PluginAssembly name="MorePlugins.dll">
</PluginAssembly>
This XML includes the minimum information necessary to use reflection to implement the loading process; that is, the name of an assembly and the name of one or more types in the assembly that you want to activate. All your code must do is find the name of the assembly in the XML and then use code similar to that found in Figure 1 to load the assembly into your AppDomain.

The XML method is easy to implement, but does not make for a useable application. For example, I would not be particularly pleased with a text editor that forced me to edit a .config file of some sort to extend the application. However, for server-side applications this method may be more suitable than using reflection.

The reflection-based discovery method is automated by using a well-known location relative to your application directory. For example, the sample application that I produced for this article searches in a subdirectory named "Plugins" for potential extensibility assemblies (to download the complete source code, go to the link at the top of this article). This particular method is more useable because the user need only copy the assembly files into the file system, and the application will bind to the new code when it launches.

There are two things that make this method more difficult to implement than the XML approach. The first is that you must establish criteria under which assemblies should be loaded. Then you have to reflect over the assembly to discover whether it meets your criteria and should be loaded into your AppDomain.

There are three useful criteria for establishing a type in an assembly as a plug-in for your code. You should employ one of these when using the reflection method for discovery.

Interface criteria Your code can search through an assembly, using reflection, to find all of the types that implement a known interface type.

Base class criteria Your code can search through an assembly, again using reflection, to find all of the types that derive from a known base class.

Custom attribute criteria Finally, custom attributes can be used to mark a type as a plug-in for your application.

In a minute I will show you how to use reflection to discover if a type in a late-bound assembly implements an interface, derives from a base class, or is attributed. But first, let's look at the design trade-offs of using these binding criteria.

The interface and base class criteria are more useful than the custom attribute approach (or any other approach to reflection-based late binding) for two reasons. First, plug-in developers are more likely to be familiar with interfaces or base classes than they are with custom attributes. More importantly, interfaces and base classes can be used to call into the late-bound code in an early-bound manner. For example, when you use an interface as criteria for binding to an object type, you can use an instance of the object through a reference of the interface type. This avoids the need to use reflection to call methods and access other members of the instance, which improves performance and codeability of your extensible application.

To discover information about a type at run time you can use the reflection type System.Type. Each instance of a type derived from Type refers to a type in the system. The Type instance can be used to discover facts about the type at run time such as its base class or the interfaces that it implements.

To obtain a set of types implemented by an assembly, you simply call the GetTypes instance method on an Assembly object. The GetTypes method returns an array of references to Type-derived objects, one for each type defined in the assembly.

To establish whether a type implements a known interface or is derived from a certain base class, use the IsAssignableFrom method of the Type object. The code snippet that follows shows how to test whether or not the type represented by the someType variable implements the IPlugin interface:

Boolean ImplementsPluginInterface(Type someType){
   return typeof(IPlugin).IsAssignableFrom(someType);
}
This principle also applies to testing for base types.

Loading an assembly to reflect on the types in the assembly works fine for discovering which types you want to plug into your application logic. But what happens if none of them match your criteria? The short answer is that there is no way to unload an assembly that's already in your AppDomain. This is the second reason that a reflection-based discovery method is slightly more difficult to implement than the XML approach.

For certain client applications it may be acceptable to litter your process with all of the assemblies that your code reflects over, even the ones that don't meet your plug-in binding criteria. But for an extensible server application its insufficient to load anything that can't subsequently be unloaded. The solution for this comes in two phases. First, you can unload an assembly so long as you unload the entire AppDomain in which the assembly was loaded. Second, you can programmatically create a temporary AppDomain for the sole purpose of loading assemblies to reflect over to find out whether you want to use the types in the assembly as plug-ins. Once you are past the discovery stage, the temporary AppDomain along with all of its assemblies can be unloaded.

This may sound like a complex solution, but it is in fact a fairly simple piece of code to implement. The PluginManager class published with this article takes this approach, and it takes less than 100 lines of code to implement. If you want to take this approach in your own extensible application, you will find the source for PluginManager useful (see the download).

Activation: Create Instances and Call Methods

Once you have established which types you want to plug into your application, you need to create instances of the objects. This can be done entirely using reflection; however, you will find reflection to be slow, impractical, and difficult to maintain. Alternatively you should use reflection to create instances of plug-in objects, cast the objects to a known interface or base class, and then use the objects in terms of the known type throughout the remainder of its lifetime. Remember that this approach fits hand-in-glove with the interface and base class criteria for discovering plug-in types.

Thanks to the CTS, managed code is object oriented and type safe. The challenge with late-bound code is that an object's type isn't necessarily known at compile time. But thanks to the CTS, you can treat any object as the base class Object and then cast to some base class or interface that's known to your application and the pluggable code. If the object is not compatible with the cast, an InvalidCastException is thrown, which your application can catch and handle accordingly.

However, before you can do any of this, you must create an instance of the object that you are binding to. Unlike early-bound objects, you can't easily use the new keyword to create an instance because the compiler expects the name of a type to be used with new, which of course you don't know for late-bound object types. The solution is the static method Activator.CreateInstance. This method will create an instance of an object given a reference to the object's Type-derived instance, and an optional array of Object references to be used as constructor parameters. The following code uses CreateInstance to create an object and return a well-known interface:

IPlugIn CreatePlugInObject(Type type){
   return (IPlugIn) Activator.CreateInstance(type);
}

Once you have an object and you have cast it to a known type, then you can call methods on the object like you would any other object through the reference variable. At this point, the objects in your late-bound code are seamlessly integrated with the rest of your application.

Secure Extensibility

If your application will be distributed widely, and anyone might be writing plug-ins, you'll have to think about security. Fortunately, the .NET Framework makes it easy to secure your code. Let's see how.

Imagine again the extensible text editor. In the ideal world, a user should be able to download a plug-in from the Internet and plug it into this application safely, even if it was designed by someone who is not a trusted third party. And when it isn't trusted, it should execute in a partially trusted state where it does not have access to system resources such as the file system or registry.

The .NET Framework can execute partially trusted code through CAS. Because managed code is just-in-time compiled, the CLR has the ability to assert that partially trusted managed code not perform an action for which it lacks permission. If partially trusted code attempts a disallowed action, the CLR throws a SecurityException exception, which the application can catch.

In certain cases code is partially trusted by default, such as controls that are distributed over the Internet and embedded in HTML documents. However, you can also take advantage of CAS so that users can safely use extension assemblies from third parties. In all cases, the CLR considers an assembly to be a unit of security, which means that your application can consist of more than one assembly, each of which might have slightly differing security permissions granted to it. This is perfect for plug-ins.

Again, the .NET Framework makes available many features, and there are a number of approaches that you can take in order to create partially trusted plug-ins. The following steps break down the simplest and most secure approach. First, create two subdirectories for plug-ins to your application. One is for fully trusted plug-ins and the other is home to partially trusted plug-ins. Then programmatically adjust the local security policy to associate a code group with your subdirectory for partially trusted plug-ins. Finally, grant code in the code group Internet permissions, which is to say that it has a subset of permissions deemed safe even for potentially malicious code.

Once you have made this code group, the CLR automatically associates the reduced permissions with any assembly loaded from your partially trusted subdirectory. Aside from having to adjust local security policy (which I will show you how to do shortly), the security for plug-ins works automatically.

Adjusting Security Policy

The .NET Framework security policy engine is very flexible and adjustable. In practice, adjustments to security policy are made manually using the .NET Framework Configuration control panel applet installed with the Framework (see Figure 4). You can also adjust policy programmatically. To write code that modifies security policy, you must start with the SecurityManager type. This helpful type gives you access to the three policy levels installed with the .NET Framework: Enterprise, Machine, and User. I suggest adding your custom code group for plug-ins to the Machine policy level. To find the machine PolicyLevel object use code like that shown in Figure 5.

Figure 5 Finding Machine Policy Level (Close Figure 5)
// Get machine policy
static PolicyLevel FindMachinePolicy(){
   PolicyLevel ret = null;
   // Get the policy hierarchy
   IEnumerator en = SecurityManager.PolicyHierarchy();
   en.Reset();
   // Find the policy named "Machine"
   while(en.MoveNext()){
      PolicyLevel level = (PolicyLevel) en.Current;
      if(level.Label == "Machine"){
         ret = level;
         break;
      }
   }
   return ret;
}
Figure 4 Security Configuration
Figure 4 Security Configuration

Code groups are arranged in a logical hierarchy. Once you have obtained a PolicyLevel object, you can traverse the code group hierarchy starting with the code group returned by the PolicyLevel.RootCodeGroup property. The root code group, by default, goes by the name ALL_CODE, which represents all managed code. You should create your custom code group as a child of the ALL_CODE code group.

The code in Figure 6 creates a custom code group for any code loaded from a given URL. The code group has Internet permissions, and also has the PolicyStatementAttribute.Exclusive and PolicyStatementAttribute.LevelFinal bits set to indicate that code that matches this code group only gets these permissions. The URL can be an HTTP, HTTPS, or FILE URL. In order to associate the new code group with a directory on the file system, you can use a file URL that is structured like this: file://d:/programs/extensible-app/partially-trusted.

Figure 6 Creating and Saving Custom Code Group (Close Figure 6)
static void MakePluginCodeGroup(
   PolicyLevel level, CodeGroup root, String url){

   // Get the internet permissiion set
   PermissionSet permissions =
      level.GetNamedPermissionSet("Internet");

   // Create a membership condition for our path
   IMembershipCondition membership =
      new UrlMembershipCondition(url);

   // Create a policy statement from permissions and condition
   PolicyStatement statement = new
      PolicyStatement(permissions,
      PolicyStatementAttribute.Exclusive |
      PolicyStatementAttribute.LevelFinal);
   // New code group
   UnionCodeGroup group =
      new UnionCodeGroup(membership, statement);
   group.Description= "Custom code group for plug-ins";
   group.Name = "Custom group";
   root.AddChild(group);
   SecurityManager.SavePolicyLevel(level);
}

The CAS features of the .NET Framework are pretty flexible but can take a while to get used to. You should be able to get the information you need to create a plug-in architecture from this article, along with the sample application code that I have included in the download. I do, however, strongly suggest using the .NET Framework Configuration control panel applet to try making changes to your systems policy so that you'll become familiar with the concepts. You can always restore the policy defaults if you go too far.

Extensible Application Design

The ExtensibleApp.exe sample application that comes with this article is an example that supports custom plug-ins. In fact, the application is really just a shell that does nothing but display an MDI window and allow the user to install plug-ins. If you are using the code in this sample as a learning tool for writing your own extensible applications, you should pay special attention to the code in PluginManager.cs. This module contains the PluginManager reusable class that handles all of the non-application-specific plug-in logic for the sample.

Figure 7 Before Plug-in
Figure 7 Before Plug-in

If you build and run the ExtensibleApp.exe sample, you will see that it lets you select DLLs to install into your application as plug-ins. The sample includes two plug-in projects, PluginProject1.dll and PluginProject2.dll. They take advantage of the API exposed by the application itself to create toolbars and menus, as well as to add a document type to the application. Figure 7 shows the application before plugging in the custom code, and Figure 8 shows the application afterwards.

Figure 8 Running a Plug-in
Figure 8 Running a Plug-in

The application uses the technologies and techniques discussed so far in this article. Additionally, it shows some approaches to design that you should consider when making your application extensible. Let's take a look at some of these considerations.

Versioning

Versioning is an important aspect of the application lifecycle. It can also have a significant effect on extensible applications. Your extensible application needs to define interfaces, base types, or attribute types that you use for finding and using plug-ins. If you include these types in your regular application EXE or DLL assemblies, then they will be versioned with the rest of your application, which can cause binding hassles for the plug-in assemblies that aren't versioning on the same schedule. But there is a solution.

The solution is that any type you use to discover or bind to plug-ins should be defined in its own assembly where nothing else lives but other plug-in-oriented types. You should also avoid putting any code (or at least not much) in the assembly because you need to version the assembly as infrequently as possible. Meanwhile, the rest of your application can be changed and versioned as needed. Both your application and plug-ins will share the glue assembly, which will only be versioned rarely.

This brings up a question about base types used with plug-ins. Unlike interfaces, base classes commonly contain some code. This is a tough problem to solve, and it is one of the primary reasons that interfaces are the preferred means of calling through to late-bound objects.

However, you should be aware that if you need to version the code in your base class and interface assembly, the .NET Framework does give you the flexibility that you need to redirect binding from the old assembly version to the new version. But this requires a change to binding policy which must be entered either in the app.config file for the application or in the Global Assembly Cache (GAC) for the entire system. Avoiding this eventuality is preferable, so manage your plug-in interfaces and base classes so that they are versioned rarely, if ever.

Robustness

Remember when writing your extensible application that although your code has undergone rigorous quality control, the plug-in code most likely has not. For this reason, any code that calls through an interface or base class reference into late-bound objects should expect the unexpected. Partially trusted plug-ins are likely to throw security exceptions even when the developer wasn't purposefully trying to cause a breach. Similarly, both partially trusted and fully trusted plug-ins are likely to have bugs because the author of the plug-in doesn't have the same inside understanding of your application that you do.

If you are designing your extensible application to support plug-ins written only by you or your team, this may not be a concern. But many extensible applications will want to recover as gracefully as possible when a plug-in object is misbehaving.

One great thing about managed code is that when an object does, in fact, fail, it fails in a well-defined manner—namely, the object throws an exception. So if you are consistent about registering for unhandled exception handlers, and handling expected exceptions around calls into plug-in code, then you can give the user a consistent experience even when a plug-in fails.

One advanced technique that you can use, if you are using interfaces to call through to your plug-in objects, is to wrap all late-bound objects in a proxy object that implements the same interface. This general-purpose proxy object would pass all calls through to the underlying plug-in, but could also wrap calls with exception handlers that are consistent about logging failure, alerting the user of the misbehaving plug-in and other such things. For ultimate plug-in robustness this is a good idea, but for many applications this degree of stability may not be necessary.

Security Considerations

The .NET Framework takes care of security for your partially trusted plug-ins so long as you set policy to restrict the permissions for the assemblies that you load. However, this works great for protecting the system, but does not automatically protect the state of your application. Partially trusted objects share a managed heap with the objects in your application, and consideration needs to be made to limit their access to your internal application objects. Following are some pointers.

Don't unnecessarily designate object types in your application as public objects. If your type is internal (the default case), then they are off limits to partially trusted code in your application. However, public types can sneak into your application without you noticing. For example, the Visual Studio wizard generates public classes when you add a Form-derived type to your project. This isn't necessary for most applications and you should delete the public keyword from these types until you find a need to add it.

Similarly, you should not unnecessarily make members of public types public or protected. Even for nonextensible applications, your members should be private until you find a need to increase their accessibility. Then, if you can get the job done with internal accessibility, do so. Only use protected and public members when you intend to expose the member externally to your assembly.

You should watch for class library types with bad or incomplete security policies. For example, the System.Windows.Forms.MainMenu class exposes a method called GetForm, which returns the form in which the menu lives. This is usually the main form of your application. Even if you did not intend to pass a reference to the main form of your application to a partially trusted plug-in, you might inadvertently allow access by giving plug-ins direct access to Menu-derived objects in the application. The CLR class library developers considered similar security problems. For example, the Form.Parent property demands a security permission of its caller before returning a reference to the parent form. Partially trusted code running with Internet permissions, for example, doesn't have access to this property by default.

As you can see, partially trusted plug-ins may not have access to the general file system or the registry, but you still have to be watchful to keep a malicious plug-in from doing something like closing your application. For client applications this sort of problem is not often critical. For server applications it can be.

In the final section, I'll briefly discuss some advanced possibilities for extensible applications. These topics are too extensive to cover in detail and are not likely to be needed by the majority of applications, but it's helpful to be aware of the possibilities.

Unloadable Plug-in Assemblies

In the section on discovering plug-ins earlier in this article, you might remember the problem of unloading unwanted assemblies. The solution to the problem was to test assemblies for usefullness in a temporary AppDomain so that they can be unloaded if need be. Then, after discovering the assemblies you do want to use, you can load them into your main AppDomain. But what if you don't want all your plug-ins living indefinitely?

For client applications it can be acceptable to load plug-in assemblies, use their types until they are no longer useful, and then ignore the extra assembly for the remainder of the application's lifetime. However, on the server side the requirements are stricter. Server-side applications must be able to run indefinitely without running out of vital resources such as process address space, so you'll need to both load and unload assemblies containing plug-in types.

To do this, you have to discover the assemblies in a temporary application domain as well as use the plugged-in type exclusively from another AppDomain. This adds a level of complexity to your application design, but also nicely separates the plug-ins from your core application logic.

It is possible to instantiate and use objects entirely from within a separate AppDomain, and the feature of the .NET Framework that implements this is called Remoting. Remoting is used to access objects across processes, as well as across the network, but is also used to access objects in different AppDomains even if they are in the same Windows® process. I can't cover Remoting completely here, but you will find extensive coverage in past issues of MSDN® Magazine, and you will also find some simple Remoting code in my sample PluginManager type.

Managed Plug-ins in Unmanaged Apps

With all of these great extensibility features at your disposal, you might feel left out in the cold if yours is a legacy application written in an unmanaged language like Visual Basic® 6.0 or C++. Well, feel left out no more. The CLR itself is a COM object and can be hosted in any Win32 process by any language capable of being a COM client.

This means that you can write your plug-in managed code—in C# or Visual Basic .NET—and then from unmanaged code you can load the runtime and your glue code which, in turn, loads plug-ins that can interact with your unmanaged application in any manner you like. Meanwhile, COM Interop allows you to seamlessly pass COM interfaces back and forth between managed and unmanaged code.

In fact, there are three ways to host and interact with managed code in your unmanaged assembly. You can use COM Interop. The CLR allows you to create COM servers with C#, Visual Basic .NET, and other managed languages. You bind to and use these managed objects from any unmanaged application. You can also use C++ with Managed Extensions (MC++), which is capable of natively mixing managed and unmanaged code in a single process. If your application is written in C++, you might look at Managed C++ for including rich extensibility features. In addition, you can host the CLR directly. The CLR itself is a COM object that you can host. The .NET Framework SDK ships with a C++ header file called MSCorEE.h which includes the necessary definitions to use the runtime as a COM object.

Again, once your unmanaged application is bound to your managed code, your managed code can use the techniques covered in this article to implement extensibility for your application.

Wrapping It Up

The .NET Framework provides some very flexible features for code reflection, late binding, and code security. These features can be mixed and matched in a number of ways to implement extensible applications. When a reliable application is also extensible it's likely to have a longer shelf-life, and may gain something of a following, which can lead to the creation of more plug-ins and wider acceptance. So take these extensibility features for a spin around the block. I think you'll like the results.

For related articles see:
.NET Framework: Building, Packaging, Deploying, and Administering Applications and Types
.NET Framework: Building, Packaging, Deploying, and Administering Applications and Types—Part 2

For background information see:
.NET Framework Security by Brian A. LaMacchia, Sebastian Lange, Matthew Lyons, Rudi Martin, and Kevin T. Price (Addison-Wesley, 2002)

Jason Clark provides training and consulting for Microsoft and Wintellect (http://www.wintellect.com) and is a former developer on the Windows NT and Windows 2000 Server team. He is the coauthor of Programming Server-side Applications for Microsoft Windows 2000 (Microsoft Press, 2000). You can get in touch with Jason at JClark@Wintellect.com.


Editor's Note: Best Practices

Editor's Note
Best Practices

Whew, what a month. As we worked on this super-fabulous issue, we've simultaneously been packing everything in the office for the move to our fancy new digs a few blocks away in the heart of Midtown Manhattan. For the most part, this has gone smoothly—although we just found out we're losing our classy 212 phone numbers for one of those lame new area codes, 646. 212 is the country's best area code. Its very mention invokes images of Swingin' New York at its finest in the mid-1970s. The air was redolent of Disco Fever, blackouts, riots, fiscal insolvency, and finally, Ed Koch. 212 says, "I've been through it all, and I'm still here." When you get a call on the second line, flashing a 212 on your caller ID, you say "I'm gonna put you on hold, Mom, this could be opportunity calling!" But when a 646 calls? "I don't recognize the number—probably a collection agency looking for my Saturn loan payment. I ain't answering that!"

So as you can see, we made the mistake of packing away our Editor's Note idea book, and we can't figure out which of the 4,215 moving boxes it's tucked away in. It's probably under a stack of Ashton-Tate demo disks we're sure will be a collector's item someday. One thing we do have on our minds (which have also been packed for shipping), is the not-entirely-new, but still vital, topic of best practices in programming.

When we solicited feedback at Tech•Ed this year, many readers told us that they wanted to see more discussion of best practices in the magazine. Best practices can be a bit of a tricky concept at times. There's not always one "right" way to get a task done, especially in computer programming, a creative art that masquerades as a science. Of course, design patterns are one way that the developer community attempts to reach consensus on the right way to do discrete tasks. But the right way to do something is necessarily driven by circumstance.

For example, is there always one right way to sort a list? Sometimes you just want a simple alphabetical sort of a set of strings. But other times, you might want to sort objects by certain properties, or even do advanced percentage hits on text lookups. You're not going to be doing a simple bubble sort when you build a relevancy-based search engine.

When we work on an article, we have it reviewed by technical experts to make sure that we're providing the right best practices advice to readers. Of course, sometimes what's considered "right" can change—when a new product or API is released, the best way to search through an XML file might also change, for example.

In the early days, Microsoft would often publish a bunch of code samples to demonstrate the use of a particular API. Since these samples were designed to highlight the way to call a specific API, rather than taking the complete sample's workings into account, they didn't always reflect the best way of getting a job done. So you might have a call that showed how to use a particular SQL call, even though there might be a better method (like a stored procedure) to accomplish the same result. There were times when multiple samples could be internally contradictory in their presentation.

Realizing the potential for error this could cause, Microsoft has dedicated a lot of effort to developing proven architectures and best practices, and the fruits of this labor are evident on the Patterns and Practices Web site (see http://www.microsoft.com/resources/practices). One interesting new release is "Improving Web Application Security: Threats and Countermeasures". Using threat modeling, which evaluates appropriate countermeasures to security threats, the article helps you to create threat-resilient ASP.NET apps through a series of best practice recommendations.

We take best practices and security seriously here. If you see something in our magazine that you don't think is a best practice, we want you to let us know. Drop us a line (mmeditor@microsoft.com) and tell us how you'd do it better. Selected letters might be discussed in future issues. But not until we've unpacked all these boxes and the area code thing sinks in, at least.

—J.T.

IntelliSense, JScript, Microsoft, MSDN, Outlook, PivotTable, PowerPoint, Visual Basic, Visual C++, Visual J++, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Generating Help
Animate Your Web Site
Interactive Charting and Graphics
Recover Corrupted Exchange Server Databases
Driver Development Studio
Integrate E-mail into .NET Framework-based Apps
Expert System Development
Localizing Web Sites
Data Tier Code Generation Tool
SQL Scripts for Multiple Servers
Present Tabular Data
Customizable Syntax Editor
Automating Patch Management
The Bookshelf

Generating Help

Excel Software has announced the availability of QuickHelp for Windows, a development tool for creating and deploying application help to Windows® 95 through Windows XP, Mac OS 9, Mac OS X, and virtually all Linux distributions. In QuickHelp you can manage, author, and test help topics in a single tabbed window running on your platform of choice, then deploy the help system. Help information resides in an XML file distributed with a native viewer executable for each platform.

QuickHelp Builder lets you author help systems and QuickHelp Viewer lets you deploy them to users. For the user, QuickHelp supports a table of contents, an index with automatic search field, word searches across topics, color-highlighted topic text with hypertext links, formatted images, and context-sensitive help from application menus and dialogs.

From the Contents panel, the user can view, expand, or collapse topics. The Index panel allows you to locate topics.

For the developer, QuickHelp provides an authoring environment that combines the Contents and Index panels with the Edit and General panels for editing the help file. The Edit panel is used to add, edit, delete, or move topics within the Contents tree. Formatted text, images, and links between topics can be added.

Each topic has fields for defining index words and context-sensitive identifier strings that link topics to application menus and dialogs. Topics or portions of a topic can be conditionally visible based on the current platform or other variables, allowing you to customize your help system for different platforms or product builds from a single source. The General panel has buttons to save, open, close, verify, import, or export help topics.

QuickHelp handles administrative activities like maintaining links between topics, reordering topics and locating bad links, missing indexes, or images. QuickHelp for Windows includes Builder and Viewer executables for Windows 95 through Windows XP, a license to distribute the QuickHelp Viewer, a printed manual, online help, and a PDF manual on CD.

http://www.excelsoftware.com

Animate Your Web Site

Lucky Monkey Designs has released KoolMoves version 4.0, a Windows-based animation and multimedia application that lets you combine and manipulate text, vector clip art, and sounds to create Flash-animated Web pages, banners, navigation systems, and frames for GIF animations.

KoolMoves allows you to place text, images, shapes, and sounds onto movie frames which then can be played in KoolMove's internal player or in your browser. A time/event chart enables the synchronization of sounds and actions associated with frames and buttons. KoolMoves automatically morphs drawings into animations that are compact and fast loading.

You can select from a set of libraries containing Web interface templates, text effects templates, action script-based text effects, buttons, animated, and non-animated vector clip art, all designed for Flash movies.

Among KoolMoves' features are Flash MX action scripting, HTML dynamic text, masking, import of Flash movies as objects, bones for character animation, nonlinear morphing, transition effects, drop shadows, and preloaders.

http://www.koolmoves.com

Interactive Charting and Graphics

B-Line Express now offers Charting Components 2.0, a Flash component set. Charting Components 2.0 enables developers to add interactive charting and graphic solutions to their Macromedia Flash front end. The components can consume static, dynamic, and real-time data from any application server including ColdFusion, ASP.NET, Java, PHP, ASP, or Flash Communication Server. The components can also work independently with local data from XML files or Flash ActionScriptArrays.

Designers and developers using Charting Components 2.0 will be able to create charts in 10 styles. A programming interface provides access to charting methods and style properties that allow you to customize charts. The component library can be extended programmatically to include additional styles, methods, and skinning effects that will give each set of charts its own UI.

http://www.blinex.com

Recover Corrupted Exchange Server Databases

Recoveronix recently published ExchangeServerRecovery 2.1, which is a recovery product for corrupted Microsoft® Exchange Server database files. The updated version of this application supports files over 30GB and can recover folders, messages, attachments, notes, contacts, tasks, appointments, and creation dates for all objects.

The program also recovers formatting for RTF and HTML messages. ExchangeServerRecovery supports Exchange Server 5.5 and Exchange Server 2000, and saves the recovered data into Personal Storage File (.pst) format to be viewed with any e-mail client compatible with Microsoft Outlook®. The program requires Data Access Objects and runs under Windows 95, Windows 98, Windows Me, Windows 2000, or Windows XP.

http://www.officerecovery.com

Driver Development Studio

Compuware Corporation has announced the availability of DriverStudio 3.0, a suite of tools for developers of Windows device drivers. DriverStudio 3.0 can integrate with the Microsoft Visual Studio® .NET IDE as well as with Visual Studio 6.0.

The DriverStudio suite of tools covers the entire driver development lifecycle, helping device driver developers write, debug, test, and tune driver code that meets quality standards for Windows Hardware Quality Labs driver certification. DriverStudio components include DriverWorks, DriverNetworks, BoundsChecker Driver Edition, TrueTime Driver Edition, and TrueCoverage Driver Edition.

DriverWorkbench has been enhanced in DriverStudio 3.0 to provide integration of Visual SoftICE with the other DriverStudio tools including BoundsChecker Driver Edition, TrueTime Driver Edition, and TrueCoverage Driver Edition. Additionally, the DriverStudio Framework tools, DriverWorks and DriverNetworks, plug into the Visual Studio 6.0 IDE. DriverWorkbench allows you to move from one tool to another without leaving the host environment, thus easing the process of testing, debugging, and optimizing device drivers.

http://www.compuware.com

Integrate E-mail into .NET Framework-based Apps

Quiksoft Corporation recently announced EasyMail .Net Edition 2.0, an update to its .NET e-mail component suite that enables application developers to integrate e-mail into their .NET Framework-based applications. Version 2.0 adds support for Quiksoft's new secure sockets layer (SSL) plug-in, which enables the e-mail component suite to communicate with SSL-enabled e-mail servers.

Other highlights included in the update are integration with Visual Studio .NET 2003, support for bulk e-mailing, address verification, importing from URLs, additional authentication protocols, and more.

EasyMail .Net Edition is written in C# and includes SMTP, POP3, IMAP4, and parsing assemblies. The program includes support for all e-mail features including attachments, SSL, MIME, Base64, Uuencode, Quoted-Printable, HTML, bulk e-mail, and more.

http://www.quiksoftcorp.com

Expert System Development

Logic Programming Associates Ltd. recently announced an update to WebFlex, its Web-based expert system development tool. WebFlex allows artificial intelligence developers to deliver commercial rule-based systems, expert systems, and knowledge-based systems on the Internet. WebFlex offers an integrated solution which runs as a CGI application on most commercial Web servers using industry standards.

Basic features include a high-level English-like Knowledge Specification Language, a frame-based hierarchy with multiple inheritance for modeling data, dynamic access to ODBC data sources using the ProData interface, and support for uncertainty using fuzzy and Bayesian rules.

Webflex also supports interleaved forward and backward chaining rules, an automated question and answer system, and access to Prolog for list processing, recursion, scientific math functions, and other add-ons.

In addition, WebFlex includes the ability to define the appearance of runtime questions using Flex KSL, the automatic generation of HTML and integration with existing HTML files, automatic generation of JavaScript for validation of user data input, and support for multiple, simultaneous users.

http://www.lpa.co.uk/wfs.htm

Localizing Web Sites

TMG Development Ltd has announced their ASP.NET Localizer toolkit, which provides design-time and run-time support for producing a professional, internationalized Web site using ASP.NET.

With ASP.NET Localizer, all you do is drop a Localizer component onto a Web Form (or UserControl) in Visual Studio and design the page in the invariant language. Then set the Language property of Localizer to a new language and type the localized strings, image source locations, and so on into the Visual Studio designer. Localizer generates a new .resx file for that language, and all the property changes are written to this new file.

You can switch between languages using Localizer, and the design surface is kept up to date with the current language selection. When the Web application is run, Localizer checks the locale and generates the Web Form using resources for that locale.

Localizer 1.0 requires Visual Studio .NET 2002 or Visual Studio .NET 2003, and the.NET Framework 1.0, or above.

http://www.winformreports.co.uk

Data Tier Code Generation Tool

2LKit has announced the release of RapTier 1.1, a data-tier code generation tool created for multitier application development. The generated code supports ADO.NET transactions and stored procedures for all database operations. The default templates generate C# or Visual Basic® .NET classes that use a data source-independent mechanism to access a database. This feature allows switching among different .NET data providers without changing the code. In addition to new features and improved existing templates, RapTier is now available in two editions: Lite and Professional.

http://www.2LKit.com

SQL Scripts for Multiple Servers

Red Earth Technologies has published Equivalent Script version 1.1, a SQL tool that develops equivalent SQL scripts for multiple database servers. Equivalent Script allows users to develop entire SQL scripts once, and then generate the equivalent SQL dialect for multiple database servers, thus eliminating the need for manual conversion.

The program's interface can be used to create and generate scripts, or they can programmatically access the document object from within their own applications. Equivalent Script generates SQL scripts for Microsoft SQL Server™, Microsoft Access, MySQL, and Oracle. Equivalent Script can be extended to cater to additional database servers.

http://www.equivalentscript.com

Present Tabular Data

FarPoint Technologies Inc. recently announced the availability of Spread for Windows Forms for development on the .NET Framework. Spread for Windows Forms allows you to present, edit, and update your tabular data. It provides validation by using its 14 cell types to alert users of invalid data immediately.

The Spread Designer allows developers or designers to design the look of the component at design time with no coding required. Users can sort data and import and export their information in several formats, including Microsoft Excel.

Spread for Windows Forms supports in-cell editing, multiple sheets, splitter bars, printing, zooming in and out of the data, import and export to Excel, cell notes, cell spans, multi-line headers, sorting, complete customization at the cell level, 200 built-in mathematical functions, and hierarchy view.

http://www.fpoint.com

Customizable Syntax Editor

Actipro Software has released its SyntaxEditor, modeled after the code editor in Visual Studio .NET, an editing control similar to a multi-line TextBox. SyntaxEditor syntax highlights text and provides features for code editing. In addition, custom language definitions can be developed and distributed with your applications.

SyntaxEditor is designed for use in IDE applications, but may be used in other scenarios. For instance, you could make the editor read-only and use it for displaying log files or status messages (like in an FTP client) for an application.

SyntaxEditor's major features include multi-language support, a custom regular expression engine, XML language definitions, the ability to save to binary for instant language domain reloading, unlimited undo/redo, and undo/redo action grouping.

You'll also find line numbering in the line number margin, word wrap (character and token wrapping modes), indicators (bookmarks, breakpoints, misspelling markers, and so on) in the indicator margin, indentation guides, and content dividers. Any type of font is supported, including non-monospace fonts. The program also contains triggers for handling keystroke/action combinations.

SytaxEditor allows for customizable dialogs and four-way or two-way split views. You can export to HTML or RTF. The program is packaged with sample projects and documentation on how to use the control as well as complete class library documentation.

http://www.actiprosoftware.com/Products/DotNet

Automating Patch Management

Spire Security LLC and Shavlik Technologies have developed a research report to help companies understand the security benefits, return on investment (ROI) potential, and cost- and time-saving advantages of replacing the manual patch management process with an automated system. The report, "Shavlik Technologies: Proactive Patch Management," examines the patch management process, the qualitative and quantitative benefits of automated patch management, and describes a typical ROI scenario using Shavlik's HFNetChkPro automatic patch management solution.

http://www.shavlik.com

The Bookshelf

Cisco Press has announced a series of self-study security titles available for CCIE Security certification preparation. The recently published Network Security Principles and Practices provides candidates with topical coverage of CCIE security exam issues. The titles CCIE Security Exam Certification Guide and CCIE Practical Studies: Security provide written exam and lab exam preparation tools, as well as topical coverage of CCIE security exam issues.

Network Security Principles and Practices takes an applied approach to building security into networks. This book is a comprehensive guide to network security threats and the policies and tools developed specifically to combat them.

CCIE Security Exam Certification Guide is the official preparation guide for the CCIE Security written exam and includes pre- and post-chapter exams, exam objective reviews, and a CD-ROM-based testing engine that allows for focused or comprehensive tests.

http://www.ciscopress.com/ccie

VB.NET Database Programming with ADO.NET by Anne Prince and Doug Lowe has been published by Mike Murach & Associates. The book is designed to lead you from the simple to the complex as it covers the range of ADO.NET capabilities. You'll learn about typed and untyped datasets, bound and unbound controls, data views, parameterized queries, relationships, data commands, schema definitions, database classes, data controls for Web applications, and Crystal Reports. The authors show 21 applications, culminating in an order-entry application that uses database classes and accesses nine different tables.

http://www.murach.com

Que Publishing has released two more books in its Microsoft Certified Application Developer series: Developing and Implementing Web Applications with Microsoft Visual Basic .NET and Microsoft Visual Studio .NET, and Developing XML Web Services and Server Components with Visual Basic .NET and the .NET Framework, both by Kirk Hausman. Both books feature test-taking strategies, study tips, and a special Cram Sheet that includes tips, acronyms, and memory joggers. They also include PrepLogic practice exams.

http://www.quepublishing.com

Sams Publishing offers the latest book from Andre LaMothe, Tricks of the 3D Game Programming Gurus: Advanced 3D Graphics and Rasterization. This book, a follow-up to Tricks of the Windows Game Programming Gurus, is intended to take the reader to the next game programming level. When readers are finished with this book, they should be able to create a full 3D, texture-mapped, lit video game for the PC with a software rasterizer they can write themselves. They will also understand the underlying principles of 3D graphics and better understand and utilize 3D hardware. This book discusses advanced topics including 1/z buffering, mip-mapping, alpha blending, texture filtering, and spatial partitioning. It is intended for an intermediate to advanced audience.

http://www.samspublishing.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: Schema From a DataSet, Exporting SQL Data to Excel, and More

Web Q&A
Schema From a DataSet, Exporting SQL Data to Excel, and More
Edited by Nancy Michell

Q How can I get a schema produced from a DataSet that is returned from a stored procedure? I know that this is possible in Visual Studio® .NET through the Generate DataSet dialog box, but I'd like to do this without using Visual Studio. How does the Visual Studio DataSet Generator do what it does? Is this functionality available from Visual Studio as a command-line tool or could I write one in C#?

A The DataReader includes a GetSchemaTable method that returns a DataTable of schema information about the resultset. Each row in the schema DataTable corresponds to a column in the resultset. You can use this schema DataTable to determine column names, data types, and so on for the resultset. If you execute a query with CommandBehavior.KeyInfo, you'll receive additional schema information such as the base table name and column name. That's probably the best way to programmatically discover this schema information (see Figure 1).

Figure 1 Get Schema (Close Figure 1)
string strConn = "Data Source=(local);Initial Catalog=Northwind;
Trusted_Connection=Yes;";
SqlConnection cn = new SqlConnection(strConn);
cn.Open();

SqlCommand cmd = new SqlCommand("CustOrdersOrders", cn);
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("@CustomerID", "ALFKI");

SqlDataReader rdr = cmd.ExecuteReader(CommandBehavior.KeyInfo);
DataTable tbl = rdr.GetSchemaTable();
rdr.Close();
cn.Close();

foreach (DataRow row in tbl.Rows)
       Console.WriteLine("Table: {0}\tColumn: {1}",
       row["BaseTableName"], row["BaseColumnName"]);

Just remember these few caveats: this won't work well if your stored procedure returns different resultsets depending on input parameters. It also won't work if your stored procedure creates and uses temporary tables. If you execute the query with CommandBehavior.KeyInfo, there's no guarantee that you'll receive additional schema information such as the base table name and column name.

Q I have some data in a SQL Server™ 2000 database and I would like to write a Windows® Forms app to allow users to export and save the data to Microsoft® Excel. How can I do this using C#? I have tried to bind SQL Server tables to an ADO.NET dataset and use WriteXML to save data into an XML file. Then I thought I could open the XML file with Excel. It seems like a simple workaround. Is there any problem doing it this way? Also, how can I export SQL Server table data to an Access database (an MDB file) using C# and/or Visual Basic® .NET?

A To export a SQL Server table into an Access database you can create a Data Transformation Services (DTS) package in SQL Server and execute it from C# or Visual Basic .NET.

See Knowledge Base article 306023, "Transfer Data to an Excel Workbook by Using Visual C# .NET," which provides sample code using ADO. You can also use XSL to transform the data into XML that fits the schema of an Excel workbook. That way, you don't have to use the Excel object model (especially if you want to do this on the server and don't have Microsoft Office installed there).

By creating an XSLT transform to convert exported XML to Excel XML Spreadsheet (XMLSS) format, you make it more Excel-friendly and allow for pre-formatting the results. Just save a simple spreadsheet as XML to get an idea of the basic schema.

However, DTS is the normal way to export data from SQL Server. Once you've created a DTS package, you can execute it programmatically with about two lines of code: one line to load the package into memory, and one line to run it. This allows you to output SQL data in native Excel and Access binary formats. Other approaches are going to be limiting or tedious to implement.

Q I have a question related to performance of XPath-based queries. Are they expensive in terms of runtime and resource utilization when compared to doing the same thing programmatically from within an application?

A In MSXML, XPath is cheap, resource-wise, and the Document Object Model (DOM) is slow. In System.Xml, the DOM is fast, and XPath may be slow (since it enforces XPath data-model rules).

Generally speaking, domain-specific languages (like XPath) beat out generic languages (like C#) for their designed task because they take advantage of special domain knowledge and internal data structures. Also, languages like XPath and SQL can throw away almost all intermediate temporary values, and choose to execute your query differently from how you specified it. C# and other generic languages are usually more constrained, and perform your program exactly the way you wrote it. For example, a descendant query could use an internal index to seek forward to all foo elements directly. You would be unable to do this if you wrote the equivalent code in C#; the common language runtime (CLR) can't do it either. But remember, when it comes to performance, you should always test, measure, and test some more.

Q I use the following piece of code in my automation scripts to start and stop tests:

DateAndTime = DatePart("m", Now) & "/" & DatePart("d", Now) & "/" &
    DatePart("yyyy", Now) & " " & CStr(UserDefinedStartTime)
        If (DateDiff("n", DateAndTime, Now) = 0)then
              NdisTest
When I moved these automation scripts onto machines with German locale settings they no longer worked! I think that the issue lies in the fact that English locales express the date as month/day/year, whereas German builds appear to express it as day/month/year. Originally, I simply tried to use time since that's all I'm really interested in. But when I use the time function, the event never actually fires at the specified time.

The bottom line is that I want a script to do something starting at 9:00 A.M. every morning. At 8:55 A.M. I want the script to stop whatever it's doing, and then start over again at 9:00 A.M. In other words, I want to create a simple loop based on the time. How can I accomplish this and remain independent of locale?

A Why use strings in the first place? Your example takes Now, which is a date, and UserDefinedStartTime (which is presumably a date, though it's difficult to tell from the code), converts them to strings, and passes them to DateDiff—which, of course, turns them right back into dates.

Also, it appears that you are trying to compare the current time to a given time. If so, then why mess around with dates when you could be using times? This should do what you want:

If DateDiff("n", UserDefinedStartTime, Time) = 0 Then

Take note of the way Visual Basic and VBScript represent dates and times. A date/time is stored in a double-precision float in which the signed integer part is the number of days since 30 December 1899 and the fractional part is the portion of the day elapsed. So 0.75 is 30 December 1899, 6 P.M.; -1.75 is 29 Dec 1899, 6 P.M.

Of course, by now you have already realized the potential problems. For example, 0.75 and -0.75 are both the same date/time. Similary, -2.999999 and -1.0 are only microseconds apart, even though there is a gap of almost two whole numbers. Hence you cannot use any simple arithmetic operation on dates whatsoever—rounding, addition, subtraction, and comparison all fail under various circumstances. This explains the necessity of helper functions like DateAdd and DateDiff.

Times are represented—of course—as the fractional part that I just discussed. So does 0.75 mean 6 P.M. or 30 December 1899, 6 P.M.? It depends on the context. That's why you're getting strange results sometimes; you are accidentally treating times as dates and the runtime is getting confused.

You'll also note that this format lacks support for both daylight savings time and time zones, has a very strange epoch date, has precision which varies enormously over its range, and has a range of a trillion trillion trillion trillion times the age of the universe.

If you want to get around these problems, here's one surefire way that works whether you give it times or dates:

Function Fire(UserTime)
    Fire = (Hour(UserTime) = Hour(Time)) And _
       (Minute(UserTime) =  Minute(Time))
End Function
This returns True if the hour and minute are equal to the hour and minute of the target; False is returned otherwise.

Q My XML code uses xs:all (see Figure 2). It validates just fine, but I get the following error when I try to add it to a schema cache:

C:\nt\admin\wmi\jobs\server\parser\schema\validate.vbs(20, 1) msxml6.dll:
file://jobs.xsd#/schema/complexType[6][@name = 'intervalWeeklyType']/
complexContent[1]/extension[1]

<all> is not the only particle in a <group> or being used as an extension.
What's wrong?

Figure 2 Using xs:all (Close Figure 2)
<xs:complexType name="intervalWeeklyType">
    <xs:complexContent>
    <xs:extension base="daysType">
      <xs:all>
        <xs:element name="WeeksInterval" type="xs:unsignedByte"
         minOccurs="0" maxOccurs="1" />
      </xs:all>
    </xs:extension>
  </xs:complexContent>
</xs:complexType>

<xs:complexType name="daysType">
  <xs:all>
    <xs:element name="Monday" type="xs:boolean" minOccurs="0"
     maxOccurs="1" />
    <xs:element name="Tuesday" type="xs:boolean" minOccurs="0"
     maxOccurs="1" />
    <xs:element name="Wednesday" type="xs:boolean"
     minOccurs="0"
     maxOccurs="1" />
    <xs:element name="Thursday" type="xs:boolean" minOccurs="0"
     maxOccurs="1" />
    <xs:element name="Friday" type="xs:boolean" minOccurs="0"
     maxOccurs="1" />
    <xs:element name="Saturday" type="xs:boolean" minOccurs="0"
     maxOccurs="1" />
    <xs:element name="Sunday" type="xs:boolean" minOccurs="0"
     maxOccurs="1" />
  </xs:all>
</xs:complexType>

A An all group can only appear at the top level and (conceptually at least) you end up with the following in your derived type:

<sequence>
  <all>
  •••
  </all>
  <all>
  •••
  </all>
</sequence>
Any complexType that uses xs:all cannot have its content model extended (you could still derive by extension and add attributes).

The MSDN® XML reference should help. In addition, see the W3C XML Schema primer at http://www.w3.org/TR/xmlschema-0.

Q What are my options for inserting binary data into SQL Server 2000 using XML? SQL Server 2000 easily generates base64-encoded binary data in response XML streams, but on insert/update operations, the conversion of a text string into the binary type is a straight character Unicode conversion. The query side is great, but I find the insert side somewhat lacking.

A If you are using SQLXML, you should be able to use Updategrams or Bulkload to upload the binary data. You will need to create an annotated mapping schema with the binary column mapped as xsd:base64Binary, and you will be able to upload the binary data which is stored as binarybase64. You can also upload binary data stored in binary format similarly using the xsd:hexBinary datatype in the mapping schema.

Unless you want to convert the binary data yourself to hex for insertion into SQL Server through OLE DB, XmlBulkload (in SQLXML 3.0) is very close to pure OLE DB calls, especially if the structure of the XML data is simple.

Q How can I use VBScript code to do a reboot on Windows NT® using Shell object, RunDll32, or something equivalent?

A You can use Windows Management Instrumentation (WMI):

Set wmi = GetObject("winmgmts:")
Set objset = wmi.instancesof("win32_operatingsystem")
For Each obj in objset
    Set os = obj
    Exit For
Next
os.reboot

Here are two Knowledge Base articles that will help. Article 149690, "ExitWindowsEx with EWX_LOGOFF Doesn't Work Properly," offers an explanation of the problems using this API to try to programmatically reboot Windows 9x. The other, Article 168796, "ExitWindows Function Declaration Incorrect in API Viewer," offers another example of code to perform this task.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Chris Beatie, Michael Brundage, Tom Cox, Derek Denny-Brown, Bruno Denuit, Roberto Di Pietro, Robert Gruen, Martin Gudgin, David He, Brian Heitt, Stephen Howard, Walter Hsueh, Chris Jensen, Douglas Laudenschlager, Eric Lippert, Taha Masood, Gray McDonald, Amar Nalla, Dare Obasanjo, Hossain Rahman, Jeffrey Richter, David Sceppa, Karol Zadora-Przylecki


Data Points: Exploring the ADO.NET DataRow

Data Points
Exploring the ADO.NET DataRow
John Papa

Code download available at: DataPoints0310.exe (136KB)
Contents
Creating and Importing Rows
Locating Rows
Summing It Up

ADO.NET provides a rich programming interface that allows developers to interact with rows through the DataTable, DataRowCollection, and the DataRow objects. Because I receive many questions regarding DataRow objects in ADO.NET, I have combined several of them and will present the answers in this month's installment of Data Points. I'll explore how you can manipulate data using the DataRow and focus on importing loading, traversing, locating, and examining data in ADO.NET and C#.

Creating and Importing Rows

The DataTable exposes several methods that you can use to manipulate DataRow objects. They include ImportRow, NewRow, Select, and LoadDataRow. I will use these methods in a sample Windows® Forms application to show you how to grab one or more rows from one DataSet and add them to a second DataSet. I'll also show you how to create and load a DataSet manually.

To begin, I load a DataSet directly from the SQL Server™ Northwind database Employee table (see Figure 1) and bind it to a DataGrid (see Figure 2). After loading the DataSet, I create a primary key constraint for the DataTable on the EmployeeID column. Since this column is an IDENTITY column in the Northwind database, I also set the AutoIncrement property to true and both the AutoIncrementSeed and AutoIncrementStep properties to -1. This tells the DataTable that whenever a row is added to this DataTable, the EmployeeID DataColumn's value should be generated by the DataTable, starting with a value of -1 and decreasing by 1 for each row. Thus, EmployeeID values in any added rows would have a value of -1, -2, -3 and so on. This is important because I can't know what EmployeeID value the database will assign when I send new rows to be inserted. Therefore, I assign a value that I know can't be assigned by the database (negative numbers). Then, when I send the data to the database to be inserted, I will ignore the EmployeeID value and will retrieve the newly created and assigned value from the database. (For more information please refer to the Data Points column in the July 2003 issue of MSDN® Magazine, "Managing Hierarchical Inserts in ASP.NET and ADO.NET").

Figure 2 Get Employees and Bind to DataGrid (Close Figure 2)
m_oCn = new SqlConnection(m_sCn);

//-- Set up the SELECT Command
string sSQL = "SELECT EmployeeID, FirstName, LastName FROM
               Employees";
SqlCommand oSelCmd = new SqlCommand(sSQL, m_oCn);

//-- Get the Employee records
oSelCmd.CommandType = CommandType.Text;
m_oDa.SelectCommand = oSelCmd;
m_oDa.Fill(m_oDs1, "Employees");

//-- Set the primary key
m_oDs1.Tables["Employees"].Constraints.Add("PK_Employees",
m_oDs1.Tables["Employees"].Columns["EmployeeID"], true);
m_oDs1.Tables["Employees"].Columns["EmployeeID"].
    AutoIncrement = true;
m_oDs1.Tables["Employees"].Columns["EmployeeID"].
    AutoIncrementSeed = -1;
m_oDs1.Tables["Employees"].Columns["EmployeeID"].
    AutoIncrementStep = -1;

//-- Bind
grdEmployees1.DataSource = m_oDs1.Tables["Employees"];
Figure 1 Loading a DataSet (Close Figure 1)

Next, I create a second DataSet manually using code, then add two rows of employees to it (see Figure 3). In this code sample I create a DataTable and add DataColumn objects to it that match the DataColumn objects from the first DataSet, m_oDs1. I also add a primary key constraint to this table to keep the two DataTables and DataSets in sync. However, notice that I set the AutoIncrementSeed to -1000 (see Figure 3). That way, when I take the rows from this manually created DataTable and copy them to the DataTable derived from the database, none of the rows will have duplicate EmployeeIDs. This is a simple, yet effective, solution for avoiding non-unique key values.

Figure 3 Creating a DataSet Manually (Close Figure 3)
DataTable oDt = new DataTable("Employees");
DataRow oRow;

//-- Create the table
oDt.Columns.Add("EmployeeID",
    System.Type.GetType("System.Int32"));
oDt.Columns.Add("FirstName",
    System.Type.GetType("System.String"));
oDt.Columns.Add("LastName",
    System.Type.GetType("System.String"));
oDt.Constraints.Add("PK_Employees", oDt.Columns["EmployeeID"],
    true);
oDt.Columns["EmployeeID"].AutoIncrement = true;
oDt.Columns["EmployeeID"].AutoIncrementSeed = -1000;
oDt.Columns["EmployeeID"].AutoIncrementStep = -1;
m_oDs2.Tables.Add(oDt);

//-- Add the rows
oRow = m_oDs2.Tables["Employees"].NewRow();
oRow["FirstName"] = "Lloyd";
oRow["LastName"] = "Llamas";
m_oDs2.Tables["Employees"].Rows.Add(oRow);
oRow = m_oDs2.Tables["Employees"].NewRow();
oRow["FirstName"] = "Layla";
oRow["LastName"] = "Lewis";
m_oDs2.Tables["Employees"].Rows.Add(oRow);
grdEmployees2.DataSource = m_oDs2.Tables["Employees"];

Next, I create a new row for the manually created DataTable using the NewRow method (again, see Figure 3). This may seem awkward because what is really happening here is that to add a new row to a DataTable I actually have to ask the DataTable for a new row, set its values, and then add the new row to the DataTable. Thus, when I call the NewRow method, the newly created row is actually detached from the DataTable. To attach it, it must then be passed to the Add method of the DataTable's Rows collection.

The following code can be executed to import the rows from the second DataSet (the manually created one) into the first DataSet (from the Northwind database).

//-- Import the rows
foreach (DataRow oRow in m_oDs2.Tables["Employees"].Rows)
{
    m_oDs1.Tables["Employees"].ImportRow(oRow);
}
This code loops through the two rows I added to the manually created DataTable and imports them via the ImportRow method one at a time into the DataSet that is derived from the Northwind Employee table.

That's all you need to import data. However, there are some important aspects of imported data to keep in mind. When data is imported from a DataTable into another DataTable, the RowState is also imported. To demonstrate this, I added two additional DataGrid objects to the Windows Form (see Figure 1). I added a second DataTable to store the RowState for each row to the first DataSet, which already contained a DataTable of employees from Northwind. Thus there are two DataTable objects in the first DataSet m_oDs, one referring to the employees and another that has the same number of rows as the first one. The second DataTable, which I call State, is used simply to store the RowState of each row from the Employee DataTable.

The second DataSet already contains a DataTable to store the employees that I manually created with code. I added a second DataTable to it to store the RowState for each of its rows. The purpose here is to demonstrate how the RowState of a row is maintained by the ImportRow method. The RowStates of the rows retrieved from the database's Employees table and stored in the first DataSet m_oDs1 are all set to "unchanged" since I do nothing to them. The RowState values of the rows that I manually added to the second DataSet m_oDs2 are both "added," since I added them myself. Thus, when I import these two rows into the m_oDs1 DataSet, the RowState for the two new rows is still "added."

As a side note, if I had invoked the AcceptChanges method on the first DataSet after I added the two rows, their RowState values would have been set to "unchanged." Therefore, when I later imported them into the m_oDs2 DataSet, their RowStates would also be "unchanged." When invoked on a DataTable, the AcceptChanges method tells the DataTable that the original values of each DataRow should be set to whatever value is current, producing an "unchanged" RowState for each row. The full code for this example includes the code to keep track of the DataGrid RowState as well.

LoadDataRow is the other method worth noting while I'm on this topic. The ImportRow method essentially copies one DataRow from a DataTable into a second DataTable, assuming both DataTable objects have the same columns. The LoadDataRow method can also be used to copy rows. The following code shows how ImportRow was used in the previous code sample to copy a DataRow from one DataTable to another DataTable:

//-- Import Row
m_oDs1.Tables["Employees"].ImportRow(oRow);
This can also be accomplished by using the LoadDataRow method, as shown here:
//-- Load Data Row
object[] aRowValues = {oRow["EmployeeID"], oRow["FirstName"],
oRow["LastName"]};
m_oDs1.Tables["Employees"].LoadDataRow(aRowValues, false);

Since rows are simply being copied, ImportRow is more practical when two DataTable objects exist with the same columns. However, LoadDataRow yields a little more flexibility. For example, if the DataTable objects do not have the same columns, LoadDataRow can be used whereas ImportRow cannot. This is because the LoadDataRow method accepts an array of values to insert, in this case, into the DataTable. Thus, if there were different columns in the two DataTable objects, LoadDataRow could simply grab the appropriate values in the proper order from the second DataTable to be inserted into the first DataTable.

However, the LoadDataRow method serves another purpose beyond inserting a row into a DataTable. In fact, LoadDataRow is most useful when it is necessary to update an existing row if one exists or to add a row if it does not yet exist. If the key value for the row is found, LoadDataRow will update that row's values instead of inserting a new row. This is just another way in which LoadDataRow shows greater flexibility over ImportRow, since in this scenario ImportRow would raise an exception if a row already existed with a duplicate key value.

Locating Rows

The next code example addresses several questions I received regarding how to find a row. Since there are several ways to find rows using ADO.NET, I'll walk through examples for three of the most prominent techniques: the Find, Contains, and Select methods. All three of these methods can be used to locate specific rows; however, they accomplish this task in very different ways.

The Find method looks for a row by the primary key value of the row. If a row is found, it is returned; if no row is found, a null value is returned. This Find method is very useful when you need to locate a row and then perform an action on it. I used the Find method in the first example in this column when I synchronized the employee DataGrid with its RowState DataGrid. In that example, I had a common value in both DataGrid objects—the EmployeeID. I created a primary key constraint on both of the DataGrid objects' underlying DataTable objects using their EmployeeID DataColumns. By doing this, I was able to loop through the rows of the employee DataTable and set the corresponding state DataTable's row to the appropriate value. Notice how the following code sample loops through each employee in the Employees DataTable and uses the Find method to retrieve the matching row from the State DataTable.

DataRow oStateRow;
foreach (DataRow oRow in m_oDs1.Tables["Employees"].Rows)
{
    oStateRow = m_oDs1.Tables["State"].Rows.Find(oRow["EmployeeID"]);
    oStateRow["RowState"] = oRow.RowState;
}

In this example, for the Find method to work properly there must be a primary key. The primary key constraint must exist on the EmployeeID DataColumn; otherwise the Find method will raise an exception. The Find method relies on the constraint to locate the desired row. Since the Find method requires a primary key constraint, there is no need to worry about locating more than one row. If the DataTable had multiple DataColumn objects that made up its primary key resulting in a compound key, I could have simply passed in an array of values representing the key values to the Find method.

If I don't need to locate and perform an action on the row, but rather just need to know if a row exists in the DataTable that matches a primary key value, I could use the Contains method. It is very useful when you need to check for the existence of a row but not retrieve it. The Contains method works like the Find method in that it can be passed a single value or an array of values, depending on how many columns make up its primary key. Contains relies on the existence of a primary key just like Find, so it is important to make sure that one exists before going down this road. Contains yields a simple way to check for the existence of a row while Find not only checks for the row, it returns it.

The third technique is to use the Select method of the DataTable object. It returns an array of DataRow objects that meet the specified criteria. The Select method has many overloaded forms, so it can be filtered using a string expression, returned in a particular sort order, or it can be filtered to show only certain row states.

One key difference between the Select and Find methods is that Find searches for one row that matches the primary key value(s) passed in. This Find method may return zero rows or one row while the Select method may return zero or more rows. Select is more flexible in this respect since it is not dependent upon the existence of a primary key constraint. It searches for all rows that meet the criteria and it returns them in an array of DataRow objects. To compare the Find and Select methods, in the following code snippet I have shown the code that uses Find from the previous example application as well as what the replacement code would look like using Select:

//-- Using the Find method
oStateRow = m_oDs1.Tables["State"].Rows.Find(oRow["EmployeeID"]);
oStateRow["RowState"] = oRow.RowState;

//-- Using the Select method
DataRow[] aRow = m_oDs1.Tables["State"].Select("EmployeeID = " +
    oRow["EmployeeID"]);
aRow[0]["RowState"] = oRow.RowState;

For this example, Find is more efficient since I am always looking for a single row. Select can be used; however, it does not have the advantage of using only the primary key value. Also, notice how in the sample code where I used the Select method, I hardcoded the row index to the first row (index of 0). I could have looped through the returned rows, but since I know that there is only going to be one row, I took the risk of hardcoding it. Obviously, this is not the best technique when looking for a single row.

The Select method can be very effective, however, when searching for an unknown number of rows. For example, if I wanted to find all employees from Seattle (assuming the City column was in my DataTable) I could easily do so by using the following code:

DataRow[] aRow = oDs.Tables["Employees"].Select("City = 'Seattle'");
I could also sort the retrieved DataRow objects in the array by specifying the sort order parameter of the Select method as follows:
DataRow[] aRow = m_oDs2.Tables["Employees"].Select("City = 'Seattle'",
    "LastName");
This would return the two employees who live in Seattle, sorted by their last names. The Select method can also filter rows using the DataViewRowState. The following code would return the employees from Seattle who were just added, sorted by last name:
DataRow[] aRow = m_oDs2.Tables["Employees"].Select("City = 'Seattle'",
    "LastName", DataViewRowState.Added);

Summing It Up

This column demonstrated how to retrieve rows, check to see if a row exists, locate a specific row based on key value, create tables manually, create a new row, and copy rows between DataTable objects. It also touched on DataRowState and how it is maintained when a row is imported using the ImportRow method. The DataRowState and the DataViewRowState are both instrumental in ADO.NET on a number of fronts, especially when dealing with concurrency, as I will discuss in a future Data Points column.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive. You can reach him at mmdata@microsoft.com.


Cutting Edge: Nested Grids for Hierarchical Data

Cutting Edge
Nested Grids for Hierarchical Data
Dino Esposito

Code download available at: CuttingEdge0310.exe (135KB)
Contents
Building Nested Grids
The NestedGrid Class
The ExpandCommandColumn Class
Rendering the Child Grid
Creating the Child View
Conclusion

In the August 2003 installment of Cutting Edge, I discussed how to extend the ASP.NET DataGrid server control to use a multi-table data container such as a DataSet object as its data source. If the DataSet contains pairs of interrelated tables, the control adds a dynamically created button column whenever the displayed table is the parent of one of these relations. When the column button is clicked, a child DataGrid is displayed, listing the child rows of the selected record according to the relation. The overall behavior is illustrated in Figure 1 and is similar to how the Windows® Forms DataGrid control works in similar situations.

Figure 1 Parent and Child DataGrids
Figure 1 Parent and Child DataGrids

The application shown in Figure 1 is a user control that comprises two DataGrid controls working together. The user control (see the August 2003 source code) contains all of the logic necessary to keep the two grids in sync. The parent DataGrid is bound to a DataSet and displays the contents of the parent table. When this happens, the user control ensures that a relation exists within the DataSet where the displayed table acts as the parent. The child DataGrid is bound to a view of the data that includes all the records in the child table which are only related to the selected record. As a result, if you have a DataSet with two tables whose relations have been set, the user control saves you from coding any extra display mechanism.

So what's wrong with this approach? Nothing, if you're just looking at basic functionality. Some readers, however, noted that they would rather not have two physically separate DataGrid controls. The user control builds a wall around the constituent controls, so the only way you can access them is by either mirroring properties and methods or by exposing the internal control as a whole. From the standpoint of programmability, having a single DataGrid control to display hierarchical data would make life significantly simpler. For one thing, you wouldn't have to worry about the configuration of the parent table. You could just use the standard interface of the DataGrid control. Any child grids that display related data can be created dynamically and displayed within the layout of the main grid.

Figure 2 Embedded Child DataGrid
Figure 2 Embedded Child DataGrid

On the other hand, remember that the DataGrid control is not designed to contain hierarchical data. Its internal layout is optimized for tabular data. The DataList control might be a good choice, but it doesn't provide native paging support and it requires a bit of code to work like a DataGrid. A quick search on Google for "nested DataGrid" returned links to articles discussing how to embed DataGrids in a DataList control, which gave me some ideas for this column. Here, I'll build a custom control that inherits from the DataGrid class. The control implements a custom column type (ExpandCommandColumn) and contains all the logic needed to display records associated with the clicked item. The expanded view is represented by a child DataGrid embedded in the parent. Figure 2 shows what it will look like.

Building Nested Grids

A hierarchical DataGrid control makes sense only if the data source is a DataSet object that contains relations between tables. For example, consider a DataSet with Customers and Orders tables with a DataRelation set between the two on the CustomerID column. As long as the DataGrid contains a button column, when you click it you could create a child view for the selected customer and bind the resulting DataView object to the child grid.

Because the new control (named NestedGrid in the sample code) inherits from the DataGrid class, you can use it whenever a DataGrid object is appropriate. However, take this last statement with a grain of salt. In general, when you derive a control from a base class there might be situations in which the derived control can't just replace the original due to its specific extensions and additions. In this column, I won't spend much time making the NestedGrid component backward compatible with the base DataGrid class. For simplicity, I'll assume that you always bind it to a DataSet object.

I make a few more assumptions regarding the NestedGrid control, which will become clear later on. In particular, you are responsible for adding a button column that will dictate the expanded/collapsed state of each row. In theory, this column can be placed anywhere in the grid. However, I assume here that the expand column is the first column in the grid. (As discussed in the August 2003 issue, you could modify the behavior to dynamically generate the column only if the DataGrid is bound to a DataSet that has related tables.)

If you've played with the DataGrid control a bit, you know that although it's extremely powerful and customizable, it doesn't happily support changes in its layout. The layout of a grid represents a table of data—a regular succession of equally sized rows. How can you embed a child grid with this restriction?

The key point to recall here is that the grid is rendered as a standard HTML table. Once the cells form a regular table layout, you can place anything within each of them, including a child table that represents a child grid (using a rowspan tag). First, you modify the number of cells that form the selected row—that is, the row where the expand command button that the user clicked is located—by removing all the cells except the one that contains the command button. This is pretty easy to do if you assume that the expand command column is the leftmost one. Once all the cells have been removed, then you can create a brand new one that spans the necessary number of columns, equal to the number of items in the Columns collection of the DataGrid control.

At this point, you have a completely custom cell for the row to expand. This custom cell can be programmatically populated with any combination of server controls. For example, you can insert a table in which the top row mimics the structure of the removed cells (typically, information about the parent row) and the bottom row contains the child DataGrid. The control in Figure 2 is created based on this scheme.

The NestedGrid Class

As I mentioned earlier, the NestedGrid class inherits from the System.Web.DataGrid class and adds a few extra properties (see Figure 3). The control also raises the custom UpdateView event whenever data binding is required. To exercise strict control over the object type assigned to the DataSource property (and ensure it is a DataSet), you can override the DataSource property, as shown in the following code:

public override object DataSource
{
   get {return base.DataSource;}
   set {
      if (!(value is DataSet)) {
         // throw an exception
      }
      base.DataSource = value;
   }
}

Figure 3 NestedGrid Properties (Close Figure 3)

Property | Description
ExpandedItemIndex | Gets or sets the zero-based index of the item currently selected. (The control supports only one item expanded at a time.)
RelationName | Gets or sets the name of the relation within the bound DataSet in which the DataMember table is the parent. This relation determines the contents of the child grid.
ScrollChildren | Indicates whether the contents of the child grid should be scrollable. By default, the contents are pageable.

A NestedGrid control is instantiated when a user clicks on a row button to expand a record (a customer) to see its details (related orders). For this reason, a nested grid must contain a button column with some specific features. For one thing, the grid must provide a handler for the ItemCommand event to handle the expand/collapse request. The handler sets the ExpandedItemIndex property to the zero-based index of the record that was clicked and updates the grid's view. So when is a good time to modify the layout of the clicked row?

The ItemDataBound event fires at the bottom of the chain of events that occur when the grid layout is created. After ItemDataBound fires, the data binding phase is almost complete and all the cells are ready for display. The layout and data you see from this point will not change any further. For exactly this reason, I decided to implement any needed changes before handling the ItemDataBound event.

Before going deep into the implementation of the control, let me share a few more notes. First, the ExpandedItemIndex property is zero-based but represents the absolute position of the clicked row. This property differs from similar grid properties (such as SelectedItemIndex and EditItemIndex) only in that it doesn't represent a page-based value. Second, the NestedGrid also implements paging internally. For the control to move through pages of the member table, you don't have to do anything but handle the UpdateView event and pass the binding data:

void UpdateView(object sender, EventArgs e) {
   BindData();
}
void BindData() {
   dataGrid.DataSource = (DataSet) Cache["MyData"];
   dataGrid.DataBind();
}

The NestedGrid class has a built-in handler for the PageIndexChanged event, as shown here:

void PageIndexChanged(object sender, DataGridPageChangedEventArgs e)
{
   CurrentPageIndex = e.NewPageIndex;
   SelectedIndex = -1;
   EditItemIndex = -1;
   ExpandedItemIndex = -1;

   if (UpdateView != null)
      UpdateView(this, EventArgs.Empty);
}

The key element in the architecture of the NestedGrid control is the button column. For the sake of simplicity, this version of the control supports only a single expanded item. This feature can be easily extended by changing the ExpandedItemIndex property from an integer to an array or collection.

The ExpandCommandColumn Class

The expand column can be rendered using either a string (such as "+/-", or "Expand/Collapse") or a bitmap. You might want to use different pictures for different applications. The most flexible way to implement this feature is to use a couple of properties such as ExpandText and CollapseText. So, should you define these properties on the NestedGrid class? In a similar scenario—in-place editing—the ASP.NET team created a custom DataGrid column and placed properties such as EditText, CancelText, and UpdateText on it. Based on this, I created my own ExpandCommandColumn class with a couple of text properties representing the HTML output for expanding and collapsing the view. The following code snippet shows how to integrate the custom column with the grid.

<cc1:NestedDataGrid id="dataGrid" runat="server" ...>
<Columns>
   <cc1:ExpandCommandColumn
      CollapseText="<img src=images/collapse.gif>"
      ExpandText="<img src=images/expand.gif>" >
      <ItemStyle Width="15px" />
   </cc1:ExpandCommandColumn>
•••
</Columns>
</cc1:NestedDataGrid>

Building a custom DataGrid column is not difficult. It doesn't require much work beyond creating a new class that inherits from DataGridColumn. In the new class, you must code any extra properties you need and override the InitializeCell method. This method is called whenever a cell is being created for the column. Everything that appears by default within a column cell is dictated by this method. The following code demonstrates the implementation of the ExpandText property:

public class ExpandCommandColumn : DataGridColumn
{
  public string ExpandText {
    get {
      object data = ViewState["ExpandText"];
    if (data != null)
          return (string) data;
    return "+";
    }
    set {
       ViewState["ExpandText"] = value;
    }
  }
  •••
}
The CollapseText property differs only in the name of the viewstate slot and its default value ("-").

It is worth noticing that the default value of server control properties should be set in the get accessor instead of in the constructor or in an initialization event such as Init or Load. This is a practice that Microsoft used throughout ASP.NET. By insulating such code inside the get accessor of the property you gain code encapsulation and a neater separation between the logic behind the value of a property and the rest of the control. Especially when the default value of a property is subject to sophisticated rules, you have a single point of control, which makes the overall code more maintainable. Speaking of best practices, bear in mind that the value returned for a property must be checked for null and normalized if needed. For example, a property of type string should never return null; it should return an empty string instead.

A DataGrid column is centered around the InitializeCell method. The method is declared public and virtual (that is, it can be overridden in derived classes) and the code within the DataGrid control calls it whenever the column needs to be rendered. Despite being declared as public, the method is usually used only by control developers. Let's have a look at the signature:

public override void InitializeCell(
   TableCell cell,
   int columnIndex,
   ListItemType itemType)

The DataGrid code calls the method, passing the object that represents the cell being created, the index of the column in the grid's Columns collection, and the type of the cell to be rendered (header, footer, item, and so on). As you can see, there's no information about the index of the cell in the grid's page. Is this information really important? Looking at the predefined types of grid columns, the answer would seem to be no. (In fact, this information is not passed along.) The predefined grid columns (bound, button, hyperlink, template columns) use one of two algorithms to populate cells. If their Text properties are set, all the cells contain a fixed and constant value; otherwise, if a data-bound property (like the DataField) is set, the contents of each cell is resolved through a data binding process.

So which category does the ExpandCommandColumn type fall into? Well, neither. Ideally, this type will render the cell text using ExpandText or CollapseText, depending on the state of the item being rendered. If the cell index matches the ExpandedItemIndex property (or belongs to the collection of expanded items), the CollapseText value is used; otherwise, the default ExpandText property is employed. How can the column's InitializeCell method know about the cell index?

The first idea that crossed my mind was to take the TableCell object that gets passed to the method, invoke its NamingContainer property, and cast the results to DataGridItem. If the object obtained is not null, that would be the container of the cell and its ItemIndex property would contain the required information. Unfortunately, things are not that easy. The naming container of the cell object is null because when InitializeCell is called, the TableCell object has not yet been added to the grid item container. Consequently, it doesn't belong to any parent container, so the NamingContainer property returns null.

To find a workaround, I pored over the list of internal overridable methods of the DataGrid control. The DataGrid's InitializeItem method proved to be the one I wanted. This method is responsible for initializing the grid columns when the grid layout is being created. The InitializeItem method is mentioned in the ASP.NET documentation on MSDN®, but not fully described. As of ASP.NET 1.x, its behavior is fairly simple. InitializeItem takes two parameters: the DataGridItem object that represents the grid row that's being rendered and an array of DataGridColumn objects (the columns of the row):

protected virtual void InitializeItem(
   DataGridItem item,
   DataGridColumn[] columns
);

The InitializeItem method loops over the columns and creates a new TableCell object for each one. This object is passed to the column-specific InitializeCell method and then added to the Cells collection of the DataGridItem object. (As you can guess, it's only at this point that the TableCell's naming container is set to a non-null value.) The code in Figure 4 shows the overridden version of InitializeItem that passes an extra flag down to the ExpandCommandColumn column class.

Figure 4 Passing a Flag to the Column Class (Close Figure 4)
public class NestedGrid : DataGrid
{
    •••
    // DataGrid::InitializeItem (overridden)
    // If the column is of type ExpandCommandColumn, this calls an
    // overloaded version of the DataGridColumn::InitializeCell
    // method. The extra information passed on indicates
    // whether the column cell is to be rendered
    // as expanded or collapsed.
    protected override void InitializeItem(DataGridItem item,
        DataGridColumn[] columns)
    {
        for (int i=0; i<columns.Length; i++)
        {
            TableCell cell = new TableCell();

            // Check the column type
            if (columns[i] is ExpandCommandColumn)
            {
                // Grab the column object
                ExpandCommandColumn col = (ExpandCommandColumn)
                    columns[i];

                // Determine if the cell is expanded/collapsed
                bool expanded = (item.ItemIndex==(ExpandedItemIndex %
                    this.PageSize));

                // Call the overloaded method (see ExpandCommandColumn.cs)
                col.InitializeCell(cell, i, item.ItemType,
                    expanded);
            }
            else
                columns[i].InitializeCell(cell, i, item.ItemType);

            // Add the newly configured cell to the grid item
            item.Cells.Add(cell);
        }
    }
    •••
}

Figure 5 contains the code that is used to initialize the cells of the ExpandCommandColumn class. Each cell is rendered as a link button with the text determined by the extra Boolean argument (see Figure 6). As with many other elements of the DataGrid control, HTML text is fully supported, so you can use images to implement the expand/collapse feature.

Figure 5 Initializing the ExpandCommandColumn Class (Close Figure 5)
public class ExpandCommandColumn: DataGridColumn
{
    •••
        // Initializes the cells in the column
        public void InitializeCell(TableCell cell, int
            columnIndex, ListItemType itemType, bool expanded)
        {
            // Call the base method
            base.InitializeCell(cell, columnIndex, itemType);

            // Determine how to populate the cell
            if (itemType == ListItemType.Item ||
                itemType == ListItemType.AlternatingItem)
            {
                // Create a link button
                LinkButton link = new LinkButton();

                // Set the command name
                link.CommandName = "Expand";

                // Determine the command text to show
                if (expanded)
                    link.Text = this.CollapseText;
                else
                    link.Text = this.ExpandText;

                // Add the button to the Controls collection of the cell
                link.CausesValidation = false;
                cell.Controls.Add(link);
            }
        }
    •••
}
Figure 6 Cell as Link Button
Figure 6 Cell as Link Button

What happens when the column's link button is clicked? If you need a column-specific behavior, then add a handler for the Click event. That code will be the first to execute after the click. Next, the event will bubble up by way of the DataGridItem class, and will result in an ItemCommand event at the DataGrid level.

Rendering the Child Grid

Although highly customizable, the DataGrid control doesn't provide facilities to modify the HTML layout of the rows. The DataGrid customization logic is built around the idea that the grid is made of columns, and rows are simply the result of columns next to each other. But here you need to modify the structure of the selected row to encompass a child DataGrid control. There are only two places in the process where the grid's layout can be altered—the ItemCreated event or, better yet, the ItemDataBound event. The ItemDataBound event fires a little later in the grid item's lifetime and is the last event you see before a new row is added to the final HTML table.

When the user clicks one of the link buttons in the command column, the ItemCommand event bubbles up to the grid. If the command name equals Expand—the command name of the column's buttons—the code first checks to see whether the index of the clicked item matches the ExpandedItemIndex property. If so, the user has clicked on an already expanded item which will then be collapsed. Figure 7 shows the code that implements this mechanism. As mentioned earlier, the ExpandedItemIndex property is an absolute index ranging from 0 to the number of items in the data source. That's why you need to compare its modulus against the item's index while composing the grid's page.

Figure 7 Expand/Collapse Mechanism (Close Figure 7)
// Command handler
private void NestedDataGrid_ItemCommand(object source,
    DataGridCommandEventArgs e)
{
    if (e.CommandName != "Expand")
        return;

    ExpandItem(e.Item);
}

// Adjust the index of the expanded item
private void ExpandItem(DataGridItem item)
{
    if (item.ItemIndex == (ExpandedItemIndex % this.PageSize))
        SetExpandedItem(item, false);
    else
        SetExpandedItem(item, true);

    OnUpdateView();
}

// Adjust the index of the expanded item
private void SetExpandedItem(DataGridItem item, bool expand)
{
    // ExpandedItemIndex is an absolute index
    // (take care of the current page index and page size)
    if (expand)
        ExpandedItemIndex =
          (this.PageSize*this.CurrentPageIndex+item.ItemIndex);
    else
        ExpandedItemIndex = -1;
}

The final step in the code shown in Figure 7 fires the UpdateView event. For the NestedGrid control, this event represents the entry point to the process of UI rendering. The client is expected to handle the event, bind any necessary data, and finally call into the grid's DataBind method. At this point, the control's lifetime is a succession of events, the first of which is DataBinding. Next, ItemCreated and ItemDataBound are fired for each item in the grid, including the header, data rows, and footer. At this stage, InitializeItem is invoked to populate the cells of each bound column.

In Figure 2, you can see that the grid makes room for another embedded grid which shows the child rows of the expanded record. Assuming that the expand column is the leftmost one, you remove all subsequent cells and replace them with a new cell in which the RowSpan attribute is properly set. You can determine the contents of this new cell, but it should contain at least the following information: the removed cells (that is, information about the record you're going to expand) and the child grid. That a handful of cells are to be removed and added again later may sound bizarre, but this compromise is necessary to wed two contrasting requirements: inserting a child table while preserving the rest of the table layout.

In my implementation, I cache text and the width of each cell that is being removed. As an alternative approach, you can consider moving the TableCell object from one Cells collection to another (see Figure 8). The new cell contains a two-row table in which the first row repeats the original cells and the second row spans the width to display a child DataGrid.

Figure 8 Moving the TableCell Object (Close Figure 8)
// Modify the layout of the cell being expanded
private void BuildChildLayout(DataGridItem item)
{
    DataGridItem row = item;

    // Assumes the Expand column is the first

    // Remove all cells but one
    int cellsToSpanOver = row.Cells.Count-1;
    ArrayList listOfText = new ArrayList();
    ArrayList listOfWidth = new ArrayList();
    for (int i=row.Cells.Count-1; i>0; i—)
    {
listOfText.Add(row.Cells[i].Text);
if (i==1) // Add the width of the column whose width is not declared
    listOfWidth.Add(HostColumnWidth);
else
      listOfWidth.Add(this.Columns[i].ItemStyle.Width);
      row.Cells.RemoveAt(i);
    }

    // Add the new cell that will host the child grid
    TableCell newCell = new TableCell();
    newCell.ColumnSpan = cellsToSpanOver;
    newCell.BackColor = Color.SkyBlue;

    // MUST BE empty. If you set a fixed width declaratively, that value
    // will override this one. For this reason, we set the width of the
    // first column after the EXPAND column dynamically. We also assume
    // that the first column after the EXPAND column is the host cell,
    // where the child grid is inserted.
    newCell.Width = Unit.Empty;
    row.Cells.Add(newCell);

    // The child layout is made of a 2-row table: header (same as the
    // previous unexpanded row) and the subgrid
    Table t = new Table();
    t.Font.Name = this.Font.Name;
    t.Font.Size = this.Font.Size;
    t.CellSpacing = this.CellSpacing;
    t.CellPadding = this.CellSpacing;
    t.BorderWidth = this.BorderWidth;

    TableRow rowHeader = new TableRow();
    t.Rows.Add(rowHeader);
    TableRow rowSubGrid = new TableRow();
    t.Rows.Add(rowSubGrid);
    newCell.Controls.Add(t);

    // Fill the header row
    for (int i=listOfText.Count-1; i>=0; i—)
    {
TableCell c = new TableCell();
c.Text = listOfText[i].ToString();
c.Width = (Unit) listOfWidth[i];
rowHeader.Cells.Add(c);
    }

    // Fill the second row
    Panel outerPanel = null;
    if (ScrollChildren)
    {
outerPanel = new Panel();
outerPanel.Height = Unit.Pixel(100);
outerPanel.Style["overflow"] = "auto";
    }

    TableCell cellSubGrid = new TableCell();
    cellSubGrid.ColumnSpan = cellsToSpanOver;
    cellSubGrid.BackColor = Color.LightCyan;
    rowSubGrid.Cells.Add(cellSubGrid);

    detailsGrid = new DataGrid();
    detailsGrid.ID = "detailsGrid";
    detailsGrid.BackColor = Color.LightCyan;
    detailsGrid.Font.Name = this.Font.Name;
    detailsGrid.Font.Size = this.Font.Size;
    detailsGrid.HeaderStyle.Font.Bold = true;
    detailsGrid.Width = Unit.Percentage(100);

    if (!ScrollChildren)
    {
detailsGrid.AllowPaging = true;
detailsGrid.PageSize = 5;
detailsGrid.PageIndexChanged += new
DataGridPageChangedEventHandler(detailsGrid_PageIndexChanged);
    }
    BindDetails(detailsGrid);

    if (ScrollChildren)
    {
outerPanel.Controls.Add(detailsGrid);
cellSubGrid.Controls.Add(outerPanel);
    }
    else
    {
cellSubGrid.Controls.Add(detailsGrid);
    }
}

When I first tested this code, it caused trouble. Before coding to ASP.NET, I verified my idea using plain HTML. I was so sure that a layout such as the one described previously made sense, I coded it within the ItemDataBound event of an ASP.NET DataGrid. To my great surprise, it did not span properly. It took me a while to figure out what was going on. The rub lies in the fact that I assigned an explicit width in pixels to each column, including the first column after the expand column:

<asp:boundcolumn runat="server"
   headertext="ID"
   datafield="ID"
   itemstyle-width="150px" />

Consider now that the new cell—the one destined to contain the child grid—is still the first after the expand column and as such inherits the width in pixels of the original ID column. To have this cell spanned over the available space, the Width property shouldn't be set; it should be left empty. No matter what you do in code (see the ItemDataBound event handler), the DataGrid internal framework always generates a style attribute in which the original width in pixels is retained. If you look at the source HTML for the cell, you see a similar design:

style="width:150px;...;width='';"

The width property is set twice (the second assignment is what you made in ItemDataBound), but the first value is the only one that matters for the browser. For this issue, I haven't found a better workaround than dropping the static width assignment. If the column needs to have a width, you can define a custom property (called HostColumnWidth in the sample) representing the width in units of the host column—the first column after the expand column. The following code snippet shows how to dynamically set the width of a column, obtaining an effect that is equivalent to setting the item's style:

if (e.Item.ItemIndex != (ExpandedItem % this.PageSize)) {
   // Equivalent to setting itemstyle-width declaratively
   e.Item.Cells[1].Width = HostColumnWidth;
   return;
}

Creating the Child View

At this point, the end of the game is fast approaching and there is one more thing left to do—populate the child DataGrid. The way you accomplish this depends on the internal layout of the hierarchical data you're managing. However, assuming that you hold your multi-level data in an ADO.NET DataSet with pairs of related tables, using child DataView objects is a viable option. (This approach is similar to what I discussed in my August 2003 column.)

The user selects the parent table to show in the grid and the relation that determines child views. A child view is created by calling the CreateChildView method on the DataRowView object that represents the parent record:

DataTable dt = ds.Tables[this.DataMember];
DataView theView = new DataView(dt);
DataRowView drv = theView[ExpandedItemIndex];
DataView detailsView = drv.CreateChildView(this.RelationName);

The set of records associated with the expanded row are grouped in a new DataView object. For example, given a customer-to-orders relationship, the set of child records are the orders issued by a given customer. The child grid is created and configured dynamically, as shown here:

detailsGrid = new DataGrid();
detailsGrid.ID = "detailsGrid";
detailsGrid.Font.Name = this.Font.Name;
detailsGrid.Font.Size = this.Font.Size;
detailsGrid.Width = Unit.Percentage(100);
detailsGrid.AllowPaging = true;
detailsGrid.PageSize = 5;
detailsGrid.PageIndexChanged += new DataGridPageChangedEventHandler(
       detailsGrid_PageIndexChanged);
BindDetails(detailsGrid);

In particular, the child grid manages paging internally. It is given a built-in handler for the PageIndexChanged event, which automatically moves the grid to the next page as the user clicks on the pager. No further code is required on the programmer's end for this feature to work.

Any event that is generated within the embedded grid is not visible outside the outermost grid. Among other things, this means that no user code could ever be written to handle the clicking on the child grid's pager bar. Is there a way to work around this structural limitation? One possibility is that you fire a new event from within the internal handler.

If you don't like paging the child grid, you can use a scrollbar and wrap the grid into a scrollable panel. In HTML 4.0, the overflow CSS attribute transforms a bunch of HTML elements into scrollable areas if the contents exceed the fixed dimensions. To make the child grid scrollable, you simply need to wrap it in a Panel (corresponding to a <div> tag) and give the panel the overflow attribute (see Figure 9). Now the header of the grid scrolls with the rest of the control. This behavior is by design and requires quite a sophisticated trick, which I will not take the time to discuss in this column.

Figure 9 Making a Scrollable Child Grid (Close Figure 9)
Panel outerPanel = null;
if (ScrollChildren) {
    outerPanel = new Panel();
    outerPanel.Height = Unit.Pixel(100);
    outerPanel.Style["overflow"] = "auto";
}

// Build the grid

if (ScrollChildren) {
    outerPanel.Controls.Add(detailsGrid);
    cellSubGrid.Controls.Add(outerPanel);
}
else
    cellSubGrid.Controls.Add(detailsGrid);

Conclusion

The ADO.NET DataSet object works like an in-memory database with a collection of tables and relations and allows you to create a hierarchical representation of data. You could render such data through a grid by using a combination of iterative and data-bound controls (such as DataList, Repeater, or Label) to easily simulate relationships. The downside of this approach is that you have to explicitly write code to perform paging. The DataGrid provides a lot of interesting features but doesn't supply any facility to render hierarchical, multi-table data. In the August 2003 Cutting Edge column, I discussed one solution based on a user control. In this column, I topped off the discussion by extending the DataGrid control itself through inheritance.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Programming ASP.NET and Applied XML Programming for .NET, both from Microsoft Press, he spends most of his time teaching classes on ADO.NET and ASP.NET and speaking at conferences. Get in touch with him at dinoe@wintellect.com.


The XML Files: OPENXML, XSLT Keys, Select versus Match, XPath, and More

The XML Files
OPENXML, XSLT Keys, Select versus Match, XPath, and More
Aaron Skonnard

Q Is it possible to write a SQL Server™ stored procedure that takes an XML document as input and inserts the information into tables?

A The process of pulling an XML document apart and inserting the data into tables is often referred to as shredding. Transact-SQL (T-SQL) makes XML shredding possible through the OPENXML statement. OPENXML creates rowset views on top of in-memory XML documents. As a rowset provider, OPENXML rowsets can be used in T-SQL statements wherever traditional rowsets may be called for. In other words, OPENXML can supply records to standard INSERT, UPDATE, or DELETE statements.

Before you can begin using OPENXML, you must have SQL Server load the XML document into memory. SQL Server provides the sp_xml_preparedocument stored procedure for doing this. It loads the supplied XML text into an internal DOM (using MSXML2.dll) and returns a handle that you can later supply to the OPENXML statement.

The handle returned by sp_xml_preparedocument is valid for the duration of the SQL Server connection or until you decide to remove it from memory by calling sp_xml_removedocument. SQL Server stores the loaded documents in its internal cache, but only allows MSXML to use one-eighth of the total memory available. Hence, it's important to call sp_xml_removedocument when you're finished working with the XML documents. The following stored procedure illustrates this process:

CREATE PROCEDURE InsertInvoice @xmlDoc text AS
DECLARE @iDoc int

-- load XML document into memory
EXEC sp_xml_preparedocument @iDoc OUTPUT, @xmlDoc

-- process XML document with OPENXML here

-- remove XML document from memory
EXEC sp_xml_removedocument @iDoc

GO

Once you have the handle to the loaded XML document, you can use OPENXML to define a rowset view of the data. This requires you to provide XPath expressions and rowset schema information that together specify the exact mapping details. For example, this XML document contains invoice information:

<Invoice ID='1000' CustomerName='Acme Computers'>
   <LineItems>
     <LineItem Sku='134' Description='Mouse' Price='9.95'
               Quantity='35'/>
     <LineItem Sku='153' Description='Keyboard' Price='19.95'
               Quantity='20'/>
     <LineItem Sku='171' Description='USB Disk' Price='14.95'
               Quantity='50'/>
   </LineItems>
</Invoice>

When using OPENXML, you must first identify which nodes in the XML document will map to rows in the new rowset. You specify this using an XPath expression that identifies an XML node-set. For example, if you want to map the LineItem elements to a rowset, the XPath would be /Invoice/LineItems/LineItem.

Then you provide the new rowset schema information in the WITH clause. OPENXML maps the columns specified in the WITH clause to the identified node's attributes by default (here, the Sku column maps to LineItem's Sku attribute, the Description column maps to LineItem's Description attribute, and so forth). For example, the stored procedure in Figure 1 automatically maps the LineItem attributes to a rowset using identical column names.

Figure 1 Stored Procedure (Close Figure 1)
CREATE PROCEDURE InsertInvoice @xmlDoc text
AS
DECLARE @iDoc int

EXEC sp_xml_preparedocument @iDoc OUTPUT, @xmlDoc
SELECT *
 FROM OPENXML(@iDoc, '/Invoice/LineItems/LineItem')
    WITH (
        Sku varchar(50),
        Description varchar(50),
        Price float,
        Quantity int
        )
EXEC sp_xml_removedocument @iDoc
GO

Executing this stored procedure by supplying the XML document shown earlier produces the following rowset:

Sku | Description | Price | Quantity
134 | Mouse | 9.95 | 35
153 | Keyboard | 19.95 | 20
171 | USB Disk | 14.95 | 50

You can also map the rowset to elements (instead of attributes) through a special flag. The third argument specifies the default mapping that should be used between the XML node-set and the relational rowset. The possible flag values and their descriptions are provided in Figure 2.

Figure 2 OPENXML Flags (Close Figure 2)

Byte Value | Meaning
0 | Defaults to attribute-centric mapping.
1 | Use the attribute-centric mapping. Can be combined with XML_ELEMENTS, in which case attribute-centric mapping is applied first, and then element-centric mapping is applied for all columns not yet dealt with.
2 | Use the element-centric mapping. Can be combined with XML_ATTRIBUTES, in which case attribute-centric mapping is applied first, and then element-centric mapping is applied for all columns not yet dealt with.
8 | Can be combined (logical OR) with XML_ATTRIBUTES or XML_ELEMENTS. In the context of retrieval, this flag indicates that the consumed data should not be copied to the overflow property @mp:xmltext.

For example, consider the following document that uses elements instead of attributes:

<Invoice>
   <InvoiceID>1000</InvoiceID>
   <CustomerName>Acme Computers</CustomerName>
   <LineItems>
      <LineItem>
         <Sku>134</Sku>
         <Description>Mouse</Description>
         <Price>9.95</Price>
         <Quantity>35</Quantity>
      </LineItem>
      ...
   </LineItems>
</Invoice>

The following call to OPENXML produces the same rowset as before, but for the document using elements instead of attributes:

-- 2 indicates an element-centric mapping
SELECT *
FROM OPENXML(@iDoc, '/Invoice/LineItems/LineItem', 2)
    WITH (Sku varchar(50),
          Description varchar(50),
          Price float,
          Quantity int)

In some situations, the default mapping mechanisms aren't sufficient. For example, they don't let you include the InvoiceID and CustomerName elements in the rowset of LineItem information. Hence, you can also specify an XPath expression that defines the exact mapping for each column. Relative XPath expressions are evaluated relative to the node identified by the first XPath expression supplied to OPENXML. The following example illustrates how to generate a rowset including the CustomerName and InvoiceID elements:

SELECT *
FROM OPENXML(@iDoc, '/Invoice/LineItems/LineItem', 2)
    WITH (CustomerName varchar(50) '../../CustomerName',
          InvoiceID int '../../InvoiceID',
          Sku varchar(50),
          Description varchar(50),
          Price float,
          Quantity int)

Executing this stored procedure against the element-centric document shown earlier produces the following rowset:

CustomerName | InvoiceID | Sku | Description | Price | Quantity
Acme Computers | 1000 | 134 | Mouse | 9.95 | 35
Acme Computers | 1000 | 153 | Keyboard | 19.95 | 20
Acme Computers | 1000 | 171 | USB Disk | 14.95 | 50

Finally, inserting these rows into a table with the same structure is as simple as supplying the selected rowset to an INSERT statement, as shown in Figure 3. There are a variety of ways to shred XML documents outside of the database, but OPENXML is the mechanism provided by T-SQL.

Figure 3 Insert into LineItems (Close Figure 3)
CREATE PROCEDURE InsertInvoice @xmlDoc text
AS
DECLARE @iDoc int

EXEC sp_xml_preparedocument @iDoc OUTPUT, @xmlDoc
INSERT INTO LineItems
SELECT *
 FROM OPENXML(@iDoc, '/Invoice/LineItems/LineItem', 2)
    WITH (Sku varchar(50),
          Description varchar(50),
          Price float,
          Quantity int,
          InvoiceID int '../../InvoiceID')
EXEC sp_xml_removedocument @iDoc
GO

Q Can you tell me how to process a namespace-qualified document using OPENXML?

A Although the examples I looked at in the previous question didn't contain any XML namespaces, OPENXML is also capable of shredding namespace-qualified documents. To enable this, you must first supply a set of namespace bindings to sp_xml_preparedocument. These bindings should contain the namespace prefixes that you intend to use in the OPENXML statement. Then you simply use those prefixes to identify namespace-qualified elements/attributes found in the XML document. For example, consider the following invoice document that contains two namespaces:

<Invoice xmlns="urn:invoices">
   <InvoiceID>1000</InvoiceID>
   <CustomerName>Acme Computers</CustomerName>
   <LineItems xmlns="urn:items">
      <LineItem>
         <Sku>134</Sku>
         <Description>Mouse</Description>
         <Price>9.95</Price>
         <Quantity>35</Quantity>
      </LineItem>
      ...
   </LineItems>
</Invoice>

To process this document properly now that it contains namespaces, you need to supply a string of XML containing namespace declarations as the third parameter to sp_xml_preparedocument, as shown here:

EXEC sp_xml_preparedocument @iDoc OUTPUT, @xmlDoc,
  '<ns xmlns:inv="urn:invoices" xmlns:li="urn:items"/>'

The string must contain a well-formed XML document, but the name of the root element doesn't matter (in this case, I called it ns). The namespace declarations on the element are the important part. With the prefixes defined (in this case, "inv" and "li"), you can start using them in the XPath expressions that constitute the OPENXML statement, as shown in Figure 4.

Figure 4 Namespace Declarations (Close Figure 4)
CREATE PROCEDURE InsertInvoice @xmlDoc text
AS
DECLARE @iDoc int

EXEC sp_xml_preparedocument @iDoc OUTPUT, @xmlDoc,
  '<ns xmlns:inv="urn:invoices" xmlns:li="urn:items"/>'

INSERT INTO LineItems
SELECT *
 FROM OPENXML(@iDoc,
   '/inv:Invoice/li:LineItems/li:LineItem', 2)
    WITH (Sku varchar(50) 'li:Sku',
          Description varchar(50) 'li:Description',
          Price float 'li:Price',
          Quantity int 'li:Quantity',
          InvoiceID int '../../inv:InvoiceID')
EXEC sp_xml_removedocument @iDoc
GO

Once you start working with namespace-qualified documents, the default element/attribute mappings will no longer work like they did in the earlier OPENXML sample since they assume no namespaces. You must specify the namespace-qualified name for each element/attribute in the WITH clause, as I just illustrated.

Q Is there any way to save the unprocessed portions of an XML document using OPENXML?

A SQL Server exposes several properties on each node in the XML document that you can access through a well-known namespace (urn:schemas-microsoft-com:xml-metaprop). These properties give you access to a given node's ID, local name stream, namespace URI, namespace prefix, the entire XML serialization, and information about parent and sibling nodes. SQL Server automatically binds the namespace to the "mp" prefix so you can use these metaproperties without any further configuration (as was necessary in the example in the previous question). Figure 5 lists the names of these properties along with their descriptions.

Figure 5 SQL Server 2000 Metaproperties (Close Figure 5)

Metaproperty Attribute | Description
@mp:id | Provides system-generated, document-wide identifier of the DOM node (element, attribute, and so on). This ID is guaranteed to refer to the same XML node as long as the document is not reparsed. An XML ID of 0 indicates that the element is a root element. Its parent XML ID is NULL.
@mp:localname | Stores the local part of the name of the node. It is used with prefix and namespace Uniform Resource Identifier (URI) to name element or attribute nodes.
@mp:namespaceuri | Provides the namespace URI of the current element. If the value of this attribute is NULL, no namespace is present.
@mp:prefix | Stores the namespace prefix of the current element name. If no prefix is present (NULL) and a URI is given, this indicates that the specified namespace is the default namespace. If no URI is given, no namespace is attached.
@mp:prev | Stores the previous sibling relative to a node, thereby providing information about the ordering of elements in the document. @mp:prev contains the XML ID of the previous sibling that has the same parent element. If an element is at the beginning of the sibling list, @mp:prev is NULL.
@mp:xmltext | This metaproperty is used for processing purposes. It's the textual serialization of the element and its attributes and subelements as used in the overflow handling of OPENXML.
@mp:parentid | Corresponds to ../@mp:id.
@mp:parentlocalname | Corresponds to ../@mp:localname.
@mp:parentnamespacerui | Corresponds to ../@mp:namespaceuri.
@mp:parentprefix | Corresponds to ../@mp:prefix.

The @mp:xmltext property provides the textual serialization of the node in question along with any attributes and descendant nodes. This metaproperty can be used for overflow handling. In fact, it can be used to capture the entire XML structure of the node or just the unprocessed portions. The following OPENXML statement illustrates how to provide an overflow column by mapping it to the @mp:xmltext property:

OPENXML(@iDoc,
   '/inv:Invoice/li:LineItems/li:LineItem', 2)
    WITH (Sku varchar(50) 'li:Sku', Description varchar(50)
          'li:Description', Extra ntext '@mp:xmltext')
In this example, the entire serialized LineItem element will show up in the Extra column for each record. If you only want the unprocessed portions of each LineItem to show up, you should change the flag parameter from 2 to 10 (refer to Figure 2 for more details), as illustrated here:
OPENXML(@iDoc,
   '/inv:Invoice/li:LineItems/li:LineItem', 10)
    WITH (Sku varchar(50) 'li:Sku', Description varchar(50)
          'li:Description', Extra ntext '@mp:xmltext')
With this in place, the Extra column will contain a string of XML that includes the Price element since it was the only node that wasn't mapped by the WITH clause.

The metaproperties listed in Figure 5 also appear when using OPENXML without a WITH clause, as illustrated here:

SELECT *
FROM OPENXML(@iDoc, '/inv:Invoice/li:LineItems/li:LineItem', 2)

This produces what's called an edge table (see Figure 6), which contains a row for each node in the subtree as identified in the XPath expression. The rows contain columns for most of the metaproperties listed in Figure 5. This provides the information to reconstruct the original XML document from the rowset if necessary.

Figure 6 Edge Table for Invoice Document (Close Figure 6)

id | parentid | nodetype | localname | prefix | namespaceuri | datatype | prev | text
7 | 5 | 1 | LineItem | NULL | urn:items | NULL | NULL | NULL
8 | 7 | 1 | Sku | NULL | urn:items | NULL | NULL | NULL
22 | 8 | 3 | #text | NULL | NULL | NULL | NULL | 134
9 | 7 | 1 | Description | NULL | urn:items | NULL | 8 | NULL
23 | 9 | 3 | #text | NULL | NULL | NULL | NULL | Mouse
10 | 7 | 1 | Price | NULL | urn:items | NULL | 9 | NULL
24 | 10 | 3 | #text | NULL | NULL | NULL | NULL | 9.95
11 | 7 | 1 | Quantity | NULL | urn:items | NULL | 10 | NULL
25 | 11 | 3 | #text | NULL | NULL | NULL | NULL | 35
12 | 5 | 1 | LineItem | NULL | urn:items | NULL | 7 | NULL
... | ... | ... | ... | ... | ... | ... | ... | ...

Q How do keys work in XSLT?

A Keys in XSLT are similar to IDs in Document Type Definitions (DTDs). If you define an attribute of type ID in a DTD, the attribute value must be unique in the document. You can also define attributes of type IDREF. In this case, the attribute value must refer to a unique ID value within the document. With this information, you can then use XPath's id function to resolve IDs and IDREF relationships in a document at run time. For example, consider an XML document with an internal DTD (see Figure 7).

Figure 7 Internal DTD (Close Figure 7)
<!DOCTYPE courses [
  <!ELEMENT courses (course*, employees)>
  <!ELEMENT course EMPTY>
  <!ELEMENT employees (employee*)>
  <!ELEMENT employee EMPTY>
  <!ATTLIST course
            instructors IDREF #REQUIRED
            name CDATA #REQUIRED
  >
  <!ATTLIST employee
            id ID #REQUIRED
            name CDATA #REQUIRED
  >
]>
<courses>
  <course instructors="e101 e102 e103 e104" name="EWS.NET"/>
  <course instructors="e101 e102 e105" name="GWS.NET"/>
  <employees>
    <employee id='e101' name="Aaron Skonnard"/>
    <employee id='e102' name="Simon Horrell"/>
    <employee id='e103' name="Dan Sullivan"/>
    <employee id='e104' name="Scott Bloom"/>
    <employee id='e105' name="Bob Beauchemin"/>
  </employees>
</courses>

The following XPath expression identifies the element that has the ID of e104 which, in this case, happens to be the employee element that has the name of Scott Bloom:

id('e104')
The following XPath expression identifies the elements with id values of e101, e102, and e104 which, in this case, happens to be the employee elements with names of Aaron Skonnard, Dan Sullivan, and Scott Bloom:
id('e101 e103 e104')
And finally, the following XPath expression identifies the employees who teach the "GWS.NET" course (specifically Aaron Skonnard, Simon Horrell, and Bob Beauchemin) through the instructors IDREF attribute attached to them:
id(/courses/course[2]/@instructors)

As you can see, this technique lets you establish relationships between different elements based on ID values. The problem with the ID/IDREF technique is that it requires DTD information to be present at run time and it's limited to attributes. To help alleviate these restrictions, XSLT introduced a new concept, called keys, which provides the same basic functionality but with more flexibility. To define a key in an XSLT document, you use the key element, which has the following structure:

<xsl:key name="qname" match="pattern" use="expression"/>
You give the key a qualified name so that you can refer to it later from the key function (which behaves like the XPath id function described earlier). You also specify a pattern, which identifies the nodes to which the key applies. And finally, you rely on the "use" attribute to specify an XPath expression that is evaluated (relative to the nodes identified by the match expression) to produce the actual key value. For example, consider the XML document containing courses and employees shown in Figure 8.

Figure 8 An XML Document (Close Figure 8)
<courses>
  <course>
    <name>EWS.NET</name>
    <instructors>
      <instructor>e101</instructor>
    </instructors>
  </course>
  <course>
    <name>GWS.NET</name>
    <instructors>
      <instructor>e101</instructor>
      <instructor>e102</instructor>
      <instructor>e103</instructor>
    </instructors>
  </course>
  <employees>
    <employee id='e101'>
      <name>Aaron Skonnard</name>
    </employee>
    <employee id='e102'>
      <name>Simon Horrell</name>
    </employee>
    <employee id='e103'>
      <name>Dan Sullivan</name>
    </employee>
  </employees>
</courses>

In this case, you probably want to create a key based on the employee element's id attribute, as illustrated here:

<xsl:key name="employeeId" match="employee" use="@id"/>
This creates a key for all employee elements within the document; the key value is evaluated from the employee element's id attribute. With the key defined, you can then use the key function to retrieve nodes with a given key. For example, the following XSLT fragment illustrates how to retrieve the instructors who teach a given course:
<xsl:template match="course">
  <h2><xsl:value-of select="name"/></h2>
  <h3>Instructors</h3>
  <ul>
   <xsl:for-each select="key('employeeId', ./instructors/*)">
     <li><xsl:value-of select="."/></li>
   </xsl:for-each>
  </ul>
</xsl:template>

That fragment will emit the following HTML, assuming the input document shown in Figure 8:

<h2>EWS.NET</h2>
<h3>Instructors</h3>
<ul>
   <li>Aaron Skonnard </li>
</ul>
<h2>GWS.NET</h2>
<h3>Instructors</h3>
<ul>
   <li>Aaron Skonnard</li>
   <li>Simon Horrell</li>
   <li>Dan Sullivan</li>
</ul>
As you can see, keys don't require DTD information to be used and they're not limited to attributes as with ID/IDREF. The fact that keys can be assigned to nodes that match an arbitrary pattern also gives you an extra degree of flexibility.

Q What's the difference between the match and select attributes in XSLT? What do they do, exactly?

A In XSLT, there are various elements that come with a match attribute (such as template and key) and others that come with a select attribute (value-of, for-each, apply-templates, and so on). At first glance, both seem to take XPath expressions, but in fact they're quite different.

Select indeed expects an XPath expression, which is used to select a nodeset for further processing. In the following example, the for-each expression identifies a set of nodes to be iterated over and the nested value-of expression writes the value of each node to the output tree:

<xsl:for-each select="//item">
  <xsl:value-of select="."/>
</xsl:for-each>

The match attribute, on the other hand, takes what's called a pattern. A pattern looks like an XPath expression because it shares the same syntax, but it's treated differently by the XSLT processor. A pattern is used for matching nodes in the tree against the specified criteria. For example, the following template matches any element node that has a grandparent named foo:

<xsl:template match="foo/*/*">
   •••
</xsl:template>
In other words, a pattern describes an "is a" relationship with the nodes in question (that is, is this node an element that has a grandparent named foo?). Here's another slightly more sophisticated example of a pattern:
<xsl:template match="f:foo[@id > 323]//text()">
   •••
</xsl:template>
In this case, the pattern identifies all text nodes that have an ancestor element named foo, from the namespace identified by f, and with an id attribute with a numeric value greater than 323.

Q Why does SelectNodes always return the nodes in document order irrespective of the axis direction (forward/reverse) that was used in the query?

A Consider the following XML document, which I'll call foo.xml:

<foo depth='1'>
  <foo depth='2'>
    <foo depth='3'>
      <bar/>
    </foo>
  </foo>
</foo>

Now consider the following code that evaluates slightly different XPath expressions against the loaded foo.xml document:

XmlDocument doc = new XmlDocument();
doc.Load("foo.xml");

XmlElement b, f1, f2;

b =  (XmlElement)doc.SelectSingleNode("//bar");
f1 = (XmlElement)b.SelectSingleNode("ancestor::foo[1]");
f2 = (XmlElement)b.SelectNodes("ancestor::foo")[0];

Console.WriteLine(f1.GetAttribute("depth"));
Console.WriteLine(f2.GetAttribute("depth"));

You might expect the console output to be the same in both cases, but it's actually different. The first Console.WriteLine outputs "3" while the second outputs "1". The reason it works this way is that axis direction, as defined by the XPath specification, is only significant within XPath predicates. In fact, XPath nodesets are by definition unordered. It's up to the XPath implementation to determine in what order to deliver nodesets.

In the call to SelectSingleNode, however, I asked for the first node in the ancestor axis within a predicate. This guarantees that I'll get the first ancestor node (in reverse document order) starting from the bar element (the foo element with depth='3'). In the call to SelectNodes, on the other hand, I asked for all ancestor foo elements and the implementation returned an XmlNodeList collection in document order (which happens to be completely legal, according to the specification). Then, my code asked for the first item in the collection, which happens to be the foo element where depth='1'. Since XPath doesn't specify any particular order in which nodes are required to be returned, languages that build on XPath often define additional semantics of their own to take care of ordering. For example, in the case of XSLT, nodesets are always processed in document order.

Q How do you query a paragraph of text to determine whether it contains a <SCRIPT> element?

A The answer to this question depends on what you mean when you ask if the paragraph contains a <SCRIPT> element. Consider the following document:

<item>
   <name>Something</name>
   <desc>This is a description that contains a <SCRIPT
     language="Javascript" src="script.js"/> element.</desc>
</item>
If the document looks like this, you would use a normal XPath expression that queries against the document's logical data model. For example, the following XPath expression identifies all element nodes in the document tree that contain a child SCRIPT element (in this case, the <desc> element):
//*[SCRIPT]

Now, consider the following document, which uses special entity references for the less-than "<" and greater-than ">" characters within the <desc> element:

<item>
   <name>Something</name>
   <desc>This is a description that contains a &lt;SCRIPT
      language="Javascript" src="script.js"/&gt; element.</desc>
</item>
When an XML processor parses this document, it replaces the "&lt;" string with the < character and the "&gt;" string with the > character. Hence, the <desc> element contains the following text when you query it using XPath:
This is a description that contains a <SCRIPT language="Javascript"
src="script.js"/> element.

So you would use the following XPath expression to search this document for all elements that contain "<SCRIPT":

//*[contains(.,'<SCRIPT')]

Q I have two documents that only differ in namespace prefixes:

<person xmlns:a="urn:names">
  <a:name>Bob</a:name>
</person>

<person xmlns:b="urn:names">
  <b:name>Sally</b:name>
</person>

It seems that most DOM APIs require the prefix when performing getElementsByTagName queries. How can I get a value for the name element without worrying about its prefix?

A When the DOM Level 1 specification was first published, it didn't include support for namespaces. Hence, the original getElementsByTagName method expected a simple tag name and was oblivious to namespaces. Some Level 1 implementations made it possible to process namespace-qualified documents using prefixed names, but you couldn't expect two implementations to produce the same results since the behavior wasn't defined by the spec.

The DOM Level 2 specification (http://www.w3.org/DOM/DOMTR#dom2) introduced full support for namespaces by adding new methods for every operation dealing with element or attribute names (where namespaces are possible). The specification outlined that the new versions of these methods should all end in "NS". For example, in DOM Level 2 there is also a getElementsByTagNameNS method. This method expects a local name (with no prefix) and the namespace name.

Although this is what the specification says, not all implementations conform completely today. For example, the Microsoft® .NET Framework implementation provides an overloaded version of GetElementsByTagName instead of a completely separate method with a distinct name. Here's how you'd select the name element in C#:

XmlNodeList nl = doc.GetElementsByTagName("name", "urn:names");
string name = nl.Item(0).Value;
This code will select the name element from the urn:names namespace regardless of the prefix used in the source document.

The MSXML 4.0 DOM implementation doesn't include the NS methods either, but it does provide the functionality you need through other methods. For example, one way to select a node regardless of its prefix is through an XPath expression. Both the MSXML 4.0 and .NET Framework implementations support XPath through the SelectNodes and SelectSingleNode functions. The following JScript® illustrates how to select the name element based on local name alone, without regard to namespace or prefix:

name = doc.SelectSingleNode("/person/*[local-name()='name']").text;

Another way to use these functions with namespace-aware documents is to configure a separate namespace prefix that you intend to use in your XPath expressions. With MSXML 4.0, you do this through the call to setProperty, as illustrated here:

doc.setProperty("SelectionNamespaces", "xmlns:x='urn:names'");
name = doc.SelectSingleNode("/person/x:name").text;

The namespace prefix you use in the XPath expression doesn't have to be the same as the one used in the source document. In fact, it doesn't matter what prefix you use as long as it's mapped to the same namespace name as is used in the document. As you can see, there are a variety of ways to accomplish what you want depending on what's offered by your implementation.

Q I've always been confused about how xsl:decimal-format and format-number work in XSLT. Can you explain?

A XSLT provides a built-in mechanism for formatting numbers. This mechanism is built on the format-number function and the xsl:decimal-format element. The format-number function has the following signature:

string format-number(number, string, string?)
The first argument is the number you want to format while the second argument is a string representing the exact format you want to use. The syntax of the format string is defined by the JDK 1.1 DecimalFormat class. You can use any of the characters shown in Figure 9 to build a format string.

Figure 9 Format String Characters (Close Figure 9)

Character | Meaning
0 | A digit; zeros show as zeros
# | A digit; zeros show as absent
. | Placeholder for decimal separator (locale-specific)
, | Placeholder for grouping separator (locale-specific)
- | Default negative prefix
% | Multiply by 100 and show as a percentage
; | Separates number formats (positive format on left, optional negative number format on right)
' | Used to quote special characters in a prefix or suffix
other | Appears literally in output

For example, the XSLT transformation in Figure 10 illustrates how to use the format-number function with a few different format strings. Calling the printNumbers template and passing in the value 9950001 produces the following results:

$9,950,001
$9,950,001.00
$9,950,001.00

Figure 10 Format Number Function (Close Figure 10)
<xsl:template name="printNumbers">
    <xsl:param name="num"/>
    <xsl:value-of select="format-number($num, '$#,###.##')"/>
    <xsl:value-of select="format-number($num, '$#,##0.00')"/>
    <xsl:value-of select="format-number($num,
                         '$#,##0.00;&lt;$#,##0.00&gt;')"/>
</xsl:template>

<xsl:template match="/">
    <xsl:call-template name="printNumbers">
        <xsl:with-param name="num" select="9950001"/>
    </xsl:call-template>
</xsl:template>

The last format string has two parts: one part for positive numbers and another part for negative numbers (separated by a semicolon). When you supply a positive number to format-number, it uses the first format string; otherwise it uses the second format string. For example, calling printNumbers with a value of -9950001 produces the following results:

-$9,950,001
-$9,950,001.00
<$9,950,001.00>

If you don't specify a negative format string (like in the first two calls to format-number) a minus sign is automatically placed in front of the formatted number, as shown in the previous output.

The optional third argument is the name of a decimal-format element that influences how the format string is interpreted by the XSLT processor. If there isn't a third argument, the function uses the default decimal-format for the transformation (which follows the default format string rules).

Here's the syntax of the decimal format element:

<xsl:decimal-format
  name = qname
  decimal-separator = char
  grouping-separator = char
  infinity = string
  minus-sign = char
  NaN = string
  percent = char
  per-mille = char
  zero-digit = char
  digit = char
  pattern-separator = char />
As you can see, xsl:decimal-format lets you control exactly what the characters mean in a given format string. Figure 11 describes what each of the attributes means along with its default value. You name the decimal-format so that you can refer to it later while calling the format-number function. If you don't give it a name, it automatically becomes the default decimal format for the entire transformation.

Figure 11 xsl:decimal-format Attributes (Close Figure 11)

Name | Default | Description
name | "" | The qualified name of the decimal-format; if there's no name, it becomes the default decimal-format
decimal-separator | . | The character used for the decimal sign
grouping-separator | , | The character used as the grouping separator
infinity | Infinity | The string used to represent infinity
minus-sign | - | The character used as the minus sign
NaN | NaN | The string used to represent the NaN value
percent | % | The character used as the percent sign
per-mille | â€° | The character used as a per mille sign
zero-digit | 0 | The character used as the digit zero
digit | # | The character used for a digit in the format string
pattern-separator | ; | The character used to separate positive and negative sub patterns in a pattern

For example, consider the following xsl:decimal-format element that switches the decimal and grouping separator characters:

<xsl:decimal-format decimal-separator = ","
     grouping-separator = "."/>
Inserting this into the XSLT transformation in Figure 10 and calling the printNumbers template with -9950001 now produces the following output:
-$9.950.001
-$9.950.001,00
<$9.950.001,00>
Notice that since I didn't give the xsl:decimal-format element a name, it automatically became the default for the transformation and was used in the calls to format-number. The functionality that results from the coordination between format-number and xsl:decimal-format offers a flexible mechanism for formatting numbers throughout your transformations.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard is an instructor and researcher at DevelopMentor, where he develops the XML and Web Services-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.


Advanced Basics: Enterprise Services, SQL Script Editing

Advanced Basics
Enterprise Services, SQL Script Editing
Ken Spencer

Code download available at: AdvancedBasics0310.exe (150KB)

Q I have lots of experience developing ASP applications that use Visual J++® components wrapped in COM running in Microsoft® Transaction Services (MTS). I am currently developing an application with the Microsoft .NET Framework and have written several classes that perform the bulk of my data access. I then use these classes in my ASPX pages. This application will have as many as 10,000 concurrent users.

When I create component assemblies do I need to use MTS and COM+ at all or does the common language runtime (CLR) take care of these classes? COM+ and MTS handle the transactions and object pooling, but can I really use components without COM+ and MTS?

A When you create an application that uses classes, should you put those assemblies in MTS or COM+? First let's define a few things for clarity. MTS provided services to COM components on Windows NT® 4.0. In Windows® 2000, MTS and many other services were exposed as COM+ 1.0 components. With the advent of the CLR, the same services were exposed to .NET Framework components via System.EnterpriseServices. The purpose of COM+ and System.EnterpriseServices is to provide services to COM components and .NET Framework components. These services include transactions, the QueueComponent, RoleBased Security, Object Pooling, Hosting, Availability, packaging, and a whole slew of others. More information can be found at COM+ Services.

Now, back to the general question. Should you use Enterprise Services at all to host your assemblies? This is a great question which, at first, seems quite simple. If you need any of the aforementioned services, then you need to use System.EnterpriseServices.

For all of the code that just reads information from a database or handles business rules and does not do any updates, you probably don't need to avail yourself of Enterprise Services at all. If you want to take advantage of things like object pooling, which Enterprise Services offers, you can use it for those features. However, you would need to carefully test your application's performance to see if it runs faster in Enterprise Services.

Second, you could use Enterprise Services to handle your database updates to take advantage of its transactional features. The question is, do you really need it? If I am handling transactions across two or more databases in a single transaction, then my rule of thumb is to use Enterprise Services for those classes. If the application only updates a single database, I usually do not use Enterprise Services, since ADO.NET handles this nicely with SQL Server™ and other databases, as long as the provider supports it. The Data Access Application Block from Microsoft (see Microsoft Application Blocks for .NET) supports transactions without using Enterprise Services. This makes it easy to handle transactions without using Enterprise Services . However, if you do plan to use multiple databases or use other transacted resources (such as MSMQ) in your application, then Enterprise Services provides an easy way to manage distributed transactions.

Another consideration is the ServicedComponent class in the .NET Framework. To use Enterprise Services, you must derive your component from this class. The following short sample from the MSDN® docs shows how simple this is to set up and how attributes can be applied to your class:

Imports System.EnterpriseServices
<Transaction(TransactionOption.Required)> Public Class Account
Inherits ServicedComponent
   <AutoComplete()> Shared Sub Main()
   End Sub
End Class
This code is nice and simple, but if you designed your application with a class hierarchy containing multiple levels of inheritance, this presents a potential problem. In this case, your base class must derive from ServicedComponent.

System.EnterpriseService is a managed namespace of these services and is provided by COM+, so if you are in the managed world and want to use these services, you would use System.EnterpriseServices to get at these services and/or express them as attributes like any other class in the .NET Framework.

Q I'm just learning to use Visual Studio® .NET 2003 and I need to execute a SQL statement that exists in a file without creating a solution/project/database reference. I want to perform file-based SQL script editing without being persistently connected to a SQL Server (like Query Analyzer), but still want to have the ability to execute statements when needed. I know there are third-party tools that do this, but does Microsoft have one?

A You don't need to buy a tool in order to execute SQL statements. Of course you can use Query Analyzer, as you mentioned. You can also use the third-party tool ISQL from the command line, but you must install it first. Then there is Visual Studio .NET, but that requires opening the IDE, creating a project, and then opening Query Builder. For a quick and easy solution, I created a simple SQL tool which you can modify.

Let's start with a look at the interface for the tool (see Figure 1). It's a Windows Form MDI application. As you can see from the interface, the tool executes queries in a manner similar to Query Analyzer. You can type in a query, select a connection string from the list, and then click Execute. The results show up in the grid below the query. Since it is an MDI application, you can open many windows with different queries.

Figure 1 Simple SQL Tool
Figure 1 Simple SQL Tool

As you requested, the tool only connects to the database to execute the query; the connection is then closed. This is because the tool uses a DataSet and data adapter to perform the work.

Now, let's walk through the tool. First, I created a new Windows-based application in Visual Studio .NET. Then I set the isMDIContainer property to True. This makes the first form the MDI parent. I then added a MainMenu and set the top-level menus to the following:

Name | Text
mnuitmWindow | &Window
mnuitmExit | &Exit
mnuitmNew | &New Window
mnuitmOptions | &Options

To mnuitmOptions I added a submenu named mnuitmConnections with a Text property of "Connections". The mnuitmWindow menu has the MDIList property set to True.

Next, I added a second Windows Form to the project and named it frmSQLWindow. The two buttons on the left are named cmdOpen and cmdSave, respectively, and use icons that are included with Visual Studio .NET. The Execute button is named cmdExecute. The combo list is named cboConnection and holds the list of connections that the tool uses.

The DataSet radio button is named rdoDataSet and has the Checked property set to True. The No Return radio button is named rdoExecuteOnly. The Textbox control showing the number 1 is named txtIterate. The control in the lower half of the form is a DataGrid named dgrdOutput, and the Dock property is set to Bottom. The Textbox above it is named txtSQL and its Dock property is also set to Bottom.

Now, let's wire up this tool. Add the code shown in Figure 2 and Figure 3 to the project. These two modules handle the data access and the loading of the data connections from an XML file. Open Form1 and set the Click event for the Exit menu to Application.Exit. Set the code for mnuitmNew to:

Dim frmSQL As New frmSQLWindow
frmSQL.MdiParent = Me
frmSQL.Show()
This will create a new child form each time you click the menu.

Figure 3 Load Data Connections (Close Figure 3)
Imports System.Data.SqlClient
Module modDB
    Friend frnConnectionString As String
    Private pvtDataSet As DataSet
    Private pvtConnection As SqlConnection
    Private pvtCommand As SqlCommand
    Private pvtSQLDataAdapter As SqlDataAdapter
    Friend WithEvents frnCurrentConnection As DataSet
    Friend Function runSQLDataSet(ByVal SQL As String) _
    As DataSet
        Try
      pvtConnection = New _
          SqlConnection(frnConnectionString)
            pvtCommand = New SqlCommand(SQL, pvtConnection)

            pvtDataSet = New DataSet

            pvtSQLDataAdapter = New SqlDataAdapter(pvtCommand)

            pvtSQLDataAdapter.Fill(pvtDataSet)

            Return pvtDataSet

        Catch ExceptionObject As Exception
            Throw New Exception(ExceptionObject.Message)
        Finally
            pvtConnection.Close()
        End Try

    End Function

    Friend Sub runSQL(ByVal SQL As String)
        Try
            pvtConnection = New _
                SqlConnection(frnConnectionString)
            pvtCommand = New SqlCommand(SQL, pvtConnection)

            pvtCommand.ExecuteNonQuery()

        Catch ExceptionObject As Exception
            Throw New Exception(ExceptionObject.Message)
        Finally
            pvtConnection.Close()
        End Try
    End Sub
End Module
Figure 2 Data Access (Close Figure 2)
Imports System.IO
Module modGeneral
    Friend sFile As String

    Friend Function LoadConnectionstrings() As DataSet
        Dim ds As New DataSet

        sFile = Application.ExecutablePath()
        sFile = Replace(sFile, "\SQLExecutor.exe", "")
        sFile &= "\ExecuteSQLConnections.xml"

        If File.Exists(sFile) Then
            ds.ReadXml(sFile)
        End If

        Return ds
    End Function
End Module

Now add this code to Form_Load:

frnCurrentConnection = New DataSet
frnCurrentConnection = LoadConnectionstrings()
Next, add the following code to mnuitmConnections_Click. This will fire up frmConnections, which allows you to maintain the connection list:
Dim frm As New frmConnections
If frm.ShowDialog() = DialogResult.OK Then
    frnCurrentConnection.Clear()
    frnCurrentConnection = LoadConnectionstrings()
End If

Open frmSQLWindow. Add the code in Figure 4 to cmdOpen_ Click to allow you to open a file and load it into the texbox. Then add the following code to cmdSave to save the SQL query:

Dim lclwriter As StreamWriter
sveSQLFile.InitialDirectory = "c:\sql"
sveSQLFile.Filter = "SQL files (*.sql)|*.txt|All files (*.*)|*.*"
sveSQLFile.FilterIndex = 2
sveSQLFile.RestoreDirectory = True
If sveSQLFile.ShowDialog = DialogResult.OK Then
lclwriter = New StreamWriter(sveSQLFile.FileName)
lclwriter.WriteLine(txtSQL.Text)
lclwriter.Close()
End If

Figure 4 Open and Load File (Close Figure 4)
Dim lclStream As StreamReader
opnSQLFile.InitialDirectory = "c:\sql"
opnSQLFile.Filter = "SQL files (*.sql)|*.txt|All files
                    (*.*)|*.*"
opnSQLFile.FilterIndex = 2
opnSQLFile.RestoreDirectory = True

If opnSQLFile.ShowDialog() = DialogResult.OK Then
    lclStream = New StreamReader(opnSQLFile.FileName)
    If Not (lclStream Is Nothing) Then
        lclStream.BaseStream.Seek(0, SeekOrigin.Begin)
        txtSQL.Text = lclStream.ReadToEnd
        lclStream.Close()
    End If
End If

Now add the code from Figure 5 to cmdExecute. This code executes the SQL query you enter. Notice that you can iterate the SQL statement by changing the iterate value to a number greater than 1. This allows you to perform some testing of your SQL code with SQL Profiler.

Figure 5 Executing a SQL Query (Close Figure 5)
Dim ds As DataSet
Dim i, iIterate As Integer
If IsNumeric(txtIterate.Text) Then
    iIterate = CInt(txtIterate.Text)
Else
    iIterate = 1
End If
If rdoDataSet.Checked Then
    frnConnectionString = cboConnection.Text
    For i = 1 To iIterate
        ds = runSQLDataSet(txtSQL.Text)
    Next
    If Not IsNothing(ds) Then
        dgrdOutput.DataSource = ds.Tables(0)
    End If
Else
    frnConnectionString = cboConnection.Text
    For i = 1 To iIterate
        ds = runSQLDataSet(txtSQL.Text)
    Next
End If

The only trick in frmConnections involves the code to save the connection strings in the list. Here is the code from cmdOK:

Dim ds As New DataSet
ds.Merge(CType(dgrdConnections.DataSource, DataTable))
ds.WriteXml(sFile)
Me.DialogResult = DialogResult.OK
Me.Close()
The line that merges the DataSet has to use DataTable as the new type instead of DataSet, as suggested in the docs. This is because the application uses the DataTable, not the DataSet as the input into the DataSource.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencer works for 32X Tech (http://www.32X.com), where he provides training, software development, and consulting services on Microsoft technologies.


Basic Instincts: Deploying Assemblies

Basic Instincts
Deploying Assemblies
Ted Pattison

Contents
Private Assemblies
The Global Assembly Cache
Deploying Assemblies with a Configured CodeBase
The Download Cache
Assembly Loading
Conclusion

This month's installment of Basic Instincts is the second in a series focused on working with assemblies. In the August 2003 issue I discussed the four parts of an assembly name. I also covered the build mechanics for building an assembly with a version number and a strong name. In this month's column I'll explore the options for deploying assembly DLLs in both development and production environments.

The four-part name of an assembly is independent of location. That is, there is nothing about the name of an assembly DLL that tells the common language runtime (CLR) or a hosting application anything about the location of the actual assembly file. This provides a valuable degree of flexibility because an assembly DLL can be deployed in several different ways and in many different places. It also means that the CLR must use something other than the assembly name to locate it at run time.

There are three primary ways to deploy an assembly on a target machine. The first technique involves deploying a DLL as a private assembly by locating it in the ApplicationBase directory. The second technique involves installing it in a machine-wide repository called the Global Assembly Cache (GAC). The third technique involves configuring an assembly DLL with a <codeBase> element that allows the CLR to download the DLL on demand from across the network the first time it's used by a hosting application.

Private Assemblies

Deploying a DLL as a private assembly is the simplest approach you can take. You deploy the DLL inside the ApplicationBase directory of the hosting application or inside a subdirectory of the ApplicationBase directory if the PrivateBinPath property is set. In most cases, it's really as easy as that.

One of the biggest advantages of private assembly deployment is that it allows for XCOPY deployment. That's because an application, its configuration file, and all of its private assemblies are contained within a single directory structure. Once your application and its private assemblies have been thoroughly tested, you can deploy the application as a whole by simply copying the ApplicationBase directory structure to the target machine using a utility such as XCOPY.EXE or a network transfer protocol such as FTP. You can even deploy an application by simply using drag and drop in Windows® Explorer. Once you have copied the ApplicationBase directory structure, the application is ready for use (assuming the common language runtime is present).

One noteworthy limitation of private assembly deployment is that it can never be deployed outside the ApplicationBase directory. You cannot share a private assembly across two or more applications that you have deployed in separate directories.

So how does the CLR find and load private assemblies? Well, the CLR discovers the physical path at run time through a search process known as probing. When the CLR starts probing for a private assembly, it determines the file name of the assembly by taking its friendly name and adding an extension of .dll.

Once the CLR determines the name of the target assembly file, it searches inside the ApplicationBase directory to see if it's there after checking the GAC for it. If it is, probing stops and the CLR loads the assembly into memory. Otherwise the probing process continues in a subdirectory of the ApplicationBase directory that has the same name as the assembly itself. For example, if the CLR were probing for an assembly named MyLibrary, it would look to see if there was a subdirectory named MyLibrary in the ApplicationBase directory that contained the assembly file MyLibrary.dll.

If the CLR finds the assembly file in this subdirectory, the assembly is loaded. If not, the CLR continues probing in the same two directories for an assembly file with an .exe extension instead of a .dll extension. That means the CLR automatically looks at four different paths while probing for a private assembly. If an application were to have an ApplicationBase directory with a path of C:\MyApp, the CLR would automatically probe for the assembly file using the following four file paths:

c:\MyApp\MyLibrary.dll
c:\MyApp\MyLibrary\MyLibrary.dll
c:\MyApp\MyLibrary.exe
c:\MyApp\MyLibrary\MyLibrary.exe
So, the CLR will automatically inspect four file paths when probing for an assembly that is culture neutral. When the CLR is probing for a satellite assembly with a cultural identifier, it will additionally look in subdirectories that have the same name as the cultural identifier itself. This extra support makes it easier to deploy multiple resource-only assemblies that have been localized for different languages.

Note that although you have the flexibility to deploy a private assembly in any subdirectory inside the ApplicationBase directory, if you choose one with a name other than the ones previously outlined, the CLR will require extra configuration information to assist it in probing. You must add a special <probing> element to the application configuration file to give the CLR a hint.

Let's say you wanted to create a subdirectory named MyAssemblies inside the ApplicationBase directory and then you wanted to deploy some of your dependent assemblies inside of it. The application would not be able to load these private assemblies until you modified your application configuration file to include the probing element, as shown here:

<configuration>
  <runtime>
     <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="MyAssemblies"/>
    </assemblyBinding>
  </runtime>
</configuration>

As you can see, the <probing> element contains a privatePath attribute that tells the CLR where to look. If you want to add more than one subdirectory to the private path, you can concatenate them together using a semicolon-delimited string.

You should also remember that the CLR looks through a predetermined sequence of directories during probing. Given a privatePath value of MyAssemblies, the CLR will now probe for an assembly named MyLibrary in the following order:

C:/Apps/MyLibrary.DLL
C:/Apps/MyLibrary/MyLibrary.DLL
C:/Apps/MyAssemblies/MyLibrary.DLL
C:/Apps/MyAssemblies/MyLibrary/MyLibrary.DLL
C:/Apps/MyLibrary.EXE
C:/Apps/MyLibrary/MyLibrary.EXE
C:/Apps/MyAssemblies/MyLibrary.EXE
C:/Apps/MyAssemblies/MyLibrary/MyLibrary.EXE

The sequence of file paths in which the CLR probes for an assembly file is important because the probing process stops once the CLR locates an assembly file with the correct file name. If you deploy an application along with one version of MyLibrary.dll in the ApplicationBase directory and a second version in the subdirectory MyAssemblies, which DLL file will the CLR load? You should see that the CLR is going to load the DLL in the ApplicationBase directory because that is always the first directory searched by the CLR during the probing process.

The Global Assembly Cache

You cannot use private assembly deployment to share an assembly DLL across several applications on the same machine. The best way to do that is to install the assembly in the GAC.

Installing an assembly DLL in the GAC eliminates path dependency problems between an application and a dependent DLL. It doesn't matter where the ApplicationBase directory for an application is located. The CLR can always find and load an assembly DLL when it has been installed in the GAC.

Another significant benefit to using the GAC is that you can install as many versions of the same assembly DLL on a single machine as you want. This is valuable because different applications can load whichever version of MyLibrary.dll works best with them. This is an example of how the GAC facilitates side-by-side assembly deployment.

The GAC has been designed to be a secure repository of assemblies. Therefore, the CLR places two important restrictions on the installation of DLLs in the GAC. First, you cannot add assemblies or remove them from the GAC unless you have either Administrators or Power Users privileges on the target machine. Second, you cannot install an assembly in the GAC unless it contains a strong name. That's because the CLR has been designed to perform a strong-name verification check on the assembly's digital signature whenever you install an assembly DLL in the GAC. This ensures that the GAC only contains assemblies that have been signed by someone in possession of the proper private key.

It is interesting to note that the CLR does not perform strong-name verification checks on an assembly's digital signature when it loads assemblies in the GAC at run time. That's because the CLR assumes that assemblies in the GAC have already been verified. Therefore, assembly deployment in the GAC provides a small performance boost. The GAC is the only place from which you can load a strongly named assembly without paying the price of strong-name verification checks at run time.

Now let's discuss how the CLR manages the GAC internally. The CLR contains a system component called the assembly manager that takes on the responsibilities of storing assembly files in the GAC and loading them at run time when they are first used by an application. The assembly manager is loaded from the system component FUSION.DLL.

I must point out that the manner in which the assembly manager stores and retrieves assembly files on a target machine should be considered a private implementation detail of the CLR. I am going to describe some of these details simply to give you a better sense of how the GAC works. You should never design applications or use deployment techniques that rely on these private details because they are likely to change in future versions of the CLR.

The assembly manager stores assembly files in the GAC using a special directory structure in the Windows file system. This directory structure is created as a subdirectory in the Windows directory with a path that looks like this:

C:\Windows\assembly\GAC

When you install assemblies into the GAC, the assembly manager creates new directories to store them in. In fact, the assembly manager will create a unique directory for each assembly that is stored in the GAC. The reason for this is that the GAC must be able to accommodate two different assemblies whose name only differs by their public key value or by their version number. After all, there could be many different assemblies with the file name MyLibrary.dll. Therefore, the assembly manager creates a unique directory for each assembly using all four parts of the assembly name. For example, imagine you install version 1.0.24.0 of MyLibrary.dll in the GAC and this assembly has a public key token of 29989d7a39acf230. When you do this, the assembly manager creates a new directory with the following path:

C:\WINDOWS\assembly\GAC\MyLibrary\1.0.24.0__29989d7a39acf230

As you can see, the assembly manager uses its own internal naming scheme for directories when it stores an assembly in the GAC. When it's time for the assembly manager to load an assembly with a specific four-part name, it knows where to locate it because it follows the same naming scheme.

When you want to install an assembly in the GAC, you are not required to interact directly with FUSION.DLL. Instead, you use a utility that has been written to interact with FUSION.DLL for you. If you want to install an assembly into the GAC on a development workstation for testing, you can use a command-line utility named GACUTIL.EXE that ships with the Microsoft® .NET Framework SDK. GACUTIL.EXE provides many command-line switches for installing and managing assemblies in the GAC. For example, the /i switch is used for the regular installation of an assembly, like so:

GACUTIL.EXE /i MyLibrary.dll

In order to install an assembly in the GAC using this technique, you must have access to the assembly file. However, it's also important to note that the assembly manager makes a copy of the assembly file when it's installed in the GAC. After installation, the assembly manager is only concerned with the copy it made. That means you can delete the original assembly file after it has been installed in the GAC. If you installed the assembly file from a CD, you can remove the CD from the drive without any problems; if you installed the assembly file from the network, you can disconnect from the network without any problems as well.

You can also examine and manage the assemblies in the GAC using a GUI-based utility called the Assembly Cache Viewer, an administrative utility that runs transparently in Windows Explorer as a Windows shell extension named SHFUSION.DLL. This utility is located in the \Windows\assembly directory (see Figure 1).

Figure 1 The Assembly Cache Viewer
Figure 1 The Assembly Cache Viewer

Note that the view supplied by the Assembly Cache Viewer does not actually show you the physical layout of the directory structure maintained by the GAC. Instead, it shows you a flattened-out view where all the assemblies are shown as a single scrollable list. You should also note that every assembly is displayed with its friendly name, version number, culture, and public key token.

The Assembly Cache Viewer provides administrators and developers alike a simple way to install and remove assemblies from the GAC. If you want to install an assembly, you can drag it from the Windows Explorer and drop it in the Assembly Cache Viewer. When you want to remove an assembly from the GAC, you can select it in the Assembly Cache Viewer and press the DELETE key on your keyboard.

Remember that FUSION.DLL is the only component that's allowed to read and write files into the GAC. When you manage assemblies using the Assembly Cache Viewer, you should understand that the Windows shell extension SHFUSION.DLL is interacting with FUSION.DLL behind the scenes to carry out your commands for you.

Deploying Assemblies with a Configured CodeBase

The final option for deploying a dependent assembly on a target machine is to configure it using a <codeBase> element. A configured <codeBase> element is powerful because it lets you download an assembly DLL from across the network. This means the CLR can download an assembly DLL to a target machine on demand the first time it is used by an application.

You can use a <codeBase> element to deploy a strongly named assembly in any directory on the target machine. You can even deploy a strongly named assembly on a file server or a Web server. You can then use a <codeBase> element to configure an app on a target machine to download that assembly on demand. If you plan on deploying your assemblies using a <codeBase> element, build them with a strong name to allow for this extra flexibility.

An example of an application-specific <codeBase> element is shown in Figure 2. Note that you can add a <codeBase> element to the machine.config file instead of the application configuration file to manage a dependent assembly on a machine-wide basis.

Figure 2 A <codeBase> Element (Close Figure 2)
<!-- MyApp.exe.config -->
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity
          name="MyLibrary" publicKeyToken="29989D7A39ACF230" />
        <codeBase
          version="1.0.24.0"
          href="http://www.AcmeCorp.com/Downloads/MyLibrary.dll"/>
      </dependentAssembly>
    </assemblyBinding>
  </runtime>
</configuration>

Examine the <codeBase> element shown in Figure 2. Note that a <codeBase> element must be placed inside a <dependentAssembly> element. The <dependentAssembly> element in this example also has an inner <assemblyIdentity> element which has attributes to identify the friendly name and public key token of the dependent assembly being configured.

While a <dependentAssembly> element only contains one <assemblyIdentity> element, it can have many inner <codeBase> elements. That's because each separate version of an assembly needs a separate <codeBase> element. The <dependentAssembly> element in Figure 2 contains only a single <codeBase> element for version 1.0.24.0, but it's possible to add additional <codeBase> elements for other versions of this assembly.

Note that the <codeBase> element contains an href attribute in addition to the version attribute. The purpose of the href attribute is to provide the CLR with a Uniform Resource Identifier (URI) that allows it to determine the location of the assembly file. If you want to configure a dependent assembly on the local file system using a <codeBase> element, you should use an href attribute with a URI that looks like this:

href="http://file:///c:\AcmeCorpSharedAssemblies\MyLibrary.dll"

If you want to configure a dependent assembly for download from a file server using a Univeral Naming Convention path name, you should configure the href attribute with a URI that looks like this:

href="http://file://AcmeCorpFileServer1\Downloads\MyLibrary.dll"

If you want to configure a dependent assembly for download from a Web server using HTTP, you should configure the href attribute with a URI that looks like this:

href="http://www.AcmeCorp.com/Downloads/MyLibrary.dll"

Keep in mind that you can also configure a <codeBase> element for a dependent assembly using the .NET Framework Configuration administrative tool, MSCORCFG.MSC, which is a Microsoft Management Console (MMC) snap-in that can be launched from a shortcut in the Administrative Tools group under the Windows Start menu. This eliminates the need to deal directly with the XML that goes into an application configuration file or into the machine.config file. By the way, when you start configuring applications and their dependent assemblies, be aware that the Microsoft .NET Framework Configuration administrative tool is a convenient GUI-based tool. This utility's UI is shown in Figure 3.

Figure 3 .NET Framework Configuration
Figure 3 .NET Framework Configuration

The .NET Framework Configuration administrative tool is convenient because it can do the work of creating an application configuration file for you. You can configure an application by simply interacting with standard Windows controls such as textboxes, radio buttons, and checkboxes. When you configure an application in this manner, the Microsoft .NET Framework Configuration administrative tool does the grunt work of creating the appropriate XML content and adding it to the application configuration file.

The Download Cache

Now let's discuss what happens when the assembly manager of the CLR downloads a dependent assembly from across the network using a <codeBase> element. The assembly manager does not load the assembly file directly into memory within a running application. Instead, it downloads the assembly file and writes it to disk in a temporary storage area known as the download cache.

There is an obvious advantage to this caching scheme. It isn't necessary to copy the assembly file across the network more than once. The assembly file only needs to be downloaded by the assembly manager and saved to disk the first time it's used by an application. The assembly manager can load the assembly file from the download cache on the local hard drive.

There are two important differences between the download cache and the GAC. The GAC is truly a machine-wide repository while the download cache is not. The download cache is actually managed by the CLR on a user-by-user basis. For example, if a user named Bob runs an application that downloads an assembly file from across the network using a <codeBase> element, the assembly manager will store the assembly file within a private subdirectory for this user under the following path:

C:\Documents And Settings\Bob\Local Settings\Application Data\

The other big difference is that the assembly manager treats the GAC as a secure and fully trusted repository of assemblies because it's local. The download cache is less secure and not fully trusted. Therefore, the assembly manager treats assemblies loaded from the download cache as mobile code that is subject to additional security restrictions. This is important to know because the CLR runs mobile code in a restricted sandbox to protect the host machine from attacks.

Assembly Loading

Now it's time to discuss what happens at run time when the assembly manager needs to load a dependent assembly into a running application. The assembly manager starts by breaking down the format string the four-part name of the assembly it's looking for. The assembly manager usually determines the format string by looking at a reference in the manifest of another assembly.

Let's walk through an example. Imagine you run MyApp.exe and it executes the first line of code that requires the CLR to load MyLibrary.dll. The assembly manager examines the assembly manifest for MyApp.exe and discovers the four-part name of the assembly file for MyLibrary.dll that was present when the consuming app was compiled. If MyLibrary.dll has a strong name, the assembly manager generates a format string containing a public key token, like this:

  MyLibrary,
  Version=1.0.24.0,
  Culture=neutral,
  PublicKeyToken=29989D7A39ACF230

At this point in the process, the assembly manager has the four-part name of the dependent assembly that was present during the compilation of MyApp.exe. In the next step, the assembly manager looks to see if there is any configuration information to redirect the application to use a different version number. Let's assume for the purposes of this discussion that the version number is not going to be redirected.

Once the assembly manager knows the four-part name of the assembly that it's looking for, it is now ready to begin searching for the assembly file. The assembly manager conducts its search in the following manner. First, it checks the config file redirect locations (like <codeBase>), then it searches for the assembly in the GAC. Next it searches for the assembly using configuration information in a <codeBase> element. And finally, it searches for the assembly by probing within the ApplicationBase directory.

The fact that the assembly manager always looks in the GAC first has an important implication. Think about what would happen if you deployed three copies of the same assembly. For example, imagine you have put one copy of the assembly DLL in the ApplicationBase directory, you have deployed a second copy using a <codeBase> element, and you have installed a third copy in the GAC. The assembly manager will always load the assembly from the GAC. Once it finds an assembly in the GAC, it stops the search.

If the assembly manager looks for a configured <codeBase> element and finds one, it then checks to see if the URI points to a location on the local hard disk. If it does, the assembly manager loads the assembly file directly from that location on disk.

If the URI in the <codeBase> element points to an assembly file on another machine, the assembly manager knows that it can only load a local copy of the assembly file from the download cache. Therefore, it checks the download cache to see if it has already been downloaded. If it hasn't, then the assembly manager copies it from across the network to the download cache. Once the assembly file is in the download cache, the assembly manager will load it.

If the assembly manager cannot find an assembly file in the GAC and it sees that the assembly has no configured <codeBase> element, it resorts to probing the ApplicationBase directory. The assembly manager conducts the probing process by looking directly in the ApplicationBase directory and then by looking in various subdirectories according to the rules outlined earlier in the section on private assemblies.

As you can imagine, many things can go wrong when the CLR attempts to load an assembly. For example, if the assembly manager cannot find the assembly file while probing within the ApplicationBase directory, it throws a FileNotFoundException to the line of code that caused the assembly manager to begin looking for the assembly in the first place. The CLR throws a FileNotFound exception when it cannot locate the assembly file configured in a <codeBase> element and when it determines that the assembly it is attempting to load does not have the correct version number, culture setting, or public key value.

Conclusion

This month I described the three primary ways to deploy an assembly DLL. Private assembly deployment is valuable because it provides all the advantages of XCOPY deployment. But it's limited because you must deploy the assembly DLL within the ApplicationBase directory structure. Assembly deployment in the GAC is useful because it lets you share an assembly DLL across several different applications regardless of where the ApplicationBase directory for each application is located. The GAC also provides the advantage of being a secure and fully trusted repository of assemblies. Finally, deploying an assembly DLL using a <codeBase> element allows you to deploy a strongly named assembly anywhere on the local hard drive or at any accessible location across the network. There are two main advantages to using a <codeBase> element. First, you are not restricted to deploying the assembly DLL within the ApplicationBase directory structure. Second, you can configure an assembly DLL to be download by the CLR on demand from across the network.

Send your questions and comments for Ted to instinct@microsoft.com.

Ted Pattison is an independent contractor living in Los Angeles, CA. Ted is the author of Programming Distributed Applications with COM+ and Visual Basic 6.0 (Microsoft Press, 2000). His new book, Building Applications and Components with Visual Basic .NET, will be available in October 2003 from Addison-Wesley. Contact Ted at TedP@SubliminalSystems.com.


.NET Column: More on Generics in the CLR

[ Editor's Update - 1/20/2006: This article refers to a beta version of Visual Studio 2005. An updated version of the article, reflecting features found in the final release of Visual Studio 2005, can be found at .NET: Introducing Generics in the CLR.]
.NET
More on Generics in the CLR
Jason Clark

Contents
How Does the Compiler Handle Generic Types?
Rules and Constraints
Generic Interfaces and Delegates
Generics in the Class Library
Conclusion

In the September 2003 installment of this column, I began a discussion of generics in the common language runtime (CLR). I introduced the concept of a generic type, covered the flexibility and code reuse that's afforded by generics, discussed performance and type safety, and presented the generics syntax in C# in a simple code sample. This month I'll dig deeper into the internal workings of the CLR with respect to generics. I'll cover type constraints, generic classes, methods, structures, and the upcoming generic class libraries.

How Does the Compiler Handle Generic Types?

Both C++ templates and the proposed generics equivalent in the Java language are features of their respective compilers. These compilers construct code from references to the generic or template type at compile time. This can cause code bloat as well as reduced type equivalence from one construction to another, even when the type arguments are the same. In contrast, CLR generics do not work this way.

Generics in the CLR are a first-class feature of the platform itself. To implement it this way required changes throughout the CLR including new and modified intermediate language instructions, changes to the metadata, type-loader, just-in-time (JIT) compiler, language compilers, and more. There are two significant benefits to the run-time expansion in the CLR.

First, even though each construction of a generic type, such as Node<Form> and Node<String>, has its own distinct type identity, the CLR is able to reuse much of the actual JIT-compiled code between the type instantiations. This drastically reduces code bloat and is possible because the various instantiations of a generic type are expanded at run time. All that exists of a constructed type at compile time is a type reference. When assemblies A and B both reference a generic type defined in a third assembly, their constructed types are expanded at run time. This means that, in addition to sharing CLR type-identities (when appropriate), type instantiations from assemblies A and B also share run-time resources such as native code and expanded metadata.

Type equivalency is the second benefit of run-time expansion of constructed types. Here is an example: referencing code in AssemblyA.dll that constructs Node<Int32> and referencing code in AssemblyB.dll that constructs Node<Int32> will both create objects of the same CLR type at run time. This way, if both assemblies are used by the same application, their constructions of the Node<T> type resolve to the same type and their objects can be exchanged freely. You should note that compile-time expansion would make this logically simple equivalency either problematic or impossible to implement.

There are a few other benefits to implementing generics at the runtime level, rather than at the compiler level. One such benefit is that generic type information is preserved between compilation and execution and is therefore accessible at all points in the code lifecycle. For example, reflection provides full access to generics metadata. Another benefit is rich IntelliSense® support in Visual Studio® .NET as well as a clean debugging experience with generic code. In contrast, Java generics and C++ templates lose their generic identity by run time.

Another benefit, and mainstay of the CLR, is cross-language use—a generic defined using one managed language can be referenced by code written in another managed language. Meanwhile, the likelihood that a language vendor will put generics support in their compilers is increased by the fact that much of the hard work is done in the platform.

Of all the fringe benefits of run-time type expansion, my favorite is a somewhat subtle one. Generic code is limited to operations that are certain to work for any constructed instantiation of the type. The side effect of this restriction is that CLR generics are more understandable and usable than their C++ template counterparts. Let's look at the constraints around generics in the CLR.

Rules and Constraints

One problem that plagues programmers who use C++ templates is the many ways in which a particular attempt at type construction might fail, including the failure of a type argument of a type parameter to implement a method called by the templated code. Meanwhile, the compiler errors in these cases can be confusing and may seem unrelated to the root problem. With run-time expansion of constructed types, similar errors would become JIT-compiler or type-load errors rather than compile-time errors. The CLR architects decided that this would not be an acceptable implementation for generics.

Instead, they decided that a generic type, such as Node<T>, must be verifiable at compile time as a valid type for any possible type instantiation of the generic type, even though type instantiation actually occurs at run time. As such, expansion errors around problematic type constructions become impossible. To achieve this goal, the architects constrained the features of generics with a set of rules and restrictions that make it possible to assure the validity of a generic type before attempting an expansion of one of its instantiations.

There are some rules that constrain the types of code that you can write generically. The essence of these rules can be summed up in a single statement: generic code is only valid if it will work for every possible constructed instance of the generic. Otherwise, the code is invalid and will not compile correctly (or if it does, it won't pass verification at run time).

At first this can seem like a limiting rule. Here is an example:

public class GenericMath {
   public T Min<T>(T item1, T item2) {
      if (item1 < item2) {
         return item1;
      }
      return item2;
   }
}
This code is not valid in CLR generics. The error that you get from the C# compiler reads something like this:
invalid.cs(4,11): error CS0019: Operator '<' cannot be applied to
operands of type 'T' and 'T'

Meanwhile, minor syntax differences aside, this very same code is allowed in C++ templates. Why the limitation with generics? The reason is that the less-than operator in C# only works with certain types. However, the type parameter T in the previous code snippet could expand to any CLR type at run time. Rather than risk having invalid code at run time, the preceding example is found invalid at compile time.

Operators aside, though, the same limitation applies to more manageable uses of types, such as method calls. This modification to the Min<T> method is also invalid generic code:

class GenericMath {
   public T Min<T>(T item1, T item2) {
      if (item1.CompareTo(item2) < 0) {
         return item1;
      }
      return item2;
   }
}
This code is invalid for the same reason as the previous example. The CompareTo method, although implemented by many types in the class library and easily implemented by your custom types, is not guaranteed to exist on every possible type that can be used as an argument to T.

But you've already seen that method calls are not completely disallowed in generic code. In Figure 1 the GetHashCode method is called on the two parameterized arguments, and the Node<T> type in Figure 2 calls the ToString method on its parameterized m_data field. Why are GetHashCode and ToString allowed, when CompareTo is not? The reason is that GetHashCode and ToString are both defined on the System.Object type from which every possible CLR type is derived. This means that every possible expansion of type T has an implementation of the ToString and GetHashCode member functions.

Figure 2 Defining Code: Generic Linked-list Node (Close Figure 2)
class Node<T> {
   T        m_data;
   Node<T>  m_next;

   public Node(T data, Node<T> next) {
      m_data = data;
      m_next = next;
   }

   // Access the data for the node
   public T Data {
      get { return m_data; }
      set { m_data = value; }
   }

   // Access the next node
   public Node<T> Next {
      get { return m_next; }
      set { m_next = value; }
   }

   // Get a string representation of the node
   public override String ToString() {
      return m_data.ToString();
   }
}
Figure 1 Indirection as a Generic Method (Close Figure 1)
using System;

class App {
   public static void Main() {
      Boolean result;

      // Constructs an instance of CompareHashCodes<T> for Int32
      result = CompareHashCodes<Int32>(5, 10);

      // Constructs an instance of CompareHashCodes<T> for String
      result = CompareHashCodes<String>("Hi there", "Hi there");
   }

   // Generic method, a "method factory" of sorts
   static Boolean CompareHashCodes<T>(T item1, T item2) {
      return item1.GetHashCode() == item2.GetHashCode();
   }
}

If generics are to be useful for anything beyond collection classes, then generic code needs to be able to call methods other than those defined by System.Object. But remember, generic code is only valid if it will work for every possible constructed instance of the generic. The resolution to these two seemingly incompatible requirements is a feature of CLR generics called constraints.

You should be aware that constraints are an optional component of a generic type or method definition. A generic type may define any number of constraints, and each constraint applies an arbitrary restriction on the types that can be used as an argument to one of the type parameters on the generic code. By restricting the types that can be used in the construction of a generic type, the limitations on the code that references the constrained type parameter are relaxed (see Figure 3).

Figure 3 Defining Code: Generic Constraint, Min and Max (Close Figure 3)
public class GenericMath {
   public static T Min<T>(T item1, T item2)
      where T : IComparable {
      if (item1.CompareTo(item2) < 0) {
         return item1;
      }
      return item2;
   }

   public static T Max<T>(T item1, T item2)
      where T : IComparable {
      if (item1.CompareTo(item2) < 0) {
         return item2;
      }
      return item1;
   }
}

// System.IComparable definition:
// public interface IComparable {
//    Int32 CompareTo(Object obj);
// }

It is valid for Min<T> and Max<T> to call CompareTo on its items because the constraint is applied to the type parameter T. In the third line of code in Figure 3, you can see the introduction of a where clause that looks like this:

where T : IComparable
This constraint indicates that any construction of the method Min<T> must supply a type argument for parameter T of a type that implements the IComparable interface. This constraint limits the variety of possible instantiations of Min<T> while increasing the flexibility of the code in the method such that it can now call CompareTo on variables of type T.

Constraints make generic algorithms possible by allowing generic code to call arbitrary methods on the expanded types. Although constraints require additional syntax for generic defining code, referencing code syntax is unchanged by constraints. The only difference for referencing code is that the type arguments must respect the constraints on the generic type. For example, the following referencing code to Max<T> is valid:

GenericMath.Min(5, 10);
This is because 5 and 10 are integers and the Int32 type implements the IComparable interface. However, this attempted construction of Max<T> produces a compiler error:
GenericMath.Min(new Object(), new Object());
Here is the error generated by the compiler:
MinMax.cs(32,7): error CS0309: The type 'object' must be convertible
to 'System.IComparable' in order to use it as parameter 'T' in the
generic type or method 'GenericMath.Min<T>(T, T)'

System.Object is an invalid type argument for T because it does not implement the IComparable interface required by the constraint on T. Constraints make it possible for the compiler to provide descriptive errors like the one shown in the preceding example, when the type argument is not compatible with the type parameter on the generic code.

At present, generics support three kinds of constraints: the interface constraint, the base class constraint, and the constructor constraint. An interface constraint specifies an interface with which all type arguments to the parameter must be compatible. Any number of interface constraints can apply to a given type parameter.

The base class constraint is similar to an interface constraint, but each type parameter may only include a single base class constraint. If no constraint is specified for a type parameter, then the implicit base class constraint of Object is applied.

The constructor constraint makes it possible for generic code to create instances of the type specified by a type parameter by constraining type arguments to types that implement a public default constructor. At this time, only default or parameterless constructors are supported with the constructor constraint.

A where clause is used to define a constraint or list of constraints for a given type parameter. Each where clause applies to only a single type parameter. A generic type or method definition may have no where clauses, or it may have as many where clauses as it has type parameters. A given where clause may include a single constraint or list of constraints separated by commas. Figure 4 shows the various syntaxes for applying constraints to generic code.

Figure 4 Defining Code: C# Generic Constraint Syntax (Close Figure 4)
// Class
class SortedCollection<T>
  where T : IComparable {...}

// Interface
interface IBlueAlgorithm<T>
  where T : IRedAlgorithm {...}

// Factory Class
class MyFactory<T,P>
  where T : new()
  where P : IConstructionPolicy{...}

// Method
public static void EvaluateItems<T>(T [] items)
  where T : IRedAlgorithm, IBlueAlgorithm<T> {...}

// Delegate
delegate void
  GenericEventHandler<S,A>(S sender, A args)
  where A : EventArgs;

The fundamental rule of generics—that either all possible instantiations of the generic are valid or the generic type itself is invalid—has a few more interesting side effects. First, there's casting. In generic code a variable of a type-parameter type may only be cast to and from its base class constraint type or bases thereof. This means that if a type parameter T has no constraints, it can only be cast to and from Object references. However, if T is constrained to a more derived type such as FileStream, then the generic definition can include casts to and from T and FileStream, as well as all bases of FileStream, down to Object. The code in Figure 5 shows this cast rule in action.

Figure 5 Custom Rule in Generics (Close Figure 5)
static void Foo<T>(T arg) {
  Object o = arg;
  T temp = (T) o;

  Int32 num;

  num = (Int32) temp;          // Compiler error
  num = (Int32)(Object) temp;  // Compiles, but throws InvalidCastException
                               // at run time if T != Int32

  temp = (T) 5;           // Compiler error
  temp = (T)(Object) 5;   // Compiles, but throws InvalidCastException at
                          // run time if T != Int32
}

In this example, T has no constraint and is considered to be unbounded. It has the implicit base class constraint of Object. Casts of T to and from the Object class, as well as casts to and from interface types, are valid at compile time. Casts to other types, such as Int32, for unbounded T are disallowed by the compiler. The casting rule for generics also applies to conversions and therefore disallows the use of casting syntax to make type conversions such as the conversion from an Int32 to an Int64; conversions like this are not supported with generics.

Here's another interesting tidbit. Consider the following code:

void Foo<T>() {
   T x = null;  // compiler error when T is unbounded
   •••
While a null assignment like this one has obvious uses, there is one possible problem. What if T is expanded to a value type? A null assignment to a value variable has no meaning. Fortunately, the C# compiler provides special syntax that provides correct results regardless of the runtime type of T:
void Foo<T>() {
   T x = T.default;  // OK for any T
}

The expression on the right side of the equals sign is called the default value expression. If T is expanded to a reference type, then T.default resolves to null. If T is expanded to a value type, then T.default is the all-bits-zero value for the variable.

Null assignment to a parameterized variable T is disallowed if T is unbounded, so one might guess that the following would also be invalid, but it is not:

void Foo<T>(T x) {
   if (x == null) { // Ok
      •••
   }
}
If null is allowed here, what happens if T is expanded to a value type? Where T is a value type, the Boolean expression in the preceding example is hardwired to false. Unlike in the null assignment case, null comparison makes sense for any expansion of T.

The premise that generic types are verifiably valid at compile time for any possible instantiation does have an effect on your generic code. However, I have found that the additional structure around generics in the CLR, by comparison to templates in C++, has a clarifying effect. All in all, constraints and their surrounding infrastructure are one of my favorite aspects of CLR generics.

Generic Interfaces and Delegates

Generic classes, structures, and methods are the primary features of CLR generics. Generic interfaces and delegates are really supporting features. A generic interface alone has limited usefulness. But when used with a generic class, struct, or method, generic interfaces (and delegates) have significant impact.

The GenericMath.Min<T> and GenericMath.Max<T> methods in Figure 3 both constrain T to be compatible with the IComparable interface. This allows the methods to call CompareTo on the parameterized arguments to the methods. However, neither of these methods as implemented in Figure 3 enjoy the full benefits of generics. The reason is that calls to non-generic interfaces for value types incur a boxing if the interface takes one or more object parameters, such as the obj parameter of CompareTo.

Each call to GenericMath.Min<T> in Figure 3 causes a boxing of the parameter to the CompareTo method if the instantiation of the method expands T to a value rather than to a reference. This is where generic interfaces can help.

The code in Figure 6 refactors the GenericMath methods to constrain T over a generic interface IComparable<T>. Now if an instantiation of Min<T> or Max<T> uses a value type as an argument to T, the interface call to CompareTo is part of the interface construction, its parameter is the value type, and no boxing occurs.

Figure 6 Defining Code: Generic Interface, Min and Max (Close Figure 6)
public class GenericMath {
   public static T Min<T>(T item1, T item2)
      where T : IComparable<T> {
      if (item1.CompareTo(item2) < 0) {
         return item1;
      }
      return item2;
   }

   public static T Max<T>(T item1, T item2)
      where T : IComparable {
      if (item1.CompareTo(item2) < 0) {
         return item2;
      }
      return item1;
   }
}

public interface IComparable<T> {
   Int32 CompareTo(T obj);
}

Generic delegates have similar benefits to generic interfaces, but are oriented to methods rather than types.

Generics in the Class Library

In addition to implementing generics in the CLR, Microsoft is planning to provide new generic library classes as part of the class libraries for the release of the CLR code-named "Whidbey". A preview release of the Whidbey CLR should be available about the time that this column goes to press. (For more details, see the September 2003 installment of this column.) Minimally, generic collection classes for implementing lists, dictionaries, stacks, and queues are expected. Additionally, the class libraries will include supporting interface types such as IList<T>, ICollection<T>, and IComparable<T>, which are generic equivalents of the simple interfaces that shipped in the Microsoft® .NET Framework 1.0 and 1.1 class libraries.

Finally, you will notice that types throughout the class library will be augmented with generic versions of new and existing functionality. For example, the System.Array class will include generic versions of its BinarySearch and Sort methods that take advantage of the generic interfaces IComparer<T> and IComparable<T>. At the time of this writing, Microsoft has not yet decided exactly how much of the existing class library will be reworked to include generic support in the next release of the runtime.

Conclusion

CLR generics are a powerful application and library development feature. Whether you choose to apply generics by using the collection classes or by architecting your app generically throughout, generics can make your code more type-safe, maintainable, and efficient. I have had fun with generics programming in managed code in these two installments of this column, and I look forward to the released product. It's exciting to see the CLR progress with the addition of significant features like generics. There are so many interesting possibilities. Stay tuned.

Send your questions and comments for Jason to dot-net@microsoft.com.

Jason Clark provides training and consulting for Microsoft and Wintellect (http://www.wintellect.com) and is a former developer on the Windows NT and Windows 2000 Server team. He is the coauthor of Programming Server-side Applications for Microsoft Windows 2000 (Microsoft Press, 2000). You can get in touch with Jason at JClark@Wintellect.com.


C++ Q&amp;A: Getting a Menu Handle, Declaring GetParam, and Filtering File Names

New information has been added to this article since publication.
Refer to the Editor's Update below.

C++ Q&A
Getting a Menu Handle, Declaring GetParam, and Filtering File Names
Paul DiLascia

Code download available at: CQA0310.exe (152KB)

Q I'm trying to handle OnMenuSelect in my MFC app. The declaration for this function is:

void OnMenuSelect(UINT nItemID, UINT nFlags, HMENU hSysMenu);

When my handler is invoked, none of the menu functions, like GetMenuItemCount, work properly. I added an ASSERT to check for a valid menu handle:

ASSERT(::IsMenu(hSysMenu));
It fails, so the handle isn't valid. How do I get the menu handle?

Frank Connors

A Congratulations; it's not every day you uncover a bug in MFC! In fact, while hSysMenu is declared as HMENU, it isn't a handle at all. What is it, then? Answer: a pointer to an MFC CMenu object. You can prove this with the following code:

void CMainFrame::OnMenuSelect(... HMENU hSysMenu)
{
  CMenu* pMenu = DYNAMIC_DOWNCAST(CMenu, (CObject*)hSysMenu);
  TRACE("OnMenuSelect, hSysMenu=%p is %s\n", hSysMenu,
    ::IsMenu(hSysMenu) ? "HMENU" : pMenu ? "CMenu" : "unknown");
  return CFrameWnd::OnMenuSelect(nItemID, nFlags, hSysMenu);
}
[ Editor's Update - 12/6/2004: This bug was fixed in MFC 7.1.] To understand how this can be, you have to delve into the mysterious workings of MFC message maps and message handler functions. When you use ON_WM_MENUSELECT, the macro expands, like so:
// ON_WM_MENUSELECT()
{ WM_MENUSELECT, 0, 0, 0, AfxSig_vwwh,
  (AFX_PMSG)(AFX_PMSGW) \
  (static_cast< void (AFX_MSG_CALL CWnd::*)
    (UINT, UINT, HMENU) > (OnMenuSelect)) }
Figure 1 TraceWin
Figure 1 TraceWin
That's a lot of gobbledygook to create a table entry whose message ID is WM_MENUSELECT and handler function is OnMenuSelect. The important thing here is the special "signature code" AfxSig_vwwh. In order to convert an HWND or HMENU to a pointer to an MFC object like CWnd and CMenu, MFC stores an enum value in your message map. The value identifies the handler's argument types and return value, as shown here:
// from afxmsg_.h
enum AfxSig
{
  AfxSig_v_uu_M,  // void (UINT, UINT, CMenu*)
  AfxSig_b_W_uu,  // BOOL (CWnd*, UINT, UINT)
  ... // etc, over 140 total
};

The naming convention is AfxSig_return type_WPARAM types_LPARAM types. For example, take a look at the declaration for OnSetCursor here:

BOOL OnSetCursor(CWnd* pWnd,
  UINT nHitTest, UINT message);
Thus, the symbol for the corresponding signature code is AfxSig_b_W_uu, which is what ON_WM_SETCURSOR uses. Here, b means the return type is BOOL, W means that WPARAM should be converted to CWnd*, and uu means that LPARAM should be converted to two UINTs from the low- and high-order words. When MFC's internal message-processing function CWnd::OnWndMsg sees a message map entry with signature code AfxSig_b_W_uu, it converts the arguments like so:
AFX_MSGMAP_ENTRY* lpEntry = // find it
switch (lpEntry->nSig) {
case AfxSig_b_W_uu:
  CWnd* pWnd = CWnd::FromHandle(reinterpret_cast<HWND>(wp));
  UINT arg1 = LOWORD(lp);
  UINT arg2 = HIWORD(lp);
  // call handler w/converted args
  *pRes = (*lpEntry->pfn)(pWnd, arg1, arg2);
  •••
The point of the signature code is that it tells OnWndMsg how to convert WPARAM and LPARAM without knowing the handler function at compile time. I've glossed over some gnarly details (MFC uses a giant union to cast the function pointer to a function with the appropriate signature), but you get the basic idea. The mechanics aren't important. What's important is that MFC uses an enum to identify the signature of each handler function; the ON_WM_XXX macros create message map entries with the proper enum codes; and CWnd::OnWndMsg uses the codes to convert WPARAM and LPARAM values to MFC objects before calling your handler function. The result is that your handler functions can deal with MFC objects instead of raw handles.

How does all this relate to your OnMenuSelect problem? Apparently one day the friendly Redmondtonians embarked on a code cleanup because many AfxSig codes are equated to other codes. For example:

// in afxmsg_.h
enum AfxSig {
  •••
  AfxSig_bWww = AfxSig_b_W_uu,
};

In other words, AfxSig_bWww is another name for AfxSig_b_W_uu. It appears the Redmondtonians adopted the underbar convention in a later release, no doubt because something like uuu is ambiguous (could be either u_uu or uu_u). To maintain backward compatibility, they kept the old names—like AfxSig_bWww. This is where the OnMenuSelect bug crept in. If you look in afxmsg_.h, you'll see the following lines:

enum AfxSig
{
  AfxSig_vwwh = AfxSig_v_uu_M,
  •••
AfxSig_vwwh means two WORDs and a HANDLE returning void. AfxSig_v_uu_M means two UINTs and a CMenu returning void. WORDs and UINTs are interchangeable, but a HANDLE is definitely not equivalent to a CMenu. ON_WM_MENUSELECT creates a message map entry with AfxSig_vwwh (LPARAM is HMENU), but CWnd::OnWndMsg reads it as AfxSig_v_uu_M and converts LPARAM to CMenu*. Oops. Whoever equated AfxSig_vwwh with AfxSig_l_uu_M forgot to change the declaration for OnMenuSelect. The code does actually run, but hSysMenu has the wrong type.

Now that I've explained the bug, what can you do about it? The easiest thing is to ignore the declaration and cast hSysMenu to a CMenu* in your handler.

void CMainFrame::OnMenuSelect(... HMENU hSysMenu)
{
  CMenu* pMenu = (CMenu*)hSysMenu;
  UINT nItems = pMenu->GetMenuItemCount();
  // etc..
}

Finally, I should point out that despite its name, hSysMenu is not the system menu, but the menu that was clicked.

Q Your column in the August 2002 issue of MSDN® Magazine has been very helpful in converting from C++ to C#. However, I need to call a function in a C/C++ DLL that has a void* as one of its parameters. The void* can be a pointer to a short, long, or character array depending on the value of another parameter, the parameter ID. How should I write the C# wrapper to handle this? Here's an example of my code in C:

// In DLL's header (.h) file:
GetParam(short param_id, void* param);
// In my program:
unsigned short width;
unsigned long exp_time;
char cam_name[50];
GetParam(PARAM_WIDTH,(void*)&width);
GetParam(PARAM_TIME, (void*)&exp_time);
GetParam(PARAM_CAM_NAME, (void*)&cam_name);

The DLL figures out the correct type for param based on param_id. It's a third-party DLL; I didn't write it, so I have no control over it. I can't figure out how to declare GetParam in C#. Do you have any suggestions?

Earl Izydore

A Man, your DLL is sooooo retro. Parameter IDs, void*s... It's like programming the 8080 (for those of us who remember what it was). And to think here we have entered the age of C# and the Web-programmable TiVo, and we're still dealing with grody code like this. Never fear, C# can handle any interop conundrum a legacy programmer can conjure. The reason you're having trouble figuring out how to declare GetParam is because one declaration isn't enough. You need three.

Figure 2 shows a DLL I wrote to simulate your situation. MyLib.dll has a single function, GetParam, that mimics what you've described. The first argument is an enum code that's 1, 2, or 3, depending on which parameter you want to obtain: width, time, or name. The value is returned through the second argument, a void*. The actual type depends on which parameter you request: short for width, long for time, and string (char*) for name.

Figure 2 MyLib.cpp (Close Figure 2)
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
// This DLL has a single function, GetParam, which gets the value of three
// different parameters: width, time and name. The parameter is returned
// through a void* pointer which can be a short, long, or string
// depending on which parameter is requested.
//
// You would of course never write such bad code as this; the example is
// intended to mimic old-style legacy code that does this sort of thing.
// The companion program test.cs shows how to call this DLL from C#/.NET
// in a typesafe manner.
//
#include <string.h>

enum {
   PARAMID_WIDTH=1,
   PARAMID_TIME,
   PARAMID_NAME
};

// This is normally defined in windows.h:
#define CALLBACK __stdcall

//////////////////
// Exported fn to get parameter. 1st arg says which param to get; 2nd
// parameter is output buffer whose type depends on param_id.
//
extern "C" __declspec( dllexport ) int GetParam(int param_id,
   void* param)
{
   switch (param_id) {
   case PARAMID_WIDTH:
      *((short*)param) = 17;         // return param is short
      break;
   case PARAMID_TIME:
      *((long*)param) = 230001;      // return param is long
      break;
   case PARAMID_NAME:
      strcpy((char*)param,"Goofy");  // return param is string (char*)
      break;
   default:
      return -1;
   }
   return 0;
}

Figure 3 shows how to call MyLib from C# in a bug-proof, type-safe manner. The trick is to use three overloaded import declarations, each with a different signature for the different parameter types. Don't forget that in C#, just like in C++, you can overload your functions. That is, you can write several functions with the same name but different argument types. This applies to interop/import declarations as well as ordinary functions—er, I mean methods. The compiler's IQ is high enough to figure out which overloaded method it should call based on the arguments you called it with. Once you realize you can write three declarations, each one uses straightforward rules from Basic Interop Programming 101. To marshal a pointer-to-short or pointer-to-long, use a reference; to marshal an in/out string, use StringBuilder. Figure 3 shows the exact syntax.

Figure 3 Test.cs (Close Figure 3)
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
// This program illustrates how to use several overloaded interop
// declarations to wrap a DLL function that takes a void* parameter that
// can be different things depending on an op code.
//
using System;
using System.Text;
using System.Runtime.InteropServices;

//////////////////
// This class wraps the C DLL (MyLib.dll).
//
class MyLib {
   public enum PARAMID { WIDTH=1, TIME, NAME };

   // Three GetParam overloads, one for each signature (return/argument
   // types). Note that in C#, a pointer-to-int/short/long is just a ref
   // parameter, and StringBuilder is used to marshal in/out character
   // arrays (strings are call-only, can't be returned).
   //
   // These imports are private. Callers must go through the GetXxx
   // methods,which eliminate the possibility of using the wrong
   // parameter ID.
   //
   [DllImport("MyLib.dll")]
   private static extern int GetParam(PARAMID param_id, ref short wid);

   [DllImport("MyLib.dll")]
   private static extern int GetParam(PARAMID param_id, ref long time);

   [DllImport("MyLib.dll")]
   private static extern int GetParam(PARAMID param_id, StringBuilder s);

   // These wrappers further eliminate the need for param_id.
   // Op codes and void*s are soooo yucky!
   //
   public static short GetWidth() {
      short wid=0;
      GetParam(PARAMID.WIDTH, ref wid);
      return wid;
   }

   public static long GetTime() {
      long time=0;
      GetParam(PARAMID.TIME, ref time);
      return time;
   }

   public static String GetName() {
      StringBuilder name = new StringBuilder();
      GetParam(PARAMID.NAME, name);
      return name.ToString();
   }
}

//////////////////
// App class with Main entry calls MyLib functions.
//
class MyApp {

   // main entry point
   [STAThread]
   static int Main(string[] args) {
      Console.WriteLine("Width is {0}", MyLib.GetWidth());
      Console.WriteLine("Time  is {0}", MyLib.GetTime());
      Console.WriteLine("Name  is {0}", MyLib.GetName());
      return 0;
   }
}

With the new declarations in hand, all you have to do is call GetParam with the proper argument type. C# knows which method to call and how to marshal the arguments. Amazing, isn't it? Of course, you still have to be careful to use the correct param_id/argument type combination. To avoid even the possibility of screwups, you can go one step further, as shown by the code in Figure 3, where the three GetParam overloads are declared private so only the wrapper class itself can call them. Instead of GetParam, programmers call new public methods GetWidth, GetTime, and GetName. For example:

// get "width" parameter
short width = MyLib.GetWidth();
MyLib.GetWidth hides param_id, thereby eliminating the possibility of an accidental param_id/type mismatch.
public static short GetWIdth() {
  short wid=0;
  GetParam(PARAMID.WIDTH, ref wid);
  return wid;
}

Any time you import retro DLLs into your C# application, it's always a win to spend a few minutes doing it right. If you wrap the DLL in type-safe declarations, you'll spare yourself the headache of chasing bugs later.

Q I'm trying to write a custom file open dialog that filters a certain file name from the dialog. I use the OFN_ENABLEINCLUDENOTIFY style and handle the CDN_INCLUDEITEM notification. Unfortunately, the dialog box always ignores my return value if the item has the SFGAO_FILESYSTEM and SFGAO_FILESYSANCESTOR attributes. In other words, if I'm dealing with normal files and directories, or folders that are the parent of a file system folder (such as My Computer), then by design the OFN_ENABLEINCLUDENOTIFY style doesn't apply. My question is, how can I work around this problem? How can I remove certain files from the listview box?

Oded Gottdenker

A Gosh, will there ever be an end to questions about the file open dialog? I guess since almost every application opens files, customizing file open is natural. And what's more natural than filtering the file names in the list? The most common way to filter files is by the file name extension—.txt, .doc, or .whatever—but what if you want to filter based on some other criterion, such as the last-modified date, owner, or whether the file name rhymes with "dimple"?

At first glance, Windows® seems to offer exactly what you want. If you create your dialog with OFN_ENABLEINCLUDENOTIFY, Windows sends your hook procedure a CDN_INCLUDEITEM notification for every item it adds to the open list. If you return FALSE, Windows excludes the item. The problem is, Windows doesn't notify you for ordinary files, only pseudo-objects like namespace extensions. When you read the documentation through a magnifying glass, the print is perfectly clear: "The dialog box always includes items that have both the SFGAO_FILESYSTEM and SFGAO_FILESYSANCESTOR attributes, regardless of the value returned by CDN_INCLUDEITEM." Apparently the Redmondtonians added CDN_INCLUDEITEM for their own purposes, which didn't include filtering ordinary file names. Sigh.

There's no need to reach for your hanky. What the Redmondtonians taketh away, you can always add on your lonesome. I modified last month's OpenDlg program to add a file filter that works for ordinary files (see C++ Q&A: Retrieving Hidden Path Names, Mouse Events in C#). Faithful readers will recall that last month I showed how to get the true path name of files listed in the open dialog's list view. I wrote a class, CFileDlgHelper, with a function GetItemPathName to get the true path name of any item in the open dialog's list view. This month I'll use CFileDlgHelper to filter the file names.

Figure 4 shows the code. CMyOpenDlg::FilterFiles is the function that does the work. It iterates the file names in the list, calling a virtual function IncludeFile for each one. If IncludeFile returns TRUE, FilterFiles does nothing. If IncludeFile returns FALSE, FilterFiles deletes the file name from the list view. Here's what it looks like in pseudocode:

 void CMyOpenDlg::FilterFiles()
{
  for (CString pathname = /* each listview file */) {
    if (!IncludeFile(fn)) {
      // delete it
    }
  }
}

Figure 4 MyDlg (Close Figure 4)

MyDlg.h
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#include "FileDlgHelper.h"

//////////////////
// My open dialog — customized to add preview, debug panes, and Select All
// button.
//
class CMyOpenDlg : public CFileDialog {
protected:
    CFileDlgHelper m_dlghelper;             // helper for file dialogs
    CEdit    m_edit1;                       // edit control for preview
    CEdit    m_edit2;                       // another for debug info

    // Helpers
    CString GetTextPreview(LPCTSTR pszPath);
    void AddText(CEdit& wndEdit, LPCTSTR lpText, BOOL bClear=FALSE);
    void ShowFileInfo();

public:
    CMyOpenDlg();

    virtual BOOL OnNotify(WPARAM wp, LPARAM lp, LRESULT* pRes);

    // Handle CDN_ notifications
    virtual void OnFileNameChange();
    virtual void OnFolderChange();
    virtual void OnTypeChange();

    virtual BOOL OnIncludeItem(IShellFolder* ishf, LPCITEMIDLIST pidl);

    // Command/message/UI handlers/overrides
    virtual int DoModal();
    virtual BOOL OnInitDialog();
    afx_msg void OnSize(UINT nType, int cx, int cy);
    afx_msg void OnSelectAll();
    afx_msg void OnUpdateSelectAll(CCmdUI* pCmdUI);

    void FilterFiles();
    virtual BOOL IncludeFile(CString pathname);

    DECLARE_DYNAMIC(CMyOpenDlg)
    DECLARE_MESSAGE_MAP()
};
MyDlg.cpp
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2003
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#include "StdAfx.h"
#include "MyDlg.h"
#include "FileDlgHelper.h"
#include "Resource.h"

#ifdef _DEBUG
#define new DEBUG_NEW
#undef THIS_FILE
static char THIS_FILE[] = __FILE__;
#endif

// IDs of controls in Explorer-style dialog, from Spy
#define ID_FILESOFTYPE        0x0441
#define ID_FILENAME            0x0442
#define ID_LOOKIN                0x0443

IMPLEMENT_DYNAMIC(CMyOpenDlg, CFileDialog)
BEGIN_MESSAGE_MAP(CMyOpenDlg, CFileDialog)
    ON_COMMAND(ID_SELECT_ALL,OnSelectAll)
    ON_UPDATE_COMMAND_UI(ID_SELECT_ALL,OnUpdateSelectAll)
    ON_WM_SIZE()
END_MESSAGE_MAP()

//////////////////
// does path ends in ".txt" ?
//
static BOOL IsTextFileName(CString fn)
{
    fn.MakeLower();
    return fn.Right(4)==_T(".txt");
}

////////////////
// Constructor
//
CMyOpenDlg::CMyOpenDlg() : CFileDialog(TRUE,
    NULL,                                          // no default extension
    NULL,                                          // ..or file name
    OFN_HIDEREADONLY|OFN_ALLOWMULTISELECT,
    _T("Text Files (*.txt)|*.txt|All Files (*.*)|*.*||"))
{
    m_ofn.lpstrTitle = _T("Select a file or folder");
}

//////////////////
// Initialize dialog.
// For Explorer, this is actually a child of the main dialog.
//
BOOL CMyOpenDlg::OnInitDialog()
{
    // subclass controls...
    m_edit1.SubclassDlgItem(IDC_MYDLGINFO1, this);
    m_edit2.SubclassDlgItem(IDC_MYDLGINFO2, this);

    // initialize helper
    m_dlghelper.Init(this);

    return CFileDialog::OnInitDialog();
}

//////////////////
// Select all text files.
//
void CMyOpenDlg::OnSelectAll()
{
    CFileDlgHelper& fdh = m_dlghelper;
    CListCtrl* plc = fdh.GetListCtrl();
    for (int i=0; i<plc->GetItemCount(); i++) {
        CString fn = fdh.GetItemPathName(i);
        if (IsTextFileName(fn)) {
            plc->SetItemState(i,LVIS_SELECTED,LVIS_SELECTED);
        }
    }
    plc->SetFocus();
}

//////////////////
// Update "Select All" button: disable if no .txt files.
//
void CMyOpenDlg::OnUpdateSelectAll(CCmdUI* pCmdUI)
{
    CFileDlgHelper& fdh = m_dlghelper;
    CListCtrl* plc = fdh.GetListCtrl();
    for (int i=0; i<plc->GetItemCount(); i++) {
        CString fn = fdh.GetItemPathName(i);
        if (IsTextFileName(fn)) {
            pCmdUI->Enable(TRUE);
            return;
        }
    }
    pCmdUI->Enable(FALSE);
}

//////////////////
// DoModal override: use my template
//
int CMyOpenDlg::DoModal()
{
    // Add my custom dialog to bottom
    m_ofn.lpTemplateName = MAKEINTRESOURCE(IDD_MYOPENDLG);
    m_ofn.Flags |= OFN_ENABLETEMPLATE;
    return CFileDialog::DoModal();
}

//////////////////
// User selected new file (CDN_SELCHANGE): show info and filter file names
//
void CMyOpenDlg::OnFileNameChange()
{
    ShowFileInfo();
    FilterFiles();
}

//////////////////
// User selected new folder (CDN_FOLDERCHANGE): show info and filter
// file names
//
void CMyOpenDlg::OnFolderChange()
{
    ShowFileInfo();
    FilterFiles();
}

//////////////////
// User selected new file type (from drop down)
//
void CMyOpenDlg::OnTypeChange()
{
    ShowFileInfo();
}

BOOL CMyOpenDlg::OnNotify(WPARAM wp, LPARAM lp, LRESULT* pRes)
{
    OFNOTIFYEX& of = *(OFNOTIFYEX*)lp;
    switch(of.hdr.code) {
    case CDN_INCLUDEITEM:
        *pRes = OnIncludeItem((IShellFolder*)of.psf,
                (LPCITEMIDLIST)of.pidl);
        return TRUE; // handled
    }
    return CFileDialog::OnNotify(wp, lp, pRes);
}

//////////////////
// This doesn't work for ordinary file system files. See explanation in
// documentation for CDN_INCLUDEITEM
//
BOOL CMyOpenDlg::OnIncludeItem(IShellFolder* ishf, LPCITEMIDLIST pidl)
{
    CString fn = m_dlghelper.GetDisplayNameOf(ishf, pidl,
        SHGDN_NORMAL|SHGDN_FORPARSING);
    return Default();
}

//////////////////
// Common helper: show information in the preview and debug panes
//
void CMyOpenDlg::ShowFileInfo()
{
    CFileDlgHelper& fdh = m_dlghelper;
    CString path = GetPathName();
    CString fldr = GetFolderPath();

    // Create debug message
    //
    CString s;
    s.Format(_T("GetPathName=%s\nGetFolderPath=%s\n"),
        (LPCTSTR)path, (LPCTSTR)fldr);

    CListCtrl* plc = fdh.GetListCtrl();
    s += _T("Selected:\n");
    int nSelected = 0;
    POSITION pos = plc->GetFirstSelectedItemPosition();
    while (pos) {
        int i = plc->GetNextSelectedItem(pos);
        CString temp;
        temp.Format(_T(" %s %s = %s\n"),
            (LPCTSTR)fdh.GetItemText(i),
            fdh.IsItemFolder(i) ? _T("(FOLDER)") : _T(""),
            (LPCTSTR)fdh.GetItemPathName(i));
        s += temp;
        nSelected++;
    }
    AddText(m_edit2, s, TRUE);

    // Create preview text
    //
    s.Empty();
    if (nSelected==1 && IsTextFileName(path)) {
        s = GetTextPreview(path);
    } else if (nSelected>1) {
        s = _T("[Multiple selected]");
    } else if (nSelected==0) {
        s = _T("[None selected]");
    }
    AddText(m_edit1,s,TRUE);
}

//////////////////
// Preview: Read first 256 characters of text file
//
CString CMyOpenDlg::GetTextPreview(LPCTSTR pszPath)
{
    const int BUFLEN=256;
    char buf[BUFLEN];
    CString text;
    CFile f;
    if (f.Open(pszPath,CFile::modeRead)) {
        int len = f.Read(buf, BUFLEN);
        text += buf;
        if (len==BUFLEN)
            text += _T("...");
    }
    return text;
}

//////////////////
// Helper adds text to an edit control that I've added to the open dialog
//
void CMyOpenDlg::AddText(CEdit& wndEdit, LPCTSTR lpText, BOOL bClear)
{
    if (wndEdit) {
        // Convert \n to \n\r for Windows brain-damaged edit control !&#%!
        // It's 2003, and I'm still writing code like this!
        //
        LPCTSTR src = lpText;

        TCHAR buf[1024];
        TCHAR* dst = buf;
        TCHAR* endbuf = buf + sizeof(buf) - 1;

        while (*src && dst < endbuf) {
            if (*src == '\n')
                *dst++ = '\r';
            *dst++ = *src++;
        }
        *dst = 0;

        wndEdit.SetSel(bClear ? 0 : -1, -1);   // end of edit text
        wndEdit.ReplaceSel(buf);               // append string..
        wndEdit.SetSel(0,0);

    }
}

//////////////////
// Dialog was sized: reposition my controls. Zzzzz.
//
void CMyOpenDlg::OnSize(UINT nType, int cx, int cy)
{
    CWnd* pDlg = GetParent();
    CRect rcDlg;
    pDlg->GetWindowRect(&rcDlg);

    // important to adjust my own size in case of places bar!!
    SetWindowPos(NULL,0,0,rcDlg.Width(),rcDlg.Height(),
        SWP_NOZORDER|SWP_NOREPOSITION);

    CWnd* pCancel = pDlg->GetDlgItem(IDCANCEL);
    ASSERT(pCancel);
    CRect rcCancel;
    pCancel->GetWindowRect(&rcCancel);

    int cxRightMargin = rcDlg.right - rcCancel.right;

    CWnd* pSelAll = GetDlgItem(ID_SELECT_ALL);
    ASSERT(pSelAll);
    CRect rc;
    pSelAll->GetWindowRect(&rc);
    rc.left = rcDlg.right - cxRightMargin - rcCancel.Width();
    rc.right = rc.left + rcCancel.Width();
    rc.bottom= rc.top  + rcCancel.Height();
    ScreenToClient(&rc);
    pSelAll->SetWindowPos
        (NULL,rc.left,rc.top,rc.Width(),rc.Height(),SWP_NOZORDER);

    for (int i = IDC_MYDLGINFO1; i<= IDC_MYDLGINFO2; i++) {
        CWnd* pInfoWnd = GetDlgItem(i);
        ASSERT(pInfoWnd);
        pInfoWnd->GetWindowRect(&rc);
        rc.right = rcDlg.right - cxRightMargin -1;
        ScreenToClient(&rc);
        pInfoWnd->SetWindowPos
            (NULL,rc.left,rc.top,rc.Width(),rc.Height(),0);
    }
}

//////////////////
// Filter files: iterate listview, removing files for which IncludeFile
// returns FALSE.
//
void CMyOpenDlg::FilterFiles()
{
    CFileDlgHelper& fdh = m_dlghelper;
    CListCtrl* plc = fdh.GetListCtrl();
    for (int i=0; i<plc->GetItemCount(); i++) {
        CString fn = fdh.GetItemPathName(i);
        if (!IncludeFile(fn)) {
            plc->DeleteItem(i);
            i—;
        }
    }
}

//////////////////
// Remove files that begin with _ (underbar).
//
BOOL CMyOpenDlg::IncludeFile(CString pathname)
{
    if (pathname.Find("\\_")>=0) {
        TRACE("CMyOpenDlg::IncludeFile: removing %s\n",(LPCTSTR)pathname);
        return FALSE;
    }
    return TRUE;
}

For my test program, CMyOpenDlg::IncludeFile removes file names that begin with _ (underbar) or live in a folder whose name begins with _. FilterFiles and IncludeFile are straightforward. The only mystery is: who calls FilterFiles? You have to call FilterFiles any time the open dialog populates its listview with file names. This can happen when the user navigates to a new folder, or changes the file name or type (for example, by selecting "*.bmp" instead of "*.jpg", or typing "foo*" in the file name field). In the first case, Windows sends CDN_FOLDERCHANGE; in the second, CDN_SELCHANGE. So CMyDlg calls FilterFiles whenever it gets one of these notifications. Figure 4 shows the details.

If you're thinking CDN_INCLUDEITEM is kind of lame, be glad you're not programming in the Microsoft® .NET Framework. There, you can't customize the file open dialog at all because OpenFileDialog is sealed, which is a fancy way to say the Redmondtonians won't let you derive from it. As soon as I'm appointed Grand Programming Poobah, I promise one of the first action items on my legislative agenda will be to remove "sealed" from the C# vocabulary. The keyword will be shipped to the Museum of Restrictive Programming Techniques where it will be displayed in a glass case. Alas, my appointment may not be soon forthcoming, so until then, happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


Talking To...: Eric Brown Discusses SQL Server—Past, Present, and Yukon

Talking To...
Eric Brown Discusses SQL Server—Past, Present, and Yukon

It's been a busy year for SQL Server. Not only did it celebrate its 10th anniversary last winter, Microsoft is preparing a new version, code-named "Yukon," that's currently undergoing a round of early beta testing. MSDN Magazine recently talked to Eric Brown, a product manager for SQL Server, about the product's past and future.

MSDN This year was the 10th anniversary of SQL Server. How did you celebrate it?

Brown We marked the 10th anniversary with celebrations at the Professional Association of SQL Server (PASS) in November last year, and at Tech•Ed in Dallas, with parties and games. You can learn about the 10 years of innovation by visiting "SQL Server 10-Year Anniversary".

MSDN Explain the significance of writing stored procedures in any language that compiles to Microsoft Intermediate Language (IL) and the significance of native XML support.

Brown SQL Server "Yukon" is empowering a new era of database applications. We see more and more customers asking for greater interoperability, greater flexibility and choice in programming models, and the ability to use developer skills in more areas. The integration of SQL Server "Yukon" with the .NET Framework represents the delivery of a more symmetrical programming model between the middle and database tiers. Writing database logic in either Transact SQL (T-SQL) or any .NET-compliant language means greater productivity for developers. Not only will they be able to take advantage of the Microsoft .NET Framework, they will also be able to create data types, functions, and aggregates—thus extending the data tier beyond the types and functions that are included in SQL Server.

XML integration is as important as the .NET Framework integration. SQL Server "Yukon" will ship with a full-fledged XML data type that will support XSD for schema validation. Moreover, support for existing and emerging open standards such as HTTP, XML, SOAP, and XQuery. This will allow SQL Server "Yukon" to facilitate communication across extended enterprise systems.

MSDN Are there any major trends you see emerging?

Brown More and more, we see customers using XML. I've talked to customers who are using Web Services to overcome application interoperability hurdles. I've seen some of our large customers using the XML features currently available in SQL Server 2000—the FOR XML clause and the Web Service Toolkit, for example—to extend their databases into the Web Services frontiers. Another big trend is in the area of reporting applications. The demand for the SQL Server 2000 Reporting Services Beta has been overwhelming. Customers are looking to improve and extend their current applications, and Reporting Services provides a natural extension.

MSDN When can we expect to see Yukon?

Brown SQL Server "Yukon" has just entered Beta 1. We will do more betas as we get feedback from customers. Our current goal is to have a public beta in the first half of 2004.

MSDN What kind of community efforts are you planning to set up around SQL Server?

Brown Community is important to us. We want to make sure that we're providing those who run and develop against SQL Server easy access to the people and information that will make their jobs simpler. We're working with PASS to hold SQL Server University classes on different SQL Server topics. Plus, don't be surprised if you see people from the SQL Server development team answering your questions in the newsgroups. We've set up a community lounge so that our customers get answers directly from the designers, program managers, developers, and testers who build SQL Server at Microsoft. Finally, we're working closely with our MVPs and the MSDN Regional Directors to make sure they're ready to answer all of the community's questions around both SQL Server 2000 and SQL Server "Yukon".

MSDN A lot of readers got hit with the Slammer virus last year. What advice can you give to help them keep servers secure?

Brown Customers should make sure that they're aware of the tools available for protecting themselves from Slammer and also from future attacks. For Slammer, we rereleased SQL Server so that by default you will not get infected by the worm. Also, we have a number of tools available for download from the Web site that will find and remove the Slammer worm from existing SQL Server installations. Finally, customers should use tools like Windows Update, the SUS Server, and the Microsoft Baseline Security Analyzer (MBSA). MBSA is a great tool to ensure that your SQL Server systems have the right security practices, and it works with Windows, IIS, and Exchange Server.