 MSDN Magazine July 2003

MSDN Magazine July 2003

DirectX 9.0: Introducing the New Managed Direct3D Graphics API in the .NET Framework

DirectX 9.0 is the latest evolution of the Microsoft 3D graphics technology for Windows. Direct3D, a major component of the DirectX Graphics subsystem, has evolved so rapidly in the last few years that the underlying programming paradigm has changed quite a bit from its origin. This article introduces the fundamental concepts of the unmanaged Direct3D architecture and illustrates how the managed Direct3D layer abstracts the unmanaged layer. Also, the author describes the Geometry, Texture, Device, and other classes and uses code from the Samples SDK.

Yahya H. Mirza and Henry da Costa GDI+: A Primer on Building a Color Picker User Control with GDI+ in Visual Basic .NET or C#

Although most developers and APIs use the RGB scheme when working with colors, it's not the only available way to represent or select colors. For instance, the standard Windows color-selection dialog box allows you to work with the HSL color scheme in an indirect way. In this article, the author describes several color selection schemes, and uses GDI+ (via the System.Drawing namespace) to create a component that makes it possible for your own applications to provide a simpler, friendlier color chooser. Along the way, you'll get tips to help you use GDI+ in your own apps.

Ken Getz Vector Graphics: Build Flexible, Lightweight XML-Based Images for ASP.NET Using Scalable Vector Graphics

Scalable Vector Graphics (SVG), a W3C graphics standard built around XML, is one of several vector graphics technologies that allows fast, lightweight drawings such as charts and graphs to be rendered on the fly in an appropriate viewer. There are many advantages to such vector graphics, including conservation of bandwidth and storage media, and flexibility. This article explains these benefits and shows you how to easily add powerful, dynamic, interactive visual elements to your Web applications.

Dennis Forbes Web Services: Extend the ASP.NET WebMethod Framework by Adding XML Schema Validation

WebMethods make the development of XML Web Services easier by encapsulating a good deal of functionality, but there is still a lot of underlying XML processing that you have to be responsible for. For example, WebMethods do not validate messages against the implied schema. Because they are not validated, the response that's returned can result in unintended consequences. To address this, the authors extend the WebMethod framework by adding XML Schema validation through a custom SoapExtension class.

Aaron Skonnard and Dan Sullivan Real-World XML: Manipulate XML Data Easily with the XPath and XSLT APIs in the .NET Framework

XPath is emerging as a universal query language. With XPath, you can identify and process a group of related nodes in XML-based data sources. XPath provides an infrastructure that is integral to XML support in the .NET Framework. The XPath navigation model is even used under the hood of the XSLT processor. In this article, the author reviews the implementation details of the XPath navigator and the XSLT processor and includes practical examples such as asynchronous transformations, sorted node-sets, and ASP.NET server-side transformations.

Dino Esposito Editor's Note: Check Out the Tablet PCs

Since we founded MSDN Magazine back in 1932, we've seen generation after generation of technological advances, from the early TV test transmissions of Felix the Cat dolls that made up the bulk of our coverage in the 1940s to the special 132-page bumper issue that coincided with the launch of "New" Coke in 1985.

New Stuff: Resources for Your Developer Toolbox

TeaLeaf Technology has announced the availability of TeaLeaf IntegriTea with Session to Script. This new feature allows scripts for testing products, such as the Microsoft® Web Application Stress tool, to be automatically generated directly from the Web sessions of real users.

Theresa W. Carey Web Q&A: Accessible Images, Image Format Converter, and More

Edited by Nancy Michell Data Points: Managing Hierarchical Inserts in ASP.NET and ADO.NET

John Papa Cutting Edge: Working with Images in the .NET Framework

Dino Esposito The XML Files: XML Namespace Collisions, XmlNodeList and Deserialization, and More

Aaron Skonnard .NET Column: Calling Win32 DLLs in C# with P/Invoke

Jason Clark Advanced Basics: Passing Data Between Objects in an Application

Ken Spencer Design Patterns: Architecture of an Autonomous Application

Sten Sundblad and Per Sundblad C++ Q&A: Parent and Child Window Captions, More MoveDlg

Paul DiLascia Talking To…: Ken Getz Talks About His Role as a Microsoft MVP


DirectX 9.0: Introducing the New Managed Direct3D Graphics API in the .NET Framework

DirectX 9.0

Introducing the New Managed Direct3D Graphics API in the .NET Framework

Yahya H. Mirza and Henry da Costa
This article assumes you're familiar with C++ and DirectX
Level of Difficulty 1 2 3
SUMMARY

DirectX 9.0 is the latest evolution of the Microsoft 3D graphics technology for Windows. Direct3D, a major component of the DirectX Graphics subsystem, has evolved so rapidly in the last few years that the underlying programming paradigm has changed quite a bit from its origin. This article introduces the fundamental concepts of the unmanaged Direct3D architecture and illustrates how the managed Direct3D layer abstracts the unmanaged layer. Also, the author describes the Geometry, Texture, Device, and other classes and uses code from the Samples SDK.

Contents
The Direct3D Architecture
The Direct3D Pipeline
Unmanaged Direct3D
Direct3D Extensions
Managed Direct3D
The Device Class
Graphics State Classes
Geometry-related Classes
Texture Classes
Managed D3DX Library
The Managed DirectX 9.0 Samples Framework
The GraphicsSample Class
Initialization and Cleanup
Frame Updating and Rendering
Conclusion

A
fundamental challenge of a hardware-accelerated graphics API is to enable application developers to take advantage of the rapid technology advances occurring in the 3D hardware space while allowing a certain amount of compatibility and uniformity across graphics hardware solutions. One way to do this is to define a standard by committee and then have each vendor support that standard. Graphics hardware vendors can innovate and create proprietary extensions through an agreed upon extension mechanism. Over time, the hardware vendor can lobby the standards body to accept their proprietary extension as part of the standard. OpenGL version 1.1 is an example of this approach to hardware interoperability. One limitation is that it can take a long time to get vendor-specific innovations incorporated into a multi-vendor standard, thereby taking the risk that the standard will become obsolete.

In DirectX® 9.0, the features of DirectDraw® and Direct3D® are combined into a single API called DirectX Graphics. The Direct3D portions of this component will be our primary focus in this article. In Microsoft® Direct3D, the programmer has two options: rely on a fixed-function pipeline or a programmable pipeline. The fixed-function pipeline itself depends on existing algorithms standardized by Direct3D. These fixed functions are exposed through a fixed set of enumeration values similar to OpenGL. This implies that the fixed-function pipelines of both Direct3D and OpenGL use internal switch statements. Some of the cases that correspond to the enumeration values in the switch statement may be hardware accelerated, based on the capabilities of the graphics card on which the runtime relies. In Direct3D, when using the fixed-function pipeline, the programmer first checks with the runtime to see if the graphics card supports a particular capability.

Because some graphics cards do not support all of the features exposed through Direct3D, a mechanism is provided in Direct3D to probe the graphics hardware. If a particular graphics capability is not supported by the hardware, the check will fail, allowing the programmer to look for a different hardware-accelerated algorithm. The key point to remember is that the Direct3D fixed-function pipeline exposes hardware-accelerated features. Although Direct3D has a software-only emulation mode called the reference device, it is designed for debugging and feature testing purposes—not for shipping applications.

The other, more interesting approach to the problem of hardware and software coevolution is the programmable pipeline. In the programmable pipeline, rather than picking a predefined enumeration value and asking Direct3D to perform the algorithm, the programmer can define his own algorithm. The runtime will dynamically compile the algorithm for the underlying graphics hardware. In this case, the Direct3D runtime has a just-in-time (JIT) compiler, which is an explicit part of the hardware device driver. Hardware vendors are responsible for providing a JIT compiler for their particular graphics hardware. Direct3D thus serves as a graphics virtual machine, which effectively virtualizes the graphics processor (GPU) with a custom graphics instruction set.

The Direct3D Architecture

Although both the managed and unmanaged Direct3D programming layers are exposed through a series of objects, you should not consider them an application-level programming framework. The primary role of the Direct3D architecture is to provide the base functionality for a higher-level solution such as a game engine or a scene graph API. To help implement these solutions, the Direct3D extension library explicitly provides additional functionality. To best understand the Direct3D architecture you must understand not only the functionality that is being abstracted but also how that functionality is organized and exposed. In the next several sections, we'll provide an overview of the basic elements of the Direct3D architecture and discuss how they are exposed through both the unmanaged COM API and the managed .NET abstraction layer.

The Direct3D Pipeline

As is common to computer hardware architectures, two performance optimization techniques used in 3D graphics architectures are pipelining and parallelizing. The algorithms exposed through Direct3D are logically organized into a pipeline. The Direct3D pipeline is illustrated in Figure 1.

Figure 1 Direct3D Graphics Pipeline (Close Figure 1)

The Direct3D pipeline should be viewed as a set of algorithms that operate on 3D geometric quantities, which in the case of Direct3D are predefined vertices and primitives. The main purpose of the pipeline is to convert geometric data into an image that's rendered on the video display. The Direct3D tessellation stage is used to tessellate (convert to triangles) a fixed set of higher-order primitives that Direct3D predefines, which include triangle patches, rectangle patches, and N patches (although triangle patches remain the most common form of geometry). Currently, the tessellation stage is not programmable, so Direct3D does not expose any mechanisms to enable procedural geometry generation on the graphics hardware. Procedural geometry provides a large number of benefits with respect to minimizing the data that is sent across the bus. In the near future, you are likely to see hardware support for a programmable tessellation stage.

The transform and lighting stage of the pipeline transforms vertex positions and normals from the model coordinate system to the world and camera coordinate systems. This occurs through the world and view transformations. Per-vertex lighting calculations are performed to determine the specular and diffuse color components. The vertex positions are then transformed by the projection transformation to create a perspective, orthographic, or other type of projection. Although the fixed-function pipeline still exposes these transform and lighting algorithms in the same API convention as before, in most graphics cards they are being implemented in the microcode of the graphics processor as opposed to being a set of hardware-accelerated algorithms. In the Radeon 9700 processor, for example, the entire transform and lighting module of the past can and should be implemented in the programmable pipeline as vertex shaders.

To improve performance during the rasterization stage, any vertices belonging to objects that are not visible to the camera are clipped. In addition, back-face culling may be performed to avoid rasterizing triangles that are facing away from the camera. Furthermore, attribute evaluation is performed to configure and select the actual algorithms that will be used during the rasterization stage. Finally, rasterization is performed to actually render the pixels.

In the pixel processing stage, you have the option of using fixed-function multi-texturing or programmable pixel shaders to determine the color value of a pixel. Fixed-function multi-texturing is exposed through a cascade of texture stages with each stage enabling a fixed set of operations to be performed on the color and alpha values of a pixel. Pixel shaders provide much more flexibility by exposing the operations performed on the color and alpha values through a custom assembly language. The algorithms implemented at the pixel processing stage include bump mapping, shadowing, environment mapping, and so on.

Frame buffer processing involves a set of memory regions known as the render surface, depth buffer, and the stencil buffer. During this stage, a series of calculations are performed to determine values such as depth, alpha, and stencil. The depth buffer is another rendering optimization used to remove hidden lines and surfaces. A depth test to determine which pixels are hidden and do not need to be rendered can be determined using either a z-buffer or a w-buffer, with each algorithm having its own pros and cons. Frame buffer processing enables a number of effects such as transparency, fog, and shadows.

One final point that needs to be stressed about the Direct3D pipeline is that its behavior can be modified through the graphics state. The graphics state is used to configure the many transformation, lighting, rasterization, pixel processing, and frame buffer processing algorithms provided by Direct3D for the purpose of rendering a frame. These states include the render, transform, sampler, and texture stage states.

Unmanaged Direct3D

The primary object that manages DirectX graphics is the Direct3D object. The Direct3D object is created through the Direct3DCreate9 function, which is the only global function exposed by the core Direct3D API. The Direct3D object implements the IDirect3D9 interface and is responsible for determining the details of a particular device. Besides device enumeration, the IDirect3D9 interface is responsible for creating a Direct3DDevice object through its CreateDevice factory method.

A Direct3DDevice object implements the IDirect3DDevice9 interface and serves as the primary workhorse of Direct3D graphics. The Direct3DDevice9 interface abstracts the entire pipeline illustrated in Figure 1. It provides methods exposing the numerous algorithms for transformation, lighting, rasterization, pixel, and frame buffer processing, supplied by a particular Direct3D-compliant hardware device. The methods in the IDirect3DDevice9 interface can be organized into the following categories: a set of properties that configures the Direct3D graphics pipeline or provides information about its current state, factory methods that create other objects in the Direct3D architecture, or a set of methods that execute the actual graphics algorithms.

The state of the Direct3D pipeline can be configured through a set of accessor methods on the IDirect3DDevice9 interface. The GetRenderState and SetRenderState methods retrieve and/or set the render state value for a device. The unmanaged D3DRENDERSTATETYPE enumerated type provides you with a set of options that can be used to configure the current render state. The GetTextureStageState and SetTextureStageState methods retrieve and/or set the state value for the currently assigned texture. The unmanaged D3DTEXTURESTAGESTATE enumerated type also provides the developer with a set of options that can be used to configure the individual texture stages. Additional attributes of the graphics state can be configured by selecting options from the unmanaged D3DTRANSFORMSTATETYPE and D3DSAMPLERSTATE enumerated types.

Geometry in unmanaged Direct3D is used through the VertexBuffer and IndexBuffer COM objects. These objects are created through factory methods on the IDirect3DDevice9 interface. A VertexBuffer object implements the IDirect3DVertexBuffer9 interface and stores an array of user-defined vertices. These vertices can be comprised of data needed for a particular application and can be declared using the Direct3D flexible vertex format. The Direct3D flexible vertex format is used through a set of macros. A fundamental limitation of flexible vertex formats is that they can only use a single stream of data and thus can be wasteful with memory. Flexible vertex formats are usually used in conjunction with the fixed-function pipeline. An IndexBuffer object implements the IDirect3DIndexBuffer9 interface and stores an array of indices into the VertexBuffer. An IndexBuffer provides an optimization to save memory by reusing shared vertices.

Direct3D Extensions

The unmanaged Direct3D extensions (D3DX) utility library provides a large body of interfaces, functions, macros, and support classes for a full range of 2D, 3D, and 4D vector math operations. Additionally, functions to support procedural textures, bump mapping, and environment mapping are provided. An important subset of Direct3D is its .X file format. Functions are provided for working with meshes, as well as animation and textures. A more recent feature of D3DX is its support for shaders and effect files. A runtime assembler is provided for shaders through the D3DXAssembleShader, D3DXAssembleShaderFromFile, and D3DXAssembleShaderFromResource API functions.

Managed Direct3D

Managed Direct3D has become an important part of the core Direct3D runtime, as opposed to an extension library like D3DX. Thus, an attempt was made to keep the managed abstractions as close as possible to the original unmanaged abstractions. To better understand managed Direct3D, it is useful to investigate how the managed layer abstracts unmanaged Direct3D. For the most part, managed Direct3D provides a one-to-one mapping of the interfaces, structures, and enumerations in the unmanaged layer to the classes, structures, and enumerations in the managed layer. To explicitly use some of the new features of managed code, such as class implementation inheritance, C# properties, indexers, and events, a few of the abstractions in the managed layer are exposed in a slightly different way than their unmanaged counterparts. In the next few sections, we will introduce the fundamental classes that are exposed by managed Direct3D.

The Device Class

Managed Direct3D provides a class-based abstraction of a device, which is implemented using the unmanaged IDirect3DDevice9 interface. A programmer writing managed Direct3D code uses methods of the Device class to access other Direct3D objects, to set the various graphics states, and to render the geometry. To perform these functions, the Device class provides many properties, methods, and events. The most important methods of the Device class, as called by 3D applications, are listed in Figure 2.

Figure 2 Device Class Methods (Close Figure 2)

Method | Description
Clear | Clears the viewport in preparation for another frame of rendering
BeginScene | Prepares the device to render a frame of primitives; BeginScene must be called before any primitive is rendered for the frame
EndScene | Signals to the device that all the primitives have been rendered for a frame; EndScene must be called after all the primitives are rendered for the frame
DrawPrimitives | Renders a primitive
DrawIndexedPrimitives | Displays the buffer that was
Present | rendered into and prepares the next buffer for rendering; Present is called after EndScene and before the next BeginScene (for the next frame)
SetStreamSource | Binds a vertex buffer to a device data stream; vertex shaders may use vertex data coming from different streams; for example, vertex positions may be in one stream, normals in another
GetVertexShaderConstant, SetVertexShaderConstant, GetPixelShaderConstant, SetPixelShaderConstant | Gets/sets the constant values that are accessible by vertex/pixel shaders
GetTexture, SetTexture | Gets/sets the texture associated with a given texture stage
GetTransform, SetTransform | Gets/sets the world, view, projection or other transform; transforms are applied to vertex positions and normals, and/or to texture coordinates

With respect to object activation, one view of why unmanaged Direct3D uses factory methods is simply that COM does not support a new operator. The standard activation pattern for COM is factory-based construction. The CoCreateInstance function simulates a new operator for COM, but it is just a wrapper that internally uses a COM factory. Another rationale for using factories as opposed to a new operator is that when a new operator is used, you bind a particular implementation of an object into the client's code; thus extensive use of the new operator can lead to versioning problems. Factory methods are often used to decouple the client or user of the code from the creator of the code. The unmanaged COM IDirect3DDevice9 interface provides 14 explicit factory methods to create the objects that are required when using Direct3D, such as pixel and vertex shaders, and vertex and index buffers.

Currently, managed Direct3D relies heavily on class construction using the new operator. Although the managed Device class only provides three factory methods, it is interesting to note the one-to-one correlation between the unmanaged IDirect3DDevice9 factory methods and the public properties exposed by the Device class. The following lines of code illustrate the three factory methods which are exposed by the managed Device class, each taking multiple arguments:

public Surface CreateRenderTarget(...);
public Surface CreateDepthStencilSurface(...);
public Surface CreateOffscreenPlainSurface(...);

Method overloading is used heavily in managed Direct3D. The next code example illustrates the unmanaged Device.ColorFill method and its managed overloaded wrappers:

typedef DWORD D3DCOLOR;

// Methods from the unmanaged IDirect3DDevice9 interface.
HRESULT ColorFill(IDirect3DSurface9 *pSurface, CONST RECT *pRect,
    D3DCOLOR color);

// Methods from the managed Device class.
public void ColorFill(Surface surface, Rectangle rect, Color color);
public void ColorFill(Surface surface, Rectangle rect, int color);

Unmanaged Direct3D, like traditional graphics systems, requires color values to be passed internally as 32-bit integer values. On the other hand, the Microsoft .NET Framework explicitly provides the System.Drawing.Color class to deal with ARGB color values and contains several methods to convert between representations. As a convenience to programmers who have been accustomed to using integer values to represent color, managed Direct3D provides method overloads using both representations wherever a color argument is required.

The unmanaged IDirect3DDevice9 interface exposes both read-only and read-write properties, both with and without arguments. In the current version of the .NET Framework, properties do not take arguments. Therefore, when mapping from unmanaged Direct3D to managed Direct3D, properties that do not take arguments map over nicely to C# properties, whereas unmanaged Direct3D properties that take arguments are simply exposed as Get and Set methods. Figure 3 lists some of the more important properties of the Device class.

Figure 3 Device Class Properties (Close Figure 3)

Property | Description
DeviceCaps | Gets a struct representing the capabilities of the hardware; this is the property to query when determining whether the hardware supports the particular features that an application may require
Indices | Gets/sets the index buffer to use for rendering primitives
Material | Gets/sets the material to use in rendering
Lights | Gets the collection of lights that can be activated for rendering
RenderState | Gets the collection of render states that are used to control the different stages of the Direct3D pipeline
TextureState | Typically used with an indexer, gets the texture state for the indexed texture stage; the texture state controls how a texture is applied
SamplerStates | Typically used with an indexer, gets the sampler state for the indexed texture stage; the sampler state controls how a texture is sampled
VertexDeclaration | Gets/sets a description of the vertex format being used with a vertex shader
VertexFormat | Gets/sets a description of the vertex format being used with the fixed-function pipeline
Viewport | Gets/sets the rectangular region on the device on which to render
VertexShader, PixelShader | Gets/sets the vertex/pixel shader to use for rendering

Finally, the managed Direct3D Device class provides five explicit events for interacting with a device. A Direct3D-based application can source events to a Device by starting an application, resizing a window, changing from windowed to full screen mode, or exiting the application. It is interesting to see the .NET delegate-based events pattern used by Direct3D. For each event declaration on the Device class, add, remove, and raise methods are provided as well as a private instance variable storing the event handler. In C#, syntactic sugar for client-side event registration and unregistration is provided through the += and -= operators, respectively (see Figure 4).

Figure 4 Device Event Handling (Close Figure 4)
public class Device : MarshalByRefObject, IDisposable
{
   // Lots of additional Properties, Methods, and Events.
   private   EventHandler DeviceCreated;
   public    void  add_DeviceCreated(EventHandler eh);
   public    void  remove_DeviceCreated(EventHandler eh);
   protected void  raise_DeviceCreated(object i1, EventArgs i2);
   public    event EventHandler DeviceCreated;
   // Similar pattern for the DeviceLost, DeviceReset, DeviceResizing,
   // and Disposing events.
}

// Client code.
PresentParameters presPar = new PresentParameters();
device = new Device(0, DeviceType.Hardware, this,
         CreateFlags.SoftwareVertexProcessing, presPar);
device.DeviceCreated += new System.EventHandler(this.OnCreateDevice);
this.OnCreateDevice(device, null);

Graphics State Classes

As discussed earlier, the graphics state for unmanaged code is configured through methods on the IDirect3DDevice9 interface. For example, in unmanaged code, the transform state can be directly accessed through the GetTransform and SetTransform methods, both of which take an enumeration value of type D3DTRANSFORMTYPE.

// From unmanaged IDirect3DDevice9 interface:
HRESULT GetTransform(D3DTRANSFORMSTATETYPE State, D3DMATRIX *pMatrix);
HRESULT SetTransform(D3DTRANSFORMSTATETYPE State,
    CONST D3DMATRIX *pMatrix);

Managed Direct3D provides some abstraction over the unmanaged layer, making the client-side code a bit cleaner. The Device class shown here illustrates a set of read-only properties:

public class Device : MarshalByRefObject, IDisposable
{
   // Additional properties, methods and events.
   public Transforms    Transform    { get; }
   public RenderStates  RenderState  { get; }
   public SamplerStates SamplerState { get; }
   public TextureStates TextureState { get; }
}
The properties return instances of Transforms, RenderStates, SamplerStates, and TextureStates classes. These instances can then be used to configure the graphics state for managed Direct3D. For example, the Device class does not directly expose a Transform property taking an enumeration value. The managed Device class provides a property returning a Transforms object. The Transforms object then exposes a set of properties to access the current world, view, and projection matrices, which themselves are elements from the managed TransformType enumeration.

The RenderStates class provides a large number of properties used to configure the particular algorithms selected for the rasterization stage. The following client-side code shows how the state of the managed Direct3D pipeline is set.

// cull, spec, dither states.
device.RenderState.CullMode        = Cull.None;
device.RenderState.SpecularEnable  = false;
device.RenderState.DitherEnable    = false;
// Filter states.
device.SamplerState[0].MagFilter   = TextureFilter.Linear;
device.SamplerState[0].MinFilter   = TextureFilter.Linear;

Notice the use of indexers to access the stages of the multi-texture cascade for the sampler states.

The sampler state specifies the filtering, tiling, and texture-addressing modes used for a particular stage and are exposed through properties on the Sampler class. Filtering is used to specify how an image is mapped to a particular Direct3D primitive. Depending on the size of the image (texture) and the size of the on-screen primitive it needs to be mapped to, the image may need to be magnified or reduced. Direct3D provides three filtering methods: MipFilter, MinFilter, and MagFilter, each with modes that can be configured by the TextureFilter enumeration. Tiling is used to assemble a set of individual textures into a single texture and reduces the need to switch textures. Tiling is supported through a set of properties on the Sampler class. Finally, the texture-addressing mode defines the action for texture coordinates outside the [0,1] boundary through elements of the TextureAddress enumeration:

public class SamplerStates
{
   private  Sampler[] m_lpTex;
   private  bool pureDevice;
   public   Sampler get_SamplerState(int index);
   internal SamplerStates(Device dev, bool pure);
   public   Sampler this[int index] { get; }
}

The TextureStates and SamplerStates classes provide a read-only indexer, which returns a TextureState instance for a particular stage. The TextureState class defines properties, which can get or specify the elements of a particular texture stage state. Properties such as ColorArgumentn or AlphaArgumentn (where n can be 0, 1, or 2) can be elements from the TextureArgument enumeration, while ColorOperation or AlphaOperation can be of a TextureOperation enumeration, as shown here:

// color = tex mod diffuse.
device.TextureState[0].ColorArgument1 = TextureArgument.TextureColor;
device.TextureState[0].ColorOperation = TextureOperation.Modulate;
device.TextureState[0].ColorArgument2 = TextureArgument.Diffuse;
// alpha = select texture alpha.
device.TextureState[0].AlphaArgument1 = TextureArgument.TextureColor;
device.TextureState[0].AlphaOperation = TextureOperation.SelectArg1;
device.TextureState[0].AlphaArgument1 = TextureArgument.Diffuse;

Geometry-related Classes

Geometry in Direct3D is defined as an array of vertex definitions. It is important to understand that different graphics algorithms require different information stored at each vertex. These vertex components need to be declared by the application depending on the effect that the programmer wants to implement. For example, if the programmer is interested in mapping a texture to geometry that Direct3D will render, then the programmer has to declare that at each vertex there will be tu and tv values (texture coordinates). Managed Direct3D provides two mechanisms for vertex definition: VertexFormats and VertexDeclarations. These mechanisms can be used to define additional data at each vertex such as colors, blending weights, and texture coordinates or any application-defined usage. As a convenience, the CustomVertex helper class is provided for commonly used vertex format definitions. The CustomVertex class provides 11 different vertex definitions most often used in Direct3D applications.

In managed Direct3D, the vertex format used by the application to define the layout of its vertices is accessed by the Device class' read and write VertexFormat property. The VertexFormat property accesses an element of the VertexFormats enumeration and is implemented simply by calling the internal unmanaged Direct3D Device COM object using the GetFVF and SetFVF methods on the IDirect3DDevice9 interface. The managed VertexFormats enumeration maps to the constant VertexFormat bits defined in d3dtypes.h, as shown here:

public struct Vertex
{
   public Vector3 position;
   public float tu, tv;
   public static readonly VertexFormats Format =
       VertexFormats.Position | VertexFormats.Texture1;
};
// From a typical managed Direct3D applications Render() method.
device.VertexFormat = Vertex.Format;

When using the programmable pipeline, custom vertex declarations are required. Unlike a vertex buffer defined using a flexible vertex format, the custom vertex declarations can be defined using multiple streams, thus saving bandwidth:

VertexElement[] declaration = new VertexElement[]
{
   new VertexElement(
       0,                         // short             stream
       0,                         // short             offset
       DeclarationType.Float2,    // DeclarationType   declType
       DeclarationMethod.Default, // DeclarationMethod declMethod
       DeclarationUsage.Position, // DeclarationUsage  declUsage
       0                          // byte              usageIndex
                    ),
   VertexElement.VertexDeclarationEnd
};
VertexDeclaration declaration = new VertexDeclaration(device,
                                declaration);

Geometry in managed Direct3D is usually created by handling the VertexBuffer.Created event. Figure 5 shows the geometry of a simple triangle taken from the Tutorial2 sample in the SDK and illustrates the use of a predefined custom vertex, CustomVertex.TransformedColored.

Figure 5 Initializing a Vertex Buffer (Close Figure 5)
public void OnCreateDevice(object sender, EventArgs e)
{
   // Additional code ...
   vertexBuffer.Created += new
      System.EventHandler(this.OnCreateVertexBuffer);
   this.OnCreateVertexBuffer(vertexBuffer, null);
}
public void OnCreateVertexBuffer(object sender, EventArgs e)
{
   VertexBuffer        vertexBuffer     = (VertexBuffer)sender;
   GraphicsStream      stream     = vertexBuffer.Lock(0, 0, 0);
   CustomVertex.TransformedColored[] vertices = new
      CustomVertex.TransformedColored[3];
   vertices[0].X     = 150; vertices[0].Y = 50; vertices[0].Z = 0.5f;
   vertices[0].Rhw = 1;
   vertices[0].Color = System.Drawing.Color.Aqua.ToArgb();
   vertices[1].X     = 250; vertices[1].Y = 250; vertices[1].Z = 0.5f;
   vertices[1].Rhw = 1;
   vertices[1].Color = System.Drawing.Color.Brown.ToArgb();
   vertices[2].X     = 50;  vertices[2].Y = 250; vertices[2].Z = 0.5f;
   vertices[2].Rhw = 1;
   vertices[2].Color = System.Drawing.Color.LightPink.ToArgb();
   stream.Write(vertices);
   vertexBuffer.Unlock();
}

Texture Classes

The texture-related classes in managed Direct3D inherit from the abstract class Resource and for the most part layer over the unmanaged texture-related interfaces. Managed Direct3D has three texture classes: Texture, CubeTexture, and VolumeTexture. The Texture class represents an image that can be mapped onto a primitive for rendering or used for bump mapping, normal mapping, or other effects. The fundamental function of a texture is to map a pair of texture coordinates into a value, which is usually a color value, but may be interpreted differently. The texture coordinates are normally part of the vertex data. The mapping from texture coordinates to color or other values is done during the pixel processing stage of the rendering pipeline. Direct3D allows up to eight textures to be used at a time. The CubeTexture class is a special kind of texture that's used for cubic environment mapping, a technique which allows an environment such as a room or exterior scene to appear reflected by shiny objects and which is implemented by using a texture to define an image for each of the six faces of a cube surrounding and reflected by the rendered objects. The VolumeTexture class represents color or other values that exist in a three-dimensional rectangular space. As such, it maps a triplet of texture coordinates into color or other values. The most important aspect of these classes is that they all support a set of overloaded methods for locking and unlocking textures.

A commonly used class for loading and saving textures is TextureLoader. The TextureLoader class contains methods for loading a texture, cube texture, or volume texture from a file or stream. Additional uses of the TextureLoader class include filling and filtering textures and computing normal maps. A useful helper method for creating textures is provided by the GraphicsUtility class: the file GraphicsUtility.CreateTexture method simply finds the path of a specified texture and then calls the TextureLoader.FromFile method to load it. The following code illustrates the usage of a GraphicsUtility class:

Texture texture = GraphicsUtility.CreateTexture(device,"banana.bmp");

Managed D3DX Library

The managed D3DX utility library has classes to handle meshes, textures, shaders, effects, text, and animation. Unlike the unmanaged D3DX library, however, it does not have math classes such as matrix, vector, and quaternion classes. In managed DirectX, the math classes are part of the DirectX library.

The mesh-handling classes are Mesh, SimplificationMesh, ProgressiveMesh, SkinMesh, and PatchMesh. Mesh and ProgressiveMesh inherit from the BaseMesh class. BaseMesh manages the mesh representation in memory (including vertex and index buffers), and provides a DrawSubset method to render the mesh. Other methods include a Clone method to copy the mesh, and methods to compute vertex normals and get adjacency information.

The Mesh class extends BaseMesh with methods to load a mesh from and save to an .X file (a standard DirectX file type) or binary stream. Methods for generating a mesh representing an n-sided polygon, cylinder, sphere, torus, 3D text in a given font, or even a teapot are also provided. The user may tessellate the mesh to get a smoother object by interpreting each triangle as an N-patch, simplify the mesh (by reducing the number of triangles), optimize the mesh (by reordering the triangles in order to exploit vertex caching in hardware and improve performance), or perform ray intersection with the mesh.

A SimplificationMesh object is constructed from a Mesh and has methods to reduce the number of faces (triangles) or vertices, thereby resulting in a simpler although lower-quality version of the original mesh. A ProgressiveMesh object is constructed from a Mesh. It allows the mesh to be simplified in real time to minimize the number of rendered triangles as the distance between the mesh object and the camera increases. The SkinMesh class (meshes animated by skeletons) provides methods to be used for loading a skin mesh from an .X file data object. Properties are included to expose the underlying mesh and skin information, such as the influence of each bone on the structure.

The PatchMesh class represents a higher-order surface such as an N-Patch, Triangle Patch, or Rectangle Patch surface. PatchMesh includes methods to create an N-Patch mesh from a Mesh object, load a PatchMesh from an .X file, tesselate the PatchMesh, and so on. The texture-handling classes consist mainly of the TextureLoader class described earlier, a RenderToEnvironmentMap class, and a RenderToSurface class. The RenderToEnvironmentMap and RenderToSurface classes are used to create cubic environment maps and textures at run time. Rendering into textures at run time enables you to get special effects such as mirror-like objects reflecting other objects in the scene or creating dual-pane rendering. The text-handling classes consist mainly of the Font class, which can be instantiated from a System.Drawing.Font object and which has methods to render the text to a Direct3D-supporting device. The animation-handling classes consist of the KeyFrameInterpolator class, which has methods that are used to interpolate an object's scale, rotation, and/or translation. The interpolation is performed by using a set of 3D vector (for scaling and translation) and quaternion keys (for rotation).

The Managed DirectX 9.0 Samples Framework

The DirectX 9.0 samples framework provides numerous classes that accelerate graphics application development by implementing commonly performed tasks. The samples framework is available in source code form. To use it, you simply add the required source files to the graphics application project.

When an application is created by the Managed DirectX Application wizard (in Microsoft Visual C#® .NET), code is generated based on the samples framework. Using the wizard to produce a graphics application is the quickest way to begin using the samples framework. The samples framework is used by the samples that come with the DirectX 9.0 SDK. The DirectX 9.0 samples show how to exploit the samples framework in addition to Direct3D.

Even when not using the samples framework directly, the code in the samples framework can be referenced to see how to drive DirectX 9.0 code in a flexible and robust manner.

The GraphicsSample Class

C# applications using the DirectX samples framework inherit the GraphicsSample class, which has code to do the following:

  • Initialize and terminate DirectX.
  • Find the best windowed or full-screen display mode to use.
  • Display a form allowing the user to change device settings (such as the display mode).
  • Handle window operations such as moving, resizing, minimizing, and maximizing.
  • Handle window activation and deactivation as the window gains and loses focus.
  • Handle user interface events such as key presses and main menu navigation events.
  • Perform frame updating and rendering.
  • Perform primitive animation in the form of pausing and single-stepping the animation.
  • Collect and display statistics.
  • Handle exceptions.

The GraphicsSample class itself inherits from System.Windows.Forms.Form, so the graphics application is ultimately a Windows Form.

The GraphicsSample class defines several methods that subclasses can override to change the default behavior or add functionality. It also has a number of fields and properties that can be used to control the behavior of the graphics application and access the device that supports DirectX Graphics.

The minimum code required to use the GraphicsSample class is shown in Figure 6 and extracted from the DirectX 9.0 samples. The Render method in Figure 6 only clears the display surface so this code will just display a black background. The wizard actually adds code to display a teapot on a blue background if the developer so chooses, but that has been omitted for brevity. In normal situations, you would replace the teapot-rendering code with rendering code specific to your application.

Figure 6 Minimal App Using GraphicsSample (Close Figure 6)
using System.Drawing;
using Microsoft.DirectX;
using Microsoft.DirectX.Direct3D;

public class MyGraphicsSample : GraphicsSample
{
   protected override void Render()
   {
      device.Clear(ClearFlags.Target | ClearFlags.ZBuffer, Color.Black,
         1.0f, 0);
   }
   static void Main()
   {
      using (MyGraphicsSample d3dApp = new MyGraphicsSample())
      {
         if (d3dApp.CreateGraphicsSample())
         {
            d3dApp.Run();
         }
      }
   }
}

The DirectX 9.0 C# wizard actually creates a separate class in which it puts the Main method, so the generated code differs from that shown in Figure 6. For the code in Figure 6 to build, it is necessary to add the D3DApp.cs file to the project so that the GraphicsSample class is available.

Initialization and Cleanup

The initialization- or cleanup-specific overridable methods are listed in Figure 7. They are used in a number of scenarios: application startup, application termination, device change, device loss, and reset. The sequences of calls that result in each of these scenarios are detailed in Figure 8.

Figure 8 Initialization and Cleanup Scenarios (Close Figure 8)

Event | Function Calls
Application startup | OneTimeSceneInitialization ConfirmDevice (one or more) InitializeDeviceObjects RestoreDeviceObjects
Application termination | InvalidateDeviceObjects DeleteDeviceObjects
Device change | InvalidateDeviceObjects DeleteDeviceObjects InitializeDeviceObjects RestoreDeviceObjects
Device loss and reset | InvalidateDeviceObjects RestoreDeviceObjects
Figure 7 Initialization and Cleanup Methods (Close Figure 7)

Method | Description
OneTimeSceneInitialization | Performs initialization that is required only when the application starts
ConfirmDevice | Checks whether a Direct3D-targeted device has the capabilities required by the application
InitializeDeviceObjects | Called when the device is initialized or has changed; initializes any resources that are not lost when the device is reset, such as PoolManaged, PoolScratch and PoolSystemMemory resources, texture resources, vertex and pixel shaders
RestoreDeviceObjects | Handles the Direct3D-targeted device's DeviceReset event; recreates all PoolDefault resources and reinitializes any device states that do not change during rendering
InvalidateDeviceObjects | Handles the Direct3D-targeted device's DeviceLost event; releases all PoolDefault resources or any objects that were created in RestoreDeviceObjects
DeleteDeviceObjects | Handles the Direct3D-targeted device's Disposing event; destroys any objects that were created in InitializeDeviceObjects

In addition to the methods that you saw in Figure 7, the GraphicsSample class has a RenderTarget property that derived classes can set before CreateGraphicsSample is called to change the render target. By default, the render target is the Windows Form itself. Derived classes can define a separate control, or window, to act as the render target.

Frame Updating and Rendering

The GraphicsSample class calls the FrameUpdate method that derived classes can override to update the scene at each frame. The FrameUpdate method may get input from a mouse, joystick, or other input device and update the camera or other scene objects accordingly. The FrameUpdate method is where you would normally put any animation-related update code such as simulation, collision detection, or game-related artificial intelligence.

The GraphicsSample class calls the Render method that any derived classes must override to render the scene. The Render method is responsible for clearing the display surface area and rendering the scene between the calls to the device's BeginScene and EndScene methods.

Figure 9 shows the Render method implementation generated by the Managed DirectX Application Wizard to render a teapot, slightly reformatted for clarity. In this simple example, the Render method sets up the lighting as well as the world, view, and projection matrices. Since those render states do not change between frames, they could just as well be set in the RestoreDeviceObjects method, which would be more efficient. Setting these unchanging render states in RestoreDeviceObjects would set them just once when the device is initialized, instead of during every frame. The code shown in Figure 9 requires another file, DXUtil.cs, found in the samples framework.

Figure 9 Rendering a Teapot (Close Figure 9)
using System.Drawing;
using Microsoft.DirectX;
using Microsoft.DirectX.Direct3D;

public class GraphicsClass : GraphicsSample
{
   private float x = 0.0f;
   private float y = 0.0f;
   private Point destination = new Point(0, 0);
   private Mesh teapot = null;
   // Additional code ...

   protected override void Render()
   {
      // Clear the backbuffer to a Blue color.
      device.Clear(ClearFlags.Target | ClearFlags.ZBuffer, Color.Blue,
         1.0f, 0);

      // Begin the scene.
      device.BeginScene();

      device.Lights[0].Enabled = true;

      // Setup the world, view, and projection matrices.
      Matrix m = new Matrix();

      if( destination.Y != 0 )
         y += DXUtil.Timer(DirectXTimer.GetElapsedTime) * (destination.Y
              * 25);

      if( destination.X != 0 )
         x += DXUtil.Timer(DirectXTimer.GetElapsedTime) * (destination.X
              * 25);

      m = Matrix.RotationY(y);
      m *= Matrix.RotationX(x);

      device.Transform.World = m;
      device.Transform.View = Matrix.LookAtLH(
          new Vector3( 0.0f, 3.0f,-5.0f ),
          new Vector3( 0.0f, 0.0f, 0.0f ),
          new Vector3( 0.0f, 1.0f, 0.0f ) );
      device.Transform.Projection = Matrix.PerspectiveFovLH(
          (float)Math.PI / 4, 1.0f, 1.0f, 100.0f );

      // Render the teapot.
      teapot.DrawSubset(0);

      // End the scene.
      device.EndScene();
   }
}

Conclusion

Managed Direct3D, part of the DirectX Graphics component of DirectX 9.0, provides a great opportunity for developers targeting the .NET Framework to add 3D capability to their applications. Not only is managed Direct3D nearly as efficient as its unmanaged counterpart, but when used in conjunction with the high-level shading language, it provides a great future for high-performance RAD game development.

For related articles see:
Using the Effects Framework
http://msdn.microsoft.com/directx
http://www.realtimerendering.com

For background information see:
Real-Time Rendering, 2nd Edition, by Tomas Akenine-Moller and Eric Haines (AK Peters, 2002), Real-Time Shading, by Marc Olano, John C. Hart, Wolfgang Heidrich, and Michael McCool (AK Peters, 2002)
The DirectX 9 Programmable Graphics Pipeline, to be published by Microsoft Press, July 2003

Yahya H. Mirza is a principle at Aurora Borealis Software. He has been working on projects using the .NET Framework at Microsoft and Source Dynamics since 2000. He is working with his company on a graphics GPU project at Pixar. He can be reached at yahya_mirza@hotmail.com.

Henry da Costa has been programming in the computer graphics field for the past 18 years. He has implemented a game engine for a start-up venture and several SDKs for Softimage products. You can reach Henry at henrydacosta@sympatico.ca.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


GDI+: A Primer on Building a Color Picker User Control with GDI+ in Visual Basic .NET or C#

GDI+

A Primer on Building a Color Picker User Control with GDI+ in Visual Basic .NET or C#

Ken Getz
This article assumes you're familiar with C# and Visual Basic .NET
Level of Difficulty 1 2 3
Code download available at: GDIColorPicker.exe (395KB)
SUMMARY

Although most developers and APIs use the RGB scheme when working with colors, it's not the only available way to represent or select colors. For instance, the standard Windows color-selection dialog box allows you to work with the HSL color scheme in an indirect way. In this article, the author describes several color selection schemes, and uses GDI+ (via the System.Drawing namespace) to create a component that makes it possible for your own applications to provide a simpler, friendlier color chooser. Along the way, you'll get tips to help you use GDI+ in your own apps.

Contents
Color Spaces
Investigating the Sample App
Converting Colors
Drawing the Color Wheel
Finishing the Drawing
Painting on Demand
Repainting is Expensive
Cleaning Up
Investigating the Sample
Conclusion

B
ecause Windows® provides a standard common dialog box for selecting colors (see Figure 1) it's easy to assume that this is the only method for selecting colors. That is not true, however, and in this article I'll provide a simple-to-use replacement for the standard color-selection dialog box. The sample's layout and appearance are easy to modify. You can lay out the indicators just about any way you like, and the sample application includes two different layouts for the dialog box. This article focuses on two specific areas: investigating different color-selection techniques and explaining the members of the System.Drawing namespace that make the sample application work.

Color Spaces

The standard Windows color-selection dialog box represents colors in just two of the many ways you might represent colors in code. This representation of color choices is often called a "color space" because color representations normally allow you to refer to colors by a set of coordinates in three-dimensional space. Figure 1 shows the standard Windows dialog box with the Define Custom Colors button selected, so the right-hand pane appears, allowing you to define your own colors. Disregarding the left-hand side of the dialog box, which simply allows you to select from a palette of preselected colors, the right-hand pane allows you to drag the color selector with your mouse, or enter colors using two different color spaces, Red/Green/Blue (RGB) or Hue/Saturation/Luminosity (HSL). In this dialog box, the individual RGB values contain 8-bit values and range from 0 all the way up to 255. The HSL values range from 0 to 240.

Figure 1 Standard Color Selection in Windows
Figure 1 Standard Color Selection in Windows

The RGB color system combines three color coordinates, along with a value that indicates the transparency of the color that's often referred to as the "alpha" component of the color, and creates a 32-bit unsigned integer value that represents the color. Note that the dialog box in Figure 1 doesn't provide any way to select an alpha component other than 255, allowing only fully opaque colors. The AlphaRGB (ARGB) value represents the color in a way that's easy for a computer to decipher. For example, with the color selected in Figure 1, the combined ARGB value is 13578293 decimal or FF30CFCB hex. Breaking this 4-byte value up into individual bytes, you'll find, as you might expect given the color in Figure 1, the values 255, 48, 207, and 203 for the alpha, red, green, and blue components of the color. Disregarding the alpha component, which will always be 255 (FF) when using this particular dialog box, it's easy to programmatically convert from the integer representation of the color to the color's individual bytes, hence the propensity for Windows, and therefore application developers, to use the RGB scheme to describe colors.

Although using the RGB color space does make it easier for the computer to convert back and forth between its internal representation of the color and the representation humans need in order to select a color, RGB doesn't provide any useful ordering of colors for selection. Some development team arbitrarily selected the layout of the color grid on the right-hand side of Figure 1—there's no reason it has to look this way. If you take a few minutes to investigate the meaning of contiguous colors within the color-selection square, watching the values for the R, G, and B components of the color as you move the mouse about the screen, you'll be able to discern a pattern. The pattern isn't obvious, however; nor could you ever guess an RGB value given the position of your selection. In a more logical color-selection scheme, the position of the selector would have some relationship to the selected color.

Back in the days before it was easy to use the Windows common dialog boxes, I remember attempting to create my own color-selection dialog box. Not knowing any better, I began by trying to create a two-dimensional grid providing the capability of selecting any RGB color. Quickly, I realized that I was attempting to solve a three-dimensional problem with two-dimensional tools. My simple solution at the time was to provide a list of available colors, the easy way out. As a matter of fact, the RGB color space consists of normal Cartesian coordinates, with the red, green, and blue portions of the colors acting as the x, y, and z values. Specifying a color using the RGB color space selects a point in three dimensions, where each component indicates the distance from black (0, 0, 0) on each of three axes. The color white (255, 255, 255) represents the corner on the color "cube" diagonally opposite the origin.

So RGB is convenient, but not for humans. Humans are accustomed to thinking about color in terms of hue (what color is it?), intensity (how strong is the color?), and brightness (is it pale or vivid?). In order to meet these needs, the standard color-selection dialog box also provides an alternative representation of the color value, the HSL format. This representation makes more sense to you and me, although it requires more work for the computer as it performs somewhat complex mathematical computations to convert HSL-formatted values to the more common RGB values. If you drag the selector left and right and then up and down within the dialog box and watch the HSL values, you'll see the pattern right away. That is, as you move from left to right across the square, the Hue value increases from 0 to 240. As you move from top to bottom in the square, the Saturation value goes from 240 to 0. As you drag the separate vertical slider from top to bottom, the Luminosity value goes from 240 (white) to 0 (black). Given these physical arrangements, you could, with relative ease, become proficient at converting from the position of the two pointers in the dialog box to a reasonable approximation of the corresponding HSL color values. Not so with RGB.

The Hue/Saturation/Value (HSV) color space is a modification of HSL in which the Value axis measures the brightness of the color, and you cannot progress beyond full brightness. In comparison, HSL allows you to move past full brightness, through pastel colors, all the way to white. HSV uses a Saturation value of 0 to provide all colors that have the same red, green, and blue components instead. (That is, shades of gray.) HSV is the color scheme demonstrated in the sample application, and it's simple to explain.

The first component of an HSV value, the Hue, corresponds to an angle on a circle. This circle contains the same colors as shown on the right-hand side of Figure 1, but arranged in circular fashion (see Figure 2). Given this layout, the color red corresponds to an angle of 0 degrees, yellow to 60 degrees, and so on. The particular variation of HSV used for the sample project scales the degrees on the circle so that values for the Hue portion of the color range from 0 to 255—the scaling is done internally, in the code.

Figure 2 Color Circle
Figure 2 Color Circle

The Saturation value corresponds to the distance from the center of the circle. Colors on the edge of the circle are fully saturated and the center of the circle contains no saturation—it's white. Figure 3 shows the range of saturation values for the color green, as if extracted from the circle shown in Figure 2. Again, my selection of 0 to 255 as the range is arbitrary and only serves to make the selection of colors using HSV feel more like the familiar RGB. In reality, the example could just as easily have used floating point values between 0 and 1 to represent saturation.

Figure 3 Saturation
Figure 3 Saturation

The Value portion contains the brightness of the selected color, and it ranges from 255 (fully bright) to 0 (black). Again, the exact values aren't important—the scaling here is just for convenience. Figure 4 shows the color green as its Value coordinate changes from 255 to 0.

Figure 4 Value
Figure 4 Value

What you really have, when working with the HSV color space, is a cone of color. The circle in Figure 2 acts as the top of the cone, and a line from each point in the circle down to the vertex of the cone, representing the color black, provides the brightness range for each color. Figure 5 shows the cone using the brightness axis for three colors: green, cyan, and purple. Any color can be represented as a Hue coordinate, measuring an angle around the circle, a Saturation coordinate as the distance from the center of the circle, and a Value coordinate as the distance from the vertex at the bottom of the cone.

Figure 5 Color Cone
Figure 5 Color Cone

Coincidentally, I ran across an application using a dialog box that allows users to select colors using the HSV color scheme (much like the one shown in Figure 6) right at the time I was attempting to teach myself how to use GDI+ in the Microsoft® .NET Framework. Fascinated by this fresh approach to selecting colors, I took on a "nights and weekends" project to create the same effect using Visual Studio® .NET and Visual Basic® .NET. That effort became the sample project discussed in this article. Along the way, I grew quite comfortable with the basics of GDI+ and many of the objects and members of the System.Drawing namespace. The remainder of this article discusses the particular GDI+ techniques, objects, and members used by this sample application. I can't describe each object and member in any detail here. The code in this article is a starting point, but the .NET Framework documentation is a major resource.

Figure 6 Sample Form
Figure 6 Sample Form

Run the sample application and load the sample form shown in Figure 6. You should be able to predict the behavior of the Hue, Saturation, and Brightness controls as you interact with the form. Click on the circle and then move the mouse in a circular fashion around the circle, keeping the distance from the center as constant as possible. You'll see just the Hue value change. Moving the mouse radially, from the center of the circle to the edge, the Saturation value increases. In either case, the Brightness value remains constant. Drag the pointer on the vertical bar to the right of the circle, and the Brightness value changes corresponding to the height of the pointer.

Investigating the Sample App

The sample includes two versions of the same project: ColorChooserVB.sln and ColorChooserCSharp.sln. You can download it from the link at the top of this article. The code is essentially the same in both projects, but I'll demonstrate with code samples from the Visual Basic .NET version. The classes in the sample projects are described in Figure 7. Each class has been saved in its own file, so it's easy to find each individual class. The ColorWheel class is the centerpiece of the application—it provides the support for drawing the various elements of the color-selection forms. Besides the fields, properties, and methods in the class, the ColorWheel class also raises the ColorChanged event whenever you move the mouse to select a new color on the color wheel or on the vertical brightness bar.

Figure 7 Sample Project Classes (Close Figure 7)

Class | Description
Main | Provides the sample form, allowing you to select from three different color-choosing dialog boxes (see Figures 1, 6, and 14)
ColorWheel | Contains the code that draws the color wheel, the brightness bar, and the selected color rectangle; handles repainting the elements as needed
ColorHandler | Contains shared methods that convert colors between RGB and HSV color spaces
ColorChangedEventArgs | Provides the ColorChanged event with a structure containing information about the selected color; inherits from EventArgs
ColorChooser1 | Sample form demonstrating the use of the ColorWheel class, using NumericUpdown controls to display current colors (see Figure 6)
ColorChooser2 | Sample form demonstrating the use of the ColorWheel class, using HorizontalScrollbar controls to display current colors (see Figure 14)

It's important to understand the design goals of the sample application. I knew that however I designed the color-selection form, the user might have a different opinion about how it should look or how it should be used. Because of this, I've taken special care to separate the specific user interface of the host form from the display of the color wheel, brightness bar, and selected color rectangle. As a matter of fact, the ColorWheel class maintains no specific information about its host. Each form or control creates a new instance of the ColorWheel class, passing Rectangle objects corresponding to the three areas that the ColorWheel instance needs to manage. The ColorWheel instance raises the ColorChanged event back to the host form or control when it determines that a new color has been selected. It's up to your host form to lay out the three areas, pass Rectangle objects corresponding to the areas to the ColorWheel class's constructor, and to call the ColorWheel class's Draw method whenever it's necessary. When the user drags the color or brightness selection pointers, or otherwise selects a new color, your form must call the ColorWheel.Draw method. See the code for the sample forms—they show two different examples of this interaction.

Converting Colors

Throughout the ColorWheel class and the sample project, the code needs to convert color values between RGB and HSV format. Generally, the code uses HSV format for most internal uses—it closely maps to the way the colors are represented on the screen—but uses RGB values when interacting with the user or when displaying the selected color on the screen. Although I made a valiant attempt at working out the conversion formulas during a cross-country flight, once I reconnected and did a quick search, I found several sites listing pretty much the same algorithm in several different languages, but none in C# or Visual Basic .NET. I converted the code to C# and Visual Basic .NET, and the ColorHandler class in the sample project provides shared/static methods that handle the conversions between RGB and HSV. The class also contains HSV and RGB structures that are used throughout the application. They are simple and provide an easy way to cart around RGB and HSV values.

Specifically, the ColorHandler class provides HSVToRGB, RGBToHSV, and HSVToColor methods that perform the color conversions. The Color structure within the .NET Framework provides FromArgb and ToArgb methods, so there's no need to create those methods here. If you're interested in the details of performing the conversions to and from the HSV color scheme, check out the code in the ColorHandler class. I haven't focused on this code here, as you're unlikely to need to modify the code for use in other projects—you can simply import the code or package it in any way you like for reuse.

It is important to note, however, that both the HSVToRGB and RGBToHSV methods expect that the individual color components—the R, G, B and H, S, V values—contain integers between 0 and 255. Although there's no reason that these exact values must be used, I chose to maintain consistency between the two color spaces, using the same range for both. If you want to handle the color values in a different way, you'll need to modify the code in the ColorHandler class.

Drawing the Color Wheel

To be honest, I started on this project mostly because I had seen an application that used an HSV color selection and was convinced that it was possible to create the circular gradient the form uses for selecting colors without writing much code using GDI+. As I suspected, it's really simple to create the gradient. Working with a PathGradientBrush object, you supply an array of points that define the edge of the gradient, indicate the center color and center point, supply a corresponding array of colors in a one-to-one mapping, and you're all set. GDI+ does all the work, filling in the designated region with the gradient you've requested. (Both the PathGradientBrush class and the LinearGradientBrush class discussed later inherit from the Brush class, and are among several means of filling a region.)

The CreateGradient procedure handles all these tasks, but in a way you might not expect. As I originally wrote it, the procedure generated the gradient each time it was called, and this procedure might be called each time you move the mouse. Of course, this inefficiency made it impossible to track the mouse effectively even on the most beefy hardware, and the current solution was born out of necessity—that is, rather than continually redrawing the gradient, the CreateGradient procedure (see Figure 8) draws the gradient once, creates an image of the gradient, and then displays the image whenever the host needs to be repainted.

Figure 8 CreateGradient and Supporting Procedures (Close Figure 8)
Private Sub CreateGradient()
  Dim newGraphics As Graphics
  Dim pgb As PathGradientBrush

  Try
    ' Create a new PathGradientBrush, supplying an array of points created
    ' by calling the GetPoints method.
    pgb = New PathGradientBrush( _
     GetPoints(radius, New Point(radius, radius)))

    ' Set the various properties. Note the SurroundColors property, which
    ' contains an array of points, in a one-to-one relationship with the
    ' points that created the gradient.
    pgb.CenterColor = Color.White
    pgb.CenterPoint = New PointF(radius, radius)
    pgb.SurroundColors = GetColors()

    ' Create a new bitmap containing the color wheel gradient, so the
    ' code only needs to do all this work once. Later code uses the bitmap
    ' rather than recreating the gradient.
    colorImage = New Bitmap( _
     colorRectangle.Width, colorRectangle.Height, _
     PixelFormat.Format32bppArgb)

    newGraphics = Graphics.FromImage(colorImage)
    newGraphics.FillEllipse(pgb, 0, 0, _
     colorRectangle.Width, colorRectangle.Height)

  Finally
    If Not pgb Is Nothing Then
      pgb.Dispose()
    End If
    If Not newGraphics Is Nothing Then
      newGraphics.Dispose()
    End If
  End Try
End Sub

Private Function GetColors() As Color()
  ' Create an array of COLOR_COUNT colors, looping through all the hues
  ' between 0 and 255, broken into COLOR_COUNT intervals. HSV is
  ' particularly well suited for this because the only value that changes
  ' as you create colors is the Hue.
  Dim Colors(COLOR_COUNT - 1) As Color

  Dim i As Integer
  For i = 0 To COLOR_COUNT - 1
    Colors(i) = ColorHandler.HSVtoColor( _
     i * 255 \ COLOR_COUNT, 255, 255)
  Next
  Return Colors
End Function

Private Function GetPoints( _
 ByVal radius As Double, ByVal centerPoint As Point) _
 As Point()

  ' Generate the array of points that describe the locations of the
  ' COLOR_COUNT colors to be displayed on the color wheel.
  Dim Points(COLOR_COUNT - 1) As Point

  Dim i As Integer
  For i = 0 To COLOR_COUNT - 1
    Points(i) = GetPoint( _
     i * 360 / COLOR_COUNT, radius, centerPoint)
  Next
  Return Points
End Function

Private Function GetPoint( _
 ByVal degrees As Double, _
 ByVal radius As Double, _
 ByVal centerPoint As Point) As Point

  ' Given the center of a circle and its radius, along with the angle
  ' corresponding to the point, find the coordinates.  In other words,
  ' convert from polar to rectangular coordinates.
  Dim radians As Double = degrees / DEGREES_PER_RADIAN

  Return New Point( _
   CInt(centerPoint.X + _
    Math.Floor(radius * Math.Cos(radians))), _
   CInt(centerPoint.Y - _
    Math.Floor(radius * Math.Sin(radians))))
End Function

The CreateGradient procedure needs to retrieve an array of points along the edge of the gradient, and the GetPoints procedure handles this calculation. GetPoints iterates through 1536 points on the edge of the circle, converting from polar to Cartesian coordinates. Although this example uses a specific number of points—1536, or 256 colors in each of six regions of the circle—you could certainly get by with fewer.

For those of you who have purged all your high school trigonometry from long-term memory, polar coordinates define locations in terms of the center of a circle, the radius of the circle, and an angle between 0 and 360 degrees on the circle. Cartesian coordinates are plain old x and y—horizontal and vertical—values. Converting between the two involves some ugly but simple math using the Sin and Cos methods of the Math class. Figure 8 contains the code for the GetPoint and GetPoints methods so if you're in a high school trig kind of mood, dig in!

The CreateGradient procedure starts by creating a new PathGradientBrush object, specifying the array of points returned by the GetPoints method:

pgb = New PathGradientBrush( _
    GetPoints(radius, New Point(radius, radius)))
The code then sets the necessary properties of the PathGradientBrush object, filling in the SurroundColors property with the array of colors returned by the GetColors method:
pgb.CenterColor = Color.White
pgb.CenterPoint = New PointF(radius, radius)
pgb.SurroundColors = GetColors()

Finally, the code creates a new Bitmap object the same size as the rectangle bounding the color wheel, then creates a new Graphics object and uses its FillEllipse method to fill the bitmap with the gradient:

colorImage = New Bitmap( _
    colorRectangle.Width, colorRectangle.Height, _
    PixelFormat.Format32bppArgb)

newGraphics = Graphics.FromImage(colorImage)
newGraphics.FillEllipse(pgb, 0, 0, _
    colorRectangle.Width, colorRectangle.Height)

Drawing the linear gradient displaying the range of brightness values for the selected color is much simpler. For this vertical bar, the code simply creates a LinearGradientBrush object, supplying the top and bottom colors, and the direction in which to fill—top to bottom in this case. The code can then take the LinearGradientBrush object and use it to fill the rectangle. The ColorWheel.DrawLinearGradient procedure does the work, as shown in Figure 9.

Figure 9 Drawing the Linear Gradient (Close Figure 9)
Private Sub DrawLinearGradient(ByVal TopColor As Color)
  ' Given the top color, draw a linear gradient ranging from black to the
  ' top color. Use the brightness rectangle as the area to fill.
  Dim lgb As LinearGradientBrush

  Try
    lgb = New LinearGradientBrush( _
     brightnessRectangle, TopColor, _
      Color.Black, LinearGradientMode.Vertical)
    g.FillRectangle(lgb, brightnessRectangle)
  Finally
    If Not lgb Is Nothing Then
      lgb.Dispose()
    End If
  End Try
End Sub

The CreateGradient and DrawLinearGradient procedures should call the Dispose method of the GDI+ managed wrapper objects instead of waiting for garbage collection, to avoid overusing window handles on older OSs. The Visual Basic .NET code handles this in the Finally block, ensuring that the objects get released correctly. C# provides a simpler mechanism, the using statement, which guarantees that the object(s) listed in the statement will have their Dispose method called automatically at the end of the block. In Visual Basic .NET, which doesn't provide a construct like the using statement, procedures require a bit more code. The following code sample shows the DrawLinearGradient procedure from the C# example, taking advantage of the using statement:

private void DrawLinearGradient(Color TopColor)
{
    using (LinearGradientBrush lgb =
        new LinearGradientBrush(brightnessRectangle, TopColor,
        Color.Black, LinearGradientMode.Vertical))
  { g.FillRectangle(lgb, brightnessRectangle); }
}
Compare this to the Visual Basic .NET version shown in Figure 9.

Finishing the Drawing

The sample code uses GDI+ to draw the square and triangular pointers and to draw the selected color rectangle. The DrawColorPointer procedure calls the DrawRectangle method of the Graphics class, passing in a Pen object and coordinates for the rectangle. I used one of the predefined Pen objects, but you can also create your own Pen, passing a color and, optionally, a width:

Private Sub DrawColorPointer(ByVal pt As Point)
  ' Given a point, draw the color selector.
  ' The constant SIZE represents half
  ' the width — the square will be twice
  ' this value in width and height.
  Const SIZE As Integer = 3
  g.DrawRectangle(Pens.Black, _
      pt.X - SIZE, pt.Y - SIZE, _
      SIZE * 2, SIZE * 2)
End Sub

The DrawBrightnessPointer procedure draws the triangular pointer, using the FillPolygon method of the Graphics class. This method allows you to specify a Brush (again, using one of the built-in brushes) and an array of points:

Private Sub DrawBrightnessPointer(ByVal pt As Point)
  ' Draw a triangle for the brightness indicator that "points"
  ' at the provided point.
  Const HEIGHT As Integer = 10
  Const WIDTH As Integer = 7

  Dim Points(2) As Point
  Points(0) = pt
  Points(1) = New Point(pt.X + WIDTH, pt.Y + HEIGHT \ 2)
  Points(2) = New Point(pt.X + WIDTH, pt.Y - HEIGHT \ 2)
  g.FillPolygon(Brushes.Black, Points)
End Sub

Every time the display needs updating—which is generally every time you move the mouse or select a new color—the UpdateDisplay procedure creates a new SolidBrush object based on the selected color, redraws the color wheel image, draws the selected color rectangle, draws the linear gradient containing brightness values, and updates the pointers. The code in Figure 10 handles these tasks, calling procedures you've already seen, along with System.Drawing methods.

Figure 10 Update the Display (Close Figure 10)
Private Sub UpdateDisplay()
  ' Update the gradients, and place the pointers correctly based on
  ' colors and brightness.

  Dim selectedBrush As Brush

  Try
    ' Draw the "selected color" rectangle.
    selectedBrush = New SolidBrush(selectedColor)

    ' Draw the saved color wheel image.
    g.DrawImage(colorImage, colorRectangle)

    g.FillRectangle( _
     selectedBrush, selectedColorRectangle)

    ' Draw the "brightness" rectangle.
    DrawLinearGradient(fullColor)
    ' Draw the two pointers.
    DrawColorPointer(colorPoint)
    DrawBrightnessPointer(brightnessPoint)

  Finally
    If Not selectedBrush Is Nothing Then
      selectedBrush.Dispose()
    End If
  End Try
End Sub

Painting on Demand

In the sample applications, it's up to the forms hosting the ColorWheel class to call the ColorWheel.Draw method every time the display needs to be updated. This might occur when the user moves the mouse or selects a new color using the input method provided by the host form. The sample forms use NumericUpdown or HorizontalScrollbar controls. No matter when or how the updating needs to occur, the ColorWheel.Draw method expects to receive a Graphics object from its host, so it has a context on which to draw its graphic elements. Although you can call the CreateGraphics method at any time, doing so presents risks. For example, the Graphics object is only valid for the duration of the Windows message and must be explicitly disposed. I've found it simplest to perform all painting from within the host's Paint event handler, mostly because the Paint event receives a PaintEventArgs object as one of its parameters, and this object provides a Graphics property containing the container's graphics context. Writing it this way also allows double buffering, as you'll see in the next section.

Handling all the updates in the Paint event centralizes all the drawing code, but it means that your apps must manage their own state and take actions based on the current state from the Paint event handler. In the sample forms, you'll find an enumeration that tracks the type of change, so the Paint event handler can call the correct overloaded version of the ColorWheel.Draw method:

Private Enum ChangeStyle
  MouseMove
  RGB
  HSV
  None
End Enum

Private changeType As ChangeStyle = ChangeStyle.None
Private selectedPoint As Point

When you move the mouse or otherwise indicate that you want to modify the selected color, the host's code must track the current state and invalidate the host. This causes a request for the region to be painted, raising the Paint event. For example, moving the mouse on the sample forms causes the following code to run:

Private Sub HandleMouse( _
 ByVal sender As Object, ByVal e As MouseEventArgs) _
 Handles MyBase.MouseMove, MyBase.MouseDown

  ' If you have the left mouse button down, then update the
  ' selectedPoint value and force a repaint of the color wheel.
  If e.Button = MouseButtons.Left Then
    changeType = ChangeStyle.MouseMove
    selectedPoint = New Point(e.X, e.Y)
    Me.Invalidate()
  End If
End Sub
The call to the Invalidate method causes the form's Paint event handler to kick in, running the code shown in Figure 11. Selecting a value from one of the NumericUpdown controls on the first sample form that corresponds to an RGB value (look back to Figure 6) runs the code shown in Figure 12. This procedure first checks to make sure it's not being called recursively. It's impossible to determine if a NumericUpdown control is being updated by a user or in code, so the sample uses a class-level variable, isInUpdate, to avoid recursive changes to the controls. The code sets the change type, calculates the RGB and HSV values, and then invalidates the form, raising the Paint event and running the Paint event handler code.

Figure 12 Color Changed (Close Figure 12)
Private Sub HandleRGBChange( _
 ByVal sender As System.Object, _
 ByVal e As System.EventArgs) _
 Handles nudRed.ValueChanged, nudBlue.ValueChanged, _
 nudGreen.ValueChanged

  ' If the R, G, or B values change, use this code to update the HSV
  ' values and invalidate the color wheel (so it updates the pointers).
  ' Check the isInUpdate flag to avoid recursive events when you update
  ' the NumericUpdownControls.
  If Not isInUpdate Then
    changeType = ChangeStyle.RGB
    RGB = New ColorHandler.RGB(CInt(nudRed.Value), _
     CInt(nudGreen.Value), CInt(nudBlue.Value))
    SetHSV(ColorHandler.RGBtoHSV(RGB))
    Me.Invalidate()
  End If
End Sub
Figure 11 Paint Event Handler (Close Figure 11)
Private Sub ColorChooser1_Paint( _
 ByVal sender As Object, _
 ByVal e As PaintEventArgs) Handles MyBase.Paint

  ' Depending on the circumstances, force a repaint
  ' of the color wheel passing different information.
  Select Case changeType
    Case ChangeStyle.HSV
      myColorWheel.Draw(e.Graphics, HSV)
    Case ChangeStyle.MouseMove, ChangeStyle.None
      myColorWheel.Draw(e.Graphics, selectedPoint)
    Case ChangeStyle.RGB
      myColorWheel.Draw(e.Graphics, RGB)
  End Select
End Sub

Repainting is Expensive

Even having optimized the drawing of the circular gradient so that the major calculations happen only once, the Draw method of the ColorWheel class needs to display the image of the gradient each time you move the mouse so it can redraw the rectangle pointer. This, too, is quite expensive and slow. There is a simple solution, however: double buffering. This technique creates an image of the form in memory. Each time you invalidate the form's image, the Windows Forms engine can figure out exactly which pixels need to be redisplayed by comparing the current image to the saved image and updating only the changed pixels. By creating an in-memory image of the display surface, the Windows Forms engine can redisplay only the necessary portion of the surface as your code calls the ColorWheel.Draw method repeatedly.

The System.Windows.Forms.Control class makes this easy—you can set bits in the ControlStyles field that affect the way the form draws itself, calling the SetStyle method. Although the SetStyle method provides a number of values you can combine to control painting, to reduce flicker by using double buffering you must set the UserPaint, AllPaintingInWmPaint, and DoubleBuffer style bits to True, like this:

Me.SetStyle(ControlStyles.AllPaintingInWmPaint, True)
Me.SetStyle(ControlStyles.UserPaint, True)
Me.SetStyle(ControlStyles.DoubleBuffer, True)

See the .NET Framework documentation on the Control.SetStyle method for more information on these and other style bits. You might also want to try commenting out these lines in the sample forms (the code in the form's Load event handlers) to verify the difference that double buffering can make.

Cleaning Up

The ColorWheel class implements the IDisposable interface and so supplies the Dispose method required by the interface. The point is that the class takes advantage of some graphics resources that should always be disposed, when the class is no longer in use (see Figure 13).

Figure 13 ColorWheel Class (Close Figure 13)
' Declared in the ColorWheel class:
Private g As Graphics
Private colorRegion As Region
Private brightnessRegion As Region
Private colorImage As Bitmap

Private Sub Dispose() Implements IDisposable.Dispose
  ' Dispose of graphic resources
  If Not colorImage Is Nothing Then
    colorImage.Dispose()
  End If
  If Not colorRegion Is Nothing Then
    colorRegion.Dispose()
  End If
  If Not brightnessRegion Is Nothing Then
    brightnessRegion.Dispose()
  End If
  If Not g Is Nothing Then
    g.Dispose()
  End If
End Sub

When working with GDI+, it's important to remember that you should call the Dispose method of any object that provides one. That way, the wrappers around unmanaged objects provided by the operating system won't hog resources.

Investigating the Sample

The sample project contains the two forms shown in Figure 6 and Figure 14, in addition to a main form that allows you to test the three different color dialog boxes shown in this article. The two forms, ColorChooser1 (see Figure 6) and ColorChooser2 (see Figure 14), each take advantage of the fact that the class that actually generates and manages the color selection, ColorWheel, requires its clients to pass in the locations of the color wheel, brightness rectangle, and selected color rectangle as parameters to its constructor. In each case, the client passes a Rectangle object containing the coordinates of the corresponding area on the client form. That is, the ColorWheel class itself has no direct interaction with its host. Figure 15 shows the ColorChooser1 form in design view, with three Panel controls acting as placeholders for the graphic elements to be displayed by the ColorWheel class. The point here is that the ColorWheel class needs to know only the locations in which to display its color wheel, brightness selector, and selected color rectangle. Code in your form passes these locations to the constructor in the ColorWheel class.

Figure 14 Another Color Picker Form
Figure 14 Another Color Picker Form

The sample forms declare a class-level variable, myColorWheel. In Visual Basic .NET, the WithEvents keyword makes it easy to hook up event handling. In C#, the sample hooks up the event handling declaratively, instead:

Private WithEvents myColorWheel As ColorWheel
As the form loads, the code within the form's Load event handler hides the panels and then retrieves their coordinates into Rectangle objects (see Figure 16).

Figure 16 Get Coordinates (Close Figure 16)
pnlSelectedColor.Visible = False
pnlBrightness.Visible = False
pnlColor.Visible = False

' Calculate the coordinates of the three required regions on the form.
Dim SelectedColorRectangle As Rectangle = _
 New Rectangle(pnlSelectedColor.Location, _
 pnlSelectedColor.Size)
Dim BrightnessRectangle As Rectangle = _
 New Rectangle(pnlBrightness.Location, _
 pnlBrightness.Size)
Dim ColorRectangle As Rectangle = _
 New Rectangle(pnlColor.Location, pnlColor.Size)
Figure 15 Design View
Figure 15 Design View

I decided to use Panel controls only because they provide a convenient way to lay out the forms. They're not required; you can build your forms by simply specifying the locations where you want the three user interface features of the ColorWheel class. I found it easiest to lay these out at design time, using the Panel controls, and then pass the coordinates of those panels to the constructor in the ColorWheel class.

Finally, the form creates a new ColorWheel object, passing in the three Rectangle objects, and hooks up the ColorChanged handler, as you'll see in the following code:

         myColorWheel = New ColorWheel( _
             ColorRectangle, BrightnessRectangle, _
             SelectedColorRectangle)
The ColorWheel class raises its ColorChanged event every time you select a new color, and the hosting form needs to react to that event, updating the display as it sees fit. The first sample form, shown in Figure 6, updates NumericUpdown controls as you move the mouse on the form. The second sample form, shown in Figure 14, updates HorizontalScrollbar controls. You aren't required, of course, to update any controls—you could simply allow the user to move the mouse, and the ColorWheel class would keep the area designated by the third rectangle that was passed by parameter to its constructor updated with the selected color.

Conclusion

If you want to take advantage of the ColorWheel class in your own applications, you could call it from any form, create a custom control that mimics the ColorDialog control, or use the class in any other way you find useful. In any case, you can simply examine the sample forms and the ColorWheel class, and go from there.

As you can see, you can easily break out of the RGB rut and offer your users or other developers an alternate means of selecting colors. In addition, you've seen in this example that you can take advantage of the powerful features provided by the System.Drawing namespace without having to write much code at all.

For related articles see:
Computer Graphics: Principles and Practice by James Foley et. al. (Addison-Wesley, 1995)
Programming Windows with C# by Charles Petzold (Microsoft Press, 2001)

For background information see:
About GDI+
Using GDI+

Ken Getz is a senior consultant with MCW Technologies and a Microsoft Regional Director for Southern California. Ken is coauthor of ASP.NET Developer's Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2001), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Vector Graphics: Build Flexible, Lightweight XML-Based Images for ASP.NET Using Scalable Vector Graphics

Vector Graphics

Build Flexible, Lightweight XML-Based Images for ASP.NET Using Scalable Vector Graphics

Dennis Forbes
This article assumes you're familiar with HTML, ASP.NET, and C#
Level of Difficulty 1 2 3
Code download available at: ScalableVectorGraphics.exe (123KB)
SUMMARY

Scalable Vector Graphics (SVG), a W3C graphics standard built around XML, is one of several vector graphics technologies that allows fast, lightweight drawings such as charts and graphs to be rendered on the fly in an appropriate viewer. There are many advantages to such vector graphics, including conservation of bandwidth and storage media, and flexibility. This article explains these benefits and shows you how to easily add powerful, dynamic, interactive visual elements to your Web applications.

Contents
Various Vector Graphics Formats
Getting Started with SVG
Dynamic Content
SVG Deployment
Creating SVGs On the Fly
Graphing with SVG
Conservation of Bandwidth
Maintaining Backward Compatibility
Conclusion

S
calable Vector Graphics (SVG) is a standard for authoring and deploying two-dimensional vector graphics using XML documents. It is a World Wide Web Consortium (W3C) recommendation that has been authored and endorsed by a number of industry organizations. Featuring the quality and versatility of vector graphics, the simplicity and extensive tool compatibility of an XML foundation, and the dynamics of a feature-rich Document Object Model (DOM) interface, SVG brings a powerful new ingredient to DHTML projects. Combining this with some creativity—and a little ingenuity—yields a limitless canvas of interactive Web possibilities, enabling you to improve the appearance, usability, and functionality of your Web apps.

Various Vector Graphics Formats

Vector graphics—which include SVG, Vector Markup Language (VML), Windows® metafiles (.wmf and .emf), and TrueType fonts—are stored and conveyed as a list of instructions detailing how to recreate an image using a sequence of shapes, lines, and transformations. This is similar to reproducing a painting by precisely recording and then recreating the sequence of brush strokes originally used to produce it. Vector graphics are usually data efficient, especially when compressed, with minimal storage and bandwidth requirements. They are rendered without pixelation when printed or scaled, even when magnified or enlarged. They are a superb choice for any graphic with limited precision detail, allowing it to be broken down into fundamental shapes and curves. On the downside, because the number of steps required to recreate an image can number in the thousands and beyond, vector graphics can be computationally demanding to render. Vector graphics are generally unsuitable for photo-realistic, highly detailed images, but are usually the best choice for graphs, diagrams, or images composed of basic shapes, such as cartoon-style images.

Raster graphics—such as Portable Network Graphics (PNG), GIF, and JPEG—are by comparison a pixel-by-pixel representation of the final image, prerendered for a specific resolution and zoom. This is similar to waiting for a painting to be completed and then taking a picture at a desired resolution and size for a specific use and output device. Raster images are consistently reproducible across platforms, with minimal computational requirements. They are ideal for highly detailed, photo-realistic images; however, they generally zoom and print poorly. Without resorting to enormous file sizes, they offer minimal dynamic programmability and are generally inefficient to store and transmit.

The highly caffeinated among you might have noticed the reference to VML earlier in this section, and indeed it does deserve special mention: VML offers many of the same advantages of SVG, such as easy text-based authoring and a foundation built on vector graphics. VML, however, has been largely deprecated in favor of other formats, with SVG garnering some of this vector graphic momentum over the past year. There are tools available for converting back and forth between SVG and VML, so it is conceivable that both could be supported simultaneously with minimal effort.

Getting Started with SVG

An SVG is, at its core, an XML document conforming to the schema defined as a part of the SVG ratification process. Defined in the specification is a variety of basic geometric shapes such as rectangles, circles, ellipses, lines, polylines, polygons, and glyph elements such as text. More complex shapes can be created by using the path element, allowing you to define shapes as a sequence of lines, arcs, and complex Bezier curves. Using these basic shapes, along with a bit of ingenuity, even the most complex geometric constructions can be created. Figure 1 illustrates the XML composition of a basic SVG document, and Figure 2 demonstrates how this SVG looks when rendered in an SVG viewer.

Figure 1 SVG Document (Close Figure 1)
<?xml version="1.0" standalone="yes" ?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN"
  "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
<svg xmlns="http://www.w3.org/2000/svg" width="400px" height="300px">
   <g id="oAll">
      <rect id="oRect" x="50" y="100" width="100" height="100"
          fill="red" stroke="black" stroke-width="2" />
      <circle id="oCircle" cx="210" cy="150" r="50"
          fill="green" stroke="black" stroke-width="2" />
      <polygon id="oPolygon" points="300,100 250,200 350,200"
          fill="blue" stroke="black" stroke-width="2" />
   </g>
   <text x="200" y="80" font-family="Verdana" font-size="25"
       fill="blue" stroke="yellow" stroke-width="0.5"
       text-anchor="middle">MSDN Magazine</text>
</svg>
Figure 2 SVG in a Viewer
Figure 2 SVG in a Viewer

While some applications (such as CorelDraw, Adobe Illustrator, and Batik SVG toolset) can display or manipulate SVGs directly, in most cases SVGs will be used as an element of an HTML document alongside other HTML text and multimedia content. The classic method for doing this is through the venerable embed tag:

<html>
  <body>
    <h1>A Page Containing an SVG</h1>
    <embed name="ExampleSVG"
    pluginspage="http://www.adobe.com/svg/viewer/install/"
    src="figure1.svg" width="400" height="300"
    type="image/svg-xml" />
  </body>
</html>
The pluginspage attribute directs the user to the applicable content handler if one isn't already installed. In this example, I have referenced the Adobe SVG Viewer.

A noteworthy point is the correlation between the width and height attributes of the embed tag, and the same on the root SVG element in the referenced graphic (a value which incidentally can be expressed in a variety of ways, including pixels, centimeters, and so on). If these figures do not match, the encapsulated graphic will be clipped or modified in some undesirable way. A technique to avoid this problem, and to provide more flexibility in general, is to refrain from tying the SVG to one specific size. Figure 3 shows a slightly modified SVG that uses a viewBox attribute rather than width and height. This sets the bounds of the canvas that I am drawing on and implies the ratio between width and height; however, it does not fix a rendering size. If you change the width and height of the embed element while keeping the relative proportions of width and height consistent, the SVG graphic should smoothly size accordingly.

Figure 3 SVG Using viewBox (Close Figure 3)
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN"
  "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 400 300">
   <g id="oAll">
      <rect id="oRect" x="50" y="100" width="100" height="100"
          fill="red" stroke="black" stroke-width="2" />
      <circle id="oCircle" cx="210" cy="150" r="50" fill="green"
          stroke="black" stroke-width="2" />
      <polygon id="oPolygon" points="300,100 250,200 350,200"
          fill="blue" stroke="black" stroke-width="2" />
   </g>
   <text x="200" y="80" font-family="Verdana" font-size="25"
       fill="blue" stroke="yellow" stroke-width="0.5"
       text-anchor="middle">MSDN Magazine</text>
</svg>

An alternative to the nonstandard HTML embed element, especially if compliance with XHTML is a requirement, is to use an object or iframe element:

  <html>
    <body>
      <object data="figure1.svg"
      codetype="image/svg-xml"
      style="width:400;height:300;">
       </object>
    <iframe src="figure3.svg" width="600" height="450" />
  </body>
</html>
The object tag presents a particularly powerful feature in that it allows cascades when unsupported types are encountered. For example, the following document tells the browser to first try to render the SVG, but failing that (if SVG functionality isn't available, for instance) to display a raster PNG as an alternative. Such a structure allows you to create a single set of pages supporting both SVG and raster content without needless branching based on the client browser's capabilities, as shown here:
<html>
  <body>
    <object data="figure1.svg" codetype="image/svg-xml"
    style="width:400;height:300;">
    <img src="figure1.png" width="400" height="300" />
    </object>
  </body>
</html>

A caveat about the object tag is in order. While developing Microsoft® Internet Explorer, Microsoft went to great lengths to make it highly extensible. This is demonstrated by the ability of the browser to instantiate a number of helper components to display multimedia Web pages (either through direct reference in the HTML, for example a specific ActiveX® object via a CODEBASE/CLASSID pair, or through the MIME type and a lookup in HKEY_CLASSES_ROOT\MIME\Database\Content Type). While this offers tremendous versatility, it can lead to trouble when overzealous applications hijack content types and then fail to render them properly when the time comes, or when they fail to reassociate Internet Explorer as the handler during deinstallation. Furthermore, I've encountered some inconsistencies and irregularities when using the object tag with SVG—problems which don't occur when using the embed or iframe tags. The embed tag, however, is being deprecated. As always, your mileage may vary, and it's just something that you should keep in mind when troubleshooting a pesky situation in which a client has installed a wide swath of helper applications.

Dynamic Content

A powerful feature of SVG is the ability to programmatically manipulate the graphical elements of an SVG through internal or external scripts. Figure 4 is an HTML document encapsulating the figure3.svg graphic which is available in the download at the link at the top of this article. This example uses the Adobe SVG Viewer plug-in for Internet Explorer. This example shows the use of ECMAScript (JavaScript) in the containing HTML document to automate the DOM interface exposed by the SVG viewer. By clicking one of the buttons on the Web page shown in Figure 5, the user can shift the colors used on the basic shapes or control the rate of rotation of a group of objects. While this particular example represents nothing more than a rehash of the infamous BLINK tag, it does hint at the breadth of possibilities that SVG offers. New objects can be added to graphics, and the existing content can be completely modified. This allows you to create graphs that are enabled with dynamic interpretation cues, for example, giving the user a better tool to fully comprehend the information.

Figure 4 HTML with Script (Close Figure 4)
<html>
<head>
<title>Automated SVG</title>
</head>
<body onload="RotateSVG();">
<script language="javascript">
var SVGDocument = null;
var Circle, Polygon, Rect, AllObjects;
var CurrentAngle = 0;
var AngleDelta = 0.0;

function InitializeObjects() {
   SVGDocument = window.document.getElementById
   ("SVGEmbed").getSVGDocument();
   Circle = SVGDocument.getElementById("oCircle");
   Polygon = SVGDocument.getElementById("oPolygon");
   Rect = SVGDocument.getElementById("oRect");
   AllObjects = SVGDocument.getElementById("oAll");
}

function RotateSVG() {
   window.setTimeout("RotateSVG();",10);
   if (AngleDelta == 0.0) return;
   if (!SVGDocument) {
      InitializeObjects();
   }
   CurrentAngle = CurrentAngle + AngleDelta;
   if (CurrentAngle > 359) CurrentAngle = CurrentAngle - 360;
   if (CurrentAngle < 0) CurrentAngle = CurrentAngle + 360;
   AllObjects.setAttribute("transform","rotate("+CurrentAngle+" 210
     150)");
}

function ShiftLeft() {
   if (!SVGDocument) InitializeObjects();
   var CircleFill = Circle.getAttribute('fill');
   Circle.setAttribute('fill',Polygon.getAttribute('fill'));
   Polygon.setAttribute('fill',Rect.getAttribute('fill'));
   Rect.setAttribute('fill',CircleFill);
}

function ShiftRight() {
   if (!SVGDocument) InitializeObjects();
   var PolygonFill = Polygon.getAttribute('fill');
   Polygon.setAttribute('fill',Circle.getAttribute('fill'));
   Circle.setAttribute('fill',Rect.getAttribute('fill'));
   Rect.setAttribute('fill',PolygonFill);
}

</script>
<center>
<h1>SVG Automation</h1>
<object id="SVGEmbed" data="figure3.svg"
   codebase="http://www.adobe.com/svg/viewer/install/"
   width="600" height="450" type="image/svg+xml"></embed>
<br />
<input type="button" value="Spin Left"
   onclick="AngleDelta=AngleDelta-0.5;" />
<input type="button" value="Color Shift Left" onclick="ShiftLeft();" />
<input type="button" value="Color Shift Right" onclick="ShiftRight();" />
<input type="button" value="Spin Right"
   onclick="AngleDelta=AngleDelta+0.5;" />
<input type="button" value="Stop" onclick="AngleDelta=0;"/>
</center>
</body>
</html>
Figure 5 Manipulating the Shapes
Figure 5 Manipulating the Shapes

SVG Deployment

Due to the cutting-edge nature of SVG and its current lack of native support across all major browsers, Web site visitors will often need to download and install a specialized viewer, such as the Adobe SVG Viewer or the upcoming Corel SVG Viewer, to see SVG content within Web pages (at least until the browsers themselves natively support SVG as they do most other graphics formats). The Adobe Viewer itself works on several platforms, including Windows, Macintosh, and Linux, and in virtually all major Web browsers. Given that SVG is a highly documented standard, I feel that it does not represent an esoteric gamble on a proprietary format, but rather an investment in the future that can yield positive results for you today.

Despite the widespread availability of SVG-equipped clients, most sites will still need to continue to support legacy clients which may not be able to install SVG-specific software. Such a solution can easily be accommodated with a graphics infrastructure that generates SVGs, but which can also create static raster graphic renderings for legacy clients. A powerful method of achieving this is detailed later in this article. Indeed, SVG can be used in the back end alongside the powerful XML classes in the Microsoft .NET Framework, even if SVGs are never directly served to Web clients.

Creating SVGs On the Fly

Coupling the graphical prowess of SVG with the wide feature set of the .NET Framework offers a powerful opportunity for Web developers. By leveraging a data back end with automatic SVG document creation, Web sites can offer real-time, aesthetically pleasing, interactive graphics, assisting users in interpreting and understanding information better. This also gives the Web application a more professional, polished look.

In order to make the dynamic creation of the root SVG document easier, and to demonstrate an XmlDocument-derived class with rudimentary SVG intelligence, I've created a trivial class which is shown in Figure 6. This helper class will be used by the following example and should be compiled with the resulting DLL placed in the Web application's bin directory (for example, C:\inetpub\wwwroot\bin):

csc /target:library figure5.cs
copy figure5.dll C:\inetpub\wwwroot\bin

Figure 6 Helper Class (Close Figure 6)
using System;
using System.Xml;

namespace Example.ExampleSvg
{
   /// <summary>
   /// XmlDocument wrapper for SVG content
   /// </summary>
   public class XmlDocumentSvg : System.Xml.XmlDocument
   {
      public XmlDocumentSvg(int Width, int Height) : base()
      {
         string BaseSvg = "<?xml version=\"1.0\" standalone=\"yes\"?>" +
         "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 20010904//EN\" "+
         "\"http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd\">" +
         "<svg width=\""+Width.ToString()+"\"
             height=\""+Height.ToString()+"\"
             xmlns=\"http://www.w3.org/2000/svg\"> "+
         "</svg>";
         this.LoadXml(BaseSvg);
      }
   }
}

Figure 7 shows the foundations of an SVG-generating Web page. It contains a simple server-side ASP.NET script that responds to requests by returning an SVG doc which contains the current time in ISO 8601 format. While this example has limited real-world use, it does show the simplicity of dynamically creating SVG documents on the server. This example concatenates strings to form the XML document. It could also be implemented by programmatically adding each element and attribute using a proper DOM approach. This sample sets the ContentType property of the Response object (it could also use the ContentType attribute of the @Page directive). This allows the client to recognize the content type without depending on tenuous and unreliable file name extensions. Having said that, it is sometimes advantageous to include a false parameter appendage on URLs, such as ?.svg, for clients such as Opera 6 that errantly look for an extension in order to determine content type.

Figure 7 Basic SVG Document (Close Figure 7)
<%@page Buffer="true" %>
<%@import Namespace="System.Xml" %>
<%@import Namespace="Example.ExampleSvg" %>
<script language="C#" runat="server">
private void Page_Load(object sender, System.EventArgs e)
{
   XmlDocumentSvg SvgXml = new XmlDocumentSvg(400,200);
   XmlNode RootNode = SvgXml["svg"];
   RootNode.InnerXml = "<text x=\"100\" y=\"100\" font-family=\"Verdana\"
       font-size=\"15\" fill=\"blue\">"
       +System.DateTime.Now.ToString("s")+"</text>";
   Response.ContentType = "image/svg+xml";
   SvgXml.Save(Context.Response.OutputStream);
}
</script>

Also notable in this example is the direct streaming of the XML document to the output stream, which eliminates any unnecessary persistence in between and maintains a high level of performance. This is a very important consideration when the page needs to serve the real-time needs of a large number of users.

The resulting SVG generated during a call to the ASP.NET page follows, with the date/time dependent on when the page is called:

<?xml version="1.0" standalone="yes"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20010904//EN"
  "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd"[]>
<svg width="400" height="200" xmlns="http://www.w3.org/2000/svg">
  <text x="100" y="100" font-family="Verdana" font-size="15"
    fill="blue">2003-03-28T122002-10-29T12:02:14</text>
</svg>
This dynamic SVG document can be included in a basic HTML page, as previously described, using the object tag:
<html>
<body>
<center>
<h1>SVG Test</h1>
<object name="ExampleSVG" codebase="http://www.adobe.com/svg/viewer/
  install/" data="figure7.aspx?.svg" width="400" height="200"
  type="image/svg+xml"/>
</center>
</body>
</html>

Graphing with SVG

Many SVGs will be dynamically generated on demand based on changing data in a database, be it GIS information containing regional land boundaries and partitions or the latest monthly sales figures. As an example of techniques to generate SVG documents from data dynamically, the namespace yafla.yaflaSVG (available by compiling and installing the yafla.yaflaSVG.cs file in svg_graphing.zip from the link at the top of this article) offers a variety of classes to assist in converting data to graphs. These graph types include pie charts (see Figure 8), as well as vertical and horizontal bar charts (see Figure 9). The class hierarchy has been built in an easily expanded object-oriented fashion, and provides numerous settings to control the UI of the generated graphs. Given that the generated SVG graphic is returned as an XML document, you could then use the broad XML capabilities within the .NET Framework to modify the documents before returning them to the caller. While these graphs are no replacement for commercial graphing products, they are an excellent introduction to the world of SVG and may actually offer real-world solutions for some readers. Barchart_example.aspx and piechart_example.aspx, also available in the download, demonstrate the basics of dynamically generating graphs using these classes, in this case creating a bar and pie chart. While these examples manually create and populate a DataTable, it is just as easy to retrieve a DataTable as a result of a query, thus achieving truly dynamic vector graphics.

Figure 8 A Pie Chart
Figure 8 A Pie Chart

Conservation of Bandwidth

While SVG documents are generally much smaller than their raster-based relatives, when representing basic graphs and charts, the intrinsic verbosity of XML can yield large files for complex graphics. Thankfully, the SVG standard includes native support for GZIP compression (a standard documented in RFC 1952), dramatically reducing the transmission and storage size—often by 80 percent or more. While GZIP does impose some additional computational overhead due to its compression and decompression, this can be alleviated on the server side by appropriate caching. On the client side, the concern is largely not relevant since modern high-power processors outpace the speed of the vast majority of network connections. Considering these factors, it becomes clear that compression is a win-win situation, producing a more responsive, snappier Web site for the client with reduced bandwidth bills for the host.

Figure 9 A Bar Graph
Figure 9 A Bar Graph

For a static SVG document, compression is as simple as GZIP-ing the file on the server (the resulting compressed file's extension is .svgz; .svg is the extension of an uncompressed SVG document), ensuring that the appropriate MIME type (image/svg+xml) is associated with both .svg and .svgz in Microsoft Internet Information Services (IIS), and ensuring that references point to the compressed copy. By doing this you'll be using a fraction of the bandwidth you would have consumed. Your clients, especially dial-up Internet users, will enjoy a much faster experience.

Things get a little more complex when compressing dynamically generated content. The first step in pursuing dynamic compression is to determine if it's already taking place. Is your Web environment enabled to stream compressed output to equipped clients? Figure 10 is a simple script that takes a URL as a parameter (for example, wscript figure9.js http://127.0.0.1/mydynamicpage.aspx) and, using the MSXML ServerXMLHTTP object, reports whether streaming compression is enabled for that URL. As an additional benefit, it also reports the MIME type that the server is returning for the request, allowing you to verify that the server is correctly configured. It is important to test since compression can be transparently taking place through load-balancing front-ends or IIS ISAPI filters compressing output when requested, and double-compression is unnecessarily wasteful in such cases.

Figure 10 Is Streaming Compression On? (Close Figure 10)
var WshShell = WScript.CreateObject("WScript.Shell");
if (WScript.Arguments.Length < 1) {
   WshShell.Popup("Usage: Pass a URL in the form of
                     protocol://domain_or_server/document");
} else {
   var XMLRequest = new ActiveXObject("msxml2.serverxmlhttp.3.0");
   XMLRequest.open("GET",WScript.Arguments(0),false);
   XMLRequest.setRequestHeader("Accept-Encoding", "gzip, deflate");
   XMLRequest.send();
   var CompressionResult;
   if (XMLRequest.getResponseHeader("Content-
      Encoding").search("gzip|deflate")!=-1) {
      CompressionResult = "***enabled***";
   } else {
      CompressionResult = "disabled";
   }
   WshShell.Popup("Compression is "+CompressionResult+" for
       "+WScript.Arguments(0)+". The content type is
       "+XMLRequest.getResponseHeader("Content-Type")+".");
}

If you want to enable dynamic SVG compression, there are a variety of .NET-accessible compression components available on the market, from feature-rich, rock-solid commercial packages, to COM Interop wrappers for existing DLLs, to excellent open-source projects. Many of these solutions are implemented as classes derived from the .NET stream objects, allowing for easy use anywhere a stream object is used. For demonstration purposes, I've chosen the open-source SharpZipLib C#-authored library, downloadable at http://www.icsharpcode.net/OpenSource/SharpZipLib.

After placing the binary DLL in the global assembly cache or in the Web application's /bin directory, streaming GZIP compression can be implemented quite easily, as shown in Figure 11 (which is a variation of Figure 7). By using the GZIP stream interface as a transport between the XML document and the HTTP response object, data is compressed while being sent, reducing the bandwidth requirements dramatically. For such a tiny SVG document, the savings achieved by GZIP compression are minimal and are likely outweighed by the computational setup costs. However, for real-world documents the benefits can be dramatic. Compressed_ barchart_example.aspx, available in the download, uses compression with the charting components. In that example, the document is reduced from 7,722 to 1,235 bytes (an 84 percent decrease).

Figure 11 GZIP Compression (Close Figure 11)
<%@page Buffer="true" %>
<%@import Namespace="System.Xml" %>
<%@import Namespace="Example.ExampleSvg" %>
<%@import Namespace="ICSharpCode.SharpZipLib.GZip" %>
<script language="C#" runat="server">
private void Page_Load(object sender, System.EventArgs e)
{
   XmlDocumentSvg SvgXml = new XmlDocumentSvg(400,200);
   XmlNode RootNode = SvgXml["svg"];
   RootNode.InnerXml = "<text x=\"100\" y=\"100\" font-family=\"Verdana\"
       font-size=\"15\" fill=\"blue\" >"
       +System.DateTime.Now.ToString("s")+"</text>";
   Response.ContentType = "image/svg+xml";
   GZipOutputStream CompressedOutput = new
       GZipOutputStream(Response.OutputStream);
   SvgXml.Save(CompressedOutput);
   CompressedOutput.Finish();
}
</script>

Maintaining Backward Compatibility

It isn't realistic to expect every user to upgrade their system with special components specifically for your Web site. The use of esoteric technologies can lead to endless support calls as users panic about being left out, or, in the case of public Web sites, the inevitable flurry of e-mail messages from advocates of the lowest common denominator furious about anything beyond HTML 1.0. To avoid this situation, one possibility is to render the SVG to a bitmapped graphic on the server, returning the SVG itself for enabled clients while returning the bitmapped graphic for everyone else. This is a credible approach if SVGs themselves are never served to the clients, but are used merely as a dynamic graphic authoring technology which is then rasterized for presentation.

You can just imagine the machinations necessary to capture its GDI output for rasterization. But whether or not the Adobe SVG Viewer can be used for server-side rendering may be addressed in their licensing agreement, which you can check at http://www.adobe.com. Another viable option can be had in the form of the Batik project (http://xml.apache.org/batik/svgrasterizer.html), a utility that can convert an SVG into a variety of formats, including raster formats like PNG, TIFF, PDF, or JPEG (though the lossy JPEG format is generally only suitable for photo-realistic images or images with smooth color gradients). The GIF format is excluded from its output repertoire, presumably because of the associated licensing issues. Batik unfortunately requires that you persist the SVG to disk, to the temporary folder, for example, where it then processes and generates a raster graphic depending upon the parameters passed. Barchart_example_rasterize.aspx, found in the download, demonstrates how you could take a width input parameter and from that it generates a raster PNG bar chart at the desired size (for example, http://localhost/barchart_example_rasterize.asp?width=750). It would be a simple exercise to add additional logic to take a parameter choosing the destination file type and allowing for the generation of the various types of graphics output at various sizes. A slightly more involved scenario would be the addition of logic to incorporate caching into the system. This would entail persisting the SVG itself to a cache object, even if only for short periods of time, with a dependent raster graphic that is regenerated whenever the source SVG changes. With such changes you can enable a powerful server-side graphics subsystem that can accommodate both your dynamic vector and raster needs, while at the same time using the caching infrastructure of the .NET Framework to achieve vast scalability and excellent response times even with a heavy load.

For performance and integration reasons, it would be preferable to develop a .NET native SVG rasterizer; however, such a project is beyond the scope of this article. Perhaps with the increasing interest in SVG this implementation may become a reality. SVG also currently lacks DHTML behaviors like VML supported. This would be another good community addition to the SVG spec.

Conclusion

By combining the graphics prowess of SVG with the power and versatility of the .NET Framework, you can turn your Web applications into graphical superheroes. With the added dynamic graphical elements, you can add clarity to your data, achieve greater organizational support, and present a more polished, professional image to the world. Hopefully this article has offered enough information to whet your appetite and to encourage further experimentation and research into exciting implementation possibilities of vector graphics in Web development.

For related articles see:
How to use VML on Web Pages
Create Snazzy Web Charts and Graphics On the Fly with the .NET Framework
Manipulate XML Data Easily with Integrated Readers and Writers in the .NET Framework

For background information see:
http://www.w3.org/TR/SVG
http://www.w3.org/TR/NOTE-VML
http://www.w3c.org/Graphics/WebCGM

Dennis Forbes is a new father and professional software developer/consultant from Toronto, Canada. His company specializes in providing solutions for the worlds of .NET and IIS. Visit his Web site at http://www.yafla.com, or contact him at dforbes@yafla.com.


Web Services: Extend the ASP.NET WebMethod Framework by Adding XML Schema Validation

Web Services

Extend the ASP.NET WebMethod Framework by Adding XML Schema Validation

Aaron Skonnard and Dan Sullivan
This article assumes you're familiar with XML, ASP.NET, and Web Services
Level of Difficulty 1 2 3
Code download available at: XMLSchemaValidation.exe (208KB)
SUMMARY

WebMethods make the development of XML Web Services easier by encapsulating a good deal of functionality, but there is still a lot of underlying XML processing that you have to be responsible for. For example, WebMethods do not validate messages against the implied schema. Because they are not validated, the response that's returned can result in unintended consequences. To address this, the authors extend the WebMethod framework by adding XML Schema validation through a custom SoapExtension class.

Contents
WebMethod Functionality
What WebMethods Miss
WebMethod Extensions
SoapExtension Configuration
SoapExtension Processing
ValidationExtension Implementation
The Schema Cache
A Better Schema Cache
Customizing Schema Definitions
Business Rules
Conclusion

A
SP.NET provides an infrastructure for mapping methods on ordinary classes, written in C# or Visual Basic® .NET, into Web Service operations (WebMethods) that support XML 1.0, XML Schema, SOAP, and Web Services Description Language (WSDL). This gives developers a familiar programming model for building Web Services without having to write any code to produce or process the XML messages used on the wire.

Although WebMethods can greatly increase productivity, you must still pay attention to certain aspects of the underlying XML processing. WebMethods do not fully validate the messages they process against the implied schema. As a result, it's possible for a given WebMethod invocation to produce a false positive (a successful response with an incorrect or unintended result) when an exception would have been expected. Also, they don't offer built-in support for declaring additional message-level constraints on business rules. The lack of validation support complicates the error handling that must be built into each WebMethod.

In this article, we'll show you how to compensate for this issue by using the WebMethod extensibility hooks built into ASP.NET. The full sample is available for download from the link at the top of this article.

WebMethod Functionality

ASP.NET makes it possible to map traditional methods to Web Service operations through the System.Web.Services.WebMethod attribute. For example, the following class exposes the CalcArea method as a Web Service:

using System.Web.Services;

[WebService(Namespace="http://example.org/geometry/")]
public class Geometry
{
    [WebMethod]
    public double CalcArea(double length, double width)
    {
        return length * width;
    }
}
To invoke this WebMethod, you must provide an .asmx file that refers to the Geometry class, as shown here:
<!-- geometry.asmx -->
<%@ WebService class="Geometry" language="C#" %>

Although it's more common to keep the source code in a separate file, it's also possible to include the class definition within the .asmx file itself. In this case, the class is just-in-time compiled the first time the .asmx file is requested.

When a request comes in for geometry.asmx, the WebServiceHandler class (.asmx HTTP handler) translates the SOAP request into a traditional method call. For CalcArea, it expects the following SOAP message:

<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CalcArea xmlns="http://example.org/geometry/">
      <length>2.3</length>
      <width>1.2</width>
    </CalcArea>
  </soap:Body>
</soap:Envelope>
This request produces the following SOAP response:
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CalcAreaResponse xmlns="http://example.org/geometry/">
      <CalcAreaResult>2.76</CalcAreaResult>
    </CalcAreaResponse>
  </soap:Body>
</soap:Envelope>

In a nutshell, the WebMethod infrastructure takes care of dispatching incoming SOAP requests to the appropriate class and method (based on either the request element's name or the SOAPAction header, configurable through SoapDocumentService.RoutingStyle) and serializing/deserializing the XML message into the common language runtime (CLR) objects, including the translation of .NET-style exceptions into SOAP Fault elements.

In addition, the infrastructure automatically generates a WSDL definition that describes the WebMethods using XML Schema types and WSDL constructs (http://localhost/geometry/geometry.asmx?wsdl). The System.Web.Services namespace contains a slew of additional attributes used to control dispatching as well as the XML Schema and WSDL mapping details.

What WebMethods Miss

If you don't take advantage of WebMethod, you have to write an HTTP handler that processes the SOAP request, extracts the width and length text representations, converts the text to a numeric value type, calculates the area, and then constructs the SOAP response. There is plenty of support in the Microsoft® .NET Framework for writing such a handler, but by using WebMethod all you have to do is calculate the area.

Although this is a significant productivity gain that most developers will want to take advantage of, it doesn't excuse you from having to deal with (or at least thinking about) certain aspects of the underlying message processing. Typically, you'll need more control over what your WebMethods will accept than what is provided by the default behavior. Consider this XML Schema complex type definition generated for the CalcArea WebMethod:

<s:element name="CalcArea">
  <s:complexType>
    <s:sequence>
      <s:element minOccurs="1" maxOccurs="1"
                 name="length" type="s:double" />
      <s:element minOccurs="1" maxOccurs="1"
                 name="width" type="s:double" />
    </s:sequence>
  </s:complexType>
</s:element>
The complex type definition states that the CalcArea element must contain a sequence of two required elements, length followed by width, both of which must be of type double.

Now consider the following unexpected SOAP request:

<soap:Envelope
    xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CalcArea xmlns="http://example.org/geometry/">
        <Length>2.3</Length>
        <Width>1.2</Width>
    </CalcArea>
  </soap:Body>
</soap:Envelope>
At first glance this message looks fine but there's actually a subtle problem with the case of the length and width fields. However, instead of producing a SOAP fault, the WebMethod produces the following successful response:
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CalcAreaResponse xmlns="http://example.org/geometry/">
      <CalcAreaResult>0</CalcAreaResult>
    </CalcAreaResponse>
  </soap:Body>
</soap:Envelope>
In fact, the following request, which is missing both the length and width elements, also produces a successful result with a value of 0:
<soap:Envelope
    xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CalcArea xmlns="http://example.org/geometry/"/>
  </soap:Body>
</soap:Envelope>

Or consider this request, where the client clearly has the wrong idea about how the WebMethod works:

<soap:Envelope
   xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <CalcArea xmlns="http://example.org/geometry/">
        <Length>2.3</Length>
        <Width>1.2</Width>
        <Length>5.0</Length>
        <Width>3.4</Width>
        <Length>9.2</Length>
        <Width>2.8</Width>
    </CalcArea>
  </soap:Body>
</soap:Envelope>
In this case the WebMethod returns a successful response with the value of 2.76, as we illustrated in the very first example.

These situations, and many others, cause WebMethods to produce false positives. For certain applications it may not be a big deal, but for others it could be catastrophic and is therefore completely unacceptable. For these applications, some form of validation is required, regardless of the implementation technique that you use.

The reason for this behavior has to do with XmlSerializer, the underlying plumbing that takes care of object deserialization. XmlSerializer is very forgiving by design. It happily ignores XML nodes that it didn't expect and will use default CLR values for expected but missing XML nodes. It doesn't perform XML Schema validation and the consequence is that details like structure, ordering, occurrence constraints, or simple type restrictions are not enforced during deserialization. This issue is described in more detail in the November 2002 installment of The XML Files.

It's possible to force XmlSerializer to perform validation during deserialization by supplying an XmlValidatingReader object to the call to deserialize, as shown here:

XmlReader xr = new XmlTextReader(fileName);
XmlValidatingReader vr = new XmlValidatingReader(xr);
vr.Schemas.Add(schemaNamespace, schemaLocation);
object validatedObject = ser.Deserialize(vr);
However, within the WebMethod infrastructure, there are two problems. First, it's not possible to get your hands on the underlying XmlSerializer, and second, the schema file doesn't exist as part of the project (the schema is derived from the CLR attributes decorating the method, and it's only generated when clients issue a GET request with the ?wsdl query string).

WebMethod Extensions

Ideally, it would be possible to enable WebMethod validation by "flipping a switch," but unfortunately it's not. In some cases, it's not even possible to detect error conditions, like missing information, from within a WebMethod implementation. Therefore, you must take advantage of the provided extensibility points in order to add validation support.

There are two general WebMethod extension mechanisms: HTTP modules and SoapExtension classes. An HTTP module is simply a class derived from IHttpModule and configured to run on a particular virtual root (using web.config) or on the entire machine (using machine.config). HTTP modules can be used to extend any type of HTTP application, not just WebMethods. HTTP modules make it possible to perform pre- and post-processing on the HTTP messages traveling through the pipeline (see Figure 1).

Figure 1 HTTP Modules
Figure 1 HTTP Modules

HTTP modules force you to work completely at the message level. Hence, it's a fairly low-level approach that requires you to implement SOAP details manually (like generating SOAP faults). Since HTTP modules are configured to run for an entire virtual root, it's quite difficult to enable functionality or supply extra configuration information on a class-per-class or method-per-method basis.

The other technique for extending WebMethods, managed entirely by the .asmx handler, is commonly referred to as the SoapExtension framework. To use this approach, you derive a new class from SoapExtension and override a few of its members. Then you configure the SoapExtension to run by using a configuration file or a custom attribute. The SoapExtension will be called before and after each WebMethod invocation, shown in Figure 2.

Figure 2 SoapExtensions
Figure 2 SoapExtensions

The hooks that allow the WebMethods to be extended are quite flexible. We are not going to look at all possible ways of using them, only at how to examine and validate messages associated with a particular method prior to invocation. The hook will validate the incoming message against the corresponding XML Schema. Based on the validation results, the message will either be allowed to proceed to the WebMethod or an appropriate SOAP fault will be returned to the client, describing the error condition. Figure 3 shows the basic flow.

Figure 3 Validating Messages Before Invocation
Figure 3 Validating Messages Before Invocation

SoapExtension is the preferred framework for extending WebMethods because it gives you much more configuration control and makes it possible for you to customize extension behavior on a method-per-method basis.

SoapExtension Configuration

You can configure SoapExtensions to run on an entire virtual root (similar to an HTTP module) or on a method-by-method basis. To configure a SoapExtension for an entire virtual root, you simply add an entry to web.config. Here's a sample web.config file that registers a custom SoapExtension called ValidationExtension:

<configuration>
  <system.web>
    <webServices>
      <soapExtensionTypes>
        <add type="ValidationExtension,
                   DevelopMentor.Web.Services"
             priority="1" group="0" />
      </soapExtensionTypes>
    </webServices>
  </system.web>
</configuration>

In order to configure a SoapExtension on a method-by-method basis, you need to write a custom attribute derived from SoapExtensionAttribute (see Figure 4). The ExtensionType property returns the type of SoapExtension to use, in this case ValidationExtension. You should take note that this attribute can only be applied to a method, and then only once. This way the compiler will catch most of the places where this attribute might be misapplied. Limiting it to a single instance on a method just makes it easier to implement the extension.

Figure 4 SoapExtension per Method (Close Figure 4)
[AttributeUsage(AttributeTargets.Method,
                AllowMultiple=false)]
public class ValidationAttribute : SoapExtensionAttribute
{
    int priority = 0;

    // used by soap extension to get the type
    // of object to be created
    public override System.Type ExtensionType
    {
        get { return typeof(ValidationExtension); }
    }
    public override int Priority
    {
        get { return priority; }
        set { priority = value;    }
    }
}

Then you can decorate your WebMethods with this attribute to tell the infrastructure to call your SoapExtension at runtime. Note that if your attribute's name ends in "Attribute", you can leave it off when it's used:

[Validation]
[WebMethod]
public double CalcArea(double length, double width)
{
    return length * width;
}
You can also design your attributes to take additional configuration information through public fields or properties—you'll see an example of this shortly. Overall, this configuration technique ties in nicely with WebMethods.

SoapExtension Processing

The ValidationExtension class must derive from the SoapExtension base class and override the required members, as shown in Figure 5. Figure 6 provides a helpful summary describing when each of the different SoapExtension methods is called by the infrastructure. However, deciding how to implement them requires a thorough understanding of the execution process.

Figure 6 SoapExtension Methods (Close Figure 6)

Method | Description
GetInitializer(serviceType) | Called once per vroot (if registered in the configuration file) the first time any Web-Method is invoked); stores returned object
GetInitializer(methodInfo, attribute) | Called once per configured WebMethod (if configured via SoapExtensionAttribute) the first time any configured WebMethod is invoked; stores returned object
Initiailize(object) | Called once per configured WebMethod invocation (receives object returned by GetInitializer)
ProcessMessage(...) | Called four times per WebMethod invocation (this is where you do the pre/post processing work)
Figure 5 ValidationExtension Class (Close Figure 5)
public class ValidationExtension : SoapExtension
{
    public override object GetInitializer(
      System.Type serviceType)
    {
        // TODO
}
    public override object GetInitializer(
      LogicalMethodInfo methodInfo,
      SoapExtensionAttribute attribute)
    {
        // TODO
}
    public override void Initialize(object initializer)
    {
        // TODO
}
    public override void ProcessMessage(SoapMessage message)
    {
        // TODO
}
}

Each time a WebMethod is invoked, the infrastructure creates a new instance of the SoapExtension class. The fact that a new SoapExtension instance is created on each invocation poses a state management issue. You can't store state in member variables since between calls they go away. You also don't want to do expensive initialization in the constructor for obvious performance reasons. Hence, the SoapExtension framework provides the GetInitializer and Initialize methods for performing one-time initialization.

The first time that any WebMethod in a virtual root is invoked, the infrastructure checks to see if any SoapExtensions have been configured in the applicable configuration files (web.config or machine.config). If so, the infrastructure creates an instance of each configured SoapExtension class and calls the first GetInitializer method shown in Figure 6. This is your chance to perform any necessary initialization, wrap the information in an object, and return the object to the infrastructure. The infrastructure stores the returned object and then passes it to Initialize, which is subsequently called on each WebMethod invocation after constructing the SoapExtension class.

During this first WebMethod invocation, the infrastructure also inspects each WebMethod to see if it also has any attributes that derive from SoapExtensionAttribute. If it does, it makes an instance of each attribute (with any supplied information) and asks via the ExtensionType property for the SoapExtension-derived class that should be called on each invocation. For each of these classes, the infrastructure creates an instance and calls the second GetInitializer method described in Figure 6, passing in the supplied attribute. This is your chance to perform WebMethod-specific initialization. The object you return from GetInitializer is stored by the infrastructure for that particular WebMethod and is passed to Initialize during each future invocation. The entire initialization process is illustrated in Figure 7.

Figure 7 Initialization Process
Figure 7 Initialization Process

There are two parts to initialization; one happens only on the first call and the other every time a method is called. The first initialization happens when GetInitializer is called and returns an initializer object which the runtime caches for future use. The second part happens when Initialize is called and also happens whenever a method is called, even the first time. Initialize is given the Initializer object for use by the ProcessMessage function. This allows the time required to create the Initializer object to be amortized over all the calls made to a method. ProcessMessage is then called four times, one for each message processing state (BeforeDeserialize, AfterSerialize, BeforeSerialize, and AfterSerialize).

Inside Initialize, you typically just want to store the supplied object in a member variable for future use in ProcessMessage, where the real pre- and post-processing occurs. There are four stages in the processing of a WebMethod: BeforeDeserialize, AfterDeserialize, BeforeSerialize, and AfterSerialize (see Figure 2). You can write code to execute during any of these stages within ProcessMessage. The WebMethod implementation is invoked between AfterDeserialize and BeforeSerialize.

ValidationExtension Implementation

To implement a validation extension, we want to perform XML Schema validation before the System.Web.Services has done anything with the message. We'll hook our logic into the BeforeDeserialize stage, as illustrated in Figure 8.

Figure 8 BeforeDeserialize (Close Figure 8)
public override void ProcessMessage(SoapMessage message)
{
  if (SoapMessageStage.BeforeDeserialize == message.Stage)
  {
    try
    {
         // perform XML Schema validation here
         // configure XmlValidatingReader
         XmlTextReader tr = new XmlTextReader(message.Stream);
         XmlValidatingReader vr = new XmlValidatingReader(tr);
         ... // load schema files in XmlValidatingReader
         while (vr.Read()) ; // read through stream
    }
    finally
    {
      // reset stream position
      message.Stream.Position = 0;
    }
  }
}

This implementation loads the raw message stream into an XmlTextReader and wraps it with an XmlValidatingReader. After loading the schema files into XmlValidatingReader's schema cache (Schemas property), it simply reads through the stream to perform validation. If XmlValidatingReader encounters a validation problem in the message, it throws an exception containing a description of the problem. The exception is translated into a SOAP Fault response message and returned to the client without invoking the WebMethod. Before returning from ProcessMessage, we also want to reset the stream's position; otherwise the infrastructure won't know how to deserialize the message properly. The only tricky part about ProcessMessage is loading the schemas into XmlValidatingReader.

The Schema Cache

In order to perform validation on the incoming requests, XmlValidatingReader needs access to the schema definitions that describe the messages. The question is, where does our code look for the schema definitions? There are a variety of possible answers. Probably the easiest solution is to load all schema files from a well-known location in the virtual root, like a child xsd directory. Then users of this extension can simply drop the required schema files into the directory and they'll be automatically picked up.

Since it could cost you some performance to load up all the schema files, this is a perfect example of where you'll want to use GetInitializer. Within GetInitializer we need to create an instance of XmlSchemaCollection and fill it up with all of the compiled schemas. We need to do the same thing regardless of which GetInitalizer version is called, so we'll benefit ourselves by creating a helper method, GetInitializerHelper, which both GetInitializer versions delegate to (see Figure 9).

Figure 9 GetInitializerHelper (Close Figure 9)
public object GetInitializerHelper()
{
  // XML Schema cache for schema validation
  XmlSchemaCollection sc = new XmlSchemaCollection();
  // load schemas from vroot cache
  HttpContext ctx = HttpContext.Current;
  if (Directory.Exists(ctx.Server.MapPath(relativeDir)))
  {
      string[] schemaFiles = Directory.GetFiles(
        ctx.Server.MapPath("xsd"), "*.xsd");
      foreach (string schemaFile in schemaFiles)
      {
        XmlTextReader r = new XmlTextReader(filePath);
        XmlSchema schema = XmlSchema.Read(r, null);
        sc.Add(schema);
      }
  }
  return sc;
}

Remember that the object returned from GetInitializer is passed into Initialize during each future WebMethod invocation. Our implementation of Initialize caches the XmlSchemaCollection object into a member variable for use within ProcessMessage:

public override void Initialize(object initializer)
{
    _context = (XmlSchemaCollection)initializer;
}
Then, going back to our implementation of ProcessMessage, we can associate the stored XmlSchemaCollection object with the XmlValidatingReader before processing the stream:
// ProcessMessage method
XmlTextReader tr = new XmlTextReader(message.Stream);
XmlValidatingReader vr = new XmlValidatingReader(tr);
vr.Schemas.Add(_context); // load schema cache
while (vr.Read()) ; // read through stream
With this, we should have enough foundation in place to begin using the ValidationExtension.

To experiment with the example described thus far, you'll need to follow these steps:

  1. Add a reference in your project to the ValidationExtension assembly (DevelopMentor.Web.Services).
  2. Annotate your WebMethods with [Validation].
  3. Create an xsd directory within the vroot.
  4. Download the SOAP 1.1 schema from http://schemas.xmlsoap.org/soap/envelope, and save it to the xsd directory.
  5. Browse to the .asmx endpoint, press Service Description (.asmx?wsdl), and then save the WSDL file to the xsd directory.
  6. Extract the schema definition from the saved WSDL file and save it to a new .xsd file (make sure you get all the namespace declarations).

Now, invoking CalcArea with an invalid request produces a SOAP Fault. For example, the invalid request containing uppercase letters (Length/Width) produces the response seen in Figure 10. Notice that the SOAP Fault contains precise information about what went wrong during validation. All false positive situations described earlier will now be detected thanks to our schema definition.

Figure 10 SOAP Fault (Close Figure 10)
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <soap:Fault>
      <faultcode>soap:Server</faultcode>
      <faultstring>Server was unable to process request. —&gt; Element
      'http://example.org/geometry/:CalcArea' has invalid child element
      'http://example.org/geometry/:Length'. Expected
      'http://example.org/geometry/:length'. An error occurred at
      (4, 3).</faultstring>
      <detail />
    </soap:Fault>
  </soap:Body>
</soap:Envelope>

A Better Schema Cache

It's easy to get the ValidationExtension to execute by either annotating with [Validation] or adding an entry to web.config. The challenge is setting up the schema cache. To help simplify this process, we decided to add a few features to our extension.

First, we automatically load the SOAP 1.1 schema from the assembly resource file. There is a call to a helper function in GetInitalizerHelper that takes care of this and adds the SOAP 1.1 schema to the XmlSchemaCollection object:

LoadSchemaFromResourceFile(sc, "ValidationResources",
   "DevelopMentor.Web.Services", "SOAP1.1");

This takes care of the SOAP schema, but there's still the issue of loading up your own schema. Since it's a bit cumbersome to extract the schema definition from the WSDL file before you can use it, we also made it possible to load schemas directly from a complete WSDL file by extending the code that automatically loads .xsd files with the following:

string[] wsdlFiles = Directory.GetFiles(
  ctx.Server.MapPath("xsd"), "*.wsdl");
foreach (string wsdlFile in wsdlFiles)
{
    ServiceDescription sd=ServiceDescription.Read(wsdlFile);
    foreach(XmlSchema embeddedXsd in sd.Types.Schemas)
       sc.Add(embeddedXsd);
}

It's also possible to just drop the WSDL file into the xsd directory and be ready to go. Also, to give you more flexibility in terms of where you want to store your XML Schema files, we've defined a few additional attributes that can be used in conjunction with the ValidationExtension: ValidationSchemaAttribute and ValidationSchemaCacheAttribute, both of which are used on the class definition. ValidationSchemaAttribute allows you to specify a schema file location (relative to vroot) that you'd like to load into the cache:

[WebService(Namespace="http://example.org/geometry/")]
[ValidationSchema("schemas/geo/geo.xsd")]
[ValidationSchema("schemas/global/math.xsd")]
[ValidationSchema("schemas/global/simple.xsd")]
public class Geometry {
    •••
}

ValidationSchemaCache allows you to specify additional directories (other than xsd) that will contain a bunch of .xsd or .wsdl files, all of which will be loaded into the cache by GetInitializer. Here's an example of it in action:

[WebService(Namespace="http://example.org/geometry/")]
[ValidationSchemaCache("schemas")]
[ValidationSchemaCache("schemas/global")]
public class Geometry {
 •••
}

The attribute class definitions are straightforward—they derive from System.Attribute and provide a public property for storing the file/directory location, respectively. GetInitializerHelper then inspects the class (serviceType) to see if they were used and if so, it retrieves the information and loads from the specified locations. Here's the code taken from GetInitializerHelper:

// load schemas from user-defined file locations
   atts = serviceType.GetCustomAttributes(
   typeof(ValidationSchemaAttribute), true);
 foreach(ValidationSchemaAttribute vsa in schemaLocations)
   LoadSchemaFromFile(sc, ctx.Server.MapPath(vsa.Location));

// load schemas from user-defined directories
   object[] atts = serviceType.GetCustomAttributes(
   typeof(ValidationSchemaCacheAttribute), true);
foreach (ValidationSchemaCacheAttribute loc in atts)
    LoadSchemasFromDirectory(sc, loc.RelativeDirectory);
This gives the user much more control over where the schema and WSDL files reside in their system.

As a final simplification, we wanted to make it possible for developers to avoid having to manage schema files altogether. The .asmx handler provides support for generating a WSDL document that describes the endpoint. We can leverage this functionality to generate a WSDL document that contains an XML Schema definition that we can use for validation purposes. This makes it possible to turn on validation for a given WebMethod without requiring the developer to manage a schema file by hand. The following method illustrates this process:

internal void LoadReflectedSchemas(
    XmlSchemaCollection sc, Type type, string url)
{
    ServiceDescriptionReflector r=
          new ServiceDescriptionReflector();
        r.Reflect(type, url);
        foreach (XmlSchema xsd in r.Schemas)
            sc.Add(xsd);
        foreach (ServiceDescription sd in r.ServiceDescriptions)
            LoadSchemasFromServiceDescriptions(sc, sd);
}

You supply the class type for which you'd like to generate WSDL and then iterate through the generated Schema collections. You can call this from either the GetInitializer method by passing serviceType or methodInfo.DeclaringType, respectively. Note that user-defined schemas always take precedence over this technique.

With all this code in place, you can cross off the last four steps listed earlier; the extension will now work. Just add the assembly to your project, annotate with [Validation], and you're finished.

Customizing Schema Definitions

Just enabling XML Schema validation will simplify WebMethod error handling tremendously. It's now possible to constrain the message structure as well as the lexical and value spaces of text values. If you don't want to worry about managing schema files, you're basically stuck with what's produced by the infrastructure. But if you're willing to take control of your schemas (using the techniques I've just described), you can use more advanced schema features. For example, consider the following WebMethod:

public void AddEmployee(string name, string id, DateTime
  bday, double salary) { ... }

The generated schema for this WebMethod will automatically constrain the values using XML schema's built-in data types. However, string and double don't precisely describe the expected formats for the id and salary values, respectively. Consider the following XSD simple type definitions:

<xs:simpleType name="SSN">
    <xs:restriction base="xs:string">
        <xs:pattern value="\d{3}-\d{2}-\d{4}" />
    </xs:restriction>
</xs:simpleType>
<xs:simpleType name="NewHireSalary">
    <xs:restriction base="xs:double">
        <xs:minExclusive value="0" />
        <xs:maxExclusive value="5000" />
    </xs:restriction>
</xs:simpleType>
These definitions precisely describe the data format expected by the operation. The SSN type utilizes a regular expression to constrain the allowed string format while NewHireSalary explicitly defines the allowed value space. To leverage these types during validation, simply modify your schema file to use them, as shown in Figure 11. Now if the client provides an id value that's not in Social Security Number format, or a salary that's not between 0 and 5000, the ValidationExtension will catch it and will return a descriptive SOAP Fault.

Figure 11 New Types (Close Figure 11)
<s:element name="AddEmployee">
  <s:complexType>
    <s:sequence>
      <s:element minOccurs="0" maxOccurs="1"
                 name="name" type="s:string" />
      <s:element minOccurs="0" maxOccurs="1"
                 name="id" type="tns:SSN" />
      <s:element minOccurs="1" maxOccurs="1"
                 name="bday" type="s:dateTime" />
      <s:element minOccurs="1" maxOccurs="1"
                 name="salary" type="tns:NewHireSalary" />
    </s:sequence>
  </s:complexType>
</s:element>

Business Rules

As we just illustrated, using the ValidationExtension enables you to take full advantage of the XML Schema language. You can control complex structural details, simple type value spaces, uniqueness, substitution mechanisms, and more as the language is quite extensive. XML Schema does, however, have some limits. One of the most apparent limitations is that you can't enforce co-occurrence constraints where the presence or value of a given node affects the constraints on other nodes.

Co-occurrence constraints are typical of most business rules in the real world. Business rules typically specify arbitrary constraints on the data beyond its type. What kind of arbitrary constraints might there be on the data? Consider the CalcArea operation where we're dealing with length and width elements. In the real world, the length is by definition greater than the width.

Although the choice is arbitrary, in a manufacturing process it is important that people have a consistent vocabulary so that boxes fit on belts and through openings, like making sure to put a tray lengthwise on the belt. Let's assume this is the case here. This rule is something that you can't enforce using XML Schema, but you can take care of it in your CalcArea method implementation. By adding a little procedural code, you can check the input parameters before you calculate the area. But since business rules typically come from corporate specifications, it would be much cleaner to declaratively include them in the class definitions.

XPath is an ideal language for defining these additional constraints. It was designed specifically for the purpose of evaluating the content of an XML document; it's easy to use and quite flexible. Furthermore, XPath is implemented for virtually all platforms, languages, and even databases in common use today. Therefore, if we use XPath as the rule language, rules can be enforced on other platforms using other languages, tools, or even databases. This means that business rules can be pushed anywhere in the chain of processing, even to the client.

Using XPath to make assertions about XML documents is not a new idea. Schematron is a proposed language for doing this in a standard manner (see http://www.ascc.net/xml/schematron). Schematron can be integrated with XML Schema definitions and evaluated during a post-validation step. Daniel Cazzulino has written an open-source implementation of Schematron for the .NET Framework that you can find at http://sourceforge.net/projects/schematron-net.

We decided that it would be useful to build this type of XPath constraint processing right into the ValidationExtension class. To do so, we defined a new attribute class called AssertAttribute that allows you to specify XPath assertions to an individual method or the entire class (applies to all WebMethods). An example of these attributes in use is shown in Figure 12.

Figure 12 Assertions (Close Figure 12)
[AssertNamespaceBinding("s",
  "http://schemas.xmlsoap.org/soap/envelope/")]
[AssertNamespaceBinding("t",
  "http://example.org/geometry/")]
[WebService(Namespace="http://example.org/geometry/")]
[Assert(@"//t:width >= 0", "width must be greater than 0")]
[Assert("(//t:length > //t:width)",
        "Length must be greater than width")]
public class Geometry
{
    [WebMethod]
    [Validation]
    [Assert("(//t:length * //t:width) > 100",
            "Area must be greater than 100")]
    [Assert("(//t:length div //t:width) = 2",
            "Length must be twice the size of width")]
    public double CalcArea(double length, double width)
    {
        return length * width;
    }
    [WebMethod]
    [Validation]
    public double CalcPerimeter(double length, double width)
    {
        return length * width;
    }
}

For more information on how this part of the project was implemented, stay tuned for our next piece where we'll continue describing the XPath assertion functionality as well as other cool topics like extending the WSDL generation process to include the business rule descriptions. However, if you're itching to take a look at these topics now, download the source code from the link at the top of this article and check out the sample.

Conclusion

The WebMethod infrastructure provides a powerful extensibility framework for adding additional user-defined behavior. We were able to add validation support to any WebMethod through the ValidationExtension and ValidationAttribute classes. This technique simplifies WebMethod error handling and fully utilizes the power and flexibility offered by XML Schema.

For related articles see:
The XML Files: A Quick Guide to XML Schema
The XML Files: A Quick Guide to XML Schema-Part 2

For background information see:
Real World XML Web Services: For VB and VB .NET Developers by Yasser Shohoud (Addison-Wesley, 2002)
Essential XML Quick Reference: A Programmer's Reference to XML, XPath, XSLT, XML Schema, SOAP, and More by Aaron Skonnard and Martin Gudgin (Addison-Wesley, 2001)
Building XML Web Services for the Microsoft .NET Platform by Scott Short (Microsoft Press, 2002)

Aaron Skonnard is an instructor and researcher at DevelopMentor, where he develops the XML and Web Service-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.

Dan Sullivan is an independent consultant and an instructor and course author at DevelopMentor. He has worked in all facets of computing, from designing processors to writing applications and system software. Dan now focuses on XML and developing Web Services. He can be contacted at dsullivan@net1plus.com.


Real-World XML: Manipulate XML Data Easily with the XPath and XSLT APIs in the .NET Framework

Real-World XML

Manipulate XML Data Easily with the XPath and XSLT APIs in the .NET Framework

Dino Esposito
This article assumes you're familiar with Visual Basic .NET
Level of Difficulty 1 2 3
Code download available at: XPathandXSLT.exe (166KB)
SUMMARY

XPath is emerging as a universal query language. With XPath, you can identify and process a group of related nodes in XML-based data sources. XPath provides an infrastructure that is integral to XML support in the .NET Framework. The XPath navigation model is even used under the hood of the XSLT processor. In this article, the author reviews the implementation details of the XPath navigator and the XSLT processor and includes practical examples such as asynchronous transformations, sorted node-sets, and ASP.NET server-side transformations.

Contents
XPath Expressions
The XPath Navigator
Documents, Navigators, and Readers
Querying for Nodes
Programming with XPathNavigator
Sorting the Node-Set
A Whistle-stop Tour of XSL in .NET
The XslTransform Class
Arguments and Extension Objects
Asynchronous XSLT
The <asp:xml> Tag
Conclusion

O
ne of the key advantages of XML is that it lets you mark portions of text with tags and attributes. You tag data inside text because you plan to retrieve it later. So how do you do this? You use XPath.

Though it doesn't have an XML-based syntax, XPath is the language defined to address parts of an XML document in a compact, relatively straightforward way. More importantly, XPath defines a common syntax so you can retrieve nodes from within classes that implement the XML document object model (DOM) as well as from XSLT.

In the Microsoft® .NET Framework, the XPath query language is fully supported through the classes defined in the System.Xml.XPath namespace. The .NET Framework implementation of XPath is based on a language parser and an evaluation engine. The overall architecture of an XPath query is similar to that of a database query. Just as with SQL commands, you prepare XPath expressions and submit them to a runtime engine for evaluation. The query is parsed and executed against an XML data source. Next, you get back some information representing the resultset of the query. An XPath expression can return a node-set (that is, an ordered collection of nodes), a Boolean value, a number, or a string.

In this article, I'll show how XPath integrates with the XmlDocument class and XSLT in the .NET Framework. I'll also dig out the intricacies of the XPathNavigator class, used by the .NET Framework to walk through XML documents.

XPath Expressions

XPath is a query language specifically designed for addressing both the elements and the text of an XML document. The XPath notation is essentially declarative. Any valid expression declares a node pattern using a notation that emphasizes the hierarchical relationship between the nodes. Similar to file system paths, XPath expressions proceed from a root (an axis in the XPath terminology) to a particular set of nodes or a value in the source document. The similarity to the file system does not end here though. XPath expressions are always evaluated in the context of a node. The context node is designated by the application and represents the starting point of the query. It is not much different from the idea of the current directory.

The context of an XPath query includes, but is not limited to, a context node and a context node-set. The context node-set is the overall set of nodes processed by the query. Typically, it is a superset of the set of nodes that is actually returned to the application. An XPath context also contains position and namespace information, variable bindings, and a standard library of functions that applications can extend. Any implementation of the XPath parser provides a function library that is used to evaluate expressions. Extension functions are defined within vendor-specific XPath implementations but can also be provided by specialized and XPath-based programming APIs such as XSL Transformation and XPointer. XPath expressions generally return node-sets, but Booleans, strings, numbers, and other types are supported.

The most frequently used type of XPath expression is the location path. A location path is an expression that looks a lot like a file system path in that it can be either absolute or relative to the context node. Absolute location paths begin with the forward slash. As shown in Figure 1, a fully qualified location path is made of three pieces: an axis, a node test, and one or more predicates. The axis information defines the initial context node-set for the expression, whereas the node test is a sequence of node names that identifies a path in the node-set. A predicate is a logical expression that defines the criteria for filtering the current node-set.

Figure 1 Location Path
Figure 1 Location Path

An XPath expression can contain any number of predicates. If no predicate is specified then all the children of the context node are returned in the query. Otherwise, the conditions set with the various predicates are logically concatenated using a short-circuited AND operator. Notice that predicates are processed in the order in which they appear so that the next predicate works on the node-set generated by the previous one.

While being processed, an XPath expression is tokenized in subexpressions called location steps and each is evaluated individually. The XPath processor is iteratively passed the subexpression and the context node-set generated at the previous step. It returns a possibly narrowed node-set that will be used as the input argument for the next subexpression. During this process, the context node, position, and size may vary while variable and function references as well as namespace declarations remain intact. Each location step is actually a location path and, as such, can be expressed in an abbreviated or fully qualified form, as appropriate. Location steps are separated by forward slashes.

In the .NET Framework, you can use XPath expressions through methods exposed by the XmlNode class on the XmlDocument class or by means of the XPathNavigator class.

The XPath Navigator

In the .NET Framework, the XmlDocument class represents the standard XML DOM (DOM Level 2 standard) as ratified by the W3C. On each child node you reach from XmlDocument, a couple of search methods are implemented. The XmlNode class provides the SelectNodes and SelectSingleNode methods that use XPath expressions to search for nodes in the document. The methods are nearly identical to similarly named methods in the COM-based MSXML library. SelectNodes returns a list of objects whereas SelectSingleNode returns only the first object that matches the search criteria. Here's how you use SelectNodes:

Dim doc As XmlDocument = New XmlDocument()
doc.Load(fileName)
Dim nodes As XmlNodeList
nodes = doc.SelectNodes(queryString)
The SelectSingleNode method differs from SelectNodes in that it returns an individual XmlNode object. I'll have more to say about these methods in a moment.

The XmlDocument support for XPath expressions has two goals. It smoothes the transition from MSXML COM code to the .NET Framework, while giving you a built-in mechanism to search for nodes on a memory-mapped XML document. The XmlDocument query API, though, is a simple high-level wrapper. The core .NET Framework API for processing XPath expressions is built around the XPathNavigator class. The navigator is an XPath processor that works on top of any XML data store that exposes the IXPathNavigable interface. Rendered through the interface defined in the XPathNavigator class, the navigator parses and executes expressions using the Select method. Unlike the XmlDocument methods, the navigator accepts expressions both as plain text and through precompiled objects. You can programmatically access the XPathNavigator object either from the XmlDocument class or the XPathDocument class. Figure 2 illustrates the two ways to access XPath functions in the .NET Framework.

Figure 2 Accessing XPath Functions
Figure 2 Accessing XPath Functions

The following code snippet shows how to create an XPathNavigator from an XML document and execute an XPath query:

Dim doc As XPathDocument
Dim nav As XPathNavigator
Dim iterator As XPathNodeIterator
doc = New XPathDocument(fileName)
nav = doc.CreateNavigator()
iterator = nav.Select(queryString)
While iterator.MoveNext()
' nav points to the node subtree
End While
The navigator returns an XPath iterator object. Iterators are nothing more than specialized enumerator objects which are used to move through the returned node-set. I'll discuss iterators shortly.

Documents, Navigators, and Readers

Prior to the advent of the .NET Framework, processing an XML file was a matter of working with documents rendered according to the SAX or the XmlDocument specification.

The XPath processor, which is the engine that parses and executes XPath queries, is built under the covers of the XPathNavigator class. As shown in Figure 2, XPath calculations are always delegated to a navigator, regardless of the high-level caller API. A navigator works on top of a particular data store, which is typically an instance of the XPathDocument class. Other data stores can be used as well as long as the data store class implements the IXPathNavigable interface, like this:

public interface IXPathNavigable {
    XPathNavigator CreateNavigator();
}
Besides the XPathDocument class, examples of data stores include XmlDocument and XmlDataDocument.

A data store is responsible for providing a navigator to explore in-memory XML content. An XPathNavigator implementation is always store specific and is built by inheriting from the XPathNavigator abstract class. Although in practice you always program navigators through the common reference type of XPathNavigator, each data store class has its own navigator object. They are internal, undocumented classes, programmatically inaccessible, and are often implemented in fairly different ways. Figure 3 lists the real navigator classes used by the three XPath data stores defined in the .NET Framework. The document-specific navigator exploits the internal layout of the document class in order to provide the navigation API.

Figure 3 Navigator Classes (Close Figure 3)

Data Store Class | Corresponding Internal Navigator Class
XPathDocument | System.Xml.XPath.XPathDocumentNavigator
XmlDocument | System.Xml.DocumentXPathNavigator
XmlDataDocument | System.Xml.DataDocumentXPathNavigator

The XPathDocument class provides a highly optimized and read-only in-memory store for XML documents. Specifically designed to implement the XPath data model, this class does not provide any identity for nodes. It simply creates an underlying tree of node references to let the navigator operate quickly and effectively. The internal architecture of the XPathDocument class looks like a linked list of node references. Nodes are managed through an internal class (XPathNode) that represents a small subset of the XmlNode class (see Figure 2).

Compared to XPathDocument, the XmlDocument class provides read-write access to the nodes of the underlying XML document. In addition, each node can be individually accessed.

The XmlDocument class also provides the ability to create a navigator object, as shown here:

Dim doc As XmlDocument = New XmlDocument()
doc.Load(fileName)
Dim nav As XPathNavigator = doc.CreateNavigator()
The XmlDocument's navigator class implements the IHasXmlNode interface. This interface defines one method, GetNode:
public interface IHasXmlNode {
XmlNode GetNode();
}
Using this method, callers can access and interrogate the currently selected node on the XmlDocument based on the XPathNavigator's position. This feature is simply impossible to implement for navigators based on XPathDocument because it does not expose the internal structure via the XmlNode class like the XmlDocument class does. This is by design. The XPathDocument minimizes the memory footprint and does not provide for node identity.

Because the GetNode method is implemented on the XPathNavigator class on the XmlDocument, callers can take advantage of it using a type cast, as shown in the following code snippet:

Dim doc As XmlDocument = New XmlDocument()
doc.Load(fileName)
Dim nav As XPathNavigator = doc.CreateNavigator()
Dim i As XPathNodeIterator = nav.Select(query)
Dim node As XmlNode = CType(i.Current, IHasXmlNode).GetNode()
At this point, the caller program has gained full access to the node and can read and update it at will.

Finally, the XmlDataDocument class is an extension of XmlDocument which is aimed at allowing the manipulation of a relational DataSet through XML. This is a neat example of the fact that the .NET Framework navigation API can be applied to XML-based data as well as XML-like (data that's structured like XML, but isn't XML) data.

If you look at the MSDN® documentation for XPath navigators, you'll see that a navigator reads data from XML-based data stores in a cursor-like fashion (forward and backward) and provides read-only access to the underlying data. In addition, it holds information about the current node and advances the internal pointer using a variety of move methods. When the navigator is positioned on a given node, all of its properties reflect the value of that node. This, of course, resembles XML readers, which I looked at in an article in the May 2003 issue of MSDN Magazine. So what's the difference between navigators and readers?

Navigators and readers are different animals. Readers are analogous to fire-hose cursors, providing basic read-only, forward-only movement. Navigators are read-only as well but they provide a significantly richer set of move methods including both forward and backward options. Navigators also provide several select methods to fine-tune the search. Readers are lower-level tools you can use to read XML-based or XML-like data, as well as build in-memory data structures. XML readers can be used to build the in-memory data structures that navigators rely upon.

Querying for Nodes

Suppose that you need to implement an XPath query in a .NET Framework-based application. Should you use XPath navigators or are you better off sticking to the XmlDocument's nodes interface? The XmlNode's SelectNodes method internally employs a navigator object to retrieve the list of matching nodes. The return value of the navigator's Select method is then used in order to initialize an internal node list object of type XPathNodeList, defined in the System.Xml.XPath namespace. As you may have already guessed, this class inherits from XmlNodeList which is a documented class. In addition, unlike SelectNodes, navigators can fully exploit compiled expressions.

The SelectNodes method always accepts the XPath expression as plain text. The string is then passed verbatim to the navigator. In only one case does the underlying navigator receive a compiled expression. If you use the overload of the SelectNodes method that handles namespace information, the XPath expression is first compiled and then passed to the processor. The overload is prototyped as follows:

Function SelectNodes( _
    xpathExpr As String, _
    nsm As XmlNamespaceManager) _
As XmlNodeList
The advantage of using compiled expressions becomes palpable when you frequently reuse the expression in the session, and they can be namespace aware. The XmlNamespaceManager is the class that allows the user to specify the prefix of the namespace to bind.

The SelectSingleNode method works as a special case of SelectNodes in that it returns only the first element of the returned node-set. Unfortunately, up until now, the implementation of SelectSingleNode has not been particularly efficient. If you need to locate only the first matching node, then calling SelectSingleNode or SelectNodes is nearly identical. Moreover, if you need to squeeze out every little bit of performance, you're probably better off using SelectNodes. The following pseudocode illustrates the current implementation of SelectSingleNode:

Function SelectSingleNode(xpathExpr As String) As XmlNode
   Dim nodes As XmlNodeList = SelectNodes(xpathExpr)
   Return nodes(0)
End Function

The method internally calls SelectNodes and returns the first node that is a match. Note, however, that XmlNodeList is built dynamically—the next node is searched for only when requested.

A better way to query for a single node would be to pass SelectNodes with a more precise XPath expression that returns a single node. The idea is to avoid generic wildcard expressions like this:

NorthwindEmployees/Employee
You should place a stronger filter on the XPath expression so it returns a properly sized subset of nodes. To get only the first node, add an extra predicate that stops the query after the first match:
NorthwindEmployees/Employee[position() = 1]
This is an XPath best practice in general and is not due to the .NET Framework implementation in particular. The approach could be refined and generalized to size the node-set as is appropriate. This query string shows how to get the first n nodes that match:
NorthwindEmployees/Employee[position() < n+1]

I would suggest using XmlNode's SelectNodes instead of an instance of XPathNavigator to execute XPath queries when you need to perform operations on the selected nodes that assume some form of node identity. If you need to further manage the node as an instance of the XmlNode class, then using SelectNodes would simplify your code.

Programming with XPathNavigator

Let's learn a bit more about the programming interface of XPathNavigator. In general, regardless of the application-level API, the sequence of steps necessary to execute XPath queries on an XML data source is roughly the same:

  1. Get a reference to an XPath-enabled document class (for example, an instance of an XPathDocument or XmlDocument class).
  2. Create a navigator object for the specified data store.
  3. Optionally, precompile the XPath expression if you plan to reuse it later.
  4. Call the navigator's Select method to action.

The programming interface of the navigator object is defined in the XPathNavigator abstract class. Although you will mostly use the navigator object to execute XPath queries, it is worth noting that the XPathNavigator class represents a more general-purpose component. The navigator is a generic interface that acts as a cursor-like explorer of any data store that exposes its content as XML. Although functionally similar to an XML reader, the navigator is not as fast and effective for simple reads since it is a tree navigator and specializes in retrieval. If you just need to read an XML document, use an XML reader; if you need to perform queries, use a navigator. Remember, right now the navigator works on a fully memory-mapped data source.

Functionally speaking, the XPathNavigator class is not much different from a pseudo-class that just groups together all the methods needed to navigate the document contents. The big difference is in the fact that XPathNavigator is a distinct component that's completely decoupled from the document class. In other words, XPathNavigator represents an XML view of some store of data that's been mapped to an XML data model.

Figure 4 enumerates the properties available on the XPathNavigator class. Like XML readers and the XMLDocument class, XPathNavigator employs name tables to store repeated strings more efficiently. The set of properties looks like the subset of properties that characterize the current node in the XmlTextReader class.

Figure 4 XPathNavigator Properties (Close Figure 4)

Property | Description
BaseURI | Gets the base URI of the current node
HasAttributes | Indicates whether the current node has any attributes
HasChildren | Indicates whether the current node has any child nodes
IsEmptyElement | Indicates whether the current node is empty (<node />)
LocalName | Name of the current node without the namespace prefix
Name | Gets the fully qualified name of the current node
NamespaceURI | Gets the URI of the namespace associated with the current node
NameTable | Gets the name table associated with the navigator
NodeType | Gets the type of the current node
Prefix | Gets the namespace prefix associated with the current node
Value | Returns a string denoting the value of the current node
XmlLang | Gets the xml:lang scope for the current node

It is worth repeating that the XPathNavigator's Select method returns an iterator object whose current element is referenced back as a navigator (XPathNavigator class). To access and process node information, you can only use the properties of the navigator. The properties in Figure 4 are read-only and, more importantly, don't map to an instance of the XmlNode class. If you need to manipulate the node as an XmlNode object (for example, to apply changes) make sure that you use XmlDocument as the data store class and then cast the current element of the iterator to IHasXmlNode. From the reference type, IHasXmlNode calls the GetNode method, which returns the XmlNode instance of the underlying node. In all other cases, access to nodes is read-only.

The navigator object provides a rich set of methods that I classify in three main groups based on their functionality: select, move, and miscellaneous. The following code snippet selects the descendants of a node. The code to get the ancestors is nearly identical:

Dim doc As XPathDocument = New XPathDocument(fileName)
Dim nav As XPathNavigator = doc.CreateNavigator()
nav.SelectDescendants(nodeName, nsUri, selfIncluded)
SelectDescendants takes the local name of the node and selects the descendants. The nsUri variable indicates the namespace URI of the descendant nodes, if any. The selfIncluded Boolean variable is a flag that indicates whether or not the context node should be included in the node set.

Figure 5 contains the list of XPathNavigator's move methods. You can jump in either direction—forward or backward, from sibling to sibling, and according to namespace restrictions. As you may have noticed, there are three different groups of move methods: for element, attributes, and namespace nodes. Only the MoveTo and MoveToRoot methods can be called on any node, irrespective of the type. In addition, attributes and namespaces also have methods to return their value: GetAttribute and GetNamespace. When selected, the navigator's Name property returns the namespace prefix. The Value property returns the URI.

Figure 5 XPathNavigator Move Methods (Close Figure 5)

Method | Description
MoveTo | Moves to the same position as the specified XPathNavigator object
MoveToAttribute | Moves to the specified attribute of the current node
MoveToFirst | Moves to the first sibling of the current node
MoveToFirstAttribute | Moves to the first attribute of the current node
MoveToFirstChild | Moves to the current node's first child
MoveToFirstNamespace | Moves to the first namespace in the current element node
MoveToId | Moves to the node with an attribute of type ID whose value matches the given string
MoveToNamespace | Moves to the namespace node with the specified prefix in the current element node; any namespace node is seen as an attribute node with the xmlns name; the real name of the namespace node is the prefix
MoveToNext | Moves to the next sibling of the current node
MoveToNextAttribute | Moves to the next attribute of the current node
MoveToNextNamespace | Moves to the next namespace in the current element node
MoveToParent | Moves to the parent of the current node
MoveToPrevious | Moves to the previous sibling of the current node
MoveToRoot | Moves to the root node of the document

Figure 6 groups all the other methods defined on the XPathNavigator class. A few of these methods have to do with XPath expressions. An XPath expression is a string that represents a location path, though it is a bit more than a plain command string. It has a surrounding context that the XPathExpression class encapsulates. The context of an expression includes the return type and the namespace information. The XPathExpression class is not publicly creatable. To get a new instance of it, you must take an XPath string expression and compile it into an XPathExpression object. The following code snippet shows how to compile an expression and display its expected return type:

Dim expr As XPathExpression = nav.Compile(xpathExpr)
Console.WriteLine(expr.ReturnType.ToString())
nav.Select(expr)

Figure 6 More XPathNavigator Methods (Close Figure 6)

Method | Description
Clone | Clones the navigator and returns a new object with the same current node
ComparePosition | Compares the position of the current navigator with the position of the specified XPathNavigator
Compile | Compiles an XPath expression
Evaluate | Evaluates the given XPath expression and returns the result
GetAttribute | Gets the value of the specified attribute, if such an attribute exists on the current node
GetNamespace | Gets the URI of the specified namespace prefix, if such a namespace exists on the current node
IsDescendant | Indicates whether the specified navigator is a descendant of the current one; a navigator is a descendant of another navigator if it is positioned on a descendant node
IsSamePosition | Indicates whether the current navigator is at the same position as the specified navigator
Matches | Determines whether the current node matches the specified XPath expression

A compiled XPath expression can be consumed by the Select, Evaluate, and Matches methods. The term "compile" here does not mean that the XPath expression becomes an executable. More simply, the action of compiling must be seen as the process that produces an object by collecting various pieces of information. An expression can return values of types other than a node-set. In this case, you use the Evaluate method to evaluate the expression and then cast the generic object returned to the specific type. Select is a more specific method in that it assumes that the return type is a node-set and inserts the nodes in an iterator.

Sorting the Node-Set

An interesting extension built into the XPathExpression class is the ability to sort the node-set before it gets passed back to the caller. To add a sorting algorithm, you call the AddSort method on the XPathExpression object. AddSort is available in two flavors:

Sub AddSort(expr As Object, comparer As IComparer)

Sub AddSort(expr As Object, order As XmlSortOrder, _
    caseOrder As XmlCaseOrder, lang As String, _
    dataType As XmlDataType)
The expr argument denotes the sort key. It can be a string representing a node name or another XPathExpression object that evaluates to a node name. In the first overload, the comparer argument refers to an instance of a class that implements the IComparer interface. The interface supplies a Compare method, which is actually used for comparing a pair of values. Use this overload if you need to specify a custom algorithm to sort nodes.

The second overload always performs a numeric or text comparison according to the value of the dataType argument. In addition, you can specify a sorting order (ascending or descending) and even the sort order for uppercase and lowercase letters (or you can ignore capitalization completely by using the value XmlCaseOrder.None). Finally, the lang argument specifies which language to use for comparison. The language name should also specify the locale. For example, to indicate U.S. English, it is preferable to use "us-en," rather than simply "en."

Let's delve a bit deeper into the IComparer interface. To sort arrays of objects, the .NET Framework provides a few predefined comparer classes including Comparer and CaseInsensitiveComparer. The comparer class compares objects (mostly strings) with respect to case. CaseInsensitiveComparer does the same but ignores case. To use both classes in your code, be sure to import the System.Collections namespace. The Comparer class has no public constructor but provides a singleton instance through the Default static property. For example:

expr.AddSort("lastname", Comparer.Default);

If needed, you can also create your own comparer class. This code shows a rather trivial Visual Basic® .NET implementation:

Class MyOwnStringComparer
Implements IComparer
   Public Function Compare(x As Object, y As Object) _
          As Integer Implements IComparer.Compare
       Dim strX As String = CType(x, String)
       Dim strY As String = CType(y, String)
       Return String.Compare(strX, strY)
   End Sub
End Class

The Compare method should return 0 if the two strings are equal; it returns a value greater than 0 if x precedes y, and a negative value if y precedes x.

The class can also be defined within the body of your application and does not require a separate assembly. This code associates a custom comparer with an XPath expression:

Dim comp As MyOwnStringComparer = New MyOwnStringComparer()
expr.AddSort("lastname", comp)
Figure 7 shows an application of this technique. After creating a navigator for an XML doc, the sample console app compiles the expression to use. The assumed data source has this schema:
<MyDataSet>
<NorthwindEmployees>
<Employee>
   <employeeid>...</employeeid>
   <lastname>...</lastname>
   <firstname>...</firstname>
   <title>...</title>
</Employee>
•••
</NorthwindEmployees>
</MyDataSet>
Sorting by a single node is easy—you just pass the node name to the AddSort method. It's more complicated to sort by multiple fields. The idea is that you indicate a comma-separated list of node names. However, the argument string must be an XPath expression that evaluates to the comma-separated list of node names. If you simply specify the sorting key as, say, "title, lastname" you'll get a runtime error because the XPath processor mistakes it for the actual node name. What's really needed is an expression that the XPath processor would translate at run time into the desired title and lastname, like this:
Dim sortKey As String = "concat(concat(title, ','), lastname)"
The concat keyword identifies one of the predefined helper functions that XPath implementations provide.

Figure 7 Sort and Compare (Close Figure 7)
Imports System
Imports System.Xml
Imports System.Xml.XPath
Imports System.Collections
Imports Microsoft.VisualBasic

Class SortNodeSetApp
    Public Shared Sub Main(args As String())
        Try
            ' Must sort?
            Dim mustSort As String = ""
            If args.Length > 0 Then
                mustSort = args(0)
            End If

            ' Get the XML file to work on    (assumes a certain schema)
            Dim fileName As String = "data.xml"

            ' Create the XPath navigator
            Dim doc As XPathDocument = New XPathDocument(fileName)
            Dim nav As XPathNavigator = doc.CreateNavigator()

            ' Compile a standard XPath expression
            Dim expr As XPathExpression
            expr = nav.Compile("/MyDataSet/NorthwindEmployees/Employee")

            ' Set the sort key (title + lastname)
            Dim sortKey As String = "concat(concat(title, ','), lastname)"

            ' Add sort information (use the default comparer)
            If mustSort = "/sort" Then
                expr.AddSort(sortKey, Comparer.Default)

                ' As an alternative, use custom comparer
                ' Dim comp As MyOwnStringComparer =
                ' New MyOwnStringComparer()
                ' expr.AddSort(sortKey, comp)

                ' Also use the other AddSort overload expr.AddSort(sortKey,
                ' XmlSortOrder.Ascending, XmlCaseOrder.None, "",
                ' XmlDataType.Text)
                    End If

            ' Perform the query
            Dim iterator As XPathNodeIterator = nav.Select(expr)

            ' Output the results
            Console.WriteLine("")

            ' Iterate on the node set
            While iterator.MoveNext()
                ' The iterator scrolls the nodes in the node set.
                ' The navigator lets you drill down into each node's
                ' subtree. Must clone the navigator to avoid misplacing the
                ' main navigator
                    Dim nav2 As XPathNavigator = iterator.Current.Clone()

                ' The output has the form:
                ' <Employee>
                '    <employeeid>...</employeeid>
                '    <lastname>...</lastname>
                '    <firstname>...</firstname>
                '    <title>...</title>
                ' </Employee>
                ' :

                Dim buf As String = ""

                ' Select the <employeeid> node and read the current value
                    nav2.MoveToFirstChild()
                buf += nav2.Value + ". "

                ' Select the <lastname> node and read the current value
                    nav2.MoveToNext()
                buf += nav2.Value

                ' Select the <firstname> node and read the current value
                nav2.MoveToNext()
                buf += ", " + nav2.Value

                ' Select the <title> node and read the current value
                nav2.MoveToNext()
                buf += "   " + Chr(9) + "[" + nav2.Value + "]"

                ' Write out the results
                Console.WriteLine(buf)
            End While
        Catch e As Exception
            Console.WriteLine("Error:" + vbTab + "{0}" + vbCrLf, e.Message)
        End Try

        Return
    End Sub

    ' Provides a custom comparer
    Class MyOwnStringComparer
        Implements IComparer
        Public Function Compare(x As Object, y As Object) As Integer
            Implements IComparer.Compare
            Dim strX As String = CType(x, String)
            Dim strY As String = CType(y, String)

            Return String.Compare(strX, sTry)
        End Function
    End Class
End Class

In Figure 7, you also see how to use iterators effectively. A key point to notice when working with XPath iterators is that your code passes over all the nodes in the node-set one at a time.

You'll probably need to drill down into the subtree of each of these nodes. To do so, first clone the navigator to avoid misplacing the main navigator, thus compromising the outermost loop:

Dim iterator As XPathNodeIterator = nav.Select(expr)
While iterator.MoveNext()
Dim nav2 As XPathNavigator = iterator.Current.Clone()
nav2.MoveToFirstChild()
•••
nav2.MoveToNext()
•••
End While
The Current property on the iterator returns the current node in the node-set. It evaluates to an instance of the XPathNavigator class and can be cloned using the Clone method.

A Whistle-stop Tour of XSL in .NET

An XML transformation is a user-defined algorithm that attempts to express the semantics of a given document in another (equivalent) syntax. The transformation process consists of the rendering of the source document based on the structure of the stylesheet. The stylesheet is a declarative and user-defined document that contains the set of rules that will be used to transform one document into another document. XSL refers to a meta-language designed for expressing stylesheets for XML documents. XSL files were originally conceived as the XML counterpart of HTML cascading stylesheets (CSS). In light of this, XSL was designed as an extensible and user-definable tool to render an XML document in HTML for display purposes. The growing complexity of stylesheets, as well as the advent of XML schemas, led to XSLT. Today, XSL is just a blanket term for a number of derived technologies that altogether better qualify and implement the original idea of styling XML documents. The various components that fall under the umbrella of XSL are the actual software entities that you use in your code: XSLT, XPath, and XSL Formatting Objects (XSLFO).

An XSLT program is a generic set of transformation rules whose output can be any text-based language, including HTML, RTF, and so on. As mentioned, XPath is a query language that XSLT programs employ to select specific parts of the source XML document. The result of XPath expressions is then parsed and elaborated by the XSLT processor. Normally, the XSLT processor works sequentially on the source document, but hands it off to XPath if access to a particular group of nodes is requested.

In the .NET Framework, the core class for XSLT is XslTransform. Located in the System.Xml.Xsl namespace, the class implements the XSLT processor. You make use of the class in two steps: first you load the stylesheet in the processor and then you apply transformations to as many source documents as you need. The XslTransform class only supports the XSLT 1.0 specification. The C# code in Figure 8 implements a command-line XSLT transformer. It takes three command-line arguments—the XML source, the XSLT stylesheet, and the output file—to set up the processor and save the results of the transformation to the output file.

Figure 8 XSLT Transformer (Close Figure 8)
using System;
using System.Xml;
using System.Xml.Xsl;

class QuickXslTransformer
{
    public QuickXslTransformer(string source, string stylesheet, string
                               output)
    {
        XslTransform xslt = new XslTransform();
        xslt.Load(stylesheet);
        xslt.Transform(source, output);
    }

    public static void Main(string[] args)
    {
         try {
         QuickXslTransformer o;
          o = new QuickXslTransformer(args[0], args[1], args[2]);
        }
         catch (Exception e)
        {
         Console.WriteLine("Unable to apply the XSLT transformation.");
         Console.WriteLine("Error:\t{0}", e.Message);
         Console.WriteLine("Exception: {0}", e.GetType().ToString());
            }

        return;
    }
}

The XslTransform Class

The XslTransform class supplies two methods specific to its activity—Load and Transform. The class is guaranteed to operate in a thread-safe way only during transform operations. In other words, although an instance of the class can be shared by multiple threads, only the Transform method can be safely called from multiple threads. The Load method is not thread-safe. The Transform method reads the shared state and can run concurrently from multiple threads. Figure 9 shows a portion of the internal architecture of the class. After loading the stylesheet, the XSLT processor needs to modify its state to reflect the loaded document. The operation does not occur atomically within the virtual boundaries created by a lock statement. As a result, concurrently running threads could, in theory, access the same instance of the XSLT processor and break the data consistency. The load operation is thread-sensitive because it alters the global state of the object.

Figure 9 XSL Transform Class
Figure 9 XSL Transform Class

In version 1.0 of the .NET Framework, the XslTransform class has a link-demand permission set attached. A link demand specifies which permissions direct callers must have in order to run the code. Caller rights are checked during just-in-time compilation:

[PermissionSet(SecurityAction.LinkDemand, Name="FullTrust")]
public sealed class XslTransform
{ ••• }

The permission set attribute for the XslTransform class is expressed by name and points to one of the built-in permission sets—FullTrust. What does this mean to you? Only callers (direct callers are involved with the check, not caller's callers) with fully trusted access to all the local resources can safely call into the XSLT processor. For example, if you call the XSL processor over a network share, a security exception is raised. In version 1.1 of the .NET Framework, the permission set has been removed.

In the overall behavior of the .NET Framework XSLT processor, three stages can be clearly identified: the load of the stylesheet document, the setup of the internal state, and the transformations. The first two stages occur within the context of the Load method. Of course, you cannot call the Transform method before a previous call to Load has successfully terminated. The Load method always works synchronously, so that when it returns you can be sure that the loading step has actually been completed. You will not receive any return value that denotes the failure or success of the operation. However, whenever something goes wrong with the Load method, some exceptions are thrown. In particular, you will get an exception of type FileNotFoundException if you are pointing to a missing stylesheet and a more generic exception of type XsltCompileException if the XSLT script contains some errors. An XsltCompileException exception provides you with the line position and number where the error occurred in the stylesheet.

The input stylesheet can be loaded from four different sources—URL, XML reader, XPathDocument, and XPathNavigator. Regardless of the source, as its first action the Load method expresses the source as an XPathNavigator object. The stylesheet must be compiled and considering the compiler's architecture, a navigator is an extremely efficient object. "Compiling" is a process that simply excerpts some information out of the original stylesheet and stores it in handy data structures for further use. The whole set of these data structures is said to be the state of the XSLT processor. Figure 10 shows the flow of the Load method.

Figure 10 Load Method Flow
Figure 10 Load Method Flow

The stylesheet compiler populates three internal data structures with the data read from the source. The object referenced as the compiled stylesheet object represents a sort of index of the stylesheet content. The other two objects are tables containing compiled versions of the XPath queries to execute and the actions that the various templates require.

The Transform method takes at least two explicit arguments—the source XML document and the output stream—plus a couple of implicit parameters. The compiled stylesheet object is, of course, one of the implicit input arguments. The second implicit parameter is the content of the XslTransform's XmlResolver property, which is designed to resolve external resources. The Transform method can also take a third explicit argument, an object of class XsltArgumentList. The argument contains the namespace-qualified arguments used as input to the transformation process.

The XML source document is normalized as an XPathNavigator and passed in down to the XSLT processor. Interestingly, the Transform method has two types of overloads. Some of the overloads work as void methods and simply write to the specified stream. Others work as functions and specifically return an XML reader object. As I'll discuss in a moment, this feature offers a very interesting opportunity: implementing asynchronous XSLT transformations. In Figure 11, you can see the flow of execution for the Transform method.

Figure 11 Transform Method
Figure 11 Transform Method

The Transform method also lets you pass arguments to the stylesheet using an instance of the XsltArgumentList class. When you pass arguments to an XSLT script in this way, you cannot specify which template call will actually use those arguments. You just pass arguments globally to the XSLT processor. The internal modules responsible for processing templates will then read and import those arguments as appropriate:

XsltArgumentList args = new XsltArgumentList();
args.AddParam("MaxNumOfRows", "", 7);

The AddParam method creates a new entry in the argument list. The method requires three parameters: the name of the parameter, the namespace URI (if the name is qualified by a namespace prefix), and an object representing the actual value. Regardless of the CLR type you use in order to pack it into the argument list, the parameter value must correspond to a valid XPath type: string, Boolean, number, node-fragment, and node-set. The number corresponds to a double type, whereas node-fragments and node-sets are equivalent to XPathNavigators and XPath node iterators.

Arguments and Extension Objects

XsltArgumentList is not a collection-based class. It doesn't derive from a collection class or implement any of the typical list interfaces like IList or ICollection. The XsltArgumentList class is built around a couple of hash tables—one to hold XSLT parameters and one to gather extension objects. An extension object is a living instance of a .NET object that you can pass as an argument to the stylesheet. For example, XSLT scripts can be extended in various ways to achieve functionality that the embedded XPath library doesn't supply. The XslTransform class supports the <xsl:eval> instruction, which allows you to insert VBScript into a stylesheet. An alternative for embedding custom code into the script is the <msxsl:script> element. This new instruction supports managed languages and provides access to the entire .NET Framework:

<msxsl:script
  language = "language"
  implements-prefix = "prefix">
•••
</msxsl:script>
Supported languages include C#, Visual Basic, and JScript®. The language attribute is not mandatory and, if not specified, defaults to JScript. The implements-prefix attribute is mandatory, however. It declares a namespace and associates the user-defined code with it. The namespace must be defined somewhere in the stylesheet. In addition, to make use of the <msxsl:script> instruction, the stylesheet must include the following namespace:
xmlns:msxsl=urn:schemas-microsoft-com:xslt

To define a simple script, you declare the extra namespaces in the stylesheet's root. For example:

<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
    xmlns:dino="urn:dino-scripts">
This declaration is necessary to call the <msxsl:script> instruction. The namespace simply groups some user-defined scripts under a single roof. The prefix "dino" is now necessary to qualify any calls to any functions defined in an <msxsl:script> block. In the body of the stylesheet, you call any function defined in an <msxsl:script> block:
<xsl:template match="lastname">
    <TD style="border:1px solid black">
      <xsl:value-of select="dino:PrepareName(., ../firstname)" />
    </TD>
</xsl:template>
If you enclose parameters in quotation marks, they will be treated as literal values. To ensure that the managed function receives node values, use the same expressions you would use with the select attribute of an <xsl:value-of> instruction.

The <msxsl:script> instruction is not always the best solution. It supports only a limited set of namespaces which are imported as part of the XSLT processor. For example, you cannot use the System.Data namespace. Script is good for one-off operations such as parsing strings or retrieving the current time. For a more powerful alternative, consider extension objects.

An extension object is simply a managed class that has some public methods. The only requirement is that callable methods accept arguments that are of, or can be coerced to, XPath types. Unlike embedded scripts, which are natively defined in the body of the stylesheet, extension objects are external resources that must be plugged into the stylesheet in some way. Extension object arguments must be passed as shown in the following code:

ExtensionObject o = new ExtensionObject();
// *** set properties on the object if needed
XsltArgumentList args = new XsltArgumentList();
args.AddExtensionObject("urn:dino-objects", o);
XslTransform xslt = new XslTransform();
xslt.Transform(doc, args, writer);

In the XSLT script, you reference methods on the extension object as you would with an embedded script. In the following code snippet, DoSomething is a method on an extension object associated with the namespace prefixed by "dino":

<xsl:template match="lastname">
    <TD style="border:1px solid black">
      <xsl:value-of select="dino:DoSomething(., ../firstname)" />
    </TD>
</xsl:template>
Methods of extension objects are processed as static method calls, meaning that if you have multiple objects with the same method name you are better off using different namespaces.

Using extension objects is preferable to using embedded scripts for at least two reasons. First, extension objects provide much better code encapsulation, not to mention the possibility of class reuse and the fact that any managed type can be used. Second, you end up with more compact, layered stylesheets that benefit from more seamless code maintenance.

Asynchronous XSLT

The Transform method has a couple of overloads that return an XML reader:

XmlReader Transform(XPathNavigator input, XsltArgumentList args);
XmlReader Transform(IXPathNavigable input, XsltArgumentList args);
The signature and the behavior of these overloads are slightly different from the other overloads. First, the input document must be an XPathNavigator or an XPathDocument. More importantly, the methods do not accept any argument representing the output stream. In fact, the output of the transformation process is not written out to a stream, but rather created in a stream and returned to the user through an XML reader.

The overall XSLT process in the .NET Framework works by creating an intermediate data structure (the input navigator) in which the content of the stylesheet is used as the underlying foundation. Any <xsl> tag found in the stylesheet source is replaced with expanded text or any sequence of calls that results from embedded templates. The final output looks like a compiled program in which direct statements are interspersed with calls to subroutines. These statements are called output records, while templates play the role of subroutines. When the Transform method gets an output stream to write to, the XSLT processor loops through all the records and flushes the text into the stream. If an XML reader has been requested, the processor creates an instance of an internal reader class and returns that to the caller. No transformation is performed until the caller explicitly asks to read the cached output records (see Figure 12).

Figure 12 XSL Transformation Process
Figure 12 XSL Transformation Process

When the Transform method returns, the reader is in its initial state, meaning that it is not yet initialized for reading. Each time you pop an element from the reader, a new output record is properly expanded and returned. In this way, you have total control over the transformation process and can implement a number of fancy features. For example, you could provide feedback to the user, discard nodes based on runtime conditions and user roles, or cause the process to happen asynchronously on a secondary thread. For an example of asynchronous transformation, see the code download for this article.

The <asp:xml> Tag

Before I conclude, I want to review a special ASP.NET server control identified by the <asp:xml> tag. The control is the declarative counterpart of the XslTransform class. You use the XML server control to embed XML documents in a Web page. The control is handy when you need XML data islands for the client to consume. Data islands are XML data referenced or included in an HTML page. The XML data can be included inline within the HTML or stored in an external file. By combining this control's ability with a stylesheet that does browser-specific transformations, you can transform server-side XML data into browser-proof HTML. For example, the following code embeds the content of the specified XML file, as transformed by the stylesheet in the page:

<div>
   <asp:xml runat="server" id="xmldata"
      documentsource="data.xml"
      transformsource="ie5.xsl" />
</div>

The XML server control can be configured programmatically and accepts the source XML through a string (DocumentContent property) as well as through an XmlDocument object (Document property). The transformation element can be provided through a URL to an XSL file or through an instance of the XslTransform class (Transform property). If needed, you can also indicate arguments using the TransformArgumentList property. This component, along with the HttpBrowserCapabilities class, provides a great solution for a common need—data-driven Web pages and browser-specific output.

Notice that only well-formed XML data can be used with the <asp:xml> control. More simply, if you need to flush out the content of any file, you can use the Response.WriteFile method.

Conclusion

In this article, I analyzed XPath as the language of choice for performing XML queries in managed applications and discussed several aspects of its implementation. In the .NET Framework, the XPath runtime provides a common infrastructure for other pieces of the puzzle—the first of which is XSLT. I also examined key aspects of the XSLT processor and supplied a few interesting applications of it such as asynchronous processing and the ASP.NET control.

For background information see:
Applied XML Programming for Microsoft .NET by Dino Esposito (Microsoft Press, 2002)
Essential XML Quick Reference: A Programmer's Reference to XML, XPath, XSLT, XML Schema, SOAP, and More by Aaron Skonnard and Martin Gudgin (Addison-Wesley, 2001)
XPathNavigator over Different Stores
MSDN Web Services Developer Center

Dino Esposito is an instructor and consultant based in Rome, Italy. He is the author of Building Web Solutions with ASP.NET and ADO.NET and Applied XML Programming for Microsoft .NET, both from Microsoft Press (2002). Reach Dino at dinoe@wintellect.com.


Editor's Note: Check Out the Tablet PCs

Editor's Note
Check Out the Tablet PCs

Since we founded MSDN Magazine back in 1932, we've seen generation after generation of technological advances, from the early TV test transmissions of Felix the Cat dolls that made up the bulk of our coverage in the 1940s to the special 132-page bumper issue that coincided with the launch of "New" Coke in 1985. Trends have come and trends have gone, but we were truly awed when we recently opened up our first Tablet PC.

If you haven't tried one of these babies, you should take one for a spin at your friendly local hardware retailer. Even if you're not in the market for a laptop right now, it'll be worth it to see where the evolution of the personal computer will be taking us in the next five years. The units we've seen are sleek and stylish, lightweight (3.2 pounds is average), and the concept of ink markup on a screen that can be oriented to work in either portrait or landscape mode adds surprising value to the computing experience.

To demonstrate the unique power of this computing paradigm—yes, it's so amazing that we broke policy and used the word "paradigm" in this Editor's Note—we used the Train Test. "The Train Test?" you ask incredulously. It's simpler than it sounds. You can judge the power of a new technology by using it on the train ride home one evening. Of course, this Train Test should be limited to technologies like new laptop designs. The staff of MSDN Magazine is not responsible for any delays caused on commuter train lines if an overzealous reader decides to subject their homemade HERF device to the Train Test.

So there we were, sitting on the train to Long Beach one day in May, swiveling the Tablet's convertible screen around and pretending to keep to ourselves. Reaction was instant—an eight year-old boy in the next seat exclaimed, "Dad! That guy's laptop is so cool!!" Once we had the lad's attention, we provided a quick 40-minute demo of the ink programming interfaces. We would've given the kid a chance to test the handwriting recognition tools that are built into the new Windows Journal program, too, except our stop was coming up. Better luck next time, Junior!

If you're keeping track, the Train Test is now 2 for 3: Tablet PCs get immediate reactions, and cellular modems get people asking whether they actually work well. (They do, except for dead spots near Massapequa and Valley Stream.) For some reason, however, a 20GB MP3 player got bupkes, even when operated next to a commuter who was struggling with a collection of 20 CDs and an ancient disc player. And yes, even when we accompanied its use with a shrill rendition of "Walking on Sunshine" (granted, it was 7:15 am in the middle of January, so the timing might have been suboptimal).

There are lots of things you can do with a Tablet PC that make us think that this actually is the wave of the future. It runs a full version of Windows XP plus extensions to enable pen recognition. (It's not simply pressure sensitive—the pen's got some weird stuff in it that lets the machine recognize its point while ignoring your hand as it presses on the screen to write.) The machines sample your writing 120 times per second and store the information as Bezier curves, so anything you write will actually look like your writing. There are lots of interesting features built into these machines, and we'll be talking more about them in the months to come. For now, you can check out an overview and download the Tablet PC Platform SDK at the official site, http://www.tabletpc.com. Happy inking!

—J.T.

Active Directory, ActiveX, DirectDraw, Direct3D, DirectX, JScript, Microsoft, Microsoft Press, MSDN, Visual Basic, Visual C#, Visual FoxPro, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Server is a trademark of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Generate Testing Scripts from Real-world Behavior
Installations for Smart Devices
Compression and Communications Components for .NET
Multiple OS Test Platform
Visual FoxPro Macro System
Embed Screen Captures in Windows-based Apps
It's About Time
Pinpoint Hyper-Threading Bugs
The Bookshelf

Generate Testing Scripts from Real-world Behavior

TeaLeaf Technology has announced the availability of TeaLeaf IntegriTea with Session to Script. This new feature allows scripts for testing products, such as the Microsoft® Web Application Stress tool, to be automatically generated directly from the Web sessions of real users. Web applications are constantly updated with new functionality or fixes, yet the scope and thoroughness of validation conducted prior to deployment has decreased to the point of ineffectiveness. This is because up until now you couldn't create and maintain test scripts that validate realistic usage. Applications are deployed to production having passed tests based on expected behavior only to find that real users are working with the application in unique and unintended ways. As a result, production problems are occurring that should have been caught prior to deployment.

Session to Script eliminates this problem by enabling test scripts to be automatically created based on real user behaviors, loads, and inputs, thus allowing potential production issues to be caught before the application is deployed to production.

http://www.tealeaf.com

Installations for Smart Devices

InstallShield Software Corporation has announced smart device support via Service Pack 1 (SP1) for InstallShield Developer 8, adding support for the Microsoft .NET Compact Framework, Visual Studio® .NET 2003 Smart Device Projects, and additional features that enable installations for smart devices. The InstallShield Developer Smart Device supports integration and distribution of the .NET Compact Framework and the ability to target supported platforms. Smart Device support allows developers targeting the .NET Compact Framework to create their setups inside the Visual Studio .NET IDE and automatically update their setups when they rebuild applications in Visual Studio .NET.

http://www.installshield.com

Compression and Communications Components for .NET

/n software inc. has released new editions of IP*Works! and IP*Works! Zip for developers using the Microsoft .NET Compact Framework. These products provide you with more than 35 fully managed components for the .NET Compact Framework, facilitating the integration of Internet communications and file compression in applications targeting the .NET Compact Framework. IP*Works! contains royalty-free components that implement every major Internet protocol.

IP*Works! Zip is a suite of development components for adding compression and decompression functionality to Web and desktop applications using popular file and data compression standards. IP*Works! Zip allows you to integrate compression and decompression into applications using the Zip, Tar, Gzip, or Jar standards for compression. The components also feature password support and streaming compression and decompression.

http://www.nsoftware.com

Multiple OS Test Platform

VMware has released Workstation 4, which lets you run multiple versions of operating systems, including Windows® and Linux, on a single physical system in secure, portable virtual machines. Workstation 4 features improved support for debugging, allowing you to debug and test software within virtual machines. Workstation 4 supports user-level and kernel-level debuggers, including features such as data breakpoints (watchpoints) and I/O breakpoints. In essence, Workstation 4 provides the functionality of native debugging support in the CPU (for example, debug registers, single-step traps, and resume flag).

VMware's snapshot feature allows users to take a snapshot of their virtual machine to capture a running virtual computer's disk and RAM at any moment and save it to disk. Users can revert back to that snapshot at any time, enabling the virtual machine to go back in time to erase coding mistakes.

Also new in this version are Drag and Drop and Shared Folders, allowing users to share files between host and guest virtual machines via a shared folder, or drag and drop file icons between systems. With Tabbing, users can switch between virtual machines.

Users can listen to high-fidelity music with support for SoundBlaster, and stream video via the upgraded high-performance graphics.

Workstation 4 supports a number of operating systems as host and/or guest including Microsoft Windows Server™ 2003 and Red Hat Linux 8.0.

http://www.vmware.com

Visual FoxPro Macro System

FoxMasters recently announced the release of +Macros 2.0, an automation tool for developers who work in Visual FoxPro® 6.0 or 7.0 and use FoxScript procedural macros. Developers can customize Visual FoxPro by creating their own macros using the FoxScript language. FoxScript supports all Visual FoxPro features except for classes and function definitions. FoxScript also provides an additional means to manipulate the Visual FoxPro environment.

+Macros 2.0 includes many standard macros plus additional code management utilities such as auto-completing Visual FoxPro keywords; creating Visual FoxPro classes, methods, and functions from templates; inserting the current date and user name; creating comment and uncomment text blocks of any size; moving blocks of code left or right; changing the case of letters to uppercase or lowercase; and adjusting comments and assignments to make source code more readable.

+Macros supports running any of the user's own macros or standard macros with a keyboard or mouse. You can either use hot keys or select a macro from the +Macros menu, which extends the main menu of Visual FoxPro.

+Macros is a macro management system in which developers can create, modify, and delete macros, assign hot keys for quick access, organize hierarchical macro menus using drag and drop, create macros from templates, and export and import macros.

http://www.FoxMasters.com

Embed Screen Captures in Windows-based Apps

TechSmith Corporation has announced the availability of SnagIt 6.2, with features that simplify screen capturing, allowing users to embed capture capabilities into Web browsers and Windows-based applications. Users now have the option to place a SnagIt add-in on the toolbars of Microsoft Internet Explorer, Windows Explorer, and Microsoft Office applications. These add-ins enable you to capture and manipulate content for Internet broadcasts, Web sites, training materials, and business presentations.

The SnagIt add-ins can be configured with preset input and output options. The SnagIt add-ins for Internet Explorer and Windows Explorer allow users to take screen captures of an entire scrolling window, Web page, region, object, or text that can be edited. The screen captures can be sent to files, clipboards, printers, or e-mail.

In addition, corporate developers can use SnagIt's new COM Server interface to integrate SnagIt's functionality into enterprise applications. Among the most common uses for SnagIt are documentation, employee and customer training, technical support, classroom teaching and distance learning, Web development, publishing, software design, and the creation of sales and marketing materials.

SnagIt 6.2 supports all versions of Windows from Windows 95 through Windows XP.

http://www.techsmith.com

It's About Time

Mabry Software has announced two new controls for the Microsoft .NET Framework: Time/NET and WhoIs/NET. Time/NET is a time retrieval and synchronization client component. The Time/NET control implements SNTP versions 3 and 4, retrieving the current time from Internet public time servers. Time/NET also retrieves error correction data, information about the time server and server stratum, and other data specified and supported by the SNTP protocol.

The WhoIs/NET control implements the WHOIS protocol, enabling the user to query WHOIS servers for information about domains and the people who run them. WhoIs/NET has simplified data retrieval to the point where it takes only one method with one parameter to search a WHOIS database and follow any references to other WHOIS databases. Or, with a few more parameters, you can specify exactly how WhoIs/NET should behave (timeout period, whether or not to refer to alternate databases, the database to search, and what to search for).

Both controls are 100 percent C# managed code and can operate in either synchronous (blocking) or asynchronous (non-blocking) mode.

http://www.mabry.com

Pinpoint Hyper-Threading Bugs

Intel Corporation has announced Intel Thread Checker 1.0, a new tool to help developers locate hard-to-find bugs in applications designed for Intel Pentium 4 and Intel Xeon processor-based systems that include Hyper-Threading Technology.

Intel Thread Checker uses an advanced error detection engine that monitors program execution and pinpoints the location of threading bugs. Traditional tools require a developer to guess where to place traps in source code. Intel Thread Checker eliminates this guesswork and locates the exact lines of code that are causing threading errors.

The tool classifies issues into categories ranging from errors to warnings, and shows variable, source line, and call stack information for easy developer analysis. The tool is available for Intel Pentium 4 and Intel Xeon processor-based systems running Windows XP Professional and Windows 2000.

http://www.intel.com/software/products

The Bookshelf

The Web Programming CD Bookshelf, Version 1.0, published by O'Reilly, combines six books for Web programmers into one CD. Formatted in HTML, it can be read by any Web browser. The CD-ROM contains more than 4,600 searchable and cross-referenced pages, so programmers can search either the individual index for each book or the master index for the entire collection. Included are the complete, unabridged versions of these titles: Dynamic HTML: The Definitive Reference, 2nd Edition; JavaScript: The Definitive Guide, 4th Edition; Programming PHP; Web Database Applications with PHP & MySQL; PHP Cookbook, and Webmaster in a Nutshell, 3rd Edition. As a bonus, the new paperback version of Webmaster in a Nutshell, 3rd Edition is also included.

http://cdbookshelves.oreilly.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: Accessible Images, Image Format Converter, and More

Web Q&A
Accessible Images, Image Format Converter, and More
Edited by Nancy Michell

Code download available at: WebQA0307.exe (127KB)

Q I've heard warnings about the use of graphics on a Web site when you want good accessibility. What should I look out for?

A Sometimes a Web site will rely on a nice looking graphic that suggests the type of navigation it will provide. But for the visually impaired user, there needs to be much more than a simple graphic that you hope the user will click on based on visual cues. Screen readers get the text to read from an IMG tag's alt attribute, so make sure to include it. Some designers are focused on a pixel-perfect font sizing design, which rings accessibility alarm bells. If your site is to be properly accessible to partially sighted users, your fonts must resize and flow when the user selects larger text.

When your site uses a fixed font size, it can be overridden only via the accessibility options—by selecting "ignore font sizes specified in Web pages." The Microsoft Web site, looks great with the browser default text size and still honors the user's settings without resorting to the override.

MSDN® provides some good tips for creating accessible sites at Accessibility Design Guidelines for the Web. Some things to keep in mind are:

  • Provide text links in addition to any image map links.
  • Provide good keyboard navigation by following the rules.
  • Make sure you have alternatives to all controls and applets.
  • If your site uses frames, have frame-free pages as well.
  • Provide titles for your objects.

In addition, the aforementioned Web site offers information on proper keyboard navigation, file formats, and other tips. For more information on accessibility, see http://www.microsoft.com/enable.

Q How can I programmatically convert a BMP file to a JPG or GIF?

A If you use a language that's compatible with the Microsoft® .NET Framework you can do this pretty easily with the System.Drawing.Image class. You'd just have to create an image object from a file, then resave it, passing in the new graphics format.

If your images start out as BMPs, you might want to think about storing them as PNGs. JPEGs are lossy and so the quality may degrade quite badly depending on the contents of the image. GIFs are good for simple graphics containing large fill areas, but are quite limited with regard to colors (they employ a maximum palette of 256 colors). PNGs don't have either of these problems, so for automated graphics conversions, this is your best bet.

The C# code available for download from the link at the top of this article will produce a command-line tool when compiled; this can convert between any of the image formats that the .NET Framework supports.

These lines in the Main function do the conversion:

Image img = null;
img = Image.FromFile(Source);
img.Save(Dest, Format);
Here, Source is a string containing a valid path to an image file, Dest is a string containing a valid path to the output file, and Format is a System.Drawing.Imaging.ImageFormat object.

All the rest of the code available in the download is just for wrapping that up in a command-line tool. It sounds like you were already getting a file of bitmap type, so something like this could be used to convert it to PNG (or another format). The other option would be to simply use the couple of lines of source code you need, assuming your method of grabbing the screen shot is usable from managed code. Both of these techniques require the .NET Framework to be installed.

Q I've heard that ASP.NET pages (.aspx files) and user controls (.ascx files) can't be shared between applications. Is this true?

A Actually, it turns out that there is a fairly simple method for sharing these files by mapping parts of different applications to the same physical directories.

Suppose you have two applications, app1 and app2, and would like to share some user controls between them. In terms of virtual and physical paths, this is what you want to have:

Virtual | Physical
/app1 | c:\app1
/app2 | c:\app2
/app1/controls | c:\SharedControls
/app2/controls | c:\SharedControls
Note how /app1/controls and /app2/controls both map to the same physical path, achieving the desired sharing.

Here are the steps to follow to set this up:

  1. Run the Internet Information Services (IIS) manager (inetmgr).
  2. Create app1 and app2 the same way you always create applications for IIS.
  3. Still in inetmgr, right-click on the newly created app1, and choose New | Virtual Directory.
  4. Enter "controls" for the alias.
  5. Enter "c:\SharedControls" for the directory and complete the wizard.
  6. In inetmgr, right-click on the newly created "controls," and choose Properties.
  7. In the Properties dialog, click "Remove" to the right of the application name. This keeps the virtual mapping, but does not make it an application.
  8. Follow Steps 3 through 7 again, but for app2 instead of app1.

To test it, put a user control test.ascx in c:\SharedControls. Now put some pages in c:\app1 and c:\app2 that use the user control. You'll have something like this:

<%@ register tagprefix="acme" tagname="test" src="controls/test.ascx" %>
<acme:test runat=server/>
Try requesting http://localhost/app1/testpage.aspx and http://localhost/app2/testpage.aspx. They will use the same user control.

While this technique works well and can be very useful, there are some details you should be aware of before deciding to use it. The pages and user controls are shared, but they are compiled separately for each app that uses them. So while you are improving the maintainability of your sites, you are not affecting the memory usage (compared to having several copies of the controls). If the shared pages and user controls use precompiled code-behind assemblies, you will need to copy the assemblies in the bin directory of each app that uses them. As an alternative, you can sign the assemblies and place them in the global assembly cache.

Q I'm generating an XML file that has several data tables in it and I need them to link together. How can I have my XML file point to other parts of my XML file? It looks like XLink, XPointer, or XPath are what I should be using, but it would really help if I had a complete XML example to work with.

A This is a common question. XLink and XPointer don't exist in any Microsoft product, and you can't use XPath alone to query external docs (there is a document function, but it's XSLT). So how do you do it? There are two common ways. With XSLT, you'd have to use the document function. For example, you could create an XML file like the following:

<documents>
   <document>foo.xml</document>
   <document>bar.xml</document>
</documents>
To display these files, write a template similar to this:
<xsl:template match="document">
   <xsl:copy-of select="document(string(.))"/>
</xsl:template>

The other way is to use a document type definition (DTD). For example, if you create an external entity for foo.xml, you could invoke it with a simple &foo.

Which way is better? The first is, but unless you have XSLT experience there will be a bit of a learning curve. Using XSLT allows you to do very rich processing of the incoming markup, which is what you'll want the vast majority of the time. Any application that involves reporting and which essentially distills information is an ideal use of XSLT. The primary benefit of the DTD method is simplicity. With a definition for each file, the number of lines is equal to only the number of files you have, plus however many times you reference them. It's a low-cost solution.

Q Is it possible to query an XML document in a namespace-agnostic way? Let's say an XML document contains various elements from multiple namespaces and I want to query using XPath but ignore the namespace prefix. The idea is to fetch the data back in a way that allows users to issue XPath queries without remembering the namespace prefix they used when the data was persisted.

Modifying my example (see Figure 1), can I issue an XPath query with no namespace:

//bookRecord[book/Author='Asimov'][//Price $ge$ 20]
Is this possible using MSXML or System.XML? Do I need to implement a custom solution that will have overhead of extra processing and/or persistence of extra namespace prefix-related metadata?

Figure 1 Sample XML (Close Figure 1)
<Books xmlns="http://tempuri.org/schema1"
               xmlns:A1="http://tempuri.org/schema2"
               xmlns:A2="http://tempuri.org/schema3">
  <bookRecord>
  <book>
    <Author>Asimov</Author>
    <Title>NightFall</Title>
  </book>
  <A1:bookCost>
    <Author>Asimov</Author>
    <Title>NightFall</Title>
    <Price>20</Price>
  </A1:bookCost>
  <A2:bookPublisher>
    <Author>Asimov</Author>
    <Title>NightFall</Title>
    <Publisher>NightFall</Publisher>
  </A2:bookPublisher>
  </bookRecord>
<Books>

A XPath allows you to ignore namespaces, but you need to jump though some hoops to get there. You must use the local-name function and perform a string compare on the name. The performance of this operation is not that great. Here's the original query:

//bookRecord[book/Author='Asimov'][//Price $ge$ 20]
The namespace-agnostic query looks like this:
//*[local-name()='bookRecord'][*[local-name()='book']/*[local-
name()='Author']/text()='Asimov'][//*[local-name()='Price'] $ge$ 20]

Namespaces are an integral part of XML. They are not meant to be ignored. In many ways, being completely namespace agnostic is like storing a database of people's height in whole feet as integers. If you have tight control over your input, then you can normalize to a consistent unit of measure, but if you are not careful, the data becomes useless because the unit of measure was ignored. For instance, 2 is greater than 1, but 2 inches is not greater than 1 foot. XML query/transformation standards purposely make writing namespace-agnostic queries possible, but difficult. While there are cases where it may be necessary, it almost always indicates a flawed design, and it just doesn't scale.

Consider the price field. You already need to have tight control over its unit of measure. Why not also control the namespace? Is the Author field supposed to read 'Asimov', 'Isaac Asimov', or 'Asimov, Isaac'? Is <Author> always a subelement of <book>, or might it be <author>, or might it be a subelement of <bookRecord>? You already have an implied schema contract. Why not just extend it to include XML namespaces? Basically, that is what namespaces were designed for. Customers can add whatever other information they want in there, but the standardized information conforms to a schema with a standardized namespace to indicate that.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Vikas Ahuja, Derek Denny-Brown, Chris Dreher, David Ebbo, Dion Houston, Michael Kaplan, Fatih Kihtir, Serguei Kouzmine, Richie Meyer, Leonid Smetanin, and Mark Wilson-Thomas.


Data Points: Managing Hierarchical Inserts in ASP.NET and ADO.NET

Data Points
Managing Hierarchical Inserts in ASP.NET and ADO.NET
John Papa

Code download available at: DataPoints0307.exe (145KB)
Contents
Sample Application
DataGrid Setup
AutoIncrement Columns
Retrieving AutoIDs and Cascading
Reintegrating Deltas
Wrap-up

Many online applications require an interface that allows the user to enter multiple data rows at once and then send them to the database in a single shot. For example, a clerk may need to enter several orders, each with multiple line items, into their purchasing system quickly while a customer is on the phone. The application should not save the orders and their line items to the database until they've all been entered and verified by repeating back the orders to the customer. Such an application requires both orders and line items to be added and persisted in some type of local cache until the user decides to save all of the new data to the database. Some common issues that must be dealt with include determining how to persist the data prior to committing it to the database, keeping up with record changes, and dealing with columns that have auto-incrementing values.

In this installment of Data Points, I'll demonstrate how to manage hierarchical inserts using the SQL Server™ Northwind database's Orders and Order Details tables. I'll walk through an application that lets the user add several Order and Order Detail records to a relational ADO.NET DataSet with DataTable objects representing both the Orders and Order Details in the database. Along the way, I will provide sample code techniques that store changes locally, marshal only changed values to a business services object, retrieve auto-incrementing column values, and initiate atomic transactions. Then I will proceed to explain how to reintegrate the new data back into the DataSet that is bound to the UI's DataGrid, so the user sees the latest changes that were entered, including the new auto-increment values.

This column will also demonstrate the tools and techniques that ASP.NET and ADO.NET use with the sample application. I will start by explaining how the two DataGrids are set up and how the user interface interacts with its event handlers and validation rules. Next, I will discuss how the order and order detail data are retrieved from the business objects and related to each other, and how vital constraints are implemented on the data structure. After adding rows to the persisted DataSets and submitting them, I will demonstrate how to save the data within a transaction and retrieve new key values. Finally, I will show how to render those changes to the user interface layer as well as how to reintegrate those changes with the client-side DataSet.

Sample Application

The sample application that I will be referring to (see Figure 1) loads all orders for a given customer. For simplicity's sake, the application assumes the customer is always AROUT, so all examples will be reviewing the AROUT-related records and details. (Of course, it would be a simple process of replacing the hardcoded customer ID of "AROUT" with a parameter that is passed in from a user-selectable list.)

Figure 1 Sample Order Application
Figure 1 Sample Order Application

Before I jump into the technical aspects of the application, I'll describe its intended behavior and requirements. The sample application requires some basic functionality, including paging and a "View" hyperlink next to each order that drills down into the selected order's detail. The footer of each grid must also allow the user to enter a new order or order detail record and then add the record to the grid. The application should store all pending inserts of both orders and order details so they may be submitted as a group. There should also be a hyperlink to reload the data from the database, thus canceling any pending changes, and another hyperlink to save all pending changes to the database.

DataGrid Setup

So now that you know how the application must behave, let's take a look at some of the key aspects of its implementation. The application uses two ASP:DataGrid server controls to manage the interface of the order and order detail data. The Orders DataGrid, named grdOrder, contains some basic formatting settings to allow for paging and alternating row colors as well as a special footer row to allow the user to add an order to the grid. The columns in grdOrder are broken down into two categories: those that are read-only and those that can be inserted. Read-only columns, such as the column that represents the Customer ID field, are defined with the standard asp:BoundColumn element:

<asp:BoundColumn HeaderText="Customer ID" DataField="CustomerID"
                 readonly="true" visible="true"/>

Columns that can be entered so that a new order can be inserted use the asp:TemplateColumn element. The definition for the column for the OrderDate field in the grdOrder DataGrid, shown in Figure 2, tells the DataGrid to display the order date in an asp:Label element in the grid and to display the order date in an asp:TextBox element in the footer. The footer will be used to enter any new order records from the user. To make sure that a valid date is entered, an ASP.NET validator control is included as well. The txtOrderData_Validator control (asp:CompareValidator) is used by this code to compare the order date value entered by the user with the Date datatype.

Figure 2 Listing Orders for the AROUT Customer (Close Figure 2)
<asp:TemplateColumn HeaderText="Order Date" Visible="true">
    <ItemTemplate>
        <asp:Label runat="server" ID="lblOrderDate"
         text=<%#DataBinder.Eval(Container.DataItem, "OrderDate",
         "{0:d}")%> ></asp:label>
    </ItemTemplate>
    <FooterTemplate>
        <asp:TextBox runat="server" ID="txtOrderDate"
         text=<%#DataBinder.Eval(Container.DataItem, "OrderDate",
         "{0:d}")%> Width="70"></asp:textbox>
<asp:CompareValidator Runat="server" ID="txtOrderData_Validator"
            ControlToValidate="txtOrderDate"
            ErrorMessage="Please enter a valid date MM/DD/YYYY"
            Operator="DataTypeCheck"
            Type="Date"
            Display="Dynamic"
            BackColor="#ff0000"
            ForeColor="#ffffff"></asp:CompareValidator>
    </FooterTemplate>
</asp:templatecolumn>

The application must allow the user to add the order to the DataGrid, so there is an asp:TemplateColumn defined for this. The column, of type asp:Button, has a CommandName of AddOrder, will only be displayed in the footer, and will invoke the handler defined in the grdOrder DataGrid's OnItemCommand attribute. In this case, the handler for the OnItemCommand of the grdOrder DataGrid is called grdOrder_OnItemCommand.

<asp:TemplateColumn HeaderText="" Visible="true">
    <FooterTemplate>
        <asp:Button CommandName="AddOrder" Text="Add" ID="btnAddOrder"
                    Runat="server" />
    </FooterTemplate>
</asp:TemplateColumn>

Another special column in grdOrder is the column that displays the View hyperlink for each order row. This column has a CommandName of ViewOrderDetail and is linked to the same OnItemCommand handler as grdOrder.

<asp:ButtonColumn
    ButtonType="LinkButton"
    HeaderText=""
    CommandName="ViewOrderDetail"
    Text="View"/>
Figure 3 shows the code snippet in the grdOrder_OnItemCommand that executes when the View link is clicked.

Figure 3 Invoking grdOrder_OnItemCommand Handler (Close Figure 3)
public void grdOrder_OnItemCommand(object sender, DataGridCommandEventArgs e)
{
    int nOrderID;
    DataSet oDs = null;
    switch (e.CommandName)
    {
        //===========================================================
        //-- ViewOrderDetail command
        //===========================================================
        case "ViewOrderDetail":
            //-- Get the current OrderID
            nOrderID = Convert.ToInt32(((Label)e.Item.FindControl
               ("lblOrderID")).Text);
            //-- Bind the Order Details DataGrid
            BindOrderDetails(nOrderID);
            //-- Store the current Order ID
            ViewState["nLastOrderID"] = nOrderID;
            break;

The grdOrder_OnItemCommand handler catches all ItemCommand events for the DataGrid. A DataGridCommandEventArgs argument is passed to the handler within the e.CommandName parameter, making it possible to determine what caused the handler to execute. Thus, a simple switch statement makes handling ItemCommand events straightforward. Figure 3 shows that the ViewOrderDetail CommandName is evaluated, the current Order ID is retrieved, and the private method BindOrderDetails is called. BindOrderDetails (shown in Figure 4) retrieves a cached DataSet from the Session and filters it to show only the selected OrderID by creating a DataView object. In this example, the DataView accepts the table, an expression to filter the rows, a sort expression (in this case the rows will be sorted by ProductName), and the state of the data to view. Linking these two DataGrid controls together is important so that only the details for the currently selected order are displayed. The sample application accomplishes this by using the DataView to filter the appropriate rows. In Figure 4, the state of the data that will be used in the view is CurrentRows. This tells the DataView to get the current values (as opposed to the original values) which will be used for display in the grdOrderDetail object.

Figure 4 Displaying Order Details for Selected Order (Close Figure 4)
private void BindOrderDetails(int nOrderID)
{
    //-- Get the DataSet
    DataSet oDs = (DataSet)Session["oDs"];
    //-- Create a sorted and filtered DataView
    DataView oDv = new DataView(oDs.Tables["OrderDetails"], "OrderID = "
        + nOrderID.ToString(), "ProductName",
        DataViewRowState.CurrentRows);
    //-- Bind the DataView to the OrderDetail DataGrid
    grdOrderDetail.DataSource = oDv;
    grdOrderDetail.DataBind();
}

The grdOrderDetail DataGrid works similarly to the grdOrder DataGrid with a few minor exceptions. One difference is that the grdOrderDetail DataGrid uses an asp:DropDownList control to give the user a list from which to add a product to the order. Also, the grdOrderDetail DataGrid is only displayed and bound to order detail information when the user clicks on the View hyperlink in the orders grid. However, the grdOrderDetail DataGrid contains a footer for adding a row to the order details and is tied to a handler to catch its ItemCommand events.

When a new order is entered, the grdOrder_OnItemCommand handler fires and creates the new order within the orders DataTable (shown in Figure 5). The values for the new order are retrieved from the current row via its FindControl method. Then, a new DataRow object is created from the orders' DataTable using the NewRow method. The NewRow method creates a new row by using the schema from the DataTable from whence it was invoked. Therefore, the new DataRow object is an empty orders row containing all of the columns of an order.

Figure 5 Adding an Order to the Orders DataTable (Close Figure 5)
//===========================================================
//-- AddOrder command
//===========================================================
case "AddOrder":
    try
        {
            //-- Get the DataSet out of the Session
            oDs = (DataSet)Session["oDs"];
        }
        catch(Exception ex)
        {
            throw(new Exception("Ugh, no DataSet in the Session", ex));
        }

        //-- Get the values for the new Order row from the footer row's
        //-- controls
        string sCustomerID = lblCustomerID.Text;
        TextBox txtOrderDate =
           (TextBox)e.Item.FindControl("txtOrderDate");
        DateTime dtOrderDate = Convert.ToDateTime(txtOrderDate.Text);
        TextBox txtCity = (TextBox)e.Item.FindControl("txtCity");
        string sCity = txtCity.Text;
        TextBox txtCountry = (TextBox)e.Item.FindControl("txtCountry");
        string sCountry = txtCountry.Text;
        //-- Create an Order table DataRow
        DataRow oRow = oDs.Tables["Orders"].NewRow();
        //-- Set the values for the new row
        oRow["CustomerID"] = sCustomerID;
        oRow["OrderDate"] = dtOrderDate;
        oRow["ShipCity"] = sCity;
        oRow["ShipCountry"] = sCountry;
        //-- Add the new row to the Orders DataTable
        oDs.Tables["Orders"].Rows.Add(oRow);
        //-- Store the DataSet
        Session["oDs"] = oDs;
        nOrderID = (int)oRow["OrderID"];
        //-- Store the current Order ID
        ViewState["nLastOrderID"] = nOrderID;
        oRow = null;
        //-- Bind the DataSet to the Orders DataGrid
        BindOrders();
        //-- Clear the Order Details DataGrid
        grdOrderDetail.DataSource = null;
        grdOrderDetail.DataBind();
        break;

The next step is to fill the new order row and add it into the orders DataTable. At first you'll probably think that when you create the NewRow method of the DataTable that it is also adding the row to the DataTable, but it doesn't. The NewRow method creates a detached row, so it can be filled properly and then added to the DataTable by passing it to the Rows collection's Add method. By detaching the row, you avoid errors that might have occurred if an empty row was added to the DataTable. (For example, errors that might occur if the DataTable had constraints on its DataColumn objects that made them not allow null values.)

AutoIncrement Columns

When an order row is added to the grdOrder DataGrid (shown in Figure 5), an Order ID is generated automatically. Since the DataTable is disconnected, it has to be told that the OrderID DataColumn needs to automatically increment its integer value when a new row is created. The following code snippet from the GetData method of the Orders class in the business service layer of the app assigns the primary key constraint to the OrderID DataColumn and tells it that it is an automatically incrementing field:

oDs.Tables["Orders"].Constraints.Add("PK_Orders",
    oDs.Tables["Orders"].Columns["OrderID"], true);
oDs.Tables["Orders"].Columns["OrderID"].AutoIncrement = true;
oDs.Tables["Orders"].Columns["OrderID"].AutoIncrementSeed = -1;
oDs.Tables["Orders"].Columns["OrderID"].AutoIncrementStep = -1;

The AutoIncrementSeed property is set to start at -1 with a step of -1 for each value. Thus, the first new row will get an OrderID of -1, the next will get -2, and so on. Why start at -1 and go south? Because the underlying OrderID field in the database is defined as an IDENTITY column and since there is no lock held on the table by the application, I cannot reliably determine what value the OrderID should receive. For example, let's assume that the last order that was created in the database had an OrderID of 10000 and the OrderID of the new row is set to 10001. The changes are then saved to the database and the OrderID is set to 10005. The DataTable had it set to 10001 and the database ended up creating 10005 because in the short time that the database was checked for the highest OrderID, several new orders were saved and created in the database. The bottom line is that the application does not care what the OrderID's value is as long as while a row is pending it can be used to uniquely identify that row and all of its related records.

Thus, by setting the initial value of the AutoIncrementSeed to -1 and increasing it by -1 (or decreasing by 1), the OrderID values that are generated by the DataColumn for new pending rows will not be the same as any existing row in the database. Because the OrderID DataColumn's values are not saved to the database, the value is irrelevant as long as it is unique and not interfering with existing OrderIDs.

Retrieving AutoIDs and Cascading

The key ingredient in building hierarchical applications is to create related DataTable objects within a DataSet. This sample application relies on the relationship between the orders and their order detail records for displaying the values and for updating the values to the database. To establish this symbiosis, a DataRelation object is created and set up to link the orders DataTable and the order details DataTable. If a relation is created on a field that is already a foreign key to the same parent table, the relation will use the existing foreign key; otherwise, it will create a new foreign key. By setting the UpdateRule of the foreign key to Rule.Cascade, when the OrderID's value in the parent table is modified, the OrderID's value in the child table will reflect the new value. Why would you change the OrderID's value in the orders DataTable? Well, you wouldn't, at least not directly. To explain why this is important, I will continue with the previous scenario of adding the new order row and order details rows and having their OrderID generated and set to -1. Once the pending new rows are saved to the database, the IDENTITY constraint on the orders table's OrderID column will generate a value, which will not be -1. Assuming that the new value was 11110, it would be necessary to get this value and put it back into the DataTable to replace the -1 value and have it reflected in the user interface's DataGrids. Otherwise it won't be possible to insert any more child order detail rows for this parent.

To accomplish this retrieval of the OrderID value into the DataSet's DataTable object, it is important to examine how the data is being saved in the stored procedure (see Figure 6). After the stored procedure prInsert_Order has inserted the new row, it retrieves the new ID value into the output parameter @nNewID. This parameter is then read and mapped to the DataSet by the SqlCommand object automatically, when the UpdatedRowSource property is set to OutputParameters, as shown here:

oInsCmd.Parameters.Add(new SqlParameter("@nNewID", SqlDbType.Int, 0,
    ParameterDirection.Output, false, 0, 0, "OrderID",
    DataRowVersion.Default, null));
oInsCmd.UpdatedRowSource = UpdateRowSource.OutputParameters;

Figure 6 prInsert_Order Stored Procedure (Close Figure 6)
CREATE PROCEDURE prInsert_Order
    @sCustomerID CHAR(5),
    @dtOrderDate DATETIME,
    @sShipCity VARCHAR(15),
    @sShipCountry VARCHAR(15),
    @nNewID INT OUTPUT
AS
    SET NOCOUNT ON

    INSERT INTO Orders (CustomerID, OrderDate, ShipCity, ShipCountry)
    VALUES (@sCustomerID, @dtOrderDate, @sShipCity, @sShipCountry)

    SELECT @nNewID = SCOPE_IDENTITY
GO

The new value for the OrderID, let's say 11110, replaces the -1 value in the orders DataTable's OrderID DataColumn. And because the foreign key was set to cascade on changes (shown in Figure 7), the value 11110 replaces the -1 value in the order details DataTable's OrderID DataColumn as well. The UpdateRowSource property and the output parameter from the stored procedure retrieve the changed value and store it in the root DataTable. In addition, the foreign key constraint's cascading UpdateRule setting makes sure that OrderID's value stays the same throughout the DataSet's appropriate DataRelation.

Figure 7 Creating the Foreign Key and Relation (Close Figure 7)
//============================================================
//-- Create the Foreign Key
//============================================================
ForeignKeyConstraint oFK = new
ForeignKeyConstraint("FK_OrderDetails_Orders_OrderID",
     oDs.Tables["Orders"].Columns["OrderID"],
     oDs.Tables["OrderDetails"].Columns["OrderID"]);
oFK.UpdateRule = Rule.Cascade; //-- The default setting
oDs.Tables["OrderDetails"].Constraints.Add(oFK);

//============================================================
//-- Create the relation between the Orders and Order Details
//============================================================
oDs.Relations.Add("OrdersToOrderDetails",
     oDs.Tables["Orders"].Columns["OrderID"],
     oDs.Tables["OrderDetails"].Columns["OrderID"]);

Another possible way to get the changed OrderID value and put it back into the DataSet is to use the SqlDataAdapter's RowUpdated event handler which fires when a row is updated. Thus it could be used in order to immediately grab the new IDENTITY value and store it in the DataSet.

An example of the handler is shown in Figure 8. It retrieves the new OrderID by hitting the database again and setting the value to the DataTable. While this technique works for most cases, using UpdatedRowSource is simpler and more reliable.

Figure 8 RowUpdated Handler Setting New OrderID (Close Figure 8)
private void m_oDa_Handle_RowUpdated(object sender,
    SqlRowUpdatedEventArgs e)
{
    if ((e.Status == UpdateStatus.Continue) && (e.StatementType ==
        StatementType.Insert))
    {
            e.Row["OrderID"].GetType() = GetIdentity();
            e.Row.AcceptChanges();
    }
}

private int GetIdentity()
{
    SqlCommand oCmd = new SqlCommand("SELECT @@IDENTITY", this.m_oCn,
        this.m_oTrn);
    object x = oCmd.ExecuteScalar();
    return (int)x;
}

Reintegrating Deltas

When the user submits the changes to be saved, it makes sense to send only the pending inserts to the business layer to be saved to the database. After all, why send more data across the network than is necessary? There could be 10 order rows and 100 order detail rows that were not changed, while only 1 order and 1 order detail were added. Thus, instead of sending all of the data down to the business layer, the GetChanges method of the DataSet retrieves and sends only the pending changes. A second DataSet (oDsDelta) is created containing the newly inserted data from the original DataSet's GetChanges method. The delta DataSet is passed to the business layer and is used to update the database.

Once the pending inserts have been saved to the database and the OrderID values have been propagated throughout the DataSet appropriately, the OrderID values in the original DataSet also need to be updated. However, the delta DataSet has been updated with the new OrderID values in the business layer (in the Order class) and the original DataSet is back in the calling Web Form's ASP.NET code. So to reintegrate the changes, the delta DataSet must first be returned from the Order class's SaveData method so that both DataSet objects are at the same level.

//-- Submit modified rows for all DataTables to the database and
//-- retrieve the changes
DataSet oDsDelta = oOrder.AddData("sa", "",
    oDs.GetChanges(System.Data.DataRowState.Added));

The goal is to get the new values back into the original DataSet. One way to accomplish this is to get the OrderID value from the delta DataSet for the new order, find the corresponding row in the original DataSet, and set its OrderID value. If there are several new order rows to synchronize, you can use a loop to set all of the values. However, if there are several fields, not just the OrderID field, and they need to be reintegrated (timestamp fields are one example), another technique can be implemented. The DataSet has a Merge method which does the job quite nicely. The Merge method accepts the delta DataSet as its argument and looks for matching rows in the original and the delta DataSet. When a matching row is found, the differing values in the delta DataSet's rows are overwritten into the original DataSet.

The only catch with using the Merge method in this sample application is that a match is not found since the primary key values differ. Therefore, the delta DataSet's rows would be added to the original DataSet. The original DataSet would end up with two versions of the order as well as duplicates for each of the new order detail rows. To avoid this, you must prevent AcceptChanges from being called on the delta DataSet's rows before calling Merge. By preventing AcceptChanges from being called, the inserted rows will have the old identity value as their original value, and the value from the database as the current value. When the merge is done, the DataSet will match rows based on the original value, and update the rows with the new value.

In order to prevent AcceptChanges from being called, you register a RowUpdated handler on the DataAdapter that sets the row's status to SkipCurrentRow. This prevents further processing (specifically, calling of AcceptChanges) for the newly inserted row, as shown in Figure 9.

Figure 9 Registering a RowUpdated Handler (Close Figure 9)
//-- Register this event handler on the DataAdapter
//   prior to calling Update() as follows:
// oDa.RowUpdated +=
//     new SqlRowUpdatedEventHandler(m_oDa_Handle_RowUpdated);

private void m_oDa_Handle_RowUpdated(object o, SqlRowUpdatedEventArgs e)
{
    //-- See if the change is an insert
    if (e.StatementType == StatementType.Insert)
    {
        //-- Prevent AcceptChanges from being called on this row
        e.Status = UpdateStatus.SkipCurrentRow;
    }
}

Wrap-up

While there is some code that has to be written to achieve the desired functionality using ADO.NET in a Web environment, developing this functionality with classic ADO would be much more daunting. ADO.NET has certainly brought with it a more involved model for developers. However, with this also comes a more in-depth set of features that helps resolve common architectural issues that developers face.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive. You can reach him at mmdata@microsoft.com.


Cutting Edge: Working with Images in the .NET Framework

Cutting Edge
Working with Images in the .NET Framework
Dino Esposito

Code download available at: CuttingEdge0307.exe (343KB)
Contents
The System.Drawing.Imaging Namespace
Working with Images
Supported Formats and Compression
Images and Data Binding
Summing It Up

Image processing has never been easy within Win32®; only the bravest developers ever attempted to play with the Graphical Device Interface (GDI) API. With the Windows® 2000 shell common controls, developers using Win32 gained the ability to display GIF and JPEG images using the ListView control, which supports background images and accepts common-use formats such as BMP, GIF, and JPEG. You can put an empty ListView control in your Win32 dialog, set its background image, and you're finished. The code in Figure 1 demonstrates this.

Figure 1 Using ListView to Display an Image (Close Figure 1)
BOOL DisplayFile(HWND hwndImage, LPCTSTR szFile)
{
    // The window must be a listview
    TCHAR szClass[MAX_PATH];
    GetClassName(hwndImage, szClass, MAX_PATH);
    if (lstrcmpi(szClass, _T("SysListView32")))
        return FALSE;

    LVBKIMAGE lvbi;
    ZeroMemory(&lvbi, sizeof(LVBKIMAGE));
    lvbi.ulFlags = LVBKIF_STYLE_NORMAL|LVBKIF_SOURCE_URL;
    lvbi.pszImage = (LPTSTR)szFile;
    lvbi.cchImageMax = lstrlen(szFile);

    SendMessage(hwndImage, LVM_SETBKIMAGE, 0,
        (LPARAM) (LVBKIMAGE*)&lvbi);

    return TRUE;
}

Overall, developers had to either struggle with images in Win32-based applications or buy third-party libraries. The situation is radically different with the Microsoft® .NET Framework thanks to a number of managed classes that encapsulate the new GDI+ functionality. The GDI+ subsystem is a native component of the Windows XP and Windows Server™ 2003 operating systems that is available to 32-bit and 64-bit Windows-based apps. For earlier versions of Windows, download a backward-compatible version of the GDI+ libraries (gdiplus.dll) from http://www.microsoft.com/msdownload/platformsdk/sdkupdate (look under Core SDK downloads).

As I mentioned, the .NET Framework encapsulates the whole spectrum of GDI+ functionality in a number of managed classes that wrap GDI+ low-level functions. In this column, I'll focus on the imaging API to show how to accomplish a few common tasks. In doing so, I'll discuss some GDI+ infrastructure, and review the internal architecture of the library. Before going on, let me clarify a point about GDI+: the core services of GDI+ are still in unmanaged code. The classes in the .NET Framework are merely wrappers around them. This is something that is expected to change in the upcoming version of Windows, code-named Longhorn.

The System.Drawing.Imaging Namespace

In the .NET Framework, GDI+ functions are defined in the System.Drawing namespace. The namespace includes three child namespaces, which encompass 2D drawing, imaging, and text manipulation. In System.Drawing you find the basic drawing primitives to handle fonts, brushes, pens, and bitmaps. Advanced 2D and vector graphics effects are provided by the classes in System.Drawing.Drawing2D. In particular, you'll find gradients, blending, paths, and regions. Image rendering, color manipulation, and metafile support is available in System.Drawing.Imaging. Finally, the classes in System.Drawing.Text manage an application's privately and publicly installed font collections.

The base class for representing and handling images is Image. It is an abstract class that provides core functionality for three more specific types of images: bitmaps, metafiles, and icons. In the .NET Framework, a bitmap is not necessarily a .bmp file. The Bitmap class encapsulates a GDI+ map of bits, namely the pixel data for a graphic image and all of its attributes. A Bitmap object is an object used to work with images defined by pixel data. Properties of a bitmap include width and height, the color palette, the raw format (GIF, BMP, JPEG, and so forth), the physical size, the pixel format (alpha or the number of bits per pixel). As you can see from even this quick description, the Bitmap object is a general representation of a pixel-based image as opposed to a record-based image like a metafile or icon.

The Bitmap class provides useful methods to rotate or flip the bits, save to a number of formats, and load from a number of data stores. The Bitmap class also features a GetThumbnailImage method. If the Bitmap object contains thumbnail data, then a thumbnail image is created from that data; otherwise, a thumbnail is dynamically generated by scaling the main image.

The MetaFile class defines a graphic metafile—a file made of records that describe a sequence of graphics operations. A metafile is created and displayed by recording and playing back these records. Metafiles are useful for rendering images that can be easily expressed with a set of vector operations such as drawing lines and curves. Obviously, this is not possible in all cases. Photographic images and small icons can't be drawn well with vector graphics. Let's focus more on pixel-based images and the Bitmap class.

A bitmap can be constructed dynamically by allocating a graphic memory context or it can be loaded from existing data. The Bitmap class features a long list of constructors and a few static methods to let you import pixel information from just about anywhere. The Bitmap constructor accepts data from disk files, streams, and other Image-based objects. It also features a couple of static methods to import a Bitmap from Win32 resources and HICON handles. This latter feature guarantees a good deal of interoperability between managed code and Win32 DLLs.

In GDI+, just as in Win32 GDI programming, disposing of graphics objects when you're finished with it is a fundamental rule. It is not as critical as in Win32, where you're responsible for cleaning up the resources you use, but it's still important and highly recommended. In the .NET Framework, the system garbage collector frees any objects that have gone out of scope, but since graphical resources can be expensive in terms of memory, freeing them as soon as possible is always the best practice. With the garbage collector, you will not know the exact moment the cleanup takes place. Thus, to optimize the application's performance consider releasing the resources right away by using the object's Dispose method. All GDI+ graphics objects provide an effective implementation of the Dispose method.

Working with Images

Assuming that you have a PictureBox control in your Windows Forms application, displaying an image from file is as easy as running the following code:

picture.Image = new Bitmap(fileName);
The PictureBox control has two features that make it a really compelling control to use in custom applications. Through the SizeMode property, the PictureBox control allows you to center the image in the client area of the picture box. Furthermore, by assigning a different value to the property you can also stretch the image to fit into the specified area. Valid values for the property are taken from the PictureBoxSizeMode enumeration, whose elements are listed in Figure 2.

Figure 2 PictureBoxSizeMode Properties (Close Figure 2)

Property | Description
AutoSize | The PictureBox control takes the size of the image that it contains. If the form is not large enough and not resizable, the image is clipped.
CenterImage | The image is displayed in the center if the PictureBox is larger than the image. If the image is larger than the PictureBox, the picture is placed in the center of the PictureBox and the outside edges are clipped.
Normal | The image is placed in the upper-left corner of the PictureBox. The image is clipped if it is larger than the PictureBox that contains it.
StretchImage | The image within the PictureBox is stretched or shrunk to fit the size of the PictureBox. The aspect ratio of the image is not maintained.

By default, the Image is placed in the upper-left corner of the PictureBox control, and any part of the image too large for it is clipped. The AutoSize option automatically resizes the control so that it exactly fits the image. If the user resizes the form, the PictureBox control is resized accordingly. You should note that in this case the new size of the picture box can easily overwrite existing controls, causing discrepancies. If you opt for the StretchImage mode, the image will be stretched or shrunk regardless of its aspect ratio. (See Figure 3 for a good and a bad example.)

Figure 3 StretchImage Mode Example
Figure 3 StretchImage Mode Example

If an image is too large for the area you have reserved for it on the form, the simplest way to stretch it while maintaining its aspect ratio is to create a thumbnail:

Image img = picture.Image;

// Reduce each dimension by half
Image newImg = img.GetThumbnailImage(img.Width/2,
               img.Height/2, null, IntPtr.Zero);

// Show the image
pictureBox1.Image = newImg;
Note that using the GetThumbnailImage method to scale an image is not necessarily a good idea, especially for reducing by a small amount. If the image has embedded thumbnails (say from a digital camera) GetThumbnailImage takes it and scales it up to the desired size. As you can imagine, this can introduce a lot of artifacts. In this case, it would be much better to create a bitmap of the desired size and call DrawImage to draw it.

When creating a thumbnail, you can specify the new width and height of the image. It suffices that you indicate values obtained by applying a common scale factor. Ideally, you would specify the factor that ensures that the largest dimension (width or height) will fit in the space. The previous code snippet simply sizes the image 75 percent smaller, shrinking it in half in each dimension.

While GetThumbnailImage is a good way to obtain an image of a given size, you can use the DrawImage method of the Graphics class to render cropped and scaled images. One of the method overloads receives a Bitmap object and a Rectangle object. The rectangle indicates the area in which the image will be rendered. If the size of the destination rectangle is different from the size of the original image, the image is scaled to fit the rectangle.

Another interesting point is that even when an image is bound to a PictureBox control, you can obtain its reference and manipulate it in memory. To apply changes back to the picture box, simply call the Refresh method. Earlier, I mentioned two compelling features of the PictureBox control. One is the SizeMode property; the other is the control's ability to refresh its client area automatically. If you've ever developed graphic interfaces in Win32 using C/C++ and the SDK, this will be familiar to you. Due to the characteristics of the Windows OS, the output of a window needs to be refreshed when two windows overlap or one is partially moved off the screen. A control—and the PictureBox is no exception—incorporates that code and makes it transparently available to callers.

Rotating and flipping images is easy, too. All you have to do is call the RotateFlip method:

picture.Image.RotateFlip(RotateType.Rotate90FlipNone);
picture.Refresh();
The RotateFlip method works on the source image; it doesn't create or return a new one. For this reason, if you're working on an image bound to a PictureBox control, you have to refresh the control to display changes. The RotateFlip enumeration includes elements to control both the angle of the rotation and the direction of the flipping. The previous code rotates the image 90 degrees clockwise and applies no flipping.

GDI+ realizes a complete abstraction over the physical format of the image, making it possible for you to easily code nice features such as writing a copyright note on a displayed image. To modify the bits of the image you must first obtain a Graphics object. The Graphics object is the GDI+ counterpart of the GDI device context. You should think of it as the central control from which to call into a lot of primitives. Everything you draw or fill through a Graphics object acts on a particular canvas. Typical drawing surfaces include the window background (including the control's background), in-memory bitmaps, and printers. You can obtain a Graphics object directly from the canvas.

If you want to make in-memory manipulations, get the Graphics object from the bitmap, like so:

Graphics g = Graphics.FromImage(picture.Image);
g.DrawString(text,
    new Font("verdana", 8),
    new SolidBrush(Color.Black),
    0,0);
g.Dispose();
picture.Refresh();
You must dispose of the Graphics object to make sure you can use the bitmap to refresh. If the Graphics object hasn't been freed, the PictureBox's Image object will still be locked and not refresh properly.

This code snippet opens the bitmap that's bound to the PictureBox control and gets a Graphics object for it. Next, it writes a string of text using the specified font and brush starting at the upper-left corner. Since the modification has been made to the Bitmap object associated with the control, there's no need to rebind. The Refresh method is enough because now the text is part of the image and remains there even if you rotate or stretch the image (see Figure 4).

Figure 4 In-memory Manipulation
Figure 4 In-memory Manipulation

Using a similar approach, you can also create new images from scratch and use them both as in-memory objects or save them to disk. The following code creates a color gradient and replaces the background of the picture box:

int width = pictureBox1.Width;
int height = pictureBox1.Height;
Graphics g;
g = pictureBox1.CreateGraphics();
Rectangle area;
area = new Rectangle(0, 0, width, height);
LinearGradientBrush brInterior;
brInterior = new LinearGradientBrush(area, Color.SkyBlue,
    Color.AntiqueWhite, LinearGradientMode.Horizontal);
g.FillRectangle(brInterior, 0, 0, width, height);
Figure 5 shows the final result. If you move the picture box off the screen, though, the clipped area is not automatically repainted. A better approach that also solves this issue is to bind the dynamically created image to the picture box instead of redrawing:
int width = pictureBox1.Width;
int height = pictureBox1.Height;
Bitmap bmp = new Bitmap(width, height);
Graphics g = Graphics.FromImage(bmp);
Rectangle area = new Rectangle(0, 0, width, height);
LinearGradientBrush brInterior;
brInterior = new LinearGradientBrush(area, Color.SkyBlue,
    Color.AntiqueWhite, LinearGradientMode.Horizontal);
g.FillRectangle(brInterior, 0, 0, width, height);
pictureBox1.Image = bmp;

Figure 5 Replacing the Background
Figure 5 Replacing the Background

You create a new Bitmap object as an empty block of memory whose size equals the size of the PictureBox. The Graphics object is obtained from the bitmap using the FromImage static method. In this way, you declare the newly created Bitmap object as the backing for the image being drawn. When you're done with the graphic work, bind the instance of the Bitmap class to the picture box.

Supported Formats and Compression

Figure 6 lists all the image formats that the .NET Framework supports. BMP is the standard image format used by Windows to store images. Additionally, you can set the number of colors a BMP image can support. The number of bits used to store each pixel determines the total number of possible colors that could be assigned to that pixel and subsequently the color map of the image itself. The number of bits per pixel for BMP files is specified in the file header and can vary from 1 to 64. Commonly used values are 1, 4, 8, and 24. BMP files are usually not compressed and therefore are not well-suited for transfer across the Internet. They might also contain a color table or color palette. A color table represents a small array of colors (usually 16 or 256) being used by the pixels in the image. Pixels are not represented with color information; they simply point to an entry in the table. Each pixel is represented by a 4-bit number or 8-bit number so there are 16 or 256 colors in the palette. Each color in the table is represented by a 24-bit number that is 8 bits for each base color component (red, green, blue).

Figure 6 Image Formats in the .NET Framework (Close Figure 6)

Format | Description
BMP | Bitmap
EMF | Enhanced Windows Metafile
EXIF | Exchangeable Image File
GIF | Graphics Interchange Format
Icon | Windows Icon
JPEG | Joint Photographic Experts Group
MemoryBmp | Memory Bitmap
PNG | W3C Portable Network Graphics
TIFF | Tag Image File Format
WMF | Windows Metafile

The GIF format is a standard on the Web. GIF images are optimized for images that contain lines and curves, blocks of solid color, and sharp boundaries between colors. As you can see, this description is tailored to drawn images commonly used on Web sites. GIF images are compressed to save space, but the compression algorithm is not lossy, meaning that a decompressed image is exactly the same as the original. The GIF format supports a maximum of 256 colors (8 bits per pixel).

GIF images can also have transparent colors—only one color in a file can be designated as such. This means that pixels of that color will be ignored at rendering time. As a result, in those positions the image will have the same background color of any Web page or form that lies directly beneath it. In addition, a sequence of GIF images can be stored in a single file to form an animated GIF.

JPEG is a compression scheme that works well for photographic images. Unlike the GIF format, the JPEG compression algorithm doesn't preserve all the original information. To save more space, a JPEG compressed image discards some raw information. A decompressed image, therefore, is different from the original. However, because JPEG compression is mostly used to store natural scenes, the human eye often doesn't notice any difference.

JPEGs store 24 bits per pixel and are capable of displaying 16 million colors, but do not support transparency or animation. The level of compression in JPEG images is configurable. You should note that higher compression levels result in smaller files because more data is discarded. Subsequently, the smaller the file, the poorer the reproduction. The optimal compression ratio really depends on the image and its use. In general, a 15:1 ratio generates an imperceptible loss of data. This doesn't mean that you can't compress at 30:1 with good results. The contents of the image—the scene and the disposition of the pixels—is an influential factor. Blocks of relatively solid colors normally suffer under high compression ratios, but they are uncommon in real-world images.

Comparing GIF and JPEG images, you should note that they have a rather specular (or mirror-image) set of characteristics; JPEGs perform poorly where GIFs are ideal, and vice versa. GIFs are recommended for text, geometric figures, and images that have large regions of different colors. GIFs maintain sharp differences between close colors while JPEGs attempt to smooth and blur differences. More important than this, though, is the overall resolution of the image. The higher the original resolution is, the more data is available to compress.

The EXIF format is frequently used by digital cameras to store captured images. EXIF is an extension of the JPEG format that adds extra information such as the date of the shot, model of camera, and other user information.

The PNG format has been introduced by the W3C as a better replacement for the GIF format. It's an unlossy compressed format that supports true-color (24 bits per pixel) and grayscale images. In addition, it provides an alpha color component which lets you indicate the level of blending you want between original pixels and the background color. It also improves the GIF's ability to store images progressively so they can be rendered as they arrive over a network connection.

The .NET Framework also supports TIFF images, an extremely flexible format supported by a variety of image-processing applications. TIFF files can store images with an arbitrary number of bits per pixel and using a variety of compression algorithms. Information related to the image (type of compression, orientation, colors, and so on) can be stored in the file and arranged through the use of tags. The TIFF format can be extended as needed by the approval and addition of new tags. At the highest level of abstraction, TIFF images look like XML files in that both support tags and use them to identify pieces of information by name instead of by position. As a result, two TIFF images can have different byte layouts yet can both be correctly processed by the same application.

This code shows how to save a dynamically created image:

Bitmap bmp = new Bitmap(width, height);
Graphics g = Graphics.FromImage(bmp);

// Work on the Graphics object and create the image
CreateTheImage(g);
g.Dispose();

// Save to JPEG
bmp.Save("file.jpg", ImageFormat.Jpeg);
bmp.Dispose();
To convert an existing image to another format, use this code:
Bitmap bmp = new Bitmap(fileGif);
bmp.Save(fileJpeg, ImageFormat.Jpeg);

The Save method is more powerful than the previous code samples show. It can write to a variety of sources, not just disk files. For example, you can create dynamic images (such as charts and banners) over the Web and save them to the ASP.NET Response.OutputStream property, like so:

Response.ContentType = "image/jpeg";
Bitmap bmp = new Bitmap(width, height);
Graphics g = Graphics.FromImage(bmp);
CreateTheImage(g);
g.Dispose();
bmp.Save(Response.OutputStream, ImageFormat.Jpeg);
bmp.Dispose();
A similar approach can be taken using ASP nearly identically.

When saving to JPEG images, you can also control the compression ratio of the algorithm. You must use a different overload of the Save method, as you can see here:

public void Save(
   string filename,
   ImageCodecInfo encoder,
   EncoderParameters encoderParams
);

The first step is to get the ImageCodecInfo structure for the JPEG image. The GDI+ interface provides no direct method to get this object. You must resort to a little trick—enumerate all the image encoders and check their MIME type properties against the JPEG MIME type string (image/jpeg). The ImageCodecInfo structure contains information inherent in the encoding and decoding of the image (see Figure 7).

Figure 7 ImageCodecInfo Structure (Close Figure 7)
mimeType = "image/jpeg";
ImageCodecInfo GetEncoderInfo(string mimeType)
{
   int j;
   ImageCodecInfo[] encoders;
   encoders = ImageCodecInfo.GetImageEncoders();
   for(j = 0; j < encoders.Length; ++j)
   {
      if(encoders[j].MimeType == mimeType)
         return encoders[j];
   }
   return null;
}

The EncoderParameters argument represents an array of encoding parameters. Each element of the array is an EncoderParameter type. Possible parameters are listed as members of the static class Encoder. For example, the parameter Compression lets you choose a compression engine for TIFF images. The Quality parameter lets you choose the desired quality of the JPEG compression. This code compresses a JPEG with a 40:1 ratio:

// Set the quality to 40 (must be a long)
Encoder qualityEncoder = Encoder.Quality;
EncoderParameter ratio = new EncoderParameter(qualityEncoder, 40L);

// Add the quality parameter to the list
codecParams = new EncoderParameters(1);
codecParams.Param[0] = ratio;

// Save to JPG
bmp.Save(fileName, jpegCodecInfo, codecParams);

Images and Data Binding

The .NET Framework provides great support for data binding for both ASP.NET and Windows Forms applications. Binding images to controls in ASP.NET is a bit harder that it is in Windows Forms because of the limitations of the <img> tag in HTML. The tag only accepts a file name or a URL. While using URLs allows you to return images over a stream, it still requires a second round-trip to the browser. Data binding is more direct for Windows Forms applications. To bind a database BLOB field to the Image property of a PictureBox control, start by creating a Binding object, like so:

Binding b = new Binding("Image", dataTable, "photo");
pictureBox1.DataBindings.Add(b);
You must have the BLOB field stored in a DataTable field. In the previous code snippet, that field is named photo. Notice that creating a Binding object isn't sufficient to bind the property to the control. You must first add the Binding object to the DataBindings collection of a particular control. Of course, you must ensure that the control you bind to exposes a public property as in the Binding object. So far, the description of the data binding mechanism is quite general and applies to all controls and data types.

At this point, the .NET Framework will try to bind Image with the contents of the BLOB field. If you run that code, you'll get an exception because the system tries to cast the data type representing the BLOB field—an array of bytes—to the type of the Image property, the Image class. The cast is logically possible but can't happen automatically. You need to inject code in the middle of the process to transform the array of bytes into a valid Image object. Thus, an actual conversion needs to take place inside a handler for the Format event. The Binding class supplies the Format event just to let you manually bind properties and raw data (see Figure 8).

Figure 8 Binding Properties and Data (Close Figure 8)
void Format_Blob(object sender, ConvertEventArgs e)
{
    // Cast the raw data
    Byte[] img = (Byte[]) e.Value;

    // Copy raw data to a memory stream
    MemoryStream ms = new MemoryStream();
    int offset = 0;
    ms.Write(img, offset, img.Length - offset);

    // Create a bitmap from the memory stream
    Bitmap bmp = new Bitmap(ms);
    ms.Close();

    // Bind the bitmap
    e.Value = bmp;
}

The ConvertEventArgs class contains a Value property which is a read/write buffer. It contains the raw data and can be modified to contain the data to bind. The Image property of the PictureBox control can only be bound to an Image object, so you simply need to create an image from the array of bytes. Unfortunately, this can't be done directly because the Bitmap class doesn't have a constructor that accepts arrays of bytes. For this reason, you first create a MemoryStream object, fill it with the bytes of the image, and then create the bitmap from the memory stream. The newly created Bitmap object is then placed in the Value buffer and successfully bound to the Image property of the PictureBox control. The reverse must be done to save an image to a BLOB field in a disconnected DataTable. In this case, use the Parse event instead of Format. The logic and the signature are identical.

Summing It Up

The support for images in the .NET Framework via GDI+ is even greater than I've described here. I haven't even discussed image processing capabilities, color and text manipulation, paths, regions and gradients, metafiles, effects, or alpha blending. When you explore those areas yourself, you'll see that the .NET Framework greatly improves the experience of graphics manipulation.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Building Web Solutions with ASP.NET and ADO.NET, Applied XML Programming for .NET, and Programming ASP.NET all from Microsoft Press, he spends his time teaching classes on ASP.NET and speaking at conferences. Reach Dino at dinoe@wintellect.com.


The XML Files: XML Namespace Collisions, XmlNodeList and Deserialization, and More

XML Files
XML Namespace Collisions, XmlNodeList and Deserialization, and More
Aaron Skonnard

Code download available at: XMLFiles0307.exe (350KB)

Q I am developing a public Web Service that will provide a single interface to all of my organization's internal systems. As a result, I have built several intermediary Web Services that, in turn, get wrapped by the public Web Service. If I use the same class name in any of these Web Services, I get an error when browsing to the WSDL file although it compiles just fine. How do I resolve this?

A The problem you're running into has to do with an XML namespace collision. When working with ASP.NET Web Services, you have to realize that there are two types of namespaces in play. The first type is a namespace in the Microsoft® .NET Framework—a namespace that's used in C#, Visual Basic® .NET, and other .NET-targeted languages for distinguishing among types at compile time. The second type is an XML namespace—a namespace that's used in XML documents to distinguish between XML types, elements, and attributes when processing XML documents.

When you write an ASP.NET Web Service class, you're always using both types of namespaces so it's important to understand the difference between the two and how they map to each other. Consider the following class:

namespace Foo
{
   public class MyService : WebService {
   ... // WebMethods go here
}

The Framework-friendly namespace in this case is called Foo and it contains a single class called MyService, which is going to contain some WebMethod operations. Since I didn't explicitly specify an XML namespace, the infrastructure will automatically use http://tempuri.org as the service's XML namespace. You definitely don't want to use this namespace but rather a unique one of your choosing. Hence, when you browse to the .asmx documentation page, you'll see a warning about this along with a helpful description of how to fix the problem.

The XML namespace of the Web Service class must be explicitly assigned through the [WebService] attribute, as illustrated here:

namespace Foo
{
   [WebService(Namespace="urn:example-org:myservice")]
   public class MyService : WebService {
      •••
   }
}
If you compile the project and browse back to the .asmx documentation page, the warning goes away because it assumes you know what you're doing now. All of the classes you use within this class now will also be associated with this XML namespace, regardless of their namespace in the .NET Framework, unless you explicitly assign them a different XML namespace of their own. For example, consider Figure 1, which contains a few new class definitions used in the MyService class.

Figure 1 MyService Class (Close Figure 1)
namespace Foo
{
   [WebService(Namespace="urn:example-org:myservice")]
   public class MyService : WebService {
      [WebMethod]
      public Bar.MyClass GetBarMyClass() {
         return new Bar.MyClass();
      }
      [WebMethod]
      public Baz.MyClass GetBazMyClass() {
         return new Baz.MyClass();
      }
}

namespace Bar {
    public class MyClass {
       •••
    }
}

namespace Baz {
    public class MyClass {
       •••
    }
}

Since the two MyClass types are used in the MyService class, they're automatically included in the service's XML namespace, in this case urn:example-org:myservice. And since the class names are the same in this case (MyClass), it causes a duplicate in the XML namespace. Since this is illegal in XML Schema, the .asmx infrastructure automatically throws an exception when such situations are detected at run time. However, the code compiles fine, as you noted, since the two MyClass types are in different namespaces in C#.

The poor compiler has no idea about the additional XML Schema and WSDL semantics that cause this problem to surface. The .asmx infrastructure is smart enough, however, to detect these issues when it first performs reflection on the class. For example, if you browse to the .asmx documentation page, you'll see the following error information describing the problem:

Types Bar.MyClass and Baz.MyClass both use the XML type name, MyClass,
from namespace urn:example-org:myservice.
The error message also explains that you need to perform additional Framework-to-XML mappings to resolve the situation:
Use XML attributes to specify a unique XML name
and/or namespace for the type.

In other words, if you really need to use these two different classes that happen to have the same name but mean different things, you need to put them in different XML namespaces. You can accomplish this using the [XmlType] and [XmlRoot] attributes, as illustrated in Figure 2.

Figure 2 XmlType and XmlRoot Attributes (Close Figure 2)
namespace Bar {
    [XmlType(Namespace="urn:bar")]
    [XmlRoot(Namespace="urn:bar")]
    public class MyClass {
       •••
    }
}

namespace Baz {
    [XmlType(Namespace="urn:baz")]
    [XmlRoot(Namespace="urn:baz")]
    public class MyClass {
       •••
    }
}

The [XmlType] attribute controls the name of the XML Schema type name and namespace while the [XmlRoot] attribute controls the name of the root element declaration mapped to each type. The various attribute declarations that I've shown place each MyClass type in a different XML namespace, thus avoiding the name collision that caused the problem. Now if you browse to the .asmx documentation page, everything should work just fine and the documentation will appear.

In cases where two classes really represent the same thing, you should probably eliminate one of them and just have one class that you reference from the different services in your system. In this case, you should still specify the type's XML namespace using the [XmlType] and [XmlRoot] attributes so it's not automatically incorporated into the XML namespace of the service where it's used. Doing so allows you to maintain the identity of the type within XML documents. This topic is discussed in detail at Sharing Types.

Q I'm working on a Web Service and want to return a nodeset from one of its methods. I had originally intended to just return an XmlNodeList, but it doesn't look like it's XML serializable. What's the best way to do this?

A XmlNodeList doesn't contain an Add method, so it's impossible for XmlSerializer to deserialize an instance of it. XmlSerializer, however, can serialize XmlNode objects. It can also serialize arrays of any serializable type. Since XmlNodeList is essentially just an array of XmlNode objects, it's not hard to imagine how you could accomplish this with some code that translates XmlNodeList objects into XmlNode array objects. For example, check out the following WebMethod that returns an array of XmlNodes to the client:

[WebMethod]
public XmlNode[] GetNodes()
{
    XmlDocument doc = new XmlDocument();
    XmlNodeList nl = doc.SelectNodes("//*");
    XmlNode[] nodes = new XmlNode[nl.Count];
    for (int i=0; i<nl.Count; i++)
        nodes[i] = nl[i];
    return nodes;
}

It would be easy enough to generalize this in a class called XmlNodeArray that automatically performs the translation when it's constructed:

public class XmlNodeArray
{
    public XmlNode[] nodes;
    public XmlNodeArray() {}
    public XmlNodeArray(XmlNodeList nl)
    {
        nodes = new XmlNode[nl.Count];
        for (int i=0; i<nl.Count; i++)
            nodes[i] = nl[i];
    }
}
Then you can return any XmlNodeList object by simply wrapping it with a new XmlNodeArray, as illustrated here:
[WebMethod]
public XmlNodeArray GetMoreNodes()
{
    XmlDocument doc = new XmlDocument();
    return new XmlNodeArray(doc.SelectNodes("//*"));
}

Q Is there an XmlReader implementation that sits over an underlying XPathNavigator?

A Unfortunately, System.Xml doesn't include an XmlReader adapter for XPathNavigator. It's fairly easy to write such an adapter, however, by deriving a new class from XmlReader that delegates calls to an underlying XPathNavigator cursor. I wrote one of these as part of a previous installment of The XML Files. The code was in the download for that column, but Figure 3 shows it again.

Figure 3 NavigatorReader Adapter (Close Figure 3)
namespace Developmentor.Xml
{
  using System;
  using System.Xml;
  using System.Xml.XPath;
  using System.IO;
  using System.Collections;

  public class NavigatorReader : XmlReader
  {
    private XPathNavigator cursor;
    ...
    public NavigatorReader(XPathNavigator nav)
    {
      this.cursor = nav;
      this.cursor.MoveToRoot();
    }

    public override XmlNodeType NodeType
    {
      get
      {
        // map the XPathNodeType values to the XmlNodeType values
        switch(cursor.NodeType)
        {
        case XPathNodeType.Root:
          return XmlNodeType.Document;
        case XPathNodeType.Element:
          if (IsEndElement)
            return XmlNodeType.EndElement;
          else
            return XmlNodeType.Element;
        case XPathNodeType.Attribute:
          return XmlNodeType.Attribute;
        case XPathNodeType.Text:
          return XmlNodeType.Text;
        case XPathNodeType.Comment:
          return XmlNodeType.Comment;
        case XPathNodeType.ProcessingInstruction:
          return XmlNodeType.ProcessingInstruction;
        case XPathNodeType.Whitespace:
          return XmlNodeType.Whitespace;
        case XPathNodeType.SignificantWhitespace:
          return XmlNodeType.SignificantWhitespace;
        default:
          return XmlNodeType.None;
        }
      }
    }

    public override string Name
    {
      get
      {
        return cursor.Name;
      }
    }

    public override string LocalName
    {
      get
      {
        return cursor.LocalName;
      }
    }

    public override string NamespaceURI
    {
      get
      {
        return cursor.NamespaceURI;
      }
    }

    ... // remainder ommitted
}

It's also possible to write an adapter that goes the other way, where you want to adapt an XPathNavigator to an underlying XmlReader. This gives you the benefit of being able to perform XPath queries on a forward-only stream. I've provided each of these examples for download from the link at the top of this article.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard is an instructor/researcher at DevelopMentor, where he develops the XML and Web Service-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.


.NET Column: Calling Win32 DLLs in C# with P/Invoke

.NET
Calling Win32 DLLs in C# with P/Invoke
Jason Clark

Code download available at: NET0307.exe (133KB)
Contents
Enter P/Invoke
Style
The DLL Import Attribute
Optional DllImportAttribute Properties
Data Marshaling
Marshaling Numerical and Logical Scalars
Parameters that are Pointers
Marshaling Opaque Pointers: a Special Case
Marshaling Text
Summing it Up

I have noticed a trend in my programming of late, and that trend has inspired the topic of this month's column. Recently, I have done a fair amount of Win32® Interop in my Microsoft® .NET Framework-based apps. I am not saying that my apps are full of custom interop code, but from time to time I bump into a minor, but nagging, inadequacy in the .NET Framework Class Library that can quickly be alleviated by a call into the Windows® API.

As I think about it, any feature limitation in the .NET Framework version 1.0 or 1.1 class library that is not shared by Windows doesn't come as a huge surprise. After all, 32-bit Windows, in all of its incarnations, is a mature operating system that has served a wide breadth of customers for over a decade. The .NET Framework is, in comparison, a newcomer.

As an increasing developer base moves its production applications to managed code, it seems only natural that there will be even more occasions for developers to dip down into the underlying operating system for some critical tidbit of functionality—at least for the time being.

Thankfully, the interop features of the common language run-time (CLR), called Platform Invoke (P/Invoke), are very complete. In this column I am going to focus on the practical use of P/Invoke for calling Windows API functions. P/Invoke is used as a noun when referring to the COM Interop functionality of the CLR and is used as a verb when referring to the use of this feature. I am not going to address COM Interop directly because it is paradoxically both more accessible and more complex than P/Invoke, making it less straightforward as a column topic.

Enter P/Invoke

I'll start by looking at a simple P/Invoke example. Let's see how to call the Win32 MessageBeep function whose unmanaged declaration is shown in the following code:

BOOL MessageBeep(
  UINT uType   // beep type
);

You'll need the following code to add to a class or struct definition in C# in order to call MessageBeep:

[DllImport("User32.dll")]
static extern Boolean MessageBeep(UInt32 beepType);
Surprisingly, this code is all that's required before your managed code can suddenly call the unmanaged MessageBeep API. It's not a method call, but an extern method definition. (Also, it's as close to a straight port from C that C# will allow, so it is a helpful starting point for introducing some concepts.) A possible call from managed code might look like this:
MessageBeep(0);

Now, notice that the MessageBeep method was declared as static. This is a requirement for P/Invoke methods because there is no consistent notion of an instance in the Windows API. Next, notice that the method is marked as extern. This is your hint to the compiler that you mean for the method to be implemented by a function exported from a DLL, and therefore there is no need for you to supply a method body.

Speaking of missing method bodies, did you notice that the MessageBeep declaration doesn't contain a body? Unlike most managed methods whose algorithms are comprised of intermediate language (IL) instructions, P/Invoke methods are nothing more than metadata that the just-in-time (JIT) compiler uses to wire managed code to an unmanaged DLL function at run time. An important piece of information required to perform this wiring to the unmanaged world is the name of the DLL from which the unmanaged method is exported. This information is provided by the DllImport custom attribute that precedes the MessageBeep method declaration. In this case, you can see that the MessageBeep unmanaged API is exported by the User32.dll in Windows.

So far I have exhausted all but two of the lessons to be learned from the call to MessageBeep which, as you'll recall, looked like the code shown in the following snippet:

[DllImport("User32.dll")]
static extern Boolean MessageBeep(UInt32 beepType);
The final two lessons are the related topics of data marshaling and the actual method call from managed code into the unmanaged function. The call into the unmanaged MessageBeep function can be performed by any managed code that finds the extern MessageBeep declaration within scope. The call is made like any other call to a static method. It is this commonality with any other managed method calls that introduces the requirement of data marshaling.

One of the rules of C# is that its call syntax can only access CLR data types such as System.UInt32 and System.Boolean. C# is expressly unaware of C-based data types used in the Windows API such as UINT and BOOL, which are just typedefs of the C language types. So while the Windows API function MessageBeep is documented as the following

BOOL MessageBeep( UINT uType )
the extern method must be defined using CLR types, as you saw in the preceding code snippet. This requirement to use CLR types that are different from, but compatible with, the underlying API function types is one of the more difficult aspects of using P/Invoke. Therefore, I'll devote a whole section to data marshaling a little later in this column.

Style

So making P/Invoke calls to the Windows API is easy in C#. And if the class library refuses to make your application beep, by all means call Windows to do the job, right?

Right. But the approach you choose does matter; and it matters a lot! In general, if the class library offers a way to achieve your goals, it is preferable to use that API rather than making direct calls to unmanaged code because of the significant difference in style between the CLR types and Win32. I can sum up my advice on this matter in a single sentence. When you P/Invoke, don't subject your application logic directly to any extern methods or artifacts thereof. If you follow this little rule, more often than not you'll be spared lots of hassles in the long run.

The code in Figure 1 shows the minimum additional code for the MessageBeep extern method I've been discussing. There isn't any rocket science in Figure 1, just some common-sense improvements to a bare extern method that make life a little easier. Starting from the top, you will notice that an entire type named Sound is devoted to MessageBeep. If I need to add support for playing waves using the Windows API function PlaySound, I could reuse the Sound type. However, I am not offended by a type that exposes a single public static method. This is application code, after all. Notice also that Sound is sealed and defines an empty private constructor. These are just details to keep a user from mistakenly deriving from or creating an instance of Sound.

Figure 1 MessageBeep, Interop Done Well (Close Figure 1)
namespace Wintellect.Interop.Sound{
   using System;
   using System.Runtime.InteropServices;
   using System.ComponentModel;

   sealed class Sound{
      public static void MessageBeep(BeepTypes type){
         if(!MessageBeep((UInt32) type)){
            Int32 err = Marshal.GetLastWin32Error();
            throw new Win32Exception(err);
         }
      }

      [DllImport("User32.dll", SetLastError=true)]
      static extern Boolean MessageBeep(UInt32 beepType);

      private Sound(){}
   }

   enum BeepTypes{
      Simple = -1,
      Ok                = 0x00000000,
      IconHand          = 0x00000010,
      IconQuestion      = 0x00000020,
      IconExclamation   = 0x00000030,
      IconAsterisk      = 0x00000040
   }
}

The next feature of the code in Figure 1 is that the actual extern method where P/Invoke occurs is a private method of Sound. This method is exposed only indirectly by the public MessageBeep method, which takes a parameter of type BeepTypes. This extra level of indirection is a critical detail that provides the following benefits. First, should a future managed method of beeping be introduced in the class library, you can re-tool your public MessageBeep method to use the managed API without having to change the rest of the code in your application.

A second benefit of the wrapper method is this: when you P/Invoke, you waive your right to the protection from access violations and other low-level catastrophes, normally provided by the CLR. A buffer method, even if it does nothing but pass parameters through, allows you to protect the rest of your application from access violations and the like. The buffer method localizes any potential bugs introduced by the P/Invoke call.

The third and final benefit of hiding your private extern methods behind a public wrapper is the opportunity to add some minimum CLR style to the method. For example, in Figure 1 I converted a Boolean failure returned by the Windows API function into a more CLR-like exception. I also defined an enumerated type named BeepTypes whose members correspond to the define values used with the Windows API. Since C# doesn't support defines, managed enumerated types are used to avoid scattering magic numbers throughout your application code.

This final benefit of a wrapper method is admittedly minor for a simple Windows API function like MessageBeep. But as you begin to call into more complex unmanaged functions, you will find a manual translation from the Windows API style to a more CLR-friendly approach increasingly beneficial. The more you plan to reuse your interop functionality throughout your applications, the more design thought you should put into the wrapper. Meanwhile I see no shame in non-object-oriented static wrapper methods with CLR-friendly parameters.

The DLL Import Attribute

OK, it's time to dig a little deeper. The DllImportAttribute type plays an important part in the P/Invoke story for managed code. The DllImportAttribute's primary role is to indicate to the CLR which DLL exports the function that you want to call. The name of the DLL in question is passed as the single constructor parameter to the DllImportAttribute.

If you are unsure which DLL defines the Windows API function that you are trying to use, the Platform SDK documentation provides your best source for help. Near the bottom of the topic text for a Windows API function, the SDK documentation names the .lib file that a C application would link to in order to use the function. In all but a very few cases, the .lib file has the same name as the system DLL file in which the function is defined. For example, if the function requires C applications to link to Kernel32.lib then the function is defined in Kernel32.dll. You can find the Platform SDK documentation topic for MessageBeep at MessageBeep. Toward the bottom of the topic, notice that it says that the library file is User32.lib; this indicates that MessageBeep is exported from User32.dll.

Optional DllImportAttribute Properties

In addition to indicating the host DLL, the DllImportAttribute also includes a handful of optional properties, four of which are particularly interesting: EntryPoint, CharSet, SetLastError, and CallingConvention.

EntryPoint You can set this property to indicate the entry point name of the exported DLL function in cases where you do not want your extern managed method to have the same name as the DLL export. This is particularly useful when you are defining two extern methods that call into the same unmanaged function. Additionally, in Windows you can bind to exported DLL functions by their ordinal values. If you need to do this, an EntryPoint value such as "#1" or "#129" indicates the ordinal value of the unmanaged function in the DLL rather than a function name.

CharSet When it comes to character sets, not all versions of Windows are created equal. The Windows 9x family of products lack significant Unicode support, while the Windows NT and Windows CE flavors use Unicode natively. And sitting on top of these operating systems, the CLR uses Unicode for its internal representation of String and Char data. But never fear—the CLR automatically makes the necessary translations from Unicode to ANSI when calling into Windows 9x API functions.

If your DLL function doesn't deal with text in any way, then you can ignore the CharSet property of the DllImportAttribute. However, when Char or String data is part of the equation, set the CharSet property to CharSet.Auto. This causes the CLR to use the appropriate character set based on the host OS. If you don't explicitly set the CharSet property, then its default is CharSet.Ansi. This default is unfortunate because it negatively affects the performance of text parameter marshaling for interop calls made on Windows 2000, Windows XP, and Windows NT®.

The only time you should explicitly select a CharSet value of CharSet.Ansi or CharSet.Unicode, rather than going with CharSet.Auto, is when you are explicitly naming an exported function that is specific to one or the other of the two flavors of Win32 OS. An example of this is the ReadDirectoryChangesW API function, which exists only in Windows NT-based operating systems and supports Unicode only; in this case you should use CharSet.Unicode explicitly.

Sometimes it is unclear whether a Windows API has a character set affinity. A surefire way to find out is to check the C-language header file for the function in the Platform SDK. (If you are unsure which header file to look in, the Platform SDK documentation lists the header file for each API function.) If you find that the API function is really defined as a macro that maps to a function name ending in A or W, then character set matters for the function that you are trying to call. An example of a Windows API function that you might be surprised to learn has A and W versions is the GetMessage API declared in WinUser.h.

SetLastError Error handling is an important but frequently avoided aspect of programming. And when you are P/Invoking, you are faced with the additional challenge of dealing with the differences between Windows API error handling and exceptions in managed code. I have a few suggestions for you.

If you are using P/Invoke to call into a Windows API function for which you use GetLastError to find extended error information, then you should set the SetLastError property to true in the DllImportAttribute for your extern method. This applies to the majority of extern methods.

This causes the CLR to cache the error set by the API function after each call to the extern method. Then, in your wrapper method, you can retrieve the cached error value by calling the Marshal.GetLastWin32Error method defined on the System.Runtime.InteropServices.Marshal type in the class library. My advice is to check for error values that you expect from the API function and throw a sensible exception for these values. For all other failure cases, including those where failure wasn't expected at all, throw the Win32Exception defined in the System.ComponentModel namespace and pass it the value returned by Marshal.GetLastWin32Error. If you take a look back at the code in Figure 1, you will see that I took this approach in my public wrapper around the extern MessageBeep method.

CallingConvention The last and probably least important of the DllImportAttribute properties that I will cover here is CallingConvention. This property lets you indicate to the CLR which function calling convention it should use for parameters on the stack. The default value of CallingConvention.Winapi is your best bet and will work in most cases. However, if the call isn't working, you might check the declaring header file in the Platform SDK to see if the API function you are calling is one of the odd APIs that bucks the calling convention standard.

In general, the calling convention of a native function, such as a Windows API function or a C-runtime DLL function, describes how the parameters are pushed onto and cleaned off the thread's stack. Most Windows API functions push the last parameter of a function onto the stack first, and then it is the called function's job to clean up the stack. By contrast, many of the C-runtime DLL functions are defined to push the method parameters onto the stack in the order they appear in the method signature, leaving stack cleanup to the caller.

Fortunately you only need a peripheral understanding of calling conventions to get P/Invoke calls to work. In general, starting with the default, CallingConvention.Winapi, is your best bet. And then, with C-runtime DLL functions and a very few functions you might have to change the convention to CallingConvention.Cdecl.

Data Marshaling

Data marshaling is a challenging aspect of P/Invoke. When passing data between the managed and unmanaged worlds, the CLR follows a number of rules that few developers will encounter often enough to memorize. Mastery of the details, though, is normally unnecessary unless you are a class library developer. Application developers who need to interop only occasionally should still understand some fundamentals of data marshaling to get the most out of P/Invoke on the CLR.

For the remainder of this month's column I will discuss data marshaling of simple numeric and string data. I will start with the most basic marshaling of numeric data and work my way up through simple pointer marshaling and string marshaling.

Marshaling Numerical and Logical Scalars

The majority of the Windows OS is written in C. As a result, the data types used with the Windows API are either C types or C types that are relabeled through a type definition or macro definition. Let's look at data marshaling without pointers. To keep things simple, I'll focus first on numbers and Boolean values.

When passing a parameter by value to a Windows API function, you need to know the answers to the following questions:

  • Is the data fundamentally integral or floating-point?
  • If the data is integral, is it signed or unsigned?
  • If the data is integral, how wide is it in bits?
  • If the data is floating-point, is it single or double precision?
Sometimes the answers are obvious, and other times they aren't. The Windows API redefines the fundamental C data types in a variety of ways. Figure 2 lists some common C and Win32 data types, along with their specifications and a common language runtime type with a matching specification.

Figure 2 Non-Pointer Data Types (Close Figure 2)

Win32 Types | Specification | CLR Type
char, INT8, SBYTE, CHARâ€ | 8-bit signed integer | System.SByte
short, short int, INT16, SHORT | 16-bit signed integer | System.Int16
int, long, long int, INT32, LONG32, BOOLâ€ , INT | 32-bit signed integer | System.Int32
__int64, INT64, LONGLONG | 64-bit signed integer | System.Int64
unsigned char, UINT8, UCHARâ€ , BYTE | 8-bit unsigned integer | System.Byte
unsigned short, UINT16, USHORT, WORD, ATOM, WCHARâ€ , __wchar_t | 16-bit unsigned integer | System.UInt16
unsigned, unsigned int, UINT32, ULONG32, DWORD32, ULONG, DWORD, UINT | 32-bit unsigned integer | System.UInt32
unsigned __int64, UINT64, DWORDLONG, ULONGLONG | 64-bit unsigned integer | System.UInt64
float, FLOAT | Single-precision floating point | System.Single
double, long double, DOUBLE | Double-precision floating point | System.Double
â€ In Win32 this type is an integer with a specially assigned meaning; in contrast, the CLR provides a specific type devoted to this meaning.

In general, as long as you select a CLR type whose specification matches that of the Win32 type for the parameter, your code will work. There are some special cases, however. For example, the BOOL type definition in the Windows API is a signed 32-bit integer. However, BOOL is used to indicate a Boolean value of true or false. While you could get away with marshaling a BOOL parameter as a System.Int32 value, you will get a more appropriate mapping if you use the System.Boolean type. Character type-mapping is similar to BOOL in the sense that there is a specific CLR type, System.Char, to address character meaning.

With this information, it might be helpful to step through an example. Sticking with the beep theme, let's try the Kernel32.dll low-level Beep, which makes a beep through the computer's speaker. The Platform SDK documentation for the method can be found at Beep. The native API is documented as follows:

BOOL Beep(
  DWORD dwFreq,      // Frequency
  DWORD dwDuration   // Duration in milliseconds
);

In terms of parameter marshaling, your job is to figure out what CLR data types are compatible with the DWORD and BOOL data types used with the Beep API function. Reviewing the chart in Figure 2, you'll see that DWORD is a 32-bit unsigned integer value, as is the CLR type System.UInt32. This means that you can use UInt32 values for the two parameters to Beep. The BOOL return value is an interesting case because the chart tells us that in Win32, BOOL is a 32-bit signed integer. Therefore, you could use a System.Int32 value for the return value from Beep. However, the CLR also defines the System.Boolean type for the semantic meaning of a Boolean value, so you should use that instead. The CLR will marshal the System.Boolean value as a 32-bit signed integer by default. The extern method definition shown here is the resulting P/Invoke method for Beep:

[DllImport("Kernel32.dll", SetLastError=true)]
static extern Boolean Beep(
   UInt32 frequency, UInt32 duration);

Parameters that are Pointers

Many Windows API functions take a pointer as one or more of their parameters. Pointers increase the complexity of marshaling data because they add a level of indirection. Without pointers, you pass data by value on the thread's stack. With pointers, you pass data by reference, by pushing a memory address to the data onto the thread's stack. The function then accesses the data indirectly through the memory address. There are multiple ways to express this additional level of indirection using managed code.

In C#, if you define a method parameter as ref or out, then the data is passed by reference rather than by value. This is true, even if you are not using Interop, but are just calling from one managed method to another. For example, if you pass a System.Int32 parameter by ref, then you pass the address to the data on the thread's stack rather than the integer value itself. Here is an example of a method defined to receive an integer value by reference:

void FlipInt32(ref Int32 num){
   num = -num;
}
Here, the FlipInt32 method takes the address of an Int32 value, accesses the data, negates it, and assigns the negated value to the original variable. In the following code, the caller's variable x would have its value changed from 10 to -10 by the FlipInt32 method:
Int32 x = 10;
FlipInt32(ref x);

This ability, in managed code, can be reused to pass pointers to unmanaged code. For example, the FileEncryptionStatus API function returns file encryption status as a 32-bit unsigned bitmask. The API is documented as shown here:

BOOL FileEncryptionStatus(
  LPCTSTR lpFileName,  // file name
  LPDWORD lpStatus     // encryption status
);
Notice that the function doesn't return the status using its return value, but instead returns a Boolean value indicating whether the call succeeded. In the success case, the actual status value is returned through the second parameter. The way this works is that the caller passes the function a pointer to a DWORD variable, and the API function populates the pointed-to memory with the status value. The following snippet shows a possible extern method definition to call into the unmanaged FileEncryptionStatus function:
[DllImport("Advapi32.dll", CharSet=CharSet.Auto)]
static extern Boolean FileEncryptionStatus(String filename,
   out UInt32 status);

The definition uses the out keyword to indicate a by-ref parameter for the UInt32 status value. I could have selected the ref keyword here as well, and in fact both result in the same machine code at run time. The out keyword is simply a specialization of a by-ref parameter that indicates to the C# compiler that the data being passed is only being passed out of the called function. In contrast, with the ref keyword the compiler assumes that data may flow both in and out of the called function.

Another cool aspect of out and ref parameters in managed code is that the variable whose address you pass as the by-ref parameter may be a local variable on the thread's stack, an element of a class or structure, or it can be a reference to an element in an array of the appropriate data type. This caller flexibility makes by-ref parameters a good starting point for marshaling pointers to buffers, as well as to single data values. I would only consider marshaling a pointer as a more complex CLR type such as a class or an array object after I found that a ref or an out parameter did not meet my needs.

If you are unfamiliar with C syntax or making calls into the Windows API functions, then sometimes it can be difficult to know if a method parameter requires a pointer. A common indicator is if the parameter type starts with the letter P or the letters LP such as LPDWORD or PINT. In both of these examples the LP and P indicate that the parameter is a pointer, and the data type that they point to would be DWORD or INT, respectively. In some cases, however, the API function is defined as a pointer directly using the asterisk symbol (*) in C-language syntax. The following code snippet shows an example of this:

void TakesAPointer(DWORD* pNum);
As you can see, the preceding function's single parameter is a pointer to a DWORD variable.

When marshaling pointers through P/Invoke, ref and out are only used with value types in managed code. You can tell a parameter is a value type when its CLR type is defined using the struct keyword. Out and ref are used to marshal pointers to these data types because normally a value type variable is the object or data, and you don't have a reference to a value type in managed code. In contrast, the ref and out keywords are not necessary when marshaling reference type objects because the variable already is a reference to the object.

If you are unfamiliar with the difference between reference types and value types you can find more information on the topic in the .NET column in the December 2000 issue of MSDN® Magazine. Most CLR types are reference types; however, all of the primitive types such as System.Int32 and System.Boolean are value types with the exceptions of System.String and System.Object.

Marshaling Opaque Pointers: a Special Case

Sometimes in the Windows API a method takes or returns a pointer that is opaque, which means that the pointer value is technically a pointer but your code doesn't use it directly. Instead, your code passes the pointer back to Windows for subsequent reuse.

A very common example of this is the notion of a handle. In Windows, internal data structures ranging from files to buttons on the screen are represented to application code as handles. A handle is really an opaque pointer or pointer-width data value that your application uses to represent the internal OS construct.

Occasionally, API functions also define opaque pointers to be of the PVOID or LPVOID types. These types in the Windows API definitions mean that the pointer has no type.

When an opaque pointer is returned to (or expected from) your application, you should marshal the parameter or return value as a special type in the CLR called System.IntPtr. When you use the IntPtr type it is not common to use an out or ref parameter because an IntPtr is intended to hold a pointer directly. However, if you are marshaling a pointer to a pointer, then a by-ref parameter to an IntPtr is appropriate.

The System.IntPtr type has a special property in the CLR type system. Unlike the rest of the basic types in the system, IntPtr does not have a fixed size. Instead, its size at run time is based on the natural pointer size of the underlying operating system. This means that in 32-bit Windows IntPtr variables will be 32 bits in width, and in 64-bit Windows the just-in-time compiler emits code that treats IntPtr values as 64-bit values. This automatic resizing is very useful when marshaling opaque pointers between managed and unmanaged code.

Remember, any API function that returns or accepts a handle is really working with an opaque pointer. Your code should marshal handles in Windows as System.IntPtr values.

You can cast IntPtr values to and from 32-bit and 64-bit integer values in managed code. However, since the pointers are supposed to be opaque when used with Windows API functions, you should have no need to do anything with the values but store them and pass them to extern methods. Two exceptions to the store-and-pass-only rule are when you need to pass a null pointer value to an extern method and when you need to compare an IntPtr value with null. To do this, rather than cast zero to System.IntPtr, you should use the Int32.Zero static public field on the IntPtr type to get the null value for comparing or assigning.

Marshaling Text

Working with textual data is common in programming. Text adds some wrinkles to the interop story for two reasons. First, the underlying operating system may use Unicode to represent strings or it may use ANSI. In some rare cases, such as with the MultiByteToWideChar API function, the two parameters to the function disagree on character set.

The second reason that working with text when P/Invoking requires you to have some special understanding is that C and the CLR each deal with text differently. In C, a string is really only an array of char values, typically terminating in a null. Most of the Windows API functions deal with strings on these terms, either as an array of char values in the ANSI case or an array of wchar values in the case of Unicode.

Fortunately, the CLR is designed to be very flexible when marshaling text so that you can get the job done, regardless of what the Windows API function expects from your application. Here are the primary considerations to keep in mind:

  • Is your application passing text data to the API function or does string data pass back from the API function to your application? Or both?
  • What managed type should your extern method use?
  • What unmanaged string format does the API function expect?
Let's address this last concern first. Most of the Windows API functions take LPTSTR or LPCTSTR values. These are modifiable and non-modifiable buffers, respectively (from the function's point of view), which contain null-terminated character arrays. The "C" stands for constant and means that information will not be passing out of the function using that parameter. The "T" in LPTSTR indicates that this parameter can be either Unicode or ANSI depending on the character set you choose and depending on the character set of the underlying operating system. Since most string parameters are one of these two types in the Windows API, the CLR defaults work for you so long as you selected CharSet.Auto on your DllImportAttribute.

However, some API functions or custom DLL functions represent strings in different ways. If you run across one of these functions, you can decorate your extern method's string parameter with the MarshalAsAttribute and indicate a string format other than the default LPTSTR. For more information on the MarshalAsAttribute, see the Platform SDK documentation topic at MarshalAsAttribute Class.

Now let's look at the direction in which string information is being passed between your code and the unmanaged function. There are two ways that you can know which direction the information is being passed in when working with strings. The first and most reliable method is to understand the purpose of the parameter in the first place. For example, if you are calling a parameter with a name like CreateMutex and it takes a string, then you can imagine that the string information passes from your application to the API function. Meanwhile, if you call GetUserName, then the function name suggests that string information passes from the function to your application.

In addition to the rationalization approach, the second way to find out which direction the information flows in is to look for the letter "C" in the API parameter type. For example, the GetUserName API function's first parameter is defined as type LPTSTR, which stands for long-pointer to a Unicode or ANSI string buffer. But CreateMutex's name parameter is typed as LTCTSTR. Notice that here you have the same type definition, but with the addition of the letter "C" to indicate that the buffer is constant and will not be written to by the API function.

Once you have established whether the text parameter is input only or input/output, you can determine which CLR type to use for the parameter type. Here are the rules. If the string parameter is input only, use the System.String type. Strings are immutable in managed code, and are well suited to be used as buffers that will not be changed by the native API function.

If the string parameter can be input and/or output, then use the System.StringBuilder type. The StringBuilder type is a useful class library type that helps you build strings efficiently, and it happens to be great for passing buffers to native functions that the functions fill with string data on your behalf. Once the function call has returned, you need only call ToString on the StringBuilder object to get a String object.

The GetShortPathName API function is great for showing when to use String and when to use StringBuilder because it takes only three parameters: an input string, an output string, and a parameter that indicates the length in characters of the output buffer.

The commented documentation for the unmanaged GetShortPathName function in Figure 3 indicates both an input and output string parameter. This leads to the managed extern method definition, also shown in Figure 3. Notice that the first parameter is marshaled as a System.String because it is an input-only parameter. The second parameter represents an output buffer, and System.StringBuilder is used.

Figure 3 GetShortPathName Declarations (Close Figure 3)
// ** Documentation for Win32 GetShortPathName() API Function
// DWORD GetShortPathName(
//   LPCTSTR lpszLongPath,      // file for which to get short path
//   LPTSTR lpszShortPath,      // short path name (output)
//   DWORD cchBuffer            // size of output buffer
// );

[DllImport("Kernel32", CharSet = CharSet.Auto)]
static extern Int32 GetShortPathName(
   String path,                // input string
   StringBuilder shortPath,    // output string
   Int32 shortPathLength);     // StringBuilder.Capacity

Summing it Up

The P/Invoke features covered in this month's column are sufficient to call many of the API functions in Windows. However, if your interop needs are significant, you will eventually find yourself marshaling complex data structures and perhaps even needing access to memory directly through pointers in managed code. In fact, interop into native code can be a veritable Pandora's box of details and low-level bits. The CLR, C#, and managed C++ offer many features to help; perhaps in a later installment of this column I will cover advanced P/Invoke topics.

Meanwhile, whenever you find that the .NET Framework Class Library won't play a sound for you or perform some other bit of magic on your behalf, you know how to lean on the good old Windows API for some assistance.

Send your questions and comments for Jason to dot-net@microsoft.com.

Jason Clark provides training and consulting for Microsoft and Wintellect (http://www.wintellect.com) and is a former developer on the Windows NT and Windows 2000 Server team. He is the coauthor of Programming Server-side Applications for Microsoft Windows 2000 (Microsoft Press, 2000). You can get in touch with Jason at JClark@Wintellect.com.


Advanced Basics: Passing Data Between Objects in an Application

Advanced Basics
Passing Data Between Objects in an Application
Ken Spencer

Code download available at: AdvancedBasics0307.exe (178KB)

Q What is the best way to pass data between layers in an application? There seems to be a myriad of choices.

A Passing data between forms, classes, Web pages, and Web Services is always a challenge. While it may seem quite simple at first, application demands may place different requirements on data at the endpoints and require certain changes to its format.

How many ways can you move data between parts of an operation? Let me take a stab at a short listing:

  • Structures
  • Classes (with serialization and without)
  • Arrays, collections
  • DataSets, DataTables, DataRows
  • XML
  • Shared variables
  • Public properties exposed from classes

There are other methods, but these are some of the common ones. Let's take a couple of these approaches and look at the pros and cons of each. Before you begin, you should think about what you are going to do with the data when it arrives at its destination. If you are going to use it with a DataSet, DataTable, or DataRow, then an easily transferable format is best. That is, a format that directly translates to the desired format, such as a DataRow.

First, let's take a look at structures and classes since they can have properties, methods, and constructors. There are some differences, however, which you can read more about in Moving to VB.NET: Strategies, Concepts, and Code by Dan Appleman (APress, 2003). Now let's take a look at some practical things I've discovered about structures and classes.

Structures are great because they also behave like lightweight classes since they sit on the stack instead of the heap, as long as you use value types. A simple structure looks like this:

Public Structure SomeData
    Dim Name As String
    Dim Description As String
End Structure
The structure is used like this:
Public MyStuff As SomeData
You can also use and return a structure from a function as shown in the following code snippet:
Public Function DoSomething(ByVal Name As String) As SomeData
    With MyStuff
        .Name = Name
        .Description = "Good stuff"
    End With
    Return MyStuff
End Function
As you can see, the function loads the structure with data, then returns it from the function.

Now, let's call the function from another application, in fact from two different applications:

Dim oMyClass As New SomeComponent.Class1()
Dim MyStuff As SomeComponent.Class1.SomeData

MyStuff = oMyClass.DoSomething(GetUserName)
txtDescription.Text = MyStuff.Description
txtName.Text = MyStuff.Name
oMyClass = Nothing
This code returns the name of the current user and the description baked into the class. Now let's call this method from a Web application. This also works just fine. Until, that is, you put the component in COM+. Then you will get an error message stating that the class is not marked for serialization. You can resolve this by adding the serialization attribute to the class, like so:
<Serializable()> _
    Public Class Class1

That solves that problem. So far so good. I'm building a multitier application for a client and the application uses structures, arrays, classes, datasets, and more at different points. In one case, I combine both structures and databases by creating a structure that contains a dataset and some other pieces of information.

The following structure can hold a DataTable and a string. You can pass this around with the data from SQL Server™ and with other data as well:

Public Structure CustomerStuff
    Dim MoreInfo As String
    Dim dt As DataTable
End Structure

The function that returns this structure is shown in Figure 1. RunSQLWithDataSet is in the downloadable sample at the link at the top of this article.

Figure 1 ReturnSomeData (Close Figure 1)
Function ReturnSomedata() As CustomerStuff
        Dim custstuff As CustomerStuff
        Dim ds As DataSet
        Const ConnectionString = _
            getTrustedConnection()
        ds = RunSQLWithDataSet( _
            "Select * from customers", ConnectionString)
        custstuff.dt = ds.Tables(0)
        custstuff.MoreInfo = "This was really cool"
        Return custstuff
    End Function

Using the new method is just like using the other one:

Dim oMyClass As New SomeComponent.Class1()
Dim MyCustomer As SomeComponent.Class1.CustomerStuff

MyCustomer = oMyClass.ReturnSomedata
txtDescription.Text = MyCustomer.MoreInfo
      DataGrid1.DataSource = MyCustomer.dt
      oMyClass = Nothing
As you can see, the structure provides a nice way to handle multiple types of data in one place. This solves the problem of passing back data returned from a function that has more than one return type. This approach is simpler than trying to create a class that holds the DataSet along with the other data. It's also simpler than trying to extend the DataSet itself to hold the additional data.

Now for the drawbacks associated with structures. I ran into two different problems. First, structures do not support events. If you try to use events in a structure, nothing happens when you run the app. So, classes rule in this regard. Second, I had a problem with structures and Visual Studio® .NET. I created a data library that contains a function similar to the one you just saw that returns a structure. It works fine when I call it from the client, so I thought it would be safe to put the structure in a separate component that could be referenced by the client and the data library. This would allow both the data library and UI to reference the third component. When I tried this approach, Visual Studio .NET kept telling me to add a reference to the project for the assembly containing the structure. The trouble was I had already added this reference! But the code would not compile. So I ended up putting the structure in the data library and letting both the middle tier and the UI reference it. It works, but not quite as cleanly as I had hoped.

Now, let's take a look at using classes to hold data. It's worth mentioning that both classes and structures can be bound to controls to make your life much easier. This makes it very simple to set up a control once and link it to a datasource for good. Since they are both synchronized, you don't have to worry about updating one or the other. For more information on data binding, see Billy Hollis's article, "Not Your Father's Data Binding".

Let's consider a simple example where you need to work with customer data. You could put this data into either a structure or a class. What's the difference? Well, that depends on what you want to do with the data. If you want to wire the data to controls and make your life easy, then classes are a good choice because they can fire events, as shown in Figure 2.

Figure 2 Using Classes (Close Figure 2)
Public Class Customer
    Private privateCustomerID As String
    Private privateCompanyName As String
    Private privateContactName As String

    Public Event CustomerIDChanged As EventHandler
    Public Event CompanyNameChanged As EventHandler
    Public Event ContactNameChanged As EventHandler

    Public Property CustomerID() As String
        Get
            Return privateCustomerID
        End Get
        Set(ByVal Value As String)
            privateCustomerID = Value
            RaiseEvent CustomerIDChanged(Me, New EventArgs())

        End Set
    End Property

    Public Property CompanyName() As String
        Get
            Return privateCompanyName
        End Get
        Set(ByVal Value As String)
            privateCompanyName = Value
            RaiseEvent CompanyNameChanged(Me, New EventArgs())
        End Set
    End Property

    Public Property ContactName() As String
        Get
            Return privateContactName
        End Get
        Set(ByVal Value As String)
            privateContactName = Value

            RaiseEvent ContactNameChanged(Me, New EventArgs())
        End Set
    End Property

    Sub Reset()
        CustomerID = ""
        CompanyName = ""
        ContactName = ""
    End Sub
End Class

There are two important features of this simple class. First, the events that signify (such as CustomerIDChanged) are fired whenever a property is modified. The data binding architecture listens for these events and when one fires, it causes the data binding to resynchronize, reloading the controls. Your code just needs to raise the event when you change a property.

This class shows another important feature. In the application I was building, I needed to reset the class and have this action zero out or reset the bound controls. I tried recreating the instance of the class like this:

Dim CurrentCustomer As New Customer()
That didn't work. I finally hit on an elegant solution: a "Reset" method which can be called any time an app needs to refresh the class. Calling Reset will automatically force all bound controls to refresh to the correct state, clearing the form for a new entry.

Now let's turn to DataSets for a minute. I'm going to call the Update method of the SQLDataAdapter and pass it a typed DataSet. No problem, right? Well, not so fast.

Let's look at a sample. This function returns a DataSet with customer data by making a call to the RunSQLWithDataSet function as shown in the following code:

Function RetrieveCustomerContacts() As DataSet
Dim ds As DataSet
ds = RunSQLWithDataSet("Select CustomerID, " & _
    " CompanyName, ContactName from customers", _
    ConnectionString, "Customers")
        Return ds
End Function

The last parameter to RunSQLWithDataSet specifies the name of the table returned in the DataSet. This is very important, as you'll see in a moment. The client application uses a typed DataSet, which is loaded via the Form_Load event of the Windows® Form with the following code:

Dim oData As New SomeComponent.Class1()
Dim ds As DataSet

ds = oData.RetrieveCustomerContacts
CustomerContactInfo1.Merge(ds.Tables(0))

The tricky part of this code is the last line, which calls the Merge method of the typed DataSet. This takes the DataSet returned from the function and merges it with the Customers table of the typed DataSet. The trick is that the schema and the name of the untyped table must match or the Merge will not work as intended; the DataSet table will be added to the DataSet as a new table, instead of being merged with the Customers table.

Let's assume you have everything named correctly and it all works as expected. As you continue coding, at some point you need to change the typed DataSet. In my example, I'd like to change it by adding a string field to reflect a new field that exists in the database. The only problem is that the field in the database is an integer. If you run this code you'll get a run-time error on the Merge method. But that problem is caught pretty easily.

What happens if you modify the typed DataSet and then try an update when one of the datatypes is incorrect? You may get an error such as "Input string in incorrect format." I got this error and it drove me nuts because the typed DataSet was complex, having 25 elements or so. The error message doesn't tell you where the error occurred. A Google search for the error string was of no use. I finally found the error by printing the typed DataSets schema and comparing it field-by-field with the tables schema. I found a discrepancy where the DataSet element and the corresponding field in the table did not match, I fixed the typed DataSet and I tried it again. This time it worked.

As mentioned earlier, there are many ways to move data around in an application, but each application's needs may be different. It's a good idea to prototype different parts of your code, working out the kinks of transferring data along the way. Then you can devise standards for different methods while adding to your own knowledge base of potential pitfalls.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencer works for 32X Tech (http://www.32X.com), where he provides training, software development, and consulting services on Microsoft technologies.


Design Patterns: Architecture of an Autonomous Application

Design Patterns
Architecture of an Autonomous Application
Sten Sundblad and Per Sundblad

Contents
What Is a Fiefdom?
Secured Gateways
An Architecture Pattern
Emissaries
Validation Against Business Rules
Loose Coupling to Clients
XML-based Web Services
Designing XML Datasets According to Usage
Using Customized Entity Classes
Mapping an Entity Object to the Database
An XML Dataset is Never Connected to the DB
Conclusion

No man is an island. That's even true of developers. Software architects, designers, and developers need to communicate with each other in order to be successful. The same is true with applications. It used to be that an application could be thought of as self-contained. Mainframe applications, and on a smaller scale PC applications, contained everything they needed. They controlled the user interface, the business logic, and the data, all without having to look outside the application or letting outsiders in.

This isn't so anymore. The applications we build today often need to integrate with existing software, and that trend is likely to continue. What's more, increasingly applications are losing control of the user interface, both because it's designed elsewhere and because it's accessing the application over the Internet.

Obviously, when we architect and design applications, we should consider this need for integration across computing platforms. Luckily, there's an architectural solution. Service-Oriented Architecture (SOA) sees an application as a service that's discoverable and accessible over the Internet without regard to the platform the consumer is running on or which programming language was used in order to create the consumer. In other words, a service provides platform- and language-independent services. Web Services is just one example of this.

What Is a Fiefdom?

A service should be a completely autonomous application. It's not an island though because you can ask it to perform services for you. But it is autonomous in the sense that it completely controls its data and it denies the outside world direct access to that data or to its objects. The only way to access its goodies is to send it a message asking it to perform a task for you. If it doesn't like the way you ask or who you are, it will refuse to perform the service. Nobody decides this on its behalf; it decides for itself.

Pat Helland, an architect at Microsoft, was one of the first people to talk about autonomous applications, but he's not the only one anymore. It's a sound concept, as we'll try to show you in this column. Pat uses the term "fiefdom" to describe an autonomous application, and we'll use it here too.

A fiefdom keeps, manages, guards, and protects its main resource—its data. The data of a fiefdom can be kept in different types of data sources, but in most cases structured data is kept in a relational database such as Microsoft® SQL Server™. A fiefdom never allows anybody or anything outside of the fiefdom to gain direct access to its data. The fiefdom might access its data for somebody outside the fiefdom, and even operate on it on their behalf, but direct access is only given to the fiefdom.

A fiefdom doesn't allow an outsider to put a lock on any of its data. Therefore, you must think of all the data that leaves a fiefdom as a snapshot rather than the real thing. For all practical purposes, current data exists only inside the fiefdom. You can't expect the data you receive from it to be current, not even the moment you receive it. Some transaction inside the fiefdom might have changed the data, in its source, even before you receive the snapshot.

Most fiefdoms insist on managing their own transactions. A fiefdom should never allow an outsider to control a transaction or part of a transaction that runs inside the fiefdom. Yet, a fiefdom can expose a service to manage part of a transaction that's controlled and coordinated by an outside party, if that part of the transaction is what is called a compensating transaction. The fiefdom exclusively controls its part of the transaction, but it does it as a service to the client, which is a transaction coordinator.

An example of a compensating transaction is one that follows the Web Services Transactions specification, Part II: Business Activity (BA). Such a transaction has two distinct scenarios. The normal scenario performs the operations that are specified by the transaction. The compensating scenario performs another set of operations that are meant to remove the effects of an already committed transaction within the fiefdom. The compensating scenario is called by a transaction coordinator, which may itself be another fiefdom. It occurs when a fiefdom has already committed its part of the transaction and the coordinator later finds out that some other part of the transaction has failed. In such a case, the effects of the part of the transaction that was run and committed by the fiefdom must be removed, even though it can't be rolled back. The locks on its records are already released, and other transactions might already have changed their state. The compensating scenario removes these effects.

Not all types of transactions are suitable for compensation. Most fiefdoms would not (and should not) take part in transactions for which a robust compensating scenario doesn't exist.

Secured Gateways

To an extent, a service-oriented application is secure by the nature of its service orientation. The only way to have the application perform anything for you is to send it a message, asking it to perform some service. It's up to the application to decide whether to perform or deny the request.

In many cases, the application might permit anyone to have a certain service or set of services performed. In these cases, all the Web Service has to do is delegate the job to a suitable component and then deliver the response to the client. In other cases, the Web Service might need to authenticate the user to find out if he's authorized to have the service performed. This could involve a simple check against Active Directory® or a SQL database at the gateway only; it could also involve security techniques such as impersonation or delegation combined with role-based security checks at different levels within the application. Keep in mind that you must always consider the risk of denial-of-service attacks, hackers trying to intrude on the system, and other malicious threats.

As part of a security initiative, Microsoft has published several security-oriented documents; one of the most important is called "Building Secure ASP.NET Applications," now also available in book form (Microsoft Press®, 2003). You can download the document (Building Secure ASP.NET Applications: Authentication, Authorization, and Secure Communication) or you can buy the book; it contains a great deal of information on how to build in-depth security for your Web-based application.

An Architecture Pattern

Clearly, fiefdom is an architecture pattern. As with other patterns, you as a software architect should be able to say "I think we should design this application as a fiefdom," and the people around you should understand immediately what you're saying. That's one of the great advantages of patterns; they help you discuss architecture and design on a higher level of abstraction and give you a common vocabulary with which to do that.

Obviously, fiefdoms don't all have to be architected the same way. The set of services they will provide affects the component types they need internally, so any fiefdom would be architected by using other patterns as well within its internal implementation. For example, most fiefdoms would use patterns such as data accessors to access the database, service agents to access other services, data transport objects to transport data through the layers, and entity managers to uphold entity-level business rules.

Emissaries

Some fiefdoms will also provide UIs to its services. Such a fiefdom might use one of the emissary patterns—either a Web Form emissary or a Windows® Forms emissary. An emissary is like a sales agent or a mortgage broker. It can display reference data to the user, allowing the user to select from valid alternatives rather than type the information. The emissary also knows many of the business rules used by the fiefdom, so it can help the user fill the request in a way that the fiefdom is more likely to accept it.

An emissary can also save session state for the user. An obvious example is an ASP.NET Web Form that saves the shopping cart for the user between calls, thus allowing the user to add items to those already in the cart, where they remain until it's time to complete the purchase.

The fiefdom, however, doesn't trust the emissary, not even when the emissary must be viewed as a part of the fiefdom. Any request coming in through an emissary is evaluated for security, completeness, and correctness, just as carefully as a request coming in without the assistance of an emissary. This is exactly what an insurance company would do with an insurance application brought in by a sales agent. The emissary is there to serve the interests of the client more than the fiefdom.

So how do you distribute work between a fiefdom and its emissaries? Well, a fiefdom must work with shared data, which means that writing operations must be serialized in order to protect the integrity of the data. In contrast, emissaries can work with local read-only copies of reference data and with per-user data. Therefore, emissaries can ignore concurrency issues. Emissaries typically work with one client at a time in the demilitarized zone, whereas the fiefdom typically resides in a protected intranet and must always be available to all clients.

Using a Web Farm, emissaries can scale up practically without limit. It's much harder to scale out a fiefdom because it handles shared data that must always be available to all.

So, if you can distribute more work to emissaries, thus off-loading the fiefdom, the scalability of your application will improve considerably. You can achieve this through thoughtful design. Think of Amazon.com. When you work your way through their site by putting books and other items into your shopping cart, this could all be emissary work. Not until you press the Submit button to finalize the purchase do you need to engage the fiefdom.

But this is true only if you design your system wisely, drawing a clear line between the fiefdom and the emissary and using performance and scaling enhancers, such as the cache object, vigorously. You must also design the fiefdom's service interfaces in a way that interaction between emissaries and the fiefdom can be minimized.

Validation Against Business Rules

Some requests sent to a fiefdom require the saving or deletion of data. There's no way a fiefdom should accept such requests without carefully validating the request itself against security policies and its data against business rules. That's why business rules and policies belong to fiefdoms, as defined by Pat Helland, and in services, as defined by SOA.

Loose Coupling to Clients

A fiefdom is loosely coupled to its clients. There's normally no connection at all between the client and the fiefdom until the client establishes one by sending a message to the fiefdom. The fiefdom acts on the message and sends a response to the client. The response might be the result of the requested operation, or it might be a refusal to perform the service. In any case, the connection between the client and the fiefdom is broken as soon as the fiefdom has responded, much like an HTTP exchange of messages. Like HTTP, there are ways to maintain the connection over an entire session, but it's normally limited to one complete message cycle.

Loose coupling is good for scalability. It minimizes the time any client is connected to the system and using the system's resources. Loose coupling is the same principle that once governed the design of Microsoft Transaction Services (MTS), which later evolved into COM+ Component Services, and is now known as .NET Enterprise Services.

XML-based Web Services

One of the ideas behind SOA is location transparency; another is independence of computing platform and programming language. Web Services are ideal for meeting the demands underlying these ideas. Since a Web Service is known by its URL, its actual location can change as long as the service provider still points to it.

Since a Web Service is completely independent of the underlying platform and programming language, it only requires the client to understand SOAP, XML, and the transport protocol used. The transport protocol is normally HTTP, but it could also be SMTP or any standard Internet transport protocol.

One could argue that any fiefdom written in a Microsoft .NET-targeted language should expose its data and functionality through Web Services. It shouldn't necessarily expose those Web Services to the outside world; their only type of direct client could very well be emissaries that are well known by the fiefdom.

The interior of a fiefdom should be built on the concept of loose coupling between the fiefdom and its clients. The fiefdom should use stateless service components rather than collections of state objects. It should, as much as possible, transport and manipulate data in the same form in which it receives it and delivers it to clients, which is in the form of XML streams. The Microsoft .NET Framework helps because it provides perfect XML-based APIs for the transport of data through a fiefdom and for the exposure of data to clients.

These APIs are comprised of the XML datasets of ADO.NET. The data access capabilities of ADO.NET make it easy to fetch data from any number of database tables, filling an XML dataset with that data. You can easily transport such a dataset between components, even over process and machine boundaries, to the consumer. In addition, because the data contained in a dataset can be regarded as an easy-to-use XML document, it's extremely easy to use it as an XML Web Service response. Finally, even if the dataset contains XML, you don't have to use XML syntax in order to access or to manipulate its data. You can use ordinary object-oriented syntax such as TrainerData.Trainers(3).LastName in order to access a property.

We sometimes hear an argument against the use of XML datasets to represent business entities that goes something like this: "We don't want to expose the structure of the database to the client." In fact, in most cases it would be extremely unwise to allow the structure of the database to influence the structure of an XML dataset. Usage needs alone should dictate the structure of a dataset. You could easily define the structure of a dataset without having a clue about the structure of the database; it's certainly possible to define all the datasets of an application before even starting on the design of the database.

Designing XML Datasets According to Usage

As an example, you should consider the View Upcoming Racedays use case belonging to a racing application that we built in our previous books and articles. The purpose of this use case is to display a list of upcoming race dates. The data needed includes Date, Track, City, and Country.

Figure 1 XML Dataset
Figure 1 XML Dataset

Knowing the data requirements for this use case, we can design an XML dataset for it. Figure 1 represents the ideal design of such a dataset. As you can see, the dataset contains only one table. The RaceDayTable data table contains all the data the use case needs. With the exception of the RaceDayId column, each column in the data table corresponds to a column shown to the client, as seen in Figure 2. (The RaceDayId column is in the data table to allow the user to select one of the race days displayed and then to have details on that race day displayed on another page.)

Figure 2 Raceday Details
Figure 2 Raceday Details

Let's see where this data comes from. As shown in Figure 3, which displays part of the database model for the application, you must access four database tables to create the one and only dataset table needed by the use case.

Figure 3 Four Database Tables
Figure 3 Four Database Tables

Just for fun, also take a look at a stored procedure that can move the data from the four database tables to the one dataset table, using one SQL SELECT statement and only one round-trip to the database, as shown in the following code:

CREATE PROCEDURE GetFutureRacedays AS

    SELECT RacingProgramId As RaceDayId, Date, TrackName As
             RaceTrackName, CityName, CountryName
    FROM RacingPrograms As RP
    JOIN RaceTracks As RT ON (RP.RaceTrackId = RT.RaceTrackId)
    JOIN Cities As Ci ON (RT.CityId = Ci.CityId)
    JOIN Countries As Co ON (Ci.CountryCode = Co.CountryCode)
    WHERE Date > GetDate()
    ORDER BY RacingProgramId
As you can see from this example, the structure of the dataset and the structure of the database are totally different. This is because the dataset is designed from the requirements of a single use case, whereas the database is designed using formal rules that help prevent anomalies from occurring when using the data.

Using Customized Entity Classes

Let's see how developers who don't want to expose the structure of the database to the client would design their objects. Chances are they will follow one of the popular design patterns for stateful objects, which represent the so-called domain model. The domain model, of course, represents the objects of the real-world domain. In our racing application, the domain model would consist of classes such as horses, trainers, jockeys, and races.

Incidentally, the real source for domain data is the database. This is where most structured data is stored. It stands to reason that the database model closely represents the domain; therefore, it is a domain model.

This is why the set of entity classes in an application, which is mainly comprised of customized entity classes, closely corresponds to the set of tables in a database that represents the same domain. This is also one reason that many developers grossly underestimate the value of real data modeling as opposed to class modeling.

Ideally, the correspondence between the class domain model and the relational domain model shouldn't be as close as it usually is; the database should be properly normalized. Normalizing in-memory entity classes the same way often leads to a lot of code that you don't really need. It also leads to increased complexity of that code and to slow operations.

Mapping an Entity Object to the Database

So why do in-memory entity classes tend to be as normalized as database tables? This is mainly because of the strategies often used to map the in-memory objects and the relational tables to each other. With few exceptions, authors who recommend a memory-based domain model also recommend a nearly one-to-one mapping between memory-based classes and database tables.

Figure 4 provides a simplified example of what a client must face in such a situation. With the exception of keys and collection classes, this is an exact replica of the database tables. It goes without saying that it takes more than one SQL SELECT statement to populate these four entity objects. It's also obvious that it takes longer to make four round-trips to the database than it takes to make one round-trip. And it's all in the interest of making the client less dependent on the database.

Figure 4 Result of Mapping
Figure 4 Result of Mapping

It should be clear that exposing XML datasets to client applications is not the same as making the client dependent on the database. In fact, it's the opposite! But you must understand the XML datasets to take full advantage of them. When you define their XSD schemas, you should concentrate exclusively on their usage; you shouldn't consider the structure of the database at all.

This is especially important in service-oriented apps. A service-oriented app exposes programming interfaces that return XML streams or XML documents to clients. This is at least true for service-oriented apps that expose Web Services. The structure of an XML document is controlled by its XSD schema, and the same is true for XML datasets which, in effect, are XML documents.

An XML Dataset is Never Connected to the DB

What some developers may not know yet is that an XML dataset is never connected to the database. It can't be. To move data between an XML dataset and a database you need some kind of proxy to map dataset tables to relational tables. ADO.NET offers such an object, which comes in different flavors for different database products. It's called a data adapter, and its only purpose is to move data between datasets and data sources. The dataset itself doesn't know anything about the database or about the SQL statements that fill it or that move data to a data source.

So why is it so easy to be confused about this? Why do so many believe that using a dataset means depending on the database? The reason is probably that the XML dataset, in a manner of speaking, is a replacement for the old recordsets used in traditional ADO, Remote Data Objects (RDO), and Data Access Objects (DAO). In all of these models, the recordset is directly connected to a data source such as a relational database. The only exception is the ADO recordset, which can be disconnected from and then reconnected to the database.

XML datasets are entirely different from the old recordsets. Their purpose is to keep and transport snapshots of data, totally independent from and in total ignorance of any database that the data might come from.

Conclusion

The next time you're involved in starting a software development project, consider architecting it as a fiefdom. If your would-be application will expose a user interface, which most applications do, you should think about architecting it as a set of emissaries. The emissaries should try as much as possible to do their jobs without consulting the fiefdom, except when it's absolutely necessary. For example, they should cache reference data and they should keep session state without bothering the fiefdom. An application built this way will be more scalable than an application that makes maximum use of server resources such as shared data.

This strategy will help you design your fiefdom to use stateless components. This is extremely useful in terms of scalability and simplicity. However, you should use stateful components only when absolutely necessary. In most applications, you should really consider using ADO.NET XML datasets as data transporters. Together with data adapters, they work beautifully with databases. They work across process and machine boundaries transparently, and they're completely based on XML. They are designed for data binding, which in many cases should be the preferred method of displaying data to users in Web Forms or Windows Forms.

If you want to learn more about Pat Helland's fiefdom and emissary architecture patterns, listen to him in person at Architect Webcasts: Autonomous Computing: Fiefdoms and Emissaries; it's well worth the time spent.

Send your questions and comments for Design Patterns to mmpatt@microsoft.com.

Sten Sundblad and Per Sundblad jointly run Sundblad & Sundblad ADB-Arkitektur in Sweden, a company specializing in .NET development in general, and architecture and design in particular. Together they wrote Designing for Scalability with Microsoft Windows DNA (Microsoft Press, 2000) and Design Patterns for Scalable Microsoft .NET Applications (Sundblad & Sundblad, 2002). More information on Sten and Per is available at http://www.2xSundblad.com. You can reach them at stens@2xSundblad.com and pers@2xSundblad.com, respectively.


C++ Q&amp;A: Parent and Child Window Captions, More MoveDlg

C++ Q&A
Parent and Child Window Captions, More MoveDlg
Paul DiLascia

Code download available at: CQA0307.exe (811KB)

Q I'm having trouble setting the multi-document interface (MDI) parent's text in C#. When the child form is in maximized state and activated, the child form's text gets appended to the parent text and is displayed in the form "Parent Text - [Child Text]". How can I override this default behavior and set custom title text?

Srinivasa Raghavan

A If you were using MFC, you could simply override the virtual function CFrameWnd::OnUpdateFrameTitle in the parent frame. But what about in the Microsoft® .NET Framework? To override the default MDI caption, you have to first understand that this behavior comes from MDI itself via the Windows core API, not MFC or the .NET common language runtime (CLR). When you create an MDI app, your frame and child windows get special window procedures—DefFrameProc and DefMDIChildProc—that handle various WM_MDIXXX messages as well as other messages like WM_SIZE, WM_SYSCOMMAND, and others to implement MDI behavior.

If you're writing pure C code, you're responsible for using DefFrameProc and DefMDIChildProc when you create your windows; MFC and the .NET Framework perform these details invisibly when you create CMDIFrameWnd/CMDIChildWnd in MFC, or when you set Form.IsMdiContainer and Form.MdiParent in the .NET Framework. Either way, it's the underlying user kernel, and in particular DefFrameProc, that concatenates the parent and child captions to create the main window caption when the MDI child is maximized. With that understanding, I'll show you how to override MDI. I'll use the C# sample Scribble MDI app that's in the MSDN® Library (search for "scribble sample" or go to Scribble Sample: Visual C# MDI Drawing Application). The basic idea is to override WM_GETTEXT in Scribble's MainWindow. First, you have to add a couple of data members, NormalText and MaximizedText, to hold—what else?—the captions for normal and maximized state, as shown in the following code:

// in Scribble.cs, MainWindow class
private String NormalText = "Scribble2";
private String MaximizedText = "Window is now maximized";
If you want other classes to access these strings, you can implement them as properties instead of data members:
private String normaltext;
public String NormalText
{
  get { return normaltext; }
  set { normaltext = value; }
}

Since MainWindow is the only class in my sample to access the strings, I didn't bother with properties. With the new data members in place, all you have to do is override WM_GETTEXT to return the maximized text if the child window is maximized, and the normal text otherwise. But how do you override WM_GETTEXT? Windows.Forms provides virtual counterparts for some, but not all, WM_XXX messages. For example, there's OnResize for WM_SIZE, but there's no OnGetText for WM_GETTEXT. But never fear—if there's no virtual function, you can always override the catchall handler WndProc. You have to know the ID of the message you're looking for—in this case, WM_GETTEXT is 0x000D—which you can find by running Spy or simply by looking in the Windows® SDK file winuser.h. Once you sink to the level of WndProc, you're basically writing in C since everything, including strings, is passed as WPARAMs and LPARAMs. For WM_GETTEXT, Message.LParam is a pointer to a char* buffer and Message.WParam is its length. This means you have to perform a little interop to copy the text string into the caller's buffer. Fortunately it's not rocket science; Figure 1 shows the details (full source code is available at the link at the top of this article). Now when the MDI child is maximized, the main window caption displays the text "Window is now maximized" as in Figure 2; otherwise the main window displays the usual caption, as shown in Figure 3.

Figure 1 Scribble (Close Figure 1)
public class MainWindow : System.Windows.Forms.Form
{
   private String NormalText = "Scribble2";
   private String MaximizedText = "Window is now maximized";

   // Handle WM_GETTEXT: Return maximized or
   // normal text, depending on
   // state of active MDI child window.
   protected override void WndProc(ref Message m)
   {
      const int WM_GETTEXT = 0x000D;
      if (m.Msg==WM_GETTEXT) {
         Form active = this.ActiveMdiChild;
         String s = active!=null &&
            active.WindowState==FormWindowState.Maximized ? MaximizedText :
            NormalText;
         char[] c = s.ToCharArray();
         IntPtr buf = m.LParam;
         int len = c.Length;
         Marshal.Copy(c, 0, buf, Math.Min((int)m.WParam, len));
         m.Result = (IntPtr)len;
         return;
      }
      base.WndProc(ref m);
   }

      ••• // rest of MainWindow unchanged from Scribble sample

}
}
Figure 2 Maximized Caption
Figure 2 Maximized Caption
Figure 3 Normal Captions
Figure 3 Normal Captions

Q After reading your suggestion to implement window dragging by handling the WM_NCHITTEST message in the December 2002 issue of MSDN Magazine, I tried it in a program that used to do all that WM_LBUTTONDOWN, WM_MOUSEMOVE, and WM_LBUTTONUP stuff. (Of course, I wanted to do it the cool way.) But then I ran into a little problem. My window used to have a context menu, which wouldn't appear anymore. So I changed the message handler (as shown in Figure 4). This works fine for my program, but I wonder if this is the right way to do it?

Figure 4 Hit-test (Close Figure 4)
//////////////////
// Revised hit-test handler to fake out Windows into thinking user has
// clicked on the caption area of a dialog or window in order to
// implement move-by-dragging-the-client-area.
//
UINT CMainFrame::OnNcHitTest(CPoint point)
{
    // Not returning HTCAPTION when the right button is pressed is
    // important to keep the ability to generate WM_CONTEXTMENU messages.
    //
    BOOL bMouseButtonsAreSwapped = GetSystemMetrics(SM_SWAPBUTTON);
    int logicalRightButton = bMouseButtonsAreSwapped ? VK_LBUTTON :
        VK_RBUTTON;
    short rightMouseButtonState = GetKeyState(logicalRightButton);
    BOOL  bRightMouseButtonIsDown = rightMouseButtonState & 0x8000;

    CRect rc;
    GetClientRect(&rc);
    ClientToScreen(&rc);
    return (rc.PtInRect(point) && !bRightMouseButtonIsDown) ? HTCAPTION :
        CFrameWnd::OnNcHitTest(point);
}
Rocco Matano

A I love it when readers answer their own questions; it saves me so much time and energy. Just to refresh the rest of the readers, the December 2002 column showed how to implement a feature that lets users move a dialog or window by dragging its client area. The trick is to override OnNcHitTest to return HTCAPTION instead of HTCLIENT, even when the mouse is actually in the client area, to fool Windows into performing its drag/move functionality. But as Rocco discovered, this messes up your context menu. If you return HTCAPTION instead of HTCLIENT, Windows doesn't know to send a WM_CONTEXTMENU. Rocco's solution is to call GetKeyState in order to check for right-button down before doing the HTCAPTION trick. Kudos to Rocco for remembering to check for swapped left/right mouse buttons—would you have remembered that?

I don't think there's anything wrong with Rocco's solution, unless by a fluke of timing—caused perhaps by a hardware interrupt or other unlikely event—the mouse button is already up by the time the window gets WM_NCHITTEST. After probing the depths of the MSDN Library and reading more than anyone cares to hear about the details of mouse-message processing in Windows, I was unable to definitively determine whether this is even possible. Call me paranoid, but somehow I'm reluctant to use GetKeyState to check for mouse button states. And who should have to worry whether the user is left-handed? Never fear, in Windows there's always another way. In this case, all you have to do is override OnNcRButtonUp:

// in OnNcRButtonUp
if (nHitTest==HTCAPTION && /* point in client area */)
  SendMessage(WM_CONTEXTMENU, ...);
else
   // default thing

In other words, if the hit-test code says the mouse is in the caption, but it's really in the client area, then you should send yourself a WM_CONTEXTMENU message. This is what Windows would've done if I hadn't fooled it by returning HTCAPTION in OnNcHitTest. Figure 5 shows the details. Note that if you want the dialog to display the system menu when the user right-clicks on the caption, your context menu handler should pass control to the base handler if the cursor isn't in the client area. This is true whether you're using the hit-test trick or not; even in a vanilla dialog, you should perform this check. Figure 6 and Figure 7 show the revised MoveDlg program from my December column, with system and context menus.

Figure 5 MoveDlg (Close Figure 5)
////////////////////////////////////////////////////////////////
// MSDN Magazine — July 2003
// Compiles with Visual Studio .NET on Windows XP. Tab size=3.
//
#include "stdafx.h"

// As in my original December 2002 column

//////////////////
// Typical dialog.
// Override OnNcHitTest to allow dragging by background.
//
class CMyDialog : public CDialog {
    •••
protected:
   // helper
   BOOL PtInClientRect(CPoint p);

   // virtual override/message handlers
   afx_msg UINT OnNcHitTest(CPoint pt);
   afx_msg void OnNcRButtonUp(UINT nHitTest, CPoint pt);
   afx_msg void OnContextMenu(CWnd* pWnd, CPoint pos);
};

BEGIN_MESSAGE_MAP(CMyDialog, CDialog)
   ON_WM_NCHITTEST()
   ON_WM_NCRBUTTONUP()
   ON_WM_CONTEXTMENU()
END_MESSAGE_MAP()

//////////////////
// Helper function to test if screen point lies in client area
//
BOOL CMyDialog::PtInClientRect(CPoint p)
{
   CRect rc;
   GetClientRect(&rc);
   ClientToScreen(&rc);
   return rc.PtInRect(p);
}

//////////////////
// Non-client hit-test handler to move window by its client area.
// If the user clicks anywhere on the client area, pretend it's the
// caption. Windows does the rest!
//
UINT CMyDialog::OnNcHitTest(CPoint pt)
{
   return PtInClientRect(pt) ? HTCAPTION : CDialog::OnNcHitTest(pt);
}

//////////////////
// User let go of right button in "non-client" area. If mouse is really
// in the client area because I tricked Windows, send myself a
// WM_CONTEXTMENU message.
//
void CMyDialog::OnNcRButtonUp(UINT nHitTest, CPoint pt)
{
   if (nHitTest==HTCAPTION && PtInClientRect(pt))
      SendMessage(WM_CONTEXTMENU, (WPARAM)m_hWnd, MAKELPARAM(pt.x,pt.y));
   else
      CDialog::OnNcRButtonUp(nHitTest, pt);
}

//////////////////
// Display context menu. If mouse is in the client area, display
// resource menu—otherwise, let base class handle it to display system
// menu. This is required even in vanilla dialog (i.e., without the
// hit-test trick).
//
void CMyDialog::OnContextMenu(CWnd* pWnd, CPoint p)
{
   if (PtInClientRect(p)) {
      CMenu menu;
      VERIFY(menu.LoadMenu(IDR_CONTEXTMENU));
      CMenu* pSubMenu = menu.GetSubMenu(0);
      pSubMenu->TrackPopupMenu(0, p.x, p.y, this);
      return;
   }
   CDialog::OnContextMenu(pWnd, p);
}
Figure 6 MoveDlg with System Menu
Figure 6 MoveDlg with System Menu
Figure 7 MoveDlg with Context Menu
Figure 7 MoveDlg with Context Menu

So which solution should you use? Rocco's has the advantage of being confined to a single function, but it seems too low level. You shouldn't have to worry about virtual key states or whether the user has swapped mouse buttons unless you're writing drivers or other grungy stuff.

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


Talking To...: Ken Getz Talks About His Role as a Microsoft MVP

Talking To...
Ken Getz Talks About His Role as a Microsoft MVP

Most Valuable Professionals (MVPs) are technical experts outside of Microsoft who volunteer their time engaging with the technical community to answer questions, provide training, and assist in product guidance. MSDN Magazine interviewed Ken Getz, an MVP since 1994, to find out more about the program and Ken's involvement in it.

MSDN What was your first experience in the computer industry?

Getz Back in 1983, I was teaching high school math and music at a prep school in Boston. I was totally burnt out and was looking for a new career. The guy in charge of the school's data processing and keeper of the school's PDP-11 quit, and I stretched the truth a bit and convinced the headmaster that I could program the thing. Heck, I couldn't even boot it (if you remember, that required some masterful dipswitch fiddling). A bunch of BASIC manuals later, and I was in business. Soon, the school bought one of the original IBM PCs, and I fell in love right away. A bunch of night courses at Harvard Extension later, I was a programmer! If only it was actually that easy. But it sounds good 20 years later.

MSDN How did you earn your MVP title?

Getz Around 1993, I had a client for whom I had been struggling writing a Visual Basic 1.0 database application—there wasn't much support in there for real database applications. Microsoft Access came out, and I knew I'd found the right tool for the job. I really dug in, wrote an application for the guy, found the CompuServe forums, and started answering questions there for fun, based on my experience with the one application. Answering the questions on the forum took on a life of its own, and when they started the program in 1994, I was among the first batch of Access MVPs selected (I think there were something like five or six Access MVPs at the time—now there are many, many more). Back in those days, the CompuServe support forums were really something—a tight-knit community, where most of the regulars knew each other and all the Microsoft support folks, as well. I really miss those times. Everyone I know who was involved says the same thing.

MSDN What does being an MVP entail?

Getz Originally, being an MVP was based on the quality and quantity of messages posted on the CompuServe forums. Over time, as online support grew and moved to MSN and then to public newsgroups, peer recommendations also became part of the process. Most MVPs I know spend time answering questions on the newsgroups. Most years, Microsoft sponsors a "get-together" on campus, allowing MVPs to meet, chat, and learn about new and forthcoming technologies. But mostly, it's about answering questions on the newsgroups, at least from my perspective.

MSDN Do you work on projects with other MVPs? What do you do together?

Getz I've been involved in projects with other MVPs for as long as I can remember. The company I work with, MCW Technologies, was started by five MVPs back in 1996 or so, and we still have three. Mary Chipman (SQL Server), Andy Baron (.NET), and I have been collaborating on courseware and training from AppDev (http://www.appdev.com) for the past few years. We've done ASP.NET, Visual Basic .NET, and ADO.NET courses, and we're looking forward to doing more.

MSDN What excites you enough to wake up every morning?

Getz I've been self-employed since 1991 and have worked pretty much every single day of that time. I love solving problems and enjoy explaining the solutions I find. After having written so much courseware—several thousand pages over the past six years—I've finally figured out how to do it, and I look forward to it most days. The point is to "tell a story"—in other words, to find a path through a topic so that you can motivate each new concept based on previous knowledge and a need to solve a particular problem. It's really made a difference in the work. I also do a lot of public speaking ("blabbing", we call it at home), and I love doing that, as well.

MSDN What's better, Visual Basic or C#?

Getz You've got to be kidding! Do you really think I'm going to answer that in a public forum? There's no way. Just what I need is a language war on my head. I'll be very happy to give you the "party line," but I'm not going to vote in this race. Which one is better? Visual Basic or C#, obviously.

Microsoft MVPs are a diverse group of individuals who are:

Recognized: Microsoft MVPs are acknowledged by peers and also by Microsoft for their active participation in Microsoft technical communities around the globe.

Credible: Microsoft MVPs have demonstrated practical expertise providing the highest quality information and content.

Accessible: Microsoft MVPs are active technical community leaders sharing their experience with peers.

Find out more about the MVP Program at http://mvp.support.microsoft.com.