 MSDN Magazine December 2004

MSDN Magazine December 2004

Tablet PC: Add Support for Digital Ink to Your Windows Applications

Check out the cool new features in Windows XP Tablet PC Edition, including a number of Ink types, and ink that's stored as ink. Here Paul Yao takes you on a tour of everything you need to know to get started.

Paul Yao Mobility: Optimize Your Pocket PC Development with the .NET Compact Framework

The .NET Compact Framework can be used to write great code and great applications. As long as you take a few things into consideration and are willing to bend a rule or two, you can have your performance cake and eat it too. In this article the authors present some neat tricks to make life as a programmer easier when using the .NET Compact Framework. Later they discuss techniques to increase performance, and decrease both load time and memory footprints. Sample code is provided.

Dave Edson and John Socha-Leialoha Vrooooom: How .NET and C# Drove an Entry in the DARPA Grand Challenge

Find out how the .NET Framework, a team of programmers, and a bunch of people from Carnegie Mellon University built an automated car to compete in the DARPA Grand Challenge. Along the way you get some inside tips on building an extensible real-time control architecture based on a whiteboard metaphor and implementing an accurate GPS-synchronized timer component for .NET.

John Hind .NET Internals: Tailor Your Application by Building a Custom Forms Designer with .NET

The design-time architecture of Windows Forms in the .NET Framework has made development much more flexible than it had been with MFC. With Windows Forms, you can drag one of your custom controls from the toolbox and drop it onto the Visual Studio design surface and even though Windows Forms knows nothing about the control, it's able to host it and let you manipulate its properties—not possible in MFC. In this article, the author discusses what's going on under the covers as you design your forms and then walks through the creation of a bare-bones forms designer.

Sayed Y. Hashimi .NET Code Tuning: Make Your Apps Fly with the New Enterprise Performance Tool

Because the common language runtime (CLR) is a black box, it's pretty hard to divine what's going on when you want to track down performance problems. Microsoft will be delivering a brand new profiler, the Enterprise Performance Tool (EPT), as part of Visual Studio 2005 Team Developer Edition that's ideal for use on a production system because it offers some very lightweight means of collecting performance data. Here John Robbins takes you on a tour.

John Robbins Editor's Note: Going Mobile

A couple of times a week, we are the recipients of advertising e-mail from a popular retailer of mobile products. Here at MSDN Magazine, we field a few dozen publicly available e-mail aliases, so it's easy for mail like this to get lost in the shuffle of our Junk E-Mail folder.

New Stuff: Resources for Your Developer Toolbox

There is more to fielding a mobile device than just putting a couple of company-specific applications on it. Here are some already written mobile applications that might be good to know about if you are tasked with deploying a mobile solution.

Marnie Hutcheson Web Q&A: Windowed and Windowless Elements, Cookie Characters, and More

Edited by Nancy Michell Advanced Basics: Calling All Operators

This month I found an interesting question in my mailbag. It went something like this: "I am writing some code working with points and other drawing objects in Visual Basic® 2003, and I just want to add an offset to a point, effectively moving the point.

Ken Getz Data Points: Efficient Coding With Strongly Typed DataSets

Someone once said to me that the hallmark of a good developer is the desire to spend time efficiently. Developers are continually pursuing ways to make coding easier and faster, and to reduce the number of errors.

John Papa Cutting Edge: Implications of Script Callbacks in ASP.NET

Script callbacks in ASP. NET 2. 0 is a feature whose time has come. Script callbacks can significantly speed up an application by limiting server postbacks. They also allow you to execute small portions of server-side code without having to manage the view state for reading or writing.

Dino Esposito Service Station: Run ASMX Without IIS

When the Microsoft® . NET Framework first shipped, it introduced a breakthrough Web services framework known as ASMX. The motivation behind the ASMX design was to simplify the process of developing Web services as much as possible so that even if you're not an XML expert, you can get a Web service up and running.

Aaron Skonnard .NET Matters: Asynchronous HttpWebRequests, Interface Implementation, and More

Stephen Toub C++ Q&A: Deleting Managed Objects, Wrapping a Library, and More

Paul DiLascia { End Bracket }: Around the World with Visual Basic

The Visual Basic team at Microsoft has spent much of the past eight months out on the road as a part of the Visual Basic® . NET User Group Tour 2004. We've had a chance to visit 13 user groups in the U.

Jay Roxe


Tablet PC: Add Support for Digital Ink to Your Windows Applications

Tablet PC

Add Support for Digital Ink to Your Windows Applications

Paul Yao
This article discusses:
  • Windows XP Tablet PC Edition
  • Setting up your Tablet PC dev environment
  • Writing code to process ink
This article uses the following technologies:
C#, .NET Framework
Code download available at: TabletPC.exe (129KB)
Contents
The Ink Input Function
Tablet PC Hardware
Speech Input
Programming with Ink
Ink Classes
Gestures
Ink Data
The Stroke Class and the Strokes Collection
Appearance and Display of Ink
Analyzing Ink
Converting Ink to Text
TicTacToe: An Ink-Enabled Sample
InkEdit Controls
A Full-Screen InkCollector
Nine InkCollectors
Assembly and Namespace References
TicTacToe Initialization and Cleanup
InkCollector Events
What's Next

T
wo of the most innovative features of the Tablet PC are the pen and microphone. While these devices are fairly novel for most PC users, they are intended to provide new choices for users, not to eliminate traditional input devices. In fact, a user may opt to use voice or pen input with some programs, mouse and keyboard with other programs, or voice when in transit.

By speaking commands a user can, for example, run a program or open or save a file. Speech can be used in other ways as well, most of which are not new, including conversion of spoken word to text and the recording of speech for later playback.

The pen can be used as a pointing device, substituting for a mouse. But the widest use of pen input has been for handwriting recognition. The handwriting recognition engine on the Tablet PC uses a neural network to analyze pen movement and a word list to cross-reference its results. Unlike earlier recognition engines, the Tablet PC recognizers work better with cursive than with block letters. The current generation of handwriting recognizers does have some limitations, however; it does not have the ability to learn from its mistakes.

So, how good is the handwriting recognition? Figure 1 shows examples of my handwriting as I wrote the phrase "Can you read this now?" I am sometimes surprised how well it can convert garbled script into the correct text and at other times I am amazed that simple words are not interpreted the way I expect them to be. The recognizer relies on context, so with its letters all connected, cursive produces the best results. Single characters and block letters produce less consistent results. Without context, the recognition engine is unsure about whether to interpret a single vertical bar as the number 1, a lowercase L, or an uppercase I.

Figure 1 Can You Read This Now?
Figure 1 Can You Read This Now?

The recognizer works in the background and its results are displayed as they are produced within the Tablet PC Input Panel. As you can see in Figure 1, some of the words I wrote were not recognized correctly. When that occurs, a user can click on a word and a correction window appears.

The Ink Input Function

Pen input does not have to be converted into text. It can be stored as a set of points in a new data type called Ink. The Ink type can contain either text or non-text information such as a flow chart, an org chart, a street map, or even a combination of text and pictures.

In fact, Windows® Journal, which is an accessory that comes with Windows XP Tablet PC Edition, could be described as an ink-enabled cross between Notepad and Paint. It allows text input as well as freehand drawing, plus a whole lot more.

Windows Journal demonstrates the advantages of electronic ink. Physical ink cannot move to allow more room to add notes in between lines of existing notes; electronic ink can. Inside Windows Journal, handwriting recognition does not occur unless a user requests it; the default is for ink to remain as ink. And this is a core improvement in Tablet PC technology.

Tablet PC Hardware

I worked with four Tablet PCs while writing this article: the Acer TravelMate C100 Convertible Tablet PC, the Compaq Tablet PC TC1000, a ViewSonic TPCV1250, and a Toshiba Portege M200. I found all of them easy to set up and use.

The hardware in a Tablet PC represents a rethinking of the elements needed in a portable computer. To save weight, serial ports, parallel ports, and floppy drives are excluded. Most have two or more USB ports and many have 1394 Firewire ports. Most of them have built-in Wi-Fi (802.11b) support; most also have built-in wired network cards.

A USB mouse and keyboard convert even the lightest slate tablet into a familiar PC. While Tablet PC display hardware is not at all unique, Tablet PC display drivers are unique because of the support for multiple display orientations. For example, on the systems I tested, two landscape and two portrait orientations were supported. Behind the display screen is another unique feature, a high-resolution digitizer.

The digitizer picks up pen movements, but not other physical pressure, so a user can rest a hand on the screen without accidentally creating unwanted input. Even when there is no physical contact between the pen and the screen, a pen waved over a Tablet PC screen causes the mouse cursor to scurry along like a puppy at meal time. The digitizer's ability to detect the pen tip is the magic that makes the pen work.

A Tablet PC pen has at least one user-activated switch, which must include a pen tip and can also include an eraser and buttons on the pen barrel. The pen tip contains the one required switch, which triggers the equivalent of a left-button click when it makes contact with the screen. For some Tablet PC pens, a second switch is provided on the "eraser side" of the pen. Pen-enabled drawing applications often use that second button to erase on-screen objects. The right-click button can sometimes be found on the barrel of the pen.

Context menus summoned with right-clicks are commonplace, but barrel buttons are not standard on all Tablet PC pens. To address this, the Tablet PC shell incorporates a user action similar to that found on the Pocket PC: tap-and-hold. When using a Tablet PC pen without a barrel button, you can tap and hold to summon context menus.

There are six distinct ways for a pen to generate character input on a Tablet PC:

  • Input Panel Keyboard
  • Input Panel Writing Pad
  • Input Panel Character Pad
  • Pen Input Panel
  • InkEdit control
  • Ink-Enabled Window
Of these, the first three can be used in any application. The last three are only supported for ink-enabled programs.

Using a pen, a user can enter character data using the onscreen keyboard (see Figure 2) The Input Panel appears when a user clicks an icon in the system taskbar and can be hidden when not needed.

Figure 2 Tablet PC Input Panel Keyboard (Close Figure 2)

The Input Panel can be docked or undocked. The three mode keys—Control, Shift, and Alt—are "sticky" so that multiple-key input is possible using the pen by itself. Incidentally, the Input Panel keyboard is not the same as the on-screen accessibility keyboard that comes with both Windows XP and a Tablet PC.

The Input Panel also supplies a handwriting area, which collects ink input. After a slight pause, or when the user clicks the Send button, ink is recognized (converted to text). The resulting text is then sent to the window with focus given to the foreground application (see Figure 3).

Figure 3 Input Panel Writing Pad (Close Figure 3)

The Input Panel Character Pad is identical in appearance to the Input Panel Writing Pad except for the presence of small vertical ticks (see Figure 4). Using the guides, text can be entered one letter at a time. For users who insist on printing individual characters, the Character Pad uses the guides to help improve the results of handwriting recognition. It is so oriented towards individual characters that it is unable to interpret cursive characters. Instead, each stroke is assumed to belong to only one character.

Figure 4 Input Panel Character Pad (Close Figure 4)

When undocked, the Tablet Input Panel is referred to as the Pen Input Panel. Ink-enabled programs are able to control the location and appearance of the undocked Input Panel through the PenInputPanel class. Through an instance of this class, an ink-enabled program can incorporate a pen input panel to capture character input. Identical to the system's handwriting input panel, which stays docked to the top or bottom of the screen, a Pen Input Panel is positioned next to the window which is to hold handwriting input the user will enter.

Figure 5 Ink in Notepad
Figure 5 Ink in Notepad

The Pen Input Panel can be used even when a program is not explicitly ink-enabled. The top of Figure 5 shows how a button appears inside Notepad (which is not ink-enabled) when a pen is pointed at its input window. When the button is clicked, the undocked Tablet Input Panel appears. The Pen Input Panel creates a unified pen-based text entry for all programs on a Tablet PC, not just those that are ink-enabled.

Figure 6 Text Drawn
Figure 6 Text Drawn

InkEdit is the built-in control that supports handwriting recognition. Your program can treat InkEdit like a regular edit control, making this the simplest way to ink-enable an application. The user sees a window that directly accepts and recognizes ink as text. Figure 6 and Figure 7 shows ink drawn in an edit control and the resulting text that has been recognized and placed into that control.

Figure 7 Text Recognized
Figure 7 Text Recognized

InkEdit replaces a standard edit control, but is built on the rich text control, not on the standard edit control. This means that an ink edit control can display text drawn using different font sizes and styles. (A standard edit control supports one font size and style.)

The greatest level of application integration with pen input involves ink-enabled windows. An ink-enabled window is an application-specific window that calls the ink support libraries to create and manipulate Ink objects. An ink-enabled window can convert ink to text, which satisfies its inclusion here as a character data collector. An ink-enabled window can also support pen-based command input called gestures. In addition to text, ink-enabled windows also support picture drawing. When text and pictures are placed in the same window, the Microsoft® Ink libraries support a Divider object that helps separate ink-based text data from ink-based pictures.

Speech Input

The Tablet PC incorporates two important uses of a microphone: text recognition and command input. To activate speech-enabled programs, use the Microsoft Speech SDK, version 5.1. Even without adding this support, the Tablet PC supports speech as input for any Windows XP-compatible program. Speech input is controlled from the Input Panel (see Figure 8).

Figure 8 Speech-Related Buttons
Figure 8 Speech-Related Buttons

As of this writing, four speech recognizers are available: Chinese (Simplified), Chinese (Traditional), English, and Japanese. For best results, a user should spend some time training the recognizer in the nuances of her speech. I also suggest you consider purchasing a high-quality microphone, which will also help.

Figure 9 Available Voice Commands
Figure 9 Available Voice Commands

When speech is enabled on a Tablet PC, a set of speech-related buttons appear on the Input Panel. The buttons marked Dictation and Command allow a user to decide whether speech should be interpreted as text to be treated as input data, as a command, or (when both buttons are off) to ignore speech input. When a user clicks on the Command Input button in the Input Panel, text is interpreted as commands. You can, for example, tell your computer to "launch Notepad," "select paragraph," or summon the Start menu by simply saying "Start menu." If you are unsure of the available commands, ask the system "What can I say?" to summon the window showing available voice commands (see Figure 9).

Programming with Ink

The ability to store ink as ink is probably the most important innovation in the Tablet PC. In addition, ink can also be used to annotate documents, spreadsheets, Web pages, and, of course, other ink drawings and images.

You can build ink-enabled applications in either managed or unmanaged code. A set of ActiveX® controls/COM components support unmanaged ink-enabled programs. A pair of managed code assemblies supports the creation of ink-enabled programs in a managed language, such as C# or Visual Basic® .NET. The .NET Framework 1.0 is included with Windows XP Tablet PC Edition, which has the distinction of being the first Microsoft operating system with built-in support for managed applications. The sidebar "Tablet PC Test and Development Systems" describes three options for setting up a development system to build Tablet PC software.

Ink Classes

The managed code interfaces are organized by a pair of managed assemblies: Microsoft.Ink.dll and Microsoft.Ink.resources.dll. These assemblies are added to the global assembly cache so they can be shared by different managed code programs. Figure 10 summarizes the core classes for building ink-enabled programs. Not every class listed is needed to create ink-enabled programs.

Figure 10 Core Ink Programming Classes (Close Figure 10)

Class Name | Description
Divider | Analyzes ink to distinguish text from pictures.
DrawingAttributes | Controls appearance of ink, such as color, line width, and so on.
Gesture | Ink interpreted as a command.
Ink | The main container to hold ink. Holds the collection of strokes that make up ink input. Supports moving ink between memory and disk. Supports clipboard actions on ink.
InkCollector | Snap-on support for creating a basic ink-enabled window.
InkOverlay | Snap-on support for adding ink-enabled support to existing application windows.
PenInputPanel | An in-place input window for adding in-place ink input for existing controls.
Recognizer | Provides language-specific conversion of ink strokes to text and the system dictionary with common words in a given language.
RecognizerContext | Organizes the input elements needed for ink-to-text conversion: a recognizer, hints about data type (factoid), and an application-specific dictionary (word list).
Renderer | Draws ink onto a drawing surface.
Stroke | A set of (x,y) point values generated by a pen. A stroke starts when a pen makes contact with the drawing surface, and includes the locations traversed by the pen until the pen is lifted from the drawing surface.
Strokes collection | Holds a set of strokes.

The ink classes operate by connecting to a window. InkCollector and InkOverlay attach to an application window and use that window's message stream to ink-enable it. When building an ink-enabled application, you have to pick one of these two classes; you then connect the class to one of your program windows.

InkCollector provides support for a pure-ink window. It is the simpler of the two and it was created to support ink-enabled programs that are written from scratch. In a pure-ink window, the InkCollector provides the primary support for managing the contents of the window.

InkOverlay is more powerful than InkCollector. With it, you can let users perform such actions as editing an onscreen graphic layout by drawing circles around items and scribbling comments in the margin. The ink overlay class allows the ink drawing model to coexist with an application-specific drawing model.

Through a CollectionMode property both InkCollector and InkOverlay accept a setting for how pen input is to be interpreted. The possible values are available in the CollectionMode enumeration. An application can choose to receive ink input, gesture input, or both, as shown in the following code.

enum CollectionMode {
    InkOnly = 0,
    GestureOnly = 1,
    InkAndGesture = 2
};

So what is the difference between ink input and gesture input? Ink is drawn and collected; it produces data. Gestures, on the other hand, are not drawn (or if they are drawn they are immediately erased) and represent commands.

Gestures

One way to think about gestures is as pen-driven commands. The ink libraries support two types of gestures: system and application. Each contains a fixed number of elements; available system gestures are defined in the SystemGesture enumeration.

   enum SystemGesture {
       Tap = 0x10,
       DoubleTap = 0x11,
       RightTap = 0x12,
       Drag = 0x13,
       RightDrag = 0x14,
       HoldEnter = 0x15,
       HoldLeave = 0x16
       HoverEnter = 0x17,
       HoverLeave = 0x18
   };
Application gestures are defined in the ApplicationGesture enumeration. A complete list of available application gestures can be found at ApplicationGesture Enumeration.

Whereas application gestures are gestures you can choose to have your application support, system gestures are surrogates for mouse events. When not processed, system gestures become mouse events. They help support non-ink-enabled programs by bridging the gap between pen input and mouse input. Application gestures generate input intended for ink-enabled programs. Traditional programs can still benefit from some of the application gestures through the Input Panel.

What if a program wants a gesture not supported in either the system or application gesture tables? There are several options. When a desired gesture corresponds to a supported letterform (say an X to mark an object for deletion) a program could collect ink and use a RecognizerContext to watch for this command. As a command, the gesture would need to be erased to avoid confusion with ink as data.

But what if a desired gesture is not built in and has no corresponding letterform? Support for such a feature could be provided by creating a custom gesture recognizer, which could either be used by itself for total control of recognized gestures, or used to supplement the gestures recognized by the Microsoft-provided gesture recognizer.

Ink Data

As I mentioned earlier, the Tablet PC development team decided to treat ink as its very own data type. Ink created by an ink-enabled program is its own portable graphic format called Ink Serialized Format (ISF). This format can be serialized and saved to disk in either binary format or XML, and it can be placed on the clipboard. Ink can also be Web-enabled (and also made accessible to non-ink-enabled programs) when converted to a GIF with embedded ISF data.

The key managed code container class for holding ink data is the class Ink. Both the InkCollector and the InkOverlay classes hold an Ink object, which manages all of the collected ink data. The Ink class provides the primary support for serializing ink and for supporting the clipboard.

The Ink class serves up ink persistence through a Save method. Despite its name, this method does not write data to disk, but rather converts ink data into a byte array. Your program must take this byte array and copy it to disk. The class also supports a Load method, which copies a byte array—one created from a previous call to Save—into an empty Ink object.

ISF data is available in four different encodings; in a sense, ISF is four formats masquerading as one. One encoding is a simple ISF stream. A second encoding is a GIF image with a simple ISF stream embedded as metadata. This second encoding makes an ink image available to applications that are not visibly ink-enabled on a Web page, for example. Ink-enabled programs can also access the simple ISF stream inside the GIF. The third encoding is a simple ISF stream stored as base-64 data in an XML-compatible manner. The fourth encoding is a GIF similarly stored as base-64 data.

Ink data can be transferred between programs via the clipboard. Data sharing has always been an important feature of Windows; the creation of a new data format raises the issue of how to allow that format to be shared. At the most basic level, one or more clipboard formats need to be supported (see Figure 11).

Figure 11 Supported Clipboard Formats for Ink (Close Figure 11)

Name | CF_ Flag | Description
HTML Format |  | Supports more than a single Ink object in a format that is compatible with both ink-enabled and non-ink-enabled programs
Bitmap | CF_BITMAP | A device-dependent bitmap, for non-ink-enabled programs
Device Independent | CF_DIB | A device-independent bitmap, for non-ink-enabled programs
Enhanced Metafile | CF_ENHMETAFILE | Draw ink into an enhanced metafile, which can be drawn by non-ink-enabled programs
Ink Serialized Format (ISF) |  | Native ink format
Metafile | CF_METAFILEPICT | Draw ink into a metafile, for use by non-ink-enabled programs
Sketch Ink |  | An embeddable OLE object containing ink that is expected to hold a drawing
Text Ink |  | An embeddable OLE object containing ink that is expected to hold handwritten text that may sub-sequently be converted to recognized text

When ink is copied to the clipboard, the possible recipients for that data are of two types: programs that are ink-enabled and programs that are not. An ink-enabled program puts a range of different data types on the clipboard. When a user pastes an Ink objects to an ink-enabled program, the recipient can request ISF from the clipboard. And when the paste operation is performed to a non-ink-enabled program, a variety of alternative graphic formats, such as bitmaps and metafiles, allow the recipient to display the ink.

The Ink object supports the creation of embeddable OLE objects. This is a clever approach because it allows ink to be incorporated into applications that are otherwise not ink-enabled. In effect, these embeddable objects allow any OLE-compatible program to host ink-enabled objects. For the present, there are two basic types of embeddable OLE Ink objects: sketch ink and text ink. Sketch ink is expected to contain a drawing; text ink is expected to contain data that may be converted to text via an ink-compatible recognizer.

The Stroke Class and the Strokes Collection

Inside the Ink class is the Strokes property, which holds a collection of Stroke objects. A stroke is a set of points generated by the movement of a pen and can contain thousands of points. But a stroke is more thank that; it can represent other data such as Pressure and Drawing attributes. When converting ink to text, the US-English recognizer appears to prefer fewer long strokes to many short strokes. That's why the recognizer appears to get more accurate results when recognizing cursive text than block-letter text.

As a collection class, the Strokes class supports the basic operations expected of any collection class. For example, the Count property returns the number of elements, and the Add and Remove methods work as expected and individual strokes can be accessed by index (0 to Count-1). The Strokes class supports ink-specific operations, as well. Some methods apply a coordinate transform to the stroke data—scaling, shearing, rotating, and scrolling (or translating). Other methods change the appearance of ink when drawn or perform tests on stroke data.

Contained within a Strokes collection is a set of references to individual Stroke objects. As the pen moves to create a stroke, intermediate data known as Packets are delivered to an Ink object. The resulting stroke data is extracted from the packets.

The Stroke class makes it possible to perform several types of operations on stroke data. You can modify individual points, test whether a mouse-click (or pen-tap) touches a stroke, and apply a coordinate transform to an individual stroke. You can compare one stroke to another stroke for the intersections or find the places that a stroke intersects with itself through the SelfIntersections property.

The Stroke class supports copying the contents of a stroke object to an external Point array. This data is available in several forms, including as a polyline array and a Bezier array. The former can be drawn with a call to a DrawLines (or Win32 Polyline) method; the latter can be drawn with a call to a DrawBeziers (or Win32 PolyBezier) method. The benefit of a Bezier curve is that rough edges in a stroke are gently smoothed out.

Appearance and Display of Ink

Built into the InkCollector and InkOverlay classes is support for the collection and drawing of ink. The appearance of ink can be controlled by modifying various properties. The DefaultDrawingAttributes property holds a reference to a DrawingAttributes object. Changes to this object are reflected in changes to all the strokes in the current Ink object. Ink can be set to be drawn with a round or rectangular pen tip and with a given color and width. Ink can even be drawn with a Boolean raster operation to change the appearance of ink based on the color of a window (XOR, AND, and so on).

A Renderer object is also built into the InkCollector and InkOverlay classes. This object draws ink as a pen is moved, and it also provides automatic redraw of ink in response to a Paint event (or Win32 WM_PAINT message). In addition, an ink-enabled program can call on this object to redraw any combination of individual strokes. A Renderer object can draw onto a bitmap or into a metafile, and it can also be used to create hardcopy output of ink on a printer.

Figure 12 shows the containment relationship between some of the key ink-related objects for a window that uses an InkCollector. The InkCollector itself is connected to a portion of a window, as defined by the window input rectangle. The InkCollector contains an Ink object, which itself contains a Stroke collection. The Stroke collection, in turn, contains a set of Stroke objects, one for each user stroke. Outside of the Ink object is a Renderer object, which provides the drawing support for ink.Tablet PC Development and Test Systems

Tablet PC development requires a Windows XP-compatible compiler. In order to build native code executables, Visual Basic 6.0 or Visual C++® 6.0 work just fine. For building managed code executables, you need Microsoft Visual Studio® 7.0 or later (or any .NET-compatible compiler).

In addition to the compiler, you may need to install an SDK. Two SDKs support Tablet PC development; the one you need depends on the type of application you are building. For ink-enabled applications, install the Tablet PC SDK; for speech-enabled applications, install the Microsoft Speech SDK.

Test ink-enabled programs using one of three system configurations: Tablet PC operating system on a Tablet PC system, Tablet PC operating system on a standard PC, and Windows XP Professional with Tablet PC SDK. The ideal test system is obviously a Tablet PC, since it comes equipped with the hardware and software that you need. The other two do provide an easier and possibly lower cost point of entry.

Testing using a standard PC with the Tablet PC operating system is available to MSDN® subscribers. This configuration serves up most software-based features, including handwriting recognizers and Tablet PC applications. Obviously, the limitations are hardware-dependent. For example, screen display rotation is not supported. The mouse can simulate pen input, but the user experience is not going to be the same as on a Tablet PC.

If you look at the installation disks for the Tablet PC operating system, you will see that Disk 1 of the Tablet PC Edition is the same as Disk 1 of Windows XP Professional. The Tablet PC-specific files are on Disk 2. You must make sure to enter the product key for Windows XP Tablet PC Edition. If you enter a Windows XP product key, then Windows XP installs without the Tablet PC enhancements.

The test environment that is the easiest to set up, but delivers the fewest features, is available by installing the Tablet PC SDK on Windows XP Professional. This configuration supports basic ink collection, which means that most of the sample programs in the SDK can run. With the recently released recognizer pack, you can now have recognition engines on machines running other OSs for dev purposes. But none of the handwriting recognizers are installed. And none of the accessories, such as Windows Journal, are available in this configuration.

The ink-enabling components themselves are written as a set of automation-enabled ActiveX controls; they are, in short, written in unmanaged code. This means you can build ink-enabled applications using your favorite COM-compatible language: Visual Basic 6.0; Visual C++ 6.0; and the various C/C++-compatible APIs including Win32®, MFC, ATL, and WTL. My focus is on managed code development, but for each of the managed code classes, an unmanaged code equivalent is available.

Figure 12 Containment Relationship of Ink
Figure 12 Containment Relationship of Ink

An important feature provided by a Renderer object is support for coordinate conversion. Ink is collected in a set of coordinates known as ink space, which is fixed as HIMETRIC units. In this mapping mode, each logical unit is set to .01 millimeters. By contrast, mouse input always arrives as pixel coordinates. The default drawing coordinates system is also pixels (GraphicsUnit set to Pixel in managed code, and in Win32 the mapping mode is set to MM_TEXT). Figure 13 summarizes the available coordinate conversion functions provided by the Renderer class.

Figure 13 Coordinate Conversion Methods in Renderer Class (Close Figure 13)

Method | Description
InkSpaceToPixel | Converts a single point from ink space to pixel coordinates
InkSpaceToPixelFromPoints | Converts an array of points from ink space to pixel coordinates
PixelToInkSpace | Converts a single point from pixel coordinates to ink space
PixelToInkSpaceFromPoints | Converts an array of points from pixel coordinates to ink space

The Tablet PC supports three distinct coordinate mapping schemes, two of which are provided in the Renderer (see Figure 14). The two coordinate mapping schemes supported by the Renderer change the way ink is drawn, without changing the underlying ink data: the object and view mappings. The third mapping scheme, stroke mapping, modifies the actual coordinates stored by individual strokes. Stroke mapping is implemented by the Stroke class and also by the Strokes collection class to modify all the strokes in a given collection.

Figure 14 Coordinate Mapping Supported by Tablet PC (Close Figure 14)

Name | Class | Methods | Description
Object | Renderer | GetObjectTransform SetObjectTransform | Modifies coordinate mapping, leaves ink and pen width unchanged
View | Renderer | GetViewTransform Move Rotate Scale SetViewTransform | Modifies coordinate mapping, changes ink and pen (cursor) width
Stroke | Stroke and Strokes collection | Move Rotate Scale ScaleToRectangle ScaleTransform Shear Transform | Changes data in underlying ink strokes

When drawing pictures or creating annotation, the storing and retrieving of ink might be all that is needed. When program data must be converted to text, an additional step is required: the analysis of ink and the conversion (or recognition) of ink. This is the final item that a programmer needs to understand to build an ink-enabled program.

Analyzing Ink

The freedom to mix text and pictures presents a challenge for developers. If a user wants to edit the hand-drawn text as a set of characters (inside an edit control, for example), the program must convert the ink to text. But first a program must distinguish pictures from text.

The Tablet PC ink libraries provide the Divider class to do this. This object uses data in a Stroke collection to help it decipher what is text and what is picture. The analysis is actually performed by the Divide method, which returns an object of type DivisionResult. That class, in turn, returns a collection of type DivisionUnits, which contains one or more DivisionUnit objects, each holding data with one of four types, as defined in the InkDivisionType enumeration:

enum InkDivisionType {
    Segment = 0,
    Line = 1,
    Paragraph = 2,
    Drawing = 3
};
Three of the ink division types are text: Segment, Line, and Paragraph. For each of these types, the ToString method returns the associated text string. The fourth division type, Drawing, is a set of strokes that contain non-text ink. The associated strokes are stored in a Strokes collection.

Converting Ink to Text

Converting ink strokes to text requires a recognizer, which runs only on Windows XP Tablet PC Edition or on a system with the Tablet SDK installed. The SDK for the Tablet PC allows ink to be drawn and saved on systems running Windows XP Professional (or Windows 2000 with SP2), but the SDK does not provide a recognizer for anything other than a development machine.

Handwriting recognizers are language specific. To be more precise, recognizers are identified both by primary language and locale. The Recognizer class contains a Languages property, which has an array of LCID (locale-identifiers) values that identify the languages a given recognizer supports. The recognizer uses a locale-specific (not language-specific) dictionary to improve its results. Thus there are US English (locale = 0x409) and UK English (locale = 0x809) recognizers.

Multiple recognizers can be installed on a single Tablet PC. To help organize available recognizers, the ink library provides a collection class named Recognizers. A program can let the system provide a default recognizer, or it can select a specific one available from the Recognizer collection.

Microsoft provides a number of handwriting recognizers, some of which come with the Tablet PC; others are available for download and installation as part of the Microsoft Tablet PC Recognizer Pack. Each recognizer generally supports a single language, although support for several related languages is possible. As of this writing, Microsoft provides nine handwriting recognizers in the Recognizer Pack: Chinese (Simplified), Chinese (Traditional), English (UK version), English (US version), French, German, Korean, Japanese, and Spanish.

Third parties can create custom recognizers. The handwriting recognizer interfaces are public and documented in the Tablet PC SDK. Custom recognizers could be used to support other languages or sets of specialized symbols such as musical notes or architectural or engineering symbols.

Once a recognizer has been selected, text recognition services can be made available by creating an instance of the Recognizer object—or, more precisely, of a RecognizerContext. Call the CreateRecognizerContext method on the Recognizer class; and it will return an object of type RecognizerContext.

A RecognizerContext provides an opportunity for an ink-enabled program to give hints to the recognizer. The selection of a recognizer by available language/locale leads to the most important hint—a dictionary of common words. The set of words in the dictionary helps improve the recognizer's accuracy.

A RecognizerContext contains three properties to improve text recognition: Factoid, Guide, and WordList. A Factoid indicates the expected type of data (see Figure 15). Some Factoids indicate that numeric data is expected; others suggest that a date, time, or telephone number is available. This level of precision assumes that a program has gathered a very precise set of pen strokes from a well-defined input area.

Figure 15 Supported Tablet PC Factoids (Close Figure 15)

NONE | DIGIT | FILENAME | HIRAGANA
DEFAULT | NUMSIMPLE | UPPERCHAR | KATAKANA
SYSDICT | CURRENCY | LOWERCHAR | KANJI_COMMON
WORDLIST | POSTALCODE | PUNCCHAR | KANJI_RARE
EMAIL | PERCENT | JPN_COMMON | BOPOMOFO
WEB | DATE | CHS_COMMON | JAMO
ONECHAR | TIME | CHT_COMMON | HANGUL_COMMON
NUMBER | TELEPHONE | KOR_COMMON | HANGUL_RARE

A second property in the RecognizerContext is the Guide property, a structure of type RecognizerGuide. This property provides a set of coordinates (in ink space) that describe the size of the drawing area, the size of any drawn guide box, along with row and column information a program supplies to assist a user. The recognizer can use these coordinates to determine the relationship of strokes to available drawing area, thereby improving certain types of recognition.

The third property in the RecognizerContext is the WordList property. This identifies an application-specific set of words to supplement the system dictionary. The Factoid property controls whether the system dictionary is used (when set to SYSDICT), whether the word list is used (when set to WORDLIST), or when neither is used (NONE).

TicTacToe: An Ink-Enabled Sample

The Tablet PC Platform SDK offers a nice set of samples that represent a starting point for your Tablet PC programming. To supplement them, I wrote TicTacToe (see Figure 16).

Figure 16 Ink-Enabled TicTacToe
Figure 16 Ink-Enabled TicTacToe

While it's a simple game, seasoned developers can tell you that outward simplicity often hides inner complexity. And sometimes, as with my game, the problem involves finding the best match between the elements of a programming interface and the real world. I arrived at the final design after exploring other designs and writing some prototype code. A review of my discarded alternative designs reveals as much about ink programming as the one that I eventually adopted.

InkEdit Controls

My first approach involved using an InkEdit control provided by the Tablet PC libraries. Built-in controls are a good place to get rich features with a minimum amount of coding. UI elements often appear simple, but the myriad of details required often make this portion of code inherently complex. Offloading the work to a built-in control helps to simplify your project.

The InkEdit control is a rich text control that has been enabled to support ink input, which then gets converted to text. By contrast, the InkPicture control is for non-text ink. Since any TicTacToe application must allow players to enter two letters, I decided to try using the InkEdit control.

The first version of my game used nine InkEdit controls. InkEdit controls are primarily built to accept and display multiple characters, and even multiple lines, of text. An edit control is less usable when you only want to accept a single character, which is what my program required. It also does not work well when you want the character drawn in a particular way, which in my case meant having an X or an O occupy the whole client area of a single InkEdit window. I experimented with increasing the font size and playing other tricks. As I increased the font size, I also increased the chances that entered text would get scrolled outside the viewing area.

Also, I wanted the X or O to be centered both horizontally and vertically in the game square. But the InkEdit class tries to align text top-left. I could probably have made accommodations for this, but it occurred to me that the appearance of Arial or Times New Roman characters had the wrong feeling for a game that I used to play with crayons on scratch paper.

The text recognition worked well, but for my purposes it worked too well. A user could write anything in a game square, which meant that I would have to validate the input, requiring more code.

At the start, I thought I could adjust for these shortcomings, but in the end I decided to try another approach.

A Full-Screen InkCollector

My next attempt was to ink-enable my program's main window. Because this was a managed code program, the main window was an instance of the Form class. The most important thing I needed was a blank canvas to draw my game grid, and the ability to connect an InkCollector object to the window to retrieve pen events, manage the ink, and convert the input to the required game input.

The results of this experiment demonstrate a key design element of ink input. All the ink-as-text seen by an InkCollector is expected to be part of a continuous text block. So when my program went to retrieve the game input, it received a string that contained a set of X and O characters that were clumped together in a single string. For example, on the third move of a game, the string "XOX" might be returned. While this may have been correct for the letters that were being written, this string excluded all information about the specific game squares that were being played. Location information for text entered was lost because the recognizer did not provide any place holders for empty squares.

Because my game needed various functionalities that were quite different from what could easily be achieved with built-in controls, I decided that I would have to build more custom elements than I had originally anticipated.

Nine InkCollectors

My final design used nine windows and nine InkCollectors. I used the panel control for each game square because, like a form, a panel provides a blank customizable canvas. While I used a recognizer to determine what had been drawn in a game square, I did not replace the ink on the display screen with a text character. Keeping the ink as ink retained the flavor of the game as experienced when played on paper.

There are other alternatives that I did not try. One would be to use nine InkPicture controls. This would have allowed me to retain the ink feel, with perhaps less work on my part. A second alternative would have been to have a single input window with nine InkCollectors. Multiple InkCollectors can be connected to a single window (as set with the SetWindowInputRectangle method), provided no overlap exists between input rectangles.

Ink-enabling a custom input window is useful when you need multiple input areas. When a program creates such a window, it has the option of leaving the ink as ink, or converting the ink to text and displaying the results as text. After all, a data entry form might consist of both ink and text windows. Ink windows would be appropriate for signatures and hand-drawn diagrams. Text windows would be appropriate for other entry fields..

My program owns the custom input windows, which required a bit more work than I had hoped. But it provided the greatest amount of control over the input, processing, and display of game data.

In the TicTacToe game, all ink input is handled by InkCollector objects. The UI consists of nine white panels arranged on a black form, which produces the effect of having lines between game squares. Each panel has its own InkCollector object.

Assembly and Namespace References

For ink programming with managed code, you need to reference the Microsoft.Ink.dll, which ships with the Tablet PC SDK. You reference the proper namespace like this:

using Microsoft.Ink;

The Ink assembly that ships with the Tablet PC SDK is an update to the version that ships with the first generation of Tablet PCs. If your Tablet PC programs reference the updated assemblies that ship with version 1.5 of the SDK, you must also ship updated assemblies with your Tablet PC programs, or you must ensure that all of your users already have the updated assemblies installed. For details on distributing the updated merge modules, refer to Knowledge Base article 331956 at Tablet PC Object Model Components Leak Resources.

The version of Microsoft.Ink.dll that ships with the first generation of devices is version 1.0.2201.0. The assembly that comes with the SDK version 1.7 (which is available online as of October 2004) has a version number of 1.7.2600.2180. (The newer version is installed on development systems during the installation of the Tablet PC SDK.)

On the subject of versions of assemblies, the first generation of Tablet PCs ships with the .NET Framework 1.0. This is not a major issue, but when building Tablet PC programs with Visual Studio .NET 2003, you must be sure to set the runtime version to 1.0; otherwise, the default is the .NET Framework 1.1.

TicTacToe Initialization and Cleanup

Figure 17 shows the initialization of the InkCollectors when the form receives a Load event. References to the nine game square panels are stored in the m_panels array, which parallels the nine InkCollectors stored in the m_inkcoll array. The two arrays allow each set of objects to be put into loops when all objects must be searched or operated on in other ways.

Figure 17 Load Event Handler for TicTacToe Form (Close Figure 17)
private void FormMain_Load(object sender, EventArgs e)
{
   // Put panels into array.
   m_panels[0] = panel1;
   m_panels[1] = panel2;
   m_panels[2] = panel3;
   m_panels[3] = panel4;
   m_panels[4] = panel5;
   m_panels[5] = panel6;
   m_panels[6] = panel7;
   m_panels[7] = panel8;
   m_panels[8] = panel9;

   for (int i = 0; i < 9; i++)
   {
      m_inkcoll[i] = new InkCollector(m_panels[i].Handle);
      m_inkcoll[i].Enabled = true;
      m_inkcoll[i].DefaultDrawingAttributes.Width *= 3;
      m_inkcoll[i].DefaultDrawingAttributes.Color = Color.Red;
      m_inkcoll[i].Stroke +=
         new InkCollectorStrokeEventHandler(InkCollector_Stroke);
   }

   // Record original size of panels to help in scaling.
   m_sizefPanelStart = new SizeF(panel1.ClientRectangle.Width,
      panel1.ClientRectangle.Height);
}

Each InkCollector object generates various ink-specific events. As each InkCollector object is created and initialized, my initialization code connects an event handler for the Stroke event to each InkCollector. The Stroke event is triggered when ink has been collected from a sequence of user actions that include touching the screen with the pen tip, moving the pen, and removing the pen tip from the screen (described later).

The last action taken by the load event handler is to record the size of the form's client area. My game uses the initial size when the size of the form changes, when the ink inside each panel must be scaled to maintain a consistent look no matter how large or small the individual game squares become. I'll describe the details of this operation later, in the discussion of the Resize event.

A key benefit of managed code is support for automatic garbage collection for managed memory. There are times, however, when objects must be explicitly freed. Ink objects are implemented in native code as COM objects, and so care must be taken to release unmanaged resources in a timely fashion. These include the InkCollector object.

Figure 18 shows the Dispose method for my form. Notice that I first removed the event handler on the InkCollector and then disposed of the InkCollector itself. This is described in more detail in Knowledge Base article 331956, which I referenced earlier.

Figure 18 Dispose Method for TicTacToe (Close Figure 18)
protected override void Dispose(bool disposing)
{
   if (disposing)
   {
      if (components != null)
      {
         components.Dispose();
      }

      // See Knowledge Base Article #331956
      for (int i = 0; i < 9; i++)
      {
         if (m_inkcoll[i] != null)
         {
            // Remove event handler.
            m_inkcoll[i].Stroke -=
               new InkCollectorStrokeEventHandler(
               InkCollector_Stroke);

            // Clean up ink collector object
            m_inkcoll[i].Dispose();
            m_inkcoll[i]= null;
         }
      }
   }

   base.Dispose( disposing );

}  // method: Dispose

InkCollector Events

The InkCollector does all the work of capturing the input related to ink entry, in much the same way that an edit/textbox control captures keyboard input. For each of these controls, events are generated by the control in response to changes in the input state.

Some of the input events for the InkCollector are surrogates for mouse input. Others are specific to ink input, such as Gesture, NewInAirPackets, NewPackets, Stroke, and SystemGesture. The two "packet" events let you know when raw input is being received; it's useful, but not for my game.

The two "gesture" events let you know when an application or system gesture has been detected. I considered using gestures in this game, but was not able to find a gesture that matched the X (although the O did have a related gesture).

The final ink input event is the Stroke event. A stroke corresponds to all of the points collected from the time the user pushes down with the pen, through all movements of the pen and until he lifts the pen.

My game looks for the Stroke event to detect when the user has generated ink. The nine InkCollector objects share a single Stroke event handler, named InkCollector_Stroke. I figured out which InkCollector sent the event by looping through the InkCollectors in the array named m_inkcoll array and comparing to the event handler's sender parameter. The result is iWindow, an index to an InkCollector in the array of InkCollectors. The same index also refers to the associated panel that the InkCollector occupies.

I created a bit mask for the current square by shifting the value 1 left by the window number, which resulted in iWindow. The value of iWindow is added to m_SqX when a player enters an X, and to m_SqO when a player enters an O. This approach makes it easy to search for a winning combination. Figure 19 shows the table of winning values that my program uses.

Figure 19 Winning Results (Close Figure 19)
// Table of winning square combinations
// Assumes the following layout of squares
//
//   1 | 2 | 3
//  ===|===|===
//   4 | 5 | 6
//  ===|===|===
//   7 | 8 | 9
private static int [] m_aiWins =
{
    0x0007, //   123
    0x0038, //   456
    0x01C0, //   789
    0x0049, //   147
    0x0092, //   258
    0x0124, //   369
    0x0111, //   159
    0x0054, //   357
    0
};

// Check for a winning combination
// from table of possible wins.
private bool CheckForWin(int sqInput)
{
    for (int i = 0; m_aiWins[i] != 0; i++)
    {
        if ((sqInput & m_aiWins[i]) == m_aiWins[i])
            return true;
    }

    return false;
}//method: CheckForWin

Once stroke data has been entered, the final step in my game is to recognize text (see Figure 20). The game looks for either X or O, and ignores anything else. My game uses the simplest possible approach to text recognition: it calls the ToString method for the Strokes collection in a single line of code, and then my program can detect what text has been written.

Figure 20 Stroke Event Handler Method for TicTacToe (Close Figure 20)
private void InkCollector_Stroke(object sender,
   InkCollectorStrokeEventArgs e)
{
   if (m_bGameOver)
   {
      MessageBox.Show("Game over — select " +
         "Game->Restart for a new game", strAppName);
      return;
   }

   int iWindow = -1;
   for (int i = 0; i < 9; i++)
   {
      if (sender.Equals(m_inkcoll[i]))
      {
         iWindow = i;
      }
   }

   // Create bit-mask for current square.
   int iSquare = 1<<iWindow;

   // Check whether square is available.
   if ((iSquare & m_sqFree) == 0)
   {
      MessageBox.Show("Square already occupied", strAppName);
      e.Cancel = true;
      return;
   }

   // Recognize input in window number iWindow.
   string str = m_inkcoll[iWindow].Ink.Strokes.ToString().ToUpper();
   if (str == "X")
   {
      // Set flags.
      m_sqX += iSquare;
      m_sqFree -= iSquare;

      // Check for winning.
      if (CheckForWin(m_sqX))
      {
         MessageBox.Show("Game over — winner is 'X'", strAppName);
         m_bGameOver = true;
      }
   }
   else if (str == "O")
   {
      // Set flags.
      m_sqO += iSquare;
      m_sqFree -= iSquare;

      // Check for winning.
      if (CheckForWin(m_sqO))
      {
         MessageBox.Show("Game over — winner is 'O'", strAppName);
         m_bGameOver = true;
      }
   }
   else
   {
      if (m_inkcoll[iWindow].Ink.Strokes.Count > 1)
      {
         MessageBox.Show("Cannot recognize input - try again.",
            strAppName);
         m_inkcoll[iWindow].Ink.DeleteStrokes();
         m_panels[iWindow].Invalidate();
      }
   }
} // method: InkCollector_Stroke

That one line of code taps into the ink recognition, but there is much more below the surface. The Tablet PC recognition engine uses a static neural network to convert the strokes to text, which means that recognition is accomplished by assigning probabilities to different potential results. Recognizers are language specific, and rely on a set of dictionaries of possible good results. My one line of code shows how easy it is to get ink converted into text.

Programs that need more sophisticated control over text recognition can get that control. Such needs are addressed with a set of recognizer classes which include the InkRecognizers collection, which lets a program get a list of the available recognizers. For example, a program that accepted multi-language input could choose a recognizer based on its support for a specific language.

Once a specific recognizer has been selected, the InkRecognizerContext class owns the actual recognition work. My program did not have to create an instance of this class, because the ToString method in the Strokes class did it for me. A program can access the results of the recognition engine by creating an instance of this class, and then asking for a list of alternative recognition results and the degree of confidence calculated for each.

As mentioned earlier, text recognition requires the assistance of an installable component known as a handwriting recognizer. Handwriting recognizers ship with the Tablet PC operating system, but not with the Tablet PC SDK. For this reason, this game requires an actual Tablet PC—or a standard PC that has Windows XP Tablet PC Edition installed.

What's Next

Before you write your first real-world Tablet PC program, I suggest you spend time using a Tablet PC to do work in the real world. This gives you a sense of all the functionality of the input devices. A Tablet PC is not just a PC without the familiar input devices. It is a new form factor, which allows a computer to go places that a computer traditionally could not go. It allows for new types of interactions between people and machines. And it allows for the creation of new types of applications. For anyone who has been thinking the use of computers can be expanded, the Tablet PC is worth serious consideration. For more up-to-the-minute information on developing for the Tablet PC, visit the Tablet PC Developer Center.

Paul Yao is a programming guru, specializing in Windows CE, and coauthor of the first book published on Windows programming, Programmer’s Guide to Windows (Sybex, 1987). His company specializes in training programmers in Windows and Windows-related technologies. Visit his Web site at www.paulyao.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Mobility: Optimize Your Pocket PC Development with the .NET Compact Framework

Mobility

Optimize Your Pocket PC Development with the .NET Compact Framework

Dave Edson and John Socha-Leialoha
This article discusses:
  • Tips for making programming in the .NET Compact Framework environment easier
  • Tricks for making your Pocket PC-based apps run faster
  • Important .NET Compact Framework classes
  • .NET Compact Framework windowing issues
This article uses the following technologies:
.NET Compact Framework, Win32, C#
Code download available at: NETCompactFramework.exe (270KB)
Contents
Essential Goodies and Our Test Bench
Bad, Wicked Dialogs
Chilling Out the Context Menu
Designable Custom Controls
Optimization
Getting Resourceful
Load Only What You Need; Chuck What You Don't
Putting It All Together—the TabControlEx Class
Adding Toolbar Buttons to TabControlEx
Conclusion

P
rogrammers get away with a lot today. With .NET, at last there is a framework that allows for really easy application development. Memory is managed with so little effort from the developer that we can easily abuse it. All the neat controls, forms, windowing systems, and GDI+ capabilities make it so easy. Easy, that is, until you have to write for the Pocket PC, a 200MHz platform with achingly slow graphics (read: no acceleration), no hard drive, and only 32MB of RAM, of which you can really only use about 2 or 3MB. Add to the hardware limitations the sinking feeling you get when you dive into the .NET Compact Framework and discover that some of your favorite features are not implemented. Suddenly your dream of writing that squeaky clean C# killer app turns into a nightmare.

Don't despair. The .NET Compact Framework really can be used to write great code and great applications. As long as you take a few things into consideration and are willing to bend a rule or two, you can have your performance cake and eat it too. We have written many applications using C# for the Pocket PC, resulting in polished, slick apps that are small and fast.

This article is divided into two main sections: goodies and optimization. In the goodies part, we will go over some neat tricks that make our lives as programmers easier, using the .NET Compact Framework. In the optimization section, we will discuss techniques to increase performance, decrease load time, and decrease memory footprints. Sample code will be provided to help you turn your app into something small and fast.

Essential Goodies and Our Test Bench

When you fire up Visual Studio® .NET to build our initial application, you'll need to make sure to add these lines of code in the main form's constructor, right after the call to InitializeComponent:

#if DEBUG
    MinimizeBox = false;
#else
    MinimizeBox = true;
#endif

This lets you actually close the app rather than just minimizing it, which can come in very handy when testing outside the debugger. The next important task is to unearth the full power of the device for which you are programming.

The .NET Compact Framework is certainly true to its name—it sure is compact. Thankfully, interop services are included, and as a result you can call any of the Win32® CE APIs on the device. Since many of these APIs require a window handle, you need to get one. Until version 2.0 of .NET Compact Framework is released (which resurrects the Handle property), use these lines of code to get the hWnd:

control.Capture = true;
IntPtr hwnd = Win32.GetCapture();
control.Capture = false;
See the sidebar "The Windows CE Namespace" for the Win32 class.

Now that the Win32 API is back in your grasp, there is another very handy use of that hWnd. You can subclass your own window with a Win32 DLL written with eMbedded Visual C++® 4.0 and play hardball. But there is something to keep in mind: the interop services do not keep a DLL loaded in memory. Deep inside the bowels of P/Invoke is a LoadLibrary/FreeLibrary pair. If you want your DLL to subclass your app (or, excuse me, any app on the system), that DLL needs to stay loaded in memory until you are ready to let it go. There is a nice solution to this problem: just call LoadLibrary on the DLL when you start, and then call FreeLibrary on the DLL when you are finished. The reference count will be bumped up a notch, and P/Invoke's LoadLibrary/FreeLibrary pair will become benign.

Figure 1 MSDN Test Bench
Figure 1 MSDN Test Bench

A lot of different topics are going to be covered in this article. We have written a Test Bench program (shown in Figure 1) that integrates many of the concepts discussed. It incorporates the Win32 class covered in the sidebar "The Windows CE Namespace" as well as the BaseControl class discussed later in this article.

Bad, Wicked Dialogs

Users and developers unaware of dialog behavior in the .NET Compact Framework can end up in La-La Land when they open a modal dialog. Dialogs in the .NET Compact Framework aren't really dialogs in the normal sense. Rather, they're simply new forms that appear on top of existing forms, but don't return control to the caller until you close the form. If you use code like this to display your dialog

  BadDialog test = new BadDialog();
   test.ShowDialog();
the dialog will appear as a top-level form. When the user switches away from your program, the fun begins. Eventually they want to get back to your app, and they go to the System | Settings | Memory | Running Programs page, where they see both your main app and the dialog box listed. If the user clicks on the top-level app instead of the dialog, the main app window will be displayed, but will be inactive as it is waiting for the dialog to be closed.

To see this happen, click the Bad Dialog button on the Test Bench application. Now go into the list of running programs from the Settings screen and activate the Test Bench application (see Figure 2). If you tap on any of the buttons, they'll just beep, as this window is disabled. You can't even close it. So your users respond in the only way they know how: a clicking fit (actually, a tapping fit on a mobile device). The way out of this pickle is to go back to the list of running programs and activate the Bad Dialog item. Just to pour salt into the wound, once you close the dialog, another application may pop up, since the z-order of the windows could have been changed while you were switching from program to program.

Figure 2 Confusing Information
Figure 2 Confusing Information

To fix these problems, as well as a few others related to the existence of multiple forms, we created a Dialogs class with a static member called ShowDialog. Here's what it does:

  • Hides the parent form in the Running Programs List by setting its caption to an empty string.
  • Sets the caption of the dialog to the same name used by the parent form, preserving the name you see in the Running Program List, no matter which dialog is topmost.
  • Traps the Activated and Closed events of the new dialog so you can keep track of which form is the "topmost" form.
  • Calls ShowDialog for the dialog.
  • Activates and brings the "topmost" form to the top whenever any of the "parent" forms is activated.

This whole process sometimes results in another application being visible momentarily when you close a modal dialog, but other than that, everything works as a user would expect. You can see how this enhanced version of ShowDialog works by clicking on the Dialog Fix button in the Test Bench application. Notice that the caption on the dialog is the same as the main form, and only one form shows up in the Running Programs List.

Chilling Out the Context Menu

Tap and hold on a typical Pocket PC-based application, and you get those dancing dots followed by a context menu. To the user, it's divine. To the programmer, it's annoying because you don't receive the MouseDown event. This means that you don't have a chance to do some pre-context menu operations, such as disallowing the context menu in the first place or changing a selection. Getting around this behavior involves playing a few tricks on the runtime. Normally, you would just set the ContextMenu property of a control to associate it with a context menu for when a user taps and holds. In our solution, we don't set the ContextMenu property of the control; instead we mimic the code built into the Control class to handle the MouseDown event and display the menu. To make this as transparent as possible for the programmer, we developed a new base control class cleverly named BaseControl. It encapsulates a lot of the behavior we want available to our custom controls. See Figure 3 for a short list of these features. The key to banishing the default behavior and replacing it with our own is to override the ContextMenu property. This way the Control.ContextMenu property is still null, while BaseControl.ContextMenu contains the information we need.

Figure 3 The BaseControl Class (Close Figure 3)

Feature | Description
Handle property | Implements Handle property that is missing from .NET Compact Framework controls, allowing access to the window handle
Gdi | Provides support for using the Gdi class to draw; there are a number of methods and properties that help with this task
Buffered drawing | Supports drawing to an off-screen bitmap instead of the screen, then copying changes to the screen; helps eliminate flicker, but does require extra memory
Context menu | Allows you to receive the MouseDown event immediately when you have a context menu

Figure 4 shows the four methods of BaseControl that get the work done: OnMouseDown, CheckContext, ShowContext, and RecognizeGesture. Your own custom control can override the OnMouseDown method if you need to change a selection or do some other action. The CheckContext method calls OnBeforeContextPopup, which in turn fires a BeforeContextPopup event. The event handler then has a chance to change or cancel the context menu, or do other application-specific things. The RecognizeGesture method does the actual work of making the dancing dots appear. ShowContext displays the context menu.

Figure 4 Mimicking the Steps to Show a Context Menu (Close Figure 4)
protected override void OnMouseDown(MouseEventArgs e)
{
    base.OnMouseDown(e);
    CheckContext(e, this, this.Handle);
}

protected void CheckContext(
    MouseEventArgs e, Control control, IntPtr hwnd)
{
    // Tap and Hold: Overrides the built-in implementation to allow
    // you to receive MouseDown before the context menu is displayed.
    m_justDidContext = false;
    if (m_context != null)
    {
        // Give a chance to cancel the context menu
        // (or change the menu that we'll show).
        ContextPopupEventArgs eContext =
            new ContextPopupEventArgs(e.X, e.Y);
        OnBeforeContextPopup(eContext);     // Give subclass a chance to
                                            // cancel or change context

        bool fShow = false;
        if (eContext.Cancel == false)
            fShow = RecognizeGesture(hwnd, e.X, e.Y);
        if (fShow) ShowContext(control, eContext);
    }
}

protected void ShowContext(
    Control control, ContextPopupEventArgs eContext)
{
    if (m_context != null)
    {
        m_justDidContext = true;

        OnContextPopup(eContext);
        m_context.Show(control, new Point(eContext.X, eContext.Y));
        OnContextClose(eContext);
    }
}

protected virtual bool RecognizeGesture(IntPtr hwnd, int x, int y)
{
    SHRGINFO info = new SHRGINFO();
    info.cbSize = Marshal.SizeOf(info);
    info.ptDown = new Point(x, y);
    info.dwFlags = SHRG_RETURNCMD;
    info.hwndClient = hwnd;

    int cmd = SHRecognizeGesture(ref info);

    return (cmd == GN_CONTEXTMENU);
}
Figure 5 Context Menu Trick
Figure 5 Context Menu Trick

On the Test Bench program, tap the Context Menu button and you will see the screen shown in Figure 5. The top grey box control is a normally created control. Tap on it and the listbox underneath will show the relevant events. As explained, if you tap and hold, no mouse down event happens. The lower half of the screen uses a control that employs our tricks. This control behaves just like the normal control, except you get that oh-so-important mouse down event at the very moment the user touches the screen.

Designable Custom Controls

Designing custom controls for the .NET Compact Framework is not quite as easy as it is for the desktop. It would be nice to be able to use the designer to create a custom control, and then use the new control while designing your application. On the desktop this all works seamlessly because the code behind the desktop System.Windows.Form.Control object has all the designer support code built in. Microsoft.WindowsCE.Forms.Control doesn't have this code because it would eat up memory.

There is a little trick you can do to make this happen, however. When designing a new custom control, start with a form instead of a control. Now you can use all the designer tools to build your custom control. Of course, there is a price for this, but it isn't too high: when using the designer for your application (not the control), you cannot add your custom control to the designer's toolbox because you've created a form, not a control. Therefore, you will need to create an instance of the "control," using code like this:

MyControl = new MyCustomControl();
Controls.Add(MyControl);
   MyControl.Location = new Point(x, y);
   MyControl.Size = new Size(dx, dy);
   MyControl.Visible = true;

Notice that we're setting both the location and the size. When you are using a form as a control, the .NET Compact Framework will automatically set its size as appropriate for using the full screen. To get around this, you'll have to set the size yourself.

There is one little problem, however. In some cases, adding a custom control that is actually a form will cause extra flicker in the caption at the top of the screen, as it first changes to show the caption from the new form and then changes back. Getting around this involves undoing our trick. We need to change the base class back to a control again! To let the control flip-flop on whether it wants to be a form or a control, you can add a conditional compilation constant and modify the class definition for each custom control you create based on this form trick.

Figure 6 DESIGN Conditional Constant
Figure 6 DESIGN Conditional Constant

Figure 6 shows the conditional constant called DESIGN, which is only set in the debug version of the compilation. Modify the class definition for your custom control (form) to look something like this:

   // DESIGN is only set in Debug builds
   #if DESIGN
       public class MyCustomControl :
          System.Windows.Forms.Form
   #else
       public class MyCustomControl :
          System.Windows.Forms.Control
   #endif

Figure 7 Custom Control
Figure 7 Custom Control

If the DESIGN constant isn't defined (which it shouldn't be for the release mode), your custom control will be compiled as a control instead of as a form. You can keep the DESIGN constant defined for debug mode and test in that mode. But if you want to, you can always remove it for testing your code, then add it back in so you can design the form again. If you remove and add it, you might need to recompile your application before Visual Studio recognizes again that it can design the form.

In the Test Bench application, tapping on the Custom Control button brings up a dialog with the custom control in it (see Figure 7). If you go to the source code for the Test Bench, you will see the MyCustomControl class in the Tests folder, which you are able to edit from the designer (see Figure 8).

Figure 8 MyCustomControl
Figure 8 MyCustomControl

Optimization

Writing small, fast code is always a good idea, but it's critical when writing for the .NET Compact Framework. You just don't have a whole lot of memory or speed available. At design time, you need to think about what code will be executed and how your data structures are going to be used. Since memory is at a premium, it's important to avoid making copies of data. And most importantly, your code should only create data when it is needed, and discard it when finished.

Another issue to take into consideration is when to use .NET code and when to use Win32 code. There are times when it is much faster and easier to use Win32 code (really!). You will be glad to know that most of the time it is easier to use .NET code. Note, though, that we are not going to encourage you to return to the age of stone knives and bearskins just to eke a little bit better performance from your applications.

In a nutshell, the .NET Compact Framework rocks when it comes to memory management, event handling, object-oriented programming, user input, internationalization, and file I/O. The places where the .NET Compact Framework has the biggest challenges are graphics, window management, and resource management. Enormous improvements in speed and functionality may be gained by abandoning the Graphics object and going back to the Win32 GDI. When it comes to window management, .NET Compact Framework code can be tweaked to improve performance immensely. It just takes a long time to create windows. If you have a tab control with many controls across many pages, it can take eons for the tab control to initialize. This is because each and every control on every page is created at load time. Later on, we will present a replacement for the tab control class that is mainly .NET code with a sprinkling of Win32 calls that results in a tab control that loads blisteringly fast and dynamically loads or discards nonvisible pages.

One way to time your application's performance is with the Environment.TickCount property, as shown here:

   int t = Environment.TickCount;
   // <do your stuff here>
   t = Environment.TickCount - t;

Of course, nothing is that easy. Since a .NET executable is actually composed of Microsoft® intermediate language (MSIL) code, there is a just-in-time (JIT) compiler involved. The first time your function is called, it may take more time than in subsequent calls. Keep this in mind when running your tests. Also note that the emulator and the actual device can give very different opinions on the amount of time it takes to get a task done. Therefore, you will need to run all of your benchmarking tests on the actual device and not through the debugger.

Our first example harkens back to a May 1995 Microsoft Systems Journal article entitled "Grab Bag of Gotchas and Goodies for Programming in Windows." ExtTextOut is still, after all these years, the fastest way to splat text onto the screen. It can also draw with a transparent background, which labels in the .NET Compact Framework do not support (just try to put a label over a PictureBox and you'll see).

Figure 9 GDI Speed Test
Figure 9 GDI Speed Test

On the Test Bench program, run the GDI Speed Test. The first time you run the test, the JIT compiler will taint the results. On our device, the conventional .NET approach using DrawString took 51ms, while using the GDI method ExtTextOut took only 22ms. If you take the time to run the test a second time, you will see a tremendous improvement in speed, as shown in Figure 9.

Getting Resourceful

When it comes time to add images and text to your application, it's a good idea to use resources instead of hand-coding the text or using external files for images. Visual Studio .NET 2003 has all the tools you need to create string resources, but it lacks built-in support for managing image resources. For that you need to turn either to the ResEditor sample application that ships with Visual Studio .NET or to a third-party application. Our favorite is a free program called Resourcer for .NET that is available at www.aisto.com/roeder/dotnet. It's simple to use and comes with source code, so you can change it if you like.

The Test Bench program loads a bitmap resource into the form when you click the Resources button. Test Bench contains a class called MyResources that makes using these resources much easier. This is all you need to do to load an image:

m_image = MyResources.Icons.LoadBitmap("resTest");

You can create a string resource file simply by using the Add New Item dialog box to add a new assembly resource file. To group the resources in one place, we created a file called strings.resx in a folder called Resources. Double-click on the strings.resx file and you'll see a screen like the one shown in Figure 10. To load a string, you just do the following:

string s = MyResources.Strings["id"];
The Windows CE Namespace

To aid in writing code using the .NET Compact Framework, we've created a library of Win32 wrapper classes and .NET Compact Framework utility classes, shown in Figure A.

The current implementation of fonts in the .NET Compact Framework doesn't work very well. A glaring example of this is that you simply can't create an eight-point font on some devices. Our FontCache class takes all the pain away from fonts. It has one public method: CreateFont. There are three overloads of this method, as shown here:

FontGdi CreateFont(string Family, float Size, FontStyle Style)
FontGdi CreateFont(
    string Family, float Size, FontStyle Style, bool forceClearType)
FontGdi CreateFont(Font fromFont)
You have the option of forcing ClearType fonts even if ClearType is not selected in the Settings dialog of your device. In addition, when you ask for a font of a certain size, you get it. As you can guess, FontCache is a cache of fonts. That means that if you call CreateFont twice with the same parameters, only one instance of the actual font is created in memory. Tap the Font Test button in the Test Bench application and you will see fonts created via the Framework on the top, and fonts created with FontCache on the bottom.

The Gdi class is quite powerful. Its methods are shown in Figure B. If you use the properties of the Gdi object to change pens, brushes, and so on, then you do not need to worry about the whole SelectObject shenanigans from Win32 days. Typical usage of the Gdi object is done in the Paint event handler. The FontCache class is utilized in this example:

private void MainForm_Paint(
    object sender, System.Windows.Forms.PaintEventArgs e)
{
    using ( Gdi gdi = new Gdi (this))
    {
        gdi.TextColor = Color.Red;
        gdi.Font = FontCache.CreateFont ( "Tahoma", 12, FontStyle.Regular );
        gdi.ExtTextOut ( 100, 100, "Wardrobe Malfunction" );
    }
}

Figure B Gdi Class (Close Figure B)

Properties | Description
IntPtr ClipRegion | Current Win32 clip region.
Color BackColor | Current background color.
BrushGdi Brush | Current brush.
FontGdi Font | Current font.
PenGdi Pen | Current pen.
Color TextColor | Current text color.
bool Transparent | Sets the background mix mode to either transparent or opaque.
Methods | Description
Gdi(IntPtr hwnd)
Gdi(IntPtr hwnd, Rectangle clipRect)
Gdi(Control control)
static Gdi FromHdc (IntPtr hdc) 		Constructors. Allows you to create a Gdi object in a variety of ways.
void Dispose() | Disposes method to clean up resources.
PenGdi CreatePen(Color c)
PenGdi CreatePen(Color color, PenMode mode) 		Creates a PenGdi object of a specified color.
BrushGdi CreateSolidBrush(Color c) | Creates a BrushGdi object of a specified color.
bool DeleteObject(IntPtr obj) | Deletes an object. Returns true if successful; false otherwise.
void DrawClipped(string text, int width, int x, int y) | Draws text. If the text is wider than the width parameter (pixels), as much text as possible is drawn with ellipses ("...") at the end.
int DrawText(string text, Win32.RECT rect, Win32.DT format) | Exact same functionality as Win32 DrawText.
bool ExtTextOut(int x, int y, string text) | Exact same functionality as Win32 ExtTextOut. Draws a single line of text into the window without clipping or wrapping. This is fast.
bool ExtTextOut(int x, int y, int width, string text) | Same as above function, except centered between x and x+width.
int FillRect(Win32.RECT rect, BrushGdi hbrush)
int FillRect(int x, int y, int w, int h, BrushGdi hbrush)
int FillRect(Rectangle rect, BrushGdi hbrush)
int FillRect(Rectangle rect, Color color) 		Fills a rectangle with the specified color.
void DrawRectangle(Rectangle rect) | Draws a rectangle with the interior using the current brush and the edges using the current pen.
void FrameRect(Rectangle rect) | Draws the edges of a rectangle with the current pen, leaving the interior untouched.
BrushGdi GetStockBrush(Win32.StockObjects type) | Gets a stock brush from the Win32 GDI.
Size GetTextExtent(string text)
Size GetTextExtent(string text, int width, out int[] extents, out int maxFit) 		Measures the size of a string, in pixels, using the current font. The text that you want to measure and the maximum width that you'll allow are passed to the method as parameters. On exit, the extents parameter is set to an array of character positions for the string, and the maxFit parameter is set to the maximum number of characters that will fit in the width. The method returns the width and height of the string.
Size GetDrawTextSize(string text, int width, Win32.DT format) | Returns the size that DrawText will calculate for text with the currently selected font.
bool DrawLine(int x1, int y1, int x2, int y2)
void DrawLine(int x1, int y1, int x2, int y2, Color color) 		Draws a line.
bool Polygon(Point[] points) | Same as the Win32 Polygon API.
bool Polyline(Point[] points) | Same as the Win32 Polyline API.
IntPtr SelectObject(IntPtr obj)
PenGdi SelectObject(PenGdi pen)
BrushGdi SelectObject(BrushGdi brush) 		Same as the Win32 SelectObject API.
Figure A Win32 Classes (Close Figure A)

Class | Description
Win32 | A wrapper around a variety of functions in Win32
Registry | A collection of functions that make it easy to access and edit the Registry
Gdi | A high-speed replacement for the Graphics object
PenGdi | A type-safe wrapper for the GDI Pen Handle
FontGdi | A type-safe wrapper for the GDI Font Handle
BrushGdi | A type-safe wrapper for the GDI Brush Handle
FontCache | A turnkey font solution that creates and manages fonts with minimal system resource usage

As we mentioned before, using image resources requires the use of an external resource editor. If you're using Resourcer, you can select Add File from the Add menu, enter the name of the image resource to be added, then select the file. If you run the sample application and click on the Resource Test button, you'll see both a string and an image loaded from the resource file.

Figure 10 Adding a String Resource File
Figure 10 Adding a String Resource File

If you are writing a commercial application, there are many reasons to put your resources into a separate DLL. The most important is that you can offer various skins. If you can offer users a variety of appearances for the application, you can increase your potential customer base. We won't go into details of how to write a skinnable application, but we will assume that strings, bitmaps, and positional information all need to go into a DLL that can easily be swapped out with another that contains a different skin.

Dynamically loading a .NET DLL with the .NET Compact Framework and getting the resources out of it is not pleasant. Even if it were, you would be fast asleep by the time your resources were loaded. There is also an annoying behavior with .NET resources: if you load an image from a .NET resource, and then call Dispose, the image is gone for good. If you try to load it again from the .NET resource, the resource loader will give you back a valid Image object that won't draw itself. The only way to get that image to recache itself is to create a new instance of the resource loader (which takes time). So let's just walk away from those resources altogether and start from scratch.

Before we talk about loading bitmaps, we need to talk about rendering them. The Win32 BitBlt function still rules the world when it comes to speed—as long as you are using a device-dependent bitmap (DDB), that is. If you try to Blt out to a device-independent bitmap (DIB), you take a huge speed hit.

Inside the system directory of your Pocket PC device is a DLL that decompresses all sorts of image formats: JPG, PNG, and GIF, just to name a few. We take advantage of this functionality in a C++ DLL we wrote that can load an image resource from a DLL and return to us the information as a DDB. The DLL has two exposed functions:

HBITMAP LoadBitmapFromFilename (HDC hdc, LPCTSTR szFileName);
HBITMAP LoadBitmapFromResource (
    HDC hdc, DWORD dwResourceID, LPCTSTR pcszClass, HMODULE hModule);

The first parameter for both of these functions is a handle to device context (HDC). We will use our eponymously named Win32 class to get the HDC of the current form, and pass that into these functions. The functions then use the HDC passed in to figure out how to return a DDB to your application. The first function is easy. The second function can be cleared up with some code snippets. Here is the code in the RC file of a C++ resource DLL:

1001    PNG    DISCARDABLE    "msdn.png"
And here is the code that would load this into your application:
[DllImport("ImageLoader.dll", CharSet=CharSet.Unicode)]
public static extern IntPtr LoadBitmapFromResource(
    IntPtr hdc, uint dwResourceID, string pcszClass, IntPtr hModule);
...
hInstance = Win32.LoadLibary ( stringDLLName );
if (hInstance != IntPtr.Zero)
{
    IntPtr hdc = Win32.GetDC ( this.Handle );
    hbmp = LoadBitmapFromResource ( hdc, 1001,
           "PNG", hInstance );
    Win32.ReleaseDC ( hwnd, hdc );
    Win32.FreeLibrary ( stringDLLName );
}

You will find that this loads a bitmap quite quickly. And since the JPG, PNG, and GIF formats are supported, your resource DLLs can be quite small. In addition to images, you can also use the Win32.LoadString method to stuff language-specific information or even positional information as strings.

Figure 11 Resource Test
Figure 11 Resource Test

In our Test Bench application, tap the Resource Test button to see a comparison (see Figure 11). In the Resource Test screen, tap the Load button to load a bitmap from a resource DLL using the code you just saw and using a normal .NET resource call. Each piece of code loads the resource, draws it, and then discards it. The first time you run the test, you will see a stunning difference in speed. Subsequent runs are not as impressive, yet there is still a difference. For example, the next run we did resulted in a load time of 30ms for the .NET Compact Framework and 7ms for our DLL.

Load Only What You Need; Chuck What You Don't

Good optimization starts with good memory management. Our favorite method in programming .NET is Dispose. Since we do not live in the blissful world of endless memory, it is important when writing your code to think about the resources it uses. For each and every form, you want to add code for when it is activated and deactivated, as shown in the following:

protected override void OnActivated(EventArgs e)
{
    base.OnActivated (e);
}

protected override void OnDeactivate(EventArgs e)
{
    base.OnDeactivate (e);
}
In these functions, you have to be sure to load and discard anything that isn't critical to the application. This is where we load and discard our bitmaps.

The same mentality goes for objects that you create and use. Any time you can free up objects when you are not using them any more, you should let them go. And everything should be created on the fly unless it is absolutely necessary for the app's well-being. This includes the forms themselves. Forms and the controls in them take up memory. When you close down a form, dispose of the form as well.

The using keyword in C# is an excellent aid for keeping you on the disposal track. Take a look at this example:

using ( MyClass thing = new MyClass ())
{
    // Do something interesting here
}
Inside the curly braces the thing object is valid. Once program control leaves the braces, the Dispose method of thing is automatically called and any resources used by the instance of MyClass are released (although the garbage collector will still be responsible for releasing the memory occupied by the thing itself). Of course, your MyClass object must support the IDisposable interface for this to work.

Solid application architecture makes this scenario easier to implement. One paradigm of programming is known as the Mediator Model. In a nutshell, apps written in accordance with the Mediator Model are arranged so that no object knows of any objects other than those directly underneath it. Objects have no idea about their parents, and they have no idea about siblings. Communication with parent objects is accomplished solely via events, and communication with direct descendants is accomplished using methods. Keeping true to this model makes it very easy to prune off entire sections of an object tree dynamically when they are not used, and makes your memory management easier. For example, let's say your application creates an object A which itself creates and uses an object B, which in turn creates and uses an object called C.

When that part of your application is no longer needed, the top-level object A simply calls B.Dispose. B.Dispose then calls the Dispose method of the C object. As a result, all of the resources from A down are removed. If you end up reusing this code in other apps, or you move it around in your current app, as long as you stick to the rules of the Mediator Model, your initialization and cleanup of objects will be robust.

In addition to keeping the overall memory footprint small, another reason to load only what you need is to improve application load time. Every unnecessary form, bitmap, or database loaded and cached away makes for a longer load time. Since Windows® CE is an instant-on type of operating system, consumers will expect the same from their applications.

Threads are your friend when it comes to reducing load time. You can use them to build up nonessential structures while the main app is doing essential activities, such as checking for registration, logging in, displaying a splash screen, or letting the user browse a main screen. If your program has many features available from a toolbar or menu, you could load the optional features in a thread, and grow or enable the menu or toolbar to show more features as the background thread is finished loading.

Putting It All Together—the TabControlEx Class

Now it's time to put together a lot of the concepts we've discussed. TabControl in the .NET Compact Framework is packed with problems. The biggest problem is that all of the pages and their controls are loaded at startup. This means that there is a long load time before the tab control comes up, and lots of memory is used. We've written a new class, which we cleverly call TabControlEx. It's a tab control built from scratch that allows for pages to be loaded only when the user clicks on the tab. TabControlEx also allows for the page to be discarded when the user clicks on another tab. The pages will automatically resize if the Input Panel appears or disappears. There are also some other nice goodies, such as better control over the visuals. See Figure 12 for a list of the properties and methods associated with the TabControlEx class.

Figure 12 The TabControlEx Class (Close Figure 12)

Property | Description
AutoSize | Setting this property to true (default) will let the tab control size itself to use the entire client rectangle of the parent form. It will also resize itself and the tab pages whenever you show or hide the input panel. Note that you must set the Form property for this to work.
Form | Set this property to the parent form if you'd like the tab control to use the entire client area of the parent form and to resize automatically when the input panel is shown or hidden.
InputPanel | Provides access to the InputPanel object being used by this tab control, if any. Setting the Form property will create an InputPanel object if the Form doesn't already have one.
PageBounds | The size and location of the area available for the pages of the tab control.
SelectColor | The color that will be used for the background of the currently selected tab.
SelectedControl | The Page control for the page that is currently visible.
SelectedPage | Gives you direct access to the control for a specific page.
SelectedIndex | The index of the selected page or -1 if no page is selected or there are no pages.
TabPages | The collection of pages assigned to the control. The controls for each page aren't actually loaded until you select the tab for that page.
TabAlign | Determines where the tabs will be drawn, along with TabPosition.
TabPosition | Determines where the tabs will be drawn, along with TabAlign.
InputPanelVisible | Allows you to see whether the input panel is visible on the screen.
Method | Description
DisposeOtherPages | Disposes all the pages except for the one you indicate, which frees them from memory.
LoadPage | Forces a specific page to be loaded, even if it's not visible.

The biggest difference with TagControlEx is that you don't create your pages and add them to a collection. Instead, you add the types of pages to the collection:

m_tab.TabPages.Add("Tab Text", typeof(MyPage));

The TabControlEx class has a private method called LoadPage that uses this type information to create an instance of the control. Inside the LoadPage method is the code that is responsible for creating the actual object:

private Control LoadPage(Type type, TabPageEx tab) {
    ...
    ConstructorInfo constructor = type.GetConstructor(new Type[] {});
    Control page = (Control) constructor.Invoke(null);
    ...
    return page;
}

Using reflection to get access to the default constructor and invoking it is an elegant means of creating different kinds of objects on the fly with one piece of code.

The LoadPage method contains some code that attempts to get an IPage interface (see Figure 13) from the control, and then calls the InitPage method:

IPage p = page as IPage;
if (p != null) p.InitPage(this);

Figure 13 IPage, IPageActivate, and IPageToolbar (Close Figure 13)
using System;

namespace WinCE
{
    public interface IPage
    {
        void InitPage(TabControlEx tabs);
        bool CanResize();
    }

    public interface IPageActivate
    {
        void Activated();
        void Deactivated();
    }

    // This is an optional interface that can be implemented by
    // TabPageStyled pages if they want their own toolbar and/or menu
    // bar.
    public interface IPageToolbar
    {
        ToolbarSection[] Sections(MyToolbar toolbar);
    }
}

You would implement the IPage interface for two reasons. First, you might want to do some initialization once your control is loaded that requires communicating with the rest of the application. The InitPage method has a single parameter, a pointer to the TabControlEx object. If your page needs to communicate with other pages on the tab, you can use the TabPages collection to find the other control if it's loaded. You can also register to receive the PageLoadedEventHandler event so you can interact with other controls after they're loaded. Finally, you could create a subclass of TabControlEx that has some extra properties or methods that your pages will use to interact with the rest of the application.

Second, not all pages will want to be resized when the input panel becomes visible. If this is the case, you can have the CanResize property of the IPage interface return false. The default behavior, resizing your page, will happen if you return true for your implementation of this property.

Sometimes you might want to know when your page becomes active or inactive so you can, for example, update values based on other changes in the application. The TabControl in the .NET Compact Framework doesn't provide an easy way to get such notification. You can get this notification very easily in your page control by implementing the IPageActivate interface (see Figure 13), which contains just two methods: Activated and Deactivated. These methods will be called only when the specific instance of your control is activated or deactivated. You can get the same information for all pages, instead of just one, using the TabControlEx's SelectedIndexChanged event. Generally when a page needs to know when it becomes active or inactive, using IPageActivate is easier. But when you want to monitor all pages from your application, SelectedIndexChanged would definitely be better.

In our Test Bench application, tapping on the Tab Control button brings up the tab control test dialog. The constructor for the TabTest object is shown in Figure 14.

Figure 14 TabTest Constructor (Close Figure 14)
public TabTest()
{
    // Required for Windows Forms Designer support
    InitializeComponent();

    // Build our TabControlEx
    m_tab = new TabControlEx();
    m_tab.Location = new Point(20, 20);
    Controls.Add(m_tab);
    m_tab.Visible = true;
    m_tab.Form = this;

    // Add the *types* of our pages
    m_tab.TabPages.Add("Test", typeof(MyCustomControl));
    m_tab.TabPages.Add("Button", typeof(MyButton));
}

Adding Toolbar Buttons to TabControlEx

The final optional interface for your tab pages provides something that is really useful, but is difficult to implement using the tab control in the .NET Compact Framework. Let's say you have a menu bar and you want to have some toolbars that are specific to each page of your tab control. With our TabControlEx it's really simple. You'll need to add an instance of MyToolbar to your form instead of Toolbar. MyToolbar is a subclass of Toolbar that supports the concepts of toolbar sections, which are toolbar buttons that can be shown or hidden as a group. The IPageToolbar interface in Figure 13 has a single method, which returns a list of toolbar sections. Figure 15 shows the code from our Test Bench sample application that sets up a couple of toolbar icons on the screen.

Figure 15 ToolbarSection (Close Figure 15)
public ToolbarSection[] Sections(MyToolbar toolbar)
{
    if (!UseToolbar) return null;

    if (m_tbrSections == null)
    {
        ToolbarSection section = toolbar.CreateSection();

        Icon cancel = LoadIcon("cancel.ico");
        Icon accept = LoadIcon("accept.ico");
        section.Add(cancel, "day").Style = ToolBarButtonStyle.PushButton;
        section.Add(accept, "accept").Style =
            ToolBarButtonStyle.PushButton;
        m_tbrSections = new ToolbarSection[] { section };
    }

    return m_tbrSections;
}

The code in the IPageToolbar.Sections method creates a new toolbar section if this is the first time that this method is called. We then add two icons to this section and return it. LoadIcon is just a helper function, as shown here:

private Icon LoadIcon(string name)
{
    string path = Path.Combine(Utility.HomePath, name);
    usin(Stream stream = File.OpenRead(path))
    {
        return new Icon(stream);
    }
}

LoadIcon gets the Utility.HomePath property, which is a one-liner that wraps up this tough-to-remember tangle of characters:

public static string HomePath
{
    get
    {
        return Path.GetDirectoryName(
            Assembly.GetExecutingAssembly().GetName().CodeBase);
    }
}

Run the Test Bench sample and choose the Toolbar Test button. You can jump back and forth between the two tabs and see how the toolbar buttons appear only for the Test page.

You may have noticed that we used the same control, MyCustomControl, for this test with a toolbar and the Custom Control test, which did not use a toolbar. The IPageToolbar.Sections method returns null if the UseToolbar property of MyCustomControl is false, which is the default. To set this to true for the toolbar test, we use a technique we mentioned earlier for communicating with pages when they're first loaded. The constructor of the ToolbarTest form is identical to the TabTest form, except for the extra code added at the end of the constructor:

public ToolbarTest()
{
    ...
    tab.PageLoaded += new
        TabControlEx.PageLoadedEventHandler(tab_PageLoaded);

    // Create an instance of our toolbar needed to allow
    // each page to have its own toolbar buttons.
    MyToolbar toolbar = new MyToolbar();
    Controls.Add(toolbar);
}

First, we register to receive the PageLoadedEventHandler event of the tab control, which allows us to set properties of the control immediately after it's been loaded. We'll use this event to set the UserToolbar property to true:

private void tab_PageLoaded(object sender, PageLoadedEventArgs e)
{
    MyCustomControl control = e.Page as MyCustomControl;
    if (control != null) control.UseToolbar = true;
}
After creating the event handler, we add an instance of MyToolbar to the form, which is a subclass of the Toolbar class that supports toolbar sections.

Conclusion

Writing applications for the .NET Compact Framework is a wonderful experience. The applications are small enough that shareware developers can once again write something on their own and sell it online to make a little bit of money. But the simplicity and small capacity of mobile devices present new challenges to programmers. Without awesome power, you have awesome responsibility. Strive from the get-go to design code that is fast, tight, and highly, highly disposable.

Dave Edson is currently wearing the hats of massage therapist, pinball machine czar, and Software Architect for Potala Software, LLC. He lives in Seattle with his wife and two kids. Every possible spare moment is spent snow skiing or wakeboarding.

John Socha-Leialoha is the Chief Software Architect for Potala Software, LLC. His past achievements include writing the Norton Commander. He lives in Seattle with his wife and son. Every possible spare moment is spent building ridiculously small, yet terrifically accurate scale models for his model railroading hobby/business.


Vrooooom: How .NET and C# Drove an Entry in the DARPA Grand Challenge

Vrooooom!

How .NET and C# Drove an Entry in the DARPA Grand Challenge

John Hind
This article discusses:
  • Autonomous vehicle control with the .NET Framework
  • An extensible real-time control architecture based on a whiteboard metaphor
  • Implementation of an accurate GPS-synchronized timer component for .NET
This article uses the following technologies:
C#, .NET, Windows XP
Code download available at: DARPAChallenge.exe (141KB)
Contents
Background
Real Time
Design Patterns
Whiteboard
Tips for Implementing Windows Services
A Design Pattern for Add-Ins
Device Implementation
Messaging
Telemetry
Implementation of RS-232 Devices
Mission Clock
Resilience
Further Developments
Conclusion

I
n 2003, I was asked to help engineer software for an autonomous vehicle to be entered in an off-road race, the DARPA Grand Challenge, which you may have read about last spring. The opportunity to be involved in a team on the cutting edge of robotics was irresistible, though I was more than a little daunted: the race called for an average speed of over 25 mph sustained for up to 10 hours, we had less than a year to design the vehicle, and the off-road course would not be known in advance. The first prize was a million dollars.

Figure 1 The Carnegie Mellon Red Team Car
Figure 1 The Carnegie Mellon Red Team Car

A vehicle called Sandstorm (see Figure 1), built by the Red Team from Carnegie Mellon University, led the field by completing 7.4 miles. My team, Team Spirit of Las Vegas, built a vehicle called Autoquad. Although our vehicle (shown in Figure 2) did not make it through the qualification stage, we learned valuable lessons that may give us another shot at the still-unclaimed prize.

Figure 2 Team Spirit of Las Vegas Car
Figure 2 Team Spirit of Las Vegas Car

Background

Once I joined the team, I reviewed the work already done and the design decisions made by the team. A Honda Rincon one-person ATV, donated by Honda R&D Americas Inc., had been chosen as the base vehicle and had been automated using electric servo motors and string-pots. The control architecture includes a computer vision system with a gyroscopic-stabilized video camera feeding an Apple Power Mac G5 running OS X, a choice dictated by the experience and code base of the engineers working on this part of the project. There is a GPS receiver from Novatel Inc., a second GPS receiver from Swiss company u-blox AG, a three-axis orientation sensor from MicroStrain, a radio telemetry system, and four BasicX microcontroller units (BX-24s) from NetMedia Inc. The BX-24s control the servo motors and the remaining sensors. A PC running Windows® XP integrates the systems using Ethernet with the G5 and uses point-to-point RS-232 links with the other devices. That same computer also provides a platform for the navigation, command, and vehicle management systems. An Edgeport eight-port USB-to-serial converter/multiplexer from Inside Out Networks was used to provide sufficient RS-232 ports. Most of the hardware was given or lent to the team by sponsoring manufacturers (see Figure 3 for a complete diagram of the system).

Figure 3 Autonomous Vehicle Control System
Figure 3 Autonomous Vehicle Control System

Real Time

The computer-generated imagery rendering for the latest Hollywood blockbuster demands high performance, but it does not need to be real time. A system that spits out frames at random intervals over an hour, sometimes producing 10 frames in one second and sometimes cogitating for a minute over a single frame is as good as one that churns out frames like clockwork. On the other hand, a video game must sense the controller states and produce 20 or more new frames a second or the illusion is broken. This is a real-time system; if a frame cannot be rendered on time it is better to move on to the next one or to compromise quality in order to get the frame out. Driving very definitely presents real-time problems. At 30 mph, a driver has well under a second to respond to an obstacle detected 40 feet off, and this response must include both planning and executing the evasive maneuver.

My role in this project was real-time integration: providing a reliable platform for running artificial intelligence (AI) code, inputting and outputting data from sensors and actuators, and routing data between diverse platforms. Several considerations pushed me into writing the "glue" code using the .NET Framework. I have found C# programming to be highly robust, an essential attribute when you are working with nearly impossible deadlines. Furthermore, it was clear to me that much of the code written by robotics and navigation experts would be in various dialects of Basic. I could easily convert this code to Visual Basic® .NET and present it back to its authors in a form they could still understand. This ease of mixing languages is generally limited to the .NET environment and is of particular benefit in volunteer "garage engineering" projects where keeping application experts inside their comfort zone is more important than architectural purity.

However, I was quite aware of the potential downside: .NET was not primarily designed for real-time applications and most commercial operating systems, including Windows XP, are not designed to be real-time systems.

Design Patterns

The code I present in this article is a mixture of XML and C#. Please note that this code is not shipping code, and that most design decisions were made to support an evolving application. I've used design patterns (generic templates) for some of the code. Text-pasting tokens are shown as uppercase labels enclosed in brackets with no embedded white space (that is, {NAME}). The entire token needs to be replaced by text appropriate to the application. {NAME}Driver might become GpsDriver.

As a practical example, I've used the singleton design pattern several times. which I've defined as follows:

public class {CLASS-NAME}
{
    private {CLASS-NAME}(){}
    static {CLASS-NAME} _singleton = new {CLASS-NAME}();
    public static {CLASS-NAME} Singleton {get {return _singleton;}}
    •••
}
I used this for the MissionClock class by substituting for CLASS-NAME and converting it to code as follows:
public class MissionClock
{
    private MissionClock(){}
    static MissionClock _singleton = new MissionClock();
    public static MissionClock Singleton {get {return _singleton;}}
    ...
}
I used the singleton design pattern to create a class of which there is always exactly one instance. The static reference ensures that one instance is created automatically by the time I first need to use it, and the private constructor ensures that no more instances can be erroneously created. This one-and-only instance is referenced using MissionClock.Singleton.

Whiteboard

The whiteboard metaphor came from the image of a group of academics collaborating on an analysis around a shared whiteboard. Each writes what they know about the problem on the board, and all can see and respond to the collective knowledge of the group. Some of the academics are in radio contact with students making new observations in the field; they update data on the board in response to changing conditions of the phenomenon being analyzed. This system works on the basis of verifiable trust—anyone can see, use, and change any data on the whiteboard, but everyone can also see who sourced or changed a particular piece of data should it prove unreliable.

I decided to organize the whiteboard metaphor into a table of parameters, each of which has an address and a state. I defined devices and parameters required for a particular application using an XML file. I built devices as independently compiled components (assemblies); they communicate with one another entirely through the whiteboard. I implemented the whiteboard on one computer running Windows XP, but devices could be implemented on the same computer, or even on different computers, with potentially heterogeneous architectures. This design, which I call Virtual Instrument and Control Panel (VICP), provides an extremely flexible platform for experimental artificial intelligence, since I could easily add new sensors, actuators, and processing modules, move them between multiple platforms and enable or disable them at run time. I could build communications, data logging, fault tolerance and human-machine interfaces (HMI) generically to work with the well-defined parameter scheme.

Here is the XML that defines the latitude and longitude parameters for the current vehicle position as reported by a GPS sensor:

<INT32 address="102" source="5" log="Y" scale="7">GPS:Latitude</INT32>
<INT32 address="103" source="5" log="Y" scale="7">GPS:Longitude</INT32>
Here I hold the whiteboard table in memory and parse the XML once on startup. This XML results in two 32-bit signed integer table slots at addresses 102 and 103. The whiteboard table records that the source of these parameters is device 5, one of the GPS devices, and that changes to the state should be logged. The text and the scale attribute are not recorded in the whiteboard. I separately parse these in the HMI process for display formatting and labeling. Any device can write any parameter; the log records device number, time of change and new state. I identify the source device to the whiteboard for automated fault handling and for update rate management (an optional move, which I describe later). In addition to the normal range of states a parameter of the specified type can take, all VICP parameters have an additional named state: unknown. I set all parameters to unknown at startup until they are written by a device. If the device identified as the source fails, the parameter is automatically changed back to unknown. Furthermore, devices can explicitly write a parameter to unknown; for example, if the GPS antenna can't see enough satellites to generate a position solution. In this way, a device depending on this parameter can never be fooled by outdated or invalid data.

Here are a few more examples of parameter definitions:

<INT16 address="88" min="0" max="1023" valmax="12.0">Battery Volts</INT16>
<UINT2 address="58" states="F|N|R">Transmission Gear</UINT2>
These create similar structures in the whiteboard, but provide different data interpretation in the HMI. Parameter 88 is interpreted as values in the range 0 to 12.0, scaled linearly from the state codes 0 to 1023 (valmin could also be specified, but when omitted is assumed to be the same as min). Parameter 58 is interpreted as having three named states: F, N, and R (forward, neutral and reverse), represented by state codes 0, 1, and 2. As always, each also has an additional named state: unknown.

To keep the communications architecture lean and mean, I decided on a limited range of data types, focusing on minimizing storage size. Each BX-24 microcontroller had to communicate over slow RS-232 links where every byte transmitted and every byte of precious RAM used for buffering had to be justified. (The MPU chips had only 400 bytes of RAM available. Yes, I did say bytes!) The parameters could be 2-bit or 8-bit unsigned integers or 16-bit or 32-bit signed integers. I chose the 2-bit minimum size simply because there was a spare bit available in the RS-232 message layout. This parameter type had five possible states (including unknown), but mostly only three were used for a Boolean parameter (true, false, and unknown). I found a few uses for the extra states, though, as in the vehicle transmission example.

Since the VICP is a multithreaded application, data locking was vital for thread safety. I implemented a very simple mechanism at the parameter level. Thus it's important that parameters remain independent of one another. Latitude and longitude are independent dimensions, but degrees and minutes of latitude and the hemisphere marker (north or south) are dependent, which is why I stored latitude in one parameter as signed degrees scaled by 107. It is easiest to see this problem with time and date—if I stored these as separate parameters, the machine's chronology would jump by 24 hours in the interval between the time and the date being updated each midnight.

I see these restrictions as advantages; in computing, particularly when dealing with the real world through sensors and actuators, it's good to remember these things are really just finite state machines because thinking of them this way means fewer mistakes. Mathematicians and theologians can play about with infinity; engineers and scientists should forget these theoretical fantasies and deal with reality. In most real-world measurements, if you are processing more than 10 to 16 bits, you are processing noise. In the present application, the only parameters that can be measured with enough resolution to justify the 32-bit parameter type are latitude, longitude, and time.

Tips for Implementing Windows Services

As the VICP will run on a machine on an unattended vehicle, I was clear from the outset that it ought to be implemented as a Windows service. This provides built-in support for boot startup and fault restart, and makes it easier to ensure that the application will never hang while awaiting a response from a nonexistent operator. It is very easy to create a service in Visual Studio® .NET, but not as easy to thoroughly debug it. By far the easiest solution I found is to test and debug the application as a console app and then convert it to a service for release, although there are other options available. I quickly found that this conversion had to be two-way, since I needed to switch back to the console version frequently to debug add-in application code. In the end, I refactored the code into three assemblies:

  • A singleton class VICPServer in a class library assembly (VICPServer.dll), which also contains the singleton class I called Reporter.
  • A .NET assembly (VICPConsole.exe) containing a simple harness to run VICPServer as a console application.
  • A .NET assembly (VICPService.exe) containing a simple harness to run VICPServer as a Windows service.
Now I could run the latest version of the application either as a service or as a console application without having to recompile.

In VICPServer, I trap all exceptions and use my own Reporter class. In the service environment, Reporter records messages in the system event log. In the console environment, the same messages are written to the console window.

I used the Windows service project type to generate VICPService and filled out some of the stub methods to hook up the application singleton, as you see here:

static void Main()
{
    System.ServiceProcess.ServiceBase.Run(VICPServer.Singleton);
}
private void InitializeComponent()
{
    components = new System.ComponentModel.Container();
    this.ServiceName = "VICPService";
    Reporter.Singleton.SetEventLog(EventLog);
}
At startup I needed to extract the XML configuration file name from the command line and then delegate the rest of the process:
protected override void OnStart(string[] args)
{
    if (args.Length > 0) VICPServer.Singleton.ConfigFile = args[0];
    VICPServer.Singleton.Start();
}
The routines that Windows uses to start and stop the service are delegated immediately:
protected override void OnStop()
{
    VICPServer.Singleton.Stop();
}
protected override void OnShutdown()
{
    VICPServer.Singleton.Stop();
}
Similarly, I used the console application project type to generate VICPConsole, and I changed the Main routine as follows:
[STAThread]
static void Main(string[] args)
{
    if (args.Length > 0) VICPServer.Singleton.ConfigFile = args[0];
    VICPServer.Singleton.Start();
    Console.WriteLine("Hit <enter> to exit.");
    Console.ReadLine();
    VICPServer.Singleton.Stop();
}
The Reporter singleton is quite simple and is shown in Figure 4.

Figure 4 Reporter Class (Close Figure 4)
public class Reporter
{
    private Reporter(){}
    static Reporter _singleton = new Reporter();
    public static Reporter Singleton {get {return _singleton;}}
    public void Report(string s)
    {
        if (_eventLog == null) Console.WriteLine(s);
        else _eventLog.WriteEntry(s, EventLogEntryType.Error);
    }
    public void SetEventLog(EventLog e){_eventLog = e;}
    private static EventLog _eventLog;
}

A Design Pattern for Add-Ins

I have implemented devices and several other types of XML configured runtime add-ins using a common design pattern. The XML looks like this:

<{ADDIN-TYPE}s>
    <{CLASS-NAME} ident="{IDENT-NUM}" enabled="Y" assy="{ASSY-NAME}" ...>
        label
    </{CLASS-NAME}>
    ...
</{ADDIN-TYPE}s>
Then I define two interfaces in an API assembly specific to the add-in type, like this:
public interface I{ADDIN-TYPE}Host{...}
public interface I{ADDIN-TYPE}Client{...}

In the host application, I create an object to act as a proxy for each add-in listed in the XML and store it in a collection. In this proxy object I implement the IHost and hold a reference to the IClient of the add-in object itself:

public class {ADDIN-TYPE}Proxy: I{ADDIN-TYPE}Host
{
    private {ADDIN-TYPE}Client _client;
    private uint _id;
    public {ADDIN-TYPE}Client Client {get{return _client;}}
    ...
}

The classes for the add-ins themselves (the clients) may be defined in the host assembly (built-in) or may be in another assembly that references the API assembly. This is completely flexible; many add-in classes may be defined in one assembly or each may have its own assembly. Many instances of the same add-in class may be used in an application. The host assembly does not need to be recompiled to make a new add-in class available; installation merely requires that you put the assembly in the correct directory. I used this design pattern for the add-in classes:

public class {CLASS-NAME}{ADDIN-TYPE}: I{ADDIN-TYPE}Client
{
    private I{ADDIN-TYPE}Host _host;

    public {CLASS-NAME}{ADDIN-TYPE}(I{ADDIN-TYPE}Host hif, string spec)
    {
        host = _hif;
        ...
    }
    ...
}

Here I pass a reference to the host interface to the constructor, which is stored for later use. I also pass the entire XML tag in string form. The latter enables additional attributes to be passed to the add-in at startup, and they can be parsed using an XML Text Reader in the constructor.

In the host program, I use an XmlTextReader to create a proxy object for each add-in specified in the XML file (see Figure 5). I then execute the LoadClient method of the proxy object to create and reference the add-in object (see Figure 6).

Figure 6 Loading an Add-in from Its Proxy (Close Figure 6)
private void LoadClient(string spec)
{
    Assembly assy = GetType().Assembly;
    string val;

    _client = null;
    _id = -1;

    XmlTextReader tr;
    tr = new XmlTextReader(spec, XmlNodeType.Element, null);
    try
    {
        tr.Normalization = true;
        tr.WhitespaceHandling = WhitespaceHandling.None;
        tr.Read();

        // Get the ID of the add-in
        val = tr.GetAttribute("ident");
        if (val == null) throw new Exception("ident not stated.");
        _id = uint.Parse(val);

        // Find and load the add-in's assembly
        val = tr.GetAttribute("assy");
        if (val != null) assy = Assembly.Load(val);

        // Create the add-in instance
        _client = (I{ADDIN-TYPE}Client)assy.CreateInstance(
            tr.Name + "{ADDIN-TYPE}", false, 0, null,
            new object[] {this, spec}, null, null);
    }
    catch(Exception e)
    {
        throw new Exception("Failed to create " + _id", e);
    }
    finally
    {
        if (tr != null) tr.Close();
    }
}
Figure 5 Loading Add-ins from an XML File (Close Figure 5)
private void Load{ADDIN-TYPE}s()
{
    string val;
    uint i;
    int depth;
    XmlTextReader tr;

    tr = new XmlTextReader("{XML-FILENAME}");
    try
    {
        tr.Normalization = true;
        tr.WhitespaceHandling = WhitespaceHandling.None;
        XmlTextReader tr1;

        _{ADDIN-TYPE}s = new {ADDIN-TYPE}Proxy[{MAX-ADDINS}];

        while(!tr.EOF)
        {
            if (tr.MoveToContent() == XmlNodeType.Element &&
                tr.Name == "{ADDIN-TYPE}s")
            {
                depth = tr.Depth;
                tr.Read();
                tr.MoveToContent();
                while (tr.Depth > depth)
                {
                    val = tr.GetAttribute("ident");
                    i = uint.Parse(val);
                    val = tr.ReadOuterXml();
                    _{ADDIN-TYPE}s[i] = new {ADDIN-TYPE}Proxy();
                    _{ADDIN-TYPE}s[i].LoadClient(val);
                    tr.MoveToContent();
                }
                break;
            }
            else tr.Read();
        }
    }
    finally
    {
        if (tr != null) tr.Close();
    }
}

Now, from the host application, I can use the client interface held in the proxy object to control the add-in. Many of the add-ins used in the VICP are multithreaded and respond to asynchronous events occurring in the real world. I could use the host interface reference stored in the add-in client to update the host application asynchronously. The host interface is of little use in single-threaded add-ins and I omit it if it isn't required.

Device Implementation

To implement devices in the VICP, I used the add-in design pattern just shown. Figure 7 illustrates the generic architecture. I implemented three general-purpose devices built into the VICP: RS-232, TCP, and LOCAL. I built the application-specific devices such as GPS into separate .NET assemblies. This means that the VICP code is generic, so it does not need to be recompiled for each application. The entire application configuration is in the XML file, which defines the devices and parameters.

Figure 7 Abstract Device Architecture
Figure 7 Abstract Device Architecture

In the current application, I create an instance of the GPS driver with the following XML:

<GPS ident="5" assy="GpsDriver" port="COM10" baud="19200"
    type="UBLOX" rate="8" block="100" timefix="Y">UBLOX GPS</GPS>
This creates device 5 using a class GpsDriver in assembly GpsDriver.dll. This custom device interfaces with the GPS hardware and uses it to update a set of parameters including latitude and longitude in the VICP (see Figure 8). Most GPS sensors use an ASCII protocol called NMEA183 over RS-232 and send update messages (called sentences in the protocol) once per second or faster. The GpsDriver assembly parses these sentences, scales the values appropriately, and uses the host interface to update the VICP.

Figure 8 RS-232 Devices
Figure 8 RS-232 Devices

As with all devices, I pass an XML extract to the constructor as a text string. I then extract some device-specific attributes to configure the driver. In this device, the rate attribute controls the rate at which the GPS position fix is updated (every eight ticks of the 32Hz clock, or four times a second). The device writes to a contiguous block of parameters starting at the address specified in the block attribute. The timefix attribute controls whether this GPS device will be used to correct the mission clock.

The XmlTextReader class makes it very easy to unpack the XML fragment passed to the device class in the constructor (see the code in Figure 9).

Figure 9 GpsDriver (Close Figure 9)
public GpsDriver(IDriverHost hif, string spec)
{
    _host = hif;
    XmlTextReader tr = new XmlTextReader(spec, XmlNodeType.Element,
        null);
    try
    {
        tr.Normalization = true;
        tr.WhitespaceHandling = WhitespaceHandling.None;

        tr.Read();
        string val = tr.GetAttribute("port");
        if (val == null)
            throw new Exception("GPS Device must specify a port");
        _port = val;

        _timeFix = false;
        val = tr.GetAttribute("timefix");
        if (val != null) _timeFix = (val == "Y");

        ...
    }
    finally
    {
        if (tr != null) tr.Close();
    }
}

Similarly, I used instances of the built-in RS-232 device to integrate the four BX-24s used in the application (see Figure 8):

<RS232 ident="4" port="COM7" baud="19200">BX-24 D</RS232>
Unlike the GPS, I had control of both ends for the MPU link and was able to design a highly robust and efficient binary messaging protocol. I implemented the PC end of this protocol in the RS-232 device and the BX-24 end using a state machine coded in Basic. In each BX-24, I created and synchronized a copy of selected parts of the whiteboard in Basic variables.

I decided to treat application code as devices in the VICP (see Figure 10). Both simply read and write the whiteboard. But the local and TCP device classes add generic threading and messaging functionality that make writing application code simpler. The local device class communicates with managed applications on the VICP machine. The TCP device communicates over TCP with both managed and unmanaged applications running on either the VICP or a remote machine. I coded a generic TCP application harness that runs either as a standalone executable or as a Windows service on the remote machine. Managed applications written to my specification are interchangeable between the local device and the TCP harnesses, enabling code to be migrated either way for performance rebalancing or other reasons. Unmanaged applications have to be written directly to a documented TCP interface and connected via the TCP device. This approach enabled integration of the video processing system, which is written in C and runs in an OS X environment.

Figure 10 Application Code Devices
Figure 10 Application Code Devices

Figure 10 uses the following XML to run an application class called Servo from the assembly DriveServo.dll in its own thread under the VICP process on the VICP machine:

<LOCAL ident="8" codeassy="DriveServo" codeclass="Servo">
    Drive Servo</LOCAL>
Alternatively, I use the following command line to run the same application class on machine 192.168.0.2 and listen for TCP contact from a VICP on machine 192.168.0.1 using port 1020:
>TcpAppConsole.exe 192.168.0.1 1020 DriveServo Servo
Then I use the following XML to tell the VICP to connect to this client:
<TCP ident="8" client="192.168.0.2" clientport="123">Drive Servo</TCP>
Neither the application nor the VICP can see any difference between these modes of operation.

Messaging

The messaging system I chose to use between the VICP and the devices is symmetrical, and essentially the same messages are used internally to the managed code, over RS-232, and over TCP. I use the messages shown in Figure 11.

Figure 11 Messaging System Commands (Close Figure 11)

Command | Description
PUT_UNKNOWN, PUT_UINT2, PUT_UINT8, PUT_INT16, PUT_INT32 | Carries the current value of a parameter. Received by a device, it is notification of the parameter state; at the VICP it is a command to change the state.
ASK_POLL | Requests the VICP or the device to respond with one immediate PUT message for a specified parameter.
ASK_ONCHANGE | Requests the VICP or the source device to send a PUT message every time a specified parameter changes state.
ASK_PERIODIC | Requests the VICP or the source device to send a PUT message for a specified parameter at a specified rate ranging from 32 times per second to once every 220 seconds.
ASK_CANCEL | Requests the VICP or the source device to cancel any ASK_ONCHANGE or ASK_PERIODIC against a specified parameter.

I do not require a device to respond to ASK messages at all, and I allow it to ignore PUT messages, but the VICP responds to PUT and ASK_POLL messages for all parameters. For the other ASK messages, I require a routing table entry to have been configured for the specified parameter against the requesting device. I do this with the following XML in the parameter definition:

<UINT2 address="58" source="3">ATV Transmission Gear
    <ROUTING destination="1" onchange="Y"/>
    <ROUTING destination="2" rate="32"/>
</UINT2>
This allows the VICP to route PUT messages for parameter 58 to device 1 and device 2. I require the initial routing policy to be set in the XML file, but once the table entry exists, the VICP responds to ASK_ONCHANGE and ASK_PERIODIC messages from the destination devices. I have attempted to intelligently manage the rate at which the source device sends updates to the VICP to match the fastest outward routing by sending ASK_PERIODIC messages to the identified source device.

I make no linkage between the PUT message size and the parameter type. For efficiency, the smallest message size that is capable of carrying the actual state is always used. For example, a PUT_UINT2 would be used to set an INT32 parameter to 0, 1, 2, or 3. For any attempt to PUT a state value that is outside the allowed range for a parameter, I set the state to unknown.

I used a simple message packing scheme for the internal and TCP representation of the messages; the RS-232 packing had some points of interest. To minimize the number of bytes in the messages, I decided to use a binary packet with variable packet size. The problem with this approach is that it's difficult for the receiver to know which byte is which in the stream of incoming messages. One approach is to have a checksum at the end of each message. The receiver then tests the hypothesis that each byte is the first byte in a message by testing the checksum. The checksum computes correctly when, and only when, the correct start byte has been located. After this, messages can be processed continuously, and reframing is only required if there is a subsequent checksum error. This is very efficient in minimizing the transmitted messages, but it requires a large buffer and a good deal of code on the receiver, which is not ideal for resource-limited BX-24s.

My alternative method uses a reserved message start byte; various coding tricks ensure this byte could not ever occur within a message. This required more transmitted bytes (one extra per message plus on average a small fraction for pattern avoidance coding), but was much easier to decode since the receiver simply unconditionally resets its state machine whenever it sees the start byte. The chart in Figure 12 shows this message-coding scheme.

Figure 12 RS-232 Message Coding Scheme (Close Figure 12)

Byte (Bits) | Value | Meaning
0 | 0xFF | Frame start.
1 |  | Parameter address bits 0..7. Parameter addresses in which this byte would be 0xFF are illegal. Such addresses are substituted with defined values in the range 4001..4095.
2 (0..3) |  | Parameter address bits 8..11.
2 (4..7) | 00xx | PUT message UINT2 with the data bits in xx.
 | 0100 | PUT message indicating parameter unavailable.
 | 0101 | PUT message UINT8 with one follow-on byte.
 | 0110 | PUT message INT16 with two follow-on bytes.
 | 0111 | PUT message INT32 with four follow-on bytes.
 | 1000 | ASK_ONCHANGE (requests ongoing PUT messages whenever the parameter changes).
 | 1010 | ASK_PERIODIC (requests ongoing PUT messages at a rate specified in the single follow-on byte).
 | 1011 | ASK_CANCEL (cancel any ASK_ONCHANGE or ASK_PUT).
 | 1111 | Illegal.
3..10 |  | Follow-on bytes (0..4) as specified. For multi-byte integers, the MSB is transmitted first. If a byte should be 0xFF or 0xAA, it is transmitted as the two-byte sequence 0xAA followed by the complement of the data byte.
Last |  | The sum of the bytes in the message before escaping and excluding the frame start byte; the bytes are added as unsigned and the result truncated to 8 bits; if the byte would be 0xFF, it is transmitted as 0x55 instead.

Telemetry

Telemetry is not a core requirement for an autonomous vehicle (in fact, it was specifically not allowed during the actual race). However, it is very useful during testing, and I always had ambitions for the VICP code beyond this specific application. The whiteboard system and the efficient, symmetrical messaging structure lend themselves quite well to telemetry for remotely controlled systems.

As illustrated in Figure 13, I simply ran the VICP service on both the vehicle and the bunker machines with an identical parameter configuration but with most devices omitted at the bunker end. Both ends have an identical telemetry device; in this case it was interfacing a point-to-point RS-232 radio modem (wired or wireless TCP or even cellular or satellite telephony could also be used).

Figure 13 Telemetry
Figure 13 Telemetry

The telemetry device is a transparent message channel. All the infrastructure for configuring and synchronizing the two parameter tables was already available. I simply built in the necessary routing tables using XML. Then the telemetry rates could be dynamically changed at run time, on an individual parameter basis, using ASK messages.

Implementation of RS-232 Devices

I built the generic RS-232 device and the specific devices for RS-232 sensors such as GPS using the CommBase .NET RS-232 library described in my article "Use P/Invoke to Develop a .NET Base Class Library for Serial Device Communications". The code in this library was the only unmanaged code I needed for this app (other instances of unmanaged code were used for reasons of programmer familiarity and preference rather than any inability of .NET to handle the requirements). I was pleased that the library easily handled the requirements of this demanding project (a total of eight simultaneous RS-232 ports with an aggregate throughput of over 20,000 bytes per second).

Two issues did come up, however, and I have addressed both of them in an updated code download for the 2002 article. The first issue is that I discovered in most versions of Windows, the usual port name syntax does not work for double-digit port numbers—for example you cannot open "COM10:". Fortunately, when I was banging my head on this problem, a correspondent e-mailed me with the solution (which is given in Knowledge Base article 115831). You have to specify "\\.\COM10" (it will not work with a trailing colon).

To make this simpler I built automatic handling of this issue right into the library. With the latest version, you can now specify the port name either as "COM10" or "COM10:" just as you can for single digit port numbers. I made the library first try to open the port name exactly as specified; if that fails, strip any trailing colon, prefix "\\.\" and try again.

Second, a more complex problem arose with the buffering strategy used for transmission. When testing the BX-24 link I found to my surprise that I could reduce the baud rate while maintaining the message transmission rate. A quick calculation showed that messages could not possibly be transmitted at this rate. The problem was that Windows XP provides a dynamic buffer for serial transmission that grows more or less indefinitely, so the messages received by the BX-24 get older and older as time goes on—not a good situation in a real-time system.

Detecting this situation required some ingenuity and lateral thinking. Using overlapped I/O in the library, I "fire and forget" messages into the transmission buffer. A separate process within the operating system takes bytes from the buffer and transmits them as fast as possible given the configured baud rate. I could count bytes into the buffer and count them out again, but this would require a lot of complex housekeeping. There is a way of receiving a notification, within the reception thread, when the transmission buffer becomes empty. To exploit this, I implemented two flags: dataQueued is set when a message is queued, and empty is set in the reception thread when the transmission buffer is reported empty. A new function in the library, IsCongested, calculates a return value which is true if data queued is true and empty is false, and then clears both flags. This function is called at regular, relatively long intervals. If data is being transmitted faster than it is being queued, the queue will become empty at least once during the interval. Alternatively, if data is accumulating in the queue it will never become empty and IsCongested will return true.

I used a handy trick for making a flag (or any other non-object variable) thread safe. I declared it as a single element array:

private bool[] _empty = new bool[1];
Now it is an object that can be locked, as in the implementation of IsCongested from the CommBase library:
protected bool IsCongested()
{
    bool e;
    if (!_dataQueued) return false;
    lock(_empty)
    {
         e = _empty[0];
         _empty[0] = false;
    }
    _dataQueued = false;
    return !e;
}

Mission Clock

I needed a clock that ticks 32 times per second for the VICP. My first attempt used the Timer component from .NET. I set the interval to 31ms for 3 ticks out of 4 and to 32 for the fourth tick to correct for the fractional shortfall. I counted these ticks over a period of time and compared the result with the real-time clock. My tick accumulator lost time at a rate of between 5 and 20 percent, with the worst losses occurring during periods of heavy disk activity. At first I was seriously worried: if Windows XP and .NET could not even keep up with 32 processing slots per second without any application code, what hope was there? After some experimentation I resolved the problem into two issues: first, significant disk activity did seem to prevent the timer code from running for longer than a whole tick (31ms); second, there seemed to be an uncertain interval between the timer expiring and the attached event code actually running. For accurate timekeeping, I could derive closed-loop corrections from the system real-time clock. Code execution slots would still be lost occasionally, but I decided I could live with this as long as I minimized disk activity on the run-time system.

I built a new MissionClock component from the ground up for this application entirely in managed code (C#). It was not a designer component, but was implemented as an ordinary class using the singleton design pattern.

I passed the MissionClock two delegates, one of which executes 32 times per second (if possible; remember that the operating system' scheduler is under no obligation to run my threads at any particular time) and the other every second. I also used the component to keep track of time parameters to give a consistent time to the entire system. These time parameters are locked to the system time, with provision for a correction to a time observation (GPS time in this application). A closed-loop system ensures that these time measures are kept accurate even if the system is unable to keep up with the 32 times per second tick rate. The time parameters are shown in Figure 14.

Figure 14 Time Parameters (Close Figure 14)

Time | Parameter
Mission time | 32-bit count incrementing 32 times per second from 0 at the start of mission
Universal time | Seconds since the start of the current half-century at the Prime Meridian
Local time | Seconds since midnight in the local time zone
Mission day | Count of local midnights since the start of the mission

I sample and store the local time zone and the half-century base for Universal Time when a mission is started to avoid discontinuities during the mission. I persist this information to an XML file when a mission is started, enabling me to reconstitute the parameters after any reboot. Figure 15 shows the closed-loop algorithm, which runs on a high-priority thread. The key is to keep track of the expected value of the computer clock in the accumulator. When I have done everything I need to do in the current time slot, I increment the accumulator by the tick interval and sleep the thread for the difference between this and the current value of the computer clock. This way, errors are corrected and do not build up. When I increment the accumulator, I check to see if it is actually greater than the current time; if not, a tick must have been missed and I loop around adding increments until caught up. In this event, the tick delegate is not run repeatedly, but the time counters are incremented. I also increment a "lost ticks" counter for diagnostic purposes. Because I like to cover my bases, I provided similar catch-up logic for the seconds counter, though I don't really expect it to ever be invoked.

Figure 15 Mission Clock Algorithm (Close Figure 15)
private void Ticker()
{
    long c; long n; bool s = false; bool t = false; bool m = false;
    uint mt = 0;
    try
    {
        while (true)
        {
            // Process stored actions:
            if (t) tr(mt);
            if (s) sr(mt);
            if (m) MissionOverflow();
            s = m = t = false;
            lock(this)
            {
                // Increase mission time by one tick:
                mt = ++_missionTime;
                t = ((tr != null) && (!_disableDelegates));

                // Second divider (self-correcting even with
                // large tick gaps):
                if (--_secondCounter <= 0)
                {
                    while (_secondCounter <= 0)
                    {
                        _secondCounter += 32;
                        _utcTime++;
                        _localTime++;

                        //Local midnight transition:
                        if (_localTime >= _secondsPerDay)
                        {
                            _localTime -= (uint)_secondsPerDay;
                            m = (++_missionDay >= _maxMissionDays);
                    }
                }
                s = ((sr != null) && (!_disableDelegates));
            }

            // The target accumulator value for this cycle:
            n = DateTime.UtcNow.Ticks + _currentOffset;

            // Determine the correction to apply to synchronize:
            c = _targetOffset – _currentOffset;
            if (c != 0)
            {
                if (c > _maxAdjust) c = _maxAdjust;
                if (c < (_maxAdjust * (-1))) c = _maxAdjust * (-1);
                _currentOffset += c;
            }

            // Advance the accumulator correcting for missed ticks:
            _accumulator += _tickDuration;
            while (_accumulator < n)
           {
                _accumulator += _tickDuration;
                _lostTicks++;
                _missionTime++;
                _secondCounter—;
            }
        }

        // Sleep for the remainder of the tick duration
        Thread.Sleep((int)((_accumulator - n)/_dateTimeMillisecond));
    }
    catch (Exception e){}
}

I use a second closed loop to apply corrections between observed GPS time (when available) and the computer clock. A public method (not shown here) takes an observed time and uses it to calculate targetOffset from the current value of the PC clock. In the Ticker algorithm, I compare this value with currentOffset value and apply a magnitude-limited correction factor to bring the two values closer together. This process allows me to avoid sudden jumps in time as a correction is applied—the duration of a tick becomes just slightly longer or shorter over a number of ticks until the correction is absorbed.

I use a lock block to provide mutually exclusive access to the private variables between the Ticker thread and other (possibly multiple other) threads that access the public methods. This makes the class fully thread safe.

Resilience

Any system required to operate for many hours in a hostile environment without human intervention demands a structured and thorough approach to software resilience. My first step was to ensure that the system could be safely rebooted at any time during the mission and then be capable of recovering all information and state to continue the mission intelligently. I designed the hardware system so that any of the BX-24s could detect main computer failure and cycle the power to reboot. I persisted very little information to disk, but what is persisted is updated on disk as soon as it changes to ensure it will be available on reboot. The parameter values are not persisted: on reboot they are set to the unknown state until the device sourcing them comes up and provides good data. The machine must know how far along the mission timeline it is; I keep track of this value by persisting the value of the PC clock and the local time zone offset at mission start and GPS time correction whenever it changes. On startup, I restore the mission clock immediately using the persisted information and the current value of the PC clock.

Rebooting the entire system is a last resort. I have exploited the system of XML-configured devices to provide much finer-grained recovery. Recall that devices are subsystems of application functionality as well as sensor and actuator subsystems. I require all devices to respond to an ASK-POLL message against a well-known pseudo-parameter by PUT_UINT2 with the named state OK. I made the VICP send this message every five seconds; if it either gets no response or a state other than OK, the device fails and the VICP sets all parameters it is registered as sourcing to the unknown state. I then attempt to stop and restart the device up to four times. If this fails, I retire the device object for garbage collection and create and initialize a brand new one.

The RS-232 device can reset a crashed BX-24 by pulsing the RS-232 CTS signal that I specified to be hardwired to a reset generator in the BX-24 hardware. The hardware was carefully designed to ensure PC failure modes could not hold the BX-24 in reset.

Further Developments

I am convinced that the future for robotic control systems lies in the symbiosis of many single-chip microcontrollers with a few PC class machines acting in a supervisory role. Microcontrollers, particularly the latest models with digital signal processor (DSP) class mathematical capabilities, have the ability to execute sophisticated closed-loop control algorithms in real time. The PCs bring large-scale data handling capacity and the convenience of a human interface to the party. However, the programming systems available for microcontrollers and DSPs are primitive in comparison to the latest generation of PC systems represented by .NET, and are point solutions not adapted to targeting multiprocessor heterogeneous hardware designs. I am working on developing these ideas into a state machine-based distributed application platform for real-time control and AI applications.

Conclusion

The main lesson I take away from this project is one of humility on behalf of the profession of software engineering. We really have a long way to go to bridge the gap between what we can do ourselves and what we can teach machines to do. Ironically, the part we find hard (navigation) is relatively easily automated, while driving, a skill that can be acquired in a few days by the most intellectually challenged teenager, remains elusive. The team I was involved with tried, and has so far failed, to do the job with roughly human-equivalent sensors. The more successful teams used expensive sensors that were much more capable than human senses. There must be some big, powerful signal processing trick that nature has discovered and we're still missing.

On the upside, I have demonstrated an effective, flexible, scalable and robust platform for experimenting with this and other related problems. The whiteboard structure combined with symmetrical messaging is a simple but extremely powerful solution to a wide range of complex control applications. With some understood limitations, notably related to the handling of rotating media, the .NET Framework on Windows XP has demonstrated more than adequate performance combined with excellent programmer productivity and flexibility.

John Hind is an independent author and consultant living and working in London, England. He specializes in microcontroller applications and control solutions. Get in touch with John at John.Hind@zen.co.uk.


.NET Internals: Tailor Your Application by Building a Custom Forms Designer with .NET

.NET Internals

Tailor Your Application by Building a Custom Forms Designer with .NET

Sayed Y. Hashimi
This article discusses:
  • Design-time environment fundamentals
  • Forms designer architecture
  • The implementation of the forms designer in Visual Studio .NET
  • Services you need to implement to write a forms designer for your own application
This article uses the following technologies:
The .NET Framework and C#
Code download available at: CustomFormsDesigner.exe (157KB)
Contents
Some Windows Forms Basics
Services and Containers
Let's Build a Forms Designer
Implementing Services
Designer Host
Designer Transactions
Interfaces
Tracking Designer Verbs and Showing Context Menus
ITypeDescriptorFilterService
Putting It All Together
Debugging the Project
Conclusion

F
or many years, MFC has been a popular framework for building Windows®-based applications. MFC includes a forms designer to make form building, event wiring, and other form-based programming tasks easier. Even though MFC is widely used, it has often been criticized for its shortcomings, most in areas that the Microsoft® .NET Framework addressed. As a matter of fact, the extensible and pluggable design-time architecture of Windows Forms in the .NET Framework has made development a whole lot more flexible than it had been with MFC.

With Windows Forms, for example, you can drag one of your custom controls from the toolbox and drop it onto the Visual Studio® design surface. Amazingly, Windows Forms knows nothing about the control, yet it's able to host it and let you manipulate its properties. You couldn't do any of that in MFC.

In this article, I'll discuss what's happening under the covers as you design your forms. Then I'll show you how to build your own bare-bones forms designer that allows your users to create a form in a manner similar to the way they do it using the forms designer in Visual Studio. In order to do that, you'll need to understand exactly what functionality is provided by the .NET Framework.

Some Windows Forms Basics

Before I get underway with the project, there are several basic concepts that are important to understand. Let's begin with the definition of a designer. A designer provides the design-mode UI and behavior of a component. For example, when you drop a button on a form, the button's designer is the entity that determines the appearance and behavior of the button. The design-time environment provides a forms designer and a properties editor to allow you to manipulate components and build user interfaces. The design-time environment also provides services that can be used to interact with, customize, and extend design-time support.

A forms designer provides design-time services and a facility for developers to design forms. The designer host works with the design-time environment to manage designer state, activities (such as transactions), and components. In addition, there are several concepts relating to the components themselves that are important to comprehend. For example, a component is disposable, can be managed by a container, and offers a Site property. It obtains these characteristics by implementing IComponent, as shown here:

public interface System.ComponentModel.IComponent : IDisposable
{
    ISite Site { get; set; }
    public event EventHandler Disposed;
}

The IComponent interface is the fundamental contract between the design-time environment and an element to be hosted on a design surface (such as the Visual Studio forms designer). For example, a button can be hosted on the Windows Forms designer because it implements IComponent.

The .NET Framework implements two types of components: visual and nonvisual. A visual component is a user interface element, such as a Control, and a nonvisual component is one without a user interface, such as one that creates a SQL Server™ database connection. The Visual Studio .NET forms designer distinguishes between visual and nonvisual components when you drag and drop components onto the design surface. Figure 1 shows an example of this distinction.

Figure 1 Visual and Nonvisual Components
Figure 1 Visual and Nonvisual Components

A container contains components and allows contained components to access each other. When a container manages a component, the container is responsible for disposing the component when the container gets disposed—a good idea because a component may use unmanaged resources, which are not automatically disposed of by the garbage collector. Container implements IContainer, which is nothing more than a few methods that let you add and remove components from the container:

public interface IContainer : IDisposable
{

    ComponentCollection Components { get; }
    void Add(IComponent component);
    void Add(IComponent component, string name);
    void Remove(IComponent component);
}

Don't let the simplicity of the interface fool you. The concept of a container is critical at design time and is useful in other situations as well. For example, you've certainly written business logic that instantiates several disposable components. This generally takes the following form:

using(MyComponent a = new MyComponent())
{
    // a.do();
}
using(MyComponent b = new MyComponent())
{
    // b.do();
}
using(MyComponent c = new MyComponent())
{
    // c.do();
}
Using a Container object, these lines are reduced to the following:
using(Container cont = new Container())
{
    MyComponent a = new MyComponent(cont);
    MyComponent b = new MyComponent(cont);
    MyComponent c = new MyComponent(cont);
    // a.do();
    // b.do();
    // c.do();
}

There is more to a container than the automatic disposal of its components. The .NET Framework defines what is called a site, which is related to a container and a component. The relationship between the three is shown in Figure 2. As you can see, a component is managed by exactly one container and each component has exactly one site. When building a forms designer, the same component cannot appear on more than one design surface. However, multiple components can be associated with the same container.

Figure 2 Relationships
Figure 2 Relationships

A component's lifecycle can be controlled by its container. In return for lifetime management, a component gains access to the services provided by the container. This relationship is analogous to a COM+ component living inside the COM+ container. By allowing the COM+ container to manage it, the COM+ component can participate in transactions and use other services provided by the COM+ container. In a design-time context, the relationship between a component and its container is established through a site. When you place a component on a form, the designer-host creates a site instance for the component and its container. When this relationship is established, the component has been "sited" and uses its ISite property to get to the services provided by its container.

Services and Containers

When a component allows a container to take ownership of it, the component gains access to the services provided by that container. A service, in this context, can be thought of as a function that has a well-known interface, can be obtained from a service provider, is stored in service containers, and is addressable by its type.

Service providers implement IServiceProvider as shown here:

public interface IServiceProvider
{
    object GetService(Type serviceType);
}
Clients obtain a service by supplying to the service provider's GetService method the type of the service they want. Service containers act as a repository for services and implement IServiceContainer, thus providing a means of adding and removing services. The following code shows the definition of IServiceContainer. Note that the service definition simply contains methods to add and remove a service.
public interface IServiceContainer : IServiceProvider
{
    void AddService(Type serviceType,ServiceCreatorCallback callback);
    void AddService(Type serviceType,ServiceCreatorCallback callback,
                    bool promote);
    void AddService(Type serviceType, object serviceInstance);
    void AddService(Type serviceType, object serviceInstance,
                    bool promote);
    void RemoveService(Type serviceType);
    void RemoveService(Type serviceType, bool promote);
}

Because a service container can store and retrieve services, they are also considered service providers and as such implement IServiceProvider. The combination of services, service providers, and service containers form a simple design pattern that has many advantages. For example, the pattern:

  • Creates loose couplings between the client components and the services they use.
  • Creates a simple service repository and discovery mechanism, which allows an application (or parts of applications) to scale well. You can build your application with exactly the necessary parts, then add additional services later without making any drastic changes to your application or module.
  • Provides the facility to implement lazy loading of services. The AddService method is overloaded to create services the first time they are queried.
  • Can be used as an alternative to static classes.
  • Promotes contract-based programming.
  • Can be used to implement a factory service.
  • Can be used to implement a pluggable architecture. You can use this simple pattern to load plug-ins and provide services to plug-ins (such as logging and configuration).
The design-time infrastructure uses this pattern quite extensively so it's important to understand it thoroughly.

Let's Build a Forms Designer

Now that you understand the basic concepts behind the design-time environment, I'll build upon them by examining the architecture of the forms designer (see Figure 3).

Figure 3 Forms Designer Architecture
Figure 3 Forms Designer Architecture

At the heart of the architecture sits the component. All other entities, directly or indirectly, work with components. The forms designer is the glue that connects the other entities. Forms designers use a designer host to obtain access to the design-time infrastructure. The designer host uses design-time services and provides some of services of its own. Services can, and often do, use other services.

The .NET Framework does not expose the forms designer in Visual Studio .NET because that implementation is application-specific. Even though the actual interfaces are not exposed, the design-time framework is there. All you have to do is provide implementations specific to your forms designer and then submit your version to the design-time environment to use.

My sample forms designer is shown in Figure 4. Like every forms designer, it has a toolbox for users to select tools or controls, a design surface to build forms, and a properties grid to manipulate component properties.

Figure 4 Custom Forms Designer Sample
Figure 4 Custom Forms Designer Sample

First I'll build the toolbox. Before I do, however, I need to decide how I want to present tools to the user. Visual Studio .NET has a navigation bar with several groups, each of which contains tools. To build the toolbox, you must do the following:

  1. Create a user interface that displays tools to the user
  2. Implement the IToolboxService
  3. Plug the IToolboxService implementation into the design-time environment
  4. Handle events, such as tool selection and drag and drop

For any real-world application, building the toolbox user interface can be quite time consuming. The first design decision you have to make is how to discover and load the tools, and there are several viable approaches. With the first method, you can hardcode the tools to be displayed. This is not recommended unless your application is extremely simple and very little maintenance will be required in the future.

The second method involves reading the tools from a configuration file. For example, the tools could be defined as follows:

<Toolbox>
    <ToolboxItems>
        <ToolboxItem DisplayName="Label"
            Image="ResourceAssembly,Resources.LabelImage.gif"/>
        <ToolboxItem DisplayName="Button"
            Image="ResourceAssembly,Resources.ButtonImage.gif"/>
        <ToolboxItem DisplayName="Textbox"
            Image="ResourceAssembly,Resources.TextboxImage.gif"/>
    </ToolboxItems>
</Toolbox>
This method has the advantage that you can add or subtract tools and not have to recompile code to alter the tools shown in the toolbox. Additionally, the implementation is fairly simple. You implement a section handler to read the Toolbox section and return a list of ToolboxItems.

The third approach is to create a class for each tool and decorate the class with an attribute that encapsulates things like display name, group, and bitmap. At startup, the application loads a set of assemblies (from a well-known location, as specified in a configuration file, or something similar), and then looks for types with a particular decoration (such as ToolboxAttribute). Types that have this decoration are loaded into the toolbox. This method is probably the most flexible and allows for great tool discovery through reflection, but it also requires a bit more work. In my sample application, I use the second method.

The next important step is to obtain toolbox images. You could spend days trying to create your own toolbox images, but it would be very handy to somehow access the toolbox images in the Visual Studio .NET toolbox. Luckily, there is a way to do that. Internally, the Visual Studio .NET toolbox is loaded using a variation of the third method. This means that the components and controls are decorated with an attribute (ToolboxBitmapAttribute) that defines where to obtain an image for the component or control.

In the sample application, the toolbox contents (groups and items) are defined in the application configuration file. To load the toolbox, a custom section handler reads the Toolbox section and returns a binding class. The binding class is then passed to the LoadToolbox method of the TreeView control that represents the toolbox, as shown in Figure 5.

Figure 5 Loading the Toolbox (Close Figure 5)
public void LoadToolbox(FDToolbox tools)
{
    // Clear out existing nodes and images associated with the tree
    toolboxView.Nodes.Clear();
    if (treeViewImgList != null) treeViewImgList.Dispose();
    treeViewImgList = new ImageList(this.components);

    // Two images are always used for category nodes and the pointer node
    treeViewImgList.Images.Add(requiredImgList.Images[0]);
    treeViewImgList.Images.Add(requiredImgList.Images[1]);

    // Assign ImageList to the TreeView
    toolboxView.ImageList = treeViewImgList;

    // If I have categories...
    if(tools!=null && tools.FDToolboxCategories!=null &&
       tools.FDToolboxCategories.Length>0)
    {
        foreach(Category cat in tools.FDToolboxCategories)
        {
            LoadCategory(cat);
        }
    }
}

private void LoadCategory(Category cat)
{
    // If I have items in the category...
    if(cat!=null && cat.FDToolboxItem!=null && cat.FDToolboxItem.Length>0)
    {
        // Create a node for the category
        TreeNode catNode = new TreeNode(cat.DisplayName);
        catNode.ImageIndex = 0;
        catNode.SelectedImageIndex = 0;

        // Add this category to the tree
        toolboxView.Nodes.Add(catNode);

        // Every category gets the selection tool node
        AddSelectionNode(catNode);
        foreach(FDToolboxItem item in cat.FDToolboxItem)
        {
            LoadItem(item,catNode);
        }
    }
}

private void LoadItem(FDToolboxItem item,TreeNode cat)
{
    if(item!=null && item.Type!=null && cat!=null)
    {
        // Load the type
        Type toolboxItemType = Type.GetType(item.Type);
        ToolboxItem toolItem = new ToolboxItem(toolboxItemType);

        // Get the image for the item and create a node for it
        Image img = GetItemImage(toolboxItemType);
        TreeNode nd = new TreeNode(toolItem.DisplayName);
        nd.Tag = toolItem;

        // Add the item's bitmap to the image list
        if(img!=null)
        {
            treeViewImgList.Images.Add(img);
            nd.ImageIndex = treeViewImgList.Images.Count-1;
            nd.SelectedImageIndex = treeViewImgList.Images.Count-1;
        }

        // Add this node to the category node
        cat.Nodes.Add(nd);
    }
}

private Image GetItemImage(Type type)
{
    // Get the AttributeCollection for the given type and
    // find the ToolboxBitmap attribute
    AttributeCollection attrCol=TypeDescriptor.GetAttributes(type,true);
    if(attrCol!=null)
    {
        ToolboxBitmapAttribute toolboxBitmapAttr =
          (ToolboxBitmapAttribute)attrCol[typeof(ToolboxBitmapAttribute)];
        if(toolboxBitmapAttr!=null)
        {
            return toolboxBitmapAttr.GetImage(type);
        }
    }
    return null;
}

The LoadItem method creates a ToolboxItem instance for the given type and then calls GetItemImage to get the image associated with that type. The method gets the attribute collection for the type to find the ToolboxBitmapAttribute. If it finds the attribute, it returns the image so it can be associated with the newly created ToolboxItem. Note that the method uses the TypeDescriptor class, a utility class in the System.ComponentModel namespace that is used to obtain attribute and event information for a given type.

Now that you know how to build the toolbox user interface, the next step is to implement IToolboxService. Since this interface is tied directly to the toolbox, it's handy to just implement this interface in the TreeView-derived class. Most of the implementation is straightforward, but you do need to pay special attention to how you handle drag and drop operations and to how you serialize toolbox items (see the toolboxView_MouseDown method in the ToolboxService implementation in the code download for this article, available from the MSDN®Magazine Web site). The final step in the process is to hook the service implementation into the design-time environment, which I'll demonstrate how to do after I discuss how to implement the designer host.

Implementing Services

The forms designer infrastructure is built on top of services. There are a set of services that you're required to implement, and then there are some that simply enhance the functionality of the forms designer if you implement them. This is an important aspect of the services pattern I talked about earlier, as well as of the forms designer. You can start by implementing the base set and then add additional services later.

The designer host is the hook into the design-time environment. The design-time environment uses the host service to create new components when users drag and drop components from the toolbox, to manage designer transactions, to look up services when users manipulate components, and so on. The host service definition, IDesignerHost, defines methods and events. In the host implementation, you provide implementations for the host service along with several other services. These should include IContainer, IComponentChangeService, IExtenderProviderService, ITypeDescriptionFilterService, and IDesignerEventService.

Designer Host

The designer host is the nucleus of the forms designer. When the constructor of the host is called, the host uses the parent service provider (IServiceProvider) to construct its service container. It is very common to chain providers in this manner in order to achieve a trickle-down effect. After the service container is created, the host adds its own services to the provider, as shown in Figure 6.

Figure 6 Adding Services to the Provider (Close Figure 6)
public DesignerHostImpl(IServiceProvider parentProvider)
{
    // Append to the parentProvider...
    serviceContainer = new ServiceContainer(parentProvider);
    loggerService = parentProvider.GetService(
        typeof(IServiceRequestLogger)) as IServiceRequestLogger;

    // Site name to ISite mapping
    sites = new Hashtable(CaseInsensitiveHashCodeProvider.Default,
        CaseInsensitiveComparer.Default);

    // Component to designer mapping
    designers = new Hashtable();

    // List of extender providers
    extenderProviders = new ArrayList();

    // Create transaction stack
    transactions = new Stack();

    // Services
    serviceContainer.AddService(typeof(IDesignerHost), this);
    serviceContainer.AddService(typeof(IContainer), this);
    serviceContainer.AddService(typeof(IComponentChangeService), this);
    serviceContainer.AddService(typeof(IExtenderProviderService), this);
    serviceContainer.AddService(typeof(IDesignerEventService), this);
    serviceContainer.AddService(
        typeof(INameCreationService), new NameCreationServiceImpl(this));
    serviceContainer.AddService(
        typeof(ISelectionService), new SelectionServiceImpl(this));
    serviceContainer.AddService(
        typeof(IMenuCommandService), new MenuCommandServiceImpl(this));
    serviceContainer.AddService(
        typeof(ITypeDescriptorFilterService),
        new TypeDescriptorFilterServiceImpl(this));
}

When a component is dropped onto the design surface, it needs to be added to the host's container. Adding a new component is a fairly involved operation because you have to perform several checks and fire off some events as well (see Figure 7).

Figure 7 Add Components (Close Figure 7)
public void Add(IComponent component, string name)
{
    if (component == null) throw new ArgumentException("component");

    // If I don't have a name, create one
    if (name == null || name.Trim().Length == 0)
    {
        // I need the naming service
        INameCreationService nameCreationService = GetService(
            typeof(INameCreationService)) as INameCreationService;
        if(nameCreationService==null)
            throw new Exception("Failed to get INameCreationService.");
        name = nameCreationService.CreateName(this,component.GetType());
    }

    // If I own the component and the name has changed, rename it
    if (component.Site != null && component.Site.Container == this &&
        name != null && string.Compare(name,component.Site.Name,true)!=0)
    {
        component.Site.Name=name;
        return;
    }

    // Create a site for the component and associate it
    ISite site = new SiteImpl(component, name, this);
    component.Site = site;

    // Fire necessary events
    ComponentEventArgs evtArgs = new ComponentEventArgs(component);
    ComponentEventHandler compAdding = ComponentAdding;
    if (compAdding != null)
    {
        try
        {
            compAdding(this, evtArgs);
        }
        catch{}
    }

    // if this is the root component
    IDesigner designer = null;
    if(rootComponent == null)
    {
        // Set the root component
        rootComponent = component;

        // Create the root designer
        rootDesigner = (IRootDesigner)TypeDescriptor.CreateDesigner(
            component,typeof(IRootDesigner));
        designer = rootDesigner;
    }
    else
    {
        designer = TypeDescriptor.CreateDesigner(
            component,typeof(IDesigner));
    }

    // Add the designer to the list and initialize it
    designers.Add(component,designer);
    designer.Initialize(component);

    // Add to container component list
    sites.Add(site.Name,site);
    compAdding = ComponentAdding;
    if (compAdding != null)
    {
        try
        {
            compAdding(this, evtArgs);
        }
        catch{}
    }
}

If you overlook the checks and events, you can summarize the addition algorithm as follows. First, a new IComponent is created for the type and a new ISite is created for the component. This establishes the site-to-component association. Note that the site's constructor accepts a designer host instance. The site constructor takes the designer host and the component so that it can establish the component-container relationship shown in Figure 2. Then the component designer is created, initialized, and added to the component-to-designer dictionary. Finally, the new component is added to the designer host container.

Removing a component requires a bit of cleanup. Again, overlooking simple checks and validation, the remove operation amounts to removing the designer, disposing the designer, removing the component's site, and then disposing the component.

Designer Transactions

The concept of designer transactions is analogous to database transactions since they both group a sequence of operations in order to treat the group as a unit of work and enable a commit/abort mechanism. Designer transactions are used throughout the design-time infrastructure to support the canceling of operations and to enable views to delay updates of their displays until the entire transaction is completed. The designer host provides the facility to manage designer transactions through the IDesignerHost interface. Managing transactions is not very difficult (see DesignerTransactionImpl.cs in the sample application).

DesignerTransactionImpl represents a single operation in a transaction. When the host is asked to create a transaction, it creates an instance of DesignerTransactionImpl to manage the single change. The host tracks the transactions while instances of DesignerTransactionImpl manage each change. If you don't implement transaction management, you will get some interesting exceptions while you work with the forms designer.

Interfaces

As I've said, components are put into containers for the purpose of lifetime management and to provide them with services. The designer host interface, IDesignerHost, defines methods to create and remove components, so you shouldn't be surprised if the host offers this service. Again, the container service defines methods to add and remove components and these overlap with the CreateComponent and DestroyComponent methods of IDesignerHost. Therefore, most of the heavy lifting is done in the add and remove methods of the container, and the create and destroy methods simply forward the calls to these methods.

IComponentChangeService defines component change, add, remove, and rename events. It also defines methods for component changed and changing events, which are called by the design-time environment when a component is changing or has changed (such as when a property is changed). This service is offered by the designer host because components get created and destroyed through the host. In addition to creating and destroying components, the host also handles component rename operations through the create method. The rename logic is simple, yet interesting:

// If I own the component and the name has changed, rename the component
if (component.Site != null && component.Site.Container == this &&
    name != null && string.Compare(name,component.Site.Name,true) != 0)
{
    // name validation and component changing/changed events are
    // fired in the Site.Name property so I don't have
    // to do it here...
    component.Site.Name=name;
    return;
}
The implementation of this interface is simple enough that you can defer the rest to the sample application.

ISelectionService deals with component selections on the design surface. When users select components, the SetSelectedComponents method is called by the design-time environment with the selected components. The implementation of SetSelectedComponents is shown in Figure 8.

Figure 8 SetSelectedComponents (Close Figure 8)
public void SetSelectedComponents(
    ICollection components, SelectionTypes selectionType)
{
    // fire changing event
    EventHandler selChange = SelectionChanging;
    if (selChange != null)
    {
        try
        {
            selChange(this, EventArgs.Empty);
        }
        catch{}
    }

    if (components == null) components = new ArrayList();

    bool ctrlDown=false,shiftDown=false;

    // I need to know if shift or ctrl is down on clicks
    if ((selectionType & SelectionTypes.Click) == SelectionTypes.Click)
    {
        ctrlDown = ((Control.ModifierKeys & Keys.Control)==Keys.Control);
        shiftDown = ((Control.ModifierKeys & Keys.Shift)==Keys.Shift);
    }

    if (selectionType == SelectionTypes.Replace)
    {
        // Discard the hold list and go with this one
        selectedComponents = new ArrayList(components);
    }
    else
    {
        if (!shiftDown && !ctrlDown && components.Count == 1 &&
            !selectedComponents.Contains(components))
        {
            selectedComponents.Clear();
        }

        // Something was either added to the selection or removed
        IEnumerator ie = components.GetEnumerator();
        while(ie.MoveNext())
        {
            IComponent comp = ie.Current as IComponent;
            if(comp!=null)
            {
                if (ctrlDown || shiftDown)
                {
                    if (selectedComponents.Contains(comp))
                    {
                        selectedComponents.Remove(comp);
                    }
                    else
                    {
                        // Put it back into the front because it was
                        // the last one selected
                        selectedComponents.Insert(0,comp);
                    }
                }
                else
                {
                    if (!selectedComponents.Contains(comp))
                    {
                        selectedComponents.Add(comp);
                    }
                    else
                    {
                        selectedComponents.Remove(comp);
                        selectedComponents.Insert(0,comp);
                    }
                }
            }
        }
    }

    // fire changed event
    selChange = SelectionChanging;
    if (selChange != null)
    {
        try
        {
            selChange(this, EventArgs.Empty);
        }
        catch{}
    }
}

The selection service tracks component selections on the designer surface. Other services, such as IMenuCommandService, use this service when they need to obtain information about selected components. In order to provide this information, the service maintains an internal list that represents the currently selected components. The design-time environment calls SetSelectedComponents with a collection of components when a change has been made to the selection of components. For example, if a user selects one component and then holds down the shift key and selects another three, the method is called for each addition to the selection list. Each time the method is called, the design-time environment tells us which components are affected and how (via the SelectionTypes enumeration). The implementation looks at how the components were changed in order to determine whether components need to be added to or removed from the internal selected list. After modifying the internal selection list, I fire the Selection Changed event (see method selectionService_SelectionChanged in SelectionServiceImpl.cs) so that the properties grid can be updated with the new selections. The application's main form, MainWindow, subscribes to the selection service's selection changed events in order to update the properties grid with the selected component(s).

Also note that the selection service defines a PrimarySelection property. The primary selection is always set to the last item selected. I'll use this property in my discussion of IMenuCommandService when I talk about showing the correct designer context menu.

The selection service is one of the more difficult services to implement correctly because it has some valuable features that complicate the implementation. For example, in a real-world application it makes sense to handle keyboard events, such as Ctrl+A, and also manage issues around handling a large selection list.

The ISite implementation is one of the more important implementations and is shown in Figure 9.

Figure 9 The ISite Implementation (Close Figure 9)
public class SiteImpl : ISite, IDictionaryService
{
    private IComponent component;
    private string name;
    private DesignerHostImpl host;
    private DictionaryServiceImpl dictionaryService;

    public SiteImpl(IComponent comp, string name, DesignerHostImpl host)
    {
        if(comp==null) throw new ArgumentException("comp");
        if(host==null) throw new ArgumentException("host");
        if(name==null || name.Trim().Length==0)
            throw new ArgumentException("name");

        component = comp;
        this.host = host;
        this.name = name;

        // Create a dictionary service for this site
        dictionaryService = new DictionaryServiceImpl();
    }

    public object GetService(Type service)
    {
        // Forward request to the host
        if (service == typeof(IDictionaryService)) return this;
        return host.GetService(service);
    }

    public object GetKey(object value)
    {
        return dictionaryService.GetKey(value);
    }

    public object GetValue(object key)
    {
        return dictionaryService.GetValue(key);
    }

    public void SetValue(object key, object value)
    {
        dictionaryService.SetValue(key,value);
    }
}

You'll notice that the SiteImpl also implements IDictionaryService, which is a bit unusual because all of the other services I implemented were tied to the designer host. It turns out that the design-time environment requires you to implement IDictionaryService for every sited component. The design-time environment uses the IDictionaryService on every site to maintain a data table that is used throughout the designer framework. Another thing to note about the site implementation is that since ISite extends IServiceProvider, the class provides an implementation for GetService. The designer framework calls this method when it's looking for a service implementation on a site. If the service request is for IDictionaryService, the implementation simply returns itself, the SiteImpl. For all other services, the request is forwarded to the site's container (for example, the host).

Every component has to have a unique name. As you drag and drop components from the toolbox onto the design surface, the design-time environment uses an implementation of INameCreationService to generate each component's name. The component's name is the Name property that is displayed in the properties window when the component is selected. The definition of the INameCreationService interface is shown here:

public interface INameCreationService
{
    string CreateName(IContainer container, Type dataType);
    bool IsValidName(string name);
    void ValidateName(string name);
}

In the sample application, the CreateName implementation uses the container and dataType to calculate a new name. In a nutshell, the method counts the number of components whose type is equivalent to dataType and then uses that count with the dataType to come up with a unique name.

The services discussed so far have all dealt with components, either directly or indirectly. The menu command service, on the other hand, is specific to designers. It is responsible for tracking menu commands and designer verbs (actions), and showing the correct context menu when a user chooses a particular designer.

The menu command service handles the tasks of adding, removing, finding, and executing menu commands. In addition, it defines methods to track designer verbs and to show designer context menus for designers that support them. The core of this implementation lies in showing the correct context menu. As such, I'll defer what little implementation is left to the sample application and instead focus on how to show context menus.

Tracking Designer Verbs and Showing Context Menus

There are two types of designer verbs: global and local. Global verbs exist for all designers and local verbs are specific to each designer. An example of a local verb can be seen when you right-click a tab control on the design surface (see Figure 10).

Figure 10 The Design Surface
Figure 10 The Design Surface

Right-clicking a tab control adds local verbs that allow you to add and remove tabs on the control. An example of a global verb can be seen in the Visual Studio forms designer when you right-click anywhere on the design surface. Regardless of where and on what object you click, you always see two menu items: View Code and Properties. Each designer has a Verbs property that contains the verbs representing functionality specific to that designer. For the tab control designer, for example, the verbs collection contains two members: Add Tab and Remove Tab.

When a user right-clicks a tab control on the design surface, the design-time environment calls the ShowContextMenu method on the IMenuCommandService (see Figure 11).

Figure 11 Handling Context Menus on the Design Surface (Close Figure 11)
public void ShowContextMenu(
    System.ComponentModel.Design.CommandID menuID, int x, int y)
{
    ISelectionService selectionService = host.GetService(
        typeof(ISelectionService)) as ISelectionService;

    // Get the primary component
    IComponent primarySelection =
        selectionService.PrimarySelection as IComponent;

    if (lastSelectedComponent != primarySelection)
    {
        // Remove all non-global menu items from the context menu
        ResetContextMenu();

        // Get the designer
        IDesigner designer = host.GetDesigner(primarySelection);

        // Not all controls need a designer
        if(designer!=null)
        {
            // Add menu items for designer's verbs
            DesignerVerbCollection verbs = designer.Verbs;
            foreach (DesignerVerb verb in verbs)
            {
                CreateAndAddLocalVerb(verb);
            }
        }
    }

    // I only show designer context menus for controls
    if(primarySelection is Control)
    {
        Control comp = primarySelection as Control;
        Point pt = comp.PointToScreen(new Point(0, 0));
        contextMenu.Show(comp, new Point(x - pt.X, y - pt.Y));
    }

    // Keep the selected component for next time
    lastSelectedComponent = primarySelection;
}

This method is responsible for showing the context menu for the selected object's designer. As you can see in Figure 11, the method gets the selected component from the selection service, obtains its designer from the host, gets the verbs collection from the designer, and then adds a menu item to the context menu for each verb. After the verbs have been added, the context menu is shown. Note that when you create new menu items for designer verbs, you also attach a click handler for the menu item. The custom click handler handles clicking events for all menu items (see the method MenuItemClickHandler in the sample application).

When a user selects a menu item from a designer context menu, the custom handler is called to execute the verb associated with the menu item. In the handler, you retrieve the verb associated with the menu item and invoke it.

ITypeDescriptorFilterService

I mentioned earlier that the TypeDescriptor class is a utility class that is used to obtain information about a type's properties, attributes, and events. The ITypeDescriptorFilterService can filter this information for sited components. The TypeDescriptor class uses ITypeDescriptorFilterService when it attempts to return a sited component's properties, attributes, and/or events. Designers who want to modify the metadata available to the design-time environment for the component they are designing can do so by implementing IDesignerFilter. The ITypeDescriptorFilterService defines three methods that allow designer filters to hook into and modify a sited component's metadata. Implementing ITypeDescriptorFilterService is simple and intuitive (see TypeDescriptorFilterService.cs in the sample application).

Putting It All Together

If you've taken a look at the sample application and run the forms designer, you may be wondering how all of the services have come together. You can't build the forms designer incrementally—that is, you can't implement one service, test the application, and then write another service. You have to implement all of the required services, build the user interface, and tie them all together before you can test the application. That's the bad news. The good news is that I have done most of the work in the services I have implemented. All that's left to do is a bit of crafting.

To start, take a look at the CreateComponent method of the designer host. When creating a new component, it's important to see if it's the first component (if rootComponent is null). If it is the first component, you have to create a specialized designer for the component. The specialized base designer is an IRootDesigner because the top most designer in the designer hierarchy has to be an IRootDesigner (see Figure 12).

Figure 12 Creating the Root Designer (Close Figure 12)
// If this is the root component
IDesigner designer = null;
if(rootComponent==null)
{
    // set the root component and create the root designer
    rootComponent = component;
    rootDesigner = (IRootDesigner)TypeDescriptor.CreateDesigner(
        component,typeof(IRootDesigner));
    designer = rootDesigner;
}
else designer = TypeDescriptor.CreateDesigner(
    component,typeof(IDesigner));

designers.Add(component,designer);
designer.Initialize(component);

Now that you know that the first component has to be the root component, how do you ensure that the right component is the first? The answer is that the design surface ends up being the first component because you create this control, as a Form, during the main window initialization routine (see Figure 13).

Figure 13 Initializing the Forms Designer (Close Figure 13)
private void InitWindow()
{
    serviceContainer = new ServiceContainer();

    // create host
    host = new DesignerHostImpl(serviceContainer);

    AddBaseServices();

    Form designSurfaceForm = host.CreateComponent(
        typeof(Form),null) as Form;

    // Create the forms designer now that I have the root designer
    FormDesignerDocumentCtrl formDesigner = new FormDesignerDocumentCtrl(
        this.GetService(typeof(IDesignerHost))as IDesignerHost,
        host.GetDesigner(designSurfaceForm) as IRootDesigner);
    formDesigner.InitializeDocument();
    formDesigner.Dock=DockStyle.Fill;
    formDesigner.Visible=true;
    formDesigner.Focus();
    designSurfacePanel.Controls.Add(formDesigner);

    // I need to subscribe to selection changed events so
    // that I can update our properties grid
    ISelectionService selectionService = host.GetService(
        typeof(ISelectionService)) as ISelectionService;
    selectionService.SelectionChanged += new EventHandler(
        selectionService_SelectionChanged);

    // Activate the host
    host.Activate();
}

Handling the root component is the only tricky part of the adhesive between the designer host, the design-time environment, and the user interface. The rest is easily understood by spending a bit of time reading the code.

Debugging the Project

Implementing a forms designer is not a trivial exercise. There is little existing documentation on the subject. And once you figure out where to start and what services to implement, debugging the project is going to be painful because you have to implement a set of required services and plug them in before you can start to debug any of it. Finally, once you implement the required services, the error messages that you get are not very helpful. For example, you may get a NullReferenceException on a line that calls the internal design-time assemblies, which you can't debug, so you're left wondering which service failed where.

Additionally, because the design-time infrastructure is built on top of the service pattern I discussed earlier, debugging services can be a problem. A technique to mitigate debugging pains is to log service requests. Logging which service request was queried, whether it passed or failed, and where it was called from (taking advantage of Environment.StackTrace) within the framework can be a very useful debugging tool to add to your arsenal.

Conclusion

I've provided an overview of the base services that you need to implement in order to get your forms designer up and running. In addition, you've seen how to configure the toolbox based on the needs of your application by changing the configuration file. What remains is to tweak the existing services and to implement a few others depending on your needs.

Sayed Y. Hashimi, a consultant based in Jacksonville, Florida, specializes in technologies related to the .NET Framework. Get in touch with him at hashimi_sayed@hotmail.com.


.NET Code Tuning: Make Your Apps Fly with the New Enterprise Performance Tool

.NET Code Tuning

Make Your Apps Fly with the New Enterprise Performance Tool

John Robbins
This article is based on a prerelease version of Visual Studio 2005. All information contained herein is subject to change.
This article discusses:
  • The inner workings of profilers
  • Flexible features of the EPT
  • A sample app to profile
This article uses the following technologies:
.NET, Visual Studio, C#
Code download available at: EnterprisePerformance.exe (258KB)
Contents
The Philosophy of Profilers
The Philosophy of the Enterprise Performance Tool
Animated Algorithm
Getting Started with the EPT
Viewing Profiler Data

F
ast code is still in vogue. Even though I'm typing this on a machine that has the power and memory to simultaneously control a nuclear power plant, a Mars rover mission, and the air traffic over the American West, and still have plenty of horses left to work on SETI packets in the quest for extraterrestrials, it doesn't mean that developers no longer have to worry about the speed and efficiency of their code. In the old days of Win32® native development, we not only worried about speed but about those nasty access violations (and Global Protection Faults and Unrecoverable Application Errors for you old-timers) on the PC platform. While managed code has eliminated some of the worries, it just means that those performance issues you are experiencing may be a little more insidious than before. The main reason is that with managed code, we don't have the easy views into the runtime that we had in the native days.

Many times I have been working with a client, wondering how I'm going to tackle a vicious performance problem. Of course, none of these performance problems show up on any testing systems; they only show up in real-world production. Since the common language runtime (CLR) is a black box, it's pretty hard to divine what's going on if I want to begin to figure out ways to duplicate the performance problem on a test system. While there are third-party commercial performance tools on the market, most of those tools are too intrusive to even consider using on a production system. That's why I was thrilled to see that Microsoft will be delivering a brand new profiler, the Enterprise Performance Tool (EPT), as part of Visual Studio® 2005 Team Developer Edition. It's the first profiling system that I can really consider using on a production system because it offers some very lightweight means of collecting your performance data. Having led the development of a best-selling commercial profiler, I can appreciate just how hard it is to collect useful profiling data without much overhead.

In this article, I want to introduce the philosophy of EPT and show you how to get started using it. Because of the complexity of a profiler, in a future issue I will discuss how you can use EPT to track down real-world performance problems you may encounter in your co-worker's code (I know your code is perfect!). Please keep in mind that EPT is in beta (I'm using the Burton Beta 1 Refresh bits, version 40607.83) and there may be changes to the UI or to some particular steps before the product ships. Before I jump into EPT itself, I want to spend a little time talking about how profilers work in general so you can better understand what makes the Enterprise Performance Tool so special.

The Philosophy of Profilers

When you are writing a profiler, you can choose from two ways of collecting data: probing and sampling. Both are perfectly valid, but each has its trade-offs. A probe profiler collects data by inserting probes or hooks into the application so the profiler runtime is called whenever the program executes that hook. To place the probes, the profiler needs to instrument the app in the compilation step, rewrite the compiled binary, or instrument the application on the fly. To see an example of a probe profiler approach for .NET-based applications, check out Aleksandr Mikunov's phenomenal article, "Rewrite MSIL Code On the Fly with the .NET Framework Profiling API" from the September 2003 issue of MSDN®Magazine. When I get around to the discussion of EPT itself, you'll see it uses the term "instrument" to indicate the probe approach.

The key benefit to the probe profiler approach is that the inserted probes will always be called when the application executes. This way the profiler runtime will have a complete picture of the run, so building key information such as parent-child relationships between functions is guaranteed to be correct and the profiler can report perfect call trees necessary for you to easily find the call paths that took the longest time. With probe profilers, there's nothing stopping the developers from only inserting probes at the function entry and exit. Additional probes can be placed down in the source-line level so you have a complete picture of the function.

However, some drawbacks come with the detailed view provided by a probe profiler. The first is that the instrumentation scheme can be cumbersome to use and since it's rewriting at the binary level, there's plenty of areas to introduce potential errors. As you can imagine, those probes also take up space, resulting in some code bloat and slower performance. With a fully instrumented application, probe profilers can cause so much of a slowdown that it is almost impossible to run the instrumented binary on a production system, so you truly can't take advantage of the profiler when you need it most.

Sampling profilers, as the name implies, take a snapshot of what's executing in the application at predefined intervals. Most developers don't realize that Microsoft has always shipped a sampling profiler with their development environments. It's called the debugger! If you start debugging your application and every 30 seconds or so break into the debugger, you can note where the application threads are executing to get a pretty good idea of what your app is doing over a run. I've solved many production performance problems by manually doing the work of a sampling profiler.

What makes sampling profilers so valuable is that they have far less overhead than probe profilers. That means you stand a much greater chance of using them on a production system without grinding the server to a halt. The problem with sampling profilers is that it's entirely possible that all the samples taken of your application could show no code at all. For example, if you have a database-intensive application, all the samples could be from inside the database access assemblies. Finally, traditional sampling profilers that only grab the currently executing instruction for each thread make it much more difficult to determine the parent-child relationship between methods, so determining the worst performing code path is much more difficult.

The Philosophy of the Enterprise Performance Tool

With an understanding of how profilers operate, I can discuss the tack taken by EPT. Simply put: it's both a sampling and a probe profiler (which Microsoft calls instrumentation). The idea is that you'll start looking at your application performance with the sampling profiler to get the general performance characteristics so you can start focusing on the hotspots in your application. Once you have an idea of the assemblies that have some issues, you can turn to instrumented profiling to look at the specific problem areas in order to fix them. Of course, if you're doing unit performance testing, there's nothing stopping you from jumping right to instrumenting just the particular modules to look at their performance in a focused scenario.

Part of what makes the EPT sampling profiler interesting is that there are numerous items which you can use to trigger a sample. The default sampling point is clock cycles and is probably what you will almost always use. The default is to take a sample every one million clock cycles, but you can change the sampling number to any value you'd like, though the smaller you make it the more overhead EPT will incur. For your production servers, you can set the number to something quite high, like five million, to keep the overhead reasonable without completely destroying usability in the process. As you would expect, sampling every five million clock cycles will mean you need to run your application for quite a while to get a good distribution of samples across your hot spots.

If you have applications that are using lots of memory, you can choose to have the EPT sampling profiler trigger on page faults instead. That way you'll get your performance snaps when data is being swapped out of RAM, and you can see who's doing the pushing. If your initial profiler runs indicate that you are spending a great deal of time in areas outside your code, you can tell the profiler to do its sampling based on system calls instead. If you are profiling a heavily multithreaded application, this sampling statistic will snap the data when you are transitioning from user mode to kernel mode, which indicates that you have threads that are potentially blocking on kernel objects more than necessary. The final values you can use for the sampling trigger are the various performance counters supported by the CPU, such as branch counts or cache misses. This is an advanced option that very few of you will really need, but if you do need that data, it's nice to know it's there.

Those busy Redmontonians also solved the call stack problem—one of biggest problems standing in the way of a useful sampling profiler. As I mentioned earlier, most sampling profilers simply snap the currently executing instruction when taking a sample. Microsoft figured out how to incorporate a superfast stack walk into their sampling profiler portion so that you'll get the benefits of the sample with the benefits of knowing how you got there at the time of that sample. That makes correlating those snaps back to your code infinitely easier.

Before I discuss an application you can profile, I want to mention a few things that you'll hopefully find interesting. The first is that if you think that Microsoft has started from scratch on the performance tool, you're only partially right. Inside Microsoft, development teams have been using the predecessors of EPT, called Call Attribute Profiler (CAP), which used instrumentation, and the Low Overhead Profiler (LOP), a sampling profiler. Since Microsoft developed these tools to collect performance information on applications such as the operating system and the entire Office suite, they shouldn't even break a sweat with your application. Having used the predecessors of EPT, I can tell you how much easier the public versions are to use. And they have some extremely interesting features, which I'll discuss shortly.

The second item of interest has to do with the technologies supported by EPT. While some of you may be thinking that you won't be able to use EPT with your Win32 application or native code since Microsoft is very focused on the .NET Framework, the EPT team has in fact committed to supporting all Win32 native applications along with .NET code. That means that you have full sampling and instrumentation support no matter what technology you use, be it ASP.NET, Windows® Forms, MFC, or Win32 Services. You'll see that from Visual Studio .NET there's no difference in using EPT across technologies.

The actual EPT setup is trivial; simply select Enterprise Performance Tools from the Enterprise Tools tree control in the Visual Studio .NET setup. Of course, since you know that EPT is still a beta product your immediate reaction might be to run Virtual PC and safely contain everything there. However, in order to do the sampling profiling, EPT uses a kernel mode device driver to respond to the CPU performance counter interrupts and, unfortunately, Virtual PC does not implement the counters. It also does not emulate an Advanced Programmable Interrupt Controller (APIC), both of which are necessary for the kernel device driver to do its magic. The good news is that if you don't have an extra machine on which to install EPT, you're not totally out of luck because the instrumentation profiler still works. If you don't have a spare machine on which to install EPT, that's a good excuse to get your company to buy you another machine.

Animated Algorithm

To learn about any tool, you need a decent sample application that will allow you to best utilize that tool. At this point in the beta cycle, EPT does not come with any samples, but I had a perfect profiler sample already on my hard disk. A while ago, I was trying to figure out how to use multithreading in a Windows Forms application, so I wrote a great little program called Animated Algorithm that animated numerous sorting algorithms in real time. Figure 1 shows my sample application ready to sort.

Figure 1 Animated Algorithm in Action
Figure 1 Animated Algorithm in Action

Animated Algorithm lets you choose from among 15 different sorting algorithms in the combobox on the form. The Options menu lets you choose the sleep time between each element swap or set so you can slow down the graphical update.

I wrote Animated Algorithm a while ago using the Microsoft® .NET Framework version 1.1 so you won't find any fancy generics or the new BackgroundWorker item in the code. The sorting algorithms in the NSORT assembly came from an outstanding article posted on The Code Project (see Sorting Algorithms In C#) by Jonathan de Halleux, Marc Clifton, and Robert Rohde, which encapsulates the algorithms in a common structure so you can easily replace the class that does the element swapping and setting. Because of their very nice architecture, all I had to worry about was the UI portion.

For the rest of this article, I'll profile the Animated Algorithm program. It would be really cool if the EPT team shipped this as a sample application with the product. (Hint, hint.)

Getting Started with the EPT

In Visual Studio 2005 Beta 1, it certainly isn't obvious where you'll find EPT. EPT starts when you start the Performance Wizard, which you can find under the Tools menu, and is present both with and without a project open. Keep in mind that the Performance Sessions created by the Performance Wizard are not part of your projects; they're actually separate files that have their own IDE window called Performance Explorer. You can open the Performance Sessions you create by selecting the PSESS files from the File | Open dialog.

If you do not have a project open when stepping through the Performance Wizard, the resulting Performance Session is associated with the binary you specified. However, in the beta, you must have the associated project open when you specify a binary to run. I just wanted to mention this little twist because it confused me the first time I ran into it.

Once you start the Performance Wizard, the first screen you're presented asks you to choose the application you want to profile. If you have a project open that generates multiple assemblies (like Animated Algorithm), you can only pick one from the Wizard. If you want to do sampling, it's fine to pick just the one assembly because EPT sampling profiles all loaded assemblies (including those from the Framework Class Library). However, if you want to do instrumentation profiling on multiple assemblies, the Performance Wizard only selects the one assembly, so you'll need to specify the other projects or assemblies in the resulting Performance Session in the Performance Explorer. I'll show you how to do that in just a moment.

After you've chosen the assembly or project that you want to use in your Performance Session, you have to pick the profiling method. At any point in Performance Explorer you can switch between sampling and instrumentation to your heart's content; the choice you make in this page of the wizard indicates only what you want to do initially. After selecting the profiling method, the wizard is basically complete. For the final release of EPT, you'll have more options for specifying additional information in the Performance Wizard. The final release will also let you create performance sessions directly from the Performance Explorer.

Figure 2 shows the window of Performance Explorer right after completing the Performance Wizard steps to create an instrumented run of the AnimatedAlgorithms project. To add the output binaries for the other project, right-click on the Targets folder and select the Add Target Project from the context menu. If you want to add specific binaries not associated with the project, select the other option, Add Target Binary. If you've selected Add Target Project, you can choose the other projects from the open solution in the resulting dialog.

Figure 2 Performance Explorer
Figure 2 Performance Explorer

If you've selected an instrumented run, which is denoted by text in the dropdown listbox below the green start arrow, the binary instrumentation takes place before the program executes. If you don't want to instrument a particular binary for a run, right-click on the binary and uncheck the Instrument Binary menu option.

If you've selected sampling profiling and you want to attach to a running project, clicking the Attach/Detach button, the diagonal arrow to the right of the Start button, will bring up the Attach Profiler to Process dialog. With EPT you can attach to as many processes as you need in order to get a picture of your application. The Attach Profiler to Process dialog also lets you detach profiling from specific binaries. In a future issue of MSDN Magazine, I'll be talking much more about attaching to existing processes, especially for ASP.NET performance tuning.

The last button at the top of the Performance Explorer window is the ubiquitous Properties button. Before you start your profiling runs, you'll probably want to make a trip into the Performance Sessions properties to set a few key properties. The first is on the General tab and is the location where you'll want Performance Reports stored for the Performance Session. When profiling projects, the default is to store the reports in the same directory as the solution. But it's better to put your Performance Sessions and their corresponding reports into their own directories so you can more easily store particular sets of runs. That also makes it easier to analyze before and after scenarios to see the impact of the code changes you're making.

On the General tab you can also flip between instrumentation and sampling profiling, which changes the value displayed in the Performance Explorer. In my performance tuning, I like to keep a particular session devoted to a single type of profiling so there's no confusion surrounding the reports. There's nothing stopping you from creating hundreds of different Performance Session files for all sorts of particular scenarios from profiling type to single binary instrumentation. I'll whet your appetite a bit by mentioning one last item on the General tab, the very enticingly named Managed Allocation Profiling. After I get through general profiling, I'll come back to that one.

Another interesting tab on Performance Session property page is the Sampling tab (see Figure 3). Here is where you tell EPT what type of sampling you want to do. As I mentioned earlier, you have tremendous control over exactly how you want to sample.

Figure 3 Various EPT Sampling Counter Options
Figure 3 Various EPT Sampling Counter Options

When it does a profiling run, EPT instruments the binaries in places where they exist on the hard disk. If you'd like to have the instrumented binaries moved to a different location, select the Binary tab in the Performance Session property page, then check the Relocate Instrumented Binaries (which has absolutely nothing to do with REBASE-style relocation), and specify where you want the changed binaries to go.

The Instrumentation tab allows you to specify programs you want to run before and after instrumentation takes place. This can be helpful if you need to perform additional tasks on the instrumented binary such as moving it into the Global Assembly Cache or to a specific location on the Web server. The Advanced tab is undocumented in this beta release. Finally, with the Counters tab you can tell EPT to collect additional data from the CPUs in the system such as L2 or L3 Cache read misses. Obviously, these are very advanced options that only a few developers will need, but if you do need them, they can make all the difference in the world.

Before I move on to viewing Sampling data, I want to mention that the Performance Explorer window can open as many Performance Sessions as you need. This is extremely helpful when you want to look at specific before and after scenarios or would like to perform separate test runs with different instrumented binaries. When you have multiple Performance Sessions open, you should make sure to right-click on the particular session and select Set as Current Session to have that session's setting execute and the reports filed in its report node.

Viewing Profiler Data

After getting your Performance Session set to what you want to do, it's time to start profiling. I'm going to start by performing a sampling profile on Animated Algorithm to see if I can find some hotspots. The key to getting good data out of sampling is to do a long run. For Animated Algorithm, I'll run each of the 15 sort algorithms twice and I'll leave the sampling set to the default one million clock cycles.

After you've completed a run, EPT puts that run's report in the Reports folder of the Performance Session. EPT collects raw performance data and streams it out to the report file during the run with no analysis. That way you avoid all that overhead when running your application, but you will pay the price in large report files. The sampling report file for the run I just finished is 3.70MB and it took about three minutes to complete. Make sure you have lots of disk space when running EPT.

All the data analysis, which entails building the call stacks and calculating the performance numbers, occurs when you open the report file itself. For the beta, there can be some slowdowns in opening the file. It may look like the view is in an infinite loop, but if the progress bar is moving in the report window, just be patient and the file will eventually pop up.

The first view in any profiling run is the Performance Report Summary, shown in Figure 4 for the Animated Algorithm sampling run just completed. As expected, sampling takes place across the whole application so what you're looking at is what you'll see in your applications as well: most of the work takes place inside the Framework Class Library or the operating system. If you do see one of your methods in the sampling Summary view, you are probably looking at a performance problem.

Figure 4 EPT Sampling Performance Report Summary
Figure 4 EPT Sampling Performance Report Summary

Taking a quick glance at Figure 4, you're probably wondering about the difference between Inclusive Sampled and Exclusive Sampled. Exclusive Sampled means that the method was at the top of the stack when the sample was taken. In other words, it's the function that's currently executing. Inclusive Sampled means the function appears in the call stack when a sample is taken. Thus, the inclusive method is a caller of the currently executing method.

In a sampling scenario, the more often a method appears in the call stack (Inclusive Sampled), the more time that function is spending in execution, so that's where you want to focus your attention for tuning. In the case of Exclusive Sampled functions, a function that appears there frequently indicates that the function is being called quite often, but its execution may actually be quite fast. For a graphics-heavy application like Animated Algorithm, I fully expect that something from GDIPLUS.DLL will be near the top of the list just as shown. In Figure 4, the function at the offset 0x5B8D in GDIPLUS.DLL , which happens to be the FLOOR function, is called all the time to calculate where to display something on the screen. When you look at your performance runs, make sure to set up your symbol servers to get the best information possible. In writing this article, I was using an unreleased version of EPT and the symbols were not available.

Before I jump into the other views, I want to instrument Animated Algorithm and do the same run I did for the sampling profiler in order to show the Performance Report Summary for an instrumented run. As you can guess, instrumented runs generate much more data than sampling runs. For the run, I instrumented all five assemblies in Animated Algorithm and ended up with a 375MB session file.

The main difference between sampling and instrumentation data is that sampling looks across the whole process space and will show calls inside the Framework Class Library or operating system (in other words, places where you don't have source code). Instrumentation, on the other hand, only looks at the application and the methods you directly call on non-instrumented modules. For example, if you have a "Hello World!" application whose Main only calls Console.WriteLine, you'll get timing information for any work in Main as well as timing information for the length of the Console.WriteLine, but you won't get any detail into the Console.WriteLine method.

Figure 5 shows the Performance Report Summary for the instrumented run. The first table, Most Called Functions, shows the heavily used functions. The first column in the table is mislabeled as time; it really represents the number of calls to the function. The percentage column shows the percentage of total calls that were made to this particular function. In most runs, you'll see Framework Class Library or operating system functions here. If you are seeing some from your own code, you'd better understand why you are calling that particular function so often.

Figure 5 Summary for Instrumented Run
Figure 5 Summary for Instrumented Run

The Functions with Most Individual Work table lists those methods that take the most time just to execute the function, without any additional function calls. This is also known as the exclusive time for the function. For the beta, the Time column unit is clock ticks. For the final release, the units will be milliseconds. However, I think that the actual raw units for a performance run are not useful for analysis. The most important numbers are the percentages. When looking at performance problems you want to know which method took the longest as compared to all the other methods in the application. You can't look at two numbers like 3519639455 and 3492589504 with any hope of comparing them. Fortunately, the table includes the percentage, and my vote is for the EPT team to drop the raw data from the chart.

The last table, Functions Taking Longest, shows what's sometimes called wall time, stopwatch time, or elapsed time for a method. The profiler records the time at the entry point and exit point of a method and subtracts the values. This number accounts for all child methods that are called, all context switches, and sleeping done by the method. In Figure 5, you can see that System.Windows.Forms.Application.Run took the longest, as you would expect for a Windows Forms application. While many developers concentrate on the exclusive time, that's only a small piece of the whole performance picture. If your method is calling out to a database or making a Web service call, the thread your method is running on will block while waiting for those calls to return data, thus moving the thread off the CPU. By keeping a close watch on the elapsed time for your method, you'll find the parts of your code that are slowing down your application.

While the Summary views are nice, you'll be most interested in seeing where your code stacks up with the rest of the system for sampling runs or against the other methods in your application for instrumentation runs. This is the bailiwick of the Function view, which is selectable by clicking on the Function button on the bottom of the report window. You can double-click on any method in a Summary view to jump to the Function view as well.

For sampling runs, the Function view shows you the list of all function captures in at least one inclusive capture. For instrumented runs, you'll see all the instrumented methods called as part of the run. No matter what type of profiling you are doing, there's quite a bit of data displayed in the Function view so you can get a handle on how your code looks.

The sampling Function view shows the Inclusive Samples and Exclusive Samples columns by default. Since I like the percentage numbers, I right-clicked on the column headers to add Inclusive Percent and Exclusive Percent to the column headers as well. If you are sampling a multiple-process system, you may want to include other columns such as Process Name or Process ID so you can identify which method sampling goes with which process. You can also set column headers in the instrumented Function view, but you'll have a different set to choose.

When analyzing a sampling run in a Function view, I like to glance at the first couple of pages in the Function view sorted by the Inclusive Samples column to get an idea about what's been executing. If I don't see any of my methods in the first couple of pages, I right-click in the Function view and select "Group by Module" so I can get the tree report view. Sorting by a particular column will sort correctly when you have the functions grouped by module—a nice feature.

For instrumented runs, the Function view has many more columns to display. If you happen to have a 40-inch monitor, you should be able to see them all without maximizing the Visual Studio .NET window. For the rest of us, the best way to look at the Function view is to press Alt + Shift + Enter for full-screen mode.

In the columns, instrumented runs in the Function view use the "inclusive" and "exclusive" terminology that I explained earlier. However, there's another term thrown in the mix: Application. The elapsed time, as I mentioned, is the total time from one instrumentation point to another, regardless of any context switches the thread may make. The idea with Application time is that EPT will extract out the time spent in those context switches so you are seeing the time that your code was actually executing on the CPU. Figure 6 lists the definitions of the not-so-obvious columns you'll see in the instrumented Function view. You may want to tape it to your monitor until the help comes online for EPT.

Figure 6 Instrumentation Function View Columns Definitions (Close Figure 6)

Instrumentation Function View Column Name | Definition
Elapsed Exclusive Time | The execution time for the code in the method only (excludes all child function times). This includes the time the method blocked or was otherwise moved off the CPU (context switch).
Application Exclusive Time | The runtime for just the code in the method. This excludes all child methods called and any time spent off the CPU.
Elapsed Inclusive Time | The execution time for the code in the method, all child methods called, and all context switches that occurred. This is just like starting a stopwatch at the start of a method and stopping it at the end.
Application Inclusive Time | The execution time for the code in a method and all the child methods called, with no time added for when the thread was off the CPU.
Exclusive Transitions | The number of kernel mode transitions caused by the code in the method only.
Inclusive Transitions | The number of kernel mode transitions caused by the code in the method and any child methods called.

When looking at the Function view for an instrumented run, I add the columns to see the percentage values of the various timings, remove the raw number time columns, and add the two transition columns. That gives me a clearer view of the run. The first column I sort on is the % Application Exclusive Time as I want to see which function is doing most of the work. Since the instrumentation puts in probes around all child calls a method makes, it's entirely possible that you'll see a Framework Class Library or operating system on top of the list. In fact, with my Animated Algorithm run, System.Drawing.SolidBrush.ctor and System.Drawing.Brush.Dispose are listed as number one and two in the percentage of Application Exclusive Time, with 14.982 percent and 14.867 percent, respectively. The first function I wrote was the third Bugslayer.SortDisplayGraph.SorterGraph.UpdateSingleFixedElement at 12.217 percent, which draws an individual bar on the graph. Depending on the application type, I may choose to sort by other columns when looking in the function view. If there are Web service or database calls, I'd be looking at the % Elapsed Inclusive Time so I could see if there were any particular methods involved in blocking for long periods of time. For an application like Animated Algorithm, I'd also look at the percentage of Application Inclusive Time.

Based on those numbers in my instrumented run, I was curious to find out who was making those calls to the SolidBrush methods, so I right-clicked on the .ctor method and chose Show in Caller/Callee view so I could look at who was calling that method. This view, which is also available for sampling profiles, lets you see at a glance all the callers to a target method and all the methods that the target calls.

Because the .ctor method was not instrumented, the Caller/Callee view will not show any callees, but it will obviously show the callers. I double-clicked on the only caller, which happened to be the UpdateSingleFixedElement method that had the third highest Application Exclusive Time and had the view shown in Figure 7.

Figure 7 EPT Caller/Callee View (Close Figure 7)

In Figure 7, the dropdown combobox in the middle of the view is the target method, in this case UpdateSingleFixedElement. The grid above the method contains all the callers of the target method (the callers). The grid below the target method contains all the methods the target method called to do its work (the callees). If you want to look at who called a particular caller, you double-click on that caller method, it becomes the target method, and you'll see the original target method drop down into the callees section. In essence, you just have walked up the stack once.

Just based on the view in Figure 7, you can probably pick out a potential performance problem. Animated Algorithm didn't seem to have any glaring performance problems, but the fact that the SolidBrush .ctor and Dispose are taking up so much time and are both being called inside a method, UpdateSingleFixedElement, (that's called 351,872 times) indicates I'm doing something stupid by creating that brush every time through that function and should be caching it off. As I get into analyzing code with EPT in a future issue of MSDN Magazine, you'll see some other problems with Animated Algorithms as well.

The last of the common views into your data is the Callstack view. Here is where you can see, in a more hierarchical manner, the call stacks you looked at in the Caller/Callee window. For sampled runs, you'll see quite a few entries at the top level in the Callstack view because each of those represents a unique point where there was an exclusive sample. As you expand items in a sampling run, you'll also see that occasionally there are items at the same level that indicate that the function at the root had multiple call trees leading back to it. The item shown in the root position is the top of the stack.

For instrumented runs, the Callstack window will have root elements for each thread in the application. As Animated Algorithm only has two threads, you'll only see two items at the tree-root level. In the Callstack view you can see the absolute call stacks, from the first method instrumented down to the last so you get a real picture of how your application executed. Many times I've been surprised at the gap between what I thought the code did and what it does in reality.

You can spend a great deal of time analyzing your code in the Callstack window. When looking at a particular trail through your application, you can get rid of a lot of noise by selecting a particular node you're interested in going down, right-clicking, and choose the Set Root menu option. In Figure 8 I wanted to look at all the calls made by NSort.SwapSorter.Sort, so setting it as the root got the UI thread out of the way.

Figure 8 EPT Callstack View (Close Figure 8)

In a future issue I'll talk more about the final two tabs in the EPT display area, Trace, and Type. The Type view, is where you can look at the objects you've allocated in an application. It works in the beta version. Back when I was discussing the Performance Session properties, I mentioned on the General tab that there's a Managed allocation profiling section. If you select the Allocations-only radio button, EPT fills in the type view. In the beta, the report looks similar to the reports in numerous other tools, but the data collection does not seem to have as much overhead as I've seen in other tools. Finally, to understand what the Enterprise Performance Tool team is thinking and to learn more about it, make sure to check out their blog at blogs.msdn.com/profiler.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in Windows and the .NET Framework. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at www.wintellect.com.


Editor's Note: Going Mobile

Editor's Note
Going Mobile

A couple of times a week, we are the recipients of advertising e-mail from a popular retailer of mobile products. Here at MSDN Magazine, we field a few dozen publicly available e-mail aliases, so it's easy for mail like this to get lost in the shuffle of our Junk E-Mail folder. (The "Add Sender to Safe Senders List" option seems to exist primarily to satisfy the user's clicking fixation.)

So anyway, we get this mail a couple of times a week and it's one of the first things we open. Even more impressively, it's one of the only e-mails of that type that we allow Outlook to retrieve the pictures for. Advances in the mobile computing market are coming incrementally, yet regularly, and every week there seems to be a cool new gadget to whet our appetites.

But some of these devices seem a bit contrived. For instance, do you really need a handheld device that can pick up digitally transmitted music from your PC in true FM quality? We seem to remember a prototype of a similar gadget some years ago. It had a simple, two-dial interface—volume and tuning—and could actually play sound out of a small, tinny-sounding speaker attached to the unit. They had a weird name for it—a "radio." It cost $9.99, well out of reach for the average consumer. But why go that route when you can now pay hundreds for a handheld that does the same thing and only works within 100 feet of your computer? The world just doesn't appreciate the progress that's made every day.

Seriously though, there are some incredible products available in the market now. You can get a 1GB portable memory card for not much more than $100. At the risk of sounding like Crazy Grandpa Editor, we remember just 15 years ago we walked into the local Egghead and plunked down $450 for a shiny new 20MB hardcard—a hard drive welded onto an interface board. It was guaranteed to almost fit in your PC case if you left the next slot empty. Still, it was a vast improvement over the alternative, which at the time was speaking the words "one" and "zero" onto a cassette.

Much cooler stuff exists today. We recently test-drove a GPS device by Pharos that plugs into the CF slot of a Pocket PC or can use its built-in Bluetooth capabilities to communicate with the device. Being geeks, the first thing we tried was not the built-in software, but our own .NET Compact Framework-based GPS reader. Since only five people on earth seem to know how to program for Bluetooth, we started by frantically tearing through the documents to find out how to communicate natively with such a device. It turns out that our concerns were unnecessary—Bluetooth is actually translated into serial port output, so any functionality you base on a serial comm library works with a device like this.

Which brings us back to mobility, this month's theme. One of our most popular articles in the past five years has been "Serial Comm: Use P/Invoke to Develop a .NET Base Class Library for Serial Device Communications" by John Hind in the October 2002 issue. True, the .NET Framework 2.0 includes serial port functionality in its System.IO.Ports namespace, and Visual Basic 2005 makes it even more accessible through the My namespace, but the article and its accompanying code has been among the most widely downloaded pieces we've published. So, are you wondering where technology like this has been used in the real world? John demonstrates a robotic vehicle built atop .NET-based serial comm libraries in his article in this month's issue.

It's like old friends month here. Paul Yao talks about Tablet PC programming. Dave Edson, the former technical editor of MSJ, marks his triumphant return to print with some great information for any developer who uses the .NET Compact Framework for their Pocket PC development. True, it lacks the excitement of spending a couple hundred dollars to listen to your local Q Morning Zoo on the same device on which you can store the phone number you use to complain to the station about the Q Morning Zoo, but everyone has their own level of excitement.

—J.T.

Thanks to the following Microsoft technical experts for their help with this and the previous issue: Deborah Annan, Siddharth Bhatia, Justin Brown, Chris Brumme, Susan Chory, Jamie Cool, John Cunningham, Dave Fetterman, Matthew Gibbs, Frank Gocinski, Arin Goldberg, Robert Green, Cindy Gross, Martin Gudgin, Shawn Hernan, Chris Hollander, Michael Howard, Charlie Kaufman, Raja Krishnaswamy, Thomas Krueger, Ronald Laeremans, John Lambert, Gerrard Lindsay, Cory Linton, Steve Lipner, Duncan Mackenzie, Arun Moorthy, Satyajit Nath, Brian Pepin, Matt Powell, Matt Thomlinson, Anson Tsao, Dan Vallejo, Maura Van Der Linden, Ravinder Vuppula, Ting-Hao Yang, and Santosh Zachariah.

ActiveX, IntelliSense, MapPoint, Microsoft, MSDN, Outlook, Visual Basic, Visual C++, Visual Studio, Windows, Windows NT, Win32, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Marnie Hutcheson

Contents
Manage Your Personal Information
Store and Organize Your Confidential Data
Read and Write Word and Excel Files
Real-Time Access to Enterprise Data
Mobile and Wireless Development Environment
Give Presentation Capabilities to Your Mobile Apps

There is more to fielding a mobile device than just putting a couple of company-specific applications on it. Here are some already written mobile applications that might be good to know about if you are tasked with deploying a mobile solution.

Manage Your Personal Information

Chapura's TurboPasswords is an easy-to-use solution for capturing, organizing, accessing, and sharing your personal identification information in a secure source protected by a master password.

The TurboFill feature automatically fills in Web logon forms with user names and passwords, giving you secure, one-click access to your private information while browsing the Web with Microsoft® Internet Explorer.

The Password Generator in TurboPasswords creates random passwords that are less likely to be cracked. TurboPasswords lets you create and edit templates that help you organize your private information and it can be used as a standalone desktop application or synchronized with and edited on a Palm OS handheld device.

PocketCopy, also from Chapura, is a one-click solution for copying Palm Desktop Software records from the Address Book, Date Book, To Do List, and Memo Pad to Microsoft Outlook® Contacts, Calendar, Tasks, and Notes. Just choose which information you want to move into Outlook and click "Copy." There is no need to copy and paste record by record. You avoid the hassle of exporting and importing data from place to place.

Price: TurboPasswords is $29.95; PocketCopy is $19.95.

www.chapura.com

Store and Organize Your Confidential Data

eWallet Categories

eWallet 4.0, from Ilium Software, is a software utility for Windows® Mobile-based Pocket PCs, Smartphones, and Windows operating systems that makes it easy to secure storage and organization of credit card numbers, passwords, health information, PINs, and other confidential information.

This release includes interface upgrades such as a new tool bar and subcategories that make organizing and finding your information easier. You can modify the cards to suit your own needs, and then save them as templates so you can easily create your own custom cards. You can organize your information any way you prefer using the categories and subcategories that are provided.

The interface works well on all the supported mobile devices, but is the most impressive on the Pocket PC and Microsoft Smartphones.

One of the touches I liked the best on the interface is that my credit cards looked like my real credit cards, giving it a personal touch and making things much easier to identify.

You can keep multiple eWallet files, and make changes to your passwords or your cards on your mobile device or on your desktop. The eWallet cards are automatically updated and keep all changes when you sync your devices. eWallet is completely compatible across your Pocket PC, Palm Powered Handheld, Windows Mobile Smartphone, or Windows PC. It is convenient to have the same information on all your devices.

Price: $29.95; free 30-day evaluation is available.

www.iliumsoft.com

Read and Write Word and Excel Files
TextMaker Insert Menu
TextMaker Insert Menu

Take a look at TextMaker and PlanMaker from SoftMaker if you need a word processor and a spreadsheet that can operate on six different platforms (including Pocket PC) and read and write Microsoft Word and Excel files without losing any graphics or formatting.

Both programs now offer full support for the new Windows Mobile 2003 Second Edition operating system. Both programs support high-resolution, 640 ¥ 480-pixel screens introduced with Windows Mobile Second Edition, and landscape viewing of word processing and spreadsheet files. You can see more of the documents at the same time, and landscape view really helps when you are working on spreadsheets.

TextMaker is one of the only desktop-class word processors for Pocket PCs. TextMaker packs a lot of functionality into a mobile device: multilanguage spell-checking, graphics, footnotes, tables, Microsoft Word-compatible forms, character and paragraph formatting and styles, autotext, and safe conversion of Word files right on your Pocket PC.

PlanMaker for Pocket PCs is the mobile equivalent to Excel on the desktop, offering full spreadsheet power on portable devices. As with TextMaker, PlanMaker reads and writes all Excel files without loss of content or formatting.

All the calculations are preserved along with the charts, images, and even AutoShape drawings, and all are editable from inside PlanMaker. PlanMaker offers more than 320 calculation functions, complex numbers, array formulas, conditional formatting, AutoFormat, sophisticated data analysis capabilities, and more.

TextMaker and PlanMaker work on all makes and models of the Pocket PC, handheld PCs, and Linux.

Price: $49.95 each

www.softmaker.com

Real-Time Access to Enterprise Data

mEnable 2.0

mEnable 2.0, by SYWARE Inc., gives mobile users interactive, real-time access to enterprise data, allowing you to run Visual CE database applications on handheld devices and connect directly to any ODBC-enabled database residing on a corporate server.

mEnable 2.0 runs on Tablet PCs (running Windows XP Tablet PC Edition), Windows Mobile Pocket PCs, and all devices running Windows CE, Windows 9x, Windows 2000, Windows NT®, and Windows XP. mEnable 2.0 runs on any connection using the TCP/IP protocol. The network can be based on local Ethernet, Internet protocol, 802.11 wireless LAN, or Bluetooth for short-range wireless in conjunction with a cell phone's long-range capabilities.

mEnable 2.0 allows software developers to create data-driven applications that wirelessly connect Windows, Windows CE, and Windows Mobile Pocket PC devices to corporate servers in either "Sometimes Connected" or "Always Connected" modes.

In Sometimes Connected mode, you read from and write to the enterprise server when the network is available. When the connection is lost, you can work locally. When it is restored, your local database is synchronized with the server databases and you resume your connection to the server. Sometimes Connected mode is useful when you travel into buildings or areas that are out of range of your network. The local database ensures that data isn't lost when you continue to work while disconnected.

In Always Connected mode, you have continuous, real-time access to server data. No local data cache is required. Always Connected mode is the logical choice when you're interacting with a large, complex database, such as an inventory system. The mEnable application provides much more flexibility and better performance than a Web browser-based application. It is also faster and simpler to implement.

mEnable 2.0 works in conjunction with any ODBC-enabled software, including SYWARE Visual CE. Visual CE's drag and drop customization lets you create database and forms applications quickly—just decide what information you want on the form and then drag and drop the controls you need to create a custom electronic form. Little or no programming is required so mEnable can be used by business professionals as well as by developers.

Price: $499, including 10 runtime licenses. Visual CE with mEnable and one mEnable runtime license is $599. A free 30-day trial offer is available online.

www.syware.com

Mobile and Wireless Development Environment

Crossfire 5.1, recently released by AppForge Inc., is a mobile and wireless application development environment that runs on Visual Studio® .NET or Visual Basic® 6.0, and adds support for an impressive list of mobile and wireless PDAs, Smartphones, and industrial devices.

Once installed, Crossfire integrates with your Visual Basic or Visual Studio .NET environment. Building your application is a matter of selecting a new or existing Crossfire project. If it is a new project, you just select the platform on which you want to develop, then build your application using Crossfire's controls. You can develop your application, test it in Visual Studio, then deploy it to your simulator or the device, all from within Visual Studio.

Crossfire 5.1 lets you develop, test, and deploy mobile applications quickly without having to delve too deeply into the device-specific details. It integrates seamlessly with the Microsoft development environment, so your mobile projects work like any other development effort. Crossfire makes it possible to develop device-specific versions of the same application so you can deploy the latest enterprise and industrial mobile devices and still keep your older devices running.

Price: Crossfire 5.1 is $1,000. Customers who are currently using Crossfire 5.0 will receive a free upgrade to Crossfire 5.1. Free 30-day evaluation versions are available.

www.appforge.com

Give Presentation Capabilities to Your Mobile Apps

Studio for Mobile Devices from ComponentOne includes new components for building applications that use the Microsoft .NET Compact Framework.

FlexGrid for Mobile Devices is a lightweight and powerful grid component that gives your mobile device robust data presentation capabilities. ComponentOne Chart for Mobile Devices handles the details so you can easily display information with a wide variety of 2-D charts and graphs in your .NET Compact Framework-based applications.

ComponentOne Zip for Mobile Devices lets you create, open, and manage industry-standard Zip files. You can also add compression and encryption to any .NET stream, including memory, file, and network streams.

Price: Full version is $799.95; upgrade is $549.95.

www.componentone.com

All prices were confirmed at press time and are subject to change.

Send your New Stuff to newstuff@microsoft.com.

Marnie Hutcheson is president of Internet Development Associates (Ideva), a firm in Ocala, Florida, that specializes in Internet and intranet Web app design and development. She has published technical papers and books on various computing topics. You can reach her at marnie@ideva.com.


Web Q&amp;A: Windowed and Windowless Elements, Cookie Characters, and More

Web Q&A
Windowed and Windowless Elements, Cookie Characters, and More
Edited by Nancy Michell

Q I have a windowless JavaScript menu that gets overshadowed by a windowed dropdown list. How can I prevent this?

A All windowed elements paint themselves on top of all windowless elements, regardless of what their container wants. However, windowed elements do follow the z-index attribute with respect to each other, just as windowless elements follow the z-index attribute with respect to each other. See How the Z-index Attribute Works for HTML Elements for more on how this works.

So, one way to deal with this is to convert the menu to a DHTML scriptlet and place this <OBJECT> into a floating <DIV>, like so:

<div id='ctl00_MasterContentplaceholder_newdate_div'
       style='position: absolute;display:inline;z-index:8;' class=''>
  <OBJECT ID='ctl00_MasterContentplaceholder_newdate_obj'
     DATA='/aspnet_client/esupplyctl/ClientTools2.htm' style='z-index:8'
     TYPE='text/x-scriptlet' >
  </OBJECT>
</div>
An overview of scriptlets is available at Introduction to Scriptlets. Plus, the book Instant Scriptlets by Dino Esposito (Wrox, 1998) has lots of information on creating DHTML scriptlets, if you need further details on how to do this.

Another option is to use window.createPopup for your menu, which is also a windowed element. You can use an HTML Component (HTC) and an XML file defining the menu items.

Q I have been getting the following cookie-related error from an ASP.NET page: "A potentially dangerous Request.Cookies value was detected from the client: (W0073355_ID="...icVVWBPvb9OnploMFqAQ==")".

It looks like some combination of characters is raising this exception. How can I change this behavior? There is no need to flag this cookie since there isn't any text after the equal signs.

A As you know, this is a security feature designed to prevent script from being submitted with an HTML form. Although none of the base 64 characters are invalid in a cookie value, there are combinations that are not acceptable, including:

  • <{a-z}
  • <!
  • expression(
  • on{a-z}*=
  • &#
  • script{space}*.

If the HttpRequestValidationException is being raised, then the only way to work around it is to disable request validation for the pages using those cookies. However, if you disable request validation, it is very important that you perform your own validation of any input accepted from the user. The request validation feature, which was introduced in ASP.NET 1.1, is designed to be a first line of defense against cross-site scripting attacks. Without such validation, your application could be vulnerable to such attacks.

Q When I switch from ASP to ASP.NET does that mean I must move to a two-tier architecture from a three-tier architecture?

A There aren't any changes that would limit the number of tiers in your ASP.NET application. That is really a design decision. So no, switching to ASP.NET does not mean that a three-tier architecture must be turned into a two-tier version.

Actually, ASP.NET allows more tiers because you can separate the presentation code (.aspx) completely from the page logic (a .cs or a compiled class in a .dll). At that point you can choose to put your business logic in your .dll in the bin folder, or in another separate .dll, or even in the stored procedures in SQL Server™. You can have as many tiers as you want.

Note that you may want to make sure that you're clear on whether you're talking about physical or logical tiers, however. You can have a three-tier logical architecture while still maintaining a two-tier physical architecture (in order to avoid the cost of cross-process calls between the presentation and business tiers). This is true for both ASP and ASP.NET, but it's an important distinction that is sometimes lost when discussing n-tier architecture.

Q I have a query that returns data to a DataGrid depending upon which items are selected in the dropdown lists. The user can then page through the DataGrid to find the information she wants.

The problem is that when the user clicks on a row in the DataGrid it brings her to a new page, and when she clicks to return to the page with the DataGrid, the dropdowns and DataGrid are back to the original state and the user must re-query, which I don't want her to have to do.

A The best way to deal with multiple queries to the same data is to cache the DataSet in the ASP.NET cache. You can set the cache expiration policy so that it's appropriate for your data (a short expiration period if your data changes a lot, long or non-expiring if your data rarely changes). Then when you need to bind your data to the DataGrid, you can retrieve the DataSet from the cache if it's still there (memory pressure can expire data from cache) or re-query your data if it's not. The Knowledge Base article 323290 has a good example under the heading "Cache arbitrary objects in server memory".

Q I need to package a control I've built into a Microsoft® Installer (MSI) file that will include a number of DLLs, and I need to have the installer run on a user's machine when he visits my Web site. Is there anything special that I need to do to cause the OCX file to be installed and registered to execute with Microsoft Internet Explorer? Can I use the <OBJECT> tag so the user does not have to click on a link?

A For this to work in Internet Explorer, you just need to register the file like any other ActiveX® control to be used with an <OBJECT> tag. That means that in the MSI, the Class table needs to be filled out along with the Typelib table. As an alternative solution, you can use the SelfReg table.

The control itself should support IProvideClassInfo or IProvideClassInfo2 for any event handling via the <OBJECT> tag. The Knowledge Base article 200839 ("How To Enable ActiveX Control Event Handling On A Web Page") provides a further explanation.

It is acceptable to use the <OBJECT> tag if this is outside of the MSI, but users are still going to get prompts due to security constraints. You might consider just having a script redirect to the setup page if the control is not on the machine, using code like that shown in Figure 1.

Figure 1 Redirecting (Close Figure 1)
<HTML>
<SCRIPT language="JScript">

Function Test()
{
  var o;

  try
  {
    o = new ActiveXObject("theobj.theobj.1");
    o.method();
  }
  catch
  {
     alert('Object the obj not available.  Redirecting to setup
        program.');
     window.location = "/setup/setup.exe";   //user will get warning,
                                             //and so on
  }
}

Test();

</SCRIPT>

<BODY>
  Using control on this page, otherwise setting up control...
</BODY>

</HTML>

There is a document on MSDN® that discusses no-touch deployment in the Microsoft .NET Framework, but that solution also requires end-user configuration. (See No-Touch Deployment in the .NET Framework.)

Q I have an ASP.NET application that uses the Internet Explorer Web controls. Each time the user navigates to a new page or does a postback to the same page, the HTC files are retrieved again from the server. I set the Content expiration for the HTC files on the server, but Internet Explorer does not seem to cache them. Is there anything else I should be doing?

A A supported hotfix is now available from Microsoft, but it is only intended to correct the problem that is described in the Knowledge Base article 840312 ("Cached HTC Files Generate Additional GET Requests In Internet Explorer 6"). You should only apply it to systems that are experiencing this specific problem. Therefore, if you are not severely affected by this problem, Microsoft recommends that you wait for the next Internet Explorer 6.0 service pack that contains this hotfix.

Q I'm trying to troubleshoot Web connection problems and want to look at headers. Where can I find a tool to display HTTP headers in Internet Explorer?

A When you are troubleshooting connectivity issues between IIS and Web clients, you may want to view data that is not displayed in the Web browser, such as the HTTP headers that are included in the Request and Response packets. One way is to use wfetch. It's a free utility that comes with the IIS 6.0 Resource Kit and is also available as a standalone app.

You can also use wfetch to provide detailed information about the traffic between the client and server. See the Knowledge Base article 284285 ("Use Wfetch.exe To Troubleshoot HTTP Connections").

While wfetch is a great tool for some kinds of debugging, if you need a tool that integrates with Internet Explorer so you can surf and look at the headers at the same time, check out ieHTTPHeaders.

Q I have a page that shows a DataGrid based on user filters. When the user clicks on an Edit button for a particular row in the DataGrid, I use Response.Redirect to redirect the user to the details page where he updates the data. I now want to take my user to the first page and show him the DataGrid based on the same filters he had selected earlier on. What is the best way to do this? My other requirements are as follows:

  • I want to preserve some values in Page1.aspx, go to Transfer.aspx, and then when I come back to Page1.aspx I want to use those values.
  • I need to use query strings to pass values from Page1.aspx to Transfer.aspx and back.
  • Finally, I'd like to catch the URL of the calling page. I also have to consider performance.

I know I can store filter selections in session state variables and when the user comes back to the page, the DataGrid can use these session variables. However, session variables depend on cookies, so if the user doesn't have cookies enabled, I have a problem.

A One way to solve your problem is to use Server.Transfer and Context.Items. Here's the code you would use in Page1.aspx:

Context.Items.Add("pageval", "value to preserve");
Server.Transfer("Transfer.aspx");
And in Transfer.aspx you would use:
string val = (string)Context.Items["pageval"];

Server.transfer performs a transfer on the server side; the page does not go back to the client in between, so this approach would avoid the 302 page sent to the client which results in an extra round-trip between the client and server.

The HttpContext instance for a request is available while the processing occurs on the server side. The moment that the page is sent to the client, that particular HttpContext is no longer valid. It is not preserved between round-trips to the client. So, if you go to Transfer.aspx and use Server.Transfer to go back to Page1.aspx, the context would still be available. If you let the client see Transfer.aspx, then it would not be available.

Yes, you can use a query string to pass values as well. You can use this method with Server.Transfer and Response.Redirect, which is a good way to pass values back and forth. With Server.Transfer, the client will actually never see the query string, but they will see it with Response.Redirect. There are also ways to encode your values inside the query string so that critical information can be encrypted.

There's one more thing you should know about Server.Transfer. In ASP.NET 1.0, the transfer would be preserved as a POST if that was the initial request. In ASP.NET 1.1, unless the page is transferred to itself, it will always be a GET, so you would have to use the query string because the Form collection is not available anymore.

To catch the URL of the calling page you can choose to use either Request.UrlReferrer.AbsoluteUri (which gets the referring page path and query string) or Request.UrlReferrer.AbsolutePath (which gets just the referring page path).

In the page Load event of Page1.aspx, Request.UrlReferrer will be null because there is no referrer the first time a user types the URL of this page. Thus, any attempts to access properties such as AbsoluteUri will result in an exception. You can avoid this problem by only executing this code in Page1.aspx if Page.IsPostBack is true. You can use something like this:

string referrerUri = (Request.UrlReferrer != null) ?
    Request.UrlReferrer.AbsoluteUri : string.Empty;

The only real problem with using UrlReferrer is that it is not guaranteed to be there even when the page is actually being referred to because some firewalls and privacy software filter out that value. You can tack a query string parameter like "ReferrerUrl=MyPageName.aspx" onto the end of your Response.Redirect call.

For more information, visit HttpServerUtility.Transfer Method and HttpContext Class.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Gaurav Bansal, Seth Behunin, Daniel Boerner, Todd Carter, Richard Ciapala, G. Andrew Duthie, Rob Franco, Scott Golubenko, Bac Hoang, Kenneth Lassesen, Pat Long, Alexander Louie, David Lovell, Nathan Manis, Oscar Newkerk, Polita Paulus, Ranjit Sawant, Ivan S. Shestakov, Martin Sieber, and John Wadie.


Advanced Basics: Calling All Operators

Advanced Basics
Calling All Operators
Ken Getz

Contents
Creating Your Own Overloaded Operators
Some Overloading Caveats

This month I found an interesting question in my mailbag. It went something like this: "I am writing some code working with points and other drawing objects in Visual Basic® 2003, and I just want to add an offset to a point, effectively moving the point. I tried adding a Size to a Point, and the thing just won't compile. I tried the exact same thing in C#, and it works fine. What gives?"

To set the context, it's worth creating some code in Visual Studio® .NET 2003 to see what the behavior is, before attempting a solution (if you try this in Visual Studio 2005, things will go slightly differently). Open Visual Basic .NET 2003, create a new Windows® application, put a button on the form, double-click the button to enter the code designer, and add the following code to the button's Click event handler:

Dim pt As New Point(0, 0)
pt += New Size(10, 10)
It doesn't compile, does it? For fun, try the same exercise, writing the code in C# instead:
Point pt = new Point(0, 0);
pt += new Size(10, 10);
This compiles fine, right? I could go back and forth about why it works in C# and why it doesn't work in Visual Basic .NET, but the whole thing boils down to a simple design issue: namely that Visual Basic 2002 and Visual Basic 2003 don't support overloaded operators. These languages can neither create nor consume operators that have been overloaded—that is, operators that look like standard operators, but that have nonstandard behaviors defined by one of the operator's operands.

If you think about it for a second, you'll agree that any operator, even the lowly + and – operators working on integers, is merely a typographical convenience for methods. For example, when adding two integers, rather than writing the code this way

Dim x As Integer = a + b
you could imagine that under the covers, the compiler is generating code like this:
Dim x As Integer = Int32.Add(a, b)

As long as the operator is defined by the compiler, Visual Basic 2002 and Visual Basic 2003 are fine with it. For example, did you consider the + operator for strings? Although most developers don't write code like this, it's perfectly acceptable and works fine:

Dim str1 As String = "Hello, "
Dim str2 As String = "there"
Dim str3 As String = str1 + str2

Under the covers, the "+" operator for strings has been overloaded to call the String.Concat method, converting that final line of code into something like this:

Dim str3 As String = String.Concat(str1, str2)

Most people don't ever think about things like this, I'm guessing, because it all happens transparently by the compiler, and it makes using the class(es) involved more intuitive.

Now back to the original question. Because C# has supported operator overloading from its inception, and because buried somewhere in the Point structure's code there's a procedure that overloads the + operator and accepts a Point and a Size as parameters, C# developers can use the + operator to add Point and Size structures together, resulting in a new Point at a new location. So, are developers using Visual Basic simply out in the cold if they want to add a Point and a Size together? Not quite.

When you use the + operator on the Point structure in C# code, the C# compiler in turn generates a call to a Shared method on the struct that performs the actual work. In this case, the method is named op_Addition, and you call it explicitly from code in Visual Basic .NET like any other method, as you might expect. To test it, go back to your Visual Basic .NET project that didn't compile and modify your code so that it looks like this:

Dim pt As New Point(0, 0)
pt = Point.op_Addition(pt, New Size(10, 10))

Anyone would agree that this version is slightly more ungainly than the original, but at least this code works. How are you supposed to find this procedure? Check out the documentation for the Point structure. The documentation doesn't really help on its own, but if you click on the Addition Operator link on the Members page, you'll find the information that makes it look like you can use the addition, equality, and other operators. Although you can't use the actual operator symbols in the same way that you can in some other languages, you can call these operator functions explicitly. Given this extra information, when using Visual Basic, developers can call the method although they can't use the operator symbol directly.

If you spend a little more time with the documentation, you'll find that there are methods corresponding to other operators and conversions as well, as you can see in Figure 1. Why are the two conversions listed in Figure 1, and treated as operators? Let's step back and investigate how conversions work. Conversions are treated in .NET much like operators, so if you were to type the following code, whether or not you know it, you're using a widening and a narrowing conversion operator, both of which are built into the definition of the Integer class:

Dim x As Integer = 12
Dim y As Long = x
Dim z As Short = x

Figure 1 Defining Methods that Overload Operators (Close Figure 1)

Operator/Conversion | Method
Addition (+) | op_Addition(Point, Size)
Subtraction (-) | op_Subtraction(Point, Size)
Equality (=) | op_Equality(Point, Point)
Inequality (<>) | op_Inequality(Point, Point)
Point to Size conversion | op_Explicit(Point)
Point to PointF conversion | op_Implicit(Point)

If you have turned on Option Strict (and I do hope that you have), the final line won't compile. The Visual Basic .NET compiler won't convert an Integer to a Short because you might lose information in doing the conversion (going from a four-byte value to a two-byte value). The conversion from Integer to Long is a widening conversion—there's no chance data will be lost. On the other hand, the conversion from Integer to Short is a narrowing conversion—it's possible that data would be lost in such a conversion. With Option Strict on, Visual Basic doesn't allow implicit narrowing conversions like this one, thus the code won't compile.

It's interesting that this sample code only uses one operator. The assignment operator, "=", handles both the widening and the narrowing conversions. In C#, the assignment operator can be used for both Point conversion operations listed in Figure 1. The code isn't quite the same in both cases, however. Performing the widening conversion from a Point to a PointF converts from one type to the other implicitly, but performing the narrowing conversion from Point to a Size requires an explicit cast, as you see in Figure 2.

Figure 2 Narrowing Conversions from Point to Size (Close Figure 2)

C#
// Widening operations happen implicitly:
Point pt1 = new Point(0, 0);
PointF ptf = pt1;

// Narrowing conversions require explicit casting:
Point pt2 = new Point(0, 0);
Size sz = (Size) pt2;
Visual Basic .NET
Dim pt1 As New Point(0, 0)
Dim ptf As PointF = Point.op_Implicit(pt1)

Dim pt2 As New Point(0, 0)
Dim sz As Size = Point.op_Explicit(pt2)

But none of this works in Visual Basic 2002 or Visual Basic 2003 without calling the exposed methods. If I was talking only about those versions, I'd be finished now.

With Visual Basic 2005 looming, things will change in the operator overloading horizon. Visual Basic 2005 adds support for overloaded operators, making it possible to both consume and create overloaded operators. Although it's unlikely that you'll do this on a daily basis, it's nice to know that you can.

If you try my first code snippet using Visual Basic 2005, you'll notice that even with Option Strict on, the compiler doesn't complain. Try the following code, which is similar to the previous C# snippet, and you'll see that this works, as well:

' Just as in C#, a widening conversion can be implicit:
Dim pt1 As New Point(0, 0)
Dim ptf As PointF = pt1

' Just as in C#, a narrowing conversion must be explicit:
Dim pt2 As New Point(0, 0)
Dim sz As Size = CType(pt2, Size)

Creating Your Own Overloaded Operators

The canonical operator overloading example, in just about every text I've seen, is the complex number demonstration. You can find a clear description of operator overloading, using that example, by reading the excellent article by Matthew Gertz entitled "Operator Overloading in Visual Basic 2005". For this column, however, I'll work with an array of Integers in a class I'll call Vector. This is a simple example that should be relatively easy to figure out, and to be honest, it's highly unlikely that you're going to need complex numbers in many applications you write. You have the power to overload any of the operators listed in Figure 3.

Figure 3 Overloadable (Close Figure 3)

+ | + (unary)
- | - (unary)
* | \
/ | ^
& | Like
Mode | And
Or | Xor
Not | <<
>> | = (comparison)
<> | <
<= | >
>= | CType
IsTrue | IsFalse

You should note the following important facts about operator overloading, in general:

  • Only the operators listed in Figure 3 can be overloaded. You can't overload member access (.), method invocation, or any of AndAlso, OrElse, New, TypeOf...Is, IsNot, AddressOf, or GetType. Using the = sign as an assignment operator isn't overloadable either.
  • Although you generally can overload any individual operator, some work in pairs, specifically, the logical operators. If you overload =, you must overload <>. If you overload <, you must overload >, and so on.
  • Operator overload procedures must be in the same class as one of the operator's operands, declared using both the Public and Shared keywords.
  • Overloads of CType must indicate whether they're widening or narrowing conversions. The widening CType overload defines the assignment operator's overload. (That is, you must use CType when performing a narrowing conversion, but can use an assignment directly when performing a widening conversion. This technique allows you to effectively overload the assignment operator.)

In order to follow along with this demonstration, create a new Windows application in Visual Basic 2005 and add a new class to the project named Vector. In the class, add the code in Figure 4. This class contains a simple array of integers, and you must pass the size of the array you'd like in the class's constructor.

Figure 4 Vector Class Containing an Array of Integers (Close Figure 4)
Public Class Vector
    ' Create a one-dimensional array of
    ' integers that can be operated on.
    Private arrayValue() As Integer

    Public Sub New(ByVal Size As Integer)
        ReDim arrayValue(Size - 1)
    End Sub

    Public ReadOnly Property Length() As Integer
        Get
            Return arrayValue.Length
        End Get
    End Property

    Public ReadOnly Property Array() As Integer()
        ' Return the array of integers.
        Get
            Return arrayValue
        End Get
    End Property

    Default Public Property Item(ByVal Index As Integer) As Integer
        Get
            Return arrayValue(Index)
        End Get
        Set(ByVal Value As Integer)
            arrayValue(Index) = Value
        End Set
    End Property

    Public Overrides Function ToString() As String
        Dim strValue(arrayValue.Length - 1) As String
        For i As Integer = 0 To arrayValue.Length - 1
            strValue(i) = arrayValue(i).ToString
        Next
        Return String.Join(", ", strValue)
    End Function
End Class

Place a button on Form1 within your project, and add the code in Figure 5 to the button's Click event handler. The code creates two instances of the new class.

Figure 5 Creating Two Vectors (Close Figure 5)
Dim v1 As New Vector(5)
' {0, 1, 4, 9, 16}
For i As Integer = 0 To v1.Length – 1
    v1(i) = i * i
Next
MessageBox.Show(v1.ToString())

Dim v2 As New Vector(5)
' {0, 1, 2, 3, 4}
For i As Integer = 0 To v2.Length - 1
    v2(i) = i
Next
MessageBox.Show(v2.ToString())

Given the Vector class as it is, you might want to insert functionality that allows you to add one vector to the standard + operator (obviously, you could extend this concept to other operations, as well). You might also want to be able to compare two vectors and determine if their contents are identical, using the = operator. Finally, you might want to be able to convert from a single-dimensioned array of Integer values into a Vector, or from a Vector back to an array, using the CType operator. Operator overloading makes all of these possible.

To add support for adding two Vector instances, you could add code like that shown in Figure 6 to the Vector class.

Figure 6 Support Vector Addition (Close Figure 6)
Public Shared Operator +( _
        ByVal v1 As Vector, ByVal v2 As Vector) As Vector
    ' Make the output vector the same length as the shorter
    ' of the two input vectors. Differences in length
    ' could be handled in different ways.
    ' This solution is simple.
    Dim len As Integer = Math.Min(v1.Length, v2.Length)
    Dim v3 As New Vector(len)
    For i As Integer = 0 To len - 1
        v3(i) = v1(i) + v2(i)
    Next
    Return v3
End Operator

Given all these overloaded versions of the + operator, you could use code like the following to test your Vector class:

Dim v3 As Vector = v1 + v2
' 0, 2, 6, 12, 20
MessageBox.Show(v3.ToString())

If you want to overload the equality (=) operator so that you can compare two Vector instances, you can add code to your Vector class like that shown in the top half of Figure 7. Note that if you overload =, you must also overload <>.

Figure 7 Overload Various Operators (Close Figure 7)

Comparison Operators
Public Shared Operator =( _
 ByVal v1 As Vector, ByVal v2 As Vector) As Boolean
    ' Compare the size of the two arrays.
    ' If the size matches, compare each element.
    Dim retval As Boolean = False
    If v1.Length = v2.Length Then
        Dim i As Integer
        For i = 0 To v1.Length - 1
            If v1(i) <> v2(i) Then
                Exit For
            End If
        Next
        ' Get all the way through both arrays?
        If i = v1.Length Then
            retval = True
        End If
    End If
    Return retval
End Operator

Public Shared Operator <>( _
 ByVal v1 As Vector, ByVal v2 As Vector) As Boolean
    Return Not (v1 = v2)
End Operator
Conversion Operators
Public Shared Narrowing Operator CType( _
 ByVal intValues As Integer()) As Vector
    ' Convert an array of integers into a vector.
    ' I've indicated that this is a narrowing conversion,
    ' even though it's not, just to show that you must use
    ' CType when performing a narrowing conversion.
    Dim v As New Vector(intValues.GetUpperBound(0))
    For i As Integer = 0 To v.Length - 1
        v(i) = intValues.GetValue(i)
    Next
    Return v
End Operator

Public Shared Widening Operator CType( _
ByVal v As Vector) As Integer()
    ' Convert a vector into an array of integers. This is a
    ' widening conversion, because it can never cause
    ' any loss of data. Because this is a widening conversion,
    ' you can use the assignment operator to perform
    ' the conversion.
    Dim intValues(v.Length - 1) As Integer
    For i As Integer = 0 To v.Length - 1
        intValues(i) = v(i)
    Next
    Return intValues
End Operator

In order to allow for conversion to and from an array of integers, add the overloads shown in the bottom half of Figure 7 to your Vector class. To test the conversion operators, add the following code to the button's Click event handler in your form's class:

Dim intValues1() As Integer = {1, 2, 3, 4, 5, 6, 7, 8}
' Note that narrowing conversion must be explicit.
Dim v5 As Vector = CType(intValues1, Vector)
MessageBox.Show(v5.ToString())

Dim intValues2() As Integer
' Note that widening conversion can be explicit.
' After this, intValues2 will contain the same
' data as v5.
intValues2 = v5

You've now created a Vector class that has overloaded its +, =, and CType operators. I could have taken my own advice from my previous column (see Advanced Basics: Being Generic Ain't So Bad) and used a generic array instead of the Integer array, but I didn't want to confuse the issue at hand here.

Some Overloading Caveats

Lest you think that operator overloading is a trick you're going to use every day, I'd like to make something clear: you'll very seldom overload operators. When you do, they had better make your class more intuitive and easier to use, as opposed to just making you look smart. If it's not immediately clear what is going on when you execute an operation using your overloaded operator, drop it. For example, imagine an order entry system in which you're tracking Customer, Order, and OrderDetail information. Because you often want to display a total of all order amounts for a customer, you need a way to sum the Total field in the Order class across all the customer's orders. You could overload the + operator allowing you to sum the orders:

Dim total As Long = 0
For Each ord As Order In Customer1.Orders
  ' This is really the same as
  ' total = total + ord
  total += ord
Next

For this example to work, you could overload the + operator to accept a Long and an Order as operands, and return a Long as the result. But is this code really any clearer than writing it out the long way, as shown here:

Dim total As Long = 0
For Each ord As Order In Customer1.Orders
  total += ord.Total
Next

I really don't think you've gained any efficiency or clarity, and if anything, the meaning of the code has been obfuscated. In any case, be judicious in your use of operator overloading—it isn't required in most situations.

Don't think that just because some developers are programming in C#, they can't take advantage of your overloading—these features work across languages, and any overloaded operators you create in Visual Basic 2005 should work fine in C#, as well. (The same holds true in reverse, because Visual Basic 2005 can consume overloaded operators.)

Think twice before jumping into operator overloading, but don't be afraid to use this elegant technique when you need it. Until Visual Basic 2005 hits the streets, of course, you'll have to be content with calling the "op_Addition" methods of the world, but once Microsoft unleashes Visual Basic 2005, it's smooth sailing.

Ken Getz is a senior consultant with MCW Technologies. He is coauthor of ASP .NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2001), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Data Points: Efficient Coding With Strongly Typed DataSets

Data Points
Efficient Coding With Strongly Typed DataSets
John Papa

Code download available at: DataPoints0412.exe (191KB)
Contents
The Value of Being Typed
Casting Calls
Derived Classes
Typed DataSets in Enterprise Apps
Important Tidbits
Wrapping Up

Someone once said to me that the hallmark of a good developer is the desire to spend time efficiently. Developers are continually pursuing ways to make coding easier and faster, and to reduce the number of errors. Using strongly typed DataSet objects in ADO.NET can help you do just that.

This month I will discuss the pros and cons of developing a Microsoft® .NET Framework-based application using strongly typed DataSet objects. I will begin by discussing what a typed DataSet is and how it extends the DataSet, DataTable, and DataRow classes. Throughout this installment, I will refer to a sample application which includes the complete code to perform inserts, updates, and deletes on the Orders and Order Details tables of the SQL Server™ Northwind database using strongly typed DataSets. This sample multitier application (which you can downloaded from the link at the top of this article) uses a typed DataSet in several tiers including a class library used as the business logic layer, a Web service, and an ASP.NET Web application. I will wrap up by explaining some of the tips and tricks you can employ when using strongly typed DataSets.

The Value of Being Typed

Remember life before IntelliSense®? Okay, it is not as ground-breaking as the introduction of the Internet, but to a developer like me who is always looking for a faster way to code and an easier way to remember all of the property, method, and event names of an object, IntelliSense has been awesome. One area that hasn't benefited from IntelliSense is the traditional ADO Recordset. For example, in traditional ADO 2.x programming you had to use something like the following Visual Basic® 6.0 code to reference a column's value from a Recordset:

oRecordset.Fields("CompanyName").Value

Of course, to reduce code you could eliminate the Fields collection and the Value property since they are the default properties of their respective objects. However, if you misspell the name of the column, you will not be notified of the error at compile time. Also, if you forget the name of the column altogether, you will have to go back and look at your stored procedure, SQL statement, XML file, or other data source.

Another way to refer to a column in traditional ADO is by its ordinal position, using an index or a custom enumerator. An integer is easiest, but again, you can easily forget which ordinal position the column is in, or even that the columns begin at position 0, resulting in the index being off by one. Using the ordinal position makes readability an issue, too. When you take a look back at your code in a week, month, or year, there is a chance that you may not remember what column 2 represented.

Creating a custom enumerator can help with readability, and it's how I prefer to handle this problem in traditional ADO. However, your columns can still get out of sync if someone changes the order in which the columns are returned to ADO from a stored procedure. All of these are common issues that developers face in traditional ADO programming, and they can happen when using untyped DataSet objects in ADO.NET as well. These are just a few of the problems solved by strongly typed DataSets.

For example, typed DataSets contain additional methods and properties not available to untyped DataSets. Consider the following code snippet to refer to a column in ADO.NET using both typed and untyped DataSets:

//-- Untyped DataSet
string sCoName =
    oDs.Tables["Customers"].Rows[0]["CompanyName"].ToString();

//-- Strongly typed DataSet
string sCoName = oDs.Customers[0].CompanyName;

The typed DataSet code is shorter, requires neither the table name nor the column name to be accessed via a string or ordinal position, and it does not require the Rows property. You can use IntelliSense to get a list of the table names available to the typed DataSet as well as a list of the column names available to its tables, so the code is really easy to write. Notice that in the untyped DataSet code, the method ToString must be used to retrieve a string representation of the value contained within the column. In the typed DataSet, the columns are all defined as properties with their respective datatypes. As a case in point, the CompanyName property is defined as a string datatype, which makes the ToString method unnecessary. Typed DataSets make binding to controls easier since they contain the schema information within them. When binding a typed DataSet to an ASP.NET DataGrid, the properties menu reads the selected typed DataSet's schema and recognizes the DataTables and the DataColumns that the typed DataSet exposes. The Properties window fills the DataMember list with the names of the selected DataSet's DataTable objects. Likewise, the list of available fields is loaded into the DataKeyField property (as shown in Figure 1). This feature eliminates the need to write this code and reduces the chance of any misspellings or even the need to remember the names of the tables and columns in the schema (note that you can obtain this same functionality with untyped DataSets if you read the schema into the DataSet with the ReadXmlSchema method). (See the sidebar "Creating a Typed DataSet".)

Figure 1 Data Binding Properties
Figure 1 Data Binding Properties

Casting Calls

Columns of strongly typed DataSet objects are defined as a particular datatype. For example, the OrderDate column in the Northwind database's Orders table is defined as a datetime. A strongly typed DataSet created from the Northwind Orders table would have a datatype of DateTime, respective to its counterpart in the database. This explicit typing of the columns eliminates the need for explicit casting to get values in and out of DataSets, as shown in the previous code example where I retrieved the CompanyName from the Customers DataTable. This feature reduces the amount of code you need to write to get a value out of or into a DataSet. Thus, strongly typed DataSets provide the developer with faster development and fewer runtime errors, since mistakes such as improperly spelled column and field names can easily be caught at compile time.

There are several ways to refer to a DataColumn's value using the various available overloaded methods. Some execute faster than others, while still others are more readable and easier to maintain. There are differences in maintenance and performance when referring to columns in an untyped DataSet. The strongly typed DataSet offers advantages over the untyped DataSet in terms of speed and easy maintainability. The speed in accessing a typed DataSet is comparable to the faster techniques in accessing an untyped DataSet (since a typed DataSet is just a layer over an untyped DataSet) and the readability of the typed DataSet is the best: oDs.Orders[0].OrderDate. The following untyped DataSet techniques have various combinations of convenience and performance. First, you can access the value of a column using a DataColumn instance:

oDs.Tables["Orders"].Rows[0][DataColumn]
This can be the least convenient since you need to get an instance of the DataColumn first. However, it's the fastest to execute. It's a good solution to use if you'll be retrieving the value from multiple rows, as you can get the DataColumn instance once and then use it over and over. You can also access a column using its ordinal position as shown in the following line of code:
oDs.Tables["Orders"].Rows[0][Ordinal]
This can make it difficult to determine what column you are referring to; however, its execution is quite fast. A third option is to use a string and refer to the column by name:
oDs.Tables["Orders"].Rows[0][StringName]
This is more readable and maintainable than ordinal position, but it's the slowest to execute since it has to search for the column by name rather than simply indexing into an array as can be done with the ordinal position option.

Derived Classes

It takes an XML Schema Definition (XSD) file as well as a class file to create a strongly typed DataSet. The XSD file stores the XML that defines the schema for the strongly typed DataSet. Figure 2 shows the diagram view of a typed DataSet that represents an Orders DataTable, an OrderDetails DataTable, and the relationship that links the two tables. The DataSet also depicts the primary keys of the two tables. The schema that defines the typed DataSet is stored in the XSD file. The view shown in Figure 2 is just a visual representation of the XML Schema that defines the strongly typed DataSet. There is also an XML Schema Extended (XSX) file that stores the designer's layout information, which is used to draw the visual representation of the DataSet, like the one shown in Figure 2. (This strongly typed DataSet is used within the sample application that you can download from the MSDN Magazine Web site.)

Figure 2 Orders DataSet
Figure 2 Orders DataSet

A strongly typed DataSet is actually a class that inherits from the System.Data.DataSet class and adds a few extra features of its own. The class file is generated from the XSD file. You can regenerate the class file by right-clicking in the XSD's designer view and selecting Generate DataSet (alternatively, you can use the xsd.exe command-line utility). The class file actually contains a series of classes that inherit from and extend the DataSet, DataTable, DataRow, and EventArgs classes. Because of this inheritance, developers do not lose any functionality by using a strongly typed DataSet. For example, even though you can refer to a DataTable via a property with the same name as the table, you can still refer to the table through the Tables collection. The following two lines evaluate to the same DataTable object:

oDs.Tables["Orders"]
oDs.Orders

The strongly typed DataSet class file contains one class that inherits from the base DataSet. It also contains one class for every DataTable contained within the DataSet. For example, if there are both Orders and OrderDetails DataTables in a strongly typed DataSet, there will be classes called OrdersDataTable and OrderDetailsDataTable that both inherit from the DataTable object. Likewise, there would be classes named OrdersRow and OrderDetailsRow that inherit from DataRow. Because of this inheritance, these classes also expose all of the standard functionality that the base classes expose.Creating a Typed DataSet

You can create strongly typed DataSet objects through drag and drop operations, command-line utilities, and through code. I will not go into much detail on this topic since there are several good references in MSDN that detail how to create a strongly typed DataSet. For example, see MSDN Knowledge Base articles 320714 ("Create and Use a Typed DataSet by Using Visual C# .NET") and 315678 ("Create and Use a Typed DataSet by Using Visual Basic .NET") for samples in both C# and Visual Basic .NET.

There are basically four techniques you can follow to create a strongly typed DataSet:

  1. Use the Visual Studio® .NET DataAdapter configuration wizard to create a DataAdapter and a Connection; then choose to generate a DataSet from them.
  2. Create an XML schema file, which can then be used by the XSD.exe command-line utility.
  3. Use Visual Studio .NET to drag a table, view, or stored procedure from the Server Explorer onto a DataSet designer (XSD file).
  4. Finally, you can even create one by hand, since it is just XML and .NET code. But obviously the other techniques listed here are much easier.

Strongly typed DataSets also offer a few additional methods to extend the DataSet base class. For example, a typed DataSet based on the Northwind database's Orders table would expose the method FindByOrderID which would accept an integer argument that would locate the DataRow with the corresponding OrderID value. The standard Find method could still be used since all of the base class's method and properties are available, but the extended properties and methods can make writing code a bit easier on developers. Assuming that an instance of the strongly typed DataSet shown in Figure 2 was created, the extended properties and methods would look like those shown in Figure 3.

Figure 3 Extended Properties and Methods (Close Figure 3)

Extended Method/Property | Description
oDs.Orders.AddOrdersRow | Adds an instance of an OrdersRow to the Orders DataTable. Also has an overloaded version that accepts a parameter for each of the columns in the Orders DataTable.
oDs.Orders.FindByOrderID | Accepts a value for the primary key field OrderID and then locates and returns an instance of the corresponding OrdersRow.
oDs.Orders.NewOrdersRow | Creates an instance of an OrdersRow.
oDs.Orders.RemoveOrdersRow | Accepts an instance of an OrdersRow and removes it from the Orders DataTable.
oDs.Orders[0].GetOrder- | Retrieves all of the Order-
DetailsRows | DetailsRow objects that are children of the 0 row of the Orders DataTable.
oDs.Orders[0].IsOrderDateNull | Returns a Boolean value indicating if the OrderDate column of the first OrdersRow of the Orders DataTable is null.
oDs.Orders[0].SetOrderDateNull | Sets the OrderDate column of OrdersRow 0 of the Orders DataTable to null.
oDs.OrderDetails[0].OrdersRow | Retrieves the OrdersRow that is the parent of OrderDetailsRow 0 of the OrderDetails DataTable.

Typed DataSets in Enterprise Apps

When working with strongly typed DataSets in multitier applications, it is often necessary that the strongly typed DataSet be available in all (or at least most) of the tiers. Generally, it is a good idea to store the strongly typed DataSet object and its files in the lowest layer. That way, any projects and assemblies that reference that layer can also refer to the strongly typed DataSet. For example let's again refer to the sample application, in which a strongly typed DataSet exists in a class library project (the business logic layer). A Web service project has a reference to that class library project, therefore it also can refer to the strongly typed DataSet contained in the class library project.

I will point out places in the multitier solution where strongly typed DataSets are used and compare them to how they could have been written using untyped DataSets. First, take a look at Figure 4, which shows the code block that adds a new OrderDetailsRow to a strongly typed DataSet. This code block is a sample from the grdOrderDetail_OnItemCommand event from WebForm1.aspx found in the sample application. I highlighted the relevant code that uses the extended methods and properties of the strongly typed DataSet.

Figure 4 Adding an OrderDetailsRow (Close Figure 4)
// Get the values for the new Order row from the footer row's controls

int nOrderID_New = Convert.ToInt32(ViewState["nLastOrderID"]);

DropDownList lstProductID_New = (DropDownList)e.Item.FindControl(
    "lstProductID_New");
int nProductID_New = Convert.ToInt32(
     lstProductID_New.SelectedItem.Value);
string sProductName_New = lstProductID_New.SelectedItem.Text;

TextBox txtUnitPrice_New = (TextBox)e.Item.FindControl(
    "txtUnitPrice_New");
Decimal mUnitPrice_New = Convert.ToDecimal(txtUnitPrice_New.Text);

TextBox txtQuantity_New = (TextBox)e.Item.FindControl(
    "txtQuantity_New");
short nQuantity_New = Convert.ToInt16(txtQuantity_New.Text);

// Create an Order table DataRow
OrdersDataSet.OrderDetailsRow oRow =
    oDs.OrderDetails.NewOrderDetailsRow();

// Set the values for the new row
oRow.OrderID = nOrderID_New;
oRow.ProductID = nProductID_New;
oRow.ProductName = sProductName_New;
oRow.UnitPrice = mUnitPrice_New;
oRow.Quantity = nQuantity_New;

// Add the new row to the Orders DataTable
oDs.OrderDetails.AddOrderDetailsRow(oRow);

// Store the DataSet
Session["oDs"] = oDs;

// Bind the DataSet to the Order Details DataGrid
BindOrderDetails(nOrderID_New);

As another example, the following code locates an OrderDetailsRow in the typed DataSet. This code can be found in the grdOrderDetail_OnUpdateCommand event in the WebForm1.aspx file:

OrdersDataSet.OrderDetailsRow oRow;
oRow = oDs.OrderDetails.FindByOrderIDProductID(
    nOrderID_OrderDetail, nProductID_Original);
This same code could have been written using an untyped DataSet, as you can see in the following:
DataRow oRow;
oRow = oDs.Tables["OrderDetails"].Rows.Find(
    new object[]{nOrderID_OrderDetail, nProductID_Original});

To demonstrate other differences between the use of a strongly typed DataSet and an untyped DataSet, let's take a look at a few examples not found in the accompanying sample application. For example, if you wanted to check if a column contained a null value, and set it to null if it doesn't, you could use the following code with a strongly typed DataSet:

if(!oDs.Orders[0].IsOrderDateNull())
{
    oDs.Orders[0].SetOrderDateNull();}
The same task could be accomplished using an untyped DataSet, as shown in the following code:
if(!oDs.Tables["Orders"].Rows[0].IsNull("OrderDate"))
{
    oDs.Tables["Orders"].Rows[0]["OrderDate"] = Convert.DBNull;
}
Both of these code blocks will do the job, but the code using the strongly typed DataSet is a bit easier to write and read.

As another example of the differences between writing code with typed and untyped DataSets, I will show how to get all of the child rows of a row in a parent table. First, let's assume that there is a strongly typed DataSet like the one you saw earlier in Figure 2 that represents Orders and OrderDetails DataTable objects. Let's also assume that these are related to one another. If you want to loop through all of the orders and all of their related order details data, you could use the following code block:

foreach(OrdersDataSet.OrdersRow oOrderRow in oDs.Orders)
{
    Debug.WriteLine("OrderID = " + oOrderRow.OrderID);
    foreach(OrdersDataSet.OrderDetailsRow oOrderDetailRow in
        oOrderRow.GetOrderDetailsRows())
    {
        Debug.WriteLine(" — " + oOrderDetailRow.ProductName);
    }
}

This code block loops through all of the orders and displays the OrdersRow's OrderID value, then loops through that order's OrderDetailsRow objects, and displays their ProductName values. The same code could be written using an untyped DataSet using the following code block:

foreach(DataRow oOrderRow in oDs.Tables["Orders"].Rows)
{
    Debug.WriteLine("OrderID = " + oOrderRow["OrderID"].ToString());
    foreach(DataRow oOrderDetailRow in
        oOrderRow.GetChildRows("Orders2OrderDetails"))
    {
        Debug.WriteLine(" — " + oOrderDetailRow["ProductName"]);
    }
}
This code block exemplifies the fact that the code using the untyped DataSet is not as elegant as the code that was written using the strongly typed DataSet.

Important Tidbits

Keep in mind that when you use a typed DataSet, if the corresponding schema changes in the underlying database table, you will need to synchronize the schema in the typed DataSet. This is not a big deal because when you use an untyped DataSet you would still likely have to change some client code if the underlying schema changes. A huge benefit of a strongly typed DataSet in this scenario, however, is that the compiler will be able to flag most of the necessary changes in the user's code, whereas with an untyped DataSet errors might not be discovered until exceptions are thrown at runtime. In any event, it's worth noting that the typed DataSet needs to stay in sync with the schema.

I find that it is always a good idea to choose the name for your strongly typed DataSet when you create it. If you change the name after you create it, you may want to regenerate the strongly typed DataSet's class file as the classes and methods will not update their names to reflect the new names. And it is generally not a good idea to manually modify the generated class file for a typed DataSet. This code is automatically generated to reflect the schema defined in the XSD. Plus, if you regenerate the typed DataSet's class file, any manual modifications you might have made previously will be lost as the file is overwritten.

There is always an exception to the rule, and the one exception I make to changing the typed DataSet's class file or XSD's XML is when I need the typed DataSets to contain an attribute that I cannot set through the XSD designer. For example, two such properties that I set in the sample application are the AutoIncrementStep and AutoIncrementSeed. I wanted them to begin at -1 and increment by -1. There was no interface in the designer to set these properties, but I could view them. I altered the XSD's XML to include the properties in the second line in the following code:

<xs:element name="OrderID" msdata:ReadOnly="true"
    msdata:AutoIncrement="true" msdata:AutoIncrementSeed="-1"
    msdata:AutoIncrementStep="-1" type="xs:int" />
By adding this code to the XSD's XML and then regenerating the class file, my strongly typed DataSet was then able to respond with these auto-incrementing features.

There are also some important things to keep in mind if you are using strongly typed DataSets and Web services. For example, a Web service that uses a DataSet will not expose that DataSet in the autogenerated WSDL unless one of its WebMethod signatures references it in some way.

Wrapping Up

Strongly typed DataSet objects are practically self documenting since they are so easy to read. Because the names of the tables and columns that they represent are properties of the typed DataSet class, writing code with typed DataSets is more intuitive and easier to maintain. By making development time faster, easier, less prone to typing errors, and by making the code more maintainable, strongly typed DataSets are a great help to developers who want to write more effective code more efficiently.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Cutting Edge: Implications of Script Callbacks in ASP.NET

Cutting Edge
Implications of Script Callbacks in ASP.NET
Dino Esposito

Code download available at: CuttingEdge0412.exe (241KB)
Contents
Portability of Script Callbacks
Building the Sample
A Quick Client Callback FAQ

Script callbacks in ASP.NET 2.0 is a feature whose time has come. Script callbacks can significantly speed up an application by limiting server postbacks. They also allow you to execute small portions of server-side code without having to manage the view state for reading or writing.

In the August installment of Cutting Edge I began my discussion of ASP.NET 2.0 script callbacks, and this month I'll go into it further (see Cutting Edge: Script Callbacks in ASP.NET). I'll clarify some cross-platform implications of the feature and provide a realistic example that mimics the behavior of the MapPoint® Web site. As you read, remember that much of this has been based on Beta 1 of ASP.NET 2.0 and might change in future builds.

Portability of Script Callbacks

When it comes to portability, there is good and bad news. The bad news is that ASP.NET script callbacks are designed to work with browsers that can deal with dynamic page changes. Any browser that doesn't support Dynamic HTML (or similar models) can't execute callbacks since they need to refresh information without refreshing the whole page.

The good news is that there are quite a few browsers, in addition to Microsoft® Internet Explorer, that support callbacks. The list includes Netscape 7.0 and up and Safari 1.2 for Apple clients, to name just a couple. In fact, DHTML-capable browsers account for almost all of today's Web traffic.

For an ASP.NET page to run script callbacks, it needs to download some JavaScript code. Looking at the page source, you can spot the line responsible for pulling down the script:

<script src="WebResource.axd?a=s&r=WebForms.js&t={guid}"
    type="text/JavaScript">
</script>

As I explained in the August 2004 column, a JavaScript function triggers the out-of-band call. The name and prototype of this function are provided by a new method on the Page class—the GetCallbackEventReference method:

Dim callbackRef As String = _
  GetCallbackEventReference(Me, input, callback, _
  "null", "null")

The GetCallbackEventReference method has two key purposes. First, it returns a string that represents the JavaScript function call that starts the remote method invocation. You bind this str ing to a client-side event of choice—for example, a mouse click. An example of the JavaScript call is shown here:

onclick="WebForm_DoCallback('__Page', 'North', UpdateMap, null, null)"

Of particular importance are the second and third parameters. The second indicates the input parameter to the remote method; the third indicates a local JavaScript function to consume results and refresh the page upon completion of the call.

The second key action that GetCallbackEventReference performs is injecting into the page the <script> tag you saw earlier. The tag references a server-side script file that contains the definition of WebForm_DoCallback.

Take a look at the content of the WebForms.js file to understand how ASP.NET script callbacks really work. Open the temporary Internet files folder, locate a resource with the matching URL, and view the resource (see Figure 1). Alternatively, you can simply request the WebForms.js file from the Web server by copying and pasting the file's URL from the script tag in the rendered page's source into the browser's address bar.

Figure 1 Locating WebForms.js
Figure 1 Locating WebForms.js

Figure 2 shows the source code of the WebForm_DoCallback function. Note that the code in the figure is based on Beta 1 and might change in future builds. As a page author, you'll never be exposed to that code, but it helps you appreciate how smart ASP.NET 2.0 can be.

Figure 2 Callback Manager Implementation (Close Figure 2)
function WebForm_DoCallback(eventTarget, eventArgument,
    eventCallback, context, errorCallback)
{
    re = new RegExp("\\x2B", "g");

    if (__nonMSDOMBrowser)
    {
        var xmlRequest = new XMLHttpRequest();
        postData = __theFormPostData +
                   "__CALLBACKID=" + eventTarget +
                   "&__CALLBACKPARAM=" +
                   escape(eventArgument).replace(re, "%2B");
        if (pageUrl.indexOf("?") != -1) {
            xmlRequest.open("GET", pageUrl + "&" + postData, false);
        } else {
            xmlRequest.open("GET", pageUrl + "?" + postData, false);
        }
        xmlRequest.setRequestHeader("Content-Type",
            "application/x-www-form-urlencoded");
        xmlRequest.send(null);
        response = xmlRequest.responseText;

        ...
    }
    else
    {
       var xmlRequest = new ActiveXObject("Microsoft.XMLHTTP");
        xmlRequest.onreadystatechange = WebForm_CallbackComplete;
        __callbackObject.xmlRequest = xmlRequest;
        __callbackObject.eventCallback = eventCallback;
        __callbackObject.context = context;
        __callbackObject.errorCallback = errorCallback;
        postData = __theFormPostData +
                   "__CALLBACKID=" + eventTarget +
                   "&__CALLBACKPARAM=" +
                   escape(eventArgument).replace(re, "%2B");
        usePost = false;
        if (pageUrl.length + postData.length + 1 > 2067) {
            usePost = true;
        }
        if (usePost) {
            xmlRequest.open("POST", pageUrl, true);
            xmlRequest.setRequestHeader("Content-Type",
                "application/x-www-form-urlencoded");
            xmlRequest.send(postData);
        }
        else {
            if (pageUrl.indexOf("?") != -1) {
                xmlRequest.open("GET", pageUrl + "&" + postData, true);
            }
            else {
                xmlRequest.open("GET", pageUrl + "?" + postData, true);
            }
            xmlRequest.setRequestHeader("Content-Type",
                "application/x-www-form-urlencoded");
            xmlRequest.send();
        }
    }
 }

As you can see, the behavior of WebForm_DoCallback changes for two categories of browsers: Internet Explorer and everything else. The value of the __nonMSDOMBrowser variable is set to the following expression upon page loading:

var __nonMSDOMBrowser =
    (window.navigator.appName.toLowerCase().indexOf('explorer') == -1);

When the browser is Internet Explorer (version 5.0 or above) the script callback mechanism is implemented through the well-known Microsoft.XmlHttp COM object. It requires that the browser be set to run ActiveX® controls. If the browser is not Internet Explorer, a totally different approach is taken; a new component in the browser's local document object model (DOM), XMLHttpRequest, will be used.

XMLHttpRequest is a little-known object that a surprising number of Web browsers support. This object is quite similar to Microsoft.XmlHttp. Microsoft introduced the Microsoft.XmlHttp object with Internet Explorer 5.0 for Windows® back in 1999. Shortly afterwards, some architects on the Mozilla project implemented a compatible native version of it for Mozilla and Netscape 7.x. Later on, Apple did the same starting with Safari 1.2. You can get more information about the XMLHttpRequest object at www.mozilla.org/xmlextras. The programming interface of XMLHttpRequest looks nearly identical to that of XmlHttp. This is not a coincidence—the XMLHttpRequest DOM object was created to provide other browsers with the same set of functionalities as the XmlHttp COM object.

Obviously, script callbacks won't work in older browsers that do not support the DOM; likewise, the feature has a limited range of applicability (but might work) on browsers that don't support dynamic changes to content and layout. For example, it doesn't work on Netscape 4.x (no XMLHttpRequest object is defined), but works like a champ on the newest Netscape 7.2.

Technically speaking, script callbacks provide the plumbing to make a remote call to the current URL without redrawing the page and without disrupting the application flow. This is possible on a variety of browsers—basically all the fifth-generation browsers such as Internet Explorer 5.x and above, Mozilla/Netscape 6.x and above, Safari 1.2x, and Opera 7.x. Browser compatibility guarantees end once the script callback has been executed and the server-generated data has been retrieved. At this point, the JavaScript code you write to handle the results from the server determines how compatible your application is with the various browsers.

To ensure cross-platform compatibility, you should strive for a DOM syntax that provides equivalent functionality and is accepted by all browsers. I'll return to this later. For now, suffice it to say that the sample code I'm going to illustrate works with both Internet Explorer 6.0 and Netscape 7.2. To let you determine dynamically if the current browser supports ASP.NET script callbacks, the HttpBrowserCapabilities class has been extended with two new properties: SupportsCallbacks and SupportsXmlHttp.

SupportsCallbacks indicates whether the browser supports advanced script scenarios like those in which scripts are invoked in callback mode upon completion of asynchronous operations. SupportsXmlHttp indicates whether the browser supports sending and receiving XML data over HTTP. Figure 3 shows a sample page you can use to check these browser capabilities.

Figure 3 Check Browser Capabilities (Close Figure 3)
<%@ Page language="VB" %>
<%@ Import namespace="System.Web.Configuration" %>

<script runat="server">
    Private Sub Page_Load(ByVal Sender As Object, ByVal e As EventArgs)
        CheckBrowserCaps()
    End Sub

    Private Sub CheckBrowserCaps()
        Dim labelText As String = String.Empty
        Dim caps As HttpBrowserCapabilities = Request.Browser

        ' Browser name
        labelText += caps.Browser & "<hr>"

        ' Supports Callback
        If caps.SupportsCallback Then
            labelText += "Browser supports callback scripts.<br>"
        Else
            labelText += "Browser does not support callback scripts.<br>"
        End If

        ' Supports XmlHttp
        If caps.SupportsXmlHttp Then
            labelText += "Browser supports XML HTTP scripts.<br>"
        Else
            labelText += "Browser does not support XML HTTP scripts.<br>"
        End If

        Label1.Text = labelText
    End Sub
</script>

<html>
<head>
    <title>Browser Capabilities Sample</title>
</head>
<body>
    <form>
        <div>
            Browser Capabilities...
            <p/><asp:Label ID="Label1" Runat="server" />
        </div>
    </form>
</body>
</html>

Are there alternatives to ASP.NET callbacks that can run on even more browsers? Remote Scripting works on a wider range of browsers and platforms. It requires an IIS Web server and does its work using ASP. Also worth a look is the JavaScript Remote Scripting (JSRS) library, which you'll find at www.ashleyit.com/rs/main.htm. JSRS is a client-side library that lets you make hidden remote procedure calls to the server. There are server-side implementations for server environments like ASP, Cold Fusion, PerlCGI, PHP, Python, and JSP. JSRS is free and available with source code.

However, a huge advantage to using ASP.NET script callbacks is that they're part of the ASP.NET Page model. Yes, other callback mechanisms are available that can deal with a variety of server environments, but ASP.NET callbacks integrate with the page lifecycle, and as a result, a developer can leverage the context of a lightweight page environment created to run the request.

Building the Sample

Imagine you have a Web site that supplies reporting services. Chances are good that at some point you'll have to present data in a chart. Later, someone looking at the report might want to drill down into the data. You could display another image, and refresh the whole page each time this happens, but that's not very efficient. Fortunately, script callbacks can help. Before I explain how, let's take a look at Microsoft MapPoint as an example.

To use MapPoint, you navigate to the site, enter a city, and get your map (see Figure 4). Now choose a direction button and click to pan the image accordingly. You'll notice that the page remains static and only the image is refreshed. In a minute, you'll see how to implement the same feature using script callbacks.

Figure 4 The UI of the MapPoint Site
Figure 4 The UI of the MapPoint Site

While this functionality could be reproduced using only client-side JavaScript, script callbacks provide a good mix of server- and client-side execution, performance, and cross-platform support. Figure 5 shows the sample page in action in Netscape 7.2.

Figure 5 The Sample Page in Netscape 7.2
Figure 5 The Sample Page in Netscape 7.2

I built the sample page using some of the hot new technologies available in ASP.NET 2.0 like Master Pages and codebehind files. The Master Page determines the layout of derived pages by defining replaceable regions that content pages fill. The master I use here incorporates a couple of images, a page description, and a body. A content page must have a very special structure, as shown in Figure 6. At least in the Beta 1 build of ASP.NET 2.0, content pages are not allowed to contain anything but <asp:Content> tags. You can't even insert comments or client-side script blocks outside replaceable regions. To add some script code to the content page, you can define an additional placeholder and fill it with script and anything else you may need. This approach is acceptable to the ASP.NET page parser.

Figure 6 The Sample Implemented as a Content Page (Close Figure 6)
<%@ Page Language="VB" MasterPageFile="~/MsdnMag.master"
    CompileWith="Default.aspx.vb" ClassName="Default_aspx"
    Title="Cutting Edge (Dec04)" %>
<%@ Implements interface="System.Web.UI.ICallbackEventHandler" %>

<asp:Content Runat="server" ContentPlaceHolderID="PageScript">
    <script language="javascript">
        function UpdateMap(result, context)
        {
            // Unpack the result and extract image URL and
            // ID of the map control
            var o = result.split(",");

            // Set the new image
            var id = o[1];
            var img = document.getElementById(id);
            var url = o[0];
            img.src = url;
        }
    </script>
</asp:Content>

<asp:Content Runat="server" ContentPlaceHolderID="PageDescription">
    <asp:Label Runat="Server" ID="Desc">
    Using ASP.NET 2.0 script callbacks to scroll an image
    without refreshing the whole page.
    </asp:Label>
</asp:Content>

<asp:Content Runat="server" ContentPlaceHolderID="PageBody">
    <h2>Santa Claus's hometown</h2>
    <table bgcolor="#004472"><tr>
            <td colspan="3" align="center">
                <asp:Image Runat="Server" ID="GoNorth"
                    ToolTip="Move north" ImageUrl="images\gonorth.gif" />
            </td>
        </tr>
        <tr>
            <td>
                <asp:Image Runat="Server" ID="GoWest"
                    ToolTip="Move west" ImageUrl="images\gowest.gif" />
            </td>
            <td>
                <asp:Image Runat="server" ID="Map"
                    ImageUrl="images\SantaHomeTown.gif" />
            </td>
            <td align="right">
                <asp:Image Runat="Server" ID="GoEast"
                    ToolTip="Move east" ImageUrl="images\goeast.gif" />
            </td>
        </tr>
        <tr>
            <td colspan="3" align="center">
                <asp:Image Runat="Server" ID="GoSouth"
                    ToolTip="Move south" ImageUrl="images\gosouth.gif" />
            </td>
        </tr></table>
</asp:Content>

The body of the page is a 3×3 table containing a central image surrounded with navigational buttons used to change which portion of the map is displayed. Each move button is associated with a client-side event to trigger the callback mechanism. To implement graphic buttons, like the North button in Figure 5, ASP.NET supplies the excellent ImageButton control:

<asp:ImageButton Runat="Server" ID="GoNorth" ToolTip="Move north"
     ImageUrl="images\gonorth.gif" />

ImageButton derives from Image but implements the new IButtonControl interface along with IPostBackEventHandler. As a result, the control behaves like a button and, more importantly, always posts back when the user clicks on it. The control works like the Button control of ASP.NET 1.x, which always renders as a submit button. In ASP.NET 2.0, the Button control features a new Boolean property—UseSubmitBehavior—that renders it as a client-side button so that the click doesn't cause the page to post. You can build a pseudo image button using the following code:

<asp:Image Runat="Server" ID="GoNorth"
    Style="cursor:hand;"
    ToolTip="Move north" ImageUrl="images\gonorth.gif" />
The Image control must be associated with a client click event handler, as shown here:
Dim callbackRef As String = GetCallbackEventReference( _
    Me, "'North'", "UpdateMap", "null", "null");
GoNorth.Attributes("onclick") = callbackRef
The code runs during the page loading phase. Once the user clicks, the underlying ASP.NET callback manager sends the new request in the background. Figure 7 shows the RaiseCallbackEvent method that each ASP.NET page must provide as part of the ICallbackEventHandler interface. This is the server-side method that is invoked on each out-of-band call. The method is passed a string (here, "North") that represents the direction of the newly requested image. This information is passed along to the method that retrieves the new image. All the images are organized in a 3¥3 matrix, at the center of which is the main image. The main image is the map centered on the city of Rovaniemi. Two indexes (map row and column) are defined to track the current position and subsequently the URL of the image displayed. When the user pans the image, row and column indexes are updated depending on the direction required, and a new URL is retrieved from the matrix and returned to the callback manager.

Figure 7 Raised Callback Event (Close Figure 7)
Public Overridable Function RaiseCallbackEvent( _
        ByVal mapDir As String) As String _
    Implements ICallbackEventHandler.RaiseCallbackEvent

    ' Get the direction of the panning
    Dim dir As MapDirections = CType([Enum].Parse( _
        GetType(MapDirections), mapDir), MapDirections)

   ' Obtain the new image
    Dim NewImage As String = DetermineNewImage(dir)

    ' The output string contains two pieces of information:
    ' image, direction to block
    Dim buf() As String = New String(2) {}
    buf(0) = NewImage
    buf(1) = Map.ClientID

    Return String.Join(",", buf)
End Function

UpdateMap is the JavaScript callback function that executes when the remote call has completed. UpdateMap receives the URL of the new image and is called to update the Src attribute of the client-side counterpart of the Image control. You access this client-side component like so:

GoNorth.src = url;

When hosted in content pages, though, the previous code won't work because no HTML element is found in the client page that is named GoNorth. This apparently contradicts the ASP.NET page source, where you clearly have an Image control with an ID of GoNorth. What's up?

The IDs of controls defined in a Master Page placeholder have their names mangled and prefixed with the ID of the region and other information. The actual ID of the GoNorth image control is something different. In the example, it is ctl00_PageBody_GoNorth. This is why ASP.NET has provided the ClientID property since 1.0. A control's UniqueID property is used to find a control in the context of server execution, while the ClientID is used for writing client-side script to access the element rendered to the client. So, I modified the server method to make it return the client ID of the control to be refreshed. In summary, the string returned to the JavaScript callback function is a comma-separated string in which the first token contains the URL of the image and the second contains the client ID of the control to refresh.

As mentioned, the sample page needs to track row and column indexes to quickly locate the URL of the image to show as the user clicks on the displayed image. Normally, this information is placed in the view state to avoid taxing the server memory. However, to handle view state updates completely on the server would require that the entire page lifecycle be executed. This is contrary to the design of the script callback feature. While the feature does allow for avoiding a complete redraw of the client, it also allows for lightweight page execution, running only what is necessary to process the callback. The remote call posts all the input fields of the current page along with two additional values, CALLBACKID and CALLBACKPARAM, as Figure 2 shows. In the collection of input fields you'll find the view state information as well. So the view state is sent back during the remote invocation and gets correctly restored. However, any change you make on the server that alters the view state of the page is lost because the view state is not persisted upon exit. You might disagree with this design choice, but then the view state should be attached to the response you serve to the JavaScript callback and some script would be necessary to replace it on the fly. It may be possible, but is it worth the cost?

To work around the issue, avoid using the view state in the implementation of page or control properties that need be updated during script callback operations. If the properties are to survive page requests, you can use session state, as you can see in Figure 8.

Figure 8 Use Session Instead of View State (Close Figure 8)
Private Property MapRow() As Integer
    Get
        Return CType(Session("MapRow"), Integer)
    End Get
    Set (ByVal Value As Integer)
        Session("MapRow") = value
    End Set
End Property

Private Property MapColumn() As Integer
    Get
       Return CType(Session("MapColumn"), Integer)
    End Get
    Set (ByVal Value As Integer)
       Session("MapColumn") = value
    End Set
End Property

A Quick Client Callback FAQ

To finish the second leg of the ASP.NET script callback coverage, let's address a couple of side points. The first involves values you can send and receive through client callbacks. The only clear restriction you have here is that the server method RaiseCallbackEvent must receive and return a string. You can devise and implement just about any functionality around that requirement. For example, you could exchange strings that evaluate to objects. The main issue I see with this, though, is that you need to manage two different types of objects—managed .NET objects on the server and JavaScript objects on the client. The client can build and send a SOAP representation of the .NET object and the server can return the string description of a JavaScript object.

The beauty of callbacks is that they work asynchronously; therefore there should be no problem if multiple callback operations are simultaneously started. Unfortunately, this feature doesn't work in Beta 1 due to a known bug. If multiple callback events are fired in succession, only one is really fired; the others are lost.

ASP.NET script callbacks provide a good mix of power, performance, portability, and even security. Both pages and individual controls can take advantage of this feature. Now you know what's needed for a page to invoke a remote method. In a future column I'll show how to implement a custom control that takes advantage of callbacks.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the newest Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Service Station: Run ASMX Without IIS

Service Station
Run ASMX Without IIS
Aaron Skonnard

Code download available at: ServiceStation0412.exe (165KB)
Contents
ASP.NET HTTP Architecture
Hosting the HTTP Pipeline
Move Over, IIS
HttpListener: Implementing Your Own Web Server
Hosting ASMX
Hosting Models: Web Services on the Desktop
Where Are We?

When the Microsoft® .NET Framework first shipped, it introduced a breakthrough Web services framework known as ASMX. The motivation behind the ASMX design was to simplify the process of developing Web services as much as possible so that even if you're not an XML expert, you can get a Web service up and running. ASMX accomplished this by hiding most of the underlying XML and Web services details. Instead of forcing developers to deal directly with SOAP envelopes and Web Services Description Language (WSDL) files, ASMX introduced automatic mapping layers that bridge the gap with traditional .NET code.

ASMX is also tightly integrated with the popular ASP.NET HTTP pipeline. Hence, it enjoys the same benefits as traditional ASP.NET Web applications, such as a sophisticated hosting environment and process model, robust configuration and deployment options, and flexible extensibility points. As a result, ASMX is usually the first stop for most Web services developers. Most developers incorrectly assume that ASMX requires IIS; after all, it's the only use case they've ever seen. But the truth is that ASMX doesn't have any technical dependencies on IIS whatsoever.

The need to host Web services without IIS is very real. In some environments, it may not be possible to have IIS running on the machine that must host the Web service for a variety of reasons. Luckily, you can host ASMX in your own process without IIS. It's been possible to do this since the release of the .NET Framework 1.0, but you had to provide your own Web server to receive HTTP requests. Cassini is a sample Web server developed by the ASP.NET team that filled this need and allowed you to run ASP pages without IIS. However, writing your own Web server, or using a sample like Cassini, does not seem reasonable to most developers.

Since the release of Windows Server™ 2003 and Windows® XP SP2, there has been a new HTTP protocol stack available called http.sys. With http.sys and some new managed classes in the .NET Framework 2.0 (especially HttpListener), you can easily build a Web server right into your applications without requiring IIS on the machine. These advances have made it possible to run ASMX everywhere. Keep in mind that the .NET Framework 2.0 is currently in Beta and is therefore subject to change.

ASP.NET HTTP Architecture

ASP.NET was specifically designed to avoid dependencies on IIS. The underlying architecture consists of a pipeline of .NET classes that work together to process incoming HTTP messages. It's considered a pipeline because each HTTP request travels through a sequence of objects, each performing some processing.

The HttpRuntime class sits at the beginning of the pipeline and is responsible for starting the process. The pipeline begins executing when the static ProcessRequest method is called on the HttpRuntime class. ProcessRequest takes an HttpWorkerRequest object, which contains all the information in the current request. HttpRuntime uses the information in the HttpWorkerRequest to populate the HttpContext object. It then instantiates the appropriate HttpApplication class, which will invoke any IHttpModule implementations registered with the application for pre/post-processing. At this point, the appropriate IHttpHandler implementation is identified, instantiated, and invoked.

This process occurs for each HTTP request that enters the pipeline. All of the ASP.NET functionality (including that of ASMX) is contained in the pipeline classes. For example, support for processing ASMX endpoints begins when the request reaches the System.Web.Services.Protocols.WebServiceHandlerFactory class, which is responsible for identifying, compiling (if necessary), and instantiating the identified ASMX class, as well as invoking the WebMethod targeted by the incoming SOAP message.

Figure 1 HTTP Pipeline and Web Servers
Figure 1 HTTP Pipeline and Web Servers

The pipeline is completely autonomous and disconnected from IIS. It even runs in a separate process from inetinfo.exe when used in conjunction with IIS. The name of the process depends on the host OS (aspnet_wp.exe on Windows XP, w3wp.exe on Windows Server 2003). In addition to having its own process model, the pipeline also has an independent configuration story that is decoupled from the IIS metabase. The only thing the pipeline doesn't come with is its own Web server that's capable of receiving incoming HTTP requests. You still need something like IIS 5.0 or http.sys that is capable of listening for incoming HTTP messages. Even then, these components are only responsible for receiving the HTTP requests and handing them off to the ASP.NET pipeline, which handles everything from that point (see Figure 1).

Once the request makes it into the worker process, the worker process creates an HttpWorkerRequest object (representing the incoming request) and calls HttpRuntime.ProcessRequest to launch the pipeline. Because of this elegant design, it is possible to call HttpRuntime directly within your own app.

Hosting the HTTP Pipeline

The classes you need to host ASP.NET are found in the System.Web and System.Web.Hosting namespaces. The main classes you'll need to get started are ApplicationHost, HttpRuntime, and an HttpWorkerRequest-derived class. You start by calling ApplicationHost.CreateApplicationHost. This method creates a new application domain (AppDomain) capable of processing ASP.NET requests. Since you're explicitly creating the AppDomain, you must specify the virtual directory and corresponding physical directory when making the call.

In addition to creating the new AppDomain, CreateApplicationHost also instantiates an object within the new AppDomain through which you can communicate. You specify the type you would like it to instantiate when making the method call. Since the object will be used across AppDomain boundaries, it must derive from MarshalByRefObject. You'll want to use your own class that has the methods you need to interact with the AppDomain. For example, at a minimum you'll want a ProcessRequest method that makes it possible to submit a new ASP.NET request for processing.

Here's a class that could be used for this purpose:

public class MySimpleHost : MarshalByRefObject
{
    public void ProcessRequest(string file)
    {
       ... // use the ASP.NET HTTP pipeline to process request
    }
}
In this case, ProcessRequest accepts the file name of the page to be processed. Within ProcessRequest, you can use HttpRuntime to start the pipeline processing. HttpRuntime has a static method, also named ProcessRequest, which takes an argument of type HttpWorkerRequest.

HttpWorkerRequest is an abstract class, but luckily .NET ships a simple derived class called SimpleWorkerRequest, which was designed to handle simple HTTP GET requests. When you instantiate SimpleWorkerRequest, you must specify the name of the page to be processed, an optional query string, and a TextWriter to which the pipeline can write the output. Once you have an HttpWorkerRequest object, you invoke the pipeline by calling ProcessRequest, as illustrated here:

... // MySimpleHost.ProcessRequest
SimpleWorkerRequest swr =
  new SimpleWorkerRequest(page, null, Console.Out);
HttpRuntime.ProcessRequest(swr);

With MySimpleHost in place, you need to call ApplicationHost.CreateApplicationHost from within the host application to instantiate the object. Then, you can use MySimpleHost.ProcessRequest to send requests into the HTTP pipeline for processing, as shown in the following code snippet:

... // console host application
MySimpleHost msh = (MySimpleHost)
  ApplicationHost.CreateApplicationHost(typeof(MySimpleHost), "/",
    Directory.GetCurrentDirectory());
foreach (string page in args)
  msh.ProcessRequest(page);

The implementation of ApplicationHost.CreateApplicationHost expects to find the assembly of the specified type in one of two places: either in the Global Assembly Cache (GAC) or within the bin directory under the specified physical directory. There is no documented way to change this behavior short of reimplementing CreateApplicationHost. So depending on your project configuration and deployment scenario, you may need to install the assembly into one of these locations.

Figure 2 contains the code for a complete console application that hosts ASP.NET. This sample is available for download. You specify the name of the ASP.NET file along with an optional query string on the command line. Then the program passes them into the pipeline through the call to MySimpleHost.ProcessRequest.

Figure 2 Simple ASP.NET Host (Close Figure 2)
Using System;
using System.Web;
using System.Web.Hosting;
using System.IO;
...

class Program
{
    static void Main(string[] args)
    {
        if (args.Length == 0)
        {
            Console.WriteLine(
              "Usage: simplehost filename [querystring]");
            return;
        }
        string file = args[0];
        string query = (args.Length > 1) ? args[1] : "";

        MySimpleHost msh = (MySimpleHost)
            ApplicationHost.CreateApplicationHost(
                typeof(MySimpleHost), "/",
                Directory.GetCurrentDirectory());
        msh.ProcessRequest(file, query);
    }
}

public class MySimpleHost : MarshalByRefObject
{
    public void ProcessRequest(string file, string query)
    {
        SimpleWorkerRequest swr =
            new SimpleWorkerRequest(file, query, Console.Out);
        HttpRuntime.ProcessRequest(swr);
    }
}

In the download for this column, I've provided a few ASP.NET files that you can use to experiment, including one called math.asmx. When you run the app, specifying "math.asmx WSDL" on the command line, you'll see the ASMX-generated WSDL definition printed to the console window (equivalent to browsing to http://host/math.asmx?WSDL with math.asmx hosted in IIS). If you specify only "math.asmx" on the command line, it will print the ASMX-generated human-readable documentation page.

This obviously isn't a realistic example since you have to specify the ASP.NET page on the command line. In the real world this information would come in through an HTTP request. In order to support HTTP, you need to integrate a Web server with your app.

Move Over, IIS

Http.sys is a new low-level HTTP protocol stack available in Windows Server 2003 and Windows XP SP2. Http.sys is a kernel-mode component that offers its HTTP services to all applications on the machine. This means HTTP support is now deeply ingrained into the OS. Even IIS 6.0 has been rearchitected to use http.sys (see Figure 3).

Figure 3 Http.sys Architecture
Figure 3 Http.sys Architecture

Before version 6.0, IIS relied on the TCP/IP kernel and the Windows Sockets API (Winsock) to receive HTTP requests. Since Winsock is a user-mode component, each receive operation required switching between kernel mode and user mode. Http.sys is now capable of caching responses directly within the kernel. Lowering the HTTP stack into the kernel makes it possible to remove this expensive context switch when serving cached responses, boosting efficiency and overall throughput.

When http.sys receives a request, it can forward it directly to the correct worker process. Also, if the worker process isn't available to accept the request, http.sys stores it until the worker process starts up and is able to get it. This means that worker process failures don't interrupt service. When IIS 6.0 starts up, the WWW Service communicates with http.sys and registers the routing information for each configured IIS application. Whenever you create or remove applications from IIS, the WWW Service communicates with http.sys to update its routing information.

As you can see in Figure 3, http.sys lays the foundation for the IIS 6.0 Web architecture, but it's not tied to IIS in any way. Any application running on the machine can take advantage of http.sys for receiving HTTP requests. Like the WWW Service, you register your application with http.sys and begin listening for incoming HTTP requests. The .NET Framework 2.0 introduces a suite of managed classes that make this very easy to do.

HttpListener: Implementing Your Own Web Server

System.Net contains several new classes for interfacing with http.sys. HttpListener is the focal point of these classes. Using it, you can create a simple Web server (or listener) that responds to incoming HTTP requests. The listener remains active for the lifetime of the HttpListener object, although you can tell it to start and stop listening on command.

In order to begin using HttpListener, you must first instantiate it. Then you indicate which HTTP URLs the listener should process by adding URL prefixes to the Prefixes property. Each URI must contain a scheme ("http" or "https"), a host, an optional port, and an optional path. Each prefix must end with a forward slash:

HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://localhost:8081/foo/");
listener.Prefixes.Add("http://127.0.0.1:8081/foo/");
listener.Start();

Only one HttpListener can be listening for a specific URI prefix. You'll get a Win32Exception exception if you attempt to add a duplicate. When you specify a port, you can replace the host name with "*" to indicate that this listener should handle all URIs with this port unless they're matched by another HttpListener. Or you can replace the host name with "+" to indicate that the listener accepts all requests to the specified port, as illustrated here:

HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://+:8081/");
listener.Start();

You can also specify the authentication scheme used by the listener through the AuthenticationScheme property. HttpListener supports anonymous, basic, digest, and Windows authentication. It also supports secure sockets layer (SSL) connections, so you can safely use basic authentication over HTTPS, as you see here:

HttpListener listener = new HttpListener();
listener.AuthenticationSchemes = AuthenticationSchemes.Basic;
listener.Prefixes.Add("https://+:8081/");
listener.Start();

Once you've specified the URI prefixes that the listener will process, you must call the Start method as you just saw (note: you must add at least one prefix before calling Start). Start doesn't really do anything noticeable—it simply prepares the listener object to begin receiving requests.

In order to receive a request, you must call GetContext, like this:

HttpListenerContext ctx = listener.GetContext();
GetContext is a synchronous call that blocks while waiting for an incoming request to arrive. It doesn't return until one is received. HttpListener also provides an asynchronous mechanism for receiving a request through BeginGetContext and EndGetContext. GetContext and EndGetContext return an HttpListenerContext object, which represents the received HTTP request.

You can use HttpListener to integrate your own Web server with the sample application shown in Figure 2. All you have to do is forward the requests from the HttpListener to the HTTP pipeline in your code. You can add a loop that continually calls GetContext and uses the information in the returned HttpListenerContext object to invoke ProcessRequest. Then you'll use the HttpListenerContext class to look up the requested file name (using the Request.Url.LocalPath property, like "/math.asmx") and the query string (using the Request.Url.Query property, for example "?WSDL"). With this code, you've made it possible to request ASP.NET pages over HTTP, as shown in Figure 4.

Figure 4 Handling HTTP Requests with HttpListener (Close Figure 4)
MySimpleHost msh = (MySimpleHost)
  ApplicationHost.CreateApplicationHost(
    typeof(MySimpleHost), "/", Directory.GetCurrentDirectory());

HttpListener listener = new HttpListener();
listener.Prefixes.Add("http://localhost:8081/");
listener.Prefixes.Add("http://127.0.0.1:8081/");
listener.Start();

while (true)
{
   HttpListenerContext ctx = listener.GetContext();
   string file = ctx.Request.Url.LocalPath.Replace("/", "");
   string query = ctx.Request.Url.Query.Replace("?", "");
   msh.ProcessRequest(file, query);
}

After making these changes and running the program, the console application will be waiting for GetContext to return. Now you can open a Web browser and navigate to the registered URI (in this example, http://localhost:8081/math.asmx). After doing this, GetContext returns and the request is handed off to the HTTP pipeline. Then you should see the response written to the console window as you did before.

HttpListenerContext is similar to HttpContext in that it provides Request, Response, and User properties that make it easy to interface with all aspects of the HTTP message. The Request property is of type HttpListenerRequest, while the Response property is of type HttpListenerResponse. You use HttpListenerRequest to access the HTTP method, URL, headers, user-agent, body, and other parts of the request. You use HttpListenerResponse to write the HTTP response back to the client. The User property returns an IPrincipal with information about the authenticated user.

You can use these properties to extend the sample further so it's capable of writing directly to the HTTP response stream. You can do this by modifying the signature of MySimpleHost.ProcessRequest to accept a TextWriter. After that, you can wrap the HttpListenerContext.Response.OutputStream in a StreamWriter object and pass it in.

Figure 6 Browser Output
Figure 6 Browser Output

Figure 5 provides the complete code for this revised sample with an integrated HttpListener. Now when you run the sample, you can use a browser to read and navigate around the ASMX-generated document page (see Figure 6). A message is printed to the console window each time the app receives a request (see Figure 7).

Figure 5 Simple ASP.NET Host Using HttpListener (Close Figure 5)
using System;
using System.Web;
using System.Web.Hosting;
using System.Net;
using System.IO;
...

class Program
{
    static void Main(string[] args)
    {
        MySimpleHost msh = (MySimpleHost)
          ApplicationHost.CreateApplicationHost(
            typeof(MySimpleHost), "/", Directory.GetCurrentDirectory());

        HttpListener listener = new HttpListener();
        listener.Prefixes.Add("http://localhost:8081/");
        listener.Prefixes.Add("http://127.0.0.1:8081/");
        listener.Start();

        Console.WriteLine(
          "Listening for requests on http://localhost:8081/");

        while (true)
        {
            HttpListenerContext ctx = listener.GetContext();
            string page = ctx.Request.Url.LocalPath.Replace("/", "");
            string query = ctx.Request.Url.Query.Replace("?", "");
            Console.WriteLine("Received request for {0}?{1}",
              page, query);

            StreamWriter sw = new
              StreamWriter(ctx.Response.OutputStream);
            msh.ProcessRequest(page, query, sw);
            sw.Flush();
            ctx.Response.Close();
        }
    }
}

public class MySimpleHost : MarshalByRefObject
{
    public void ProcessRequest(string p, string q, TextWriter tw)
    {
        SimpleWorkerRequest swr = new SimpleWorkerRequest(p, q, tw);
        HttpRuntime.ProcessRequest(swr);
    }
}
Figure 7 Console Application Output
Figure 7 Console Application Output

This sample illustrates a hosting model capable of dealing with HTTP GET requests and query strings. SimpleWorkerRequest was only designed for this simple case, not for more advanced POST operations. Hence, the sample can't fully host ASMX endpoints, which require POST support to process incoming SOAP requests.

Hosting ASMX

To fully support ASMX endpoints, you need a custom HttpWorkerRequest that knows how to deal with the request/response streams. It should be based on the HttpListenerContext object you get back from GetContext. This is a somewhat painful task since the HttpWorkerRequest is fairly large and the documentation isn't completely up to par. As a result, I've provided a sample implementation called HttpListenerWorkerRequest in Figure 8.

Figure 8 HttpListenerWorkerRequest Implementation (Close Figure 8)
public class HttpListenerWorkerRequest : HttpWorkerRequest
{
    private HttpListenerContext _context;
    private string _virtualDir;
    private string _physicalDir;

    public HttpListenerWorkerRequest(
        HttpListenerContext context, string vdir, string pdir)
    {
        if (null == context)
            throw new ArgumentNullException("context");
        if (null == vdir || vdir.Equals(""))
            throw new ArgumentException("vdir");
        if (null == pdir || pdir.Equals(""))
            throw new ArgumentException("pdir");

        _context = context;
        _virtualDir = vdir;
        _physicalDir = pdir;
    }

    // required overrides (abstract)
    public override void EndOfRequest()
    {
        _context.Response.OutputStream.Close();
        _context.Response.Close();
        _context.Close();
    }

    public override void FlushResponse(bool finalFlush)
    {
        _context.Response.OutputStream.Flush();
    }

    public override string GetHttpVerbName()
    {
        return _context.Request.HttpMethod;
    }

    public override string GetHttpVersion()
    {
        return string.Format("HTTP/{0}.{1}",
            _context.Request.ProtocolVersion.Major,
            _context.Request.ProtocolVersion.Minor);
    }

    public override string GetLocalAddress()
    {
        return _context.Request.LocalEndPoint.Address.ToString();
    }

    public override int GetLocalPort()
    {
        return _context.Request.LocalEndPoint.Port;
    }

    public override string GetQueryString()
    {
        string queryString = "";
        string rawUrl = _context.Request.RawUrl;
        int index = rawUrl.IndexOf('?');
        if (index != -1) queryString = rawUrl.Substring(index + 1);
        return queryString;
    }

    public override string GetRawUrl()
    {
        return _context.Request.RawUrl;
    }

    public override string GetRemoteAddress()
    {
        return _context.Request.RemoteEndPoint.Address.ToString();
    }

    public override int GetRemotePort()
    {
        return _context.Request.RemoteEndPoint.Port;
    }

    public override string GetUriPath()
    {
        return _context.Request.Url.LocalPath;
    }
        // remaining methods omitted
        // download sample for more details
    ...
}

At this point, you may be tempted to go back to the previous sample and replace all instances of SimpleWorkerRequest with HttpListenerWorkerRequest. However, this would require you to pass the HttpListenerContext object into ProcessRequest. Unfortunately HttpListenerContext doesn't derive from MarshalByRefObject, which prevents it from crossing the AppDomain boundary. You'll need to redesign the sample to make this work.

First, you'll need a class that wraps an HttpListener object and makes it possible to control it from across the AppDomain. I've provided one in Figure 9 called HttpListenerWrapper. This is the type that you'll specify in the call to CreateApplicationHost from now on. It has a Configure method that instantiates the contained HttpListener object and registers the supplied URI prefixes. It has Start and Stop methods that simply delegate to the listener. And it has a ProcessRequest method that takes care of everything else—it calls GetContext, instantiates a new HttpListenerWorkerRequest, and passes it into HttpRuntime.ProcessRequest, which processes the request. You can use the following classes from the host app:

HttpListenerWrapper listener =
  (HttpListenerWrapper)ApplicationHost.CreateApplicationHost(
    typeof(HttpListenerWrapper), "/", Directory.GetCurrentDirectory());
listener.Configure(prefixes, "/", Directory.GetCurrentDirectory());
listener.Start();

while (true) listener.ProcessRequest();

Figure 9 HttpListenerWrapper Implementation (Close Figure 9)
public class HttpListenerWrapper : MarshalByRefObject
{
    private HttpListener _listener;
    private string _virtualDir;
    private string _physicalDir;

    public void Configure(string[] prefixes, string v, string p)
    {
        _virtualDir = v;
        _physicalDir = p;
        _listener = new HttpListener();

        foreach (string prefix in prefixes)
            _listener.Prefixes.Add(prefix);
    }

    public void Start()
    {
        _listener.Start();
    }

    public void Stop()
    {
        _listener.Stop();
    }

    public void ProcessRequest()
    {
        HttpListenerContext ctx = _listener.GetContext();
        HttpListenerWorkerRequest workerRequest =
            new HttpListenerWorkerRequest(ctx, _virtualDir,
                _physicalDir);
        HttpRuntime.ProcessRequest(workerRequest);
    }
}

You now have a fully functional host for ASMX endpoints. There's another complete sample in the download that illustrates this code in action. While the console app is running, you should now be able to invoke the ASMX WebMethods using SOAP or through the form provided on the HTML documentation page.

Hosting Models: Web Services on the Desktop

Now that you know how to host ASMX in a process of your choice, you're obviously wondering when and where you should use this technique. There are a variety of situations in which it's unacceptable to run IIS on a particular machine, but you may still want to host Web services on that node using the ASMX programming model. When this is the case, the technique discussed in this column is a good alternative.

Unlike inetinfo.exe, which comes with numerous services for different types of communication, http.sys is a simplified kernel focused exclusively on HTTP traffic. Thus it has a reduced attack surface. Assuming you have either Windows XP SP2 or Windows Server 2003 on your machine, you can host ASMX in your own process without IIS. This means you are able to host ASMX in a console application, a Windows Forms application, or a Windows NT® service, depending on your particular needs.

However, I should point out that when you take this approach, you're giving up the sophisticated process model provided by IIS through the w3wp.exe (or aspnet_wp.exe) worker process. This means you lose things like process management (startup, failure detection, recycling), thread pool management, and ISAPI support. When you host ASMX, you're providing the process, so you're re-sponsible for providing the process model and associated services.

Figure 10 Web Services on the Desktop
Figure 10 Web Services on the Desktop

Probably the most compelling use case for hosting ASMX in your own process is when you need Web services to run on the desktop. For example, you may have a Windows Forms application that needs to receive notifications from a Web server or some other internal enterprise-wide Windows service (see Figure 10). On the desktop, you don't need a sophisticated process model like you do on the server, but it would still be nice to take advantage of the productive programming model of ASMX. Http.sys and ASMX hosting is a great fit for this scenario. I've provided a few additional samples for download (a Windows Forms application and a Windows service that host ASMX) to illustrate these ideas.

Where Are We?

I've covered the basics for hosting the ASP.NET HTTP pipeline in a process of your choice. I've also discussed how to leverage http.sys and the new managed HttpListener class (and friends) to build a Web server right into your application. HttpListener makes it very easy to receive HTTP messages and forward them to the ASP.NET page for processing. Using these techniques gives you tremendous flexibility in running ASMX everywhere.

For more on this topic, see "New Features for Web Service Developers in Beta 1 of the .NET Framework 2.0", "HTTP.sys Response Cache", and "Download ASP.NET Cassini Sample Web Server".

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, an education and content creation company, where he focuses on XML and Web services technologies. He's a contributing editor to MSDN Magazine and the author of several books, including Essential XML Quick Reference (Addison-Wesley, 2001). Reach him at www.pluralsight.com/aaron.


.NET Matters: Asynchronous HttpWebRequests, Interface Implementation, and More

.NET Matters
Asynchronous HttpWebRequests, Interface Implementation, and More
Stephen Toub

Code download available at: NETMatters0412.exe (163KB)

Q My client and I are working on a client-side application that entails making HttpWebRequests to a server application in order to submit data. We're trying to limit the number of concurrent connections made by the client so we can limit the load placed on the server. Initially we tried making these requests from ThreadPool threads, but we continually received exceptions specifying that "There were not enough free threads in the ThreadPool object to complete the operation." So I have two questions. First, why doesn't the ThreadPool have enough threads; isn't the whole point of the ThreadPool that it should cause queued work items to block until there are threads available? And second, if we can't use the ThreadPool to do this, how can we throttle the number of concurrent connections requested?

A Excellent set of questions, and based on a quick Web search one that has stymied many developers. The first thing to be aware of is that in version 1.x of the Microsoft®.NET Framework, HttpWebRequest never makes synchronous requests. What do I mean by that? Take a look at the code for HttpWebRequest.GetResponse as coded in the Shared Source CLI (SSCLI), shown here omitting the code that checks to see if the response was previously retrieved and that accounts for timeouts:

public override WebResponse GetResponse() {
    •••
    IAsyncResult asyncResult = BeginGetResponse(null, null);
    •••
    return EndGetResponse(asyncResult);
}

As you can see, HttpWebRequest.GetResponse is simply a wrapper around the pairing of BeginGetResponse and EndGetResponse. These operate asynchronously, meaning that BeginGetResponse makes the actual HTTP request from a different thread than the one from which it was called, and EndGetResponse blocks until the request has completed. The net result of this is that HttpWebRequest queues a work item to the ThreadPool for every outbound request. So, we know that HttpWebRequest uses threads from the ThreadPool, but why does that cause a problem if the call to GetResponse is made from a ThreadPool thread? Deadlocks.

As you pointed out in your question, the ThreadPool will queue work items which will wait until threads are available to process them. For discussion's sake, let's say that the ThreadPool is a pool of only one thread (it's of course much larger by default). You queue a method that calls HttpWebRequest.GetResponse, and this method starts executing on the sole thread in the ThreadPool. The GetResponse method calls BeginGetResponse, which in turn queues a work item to the ThreadPool, and then calls EndGetResponse to wait for that work item to be completed. Unfortunately, that'll never happen. The work item BeginGetResponse queued won't execute until a ThreadPool thread becomes available, but the only thread in the ThreadPool is currently executing the call to EndGetResponse waiting for the HTTP request to complete. Deadlock. "But," you say, "this is a contrived example with only one thread in the pool, and as you said there are many more threads in the pool than that." True, but let's expand the example. What happens if there are two threads in the pool, and you quickly queue two methods that call GetResponse before either has a chance to queue their work items? Same problem, deadlock. What happens if there are 25 threads in the pool and you quickly queue 25 of these methods? Again, deadlock. See the problem?

As a workaround for this, the Framework team implemented the exception you're seeing. At the end of BeginGetResponse, just before the work item is queued, System.Net.Connection.IsThreadPoolLow is used to query how many worker threads are available in the pool. If the number is too low (less than two), an InvalidOperationException is thrown.

The good news is that in the .NET Framework 2.0, synchronous requests made with HttpWebRequest.GetResponse are truly synchronous, and this problem goes away, allowing you to queue methods that call GetResponse to your heart's content. The less-than-good news is that you're stuck with this problem in version 1.x. One solution, as you point out, is to explicitly throttle the number of work items that you have queued to or are executing in the ThreadPool at any one time. To implement this approach, you need a way to keep track of how many work items are currently outstanding and block new requests when the predetermined limit has been reached.

Semaphores are synchronization primitives that maintain a count between zero and some maximum value. Operations are provided that raise and lower the count, the trick being that attempting to lower the count when the count is already zero causes the calling thread to block until another thread comes along and increases the count. This makes semaphores great for a few different purposes. The first, and probably one of the most common scenarios discussed in a college operating systems class, is a producer/consumer model where producer threads generate products later used by consumer threads. The second, and arguably a superset of the first, is the controlling of access to shared resources that can support only a limited number of users. In such a scenario, the semaphore acts as a guard, blocking users' attempts to access a resource while the maximum allowed number of users are currently accessing it. Sounds perfect for this scenario, right?

In fact, it is. Unfortunately, version 1.x of the .NET Framework does not include a semaphore implementation. However, putting together a simple wrapper for the Win32® semaphore object requires only a handful of code, as shown in Figure 1 (note that the .NET Framework version 2.0 does include a Semaphore implementation, and a much more complete one than the class shown here). The DllImport attribute is used with P/Invoke to wrap CreateSemaphore (the Win32 function exposed from kernel32.dll and used to instantiate a semaphore object) and ReleaseSemaphore (also exposed from kernel32.dll and used to increase the count of the semaphore). A semaphore object is referred to using the handle returned by CreateSemaphore, which makes this a perfect opportunity to derive my Semaphore class from WaitHandle and to use the functionality it provides (WaitOne, for example) for waiting on synchronization objects. Using the base class's WaitOne will cause the semaphore count to be decremented, blocking if the count is already at zero and until the count is increased or until a specified timeout elapses.

Figure 1 .NET Semaphore That Wraps Win32 Semaphore (Close Figure 1)
public sealed class Semaphore : WaitHandle
{
    public Semaphore() : this(1, 1) {}

    public Semaphore(int initialCount, int maximumCount)
    {
        if (initialCount < 0 || initialCount > maximumCount)
            throw new ArgumentOutOfRangeException("initialCount");
        if (maximumCount < 1)
            throw new ArgumentOutOfRangeException("maximumCount");
        IntPtr h = CreateSemaphore(
            IntPtr.Zero, initialCount, maximumCount, null);
        if (h == WaitHandle.InvalidHandle || h == IntPtr.Zero)
            throw new Win32Exception();
        Handle = h;
    }

    public void ReleaseOne()
    {
        int previousCount;
        if (!ReleaseSemaphore(Handle, 1, out previousCount))
            throw new Win32Exception();
    }

    [DllImport("kernel32.dll", SetLastError=true)]
    private static extern IntPtr CreateSemaphore(
        IntPtr lpSemaphoreAttributes, int lInitialCount,
        int lMaximumCount, string lpName);

    [DllImport("kernel32.dll", SetLastError=true)]
    private static extern bool ReleaseSemaphore(
        IntPtr hSemaphore, int lReleaseCount, out int lpPreviousCount);
}

With this semaphore, it's now fairly straightforward to create a class that throttles items queued to the ThreadPool. Such an implementation is shown in Figure 2. The semaphore is initially configured to have an initial and maximum count equal to the maximum number of requests you want to allow to execute simultaneously. When ThreadPoolThrottle.QueueUserWorkItem is invoked, the semaphore's WaitOne method is called to decrement the count and block if the maximum number of requests are currently executing. Just as with the ThreadPoolWait class from the October 2004 installment of .NET Matters, the user-specified WaitCallback and its associated state are packed into a new state object which is then queued to the ThreadPool as the state for a private method, HandleWorkItem. When invoked by the ThreadPool, HandleWorkItem invokes the user-specified delegate with the user-specified state. It then increments the semaphore to signal that this unit of work has completed execution, allowing a blocked thread to wake up and continue with its request. Assuming nothing else is using the pool at the same time (thus decreasing the number of available threads), a class like ThreadPoolThrottle can be used to successfully throttle the number of work items queued.

Figure 2 Throttling Requests to the ThreadPool (Close Figure 2)
public class ThreadPoolThrottle : IDisposable
{
    private Semaphore _throttle;

    public ThreadPoolThrottle(int maximumAllowed)
    {
        if (maximumAllowed < 1)
            throw new ArgumentOutOfRangeException("maximumAllowed");
        _throttle = new Semaphore(maximumAllowed,maximumAllowed);
    }

    public void QueueUserWorkItem(WaitCallback callback)
    {
        QueueUserWorkItem(callback, null);
    }

    public void QueueUserWorkItem(WaitCallback callback, object state)
    {
        if (_throttle == null)
            throw new ObjectDisposedException(this.GetType().FullName);
        if (callback == null)
            throw new ArgumentNullException("callback");

        _throttle.WaitOne();
        try
        {
            QueuedCallback qc = new QueuedCallback();
            qc.Callback = callback;
            qc.State = state;
            ThreadPool.QueueUserWorkItem(
                new WaitCallback(HandleWorkItem), qc);
        }
        catch
        {
            _throttle.ReleaseOne();
            throw;
        }
    }

    private void HandleWorkItem(object state)
    {
        QueuedCallback qc = (QueuedCallback)state;
        try { qc.Callback(qc.State); }
        finally { _throttle.ReleaseOne(); }
    }

    private class QueuedCallback
    {
        public WaitCallback Callback;
        public object State;
    }

    public void Dispose()
    {
        if (_throttle != null)
        {
            ((IDisposable)_throttle).Dispose();
            _throttle = null;
        }
    }
}

Q I'm writing a tool that uses reflection. Given a method on a class, I need to figure out which interface method that class method implements. If it doesn't implement an interface, I need to know that, too. Is this possible?

A First, the question is a bit misleading. You seem to be implying that a method on a class can only be used to implement one method on one interface, but that in fact is incorrect. A class method can be used to implement methods on multiple interfaces, or even multiple methods on the same interface, though not all languages support this functionality. Take the following C# sample as an example:

interface SomeInterface1 {
    void Method1();
}

interface SomeInterface2 {
    void Method1();
}

class SomeClass : SomeInterface1, SomeInterface2 {
    public void Method1(){}
}

Here, the method SomeClass.Method1 implements two interface methods, SomeInterface1.Method1 and SomeInterface2.Method1, which both use implicit interface implementations. The C# language provides two ways in which a class method can be mapped to an interface method. One way is through explicit interface member implementation, indicated by naming the class member with both the name of the interface and the name of the interface method, as such:

class SomeClass : SomeInterface1 {
    void SomeInterface1.Method1() {}
}

This causes the implementation to be exposed publicly but only through the interface. The second way, implicit implementation, is done by matching nonstatic public methods to interface methods by name, type, and formal parameter lists. As such, whereas in C# it's possible (as shown previously) to have one class method implement a method on two different interfaces, it isn't possible in C# to have one class method implement two different interface methods from the same interface. This is because it's invalid to have two methods on the same interface that have the same name, type, and formal parameters, a situation which would be required for a single class method to map to both. Visual Basic® .NET solves this problem by forcing the developer to explicitly state what interface methods a class method is implementing. Thus, in Visual Basic .NET it is possible to have one class method implement two different interface methods, both from the same interface, as shown in the following code snippet:

Interface SomeInterface
    Sub Method1()
    Sub Method2()
End Interface

Class SomeClass
    Implements SomeInterface
    Public Sub SomeMethod() Implements _
            SomeInterface.Method1, SomeInterface.Method2
        Console.WriteLine("SomeMethod called.")
    End Sub
End Class
Here, SomeClass.SomeMethod actually implements both SomeInterface.Method1 and SomeInterface.Method2.

With that said, back to the core of your question. Yes, it is possible using reflection to determine what interface methods a given class method implements, and one possible solution for doing so is shown in Figure 3.

Figure 3 Determining the Interfaces a Method Implements Implements (Close Figure 3)
public static Type[] GetImplementedInterfaces(MethodInfo mi)
{
    ArrayList list = new ArrayList();

    Type reflectedType = mi.ReflectedType;

    foreach(Type ift in reflectedType.GetInterfaces())
    {
        foreach(MethodInfo target in
            reflectedType.GetInterfaceMap(ift).TargetMethods)
        {
            if (target.Equals(mi))
            {
                list.Add(ift);
                break;
            }
        }
    }

    return (Type[])list.ToArray(typeof(Type));
}

The Type class exposes the GetInterfaceMap method which, when passed a Type representing an interface, is used to return a mapping that denotes how that interface is mapped into the actual methods on the class that implements that interface. InterfaceMap, the type returned by GetInterfaceMap, exposes two important fields: TargetMethods and InterfaceMethods. These fields store arrays, the former being an array of MethodInfo objects representing the methods on the type that implement interface methods, and the latter being the interface methods that map to the former array. So for example, the method represented by the MethodInfo at TargetMethods[2] implements the interface method represented by the MethodInfo stored at InterfaceMethods[2].

My GetImplementedInterfaces method shown in Figure 3 takes a MethodInfo representing the method in question, and it returns an array of Type instances, each of which represents one interface that method is used to implement. First, the method's ReflectedType is retrieved, which is necessary in order to retrieve all of the interfaces that type implements as well as the interface mapping for that type's methods. You might notice that the MemberInfo class (from which MethodInfo derives) exposes a DeclaringType property in addition to ReflectedType. It's important that I not use DeclaringType for this purpose. The difference is that DeclaringType returns the type that declares this member, whereas ReflectedType returns the type that was used to retrieve the given MethodInfo. When are they different, and why does the distinction matter? For every interface implemented by the method's container class, I need to loop through each of the target methods in the interface map and see if the specified method matches any of those. If it does match, then the specified method is in the interface map for the given interface, which means that the method is used to implement that interface. However, there are certain situations in which you can obtain a MethodInfo for an interface-implementing method but where that MethodInfo won't match any MethodInfo in the TargetMethods array. Specifically, this can happen when the declaring type and the reflected type differ. Consider the following example:

interface SomeInterface {
    void Method1();
}

class Base : SomeInterface {
    public void Method1() {}
}

class Derived : Base {}

Derived.Method1 is actually implemented on the Base class. As a result, the MethodInfo retrieved using

typeof(Derived).GetMethod("Method1")
will be a different object than the MethodInfo retrieved using
typeof(Base).GetMethod("Method1")

If I were to use the GetInterfaceMap from the Base Type (which in this case is the type that declares Method1) to retrieve the interface mapping, but the MethodInfo had been retrieved using typeof(Derived), the MethodInfo won't match any of the objects in the TargetMethods array. Thus, it's important to call GetInterfaceMap on the same Type used to retrieve the specified MethodInfo, and that Type is the one returned from the MethodInfo's ReflectedType property.

Q I've been reading recently about COM threading models and how .NET fits into the picture. I know I can use the STAThreadAttribute and MTAThreadAttribute to initially set the threading model for the main thread of my application, and I know I can configure a thread using the Thread.ApartmentState property, but both of those techniques only seem to work before any calls to COM objects are made. After that, they appear to be immutable. Is that true? If so, what do I do if I need to use multiple COM objects, each of which uses a different threading model?

A The Thread.ApartmentState property can be changed as often as you like until the first calls are made to a COM object. At that point, the threading model for the current thread becomes immutable and you won't be able to change it. If you need to use COM objects that require a different threading model than that of your current thread, the standard solution is to spin up a new thread, set its ApartmentState appropriately, and then execute the code that uses that COM object on that new thread. (For a refresher in COM threading models, I suggest reading Larry Osterman's blog post at What are these "Threading Models" and why do I care?.)

To make this process a bit easier, I've implemented the class shown in Figure 4. ApartmentStateSwitcher exposes only one static method named Execute. This method takes a delegate to invoke, the parameters to pass to the delegate, and the ApartmentState required to execute this delegate. If the ApartmentState of the current thread matches the one requested by the user, the method simply invokes the delegate using the delegate's DynamicInvoke method, which provides for late-bound method invocation. However, if the apartment states differ, it invokes the delegate on a new thread with the ApartmentState set appropriately. To do this, it creates a small object that it can use to pass the state back and forth between the current and new thread. Into this object it stores the delegate and the parameters with which to invoke it.

Figure 4 ApartmentStateSwitcher (Close Figure 4)
public sealed class ApartmentStateSwitcher
{
    private Delegate _delegate;
    private object[] _parameters;
    private Exception _exc;
    private object _rv;

    private ApartmentStateSwitcher(){}

    private void Run()
    {
        try { _rv = _delegate.DynamicInvoke(_parameters); }
        catch(MemberAccessException exc) { _exc = exc; }
        catch(TargetException exc) { _exc = exc; }
        catch(TargetInvocationException exc) { _exc = exc; }
    }

    public static object Execute(
        Delegate d, object[] parameters, ApartmentState state)
    {
        if (d == null) throw new ArgumentNullException("d");
        if (state != ApartmentState.MTA && state != ApartmentState.STA)
            throw new ArgumentOutOfRangeException("state");

        if (Thread.CurrentThread.ApartmentState == state)
        {
            return d.DynamicInvoke(parameters);
        }
        else
        {
            ApartmentStateSwitcher switcher =
                new ApartmentStateSwitcher();
            switcher._delegate = d;
            switcher._parameters = parameters;

            Thread t = new Thread(new ThreadStart(switcher.Run));
            t.ApartmentState = state;
            t.IsBackground = Thread.CurrentThread.IsBackgound;
            t.Start();
            t.Join();

            if (switcher._exc != null) throw switcher._exc;
            return switcher._rv;
        }
    }
}

A new thread is then created with the appropriate ApartmentState, started, and immediately joined with the current thread, blocking the current thread until the new thread finishes execution. The main method for this new thread is Run, a private member of the class that executes the delegate with the supplied parameters. Any return value from the delegate's invocation is stored to the state class, as is any exception thrown during the process (in version 1.x of the .NET Framework, exceptions thrown on worker threads are simply eaten by the runtime, and in version 2.0, worker thread exceptions cause the AppDomain to shut down; in neither version are exceptions passed back to the main thread, so I have to do that manually). Back in the Execute method, when the worker thread finishes, the Thread.Join call completes and the state class is examined for an exception and a return value. If an exception exists, it is rethrown. If not, the return value from the delegate is returned.

Using this class is as easy as creating a delegate around whatever code you want executed (a process made much simpler with anonymous delegates in C# 2.0) and passing that delegate to ApartmentStateSwitcher.Execute, as shown here:

[MTAThread]
static void Main(string[] args) {
    PrintCurrentState();
    ApartmentStateSwitcher.Execute(
        new ThreadStart(PrintCurrentState), null, ApartmentState.STA);
    PrintCurrentState();
}

static void PrintCurrentState() {
    Console.WriteLine("Thread apartment state: " +
        Thread.CurrentThread.ApartmentState);
}
This code prints to the console:
Thread apartment state: MTA
Thread apartment state: STA
Thread apartment state: MTA

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


C++ Q&amp;A: Deleting Managed Objects, Wrapping a Library, and More

New information has been added to this article since publication.
Refer to the Editor's Update below.

C++ Q&A
Deleting Managed Objects, Wrapping a Library, and More
Paul DiLascia

Code download available at: CQA0412.exe (235KB)

Q In managed C++, can you tell me if it is safe to destroy a managed object using operator delete?

Bernie Sanders

A Yes, you can delete managed objects in managed C++, as long as you understand that all delete does is call the object's destructor, which must be explicitly defined. Calling delete does not free the object's storage. Only the garbage collector can do that. Figure 1 shows a simple program that defines a managed class with a destructor that displays a message when it runs. TESTDTOR allocates two instances of ManagedClass. It explicitly deletes the first instance but not the second. If you run TESTDTOR, you'll get results something like this:

Begin main
  ManagedClass(04A712D4)::ctor
  ManagedClass(04A712D4)::dtor
  ManagedClass(04A712E0)::ctor
End main
  ManagedClass(04A712E0)::dtor

Figure 1 testdtor.cpp (Close Figure 1)
////////////////////////////////////////////////////////////////
// MSDN Magazine — December 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
// This program illustrates what happens when you delete a managed object
// in Managed C++: the compiler calls the object's dtor.
//
// To compile: cl /clr testdtor.cpp

#include <stdio.h>
#using <mscorlib.dll>

using namespace System;

//////////////////
// Managed class WITH dtor.
//
public __gc class ManagedClass {
public:
   ManagedClass() {
      printf(" ManagedClass(%p)::ctor\n", this);
   }
   ~ManagedClass() {
      printf(" ManagedClass(%p)::dtor\n", this);
   }
};

//////////////////
// Managed class WITHOUT dtor
//
public __gc class ManagedClassNoDtor {
public:
   ManagedClassNoDtor() {
      printf(" ManagedClassNoDtor(%p)::ctor\n", this);
   }
};

void main()
{
   printf("Begin main\n");

   ManagedClass *pmc = new ManagedClass();   // create instance
   delete pmc;                               // "destroy" (force-call
                                             // dtor)
   pmc = new ManagedClass();                 // create another
// delete pmc;                               // don't destroy
                                             // (cleaned up when pgm
                                             // terminates)

   ManagedClassNoDtor *pmcnd = new ManagedClassNoDtor();
// delete pmcnd; // ERROR!: can't delete instance of class with no dtor!

   printf("End main\n");
}

This shows that the first object's destructor executes immediately when the delete statement executes; whereas the second object (at 04A712E0) isn't destroyed until after control leaves main and the system termination code invokes the garbage collector to free outstanding objects.

Figure 2 testdtor Highlights
Figure 2 testdtor Highlights

Whenever you're unsure about what happens in the .NET environment, you can always write some code, compile it, and inspect the Microsoft® intermediate language (MSIL) generated. As Figure 2 shows, defining a destructor causes the compiler to generate two methods: a Finalize method that contains your implementation (in this case, the call to printf), and the __dtor method that calls System.GC::SuppressFinalize, then Finalize. When you delete an object, the compiler generates a call to this special __dtor method. If you compile TESTDTOR with /FAs to generate an assembly listing with source code, you'll see that the delete statement compiles in the following manner:

; delete pmc;
ldloc.0  ; _pmc$
call ??1ManagedClass@@$$FQ$AAM@XZ
The strangely cryptic symbol is just the mangled name for the destructor (__dtor) function.

Seasoned C++ programmers may scratch their heads and wonder if calling delete doesn't free the object, what's the point of calling it? Good question. The only reason to call delete would be to reclaim any unmanaged resources your class uses. For example, if your object opens files or creates a database connection, you can write a destructor that closes its resources and then use delete to invoke it when you're done using the object. Alas, a better way to free resources in managed classes is by implementing the Dispose pattern, IDisposable and—if you're writing in managed C++—an auto_dispose pattern to call it. (For further information see "Tips and Tricks to Bolster Your Managed C++ Code in Visual Studio .NET" by Tomas Restrepo in the February 2002 MSDN®Magazine).

If you implement the dispose pattern, other consumers of .NET can use it. If you do your cleanup in the destructor, other languages have no way to explicitly call your cleanup code. There's no delete operator in C# or Visual Basic®.

So the upshot is that you can call delete to invoke your destructor, but it's probably not a good idea to put cleanup code in the destructor. Better to implement IDisposable, which all citizens can use. Note, though, that this behavior changes in Visual C++® 2005. For more information, see Andy Rich's discussion of deterministic finalization at Deterministic Finalization IV - Benefits, part II as well as the current C++/CLI Language Specification Standard.

Q I have an unmanaged function returning a linked list that has char* strings in it:

struct blah {
   int a, b;
   char *a, *b;
   struct blah *next;
};
struct blah *getmystruct();
Because getmystruct() allocates memory, I need to call freemystruct(struct blah *b) when I'm finished with it. I'm trying to make a wrapper that will convert it to a collection of managed types, but I'm not sure how to deal with all these pointers as they need to be freed. Could you offer any insight?

Cory Nelson

A Why, surely. You can't convert your native list to a collection of managed types with a simple dllimport statement. The interop services are good, but not that good! You need to write a wrapper that explicitly converts your list to a managed collection such as an ArrayList. I wrote a three-module program, ListWrap, that shows how. The first module, ListLib.cpp, implements a native C++ library (DLL) with two functions, AllocateList and FreeList, which allocate and free a linked list of native C++ structs. These simulate the functions getmystruct and freemystruct in your application. The second module is a managed C++ file, ListWrap.cpp, which implements a managed class ManagedNode that wraps the native C++ implementation (see Figure 3). [ Editor's Update - 1/11/2005: The ListWrap sample as published had a memory leak because ListWrap.cpp failed to delete nn->str when deleting the node. This has been corrected by calling FreeList instead of deleting the node manually.] The third module is a C# test program which calls the wrapper to show how it works. You can download the full source for ListLib.cpp and the C# test program from the MSDN Magazine Web site.

Figure 3 ListWrap.cpp (Close Figure 3)
////////////////////////////////////////////////////////////////
// MSDN Magazine — December 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
#using <mscorlib.dll>
#include "ListLib.h"

using namespace System;
using namespace System::Collections;

// Handy definition for human readability.
#define MANAGED __gc

namespace ListWrap {

//////////////////
// Managed wrapper for NativeNode mirrors definition.
//
public MANAGED class ManagedNode
{
public:
   int a, b;      // same as NativeNode::a, NativeNode::b
   String* str;   // managed String, not char*

   // static
   static ArrayList* CreateList(int nelt) {
      NativeNode* nnlist = AllocateList(nelt);
      ArrayList*  ar = new ArrayList(); // create managed array list

      // Copy each NativeNode to managed ManagedNode
      for (NativeNode* nn=nnlist; nn; nn=nn->next) {
         ManagedNode* mn = new ManagedNode();
         mn->a = nn->a;
         mn->b = nn->b;
         mn->str = nn->str; // it just works!
         ar->Add(mn);       // append to list
      }
      FreeList(nnlist); // free the whole list
      return ar;        // ..and return managed copy
   }
};

}

ListLib.cpp implements two native functions, AllocateList and FreeList, which allocate and free a list of NativeNode structs:

// from ListLib.h
struct NativeNode {
   int a, b;
   TCHAR *str;
   struct NativeNode *next;
};

The wrapper class ManagedNode in ListWrap.cpp mimics the definition of NativeNode with a couple of minor differences: the native char* is replaced with a managed String and there's no next pointer since I'll use ArrayList to implement the list structure. In code, it looks like this:

// managed equivalent of NativeNode
public __gc class ManagedNode {
public:
   int a, b;
   String* str;
};

With ManagedNode defined, the next step is to write some code that converts NativeNodes to ManagedNodes. But before you run to the keyboard, stop a moment to consider what the converter function should look like and what arguments and return value it should have. One way is to write a function that takes a native list of NativeNodes and returns a managed list of ManagedNodes, possibly destroying the native list in the process. .NET client applications would call the ListLib DLL (or your getmystruct) directly to obtain the native list as an IntPtr, then pass this IntPtr to the conversion function, as shown here:

// call DLL directly through interop
IntPtr nativeList = AllocateList(7);
// call wrapper to convert
ArrayList amanagedList = ListWrap.Convert(nativeList);
The client would be responsible for calling the DLL to free the native list in most cases, or perhaps the Convert function would do it automatically.

A different approach is to hide the DLL completely by wrapping the native function AllocateList in a wrapper that allocates the list, converts it, and frees the original native list before returning the managed list as an ArrayList. Which way is better? The advantage of the first strategy is that you only have to write a single conversion function which can be used anywhere you have a native list. The second approach requires wrapping each function that creates a list. It's a little more work if you have several functions that create lists, but it has the advantage of completely hiding all the native stuff from .NET clients. There's no need for clients to deal with IntPtrs or even import the DLL; ListWrap hides everything. This is the approach I'll take and the one I encourage you to use in your own apps. It takes more effort to fully wrap your library, but the result is more reliable and fully encapsulated.

With ManagedNode in hand, all that remains is to wrap AllocateList. The procedure is very straightforward. First, call AllocateList to allocate the native list; and then create an empty ArrayList. Next copy all the NativeNodes to ManagedNodes and add them to the managed list, deleting them as you go. Figure 3 shows the full details. One of the beauties of managed C++ is that all the code looks simple and neat, even when you're dealing with mixed objects. Copying the native char* to a managed String is just a simple assignment, as shown in the following line:

mn->str = nn->str; // String = char*: it just works!

There is no need to call a conversion function; the compiler knows what to do. CreateList deletes the native nodes as it goes. This is more storage-efficient than deleting them at the end.

By converting the entire list to managed objects (instead of exporting it with interop and StructLayout), you let managed clients live in a managed world. When in Rome, do as the Romans do! After all, one of the main reasons some programmers choose .NET is automatic garbage collection. If you export your list directly with interop, you'll have to export FreeList too, and make programmers using other .NET-based languages remember to call it.

In general, if you're exporting to the managed world, it's best to convert as much of your data as possible to managed objects. Otherwise what's the point? Your clients may as well write in C++. Of course, this rule doesn't always apply. Sometimes it's better to export structures directly and require that clients free them—for example, if copying entails an unacceptable performance or memory hit. You have to use judgment in deciding whether to go managed or native.

Q I'm using the Managed Extension for C++ to wrap an existing C++ library so that .NET-based languages can access it. In managed C++, I can write

String* s = new String();
s = _T("Hello, world");
but how can I convert a managed String back to a native TCHAR*?

Matthew Brady

A It's easy, once you know the magic voodoo. You have to call PtrToStringChars and pin the result. In code, it looks like this:

String __gc* s = S"Hello";
const wchar_t __pin* p = PtrToStringChars(s);
Don't forget to __pin the pointer returned from PtrToStringChars. Pinning is required because PtrToStringChars returns a managed (__gc) pointer to the first character of the String object in managed memory, and the garbage collector can move managed memory any time it pleases, unless you explicitly __pin it. In general, you must use __pin any time you pass a __gc pointer to a native (unmanaged) function.

Figure 4 shows a short program that converts a managed String to both wide-character and ANSI strings. To convert to ANSI, use your favorite conversion function, such as wcstombs or the ATL W2A macro. If you're using MFC CStrings, you don't have to do anything since CString has assignment operators for both char* and wchar_t:

// both will work
CString s1 = "hello, world";
CString s2 = L"Hello, world";

Figure 4 ConvertString.cpp (Close Figure 4)
////////////////////////////////////////////////////////////////
// MSDN Magazine — December 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
#include < stdio.h >
#include < stdlib.h >
#include < vcclr.h >
#using < mscorlib.dll >
using namespace System;

//////////////////
// Shows how to convert managed String to char* or wchar_t
//
int main()
{
   // A managed string
   String* s = S"Hello, world";

   // Convert to wide char
   const wchar_t __pin* p = PtrToStringChars(s);
   printf("%S\n", (wchar_t*)p);

   // Convert to char:
   size_t len = wcstombs(NULL, p, 0)+1; // calculate required length..
   if (len == 0) return -1;
   char* buf = (char*)malloc(len);      // ..allocate new string and..
   wcstombs(buf, p, len);               // ..convert
   printf("%s\n", buf);
   return 0;
}

Q I want to change the background color of tab control for my app from gray to white. I tried to make a derived class from CTabCtrl and use all functionalities, but without success. Can you help me?

Mayur Patel

A Changing the color of the tabs in your tab control is fairly easy, but giving property sheets a full color makeover requires quite a bit of work and isn't for the fainthearted. For the tabs, the basic idea is to make the control an owner-draw control and then handle WM_DRAWITEM. If you're using MFC, you can override the virtual function DrawItem.

In the March 1998 issue of Microsoft Systems Journal, I showed how to implement a tab control class, CTabCtrlWithDisable, that supports disabling the tabs. As part of disabling the tabs, CTabCtrlWithDisable changes the tab text color to light gray when the tab is disabled. This month, I borrowed some code from CTabCtrlWithDisable to implement a new class, CColorTablCtrl, that lets you change the color of the tabs (see Figure 5).

Figure 5 ColorTab.cpp (Close Figure 5)
////////////////////////////////////////////////////////////////
// MSDN Magazine — December 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
// CColorTabCtrl implements a CTabCtrl with colored tabs.

#include "StdAfx.h"
#include "ColorTab.h"

#ifdef _DEBUG
#define new DEBUG_NEW
#undef THIS_FILE
static char THIS_FILE[] = __FILE__;
#endif

IMPLEMENT_DYNAMIC(CColorTabCtrl, CTabCtrl)
BEGIN_MESSAGE_MAP(CColorTabCtrl, CTabCtrl)
END_MESSAGE_MAP()

CColorTabCtrl::CColorTabCtrl()
{
   m_clrBackground = GetSysColor(COLOR_3DFACE);
   m_clrForeground = GetSysColor(COLOR_BTNTEXT);
}

CColorTabCtrl::~CColorTabCtrl()
{
}

//////////////////
// Set bg/gf colors
//
void CColorTabCtrl::SetColors(COLORREF bg, COLORREF fg)
{
   m_clrBackground = bg;
   m_clrForeground = fg;
}

//////////////////
// Subclass the tab control: also make owner-draw
//
CColorTabCtrl::SubclassDlgItem(UINT nID, CWnd* pParent)
{
   if (!CTabCtrl::SubclassDlgItem(nID, pParent))
      return FALSE;
   ModifyStyle(0, TCS_OWNERDRAWFIXED);
   return TRUE;
}

//////////////////
// Draw the tab: use bg/fg colors.
//
void CColorTabCtrl::DrawItem(LPDRAWITEMSTRUCT lpDrawItemStruct)
{
   DRAWITEMSTRUCT& ds = *lpDrawItemStruct;

   int iItem = ds.itemID;

   // Get tab item info
   char text[128];
   TCITEM tci;
   tci.mask = TCIF_TEXT;
   tci.pszText = text;
   tci.cchTextMax = sizeof(text);
   GetItem(iItem, &tci);

   // use draw item DC
   CDC dc;
   dc.Attach(ds.hDC);

   // calculate text rectangle and color
   CRect rc = ds.rcItem;
   rc += CPoint(1,4);                   // ?? by trial and error

   dc.FillSolidRect(rc, m_clrBackground);
   dc.SetBkColor(m_clrBackground);
   dc.SetTextColor(m_clrForeground);
   dc.DrawText(text, &rc, DT_CENTER|DT_VCENTER);

   dc.Detach();
}

IMPLEMENT_DYNAMIC(CColorPropertyPage, CPropertyPage)
BEGIN_MESSAGE_MAP(CColorPropertyPage, CPropertyPage)
   ON_WM_ERASEBKGND()
END_MESSAGE_MAP()

CColorPropertyPage::CColorPropertyPage(UINT nID, COLORREF bg) : CPropertyPage(nID)
{
   SetBGColor(bg);
}

CColorPropertyPage::~CColorPropertyPage()
{

}

void CColorPropertyPage::SetBGColor(COLORREF bg)
{
   m_clrBackground = bg;
}

//////////////////
// Handle WM_ERASEBKGND: fill bg with bg color
//
BOOL CColorPropertyPage::OnEraseBkgnd(CDC* pDC)
{
   CRect rc;
   GetClientRect(&rc);
   pDC->FillSolidRect(rc, m_clrBackground);
   return TRUE;
}

To use CColorTablCtrl, create an instance in your property sheet:

class CMyPropSheet : public CPropertySheet {
protected:
   CColorTabCtrl m_tabCtrl;
};
You have to subclass the tab control in your property sheet's OnInitDialog handler (so MFC will use it), then set the foreground and background colors to whatever you like:
// in CMyPropSheet::OnInitDialog()
HWND hWndTab = (HWND)SendMessage(PSM_GETTABCONTROL);
m_tabCtrl.SubclassDlgItem(::GetDlgCtrlID(hWndTab), this);
m_tabCtrl.SetColor(WHITE, RED);

Here WHITE and RED are standard COLORREF values, namely RGB(255, 255, 255) and RGB(255,0,0). Once you've instantiated and initialized your CColorTabCtrl, the color tab control does the rest (see Figure 6).

Figure 6 The Color Tab Control
Figure 6 The Color Tab Control

CColorTabCtrl has an override for SubclassDlgItem that calls ModifyStyle to change the style to TCS_OWNERDRAWFIXED. A better place to do this would be in PreSubclassWindow because that's called whether the control is subclassed or created with CreateWindow (but I had to shrink the code to fit in the magazine, so I took a shortcut). Note that SubclassDlgItem is a simple override, not a virtual function. To set the colors, SetColor saves the colors passed in two member variables, m_clrBackground and m_clrForeground.

Once the style is set to owner-draw, Windows® sends WM_DRAWITEM messages whenever it's time to paint the tabs. MFC traps this message and calls the tab control's virtual DrawItem function, which CColorTabCtrl implements by painting the text with the new colors:

// in CColorTabCtrl::DrawItem
dc.FillSolidRect(rc, m_clrBackground);
dc.SetBkColor(m_clrBackground);
dc.SetTextColor(m_clrForeground);
dc.DrawText(...);

This is all quite straightforward, so see the source code for details. Since you probably don't want to change the tab color without changing the page color as well, I also implemented a CColorPropertyPage that lets you change the background color of the property page to match, as shown in Figure 6. For property sheets, the easiest way to change the background color is by handling WM_ERASEBKGND:

BOOL CColorPropertyPage::OnEraseBkgnd(CDC* pDC)
{
   CRect rc;
   GetClientRect(&rc);
   pDC->FillSolidRect(rc, m_clrBackground);
   return TRUE;
}

If you try this at home, you'll discover various annoying problems. First, if you change the page color, all the controls have the wrong background color, so you have to fix them too. For that you have to handle WM_CTLCOLOR and WM_ERASEBKGND. The details appear in my MSJ column from way back in May 1997.

Another problem is that the tab control still draws the edges and rounded corners of the tabs using the system 3D colors. Sigh. To fix that, there's nothing to do but handle WM_PAINT and take over the entire paint operation yourself. This includes drawing the tab offset from the others when it's selected, so it appears in the foreground. By this point, you're starting to reinvent the tab control. As every programmer using Windows knows, changing control colors is a royal pain in the brain, and once you start down this path it seems there's no end to the work that must be done. Pretty soon the standard colors will look better than they did at first, or else you'll wonder why you don't switch to using the .NET Framework where changing colors is as simple as writing:

ctl.BackColor = Color.Aquamarine;

Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at www.dilascia.com.


{ End Bracket }: Around the World with Visual Basic

{ End Bracket }
Around the World with Visual Basic
Jay Roxe

Contents
How the Heck Do You Do That?
What's the Best Way to Take Advantage of Visual Basic 6.0 Code?

The Visual Basic team at Microsoft has spent much of the past eight months out on the road as a part of the Visual Basic® .NET User Group Tour 2004. We've had a chance to visit 13 user groups in the U.S. and Canada, another 12 in Europe, and 9 in Asia and India. Some of the best memories include the 3-hour presentation in Portland that kicked off the tour, fielding questions from 300 people in California, and hanging out with developers until 1 A.M. in Croatia. For some photos from the tour, check out the Web site at Visual Basic World Tour.

In every city we've visited, developers have not been shy about asking questions. After hearing similar questions asked in different groups, I decided there must be more people out there with the same questions, so I want to take this opportunity to answer two of the most common ones.

How the Heck Do You Do That?

Over the years, we've had many people approach us wondering how we implemented a particular feature or control. As a part of this User Group Tour we've been demonstrating the Visual Basic Power Pack. This is a set of seven new controls that enhance your ability to create powerful user interfaces for your client development. It includes controls such as a BlendPanel that makes it easy to use a gradient fill, a NotificationWindow that allows you to add pop-ups (much like an instant message) to your application, and a Utility Toolbar that easily creates a user experience similar to the Microsoft® Internet Explorer toolbar. To answer the question about how we implemented these controls, we've made the source code for the Power Pack controls available under a Shared Source license agreement and put them into a GotDotNet workspace. For those of you unfamiliar with this agreement, it allows you to see the source and contribute modifications back to the community. By the time you read this, we will also have made the code for IssueVision, the very popular Smart Client sample application from DevDays 2004, available on GotDotNet.

I encourage you to take a look at the code, play with it, use it in your apps, modify it, and share it with others. Hopefully you'll be able to take advantage of some of the things we did right, and you'll send us corrections for anything we got wrong! Releasing sample code on GotDotNet is always a bit of an experiment. If we find a dynamic community forming around this, we'll add more code later. You'll need to let us know what type of code you want to see. You can find information about both the Visual Basic Power Pack and IssueVision at the Visual Basic Developer Center.

What's the Best Way to Take Advantage of Visual Basic 6.0 Code?

Many of the developers who saw us during the tour were curious about the best ways to leverage their existing Visual Basic 6.0 code base and ease the migration to Visual Basic 2005. One of the new features that will definitely help leverage existing assets is Registration-Free COM. This allows you to use COM objects without having to register them on the local machine—just use ClickOnce or XCopy deployment to get your program to any machine running Windows® XP. It's a great way to overcome the headaches you get when deploying apps that use COM components. It works for ActiveX® controls, but it can also work with nonvisual business objects. Not all COM components are suitable (add-ins and out-of-proc servers don't work), but this makes it fundamentally easier to reuse your existing COM controls as a part of your Visual Basic solution. With a couple of minor limitations, this works in the beta version of Visual Studio 2005.

In addition to the new product features, we've invited some enterprise customers with large apps built on Visual Basic 6.0 into one of our migration labs and we've worked with them to start the migration process. We're exploring ways to make the content that we present in those labs more widely available. Here are two of the most interesting observations to come out of these migrations:

  • Choose a particular piece of your application to migrate first. It doesn't matter if this is a horizontal or vertical slice of the application, but just migrate one piece and use the rest of the application through COM interop.
  • Use the Visual Basic 6.0 Code Advisor. This is a free Visual Basic 6.0 plug-in that makes suggestions on how to modify your code so that your migration is more successful. Sometimes fairly minor fixes can make a big difference.

To find a user group in your area, see the International .NET Association at www.ineta.org. Hope we'll see you out on the road soon!

Jay Roxe is the Product Manager for Visual Basic at Microsoft where he is responsible for product planning, technical evangelism, and external communication. Prior to this role, Jay was a developer and development lead on the .NET Framework.