 MSDN Magazine June 2003

MSDN Magazine June 2003

Windows Server 2003: Discover Improved System Info, New Kernel, Debugging, Security, and UI APIs

There's a lot to say about Windows Server 2003. First of all, it's the first operating system with built-in .NET Framework support, and it's the first 64-bit OS from Microsoft. But wait, there's more! There are lots of new features and APIs in this version as well. For instance, Windows Server 2003 features Hot Add Memory and a number of other arcane new tidbits. There are new APIs for handling threads, directories, and files, and new features like the low fragmentation heap for managing memory and system information. There's vectored exception handling and new UI APIs as well.OS internals expert Matt Pietrek takes a look at the additions he finds most interesting and useful so you'll have a good place to start when you dive into Windows Server 2003.

Matt Pietrek ASP.NET Pipeline: Use Threads and Build Asynchronous Handlers in Your Server-Side Web Code

Fortunately for developers, threading in ASP.NET is a lot easier than it was in ASP. In this article, the author takes a look at threading in the ASP.NET HTTP pipeline, and explains how threads are managed efficiently without the involvement of the developer. The article considers how the common language runtime threadpool is used by ASP.NET to service requests, looks at the pooling mechanisms used for handlers, modules, and applications, and covers both IIS 5.0 and IIS 6.0 and how they differ in their approach to request processing and thread allocation. Finally, how and when to use asynchronous handlers is discussed for developers who still need to use threads in their own applications.

Fritz Onion Zip Your Data: Using the Zip Classes in the J# Class Libraries to Compress Files and Data with C#

Zip compression lets you save space and network bandwidth when storing files or sending them over the wire. In addition, you don't lose the directory structure of folders you Zip, which makes it a pretty useful compression scheme. The C# language doesn't have any classes that let you manipulate Zip files, but since .NET-targeted languages can share class implementations, and J# exposes classes in the java.util.zip namespace, you can get to those classes in your C# code. This article explains how to use the Microsoft J# class libraries to create an application in C# that compresses and decompresses Zip files. It also shows other unique parts of the J# runtime you can use from any .NET-compliant language to save some coding.

Ianier Munoz .NET Remoting: Secure Your .NET Remoting Traffic by Writing an Asymmetric Encryption Channel Sink

As .NET Remoting gains popularity in the enterprise space, it must meet business demands for trustworthy computing. Remoting traffic can be secured when objects are hosted in IIS, but when they aren't hosted in IIS, custom security solutions can be developed to secure them. This article provides an in-depth look at writing channel sinks for .NET. It also details the flow of data through custom channel sinks and explains the kinds of manipulations that can be performed on that data.

Stephen Toub Smart Cleanup: Achieve More Reliable Resource Management with Our Custom C++ Classes

Managing resources in C++ is not easy. When you're unsuccessful, your app can leak all kinds of resources including file system handles, database connections, and, of course, memory. Even in garbage-collected languages like Managed C++, resource management is difficult because garbage collection only deals with memory management, not the other resources that cause performance problems.In this article, the author describes the SmartAny template library he created, which uses a policy-based approach to dynamic resource management. Readers will learn how to use the SmartAny classes and policies to ensure the proper cleanup of their resources, be they files, OS handles, or native and unmanaged objects.

Eric Niebler Editor's Note: RSS is Here!

RSS is a great, example of how a technology like XML can be used to improve the overall user experience. RSS (Rich Site Summary) is a description of a simple XML schema that can be used to describe the contents of a Web destination.

New Stuff: Resources for Your Developer Toolbox

Olero Software Inc. has announced the release of ORM. NET 1. 2, an object relational mapping tool designed for developers using the Microsoft® . NET Framework. ORM. NET auto-generates a complete data layer object model based on your SQL database schema.

Theresa W. Carey Web Q&A: Font Sizing, Internationalization in JScript, and More

Font Sizing, Internationalization in JScript, and More.

Edited by Nancy Michell Data Points: Techniques in Filling ADO.NET DataTables: Performing Your Own Analysis

How do you know which technique is best for retrieving data and populating a DataSet in ADO.NET?. Since the Microsoft .NET Framework offers so many choices on how to write the code, many developers are now taking a close look at the different options. See what they are.

John Papa The XML Files: Advanced Type Mappings

Can XmlSerializer deal with choice compositors? How about mixed content models? XmlSerializer won't serialize objects that implement IDictionary by default, so how do you get arount it? And more.

Aaron Skonnard Cutting Edge: ASP.NET Controls Templates

It's easy to create a custom control in ASP.NET by deriving a new class from an already existing control. Creating a new ASP.NET control from scratch, on the other hand, is more challenging. When you need a Web server control and none of the existing ones meet your requirements, you can derive from one of the base classes - Control or WebControl. Try it out.

Dino Esposito Advanced Basics: Adding New Features with User Controls

In past versions of Visual Basic, there were rudimentary graphics controls. In Visual Basic .NET you have the GDI+ library, which enables you to draw lines, circles, and most anything else. But how can you use the functionality of GDI+ to create lines and other graphics that respond to user mouse clicks and events? Find out.

Ken Spencer The ASP Column: Tree Controls with XSL

Manipulating the TreeView server-side control is very much like programming any other ASP.NET server-side control. There are a number of properties, methods, and events that are available both programmatically and through the designer. Find out how to take advantage of it.

George Shepherd Bugslayer: SOS: It's Not Just an ABBA Song Anymore

Hidden deep inside the Microsoft .NET Framework 1.1 you'll find Son of Strike (SOS). If your app is a pure managed code, your development and debugging tasks are easily handled by existing Microsoft tools. If you're on border between managed and native code SOS is your man.

John Robbins .NET Column: The CLR's Thread Pool

How does the thread pool work in the CLR?

Jeffrey Richter C++ Q&A: Locking Column Headers, Implementing Singleton Classes

Prevent the sizing of the column headers in an ATL composite control. Also, see how you can share a small amount of simple data among multiple processes running on the same machine without Remoting.

Paul DiLascia Resource File: MSDN Online Developer Centers

When you're looking for information about a specific technology, don't you wish you could rely on one site to point you to the most pertinent content? Now you can. Based on feedback from software developers, MSDN Online has centralized all developer information on specific technologies so you don't have to hunt in several locations.


Windows Server 2003: Discover Improved System Info, New Kernel, Debugging, Security, and UI APIs

New information has been added to this article since publication.
Refer to the Editor's Update below.

Windows Server 2003

Discover Improved System Info, New Kernel, Debugging, Security, and UI APIs

Matt Pietrek
This article assumes you're familiar with Windows, .NET, and C++
Level of Difficulty 1 2 3
Code download available at: WindowsServer2003.exe (1,024KB)
SUMMARY

There's a lot to say about Windows Server 2003. First of all, it's the first operating system with built-in .NET Framework support, and it's the first 64-bit OS from Microsoft. But wait, there's more! There are lots of new features and APIs in this version as well. For instance, Windows Server 2003 features Hot Add Memory and a number of other arcane new tidbits. There are new APIs for handling threads, directories, and files, and new features like the low fragmentation heap for managing memory and system information. There's vectored exception handling and new UI APIs as well.

OS internals expert Matt Pietrek takes a look at the additions he finds most interesting and useful so you'll have a good place to start when you dive into Windows Server 2003.

Contents
New KERNEL APIs
Processes, Threads, and Fibers, Oh My!
Vectored Exception Handling
Directories and Files
Memory and System Information
Debug APIs
Side-by-side Execution
User Interface Additions
Newly Documented Interfaces
New DEBUGHLP Additions
Windows Error Reporting
ADVAPI32
OLE Gets Old
Clustering
Real Time Client API
Conclusion

I
really am too young to be saying this, but I remember the good old days when developers eagerly awaited the release of a new version of Windows®. What new goodies might it contain, and what cool things could you do with them? With the advent of the Microsoft® .NET Framework, it's become fashionable in some circles to pretend that Windows is static and not worthy of discussion. Some people seem to think it doesn't matter which version of Windows you're running underneath.

As you might guess, I'm not in that group. While I'm as big a fan of Microsoft .NET as the next person, I still eagerly scan header files and compare the exports of system DLLs from each new version of Windows. I want to know what the fine folks on the Microsoft Windows team have been up to.

The latest and greatest operating system from Microsoft is Windows Server™ 2003, which is in its release candidate phase as I write this. In this article, I'll examine what new goodies are available in Windows Server 2003 for application-level programmers. However, before diving into the new APIs, there's some background information that helps put all this in context.

Let's first establish what "new" means. Windows Server 2003 is intended to replace the Windows 2000 Server family (Server, Advanced Server, and Datacenter Server.) As such, many of the OS features that originally appeared in Windows XP technically fall into the "new" category for the purposes of this article.

Actually, the majority of the new APIs (relative to Windows 2000) first appeared in Windows XP, rather than in Windows Server 2003. There's a good reason for that. Many moons ago, the successor to Windows 2000 was codenamed Whistler, and its betas included both workstation and server versions. In 2001, Microsoft decided that the server version needed to bake a bit longer, and so released the consumer and workstation versions as Windows XP. The intention was that the server versions would be released shortly thereafter. As you now know, that delay turned into a year and a half. Some of this time can be accounted for by the Microsoft Trustworthy Computing Initiative, wherein developers at Microsoft stopped new development in order to look for security issues in their code.

The key point is that up to a certain date, both the workstation and server versions of Whistler were essentially feature complete and used the same code base. The time between the release of Windows XP and Windows Server 2003 was primarily spent making it as robust as possible. Thus, it's not at all surprising that many of the new Windows Server 2003 features and APIs are also available in Windows XP.

The version numbers used in the Platform SDK header files give evidence of how little the public interface to the OS (that is, the APIs) has changed since Windows XP. To enable Windows XP (and later) APIs, you define _WIN32_WINNT to 0x0501 (that is, Windows XP is internally thought of as Windows 5.01.) For Windows Server 2003, the required #define value for _WIN32_WINNT has only been bumped to 0x0502. You'll see _WIN32_WINNT #defines used in the sample programs later.

Windows Server 2003 may hold the record for the most number of name changes during its incubation. It started out codenamed Whistler and then became "Windows 2002 Server." Subsequently, the .NET initiative permeated all things Microsoft, and it was rechristened "Windows .NET Server." After another delay, the name changed to "Windows .NET Server 2003." Finally, wiser heads prevailed, and the name became "Windows Server 2003." It wouldn't surprise me if I missed a name change in there somewhere.

The name changes bring up an important point, however: Windows Server 2003 is the first operating system to include the .NET Framework as an integral part of the operating system. For backward compatibility, the system has both the shiny new .NET Framework 1.1, as well as the original version 1.0 of the .NET Framework. [ Editor's Update - 2/15/2004: Windows Server 2003 only includes the .NET Framework 1.1, although version 1.0 can be installed on it manually.] Since .NET has been covered so extensively in MSDN® Magazine and elsewhere, I won't count the .NET Framework as a new API for the purposes of this article.

Besides being the first system with built-in support for the .NET Framework, Windows Server 2003 also holds the distinction of being the first Microsoft 64-bit server OS. Windows XP Professional has a 64-bit version, but the demand for 64-bit Itanium-based workstations hasn't been that great so far. With a 64-bit server OS finally available, companies with huge databases will probably start the transition to 64-bit Windows. A 64-bit version of SQL Server™ is slated to ship alongside the 64-bit versions of Windows Server 2003.

Windows Server 2003 will ship in six configurations initially. For the Intel x86 CPU, the low end is the Web Edition server for basic Web Services. Next up the ladder is the Standard Edition, intended as a departmental server. Up a notch from that is the Enterprise Edition, targeted at medium to large enterprises. Finally, for massive database systems operating on up to 32 processors there's the Datacenter Edition. The remaining two configurations are the 64-bit Itanium versions of both Enterprise Edition and Datacenter Edition.

As an interesting aside, the 64-bit versions of Windows Server 2003 won't include the .NET Framework. Apparently, the 64-bit version of .NET isn't yet ready to ship, so it will be included in a later version.

There are lots of new features in Windows Server 2003 that I won't go into any detail on, but are still worth mentioning, just for their coolness. For example, Internet Information Services (IIS) is now at version 6.0. It has had a major architectural change and big performance gains are attained by doing more work with a kernel-mode listener. If desired, you can run it in the three IIS 5.0 protection levels as well.

Other exotic new features in Windows Server 2003 include, but are not limited to, Volume Shadow Copy, Hot Add Memory, and Non-Uniform Memory Access (NUMA) support.

The Volume Shadow Copy service provides a way to do complete backups, even on files that are open at the time of the backup. While backups aren't my favorite topic, I think it's pretty cool how this functionality was implemented transparently using drivers. Volume Shadow Copy, as well as many other additions, are covered in the article "Windows XP: Kernel Improvements Create a More Robust, Powerful, and Scalable OS," by Mark Russinovich and David Solomon in the December 2001 issue.

Hot Add Memory is one of those totally exotic features that make you wonder what those wacky hardware guys will dream up next. It's the ability to add RAM to a system while it's running. The OS automatically detects and starts using the RAM when you plug it in. For this to work, however, you need to be running a system designed to support it. In the Microsoft documentation, you'll find the Surgeon General's warning to not just blindly add RAM to any old system while it's turned on—bad things will happen.

NUMA is a high-end technology used in enterprise-class multiprocessor systems. The idea is that memory and processors are grouped together into cells. A processor accessing memory within its cell can access that memory faster than memory in another cell. The NUMA support in Windows causes the scheduler to attempt to keep related processes running in the same cell.

Before jumping into the blood and guts of new APIs in Windows Server 2003, there's one obligatory disclaimer I need to add. What follows is subjective and not comprehensive. I went through hundreds of header files and beta documentation pages and culled what I thought was most interesting. I had to make a lot of tough choices as to what to include and what to leave out. Generally, for me to select a feature to include here, it had to be a user-mode API that's not completely esoteric or tied to one particular program. There's lots of new functionality at the device driver level, but that's outside the scope of this article.

New KERNEL APIs

I'll begin the tour with kernel-level functionality. You might think that this means KERNEL32 (I'll get to that soon enough). True system programming fanatics know that NTDLL is the real guts of the user-mode kernel. I've compared the exports from the Windows Server 2003 version of NTDLL.DLL to the Windows 2000 version. As you might expect, lots of APIs were added and a few have disappeared. Big deal, they're all undocumented, right? Not so fast!

As I was comparing all those .H files between different versions of the Platform SDK, I came across a very interesting file in the October 2002 version. It's named WINTERNL.H. This is definitely a file to hold on to. It could disappear in future Platform SDKs. At the top of WINTERNL.H is a big three-paragraph warning about how all the data structures and APIs are subject to change and intended for use only by Windows core components. (Not like we haven't seen this before, eh?)

Anyhow, so what goodies are in WINTERNL.H? Unfortunately, not as many as you'd like. But there are tantalizing hints of well-known, yet still undocumented data structures. For example, you'll see both the Process Environment Block (PEB) and Thread Environment Block (TEB) defined. However, most of the fields are listed as reserved. Likewise, the structures and prototypes necessary to call NtQueryInformationProcess and NtQueryInformationThread are there. However, there are a variety of books and Web sites that provide many more details about these APIs and structures than WINTERNL.H does. Admittedly, nothing in this file is new in Windows Server 2003. However, the SDK that accompanies Windows Server 2003 is where this file first appeared.

Why am I making a big fuss about this? It's significant because Microsoft is finally admitting that there are lots of interesting things that aren't documented. A few other APIs in WINTERNL.H are worth noting: NtCreateFile, NtOpenFile, NtClose, and NtWaitForSingleObject. These APIs are the guts of the user mode implementation of common KERNEL32 APIs. Likewise, RtlUnwind is a key API used in structured exception handling (SEH), as described in my January 1997 article in Microsoft Systems Journal "A Crash Course on the Depths of Win32® Structured Exception Handling". It's highly unlikely that RtlUnwind will change. If it did, a very large percentage of existing applications would fall flat on their faces.

Processes, Threads, and Fibers, Oh My!

Moving on to KERNEL32 land, my first group of APIs, shown in Figure 1, relate to processes and threads. With one exception, they all retrieve some piece of information about a process or thread. GetThreadId takes a thread handle and returns the associated thread ID. It's a similar story for GetProcessId. It's hard to believe these functions weren't in the Win32 API 10 years ago! IsWow64Process tells you if the calling process is a 32-bit process running under 64-bit Windows.

Figure 1 Process and Thread APIs (Close Figure 1)

GetProcessHandleCount
IsProcessInJob
IsWow64Process
CreateJobSet
GetProcessIdOfThread
GetThreadId
GetProcessId
GetThreadIOPendingFlag
GetCurrentProcessorNumber
SetProcessWorkingSetSizeEx
GetProcessWorkingSetSizeEx
RestoreLastError

GetProcessHandleCount returns the number of handles that the specified process has open. It's the same value seen in the Performance Monitor data or in the Task Manager. RestoreLastError is an enigma. It's code is identical to SetLastError. It's unclear to me why it was made into a separate API.

To demonstrate some of these new APIs, check out the ProcessesAndThreads program in Figure 2. The code is self-explanatory, so I won't delve into it here. To build it (and the other sample programs), you will need at least the October 2002 Platform SDK, with its Include and Lib directories at the head of the compiler and linker's directory search order. If you're running Windows XP, you can build it to use just the Windows XP subset of the APIs. Simply uncomment the #define W2K3SERVER line near the top and rebuild.

Figure 2 ProcessesandThreads.cpp (Close Figure 2)
//-------------------------------------------------------------------
// Matt Pietrek
// MSDN Magazine, 2003
// Program: ProcessesAndThreads
// Purpose: A demonstration of the Windows XP/Windows Server 2003 Process/
// Thread APIs
//-------------------------------------------------------------------
#include "stdafx.h"

// comment out to get just the XP APIs
#define W2K3SERVER  1

int main(int argc, char* argv[])
{
    DWORD dwThreadID = GetCurrentThreadId();
    DWORD dwProcessID = GetCurrentProcessId();
    printf( "ProcessId: %X  ThreadId: %X\n", dwProcessID, dwThreadID );

    HANDLE hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, dwProcessID );
    if ( !hProcess )
    {
        printf( "unable to open process\n" );
        return 0;
    }

    // Show OpenThread()
    HANDLE hThread = OpenThread( THREAD_ALL_ACCESS, FALSE, dwThreadID );
    printf( "hProcess: %IX  hThread: %IX\n", hProcess, hThread );

    // Show GetProcessHandleCount()
    DWORD dwHandleCount;
    GetProcessHandleCount( hProcess, &dwHandleCount );
    printf( "Handle Count: %u\n", dwHandleCount );

    // Show IsProcessInJob()
    BOOL bIsInJob;
    IsProcessInJob( hProcess, NULL, &bIsInJob );
    printf( "IsProcessInJob: %s\n", bIsInJob ? "true" : "false" );

    // Show GetProcessId()
    printf( "Process ID from hProcess: %X\n", GetProcessId( hProcess ) );

    // Show IsWow64Process
    BOOL bIsWow64Process;
    IsWow64Process( hProcess, &bIsWow64Process );
    printf( "IsWow64Process: %s\n", bIsWow64Process ? "true" : "false" );

    BOOL bIOIsPending;
    GetThreadIOPendingFlag( hThread, &bIOIsPending );
    printf( "IoIsPending: %s\n", bIOIsPending ? "true" : "false" );

#ifdef W2K3SERVER
    // Show GetProcessWorkingSetSizeEx()
    DWORD wsFlags;
    SIZE_T wsSizeMin, wsSizeMax;
    GetProcessWorkingSetSizeEx( hProcess, &wsSizeMin, &wsSizeMax,
                                &wsFlags );
    printf( "Working Set: min:%IX  max:%IX  flags:%X\n",
            wsSizeMin, wsSizeMax, wsFlags );

    // Show GetProcessIdOfThread()
    DWORD dwProcessIdFromThread = GetProcessIdOfThread( hThread );
    printf("Process ID obtained from thread ID: %X\n",
           dwProcessIdFromThread);

    // Show GetThreadId();
    printf("Thread ID obtained from thread handle:
           %X\n",GetThreadId(hThread));

#endif

    return 0;
}

In addition to threading support, Windows Server 2003 adds some new Fiber APIs, shown in Figure 3. The big news here is the addition of fiber local storage (FLS). The APIs are used identically to their thread local storage (TLS) counterparts. A slot is allocated via the FlsAlloc function. To set or retrieve values, use the FlsGetValue and FlsSetValue function. When you are done with the slot, you call FlsFree.

Figure 3 Fiber APIs (Close Figure 3)

FlsAlloc
FlsGetValue
FlsSetValue
FlsFree
ConvertFiberToThread
ConvertThreadToFiberEx
CreateFiberEx

Just for grins, I looked into the implementation of the FLS functions. At offset 0xFB4 in the Thread Environment Block is a pointer to a data structure. Eight bytes into this structure is an array of 128 slots. These slots are conceptually the same as the TLS slots. As the thread switches from fiber to fiber, the pointer at offset 0xFB4 is updated accordingly.

The ConvertFiberToThread API undoes the effect of ConvertThreadToFiber. Once called, no more fiber functions can be called on the thread. The remaining two APIs listed in Figure 3 are just "Ex"-tended versions of existing APIs. CreateFiberEx is just like CreateFiber, but with the ability to specify the stack reserve size. ConvertThreadToFiberEx is interesting, though. In the original fiber implementation, the floating point, MMX, and SSE registers weren't saved and restored across fiber switches to improve performance. The new API lets you specify that these registers need to be saved and restored as well.

Vectored Exception Handling

Perhaps the most exciting addition to KERNEL32 is vectored exception handling (VEH), which provides additional flexibility in how to process exceptions. I covered vectored exception handling in depth in my September 2001 Under The Hood column in MSDN Magazine, so I'll just give a brief synopsis here along with an illustration of the process in Figure 4.

Figure 4 Vectored Exception Handling
Figure 4 Vectored Exception Handling

Traditional structured exception handling (SEH) with its __try/__except mechanisms is inherently thread specific. Exceptions can only be handled by the thread that set up a handler. (The compiler and OS handle all the messy details of this and expose the relatively simple __try/__except syntax to you.) More importantly, with SEH you might set up a handler, only to have the exception grabbed first by another handler that doesn't know how to deal with the exception properly.

Vectored exception handling works more like a traditional notification callback scheme. To handle exceptions, call the AddVectoredExceptionHandler API, passing it the address of your exception callback function. When an exception occurs, the callback function receives a pointer to an EXCEPTION_POINTERS structure. This is the same structure that SEH callbacks can receive via the GetExceptionInformation API. From fields in the EXCEPTION_POINTERS structure, you can learn the exception code (for instance, 0xC0000005) and the register values (via the included CONTEXT structure).

The VEH callback chooses to either handle the exception or chain it onto the next handler in the list. It determines what happens by returning the appropriate value from the callback. Each process has a linked list of VEH callbacks. As part of processing an exception, the OS walks the VEH list and calls the handlers. To remove a handle from the list, use the RemoveVectoredExceptionHandler API.

How does vectored exception handling coexist with SEH? Good question! Immediately before walking the SEH chain, the system walks the vectored exception handler list. Put another way, VEH handlers have priority over SEH handlers. To see vectored exception handling in action, check out the VEHDemo program in Figure 5. VEHDemo installs a couple of vectored exception handlers and uses a structured exception handler to show how VEH and SEH work together. The resulting output from running the VEHDemo program is shown in Figure 6.

Figure 6 VEHDemo Output (Close Figure 6)
In VectoredExceptionHandler, EIP =00411BBA
In VectoredExceptionHandler, EIP =00411BBA
In VectoredExceptionHandler, EIP =00411BBA
In SecondVectoredExceptionHandler - handling it
In VectoredExceptionHandler, EIP =00411BBB
In VectoredExceptionHandler, EIP =00411BBB
In VectoredExceptionHandler, EIP =00411BBB
In SecondVectoredExceptionHandler - NOT handling it
Caught the exception in Function1
Figure 5 VEHDemo.cpp (Close Figure 5)
//-------------------------------------------------------------------
// Matt Pietrek
// MSDN Magazine, 2003
// Program: VEHDemo
// Purpose: A demonstration of Vectored Exception Handling
//-------------------------------------------------------------------

#define WIN32_LEAN_AND_MEAN
#define _WIN32_WINNT 0x0501
#include <stdio.h>
#include <tchar.h>
#include <windows.h>

LONG
WINAPI VectoredExceptionHandler(PEXCEPTION_POINTERS ExceptionInfo)
{
    // Stupid handler that does nothing except print out the faulting EIP
    printf( "In VectoredExceptionHandler, EIP =%p\n",
            (PVOID)(DWORD_PTR)ExceptionInfo->ContextRecord->Eip );

    // Don't handle the exception here.  Let normal processing continue
    return EXCEPTION_CONTINUE_SEARCH;
}

LONG
WINAPI SecondVectoredExceptionHandler(PEXCEPTION_POINTERS ExceptionInfo)
{
    // If the faulting EIP points to a HLT instruction, just skip past
    // it, to show us "Handling" an exception
    if ( *(PBYTE)ExceptionInfo->ContextRecord->Eip == 0xF4 )
    {
        printf( "In SecondVectoredExceptionHandler - handling it\n" );
        ExceptionInfo->ContextRecord->Eip++;
        return EXCEPTION_CONTINUE_EXECUTION;
    }
    else    // For everything else, we won't handle the exception here
    {
        printf("In SecondVectoredExceptionHandler - NOT handling it\n");
        return EXCEPTION_CONTINUE_SEARCH;
    }
}

int _tmain(int argc, _TCHAR* argv[])
{
    // Add a vectored exception handler.  In fact, add the same
    // handler 3 times, to prove that they handlers are stored
    // in a linked list.  This handler always returns "keep looking"
    AddVectoredExceptionHandler( 1, VectoredExceptionHandler );
    AddVectoredExceptionHandler( 1, VectoredExceptionHandler );
    AddVectoredExceptionHandler( 1, VectoredExceptionHandler );

    // And a second vectored handler that knows how to handle a HLT
    // caused exception, but nothing else
    AddVectoredExceptionHandler( 0, SecondVectoredExceptionHandler );

    __try
    {
        __asm   HLT         // Cause a privileged instruction fault
        *(int *)0 = 0;      // Access Violate by dereferencing NULL ptr
    }
    __except( EXCEPTION_EXECUTE_HANDLER )
    {   // This handler will catch the *(int *)0 = 0 fault
        printf( "Caught the exception in Function1\n" );
    }

    return 0;
}

Directories and Files

A few new APIs have been added in the File and Directory category, shown in Figure 7. SetDllDirectory adds an arbitrary set of directories that will be searched when the system looks for a DLL. The system searches the specified path(s) after the application load directory but before anywhere else, such as the system directory. The documentation for SetDllDirectory describes the exact search order and makes for interesting reading. GetDllDirectory returns whatever value has been previously set by calling SetDllDirectory. GetSystemWow64Directory is used for finding the 32-bit system directory when on a 64-bit system.

Figure 7 Directory and File APIs (Close Figure 7)

GetDllDirectory / SetDllDirectory
NeedCurrentDirectoryForExePath
GetSystemWow64Directory
SetFileShortName
CheckNameLegalDOS8Dot3
SetFileValidData
GetVolumePathNamesForVolumeName
FindFirstStream / FindNextStream
ReOpenFile

A little-known fact about the NTFS file system is that it supports multiple streams in a file. This is tricky to explain in a limited space, but the gist is that multiple files can be collectively referred to by a single file name. Most files only have a single default stream associated with them, and that's what most Win32 APIs report about. To create a stream other than the default stream, simply append a colon (:), followed by the stream name. For instance, you could use Notepad to create a file/stream called abc.txt:MyStream. In the Windows Explorer window, you'll see a zero-byte abc.txt file. However, the abc.txt:MyStream exists. The normal Win32 APIs simply don't report any information about it.

In Windows Server 2003, this situation is remedied somewhat. The FindFirstStream and FindNextStream APIs enumerate through all the streams in a file. To demonstrate their use, I wrote the FindFirstStream program shown in Figure 8. To use it, simply pass it a file name. If there are any streams beyond the default unnamed stream, it lists them all. Here is the output for a file, a.txt, with three streams—abc, def, and ghi:

a.txt:abc:$DATA
a.txt:def:$DATA
a.txt:ghi:$DATA

Figure 8 FindFirstStream.cpp (Close Figure 8)
//-------------------------------------------------------------------
// Matt Pietrek
// MSDN Magazine, 2003
// Program: FindFirstStream
// Purpose: A demonstration of the Windows XP/Windows Server 2003 File
// Stream APIs
//-------------------------------------------------------------------
#include "stdafx.h"

int main(int argc, char * argv[])
{
    if ( argc != 2 )
        return 0;

    WIN32_FIND_STREAM_DATA streamData;

    wchar_t wszFileName[512];
    mbstowcs( wszFileName, argv[1], 0xFFFFFFFF );
    HANDLE h = FindFirstStreamW( wszFileName, FindStreamInfoStandard,
                                &streamData, 0 );
    if ( h == (HANDLE)ERROR_INVALID_HANDLE )
    {
        // printf( "unable to being stream enum for file %s\n", argv[1] );
        return 0;
    }
    while ( 1 )
    {
        if ( !FindNextStreamW( h, &streamData ) )
            break;
        printf( "%ls%ls\n", wszFileName, streamData.cStreamName );
    }

    // Do we need to close the stream handle???

    return 0;
}

The ReOpenFile API is used for taking an existing file handle and getting another handle that has a different set of access rights. A typical use is where you have code that has only a file handle and doesn't know the associated file name. If your code needs access rights or a sharing mode different than the existing handle has, ReOpenFile provides a way to attempt to obtain those rights. Of course, ReOpenFile ensures that the newly requested access rights and sharing mode are legal. It also lets you guard against pipe impersonation attacks.

CheckNameLegalDOS8Dot3 has one of those fluky API names. This API is useful for checking if a file name can be used on a file allocation table (FAT) file system volume.

You may recall that with the advent of long file names (beyond 8.3), the OS needed a way to refer to the file using standard 8.3 conventions. The system has an algorithm for mapping between the long and short versions of the name. These files are easily picked out because the short version ends with a tilde (~), followed by a number (for instance, "foobar~1.txt"). The new SetFileShortName API allows you to override the system's default short file name. To use it however, the target file needs to be on an NTFS volume.

Memory and System Information

In the memory allocation arena, Windows Server 2003 and Window XP have a feature called the low-fragmentation heap. This heap algorithm avoids fragmentation by allocating all blocks from 128 predetermined different block-size ranges, called buckets. When an application needs to allocate memory from the heap, the heap chooses the bucket that's able to hold the requested block with the least wasted space. The system uses the traditional heap for blocks greater than 16KB. To use the low-fragmentation heap, call HeapSetInformation, passing it the appropriate heap handle and flag value. All heaps default to "normal" Win32 heap behavior until HeapSetInformation is called. To determine which behavior a heap is using, call the HeapQueryInformation API.

In the area of system information, there's a motley collection of new interfaces, as you can see in Figure 9. GetSystemRegistryQuota provides you with the current size of the registry, as well as the maximum allowed size. GetSystemTimes returns the length of time that all processors have spent in the idle state, in kernel mode, and in user mode.

Figure 9 System Information APIs (Close Figure 9)

GetLargePageMinimum
GetSystemRegistryQuota
GetSystemTimes
GetNativeSystemInfo
TzSpecificLocalTimeToSystemTime
SetFirmwareEnvironmentVariable
GetFirmwareEnvironmentVariable
CreateMemoryResourceNotification
QueryMemoryResourceNotification
GetLogicalProcessorInformation

The GetNativeSystemInfo is intended for 32-bit programs running under 64-bit Windows. It returns a SYSTEM_INFO structure filled in as if it were called from a native 64-bit program. For instance, running an x86 program on an Itanium machine, the SYSTEM_INFO.dwPageSize would be 8192 bytes, rather than the 4096 bytes that you'd get from calling GetSystemInfo. The SystemInfo program in Figure 10 shows GetSystemInfo being used, as well as a few of the other new system information APIs.

Figure 10 SystemInfo.cpp (Close Figure 10)
//-------------------------------------------------------------------
// Matt Pietrek
// MSDN Magazine, 2003
// Program: SystemInfo
// Purpose: A demo of the Windows XP/Windows Server 2003 System Info APIs
//-------------------------------------------------------------------
#include "stdafx.h"

// Uncomment to get Windows Server 2003 APIs as well
// #define W2K3SERVER   1

int main(int argc, char * argv[])
{
    // Show GetSystemRegistryQuota();
    DWORD dwQuotaAllowed, dwQuotaUsed;
    GetSystemRegistryQuota( &dwQuotaAllowed, &dwQuotaUsed );
    printf( "Quota allowed: %uK, Quota used: %uK\n",
            dwQuotaAllowed/1024, dwQuotaUsed / 1024 );

    // Show GetSystemTimes()
    FILETIME IdleTime;
    FILETIME KernelTime;
    FILETIME UserTime;
    GetSystemTimes( &IdleTime, &KernelTime, &UserTime );

    const DWORD filetimeDivisor = 1000000000 / 100;

    printf( "IdleTime:   %I64u seconds\n",
        *(PDWORD64)&IdleTime / filetimeDivisor );

    printf( "KernelTime: %I64u seconds\n",
        *(PDWORD64)&KernelTime / filetimeDivisor );

    printf( "UserTime:   %I64u seconds\n",
        *(PDWORD64)&UserTime / filetimeDivisor );

    // Show GetNativeSystemInfo().  If you're running in a WOW session
    // (e.g., an X86 app running on IA64), it'll return the capabilities
    // of the actual OS, not the WOW info.  If not running in WOW, you'll
    // get the same info as GetSystemInfo
    SYSTEM_INFO sysInfo;
    GetNativeSystemInfo( &sysInfo );
    printf( "Native System Info - Processor type: %u  page size: %X\n",
            sysInfo.dwProcessorType, sysInfo.dwPageSize );

#ifdef W2K3SERVER
    printf( "Large Page Minimum size: %IX\n", GetLargePageMinimum() );
#endif
    return 0;
}

GetLogicalProcessorInformation returns information that is pertinent both to NUMA systems and to Intel's Hyperthreaded CPUs (wherein a single CPU has multiple execution units). The API returns an array of SYSTEM_LOGICAL_PROCESSOR_INFORMATION structures.

The CreateMemoryResourceNotification is a way for applications to receive a notification when the available physical memory is running low without having to constantly poll the value. The API creates a handle that can be passed to the WaitForXxx family of functions. The handle is signaled when free memory drops below a threshold. According to the documentation, the threshold is 32MB per 4GB of memory on the system. You can also check the memory status directly using the QueryMemoryResourceNotification. The system can also notify you when available free physical memory is high, but I doubt this will be a heavily used capability.

Debug APIs

In the realm of debugging, there's a small set of new APIs. The most exciting has to be DebugSetProcessKillOnExit. Up until now, if you were acting as a debugger for another process, there was no way to stop acting as a debugger. You simply couldn't detach from a process being debugged. When you act as a debugger for another process, one of your threads is the debug thread, and processes all the debug notification messages. Normally, if this thread terminates, the debuggee process terminates as well. The DebugSetProcessKillOnExit API changes this behavior. By passing FALSE, you tell the system to stop requiring the debug thread to process messages for the debuggee.

In a related vein, you'll find DebugActiveProcessStop, which tells the system to detach the specified process from whatever process is debugging it. It can only be called by the thread that called DebugActiveProcess or CreateProcess. Since a debugger thread can conceivably debug multiple processes at the same time, DebugActiveProcessStop needs the parameter that indicates which process to detach from.

The DebugBreakProcess is just like DebugBreak, except that it applies to the specified process rather than to the current thread. The API works by creating a thread in the target process, much like CreateRemoteThread does. The newly created thread invokes a breakpoint instruction, which causes the normal SEH mechanism to take over. For developers, this usually means that the Just-In-Time debugging dialog comes up.

The final new debugging API is CheckRemoteDebuggerPresent. It's similar to the IsDebuggerPresent API in that it tells you if a process is running under the control of a debugger process. IsDebuggerPresent tells you if your process is being debugged, while CheckRemoteDebuggerPresent lets you query about any process for which you have a handle.

Side-by-side Execution

Much has been made of the side-by-side installation and execution feature in the .NET Framework. However, these same capabilities are built into Windows Server 2003 and Windows XP. The key to these capabilities are the new activation context (ActCtx) APIs, shown in Figure 11.

Figure 11 ActCtx APIs (Close Figure 11)

CreateActCtx
AddRefActCtx
ReleaseActCtx
ActivateActCtx
DeactivateActCtx
GetCurrentActCtx
QueryActCtx
ZombifyActCtx
FindActCtxSectionString
FindActCtxSectionGuid

An activation context is a set of system-managed data structures that contain information used to cause an application to use a particular DLL version or COM object instance, based upon a manifest file. Manifest files use the XML format (no surprise there!) and look very similar to .NET manifests. The use of activation contexts could easily be an article on its own, so I'll defer to the SDK documentation. However, it's interesting to note that certain system DLLs are enabled for side-by-side execution, with their corresponding .H files now using the activation context APIs. For a prime example, examine a recent version of COMMCTRL.H. There are a ton of inline functions with names like IsolationAwareImageList_Add. These inline functions show the activation context APIs in action. You'll also see some clever tricks with C++ macros to keep existing code compiling without any changes.

There's one final Kernel32 API that doesn't fit nicely into any other category. GetModuleHandleEx should have been included in the Win32 API years ago. The key capability it adds is finding an HMODULE when given an address within that module. If you've ever written debugging or diagnostic code, you've probably been in the situation where you had a code address and needed to determine which DLL it came from. It was possible to accomplish this in an awkward way using VirtualQuery, but GetModuleHandleEx is much more elegant.

Unlike its predecessor API (GetModuleHandle), GetModuleHandleEx affects the module's reference count, unless you explicitly tell it not to. Depending on the specified flags, it can increment the reference count, leave it unchanged, or pin the DLL in memory for the lifetime of the process. The GET_MODULE_HANDLE_EX_FLAG_PIN flag addresses one of my nagging concerns. Let's say you called GetModuleHandle to retrieve a module handle. In a multithreaded program, it's possible that another thread could unload the DLL and load another DLL at the exact same address. All this can happen between the time the first thread gets the HMODULE and when it starts using it. By pinning the module in memory, you're guaranteeing that the HMODULE you get will be valid when you use it later.

User Interface Additions

On the user interface side of things, the biggest new addition to USER32 is the raw input APIs. They offer an alternative to the keyboard and mouse as the only way to receive input from users. Using the raw input APIs, devices such as joysticks, microphones, or touch screens are on equal footing with the keyboard and mouse.

In the standard Windows input model, the keyboard and mouse drivers create low-level scan codes and motion events. The system takes these low-level events and translates them into higher-level messages, such as WM_CHAR or WM_APPCOMMAND. While this is great for keeping input capture simple, it's not very adaptable to other input devices.

The new APIs for raw input are shown in Figure 12. By default, applications don't receive raw input. Instead, you must register to receive the input through the RegisterRawInputDevices API, which takes an array of all the devices that you're interested in.

Figure 12 USER32 APIs (Close Figure 12)

Raw Input APIs
GetRawInputData
GetRawInputDeviceInfo
GetRawInputBuffer
RegisterRawInputDevices
GetRegisteredRawInputDevices
GetRawInputDeviceList
DefRawInputProc
Other New APIs
PrintWindow
IsGUIThread
GetWindowRgnBox
BroadcastSystemMessageEx
DisableProcessWindowsGhosting

When a device has input, the system posts a WM_INPUT message to the program's message queue. The program reads the input in either an unbuffered mode (one message at a time) or in a buffered mode (where multiple messages are read at once). As you'd expect, there are APIs for enumerating all the raw input devices and querying for information about them.

There are also a handful of other new APIs in USER32 that I found interesting (see Figure 12). The PrintWindow API copies the contents of the specified HWND into the specified device context (DC). IsGUIThread returns (and optionally sets) a value that tells whether the calling thread is a GUI thread, meaning that the thread has called into Win32K.SYS and has a larger kernel-mode stack. BroadcastSystemMessageEx is like BroadcastSystemMessage, except that it returns more information about the window that had denied a request.

Although it's available as a redistributable for earlier operating systems, the Text Services Framework (TSF) makes its debut with Windows Server 2003 and Windows XP. The Text Services Framework is an extensible system for reading and writing text in a manner that's independent of the input/output device. The most readily accessible example of the TSF is as a way of allowing applications to receive text input from devices such as pens or microphones.

Each different type of text input/output device becomes a "text service." Between the text services and the applications is the TSF manager. To draw an analogy to databases, each text service is like an ODBC driver, and the TSF manager plays the role of the ODBC manager. The TSF is composed of a few APIs and many dozens of interfaces. An entire article—if not a book—could be devoted to them, so I won't attempt to describe them further here.

On a more graphical slant, Microsoft has put a friendly object-oriented face on the GDI. Sure, application frameworks like MFC have done this for years, but the new GDI+ API is part of the core operating system and doesn't require you to pull in all the baggage of an application framework. In broad strokes, the GDI+ functionality falls into the following four categories: two-dimensional vector graphics (lines and curves), imaging (bitmaps), typography (display of text), and matrix transformations.

Although the GDI+ APIs are technically just like any other Win32 APIs, it's unlikely that you'd call them directly (at least from C++ code). Instead, the Platform SDK has a set of header files (with names like GDIPlus.H) that define approximately 40 C++ classes. Typical of these classes are Bitmap, Font, and Region. The methods of these classes are usually inline functions that call the underlying API in GDIPlus.DLL.

Here's a typical use of the GDIPlus classes in C++ code:

VOID OnPaint(HDC hdc)
{
   Graphics graphics(hdc);
   Pen      pen(Color(255, 0, 0, 255));

   graphics.DrawLine(&pen, 0, 0, 200, 100);
}
Note that there's no BeginPaint/EndPaint tedium. Everything is very object-oriented. The Graphics object is the GDI+ equivalent of the device context handle (HDC). The Pen object takes care of the creation and destruction of the underling GDI pen for you.

The best way to get a feel for GDI+ is to browse through GDIPlus.H and the files it references. To use GDI+, you'll need to #include GDIPlus.H in your source file and add the GDIPlus.lib file to your linker line. As a side note, GDIPlus.DLL is side-by-side enabled. Thus, you won't find it in the \windows\system32 directory. Instead, you'll find the various versions of it in subdirectories under \windows\winsxs\.

Newly Documented Interfaces

Microsoft has recently documented several hundred APIs and COM interfaces that were in Windows 2000 or earlier. These APIs can be found at Settlement Program Interfaces. Although not technically new, there's documentation for them, so you can now safely call them in Windows XP and later. To be honest, when I looked at the APIs, I thought that with a few exceptions, there was really nothing scandalous or sizzling. Looking at them critically, you'll see that they're mostly rounding out existing API sets. In addition, some of the APIs in the list are already documented, but new details were included.

By far, the biggest subset of APIs fall within the SHELL32 library. There's approximately 110 APIs here, but judging from the names, many are of limited use. CDefFolderMenu_Create2 anybody? On a more promising note, there are more than 20 new COM interfaces, including IMenuBand, IShellItem, and IShellTaskScheduler.

The newly documented APIs also have a fair selection of new cryptography-related functions. WININET.DLL has five new APIs, mostly having to do with proxy support. The DirectShow functionality has a dozen or so new APIs added to it. Finally, some of the APIs such as NtQuerySystemTime and RtlUnwind are described in the WINTERNL.H file mentioned earlier.

New DEBUGHLP Additions

Long-time readers of my articles and columns for MSDN Magazine know that DBGHELP.DLL is one of my favorite DLLs. It's undergone quite substantial changes since its Windows 2000 incarnation. It does so many new things that it's really quite hard to know where to begin.

One of the coolest things in DbgHelp isn't an API that you call. Have you ever been frustrated when your debug symbols get out of sync with the DLLs on your system? Thanks to DBGHELP.DLL, this is (mostly) a thing of the past. The new feature is called the symbol server. When you ask DbgHelp to load the symbols for a module, it calls a symbol server DLL to locate the debug files if it can't find the debug file locally. The symbol server DLL is free to locate the debug files in whatever manner it sees fit. Conceptually, anybody can write a symbol server DLL, and DbgHelp will use it.

In reality, Microsoft has created the one symbol server DLL (SymSrv.DLL) that just about everybody will use. In addition, they've put debugging symbols for just about every relevant version of Windows on a publicly available Web server. The end result is that debuggers and tools can dynamically obtain debug files with no extra effort on their part. All that's necessary is to use DbgHelp.DLL to access the symbols. SymSrv.DLL is part of the Debugging Tools for Windows, which can be downloaded from the MSDN site (see Microsoft Debugging Tools).

SymSrv.DLL automatically downloads the appropriate PDB file the first time it's needed and stores it locally. It ensures that the PDB file it downloads is the correct version for the DLL. When storing the PDB files locally, it uses a directory naming scheme that allows the PDBs for multiple versions of the DLL to coexist.

The symbol server functionality is available to users of Visual Studio® .NET. All that's required is to put SymSrv.DLL in the same directory as the IDE (DevEnv.exe) and set an environment variable. DBGHELP.DLL by default uses the path in the _NT_SYMBOL_PATH to locate symbols. To indicate that the symbol server should be used, _NT_SYMBOL_PATH should be something like the following:

symsrv*symsrv.dll*c:\winnt\symbols*
  http://msdl.microsoft.com/download/symbols

Obviously, you'll want the path portion ("c:\winnt\symbols" in this example) to point at a valid directory on your hard drive. Assuming you have everything set up properly, this should work well. I've done this successfully on a number of machines, but unfortunately I can't provide support if you have problems.

The next big thing in DbgHelp is type support. My March 2002 Under The Hood column covers this in much more detail, so I'll just mention it in passing here. The type support extends beyond the primitive types to include user-defined types. Using the new SymFromAddr and SymFromName APIs, you can obtain a type index. This type index is then passed to SymGetTypeInfo to obtain information about the type. SymGetTypeInfo is a fairly difficult API to understand, so I again direct you to the aforementioned column on this subject. All the user-defined types in a given symbol table can be enumerated with the SymEnumTypes.

If you're a longtime DbgHelp user, you may notice that many of the new APIs run parallel to existing interfaces. For example, SymEnumSymbols would seem to do the same thing as SymEnumerateSymbols. The reason for the new APIs is that the old APIs didn't provide very complete information about the symbols. The newer APIs always use a SYMBOL_INFO structure, which has much more complete information about a symbol.

Another exciting addition to DbgHelp is an awareness of local variables and parameters. In the past, you could enumerate symbols for a module, but only global symbols. With the new SymEnumSymbols API, you can enumerate locals and parameters. The not-so-obvious trick to doing this is to call the SymSetContext function beforehand. The call to SymSetContext is where you specify an address within the particular function you're interested in. Under the hood, DbgHelp locates the enclosing function's local variables and parameters and enumerates only them.

The MiniDumpWriteDump API is exciting. With a single call, you can create your own dump files. These are the very same files that you'd get from a Dr. Watson crash or from the UserDump tool. These dump files can be loaded into WinDBG or Visual Studio .NET for post-mortem debugging. A typical reason to create a dump file would be if your program encounters some unexpected condition in the field. The user can send you back the file for perusal in the debugger of your choice.

There are many more new DbgHelp APIs, but I'll mention just a few more in this section. You're now able to enumerate source file lines via the SymEnumLines API. SymAddSymbol and SymDeleteSymbol let you dynamically extend the symbols defined in the symbol file. The SymFromToken API returns a SYMBOL_INFO, given a .NET metadata method token. This is an important first step for DBGHELP in supporting .NET Framework-based debug information.

With all these new features, it would be a shame not to show off some of them. The DBGHELP51 program (included with this month's download at the link at the top of this article) uses some of the new APIs just discussed, including SymEnumSymbols, SymEnumLines, and SymGetTypeInfo. To test it, run DBGHELP51.EXE, passing it the name of an EXE file that has debug information for it. If the debug information loads correctly, DBGHELP51.EXE first lists all the global symbols. If type information is available, the type name follows the symbol name. The second part of the output is any source file and line number information, if found.

Windows Error Reporting

A new DLL with only two exported APIs? That's what we have with the Windows error reporting API. As you've probably noticed over the years, Microsoft has made significant efforts to make application faults less scary to the typical user. For example, starting in Windows XP, when a program encounters an unhandled exception, a dialog pops up asking if you want to send a report to Microsoft. The new error reporting API lets applications integrate better with the system in this regard.

The first API is ReportFault, which is used in applications that catch their own exceptions with try blocks. It lets them tie into the system's fault reporting mechanism. ReportFault takes an EXCEPTION_POINTERS structure as a parameter. Calling ReportFault should cause the same system actions that would occur if you hadn't caught the exception. The return value from ReportFault is a code indicating what the system did. For example, one return code, frrvLaunchDebugger, indicates that a debugger was launched to attach to the program.

The second API is AddERExcludedApplication, which prevents the system from reporting faults on the specified executable. For example, if you called the API with the string foo.exe, any program called foo.exe that had an unhandled exception wouldn't be reported as such. The parameter to AddERExcludedApplication should be just a simple EXE name, without any path information.

ADVAPI32

Given the emphasis on security at Microsoft, it's not surprising that there's a new set of interfaces known as the credential APIs. They obtain and manage information such as user names and passwords. They can request Windows XP account information to be used in place of the credentials established while logging on. Such requests typically occur when the logon credentials do not have permissions required by the application.

Figure 13 shows the credential APIs, which come from the new WINCRED.H file. To show some basic functionality, I wrote the Credential program which enumerates all the current credentials and displays basic information for each one (see Figure 14). Try running it on your system. What you see might surprise you.

Figure 14 Credential (Close Figure 14)
//-------------------------------------------------------------------
// Matt Pietrek
// MSDN Magazine, 2003
// Program: Credential
// Purpose: A demonstration of the Windows XP/ Windows Server 2003
// Credential APIs
//-------------------------------------------------------------------
#include "stdafx.h"

int _tmain(int argc, _TCHAR* argv[])
{
    PCREDENTIAL * pCreds;
    DWORD cCreds;

    CredEnumerate( NULL, 0, &cCreds, &pCreds );

    for ( DWORD i = 0; i < cCreds; i++ )
    {
        PCREDENTIAL p = pCreds[i];
        printf( "Flags:%X  Type:%X  Target: %s  User: %s\n",
            p->Flags, p->Type, p->TargetName, p->UserName );
    }

    CredFree( pCreds );

    return 0;
}
Figure 13 Credential APIs (Close Figure 13)

CredRead
CredWrite
CredRename
CredEnumerate
CredDelete
CredFree
CredGetSessionTypes
CredGetTargetInfo
CredMarshalCredential
CredIsMarshaledCredential
CredReadDomainCredentials
CredWriteDomainCredentials
CredProfileLoaded
CredUnmarshalCredential
CredUIPromptForCredentials
CredUICmdLinePromptForCredentials
CredUIParseUserName
CredUIConfirmCredentials
CredUIStoreSSOCred

Another new set of security APIs in ADVAPI32 is the safer APIs. The goal is to make it easy for programs which launch other programs to query security policy for approval before an executable is launched. This capability isn't limited to executables, as other sorts of active content such as scripts can also be verified. The APIs are especially useful for handling e-mail attachments. Figure 15 shows the safer APIs, which are defined in WinSafer.H. As it stands now, the documentation for these APIs is lacking in practical examples of how to use the functions.

Figure 15 More APIs (Close Figure 15)

Safer APIs
SaferGetPolicyInformation
SaferSetPolicyInformation
SaferCreateLevel
SaferCloseLevel
SaferComputeTokenFromLevel
SaferIdentifyLevel
SaferGetLevelInformation
SaferSetLevelInformation
SaferRecordEventLogEntry
Event Tracing APIs
TraceMessage
TraceMessageVA
EnumerateTraceGUIDs
FlushTrace

Also new in ADVAPI32 are a few new event tracing APIs (see Figure 15). Event tracing was introduced in Windows 2000. As you'd expect, TraceMessage sends an event to the designated trace session. TraceMessageVA is essentially the same API, except that it takes a variable number of arguments. EnumerateTraceGUIDs returns information about the system's event trace providers, while FlushTrace does just what its name implies.

OLE Gets Old

Traditionally, the OLE32 and OLEAUT32 DLLs have been hotbeds of new APIs and interfaces. Since Windows 2000, it's slowed significantly due to the focus on the .NET Framework. One new API that looks interesting is CoRegisterInitializeSpy. You provide an interface implementation of type IInitializeSpy, and its methods are called before and after CoInitialize(Ex) and CoUninitialize on the thread for which the spy is registered.

The CoGetContextToken API returns the IObjContext for the current context. It's interesting primarily because this value is stored in the ReservedForOle field in the TEB, which is finally documented in WINTERNL.H.

The CoFreeUnusedLibrariesEx function is like its predecessor function, with the added capability of freeing unused libraries immediately rather than waiting the default 10 minutes. Finally, the new CoInvalidateRemoteMachineBindings API tells the OLE Service Control Manager to flush any cached remote procedure call binding handles for the specified machine. Beyond these few APIs, there's nothing else new in OLE and nothing at all new in OLEAUT32.

Clustering

The clustering API has a small number of new functions worth mentioning. Clustering is the ability to keep resources (such as applications, disks, and file shares) highly available by using more than one physical resource to represent a single logical resource to the outside world. The majority of the new clustering APIs are what I call the "EnumCount" subset. In a nutshell, there are five types of cluster objects: Group, Network, Node, Resource, and Resource Type. These objects can be enumerated via handle-based APIs. The new APIs here (for instance, ClusterNodeGetEnumCount) return the number of objects represented by the enumeration handle.

The two remaining new APIs are EvictClusterNodeEx and SetClusterServiceAccountPassword. EvictClusterNodeEx is like its predecessor function, with the addition of a timeout capability. The SetClusterServiceAccountPassword changes the password for the cluster service user account on all online nodes.

Real Time Client API

The real time client (RTC) API has one of those highfalutin names that doesn't really convey what it does. Although the RTC API encompasses quite a lot of functionality, I find it easiest to think of it primarily as the instant messaging (IM) API. The documentation paints a very broad picture, however, so it's definitely worth reprinting as follows:

The RTC Client API enables you to build applications that can make PC-to-PC, PC-to-phone, or phone-to-phone calls or create IM sessions over the Internet. Both voice and video calls can be established on PC-to-PC calls. Presence information on a list of contacts is also supported. Application sharing and whiteboard can be added to enhance the communication capabilities of any type of session.

So what does the RTC API look like? No surprises here in that it's COM-based. The root interface for working with the RTC API is the IRTCClient, which you obtain using CoCreateInstance. From that interface, you can create (or provide) other interfaces such as IRTCSession, IRTCParticipant, IRTCBuddy, IRTCProfile, and so on. Altogether, there are more than two dozen RTC interfaces. If you were so inclined, you could create your own custom instant messaging client using the RTC interfaces.

Conclusion

With all I've shown, it's pretty clear that Windows Server 2003 is a significant improvement over Windows 2000. I've focused on the user-mode programming APIs, but there have also been major changes and additions under the hood, both in performance and reliability. I'm personally very excited to see functionality like vectored exception handling, side-by-side execution, and better support for debug symbols making their way into Windows.

Consider this: Windows XP is the newest client operating system from Microsoft. Since Windows Server 2003 is a superset of Windows XP in all ways, it makes sense to at least consider using the new APIs that Windows XP brings to the table. I'm personally running Windows XP on some of my machines and Windows Server 2003 on others, and I can't tell the difference in my everyday work. I hope that you'll take the jump and explore Windows Server 2003 for yourself.

For background information see:
Windows Server 2003

Matt Pietrekis a software architect and writer. He works for the Compuware/NuMega lab as a lead architect for the BoundsChecker and Distributed Analyzer products. He has authored three books on Windows system programming and is a contributing editor for MSDN Magazine. His Web site (http://www.wheaty.net) has an FAQ and information on previous articles and columns.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


ASP.NET Pipeline: Use Threads and Build Asynchronous Handlers in Your Server-Side Web Code

ASP.NET Pipeline

Use Threads and Build Asynchronous Handlers in Your Server-Side Web Code

Fritz Onion
This article assumes you're familiar with C#, ASP.NET, and Multithreading
Level of Difficulty 1 2 3
Code download available at: Threading.exe (121KB)
SUMMARY

Fortunately for developers, threading in ASP.NET is a lot easier than it was in ASP. In this article, the author takes a look at threading in the ASP.NET HTTP pipeline, and explains how threads are managed efficiently without the involvement of the developer. The article considers how the common language runtime threadpool is used by ASP.NET to service requests, looks at the pooling mechanisms used for handlers, modules, and applications, and covers both IIS 5.0 and IIS 6.0 and how they differ in their approach to request processing and thread allocation. Finally, how and when to use asynchronous handlers is discussed for developers who still need to use threads in their own applications.

Contents
Threading in ASP.NET
Thread Dispatching in IIS 5.0 and 6.0
The Need for Asynchrony
Asynchronous Handlers
Asynchronous Handler with a Delegate
Asynchronous Handler with Custom Threads
Asynchronous Handler with a Custom Thread Pool
Asynchronous Pages
Conclusion

I
n traditional ASP, developers faced more threading issues than they knew what to do with. Because ASP was built on top of COM, there were very specific rules about the threading requirements of objects. Objects used by pages needed to be apartment threaded for maximum efficiency. Conversely, objects living in Session and Application state needed to be context-agile and able to aggregate the freethreaded marshaler, as well as have their state protected against concurrent access.

If you look at the documentation for ASP.NET, however, you will find very little information about threading requirements. Does this mean that all threading issues are solved and that ASP.NET developers can live carefree lives building .aspx pages and Microsoft® .NET Framework classes without cluttering their minds with concurrency issues? Well, yes, in most cases it does.

In this article, I will explore the details of threading in the HTTP pipeline and how it efficiently manages threads without burdening the developer. I will also take a look at how ASP.NET uses the common language runtime (CLR) thread pool to service requests, the pooling mechanisms used for handlers, modules, and applications, and some of the differences in request processing and thread allocation between Microsoft Internet Information Services (IIS) 5.0 and IIS 6.0. For those developers who just don't feel complete if they don't use threads in their applications, I will also introduce asynchronous handlers and discuss how and when to use them.

Threading in ASP.NET

To efficiently service multiple client requests, Web servers make extensive use of concurrency by launching multiple processes and/or spawning multiple threads to service requests and perform work. ASP.NET is no exception and uses multiple threads within each worker process to service requests. In spite of this fact, ASP.NET developers need not concern themselves with the multitude of issues that come with developing in a multithreaded environment. Page requests are always serviced on the same thread, and a distinct instance of the Page class is always created to service any new requests. Distinct instances of application and module objects are also used to service each request. It is important, however, to understand how threads are used to service requests so that you don't make incorrect assumptions about which objects in your applications may be accessed by concurrent threads.

To begin with, ASP.NET uses the process-wide CLR thread pool to service requests (for more background on the CLR thread pool, see the .NET column in this issue). The size of this pool is configurable in the processModel element of machine.config, and is set to a default of 25 worker threads and 25 I/O threads:

<processModel enable="true"
              •••
            maxWorkerThreads="25"
            maxIoThreads="25" />
Threads are processed differently depending whether you are using IIS 5.0 or IIS 6.0, which I will discuss shortly. For each request that comes in, a new instance of the appropriate HttpApplication-derived class is created, as are the associated modules for that app. To avoid reallocating apps and modules too often, each AppDomain maintains a pool of apps and modules. The maximum size of the app pool is the same as the size of the thread pool, so by default, up to 25 requests per worker process can be processed concurrently, each with its own app and module set. Figure 1 shows a snapshot in time of the ASP.NET worker process. In this scenario, there are two active apps in the worker process, each with a dedicated AppDomain. Each app is currently processing two requests, and each is using two threads from the CLR thread pool to service those requests.

Figure 1 Threading and Pooling in the HTTP Pipeline
Figure 1 Threading and Pooling in the HTTP Pipeline

Several aspects of this architecture may influence the way you construct your ASP.NET applications. First, the fact that applications and modules are instantiated multiple times means that you should never rely on adding fields or other states because they will not be shared across multiple requests, as you might think. Instead, use one of the many state repositories available in the pipeline, such as the application-wide cache, the session state bag, the appliction state bag, or the per-request Items collection of the HttpContext class. Additionally you can use static data if properly synchronized.

By default, most handlers created to service requests are not pooled. You can pool handlers and even control pooling on a per-handler basis through the IsReusable property of IHttpHandler, but the only handlers that are pooled implicitly are custom handlers that you write with no designated handler factory. The PageHandlerFactory class does not perform pooling, nor does the SimpleHandlerFactory class, which instantiates .ashx-defined handlers. Typically, therefore, each request is serviced by a freshly allocated instance of the appropriate handler class, which is discarded after the request completes.

Thread Dispatching in IIS 5.0 and 6.0

Requests in IIS 5.0 are typically serviced over I/O threads, or threads performing asynchronous I/O because requests are dispatched to the worker process using asynchronous writes to a named pipe. When the request is serviced in the worker process, it will typically use the thread on which the asynchronous read was performed. The threads used to service requests in the worker process are drawn from the process-wide CLR thread pool, although thread pool threads tied to I/O completion ports incorrectly return false from the IsThreadPoolThread property of the Thread class, so don't count on that property.

Although requests are typically serviced on I/O threads in IIS 5.0, under heavy loads, some requests are passed off to normal worker threads from the pool, so both types of threads are used simultaneously. The default size of the CLR thread pool is 25, with separate limits for I/O threads and worker threads. You can change the upper number of threads allocated in the CLR thread pool by modifying the processModel element found in machine.config, as was shown earlier.

Things change dramatically with IIS 6.0 (which ships with Windows® Server 2003). To begin with, inetinfo.exe is no longer used to service HTTP requests (although it remains in place to service other protocols, like FTP). Instead, HTTP requests are queued in a kernel-mode service, http.sys, which takes care of dispatching each request to the appropriate application queue. In addition, IIS 6.0 supports application pools, which are groupings of applications that will share a single worker process, now named w3pwp.exe. Application pools give you the option of having as many distinct worker processes as you want on any given server, depending on how much you care about process isolation between your Web applications.

This paradigm changes the way requests are handled in ASP.NET. Instead of dispatching requests from inetinfo.exe to the ASP.NET worker process, http.sys directly queues each request in the appropriate process. Thus all requests are now serviced by worker threads drawn from the CLR thread pool and never on I/O threads. Contrary to comments found in machine.config on Windows Server 2003, the processModel element is still used to initialize the upper bound of the thread pool, although all other attributes are ignored as they are now stored in the IIS 6.0 metabase.

The Need for Asynchrony

Many factors influence the scalability of a Web application. In general, any time multiple concurrent operations request exclusive access to a shared resource, the scalability of the system is jeopardized. High on the list of resources that are shared on a system is the CPU itself (or CPUs on a multiprocessor box). A server running ASP.NET has its CPU shared among concurrent requests by way of a thread pool. The purpose of the thread pool is to efficiently allocate CPU time for processing requests with an upper limit on the total number of threads allocated. This is an important restriction to have in place, as creating an unbounded number of threads can easily grind a system to a halt when a large number of requests are made concurrently.

However, in cases where threads from the thread pool are used to perform work that is not CPU-intensive (like making a request to a remote database or invoking a remote Web Service), it is possible to saturate the thread pool without high CPU utilization. In this scenario, the thread pool is in fact reducing the scalability of the system because requests may be delayed (or possibly rejected) even though the server was not busy processing other requests.

As an example of swamping the thread pool, consider the following ASP.NET page, slow.aspx, that artificially delays its response by two seconds (with non-CPU-bound activity) and prints out the thread on which it was serviced:

<!-- File: slow.aspx -->
<%@ Page Language="C#" %>
<%@ Import Namespace="System.Reflection" %>
<%@ Import Namespace="System.Threading" %>

<script runat="server">
  protected void Page_Load(object src, EventArgs e)
  {
    System.Threading.Thread.Sleep(2000);
    Response.Output.Write("SlowResponse, threadid={0}",
                    AppDomain.GetCurrentThreadId());
  }
</script>

Another ASP.NET page, fast.aspx, is nearly identical, except that it does not sleep for the two seconds that slow.aspx does:

<!-- File: fast.aspx -->
<%@ Page Language="C#" %>
<%@ Import Namespace="System.Reflection" %>

<script runat="server">
  protected void Page_Load(object src, EventArgs e)
  {
    Response.Output.Write("FastResponse, threadid={0}",
                    AppDomain.GetCurrentThreadId());
  }
</script>

Using the Web Application Stress tool against a server hosting these two pages, you can test the average response time under heavy request traffic. An initial test issuing requests on 100 concurrent threads from a separate client machine to only the fast.aspx page results in an average response time of 0.0824 seconds, servicing 52,190 requests during a one-minute time interval. Running the client against both fast.aspx and slow.aspx, however, reduces the average response time for fast.aspx requests to 4.47 seconds (and slow.aspx is 6.54 seconds), with the total number of requests serviced in one minute reduced to 1062.

While the numbers are dramatically different across the board, the most disturbing result is that it took more than four seconds on average to serve fast.aspx, which you know takes very little time to process. Although contrived, this example shows the worst case scenario when you have pages that are relatively slow to serve fighting for the same request threads that more responsive pages contend for—a dramatic increase in response times. If the slow pages in your application are sluggish due to intensive CPU usage, there is not much you can do about it except to throw in more hardware. If, however, the slow pages in your application are slow because they are waiting on non-CPU-bound operations to complete, then the problem is not the lack of CPU power, but the fact that slow requests are saturating the thread pool so that other requests must be queued until a thread is released.

You have a few options if you discover that your system's scalability is being adversely affected by the ASP.NET thread pool. You could change the upper limit of the thread pool. As you have seen, the default upper limit is set to 25 worker threads and 25 I/O threads. Of course, increasing the upper bound in the thread pool is a somewhat coarse-grained solution to the problem. If you can find the magic number that keeps your CPU utilization high and only delays or denies requests when the server is truly too busy to process more requests, you are very fortunate. The number of threads necessary to keep the server truly busy is not static, but will fluctuate based on factors like demand and process complexity.

Another potential solution is to carefully identify the requests in your system that take longer to complete and perform non-CPU-bound operations and to allocate distinct threads to service them, freeing up the original thread pool thread to service additional requests. This brings me to asynchronous handlers.

Asynchronous Handlers

While most ASP.NET pages and handlers are serviced synchronously on threads drawn from the process-wide thread pool, it is possible to create handlers (and even pages) that service requests asynchronously. Asynchronous handlers implement the IHttpAsyncHandler interface, which derives from IHttpHandler implemented by synchronous handlers:

public interface IHttpAsyncHandler : IHttpHandler
{
  IAsyncResult BeginProcessRequest(HttpContext ctx,
                                   AsyncCallback cb,
                                   object obj);
  void EndProcessRequest(IAsyncResult ar);
}

Handlers that implement this interface must implement two additional methods besides the standard methods of IHttpHandler. The first, BeginProcessRequest, is called by the application class instead of directly calling the ProcessRequest. It is then up to the handler to launch a new thread to process the request and then immediately return from the BeginProcessRequest method, passing back a reference to a valid IAsyncResult instance so that the runtime knows the operation is complete. The other method, EndProcessRequest, is called when the request processing is complete and can be used to clean up any allocated resources if necessary.

Asynchronous Handler with a Delegate

If you have done any work with asynchronous execution in the .NET Framework, you know that the simplest and most often recommended way to perform work asynchronously is to use asynchronous delegate invocation. Invoking a delegate asynchronously by calling BeginInvoke implicitly uses a thread from the process-wide thread pool to execute the delegate function. Implementing an asynchronous handler using asynchronous delegate invocation is actually quite straightforward, as shown in Figure 2.

Figure 2 HTTP Handler Implementation (Close Figure 2)
<!-- File: AsyncDelegate.ashx -->
<%@ WebHandler Language="C#"
    Class="EssentialAspDotNet.HttpPipeline.AsyncHandler" %>

using System;
using System.Web;
using System.Threading;
using System.Diagnostics;
using System.Reflection;

namespace EssentialAspDotNet.HttpPipeline
{
  public delegate void ProcessRequestDelegate(HttpContext ctx);

  public class AsyncHandler : IHttpAsyncHandler
  {
    public void ProcessRequest(HttpContext ctx)
    {
       System.Threading.Thread.Sleep(2000);
       ctx.Response.Output.Write(
                    "AsyncDelegate, threaded={0}",
                    AppDomain.GetCurrentThreadId());
    }

    public bool IsReusable
    {
      get { return true;}
    }

    public IAsyncResult BeginProcessRequest(HttpContext ctx,
                AsyncCallback cb, object obj)
    {
      ProcessRequestDelegate prg =
               new ProcessRequestDelegate(ProcessRequest);
      // Fire-and-forget
      return prg.BeginInvoke(ctx, cb, obj);
    }

    public void EndProcessRequest(IAsyncResult ar)
    {
    }
  }
}

My implementation of IHttpAsyncHandler.BeginProcessRequest calls BeginInvoke on an instance of my ProcessRequestDelegate which has been initialized with a reference to my ProcessRequest method. This has the effect of executing ProcessRequest on a separate thread, distinct from the request thread.

Running the same stress test I performed earlier with my slow.aspx page, this time using my new asyncDelegate.ashx handler instead of fast.aspx, produces the following results. The average response time for fast.aspx is 4.14 seconds and 7.86 seconds for asyncDelegate.ashx, with the total number of serviced requests at 957.

Disappointingly, the response time did not really improve for requests made to fast.aspx. The reason for this is that asynchronous delegate invocation completely defeats the purpose of building an asynchronous handler because it draws from the same process-wide CLR thread pool that ASP.NET uses to service requests. While the primary request thread is indeed returned to the thread pool, another thread is taken out to perform the asynchronous delegate execution, which means that there is a net gain of zero threads for servicing additional requests, rendering the handler useless. You will run into the same problem if you use ThreadPool.QueueUserWorkItem to perform the asynchronous request processing because it also draws threads from the same process-wide CLR thread pool.

Asynchronous Handler with Custom Threads

To build a truly effective asynchronous handler, you must spawn an additional thread by hand in response to BeginProcessRequest. There are three important aspects to building a successful asynchronous handler. First, construct a class that supports IAsyncResult to return from BeginProcessRequest. Then, spawn the thread to perform your request processing asynchronously. Finally, notify ASP.NET that you are finished processing the request and are ready to return the response.

I will begin the construction of an asynchronous handler by building a class that supports IAsyncResult. This class is going to be returned from the call to BeginProcessRequest, and it will later be passed into my implementation of EndProcessRequest, so among other things, it is a useful place to store request-specific state that I may need to use during the processing of a request. The complete IAsyncResult interface to be implemented is shown here:

public interface IAsyncResult
{
  public object     AsyncState             { get; }
  public bool       CompletedSynchronously { get; }
  public bool       IsCompleted            { get; }
  public WaitHandle AsyncWaitHandle        { get; }
}

In this example, I'll store a reference to the HttpContext object associated with the request, a reference to the AsyncCallback delegate passed into BeginProcessRequest (which I'll later invoke to complete the request), and a generic object reference for extra data that may be used by the caller of BeginProcessRequest. The other element that must be implemented in this class is a synchronization object that threads can wait on to be signaled when the operation completes. I will use the common technique of supplying a ManualResetEvent that fires when my request is complete, but allocate it only if someone requests it. Finally, my class will have a convenience method called CompleteRequest, which will trigger the ManualResetEvent if it was created, invoke the AsyncCallback delegate, and set the IsCompleted flag to true. The complete class definition for AsyncRequestState is shown in Figure 3.

Figure 3 AsyncRequestState Class Definition (Close Figure 3)
// From file: AsyncThread.ashx
class AsyncRequestState : IAsyncResult
{
  public AsyncRequestState(HttpContext ctx,
                           AsyncCallback cb,
                           object extraData )
  {
    _ctx = ctx;
    _cb = cb;
    _extraData = extraData;
  }

  internal HttpContext _ctx;
  internal AsyncCallback _cb;
  internal object _extraData;
  private bool _isCompleted = false;
  private ManualResetEvent _callCompleteEvent = null;

  internal void CompleteRequest()
  {
    _isCompleted = true;
    lock (this)
    {
      if (_callCompleteEvent != null)
        _callCompleteEvent.Set();
    }
    // if a callback was registered, invoke it now
    if (_cb != null)
      _cb(this);
  }

  // IAsyncResult interface property implementations
  public object AsyncState
  { get { return(_extraData);   } }
  public bool CompletedSynchronously
  { get { return(false);        } }
  public bool IsCompleted
  { get { return(_isCompleted); } }
  public WaitHandle AsyncWaitHandle
  {
    get
    {
      lock( this )
      {
        if( _callCompleteEvent == null )
        _callCompleteEvent = new ManualResetEvent(false);

        return _callCompleteEvent;
      }
    }
  }
}

The next step is to spawn a new thread to process my request. The method I call on this new thread will need access to the state I cached in the AsyncRequestState class just shown. Unfortunately, the ThreadStart delegate used to spawn new threads in the .NET Framework does not take any parameters. To get around this, I create another class with the necessary state cached as data members (in this case, simply a reference to the AsyncRequestState object for this request) and with an instance method that can be used to initialize the ThreadStart delegate. This class, called AsyncRequest, is shown in Figure 4. Note that the ProcessRequest method I define in this class is the method that will be called from the manually created thread, and when it completes, it signals that the request processing is complete by invoking CompleteRequest on the AsyncRequestState object.

Figure 4 AsyncRequest (Close Figure 4)
// From file: AsyncThread.ashx
class AsyncRequest
{
  private AsyncRequestState _asyncRequestState;

  public AsyncRequest(AsyncRequestState ars)
  {
    _asyncRequestState = ars;
  }

  public void ProcessRequest()
  {
    // This is where the non-cpu-bound activity would take place, such as
    // accessing a Web Service, polling a slow piece of hardware, or
    // performing a lengthy database operation. I put the thread to sleep
    // for two seconds to simulate a lengthy operation.
    Thread.Sleep(2000);

    _asyncRequestState._ctx.Response.Output.Write(
            "AsyncThread, {0}",
            AppDomain.GetCurrentThreadId());

    // tell asp.net I am finished processing this request
    _asyncRequestState.CompleteRequest();
  }
}

Finally, I am ready to build the asynchronous handler class itself. This class, which I have called AsyncHandler, must implement all of the methods of the IHttpAsyncHandler interface shown earlier, which derives from IHttpHandler for a total of four methods. The ProcessRequest method is not used and will never be called because I implement BeginProcessRequest. In BeginProcessRequest, I create a new instance of the AsyncRequestState class, initializing it with the HttpContext object, the AsyncCallback delegate, and the generic object reference each passed in as parameters. I then prepare a new AsyncRequest object, initialized with the freshly created AsyncRequestState object, and launch a new thread. The implementation of EndProcessRequest will not do anything in this example, but it can be used in general to perform any cleanup or last-minute response additions. The AsyncHandler class definition is shown in Figure 5.

Figure 5 AsyncHandler Class Definition (Close Figure 5)
// From file: AsyncThread.ashx
public class AsyncHandler : IHttpAsyncHandler
{
  public void ProcessRequest(HttpContext ctx)
  {
    // not used
  }

  public bool IsReusable
  {
    get { return false;}
  }

  public IAsyncResult BeginProcessRequest(HttpContext ctx,
                                          AsyncCallback cb,
                                          object obj)
  {
    AsyncRequestState reqState =
                      new AsyncRequestState(ctx, cb, obj);
    AsyncRequest ar = new AsyncRequest(reqState);
    ThreadStart ts = new ThreadStart(ar.ProcessRequest);
    Thread t = new Thread(ts);
    t.Start();

    return reqState;
  }

  public void EndProcessRequest(IAsyncResult ar)
  {
    // This will be called on the manually created thread in response to
    // calling the ASP.NET AsyncCallback delegate once the request has
    // completed processing. The incoming IAsyncResult parameter will be
    // a reference to the AsyncRequestState class, so the Context can be
    // accessed through that class. Note - you *cannot* access the
    // current context using the HttpContext.Current property, as you are
    // running on your own thread which has not been initialized with a
    // context reference.
    AsyncRequestState ars = ar as AsyncRequestState;
    if (ars != null)
    {
     // here you could perform some cleanup, write something else to the
     // Response, or whatever else you need to do
    }
  }
}

When I build this handler and register it as an endpoint, as I did with the earlier handlers, it will successfully process requests asynchronously from the calling request thread of ASP.NET. Figure 6 shows the sequence of events that will take place when a request mapped onto the asynchronous handler is made. First the application class sees that my handler implements IHttpAsyncHandler, so instead of calling the synchronous ProcessRequest method, it invokes BeginProcessRequest on the handler, passing in the current context and an asynchronous callback delegate for completion call back. The handler then creates a new AsyncRequestState object, initialized with the parameters passed into BeginProcessRequest. Next, the handler creates a new AsyncRequest object, initialized with the AsyncRequestState object, and launches a new thread using the AsyncRequest.ProcessRequest method as the entry point. The handler then returns the AsyncRequestState object back to the application object and the calling thread is returned to the thread pool while the hand-created thread continues processing the request.

Figure 6 Async Handler Operation Phase 1
Figure 6 Async Handler Operation Phase 1

Once the ProcessRequest method of the AsyncRequest object is finished performing its lengthy tasks, it calls the CompleteRequest method of the AsyncRequestState class. This in turn will fire the AsyncCallback delegate originally passed into the BeginProcessRequest method, signaling that the response is prepared and ready for return. The first thing the AsyncCallback delegate will do is call the EndProcessRequest method on the asynchronous handler class. Once that returns, it will trigger the completion of the request by sending back the prepared response. All of this processing happens on the secondary thread that I created in the handler, not on a thread pool thread. Figure 7 shows the steps involved in completing the request to the asynchronous handler.

Figure 7 Async Handler Operation Phase 2
Figure 7 Async Handler Operation Phase 2

When I ran the same stress test I performed earlier with the slow.aspx page, this time using the new AsyncThread.ashx handler instead of AsyncDelegate.aspx, the following results are produced. The average response time for fast.aspx is .012 seconds and 2.014 seconds for AsyncThread.ashx, with the total number of serviced requests at 6132.

The average response time for requests made to fast.aspx is much improved because the thread pool is not tied up servicing requests to the slow handler.

Asynchronous Handler with a Custom Thread Pool

There is one remaining problem with this asynchronous handler implementation—it has the potential to create an unbounded number of threads. If a large number of requests are made to the asynchronous handler, all of which take a significant amount of time to service, then you could easily end up creating more threads than the underlying operating system can handle. To deal with this, I need to provide a secondary thread pool to service the requests in a bounded fashion. Details of creating custom thread pools are beyond the scope of this article, but a full example written in C# by Mike Woodring is available for download at http://www.bearcanyon.com/dotnet/threadpool.zip. This custom thread pool also supports propagation of call context and HttpContext, which means that you can access HttpContext.Current in the asynchronous method, unlike the previous example. The class implementing the thread pool is called DevelopMentor.ThreadPool, and its constructor takes an initial thread count, a maximum thread count, and a string to give the pool a name. To issue asynchronous requests with this pool, call PostRequest and pass in a reference to a WorkRequestDelegate.

To rewrite the asynchronous handler to use this custom thread pool there will be two major changes that just spawned new threads. First, add a static member of type DevelopMentor.ThreadPool to the handler class and a static constructor to initialize the thread pool once per AppDomain. Then change the implementation of BeginProcessRequest to call PostRequest on the thread pool with a new instance of WorkRequestDelegate (see Figure 8). If you run the same stress test performed earlier with the slow.aspx page, this time using the new AsyncPool.ashx handler instead of AsyncThread.aspx, you get the following results. The average response time for fast.aspx is .055 seconds and 12.24 seconds for AsyncThread.ashx with the total number of serviced requests at 1042.

Figure 8 Calling PostRequest (Close Figure 8)
<!-- File: AsyncPool.ashx -->
<%@ WebHandler Language="C#"
    Class="EssentialAspDotNet.HttpPipeline.AsyncHandler" %>

namespace EssentialAspDotNet.HttpPipeline
{
  // AsyncRequestState and AsyncRequest remain the same
  // as in the previous example

  public class AsyncHandler : IHttpAsyncHandler
  {
    static DevelopMentor.ThreadPool _threadPool;

    static AsyncHandler()
    {
      _threadPool =
        new DevelopMentor.ThreadPool(2, 25, "AsyncPool");
      _threadPool.PropogateCallContext = true;
      _threadPool.PropogateThreadPrincipal = true;
      _threadPool.PropogateHttpContext = true;
      _threadPool.Start();
    }

    public void ProcessRequest(HttpContext ctx)
    {
     // not used
    }

    public bool IsReusable
    {
      get { return false;}
    }

    public IAsyncResult BeginProcessRequest(HttpContext ctx,
                     AsyncCallback cb, object obj)
    {
      AsyncRequestState reqState =
                     new AsyncRequestState(ctx, cb, obj);
      _threadPool.PostRequest(
                     new DevelopMentor.WorkRequestDelegate(ProcessRequest),
                     reqState);

      return reqState;
    }

    public void EndProcessRequest(IAsyncResult ar)
    {
    }

    void ProcessRequest(object state, DateTime requestTime)
    {
      AsyncRequestState reqState = state as AsyncRequestState;

      // Take some time to do it
      Thread.Sleep(2000);

      reqState._ctx.Response.Output.Write(
                   "AsyncThreadPool, {0}",
                    AppDomain.GetCurrentThreadId);

      // tell asp.net you are finished processing this request
      reqState.CompleteRequest();
    }

  }
}

Note that the average response time for requests made to fast.aspx is still minimal, and there is no longer the danger of creating an unbounded number of threads. There is some additional management involved with maintaining a second thread pool, which results in the extended request times for the slow handler.

Asynchronous Pages

So far, all of the examples of asynchronous request handling have involved building custom handlers (in this case, using .ashx files). There will be many cases where you can identify normal pages (.aspx) in your ASP.NET applications that are also good candidates for asynchronous execution because they perform non-CPU-bound tasks that take significant amounts of time. Such pages are also potential bottlenecks for the ASP.NET thread pool. The Page class itself implements the IHttpHandler interface, and thus services requests synchronously. To change a given .aspx page to service requests asynchronously, it is necessary to implement IHttpAsyncHandler in the Page-derived class and then use the techniques described in this article to invoke the ProcessRequest method of the Page class on an alternate thread.

One convenient way to make any page asynchronous is to create an alternate base class from which the primary page can derive and then specify this alternate base class as the base class from which the page should derive using the Inherits attribute of the @Page directive. The AsyncPage class definition shown in Figure 9 defines such a base class. Note that this implementation uses the custom thread pool shown earlier to service requests to the page asynchronously. This implementation of BeginProcessRequest invokes the PostRequest method of the thread pool with a new WorkRequestDelegate initialized with a reference to my overload of ProcessRequest. In the overload, I invoke the base class scope of ProcessRequest (with the original signature as defined in the IHttpHandler interface), which performs the standard request processing of a page, only this time it will be performed on the thread pool thread instead of the primary request thread (see Figure 9).

Figure 9 AsyncPage (Close Figure 9)
namespace EssentialAspDotNet.HttpPipeline
{
 public class AsyncPage : Page, IHttpAsyncHandler
  {
    static protected DevelopMentor.ThreadPool _threadPool;

    static AsyncPage()
    {
      _threadPool =
       new DevelopMentor.ThreadPool(2, 25, "AsyncPool");
      _threadPool.PropogateCallContext = true;
      _threadPool.PropogateThreadPrincipal = true;
      _threadPool.PropogateHttpContext = true;
      _threadPool.Start();
    }

    public new void ProcessRequest(HttpContext ctx)
    {
      // not used
    }

    public new bool IsReusable
    {
      get { return false;}
    }

    public IAsyncResult BeginProcessRequest(HttpContext ctx,
                                            AsyncCallback cb, object obj)
    {
      AsyncRequestState reqState =
             new AsyncRequestState(ctx, cb, obj);
      _threadPool.PostRequest(
             new DevelopMentor.WorkRequestDelegate(ProcessRequest),
             reqState);

      return reqState;
    }

    public void EndProcessRequest(IAsyncResult ar)
    {
    }

    void ProcessRequest(object state, DateTime requestTime)
    {
      AsyncRequestState reqState = state as AsyncRequestState;

      // Synchronously call base class Page.ProcessRequest
      // as you are now on a thread pool thread.
      base.ProcessRequest(reqState._ctx);

      // Once complete, call CompleteRequest to finish
      reqState.CompleteRequest();
    }
  }
}

As an example of using this new asynchronous page base class, consider the following ASPX page that inherits from the AsyncPage class. This base class could be applied to any ASPX page in a similar fashion to turn it into an asynchronously executed page:

<!-- AsyncPage.aspx -->
<%@ Page Language="C#"
    Inherits="EssentialAspDotNet.HttpPipeline.AsyncPage" %>
<%@ import Namespace="System.Reflection" %>
<%@ import Namespace="System.Threading" %>

<script runat="server">
protected void Page_Load(object src, EventArgs e)
{
   System.Threading.Thread.Sleep(2000);
   Response.Output.Write("AsyncPage, {0}",
                         AppDomain.GetCurrentThreadId);
}
</script>

The results of performing the stress test with fast.aspx and the asynchronous page, AsyncPage.aspx, are as follows with results matching those of the asynchronous custom handler using a custom thread pool. The average response time for fast.aspx is .054 seconds and 11.92 seconds for AsyncPage.aspx, with the total number of serviced requests at 965.

Conclusion

Building asynchronous handlers instead of synchronous ones actually adds additional overhead to request processing, so you should take the time to determine if the asynchronous capability of these types of handlers is really what you need. The purpose of an asynchronous handler is to free up an ASP.NET thread pool thread to service additional requests while the handler is processing the original request. This only makes sense if the work of servicing the request requires a significant amount of non-CPU-bound time to complete. For example, if the completion of a request depends on the completion of a number of remote procedure calls, or perhaps Web Service invocations, that would be a candidate to implement an asynchronous handler. Building asynchronous handlers to service CPU-intensive requests only adds threads to compete with the ASP.NET thread pool threads and may actually slow down the overall processing time for the request.

For related articles see:
The ASP Column: HTTP Modules
HTTP Pipelines: Securely Implement Request Processing, Filtering, and Content Redirection with HTTP Pipelines in ASP.NET
IIS 6.0: New Features Improve Your Web Server's Performance, Reliability, and Scalability

For background information see:
ASP.NET QuickStart Tutorial
Building Secure ASP.NET Applications: Authentication, Authorization, and Secure Communication

Fritz Onionis on the technical staff at DevelopMentor, where he researches, teaches, and writes courses on the .NET Framework including Essential ASP.NET. This article is adapted from Essential ASP.NET with Examples in C# (Addison-Wesley, 2003). A second edition, Essential ASP.NET with Examples in Visual Basic .NET, is available for those averse to semicolons.


Zip Your Data: Using the Zip Classes in the J# Class Libraries to Compress Files and Data with C#

Zip Your Data

Using the Zip Classes in the J# Class Libraries to Compress Files and Data with C#

Ianier Munoz
This article assumes you're familiar with C# and Windows Forms
Level of Difficulty 1 2 3
Code download available at: ZipCompression.exe (150KB)
SUMMARY

Zip compression lets you save space and network bandwidth when storing files or sending them over the wire. In addition, you don't lose the directory structure of folders you Zip, which makes it a pretty useful compression scheme. The C# language doesn't have any classes that let you manipulate Zip files, but since .NET-targeted languages can share class implementations, and J# exposes classes in the java.util.zip namespace, you can get to those classes in your C# code. This article explains how to use the Microsoft J# class libraries to create an application in C# that compresses and decompresses Zip files. It also shows other unique parts of the J# runtime you can use from any .NET-compliant language to save some coding.

Contents
Zip Files and C#
The Solution
SharpZip
Enumerating Zip Entries
Decompressing Zip Files
Creating and Modifying Zip Files
Low-level Zip Compression
Other J# Goodies
Application Deployment
Conclusion

Z
ip is a popular standard for data transfer and storage because it saves both disk space and network bandwidth. Typical text and database files can be compressed to as little as 10 percent of their original size. Even though binary files don't compress as well, 50 percent compression is often achieved.

An additional advantage of Zip files is that a single file can contain multiple files while preserving the directory structure. This allows you to send a full directory tree attached to an e-mail message and have the recipient recover the original file structure.

The Zip data format is open and not subject to patents or other legal issues. Developers are free to create applications that manipulate Zip files and to use the low-level Zip compression algorithms for temporarily reducing the size of their own custom data. The authors of the Zip data specification made the compression and decompression algorithms available to developers in a library named zlib (http://www.gzip.org/zlib). This library was adopted by the Java platform in version 1.1 of the Java Development Kit (JDK) to form the basis of the Java Archive (JAR) file format, so since JDK version 1.1, the standard Java language APIs include the necessary classes for manipulating Zip files. You can find these classes under the java.util.zip namespace.

Zip Files and C#

I wanted to use Zip compression in my applications written in C#. Unfortunately, the Microsoft® .NET Framework does not currently include any classes for manipulating Zip files. I did, however, find several products related to Zip compression. For example, #ziplib (formerly NZipLib, http://www.icsharpcode.net/OpenSource/SharpZipLib/default.asp) is a port of zlib library to C#. Its license allows developers to include this library in commercial, closed-source applications. However, as MSDN Magazine goes to press, #ziplib is in its prerelease state (version 0.31).

Another solution would be to use the unmanaged zlib as a Windows® DLL and write the necessary Interop wrappers for it, but since compression involves passing around a significant amount of data during each function call, coding Interop wrappers for optimum performance would be a difficult process. Although other libraries are available, they are not free.

The Solution

The .NET Framework was designed with language interoperability in mind. Any managed component that follows certain rules can be used correctly from any .NET-compliant programming language that implements the necessary functionality. The set of rules and language features required for interoperability is known as the Common Language Specification (CLS).

All .NET language compilers implemented by Microsoft are CLS-compliant, including Microsoft Visual J#™ .NET—a development tool for Java-language developers who want to build applications and services on the Microsoft .NET Framework. (Visual J# .NET has been independently developed by Microsoft. It is neither endorsed nor approved by Sun Microsystems, Inc.) This is why you can use .NET Framework classes in Windows Forms and ASP.NET applications in J#.

As you will see later in this article, some of the classes exposed by the J# runtime are not actually CLS-compliant, but you can still gain access to most of the J# classes from other languages to use specific features that are not implemented by the .NET Framework. Since J# implements the JDK version 1.1.4, it is no surprise that the java.util.zip namespace is available to developers through the J# runtime. In the next section of this article I will present an application written in C# that uses the java.util.zip classes to compress and decompress Zip files for saving space locally and bandwidth over the wire.

All the sample code in this article was developed with Microsoft Visual Studio 2002 and the J# runtime version 1.0 (see the link at the top of this article).

SharpZip

I wrote SharpZip, one of the sample applications accompanying this article, in C#. It's a simplified utility for handling Zip files that allows you to create Zips or open existing ones to extract, append, and delete files (see Figure 1).

Figure 1 SharpZip App
Figure 1 SharpZip App

Before looking into the code, you need to ensure that the J# runtime is properly installed on your system. You don't need to install the full Visual J# .NET product. You can download and install only the J# 1.0 redistributable package, which is available at http://msdn.microsoft.com/vjsharp/downloads/howtoget.asp.

The java.util.zip namespace is implemented in the vjslib.dll assembly. This assembly can be found in the C:\WINNT\Microsoft Visual JSharp .NET\Framework\v1.0.4205\ directory (you will need to replace WINNT with your actual Windows directory).

When you include a reference to vjslib.dll in your project, you can start using the J# namespaces from your code and navigate through the JDK namespaces with the Object Browser (see Figure 2). The important classes are java.util.zip.ZipFile, java.util.zip.ZipEntry, and java.util.zip.ZipOutputStream. These classes, shown in Figure 3, let you manipulate Zip files at the file level.

Figure 3 ZipFile, ZipEntry, and ZipOutputStream Classes (Close Figure 3)

ZipFile | Description
close() | Closes the Zip file
entries() | Returns an enumeration of the Zip file entries
getEntry(String name) | Returns the zip file entry for the specified name, or null if not found
getInputStream(ZipEntry entry) | Returns an input stream for reading the contents of the specified Zip file entry
getName() | Returns the path name of the Zip file
size() | Returns the number of entries in the Zip file
ZipEntry | Description
clone() | Returns a copy of this entry
getComment() | Returns the comment string for the entry, or null if none
getCompressedSize() | Returns the size of the compressed entry data, or -1 if not known
getCrc() | Returns the CRC-32 checksum of the uncompressed entry data, or -1 if not known
getExtra() | Returns the extra field data for the entry, or null if none
getMethod() | Returns the compression method of the entry, or -1 if not specified
getName() | Returns the name of the entry
getSize() | Returns the uncompressed size of the entry data, or -1 if not known
getTime() | Returns the modification time of the entry, or -1 if not specified
hashCode() | Returns the hash code value for this entry
isDirectory() | Returns true if this is a directory entry
setComment(String comment) | Sets the optional comment string for the entry
setCompressedSize(long csize) | Sets the size of the compressed entry data
setCrc(long crc) | Sets the CRC-32 checksum of the uncompressed entry data
setExtra(byte[] extra) | Sets the optional extra field data for the entry
setMethod(int method) | Sets the compression method for the entry
setSize(long size) | Sets the uncompressed size of the entry data
setTime(long time) | Sets the modification time of the entry
toString() | Returns a string representation of the Zip entry
ZipOutputStream | Description
close() | Closes the Zip output stream as well as the stream being filtered
closeEntry() | Closes the current Zip entry and positions the stream for writing the next entry
finish() | Finishes writing the contents of the Zip output stream without closing the underlying stream
putNextEntry(ZipEntry e) | Begins writing a new Zip file entry and positions the stream to the start of the entry data
setComment(String comment) | Sets the Zip file comment
setLevel(int level) | Sets the compression level for subsequent entries which are DEFLATED
setMethod(int method) | Sets the default compression method for subsequent entries
write(byte[] b, int off, int len) | Writes an array of bytes to the current Zip entry data
Figure 2 Namespaces in the Object Browser
Figure 2 Namespaces in the Object Browser

When using the methods outlined in this article, the names might look strange to you because the naming convention that Java uses for identifiers other than classes and interfaces is different from those used in C#. In Java, namespace and method names are written using lower camel case, where the first letter is lowercase and the rest of the words are initial-capitalized, as in "nextElement." However, I'm sure you'll get the hang of it.

Enumerating Zip Entries

The entries method of the java.util.zip.ZipFile class returns an object that implements the java.util.Enumeration interface. The application then steps through the enumeration to retrieve instances of ZipEntry representing individual entries in the Zip file. The ZipEntry class exposes all the necessary information such as file name, compression method, time stamp, original and compressed size, and so on (see Figure 4).

Figure 4 Enumerating Zip Entries (Close Figure 4)
private void DisplayEntries()
{
    zipListView.BeginUpdate();
    try
    {
        zipListView.Items.Clear();
        foreach(ZipEntry entry in new EnumerationAdapter(new
          EnumerationMethod(CurrentFile.entries)))
        {
            if (!entry.isDirectory())
            {
                string name = entry.getName();
                ListViewItem item = new
                    ListViewItem(Path.GetFileName(name));
                item.SubItems.Add(entry.getSize().ToString());
                item.SubItems.Add(entry.getCompressedSize().ToString());
                item.SubItems.Add(Path.GetDirectoryName(name));
                item.Tag = entry;

                zipListView.Items.Add(item);
            }
        }
    }
    finally
    {
        zipListView.EndUpdate();
    }
}

Please note that although the java.util.Enumeration interface is similar to the System.Collections.IEnumerator interface, Java enumerators advance to the next element when you retrieve the current object by calling nextElement, while .NET enumerators advance when you check for the availability of more elements in the MoveNext call. Another important difference is that the Enumeration interface does not provide a method for restarting the traversal.

One advantage of .NET enumerators is that you can access the current element multiple times. On the other hand, Java enumerators allow you to check multiple times for completion, but this is not very useful under most circumstances. Both Java and .NET enumerators are well designed in that they prevent you from forgetting to advance to the next element inside the enumeration loop.

I decided to write a class for wrapping Java enumerators so I could use the C# foreach statement with them. I named this class EnumerationAdapter. The Reset method is emulated by again calling the method that returns the Java enumerator. To do this, the wrapper class constructor takes as a parameter a delegate to the java.util.Enumeration interface instead of the java.util.Enumeration interface itself.

Decompressing Zip Files

The first thing the SharpZip application does when extracting files is to prompt the user for a directory where the files should be created. You may have noticed that the application shows the Browse for Folder dialog box. I was tempted to use the System.Windows.Forms.Design.FolderNameEditor.FolderBrowser class, but the documentation states that this type supports the .NET Framework infrastructure and isn't intended to be used directly, so I used the Shell32 object through COM Interop by importing the Microsoft Shell Controls and Automation type library.

Extracting the original files from a Zip file (unzipping) is a simple operation: just call getInputStream on the ZipFile object, passing the entry for which you want to get the compressed file. The getInputStream method will produce an InputStream from which you can read the content of the archived entry.

The ExtractZipFile helper function does the job for you. Directories are stored in Zip files using separate entries, but the file name on each entry contains the directory information as well, so ExtractZipFile ignores directory entries and extracts the necessary path information from the file names.

To save individual files to disk, just write to a file the content of the InputStream that corresponds to the entry of interest. This time I decided not to wrap Java streams around a custom System.IO.Stream class because the java.io namespace has fairly good support for streams. In particular, java.io.FileOutputStream allows you to create a file to which you can copy the desired entry.

The CopyStream helper function in Figure 5 copies the contents of a java.io.InputStream object to a java.io.OutputStream object. This helper function is used by other parts of the SharpZip application as well. You should note, though, that this example doesn't check whether the output files already exist before overwriting them. You may want to prompt the user by asking whether the file should be overwritten or not.

Figure 5 ZipUtils.cs (Close Figure 5)
using System;
using System.Collections;
using java.util;
using java.util.zip;

namespace CsZip
{
    public delegate Enumeration EnumerationMethod();

    /// <summary>
    /// Wraps java enumerators
    /// </summary>
    public class EnumerationAdapter : IEnumerable
    {
        private class EnumerationWrapper : IEnumerator
        {
            private EnumerationMethod m_Method;
            private Enumeration m_Wrapped;
            private object m_Current;

            public EnumerationWrapper(EnumerationMethod method)
            {
                m_Method = method;
            }

            // IEnumerator
            public object Current
            {
                get { return m_Current; }
            }

            public void Reset()
            {
                m_Wrapped = m_Method();
                if (m_Wrapped == null)
                    throw new InvalidOperationException();
            }

            public bool MoveNext()
            {
                if (m_Wrapped == null)
                    Reset();
                bool Result = m_Wrapped.hasMoreElements();
                if (Result)
                    m_Current = m_Wrapped.nextElement();
                return Result;
            }
        }

        private EnumerationMethod m_Method;

        public EnumerationAdapter(EnumerationMethod method)
        {
            if (method == null)
                throw new ArgumentException();
            m_Method = method;
        }

        // IEnumerable
        public IEnumerator GetEnumerator()
        {
            return new EnumerationWrapper(m_Method);
        }
    }

    public delegate bool FilterEntryMethod(ZipEntry e);

    /// <summary>
    /// Zip stream utils
    /// </summary>
    public class ZipUtils
    {
        public static void CopyStream(java.io.InputStream from,
            java.io.OutputStream to)
        {
            sbyte[] buffer = new sbyte[8192];
            int got;
            while ((got = from.read(buffer, 0, buffer.Length)) > 0)
                to.write(buffer, 0, got);
        }

        public static void ExtractZipFile(ZipFile file, string path,
            FilterEntryMethod filter)
        {
            foreach(ZipEntry entry in new EnumerationAdapter(new
                EnumerationMethod(file.entries)))
            {
                if (!entry.isDirectory())
                {
                    if ((filter == null || filter(entry)))
                    {
                        java.io.InputStream s = file.getInputStream(entry);
                        try
                        {
                            string fname =
                              System.IO.Path.GetFileName(entry.getName());
                            string newpath = System.IO.Path.Combine(path,
                              System.IO.Path.GetDirectoryName(entry.getName()));

                            System.IO.Directory.CreateDirectory(newpath);

                            java.io.FileOutputStream dest = new
                              java.io.FileOutputStream(System.IO.Path.Combine(
                              newpath, fname));
                            try
                            {
                                CopyStream(s, dest);
                            }
                            finally
                            {
                                dest.close();
                            }
                        }
                        finally
                        {
                            s.close();
                        }
                    }
                }
            }
        }

        public static ZipFile CreateEmptyZipFile(string fileName)
        {
            new ZipOutputStream(new
              java.io.FileOutputStream(fileName)).close();
            return new ZipFile(fileName);
        }

        public static ZipFile UpdateZipFile(ZipFile file,
            FilterEntryMethod filter, string[] newFiles)
        {
            string prev = file.getName();
            string tmp = System.IO.Path.GetTempFileName();
            ZipOutputStream to = new ZipOutputStream(new
                                 java.io.FileOutputStream(tmp));
            try
            {
                CopyEntries(file, to, filter);
                // add entries here
                if (newFiles != null)
                {
                    foreach(string f in newFiles)
                    {
                        ZipEntry z = new ZipEntry(f.Remove(0,
                          System.IO.Path.GetPathRoot(f).Length));
                        z.setMethod(ZipEntry.DEFLATED);
                        to.putNextEntry(z);
                        try
                        {
                            java.io.FileInputStream s = new
                              java.io.FileInputStream(f);
                            try
                            {
                                CopyStream(s, to);
                            }
                            finally
                            {
                                s.close();
                            }
                        }
                        finally
                        {
                            to.closeEntry();
                        }
                    }
                }
            }
            finally
            {
                to.close();
            }
            file.close();

            // now replace the old file with the new one
            System.IO.File.Copy(tmp, prev, true);
            System.IO.File.Delete(tmp);

            return new ZipFile(prev);
        }

        public static void CopyEntries(ZipFile from, ZipOutputStream to)
        {
            CopyEntries(from, to, null);
        }

        public static void CopyEntries(ZipFile from, ZipOutputStream to,
          FilterEntryMethod filter)
        {
            foreach(ZipEntry entry in new EnumerationAdapter(new
              EnumerationMethod(from.entries)))
            {
                if (filter == null || filter(entry))
                {
                    java.io.InputStream s = from.getInputStream(entry);
                    try
                    {
                        to.putNextEntry(entry);
                        try
                        {
                            CopyStream(s, to);
                        }
                        finally
                        {
                            to.closeEntry();
                        }
                    }
                    finally
                    {
                        s.close();
                    }
                }
            }
        }
    }
}

Also note that there is no support for password-protected files. You could create your own encryption mechanism using the classes in the System.Security.Cryptography namespace. If you do this, be aware that the resulting file will not be compatible with standard Zip utilities such as WinZip.

Creating and Modifying Zip Files

The java.util.zip.ZipOutputStream class allows you to compress data and write the result to an underlying java.io.OutputStream object. The SharpZip application is intended to work with files, so it writes the compressed data to a new java.io.FileOutputStream object, but you could easily derive your own class from java.io.OutputStream or use one of the standard classes to write the compressed data directly to the network or other storage mediums.

The CreateEmptyZipFile helper function creates a Zip file and closes it immediately. The result is an empty Zip file with no entries in it. Appending or deleting items is not so simple because the java.util.zip package does not provide random access to Zip files. For deleting files, you should copy the entries you want to preserve to a new Zip file. For adding files, you should copy all entries to a new Zip file and then append the new entries. Copying an entry involves decompressing the entry from the source file as I've described and compressing it again to the destination file.

Create a new instance of ZipEntry for each of the files that you want to add and call setMethod on the entry to set the compression method to use. The supported methods are ZipEntry.DEFLATED, which compresses data using the deflate algorithm, and ZipEntry.STORED, which stores data without applying any compression. Then call ZipOutputStream.putNextEntry, passing in the new entry, and write its data by calling the write method on the ZipOutputStream object. When you are finished with the current entry, call ZipOutputStream.closeEntry and proceed to the next entry.

The UpdateZipFile function in Figure 5 implements both updating and deleting by calling a delegate for each entry so that you can choose which entries should be copied to a temporary file. Finally, the new entries are added to the Zip file.

Low-level Zip Compression

You can use the java.util.zip classes to compress not only files but your application data as well. To illustrate this, I created a pair of functions to compress and decompress a string using the java.util.zip.Deflater and java.util.zip.Inflater classes.

The compression function creates an instance of the java.util.zip.Deflater class. A parameter in the constructor defines the level of compression that is desired. Next I call the Deflater.setInput class, passing the data to compress as an array of signed bytes (sbyte), and then I call Deflater.finish.

Please note that, in contrast to C#, the byte data type in Java is signed—there is no unsigned byte data type in Java. This is why all methods of the J# runtime that deal with buffers take arrays of sbytes as parameters.

Fortunately, the com.ms.vjsharp.struct namespace includes the JavaStructMarshalHelper class which, among other things, helps you when performing array conversions. The CompressString function calls the convertToByteArray method to convert a string to an array of signed bytes. To get the actual compressed bits I just keep calling Deflater.deflate until Deflater.finished returns true to signal that all the input data has been consumed. I collect the resulting data inside the compression loop using an instance of java.io.ByteArrayOutputStream. As a general rule, it is preferable to use the JDK classes when handling Java types in C#. It's the best way to avoid repeatedly converting arrays between sbyte and byte.

The code for decompressing the string looks very similar to the code used for compression. This time you create an instance of the java.util.zip.Inflater class and call the setInput method, passing in the compressed data. The decompression loop continually calls Inflater.inflate until Inflate.finished becomes true, signaling that all the input data has been decompressed. Finally, call JavaStructMarshalHelper.convertToString to convert the array of unsigned bytes to the string to be returned by the function.

The CsZipLL sample application (LL stands for low level) creates a long string and compresses it down to about half its size. You could use these functions, for example, to code a SOAP extension to reduce the network bandwidth required by your Web Services.

Other J# Goodies

Although this article focuses on handling Zip files, this principle can be applied to other areas where the J# runtime libraries provide functionality not available from the .NET Framework standard assemblies.

Since J# offers developers a migration path to the .NET Framework for their Visual J++® projects, J# also implements many of the Visual J++-specific features, such as J/Direct®. J/Direct is a technology that allows Java language programs to call native Windows code. As is the case in Visual J++, the com.ms.win32 namespace in J# gives you access to most of the Windows API functions, data types, and constants.

The User32, Kernel32, and Gdi32 classes contain the core of the Win32® API functions. The constants are defined as static fields in a number of interfaces named winx, where x is the first letter of the constant. For example, the SW_SHOW flag for the ShowWindow API can be found in the com.ms.win32.wins interface.

For an interface to be CLS-compliant it must not contain fields, and the com.ms.win32.winx interfaces fail this test. Because C# does not allow fields in interfaces, neither IntelliSense® nor the C# compiler can see these constants, but you can still access these fields using Reflection, as shown here:

private int GetWin32IntConstant(string name)
{
    System.Reflection.Assembly asm =
    System.Reflection.Assembly.GetAssembly(typeof(com.ms.win32.wina));
    Type t = asm.GetType("com.ms.win32.win" + char.ToLower(name[0]),
                         true);
    System.Reflection.FieldInfo info = t.GetField(name);
    return int.Parse(info.GetValue(null).ToString());
}

Retrieving Windows API constants using this technique is slow, so you should be careful when using this approach. Another problem is that since constants are not resolved at compile time you get runtime errors every time you misspell them. In any case, having most of the Windows API already declared in a .NET assembly may save you a lot of work. The SharpZip sample program, for example, displays the system icon associated with the extension of each file. To do this, the code calls the SHGetFileInfo API defined in the com.ms.win32.Shell32 interface to obtain the handle of the icon (see Figure 6).

Figure 6 Calling SHGetFileInfo (Close Figure 6)
private Icon IconFromFileType(string path)
{
    Icon Result = null;
    com.ms.win32.SHFILEINFO info = new com.ms.win32.SHFILEINFO();

    int flags = GetWin32IntConstant("SHGFI_ICON") |
                GetWin32IntConstant("SHGFI_USEFILEATTRIBUTES") |
                GetWin32IntConstant("SHGFI_SMALLICON");
    if (com.ms.win32.Shell32.SHGetFileInfo(path, 0, info,
        1024 /* struct size */, flags) != 0)
    {
        // the Icon does not own the handle...
        Icon tmp = Icon.FromHandle(new IntPtr(info.hIcon));
        try
        {
            // ...that's why we copy it
            Result = new Icon(tmp, tmp.Size);
            tmp.Dispose();
        }
        finally
        {
            com.ms.win32.User32.DestroyIcon(info.hIcon);
        }
    }
    return Result;
}

Note that when you create a System.Drawing.Icon object from a handle, the new Icon does not own the handle. This means that you must free the associated resources by calling the DestroyIcon API. Since I didn't want to store the icon handle for the lifetime of the Icon object, I chose to make a copy of the object.Icons created by using the copy constructor on their handle.

Although the com.ms.win32 namespace is huge, you should be aware that it does not include every single Windows API function and data structure. For example, one notable omission from the com.ms.win32.Shell32 interface is the SHBrowseForFolder API, which would have allowed us to display the "Browse for Folder" dialog box without using the Microsoft Shell Controls and Automation COM library.

Also note that handling callbacks is somewhat complicated due to the fact that the Java language does not support delegates. For every callback type, an abstract class that defines the function prototype is provided. You must derive from this class to implement the code that handles the callback and then pass the API call an instance of this class (see Figure 7). Another minor difficulty related to the Java language is that parameters passed by reference are declared as arrays, but this only affects only the code that calls the functions, not the underlying functionality.

Figure 7 Passing Callbacks to the Win32 API (Close Figure 7)
using System;
using com.ms.win32;

namespace CsWinApi
{
    class MyEnumProc : WNDENUMPROC
    {
        public override bool wndenumproc(int hwnd, int lParam)
        {
            java.lang.StringBuffer str = new java.lang.StringBuffer(255);
            User32.GetWindowText(hwnd, str, str.capacity());
            string txt = str.ToString();
            if (txt != string.Empty)
                Console.WriteLine(txt);
            return true;
        }
    }

    class AppMain
    {
        [STAThread]
        static void Main(string[] args)
        {
            User32.EnumWindows(new MyEnumProc(), 0);
            Console.ReadLine();
        }
    }
}

Finally, some API calls are poorly translated. One example is waveOutOpen (defined in the Winmm class). The dwCallback parameter is used in C++ to pass an event handle, a window handle, a thread ID, or a callback function, depending on the value of the fdwOpen parameter. Since the J/Direct wrapper declares the dwCallback parameter as Int32 and there is no way to typecast a callback (delegate) to Int32, you must use another notification mechanism, such as event handle, window handle, or thread id.

There are some other interesting things in the core J# package. For example, the java.math.BigDecimal and java.math.BigIntegers classes allow you to manipulate arbitrarily large numbers, which could be extremely helpful if you are writing applications that deal with cryptographic algorithms or scientific calculations.

The CsMath sample project shows how to use java.math.BigDecimal to calculate Pi with an arbitrary number of digits after the decimal point using Machin's formula. To make the code more readable I wrapped the java.math.BigDecimal in my own BigDecimal class and defined the most commonly used operators.

Application Deployment

Applications that use this technique require both the J# runtime and the .NET Framework on the target computer. Just as with the .NET Framework, Microsoft provides a redistributable package that you can deploy along with your application setup.

Microsoft has indicated continued support for J# for desktop operating systems. However, there is currently no support for the .NET Compact Framework in J#, so you cannot apply the techniques described in this article to applications targeting Smart Devices. Copying the assemblies to your local project directory will not work because the J# runtime assemblies rely heavily on native calls. You can, however, fully exploit the J# runtime for Web applications that use Mobile Web Controls.

Conclusion

The J# runtime includes many useful classes that you can use from other languages in the .NET Framework. Some of these classes allow you to handle Zip files, perform high-precision mathematical calculations, or call the Windows API. Although most of this functionality can be achieved by using third-party libraries, the J# runtime is fully supported by Microsoft, and it is free!

For related articles see:
Java 911: Parlez-vous J/Direct?

For background information see:
http://msdn.microsoft.com/vjsharp/
What is the Common Language Specification?

Ianier Munozis a software architect and analyst at Dokumenta, a consulting firm based in Luxembourg. He is also the author of Chronotron and other popular software. You can reach him at http://www.chronotron.com.


.NET Remoting: Secure Your .NET Remoting Traffic by Writing an Asymmetric Encryption Channel Sink

New information has been added to this article since publication.
Refer to the Editor's Update below.

.NET Remoting

Secure Your .NET Remoting Traffic by Writing an Asymmetric Encryption Channel Sink

Stephen Toub
This article assumes you're familiar with C# and .NET Remoting
Level of Difficulty 1 2 3
Code download available at: NETRemoting.exe (150KB)
SUMMARY

As .NET Remoting gains popularity in the enterprise space, it must meet business demands for trustworthy computing. Remoting traffic can be secured when objects are hosted in IIS, but when they aren't hosted in IIS, custom security solutions can be developed to secure them. This article provides an in-depth look at writing channel sinks for .NET. It also details the flow of data through custom channel sinks and explains the kinds of manipulations that can be performed on that data.

Contents
Support for HTTPS
Remoting Execution and Extensibility
One Sink or Two?
Designing the Protocol
Implementing the Key Exchange
Asynchronous Processing
Server Cleanup
Creating the Sinks
Configuration
Conclusion

R
emoting is the .NET solution to distributed computing. As a successor to DCOM, .NET Remoting fills many of the gaps left by its predecessor and at the same time adds an extraordinary amount of well-conceived functionality. As with any product, however, the initial release rarely contains the entire feature set desired by its users, and in fact user needs may continue to evolve ahead of product releases throughout the life of the product. To this end, .NET Remoting has been built from the ground up to be extensible in the sense that functionality can be added with relative ease by those who need it. One such feature is the ability to secure and encrypt remoting traffic.

With the popularity of remoting in the enterprise space on the rise, there are questions concerning the capabilities of the Microsoft® .NET Framework to encrypt remoting traffic. The most common question I've seen in this arena is whether .NET Remoting supports secure sockets layer (SSL). Unfortunately, the answer is not a simple yes or no.

Support for HTTPS

In conversations about remoting, the word "channel" is used frequently. Its definition can vary, but in general, a channel is the series of sinks through which a remoting call is made. The HTTP channel for .NET is actually a combination of two channels, one used by the client and one used by the server. Each has its own set of sinks, referred to as a "sink chain." The client sink chain ends with the HttpClientTransportSink, which uses the HttpWebRequest and HttpWebReponse classes from the System.Net namespace to handle all HTTP traffic. This brings with it a bonus—since HttpWebRequest and HttpWebResponse support HTTPS, so too does the HTTP client channel. This means that if the server channel supported SSL, you'd be all set.

Just as the client transport sink is an HTTP client, the HttpServerTransportSink is nothing more than a specialized Web server. You can see this by setting up a simple remoting host and connecting to it with a Web browser.

At a bare minimum, a remoting server that can serve up an object requires the registration of a channel and the publishing of an object or class type in some fashion (well-known, activated, and so on). This is done by using classes from the System.Runtime.Remoting namespace:

static void Main(string[] args)
{
    ChannelServices.RegisterChannel(new HttpChannel(8124));
    RemotingConfiguration.RegisterWellKnownServiceType(
        typeof(Person), "Person.soap", WellKnownObjectMode.Singleton);
    Console.WriteLine ("Press enter to stop the server...");
    Console.ReadLine();
}

To show you that this application is indeed a Web server, I'll use Microsoft Internet Explorer to connect to the designated endpoint on the running server in order to request the Web Services Description Language (WSDL) document for the remoted type (as you can do with a Web Service hosted in ASP.NET). The resulting document is shown in Figure 1.

Figure 1 The WSDL
Figure 1 The WSDL

I connect to the remoting server and make an HTTP request for "/Person.soap?wsdl" in order to retrieve the WSDL for Person, a custom class that derives from MarshalByRefObject. This request is handled on the server by the SdlChannelSink, a member of the server channel sink chain, which generates the appropriate WSDL for the Person class. This description is sent back to the client as an HTTP response:

C:\telnet localhost 8124
GET /Person.soap?wsdl HTTP/1.0

HTTP/1.1 200 OK
Content-Type: text/xml
Server: MS .NET Remoting, MS .NET CLR 1.0.3705.288
Content-Length: 8683
Connection: Close

<?xml version='1.0' encoding='UTF-8'?>
<definitions name='Person'
 targetNamespace='http://schemas.microsoft.com/clr/nsa ...

So now that you know that HttpServerTransportSink is a Web server, the question remains—can you connect to the server using HTTPS? Unfortunately, you can't. The HttpServerTransportSink does not support SSL. However, custom Windows services and other applications like the one I just showed are not the only vehicles for hosting remoted objects. Microsoft Internet Information Services (IIS) in conjunction with ASP.NET can also remote objects, and when that happens, the framework benefits from the support of HTTPS in IIS. An ASP.NET-hosted object can be accessed with a URL such as https://localhost:8124/MyApplication/Person.soap, forcing all sensitive traffic to be encrypted.

Unfortunately, using IIS to host objects will limit you to using HTTP. So if you're using a different transport, or if you're not hosting the objects in IIS, how can you ensure that prying eyes are unable to read your data? One of the beauties of the .NET Remoting framework is its extensibility—the number of ways that you can extend and modify its functionality.

Remoting Execution and Extensibility

When a client requests a reference to a remote object, it is returned a reference to a TransparentProxy—an object that appears to have the same methods and properties as the remote object and that appears to behave in a similar fashion, but which begins the process of delegation when a call is placed.

A call made on the TransparentProxy is passed as a MessageData object into another proxy, an object of a type derived from RealProxy (often, but not necessarily, a RemotingProxy). This proxy in turn creates an IMessage from the MessageData and forwards the message containing information on the method call through a series of message sinks (classes that implement IMessageSink or IDynamicMessageSink) which can all modify or substitute for the message being sent.

The final IMessageSink in the chain should be the formatter sink, which will often implement IClientFormatterSink, a combination of IClientChannelSink and IMessageSink. This sink is responsible for creating transport headers as well as for taking the IMessage and serializing it into a stream. This stream and headers collection, along with the IMessage, which is used only for reference, are then passed through a series of IClientChannelSinks, each of which can substitute a new stream of data to replace the one supplied by the previous sink (at this stage of processing, the IMessage has already been serialized into the stream and thus only serves an informational purpose). The last of these sinks, the transport sink, takes the headers and the stream and uses a means of transport, such as an HTTP connection or a named pipe, to send the data along to the server.

A similar process happens on the server side, but in reverse. The server's transport sink receives both the headers and the stream from the client's transport sink. This data is processed by a series of IServerChannelSink objects (including the SdlChannelSink, which generates the WSDL you saw earlier) just like it was in the client. The formatter, also an IServerChannelSink, takes the headers collection and the stream and deserializes them into a clone of the original IMessage, which is then forwarded through a series of IServerChannelSink, IMessageSink, and IDynamicMessageSink objects until it reaches the final sink, the StackBuilderSink. This sink uses reflection to execute the request and to obtain a return message. The return message works its way back through the sinks, across the transport layer, and back through the client-side sinks until it reaches the proxies where the return value and the out parameters are set.

As you can see, a lot goes on behind the scenes when a remoted object is used. One benefit of this, as shown in Figure 2, is that there are many places to plug in custom functionality. Formatters can be written to serialize and deserialize the IMessage in a special format (if the built-in SOAP and Binary formatters do not suit one's needs). Additional IMessageSinks can be written to modify the actual message being sent. IClientChannelSinks and IServerChannelSinks can be written to modify the streams and the headers collection. Custom RealProxies can be written to trace or modify which methods actually get executed. And Transport sinks can be written to send and receive the data using any transport layer the developer deems appropriate.

Figure 2 Remoting Flow and Interception Points (Close Figure 2)

For encryption sinks, you need to modify the data being transferred between client and server. You don't need any special transport or formatter, nor do you need to modify the message before it is serialized. The best place to plug in (and the most common interception point I've seen used) is between the formatter and the transport sink. This can be accomplished by implementing custom IClientChannelSink and IServerChannelSink classes to handle the encryption of the streams passing through them.

One Sink or Two?

Sometimes it is necessary to create both a client sink and a server sink. When to do this depends on the application as there is no rule stating that all sinks must come in pairs. For applications that require work to be done on both sides, such as an encryption or compression channel, a sink pair is, of course, required. But there are many situations where only one sink is necessary. For example, ASP.NET has a great output caching system whereby the result of the execution of a page is cached. This cached result can be served up in the future when that page is requested rather than having to execute the page again. The remoting framework does not ship with any such caching system built into it, but a server-side sink could be written to perform this processing. The sink could examine the IMessage object passed to it, look in its own cache object to determine whether a previously stored execution exists, and either forward the request up the chain if one doesn't, or immediately return a ReturnMessage based on what is in the cache. The client doesn't need to know that caching is taking place and therefore a client sink isn't necessary (of course, one could also implement a client-side cache, in which case only the client sink would be necessary). A sink that does just this task is available with the code download for this article at the link at the top of this article. The SdlChannelSink mentioned earlier also executes exclusively on the server side.

It's important to note from this example that a sink isn't required to forward anything up the sink chain. While most sinks do so in most cases, the decision is up to the sink. As such, a sink can determine that a message or stream should not be forwarded along the chain, that it should create its own response stream and headers, and immediately send those back down the chain without the message ever reaching the actual destination. For a caching sink, this makes it possible to serve up information from a cache rather than always forwarding on to the next sink. On the client side, a sink can choose to create its own stream and headers and send those to the server instead of forwarding on what was supplied to it by the previous sink. This enables a sink pair to send multiple messages back and forth without those messages even making it out of the chain. Sounds like it would be useful for a security handshake, doesn't it?

Designing the Protocol

Developing a symmetric encryption channel to use 3DES, RC2, or any of the other symmetric encryption algorithms provided in the System.Security.Cryptography namespace, requires two sinks. One will be placed on each side between the formatter and the transport; because you want to work on the serialized stream, you need the sink to come after the formatter on the client and before the formatter on the server. Figure 3 shows the various important parts of these sinks.

Figure 3 Client and Server Sinks (Close Figure 3)
// IClientChannelSink.ProcessMessage
public void ProcessMessage(IMessage msg,
    ITransportHeaders requestHeaders, Stream requestStream,
    out ITransportHeaders responseHeaders, out Stream responseStream)
{
    // Encrypt the stream
    requestStream = CryptoHelper.Encrypt(requestStream, _provider);

    // Send the stream to the next sink in the chain
    _next.ProcessMessage(msg,
        requestHeaders, requestStream,
        out responseHeaders, out responseStream);

    // Decrypt the response stream
    responseStream = CryptoHelper.Decrypt(responseStream, _provider);
}

// IServerChannelSink.ProcessMessage
public ServerProcessing ProcessMessage(
    IServerChannelSinkStack sinkStack, IMessage requestMsg,
    ITransportHeaders requestHeaders, Stream requestStream,
    out IMessage responseMsg, out ITransportHeaders responseHeaders,
    out Stream responseStream)
{
    // Decrypt the incoming request stream
    requestStream = CryptoHelper.Decrypt(requestStream, _provider);

    // Forward on the request
    ServerProcessing result = _next.ProcessMessage(
        sinkStack, requestMsg, requestHeaders, requestStream,
        out responseMsg, out responseHeaders, out responseStream);

    // Encrypt the response stream
    responseStream = CryptoHelper.Encrypt(responseStream, _provider);

    return result;
}

Because this is symmetric encryption, both sinks must share the same key and initialization vector (IV) information (used to populate the SymmetricAlgorithm provider object passed to the CryptoHelper functions). Either both sinks must be provided with this key information up front, or one of them must be able to dynamically generate the key information and securely send it to the other sink.

Algorithms such as RSA, ElGamal, and Rabin are asymmetric encryption algorithms—also known as public key encryption. These algorithms are based on the concept of key pairs, a set of keys mathematically related such that a message encrypted with one key can only be decrypted by the other. Anyone who wants the ability to receive and decrypt encrypted messages creates a key pair. One key, the public key, is distributed to the world. The private key, on the other hand, should be carefully guarded. Anyone can use the public key to encrypt a message, and this encrypted message can only be decrypted by the private key.

Figure 4 Key Exchange Algorithm
Figure 4 Key Exchange Algorithm

Using asymmetric encryption, the two sinks can solve their key exchange predicament using the process shown in Figure 4. The client sink dynamically generates a key pair and sends the public key to the server sink. The server sink then dynamically creates a symmetric key, encrypts it with the client's public key, and sends the encrypted key back to the client. The client sink can now use his private key to decrypt the symmetric key; almost magically, both sinks now have the same symmetric key and can proceed to share information. Those familiar with this process will note that it is very similar to how a browser establishes a secure connection to a Web server using SSL.

Implementing the Key Exchange

As noted earlier, formatters take an IMessage and generate two output objects, an ITransportHeaders and a Stream. The stream contains a serialized version of the IMessage while the ITransportHeaders object contains out-of-band headers (name/value pairs) to be sent to the server by the transport sink. Initially, this collection is populated with formatter-supplied headers, such as a SOAPAction. The use of this collection, however, is in no way limited to the formatter. Any of the sinks between the formatter and the transport can add headers into this collection and they will all be dealt with similarly by the transport sink. This is a great place to put handshake information that the client and server will need for communication, such as handshake state, encryption keys, and a connection identifier.

You have to keep in mind when implementing the solution that both the client and the server must be thread-safe. When remote requests are made on an object on multiple threads, those requests are not serialized in any fashion before reaching your sink (unless a previous sink has decided to synchronize access). You need to be mindful of this fact when storing state information. In addition, one server can be used by many clients. Since the client and server will need to store a shared encryption key (and since it would not be secure for the server to distribute the same shared key to all of its clients), the server will need to maintain a list of all clients and their associated information. Both the client and the server will also need to store a unique identifier so that the server can look up the client's key.

Implementation on the client is now fairly clear cut (see Figure 5). ProcessEncryptedMessage is called by ProcessMessage to handle most of the work. When ProcessMessage is called, the client must check to see if it has already received a key from the server. If it has, it can encrypt the stream containing the serialized message and forward it to the server, decrypting the response it gets back. The client must also be mindful of the fact that the server may have deleted its connection information (or possibly the server may have been restarted, causing it to lose all of its connection information), and as such the client must be ready to try again should the transaction fail. If the client has not received a shared key from the server, or if the server sends back a message informing the client that it was an unknown entity, the client must initiate a handshake with the server. As I've outlined, the client will generate a unique identifier for itself as well as an RSA key pair, and it will send the identifier and the public key to the server. When it receives the encrypted shared key back from the server, it can use its private key to retrieve the shared key, store it for later use, and continue by encrypting the original message and sending it to the server.

Figure 5 Client Sink's ProcessEncryptedMessage (Close Figure 5)
private bool ProcessEncryptedMessage(
    IMessage msg, ITransportHeaders requestHeaders, Stream requestStream,
    out ITransportHeaders responseHeaders, out Stream responseStream)
{
    // Encrypt the message.
    Guid id;
    lock(_transactionLock)
    {
        id = EnsureIDAndProvider(msg, requestHeaders);
        requestStream =
            SetupEncryptedMessage(requestHeaders, requestStream);
    }

    // Send the encrypted request to the server
    _next.ProcessMessage(
        msg, requestHeaders, requestStream,
        out responseHeaders, out responseStream);

    // Decrypt the response.
    lock(_transactionLock)
    {
        responseStream =
            DecryptResponse(responseStream, responseHeaders);
        if (responseStream == null &&
            id.Equals(_transactID)) ClearSharedKey();
    }

    // Return whether we were successful
    return responseStream != null;
}

The server requires a connectionless implementation. It must be ready to receive requests in various handshake states from any number of clients. As shown in Figure 6, when a request comes in, the server will look up the ID and handshake state in the transport headers. Based on these two parameters it can choose the proper course of action. If the client is requesting a shared key, the server will create one and send it back to the client encrypted with the client's public key (also making sure to store the key for later use). If the client is requesting the processing of an encrypted message, the server will attempt to locate the client's key, and if found, use it to decrypt the request, forwarding it along the sink chain, encrypting the received result, and sending it back to the client. If the key wasn't found, it must alert the client to this fact. An example of this initial conversation is diagrammed in Figure 7.

Figure 6 Server Sink Process Message (Close Figure 6)
public ServerProcessing ProcessMessage(
    IServerChannelSinkStack sinkStack, IMessage requestMsg,
    ITransportHeaders requestHeaders, Stream requestStream,
    out IMessage responseMsg, out ITransportHeaders responseHeaders,
    out Stream responseStream)
{
    // Get header information about transaction
    string strTransactID = (string)requestHeaders[CommonHeaders.ID];
    Guid transactID = (strTransactID == null) ?
        Guid.Empty : new Guid(strTransactID);
    SecureTransaction transactType = (SecureTransaction)Convert.ToInt32(
        (string)requestHeaders[CommonHeaders.Transaction]);

    // For reference, find out who is connecting to us.
    IPAddress clientAddress =
        requestHeaders[CommonTransportKeys.IPAddress] as IPAddress;

    // Push this sink onto the sink stack
    sinkStack.Push(this, null);

    // Process the transaction based on its type
    ServerProcessing processingResult;
    switch(transactType)
    {
        // Client is requesting a shared key
        case SecureTransaction.SendingPublicKey:
            processingResult = MakeSharedKey(
                transactID, requestHeaders, out responseMsg,
                out responseHeaders, out responseStream);
            break;

        // Client is sending us an encrypted message
        case SecureTransaction.SendingEncryptedMessage:
            if (PreviousTransactionWithClient(transactID))
            {
                processingResult = ProcessEncryptedMessage(
                    transactID, sinkStack, requestMsg,
                    requestHeaders, requestStream,
                    out responseMsg, out responseHeaders,
                    out responseStream);
             }
             else
             {
                 processingResult = SendEmptyToClient(
                    SecureTransaction.UnknownIdentifier,
                    out responseMsg, out responseHeaders,
                    out responseStream);
             }
             break;

        // Client is sending us a plaintext message
        case SecureTransaction.Uninitialized:
            if (!RequireSecurity(clientAddress))
            {
                processingResult = _next.ProcessMessage(
                    sinkStack, requestMsg, requestHeaders, requestStream,
                    out responseMsg, out responseHeaders,
                    out responseStream);
            }
            else throw new SecureRemotingException("Security required.");
            break;

         // Uh oh.
         default:
             throw new SecureRemotingException("Invalid request.");
    }

    // Take us off the stack and return the result.
    sinkStack.Pop(this);
    return processingResult;
}
Figure 7 Client/Server Conversation
Figure 7 Client/Server Conversation

Your well-behaved server should also be conscious of the possibility that not all clients will be sending encrypted messages. If it detects that the necessary transport headers do not exist (suggesting that the client application is not using the client sink), it can pass the data on to the next sink without any modifications. Even though this can be done, you don't necessarily want to do so. What if you wanted to enforce a requirement that connecting clients must be encrypting traffic, or what if you only wanted to enforce it for some clients? Both the HttpServerTransportSink and the TcpServerTransportSink help you out here by providing the IP address of the connected client. You can grab the IP address from the transport headers, as shown here:

IPAddress clientAddress =
requestHeaders[CommonTransportKeys.IPAddress] as IPAddress;
Then you can use this address to decide whether you are going to accept unencrypted traffic:
case SecureTransaction.Uninitialized:
    if (!RequireSecurity(clientAddress))
    {
        processingResult = _next.ProcessMessage(
        sinkStack, requestMsg, requestHeaders, requestStream,
        out responseMsg, out responseHeaders, out responseStream);
    }
else throw new SecureRemotingException("Security required.");
    break;

Asynchronous Processing

In version 1.0 of the .NET Framework, there is no server support for asynchronous processing in custom channel sinks (even though there is the conspicuously named method AsyncProcessResponse on the IServerChannelSink interface). All requests to your server sink will be synchronous, even if the client makes an asynchronous request; therefore, you don't need to add any functionality to your server sink to support asynchronous requests.

Implementing async handling on the client is not quite as simple. Client-side, ProcessMessage is only used for synchronous requests. For one-way and asynchronous calls, the processing logic is divided between two of the interface methods on IClientChannelSink: AsyncProcessRequest and AsyncProcessResponse. AsyncProcessRequest provides the sink an opportunity to modify the stream and headers being sent to the server, while AsyncProcessResponse allows the modification of the stream and headers being sent from the server.

In my implementation (see Figure 8) AsyncProcessRequest can't always be entirely asynchronous. If no connection information has been established with the server, AsyncProcessRequest calls EnsureIDAndProvider to ensure that this information is retrieved. To do so it must make a synchronous round-trip to the server just as ProcessMessage does. With any luck, this is only done once and the rest of the requests made through AsyncProcessRequest will all be asynchronous. Unlike with synchronous processing, here you can't make an additional asynchronous request should the initial one fail. Since AsyncProcessResponse must return the output headers and stream, and as you can't remake an asynchronous request, the second request must be synchronous to try again. To accomplish this, AsyncProcessRequest pushes the arguments supplied to it onto the sink stack as the state that will be given to AsyncProcessResponse. If AsyncProcessResponse gets back an invalid response from the server, it can use this state data to make a synchronous attempt using ProcessMessage.

Figure 8 Client's AsyncProcessRequest (Close Figure 8)
public void AsyncProcessRequest(
    IClientChannelSinkStack sinkStack, IMessage msg,
    ITransportHeaders headers, Stream stream)
{
    AsyncProcessingState state = null;
    Stream encryptedStream = null;
    Guid id;

    lock(_transactionLock)
    {
        // Establish connection information with the server
        // (hopefully only done on first request)
        id = EnsureIDAndProvider(msg, headers);

        // Saves state information just in case something goes wrong
        state = new AsyncProcessingState(msg, headers, ref stream, id);

        // Encrypting the stream
        encryptedStream = SetupEncryptedMessage(headers, stream);
    }

    // Push ourselves onto the stack with the necessary state and
    // forward on to the next sink
    sinkStack.Push(this, state);
    _next.AsyncProcessRequest(sinkStack, msg, headers, encryptedStream);
}

I have purposely limited my discussion of one-way methods because they are not supported by this sink pair. Due to their very nature, the client sink never receives a response for a one-way method and therefore the required round-trip to the server to establish a shared key is not possible without implementing a good deal of workaround logic.

Server Cleanup

The server needs to maintain connection information for all clients that connect to it, otherwise it would be unable to decrypt anything sent by the client because the server would lack the shared key necessary to do so. If many clients are connecting to this server (clients which may or may not stay active for a prolonged period of time), the server's connection table could fill with outdated information. Thus the server should clean this table periodically. In my implementation I've chosen to record the last access time of the client in addition to the key and use this timestamp to determine if the client's last access time is older than some preset threshold. Figure 9 shows the code that periodically runs through every client connection and, if it is out of date, removes it from the table. [ Editor's Update - 12/8/2004: The code in Figure 9 has been fixed to correctly start the timer.] If a client whose information was removed later attempts to send an encrypted message, the server will send a response indicating that it has no information on the client's identity. The client and server will then proceed with a new handshake, starting the whole process over again.

Figure 9 Server's Connection Sweeper (Close Figure 9)
private void StartConnectionSweeper()
  {
  // Create and start a timer that will execute SweepConnections
  if (_sweepTimer == null)
  {
    _sweepTimer = new System.Timers.Timer(_sweepFrequency*1000);
    _sweepTimer.Elapsed += new ElapsedEventHandler(SweepConnections);
    _sweepTimer.Start();
    }
  }

private void SweepConnections(object sender, ElapsedEventArgs e)
{
  lock (_connections.SyncRoot)
  {
    ArrayList toDelete = new ArrayList(_connections.Count);

    // Find all entries that need to be deleted
    foreach(DictionaryEntry entry in _connections)
    {
      ClientConnectionInfo cci = (ClientConnectionInfo)entry.Value;
      if (cci.LastUsed.AddSeconds(_connectionAgeLimit).CompareTo(DateTime.UtcNow)
        < 0)
      {
        toDelete.Add(entry.Key);
        ((IDisposable)cci).Dispose();
      }
    }

          // Delete the out-of-date entries found above
          foreach(Object obj in toDelete) _connections.Remove(obj);
          toDelete = null;
    }
}

This same process could be used to provide another level of security. In addition to the last access time, the server could record the number of accesses by the client or the amount of data that had been transferred. When these counts hit a preset limit, the server could perform the same cleanup operation and delete the client's connection information. The client could do the same, deleting the shared key and proceeding as if it had never received one. This would force the client and server to renegotiate a new shared key, similar to the process used by IPSec's automatic key renewal.

Creating the Sinks

Now that I've written the sinks, I need to help the framework instantiate them. The .NET Remoting framework does not create the sinks directly. Instead, it creates "sink providers" that are then responsible for creating the actual sinks. Providers and sinks almost always form a one-to-one relationship, meaning that a single provider is used to create a single sink; however, there is nothing preventing a provider from creating more than one sink.

The biggest benefit of having the job of sink instantiation relegated to the providers is that it adds another level of indirection over which the developer of the sink has a lot of control. If the provider implements the correct constructor, the framework will pass to the provider an IDictionary containing attributes and elements that have been specified for the provider in the remoting configuration file (this dictionary can also be provided programmatically if a remoting file is not used). The provider can then take this information and use it to create a sink which can have any constructor the developer desires. The only requirement is that the provider must inform the sink of the next sink in the chain in order for it to be able to forward processing requests.

Figure 10 Using Sink Providers to Instantiate Sinks
Figure 10 Using Sink Providers to Instantiate Sinks

When the providers are created by the framework, they themselves are created in a chain. The IClientChannelSinkProvider and IServerChannelSinkProvider interfaces both require the implementation of a Next property which is used by the framework to specify the next provider. The framework instantiates the providers and calls the first provider's CreateSink method. The CreateSink method is responsible for creating the actual sink and for returning a reference to it (see Figure 10). As the sink must be informed of the next sink in the chain, the CreateSink method must forward the call to the next provider in the chain, which will then return a reference to its sink:

   public IServerChannelSink
      CreateSink(IChannelReceiver channel)
   {
       IServerChannelSink nextSink = null;
       if (_next != null)
       {
   if ((nextSink = _next.CreateSink(channel))
          == null) return null;
       }
       return new SecureServerChannelSink(nextSink, ... );
}

The sink returned by the first provider's CreateSink now begins the complete linked list of sinks.

Configuration

Configuration of an HTTP channel can be done through a configuration file as follows:

<channels>
    <channel ref="http" port="8124" />
</channels>
This directs the framework to use the HTTP channel and its default formatter, the SOAP formatter. To override the default, you can be explicit about which formatter you want to use:
<channels>
    <channel ref="http" port="8124">
        <serverProviders>
<formatter ref="binary" />
        </serverProviders>
    </channel>
</channels>

When the <serverProviders/> tag (or <clientProviders/> on the client) is used as an element of <channel>, it tells the framework that instead of using the default sink chain, the channel should be set up to use the providers specified in this list and in the order that they're listed. In this case, you're telling the framework to use the binary formatter. To add more providers, you only need to add additional elements to the server provider list:

<serverProviders>
    <provider type=
     "MsdnMag.Remoting.SecureServerChannelSinkProvider,SecureChannel" />
    <formatter ref="binary" />
</serverProviders>

Notice that I've added the provider before the formatter as I need to be able to modify the stream before it hits the formatter. In the client configuration file, the provider must come after the formatter for the opposite reason.

As I've created the provider with the constructor that allows it to take an IDictionary of parameters, I can add attributes to the providers tag that will modify the behavior of the provider and sink, by dictating things such as the encryption algorithm to use and how frequently the connection sweeper should run.

When I use RemotingConfiguration.Configure on the client and on the server with the appropriate configuration files, the custom sinks will be part of the sink chains.

Conclusion

The channel developed in this article isn't a perfect solution. The sinks take out some pretty big locks in order to achieve thread safety. There is no authentication or message signing. There are no attempts made to limit man-in-the-middle or denial of service attacks. But I hope the point is still clear. This article provides a glimpse into the types of solutions that are possible with .NET Remoting. While many security channels could benefit from these kinds of techniques, the possibilities for productivity and creativity are limitless: transport sinks to support named pipes or MSMQ, IMessageSinks to cache return values, compression channels, logging channels, notification channels—the list goes on. I imagine that in the near future the remoting community will start to see a wide variety of Microsoft and third-party remoting extensions to fill some of these voids, but that certainly doesn't prevent you from starting work today on your dream sink.

For related articles see:
.NET Remoting Security Solution, Part 2: Microsoft.Samples.Runtime.Remoting.Security Assembly
Advanced Remoting
Advanced .NET Remoting (C# Edition), by Ingo Rammer (APress, 2002)
Microsoft .NET Remoting, by Scott McLean, et al. (Microsoft Press, 2002)

For background information see:
.NET Remoting: Design and Develop Seamless Distributed Applications for the Common Language Runtime
An Introduction to Microsoft .NET Remoting Framework
Microsoft .NET Remoting: A Technical Overview

Stephen Toubis a developer and a consultant with Microsoft in New York. He is a graduate of and a former Teaching Fellow at Harvard University. Reach Steve at stoub@microsoft.com.


Smart Cleanup: Achieve More Reliable Resource Management with Our Custom C++ Classes

Smart Cleanup

Achieve More Reliable Resource Management with Our Custom C++ Classes

Eric Niebler
This article assumes you're familiar with C++ Templates and Managed C++
Level of Difficulty 1 2 3
Code download available at: ResourceManagement.exe (196KB)
SUMMARY

Managing resources in C++ is not easy. When you're unsuccessful, your app can leak all kinds of resources including file system handles, database connections, and, of course, memory. Even in garbage-collected languages like Managed C++, resource management is difficult because garbage collection only deals with memory management, not the other resources that cause performance problems.

In this article, the author describes the SmartAny template library he created, which uses a policy-based approach to dynamic resource management. Readers will learn how to use the SmartAny classes and policies to ensure the proper cleanup of their resources, be they files, OS handles, or native and unmanaged objects.

Contents
Overview
Sample Usage
Class Prototypes
The auto_any Class
The shared_any Class
The scoped_any Class
Managed Code
Conclusion

W
hen I was very young, my mother used to joke that she could tell what path I had taken through the house by the debris I left in my wake. She referred to my carelessly discarded items as "The Trail." After many years of coaching, she taught me to clean up after myself and to put things in their proper place when I was done with them. It's good that I learned this lesson early, because these days my career as a C++ programmer depends on my ability not to leave a trail.

Resource management in C++ is hard. If you forget to clean up after yourself, your application will hemorrhage memory, file system handles, and database connections, if it runs at all. Add exceptions to the mix and it can be very hard indeed to avoid leaving a trail. Garbage-collected languages, all the rage these days, promise to sweep all this mess away, but they can leave an untidy trail behind them, too. Garbage collection (GC) helps with memory management, but memory is only one type of resource that needs attention. GC does nothing to help with windows, handles, critical sections, and the like. Quite the contrary, running finalizers nondeterministically on a separate thread can actually make it harder to clean up these resources correctly.

To automate my housekeeping chores, I developed a generic resource management library which I call SmartAny. The SmartAny library takes a back-to-basics approach to resource management. With SmartAny, I can bind a "cleanup" action to any arbitrary resource and have that cleanup action execute deterministically and synchronously when the resource is no longer needed. SmartAny is useful in both native and managed code. In managed code, it complements and integrates seamlessly with the garbage collector so that I can have deterministic finalization when I need it and let the garbage collector handle the rest. With SmartAny, cleaning up after myself is a snap.

Overview

Most people know about smart pointers like std::auto_ptr. The idea behind them is simple—a wrapper object "owns" the pointer and calls delete on it in the destructor. This way, the memory is guaranteed to be cleaned up. Come hell or high water, return statements, or exceptions, the memory will be freed.

The SmartAny library extends this simple idea to any resource type and to any cleanup action. It is a generic and extensible resource management framework. With the classes in SmartAny, you can manage file handles the same way you manage sockets, the same way you manage...—well, you get the idea.

SmartAny contains three helper classes: auto_any, shared_any, and scoped_any. The auto_any class is analogous to std::auto_ptr. It gives your resources strict ownership semantics; that is, only one auto_any can own a resource at a time, and copying an auto_any implies transfer of ownership. The shared_any class is analogous to boost::shared_ptr. It is a reference-counted resource, which gets cleaned up when the reference count drops to zero. It is appropriate for use in standard containers. The scoped_any class is analogous to boost::scoped_ptr. It is like auto_any, except that copy and assignment are disallowed, making it harder to misuse.

These helper classes work seamlessly with both managed and unmanaged types. For managed types, you can choose a policy that calls the destructor deterministically or one that calls the Dispose method instead. You can even write your own policy that does both. Since the SmartAny usage is the same for managed and unmanaged types, it makes it much easier to write and read mixed code. As you read this article, you may want to take a look at http://www.boost.org and some of the libraries provided there. I found the site very useful and can thank the authors for the design of SmartAny, which was strongly influenced by the smart pointers there.

Sample Usage

Enough theory; let's see some code. The code in Figure 1 uses the SmartAny library to manage the lifetime of a Win32® file handle. Notice the type auto_hfile. It is a typedef provided in auto_any.h. An auto_hfile is an auto_any that wraps a HANDLE, uses the close_handle policy, and has invalid_handle_t as its invalid state. I will discuss each of these individually.

Figure 1 Managing Handle Lifetime (Close Figure 1)
#include <windows.h>
#include "auto_any.h"

// return an auto_hfile by value
auto_hfile open_file( char const * szFileName )
{
    // auto_hfile is a typedef for
    // auto_any<HANDLE,close_handle,invalid_handle_t>
    return auto_hfile( CreateFile( szFileName, /*...*/ ) );
}

int main()
{
    auto_hfile hFile = open_file( "my_file.txt" );
    if ( hFile )
    {
        // File open succeeded, write something to it
        WriteFile( get( hFile ), /*...*/ );
    }
    // File is closed automatically when hFile is destructed
    return 0;
}

Now note that open_file returns an auto_hfile by value. That's perfectly OK. The ownership of the HANDLE gets passed to the hFile local variable. It works just like a std::auto_ptr.

Also, notice the if statement. This checks to see if the hFile is in a valid state. It isn't comparing the hFile to NULL, but rather to the invalid_value_t template parameter. This means that you can forget about whether you should be comparing to NULL or INVALID_HANDLE_VALUE; if you're using the right auto_any typedef, then the right thing just happens. It's very easy for you to pick the right auto_any typedef because they use an obvious and consistent naming scheme.

Finally, notice the WriteFile statement. The WriteFile API is expecting a HANDLE, not an auto_hfile. Rather than providing an implicit conversion from auto_hfile to HANDLE, there is a get function which takes an auto_any and returns the contained resource. See the sidebar "SmartAnyRationale" for the reasons why there is no implicit conversion operator and why get isn't a member function in the auto_any class.

Class Prototypes

Behind the typedefs, the SmartAny classes have a policy-based template design that makes them flexible and extensible. Figure 2 shows their prototypes. The type T is the contained resource. In the case of auto_hfile, T is HANDLE.

Figure 2 SmartAny Classes (Close Figure 2)
template< typename T, class close_policy,
          class invalid_value = null_t, int unique = 0 >
class auto_any;

template< typename T, class close_policy,
          class invalid_value = null_t, int unique = 0 >
class shared_any;

template< typename T, class close_policy,
          class invalid_value = null_t, int unique = 0 >
class scoped_any;

The close_policy class cleans up the resource. It is a class that has a static member function named close that takes an object of type T as a parameter. In the case of auto_hfile, the close policy is close_handle, which is defined in Figure 3.

Figure 3 Close Policy (Close Figure 3)
// a generic close policy that uses a ptr to a function
template<typename Fn, Fn Pfn>
struct close_fun
{
    template<typename T>
    static void close( T t )
    {
        Pfn( t );
    }
};

// close_handle calls the CloseHandle() API
typedef close_fun<BOOL (__stdcall *)( HANDLE ),CloseHandle>
close_handle;

The invalid_value parameter defines the invalid state for the contained resource. It is the state that the wrappers have before they are initialized and after they have been cleaned up. Instances of the invalid_value type are implicitly convertible to type T and evaluate to the invalid value for type T. Often, the invalid state is NULL, 0, or a default constructed object. All of those cases are covered by null_t, which is the default for the invalid_value policy. In the majority of cases, null_t is exactly what you want, so you don't need to worry about it.

SmartAny Rationale

Once you start using SmartAny, you'll begin to notice some things about its design that might strike you as unconventional. Here are some of SmartAny's features and the rationale behind them.

No Implicit Conversions

An auto_hfile doesn't have an implicit conversion to HANDLE, so you cannot use an auto_hfile transparently as if it were a HANDLE. The same is true for the other SmartAny wrappers and their contained resources. This is by design since SmartAny's intended purpose is to make it easier to manage your resources correctly. An implicit conversion would work against this purpose. If the implicit conversion were available, then code like this would be able to compile:

   auto_hfile hFile( CreateFile( /*...*/ ) );
   /* lots o' code here */
   CloseHandle( hFile ); // oops!

Notice that with an implicit conversion, it is very easy to accidentally use an auto_hfile as a parameter to CloseHandle. This is bad because the hFile object thinks it owns the handle and will try to close it again when it goes out of scope. It's generally a bad thing to free your resources twice. It is especially easy to make this mistake when you begin using SmartAny in a legacy code base. For this reason, the implicit conversion is not provided, and therefore you are required to use the get accessor function to retrieve the contained resource.

OK, One Implicit Conversion

If you snoop around in SmartAny's source code, you will notice that the SmartAny wrappers actually do have an implicit conversion, but it is to a strange type called safe_bool. It is this conversion that allows you to test the wrappers for validity in conditional statements. A conversion to bool would accomplish the same thing, so why the "safe_bool"? The problem is that if you allow an implicit conversion to bool, you also allow a conversion to any intrinsic type that bool can be converted to (char, int, and so on), and this can happen when you least expect it.

The solution is to provide an implicit conversion to some obscure pointer type. The type actually used is a pointer to a member function on a dummy struct. It looks like this:

   struct dummy_struct
   {
       void dummy_memfun() {}
   };

   typedef void (dummy_struct::*safe_bool)();

   safe_bool const safe_true  = & dummy_struct::dummy_memfun;
   safe_bool const safe_false = 0;
Now, I define the implicit conversion operator as:
   // for use when auto_any appears in a conditional
   operator safe_bool() const
   {
       return valid() ? safe_true : safe_false;
   }

The pointer type is convertible to bool, so you can use auto_any objects in conditionals, but the pointer type is not convertible to anything else, so you don't get any nasty surprises.

No Named Member Functions

If you've used std::auto_ptr, you know that get, reset, and release are member functions. You may be surprised to discover that the SmartAny wrappers do not have these member functions. Rather, SmartAny provides get, reset, and release as friend functions. Wrappers like std::auto_ptr that have named member functions can lead you astray, as the following code fragment clearly illustrates:

   struct mine
   {
       void release() { /*...*/ }
   };

   std::auto_ptr< mine > p( new mine );

   p->release();
   p.release();

Notice the last two lines. At a glance, you might not see the difference. Although they look very similar, they are doing two completely different things. The first is calling mine::release, and the second is calling std::auto_ptr::release. It's very easy to type one when you meant to type the other, or accidentally transpose the two lines, with disastrous results that will only expose themselves at run time.

Had release been implemented as a friend function, the two calls would have to be written as:

   p->release();
   release( p );
This method is less prone to error since it's just about impossible to type one when you mean to type the other since the syntax is not so similar. As a rule, all named SmartAny "member functions" are actually implemented as friend functions.

No Direct Assignment

There is no assignment operator for assigning a resource directly to a SmartAny wrapper. As with std::auto_ptr, you must use reset to assign a new resource to an existing wrapper. The rationale is that assigning to a smart wrapper is not just a simple copy; it implies a transfer of ownership. In addition, if the wrapper is already managing a resource, that resource will be freed before the assignment takes place. Innocuous-seeming code like "foo = bar" is not at all suggestive of what is really going on. Code like "reset( foo, bar )" states your intention in no uncertain terms. This is a flashing neon sign to future maintainers of your code that screams that it is releasing the resource held by foo and taking ownership of bar.

To smooth over a wrinkle in the Win32 API regarding file handles, auto_hfile uses a different invalid_value policy. Consider the following code snippet:

auto_hfile hFile( CreateFile( /*...*/ ) );
if ( hFile ) { /*...*/ }
Seasoned programmers who use Win32 know that the CreateFile API returns INVALID_HANDLE_VALUE on failure, not NULL. They want the if statement to test the file handle against INVALID_HANDLE_VALUE. Fortunately, auto_hfile accomplishes this by using invalid_handle_t, defined in Figure 4.

Figure 4 invalid_handle_t (Close Figure 4)
// value_const is a general invalid_value policy that evaluates to a
// literal constant.
template<typename T,T value = T(0)>
struct value_const
{
    operator T const() const
    {
        return value;
    }
};

typedef value_const<HANDLE,INVALID_HANDLE_VALUE>
invalid_handle_t;

When you test an auto_hfile for validity in a conditional, you are really testing against an invalid_handle_t. Code like "if ( hFile )" means the same thing as "if ( get( hFile ) != invalid_handle_t() )", which amounts to "if ( get( hFile ) != INVALID_HANDLE_VALUE )", which is exactly right.

You will usually not have to worry about the last template parameter (unique). Its purpose is to provide an extra level of type safety. There are some resources that the Win32 API treats polymorphically, even though they are very different conceptually. For example, events and threads are both represented by HANDLEs and are cleaned up by calling CloseHandle. Many programs don't need the ability to treat threads and events polymorphically, and in fact it can be dangerous to do so. The SmartAny library employs the unique template parameter to give threads and events strong types, preventing you from using one in situations where the compiler is expecting the other.

The auto_any Class

The auto_any class is modeled after std::auto_ptr. It has very useful but somewhat counterintuitive copy semantics. In particular, only one auto_any is allowed to own a particular resource at a time. When you assign from one auto_any to another, both objects change. One object gives up ownership, and the other accepts it. When I speak of "ownership," I mean the object contains a valid resource. After an auto_any has given up ownership, it no longer contains a valid resource (see Figure 5). This has profound implications. If you pass an auto_any by value to a function, you are giving ownership to the function. Your local auto_any will become invalid (oops!), and the resource will be automatically cleaned up when the function returns (oops!). It's the roach motel of functions—resources go in, but they don't come out. Sometimes this is exactly what you want; often it isn't.

Figure 5 Transferring Ownership (Close Figure 5)
auto_event e1( CreateEvent( /*...*/ ) );
// e1 is valid here
assert( e1 );
auto_event e2;
// e2 is not valid yet
assert( ! e2 );
// transfer ownership!
e2 = e1;
// now e2 is valid ...
assert( e2 );
// ... and e1 is not!!!
assert( ! e1 );

Returning auto_anys by value is a far more useful idiom. Such a function allocates a resource, wraps it in an auto_any, and returns it by value, unambiguously passing ownership to the caller. If the caller ignores the return value, the resource is freed automatically, eliminating a whole category of resource leaks.

The nonstandard copy semantics of auto_any make it unfit for the standard containers. If you would like a smart wrapper that is safe to put in a container, shared_any is your best bet.

All auto_any objects have the same memory footprint as the type they contain. So, for example, sizeof( auto_event ) == sizeof(HANDLE ). This is useful because an array of auto_events can be cast to a HANDLE* and passed to an API such as WaitForMultipleObjects.

The shared_any Class

A shared_any object uses reference counting to determine when it should clean up a resource. It has standard copy semantics, so it is safe to put shared_anys into standard containers. When you copy a shared_any, it increments its reference count. When a shared_any goes out of scope, it decrements its reference count. If the reference count goes to zero, the contained resource is deallocated (see Figure 6).

Figure 6 Shared_any Class
Figure 6 Shared_any Class

A shared_any object does not ordinarily have the same memory footprint as the contained resource. If you have an array of shared_events, you cannot safely cast the array to a HANDLE*. Use auto_any or scoped_any for that. However, if you wrap a COM pointer in a shared_any (for example, if you use the close_release_com policy), then you automatically get intrusive reference counting (see Figure 7). That is, instead of maintaining a separate reference count, shared_any will use COM's reference counting via IUnknown::AddRef and Release. These "COM-ified" shared_anys are more efficient, and they have the same memory footprint as the bare COM pointer. You can assign an auto_any to a shared_any. If you do, the auto_any gives up ownership, and the shared_any takes ownership with a reference count of 1.

Figure 7 Shared_any Object
Figure 7 Shared_any Object

The scoped_any Class

A scoped_any object is the simplest type of smart wrapper. Like auto_any, it guarantees that only one scoped_any can own a particular resource at a time. Unlike auto_any, it accomplishes this by disallowing copying and assignment altogether. This makes it less flexible than auto_any, but much more difficult to use incorrectly. If all you want is a way to guarantee that a resource will get deallocated whenever a scope is left, scoped_any is a solid choice. You cannot make the mistake of passing a scoped_any to a function by value—the compiler won't let you. Since copy and assignment are disallowed, you can't put a scoped_any into a standard container. Figure 8 will help you choose the right tool for the job.

Figure 8 Choosing a Helper Class (Close Figure 8)

Requirement | auto_any | shared_any | scoped_any
Strict ownership | Yes | No | Yes
Standard copy semantics | No | Yes | No
Pass and return by value | Yes | Yes | No
Safe for standard containers | No | Yes | No
Same footprint as contained type | Yes | No | Yes
Works with managed types | Yes | Yes | Yes

Managed Code

The SmartAny library works equally well with managed types. Each of the three SmartAny classes can contain a managed pointer. You can choose the close_delete policy, which will invoke the destructor, or the close_dispose policy, which will invoke the Dispose method. Or you could write your own to do both. Figure 9 shows an example of using auto_any with a managed object.

Figure 9 Using auto_any with a Managed Object (Close Figure 9)
#using   <mscorlib.dll>
#include <windows.h>
#include "auto_any.h"

__gc struct MyObject : System::Object, System::IDisposable
{
    virtual ~MyObject()
    {
        System::Console::WriteLine(S"~MyObject!!!");
    }
    virtual void Dispose()
    {
        System::Console::WriteLine(S"Dispose!!!");
    }
    void Hello()
    {
        System::Console::WriteLine(S"Hello, world!");
    }
};

int main()
{
    // This object is automatically disposed
    auto_any<MyObject*,close_dispose> pO1( new MyObject );

    // This object is automatically deleted
    auto_any<MyObject*,close_delete> pO2( new MyObject );

    // Just like any smart pointer, this does what you expect
    pO2->Hello();

    return 0;
}

As you can see, the syntax for dealing with managed types is exactly the same as dealing with unmanaged types. This lets you have deterministic finalization with your managed objects with code that is familiar and easy to write and maintain.

Conclusion

The SmartAny library comes with a host of useful typedefs, canned cleanup policies, invalid-value policies, and accessor functions. You can download the full source code from the link at the top of this article.

For related articles see:
/msdnmag/issues/1100/gci

For background information see:
Modern C++ Design: Generic Programming and Design Patterns Applied by Andrei Alexandrescu (Addison-Wesley, 2001)

Eric Nieblerstudied Computer Science and Mechanical Engineering at the University of Virginia. He has worked at Microsoft in various capacities since 1996; first, as a tester in the Windows 2000 group, then as a developer in Microsoft Research. These days, he calls the Visual C++ Libraries Team home. He can be reached at ericne@microsoft.com.


Editor's Note: RSS is Here!

Editor's Note
RSS is Here!

RSS is a great, example of how a technology like XML can be used to improve the overall user experience. RSS (Rich Site Summary) is a description of a simple XML schema that can be used to describe the contents of a Web destination. Its most common use is to provide updated summaries of the latest headlines on a site, including a headline, summary, and links to the actual articles. As Cookie Monster might say, "So crazy, just might work."

The sites themselves are responsible for providing their own XML-formatted RSS feeds. This information can be provided in a static file or dynamically. It's the client's job to turn the XML into a list of headlines for the user. A client can graft any combination of feeds into one list and update it regularly. (And, of course, a client can actually be an intermediary, like a Web app.)

RSS Headline Display
RSS Headline Display

So what's this have to do with us? We've been working on several Web projects of late to make our site more valuable to readers. One such enhancement—the MSDN Magazine RSS feed—is now live. We've been experimenting with creating an RSS feed from our site so you can get an XML summary of each month's articles.

To get to the new MSDN Magazine feed, go to http://msdn.microsoft.com/msdnmag/rss and check out our sample offering. (Right now it's the current issue's contents.) If you don't have a client installed, you can look at it directly in your browser, where it'll show up as plain XML. The feed for an article might look like this:

<rss version="2.0">
  <channel>
    <title>MSDN Magazine</title>
    <link>http://msdn.microsoft.com/msdnmag/</link>
    <description>The Microsoft Journal for Developers</description>
    <language>en-us</language>
    <ttl>1440</ttl>
    <item>
      <title>ASP.NET: Create Snazzy Web Charts and Graphics On the Fly
             with the .NET Framework</title>
      <link>http://msdn.microsoft.com/msdnmag/issues/02/02/ASPDraw/
      </link>

<description>Knowledge workers can understand data more
effectively when raw numbers are presented in a graphical format. This
is especially true when displaying database information on a Web page,
where a simple chart can make the difference between a dry presentation
and a vivid data source. In the past, creating dynamic, data-based
charts on the fly in ASP required purchasing a third-party, image-
generating COM component. Now with ASP.NET, developers can access the
.NET Framework's drawing classes directly with C# to create dynamic
images and charts.
</description>
       ...
      <pubDate />
</item>

Once you have an RSS client (the one shown here is called FeedExpress), you'll get a headline display comparable to what you see in the screenshot. There's a wide variety of software available now—if you don't have any, just Google up "RSS 2.0 software." MSDN has a sample news aggregator at Building a Desktop News Aggregator.

We always welcome suggestions for making our Web site more useful. Drop a line to our Webmaster, John Caricato, at mmeditor@microsoft.com if you have ideas you'd like to see in action.

Sample Update Last month, we introduced a small sample application—an MP3 playlist maker written in Visual Basic .NET. We've received lots of mail on this, so here's an update.

The original version was written with Visual Studio .NET 2003, so it didn't load for everyone who tried it. We've created a version that's compatible with the release build of Visual Studio .NET 2002. Since the FolderBrowserDialog object is not directly supported, it's been removed from the code—you'll have to type the path into the edit box yourself or write an overload. Depending on your system resources, there's also a memory ceiling that the program may or may not hit. (The app will start to churn virtual memory at some point.) If you experience this firsthand and come up with a fix for it, drop us a line and you may be featured in a future Editor's Note. You can download the updated code now at EditorsNote0304.exe.

—J.T.

ActiveX, IntelliSense, J/Direct, JScript, Microsoft, Microsoft Press, MSDN, MS-DOS, Visual Basic, Visual C++, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Visual J# and Windows Server are trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Object Relational Mapping
Side-by-side File Comparison
Monitor Web Site Traffic
Encryption Toolkit
Compare and Synch SQL Databases
Defect Tracking and Squashing
Validate Links
Keep Up with Changing Web Standards
Defect Tracking, Server Balancing, and Beta Management
Build Management
WSDL Editor
The Bookshelf

Object Relational Mapping

Olero Software Inc. has announced the release of ORM.NET 1.2, an object relational mapping tool designed for developers using the Microsoft® .NET Framework. ORM.NET auto-generates a complete data layer object model based on your SQL database schema. The generated data layer is compiled as a run-time component and exposes all tables as classes and all columns as properties.

Using the built-in DataManager object, users can retrieve data from multiple tables based on complex criteria without using stored procedures or embedded SQL code. In addition, data updates, inserts, and deletes can be saved to the database with one call.

All SQL code is automatically implemented, making retrievals, updates, and modifying even complex relational joins easier. The resulting data layer can be used with every method and property accessible using the IntelliSense® feature that's part of Visual Studio® .NET. ORM.NET allows you to model your application using a flexible object-oriented methodology while keeping your data in a normalized relational data store.

Side-by-side File Comparison

PrestoSoft has released ExamDiff Pro 3.0, a visual file and directory comparison tool, offering visual binary file comparison, word wrap, regular expression filters, named sessions, word- and character-level diffs, and file editing. Other features include print and print preview, directory snapshots for later comparison, and plug-ins that allow the preprocessing of files before comparing them.

ExamDiff Pro also includes recursive comparisons, custom reports, file management support, drag and drop support, search and replace (with regular expression support), shell extension support, internal file editing and synchronizing, and custom color and font settings.

The program uses a double-pane view that allows for side-by-side comparisons with color-coded line numbers, indicating whether each line is added, deleted, or changed.

Monitor Web Site Traffic

LastBit Software has released TrafMeter 3.40, its Web site traffic reporting and monitoring utility. TrafMeter lets you set up an unlimited number of filters for the captured traffic, defining the network packets that are used for tracking and reporting purposes. Different rule sets can be used for single hosts, subnets, or specific groups of hosts such as for all hosts that are in your LAN or for all hosts not in your LAN.

TrafMeter also supports counting of custom IP packets with a flexible filter engine by any IP protocol or by any TCP/UDP port. TrafMeter provides real-time visual presentation of traffic activity via charts, with different colored lines representing different filters. The bandwidth of the filtered packets can also be displayed. TrafMeter can also be used as a troubleshooting tool for local networks. In addition, you can include MAC addresses from Ethernet headers and TOS field from IP headers in your traces.

TrafMeter can generate XML traffic reports for immediate viewing or store the traffic counters in a database. TrafMeter is compatible with most popular databases and allows you to exchange data between them using an ADO interface. Since the traffic reports are generated in XML, you can create reports based on traffic counters by using your custom XML stylesheets.

Encryption Toolkit

Nickellie LLC recently announced the introduction of the Nickellie Encryption Toolkit 2003, an encryption library for developers using Windows® and Pocket PC software. It includes tools needed to encrypt and decrypt data and files from within applications. The toolkit also includes COM objects with a standard interface for both the Pocket PC and Windows desktop platforms. This enables you to use the same source code on both platforms and encrypt files and data to be transmitted between Pocket PC devices, between desktop computers or servers, or between a Pocket PC and a desktop computer or server. The toolkit can be used for standalone Windows or Pocket PC applications and client/server applications.

The toolkit supports both 40-bit and 128-bit symmetric key encryption via the Microsoft Base and Enhanced Cryptographic Providers. The COM interface enables encryption from Visual Basic®, eMbedded Visual Basic, Visual C++®, and eMbedded Visual C++.

The Nickellie Encryption Toolkit includes documentation, sample code, and technical support, and is available in three editions: Trial, Business, and Unlimited.

Compare and Synch SQL Databases

Adept SQL Tools has released AdeptSQL Diff, which allows users to visually compare two Microsoft SQL Server™ databases, then produce and execute synchronization SQL scripts for a selected portion or for the whole schema. The program scans both databases and displays the combined schema tree view, with all differences marked in color. Navigating through the database schema tree, you can see SQL definitions for individual objects (tables, view, stored procedures, and so on) compared side-by-side in the larger panel to the right of the schema tree.

The schema matching engine generates effective update SQL for any schema difference you need to synchronize, trying to preserve the table data whenever possible. The generated scripts are displayed in a separate debugger-style popup window where you can edit them, execute them either uninterrupted or step-by-step, set breakpoints, turn on transactions, and roll back on errors. With a single database, you can use AdeptSQL Diff to reverse-engineer your database schema.

Defect Tracking and Squashing

MCPCentral.com is shipping OnTime Defect Tracker and Feature Management 2.0, its testing and development tool. OnTime is a .NET and SQL-based multiuser defect tracking and feature management system designed for small to mid-sized development and test teams of 5 to 100 users. The new version includes features such as role-based security, file and image attachments, additional reporting, and faster performance over the network.

With role-based security, users can be grouped according to job title, allowing for different security permissions with granular control to features such as add, edit, and delete of projects, defects, features, and users. Screenshot images and other related files can be attached to a defect or feature. OnTime can store the path to a file or it can store the entire file in the SQL Server database, making the file retrievable by any user.

The new Stamp feature inserts the date and name of the person who enters or edits an entry. The Project Summary report outlines the current state of each project. Smart client-side data caching enhancements have been added to reduce round-trips to the SQL database server. Using the optional OnTime .NET Web Services SDK, software developers can incorporate defect tracking and reporting into existing applications.

Validate Links

REL Software recently announced the availability of Web Link Validator 3.1, its management software for Webmasters and corporate IT professionals. The product is compatible with Windows 9x, Windows NT®, and Windows XP. Web Link Validator performs a thorough analysis of Web sites, including link integrity testing and analysis of slow-loading, deep, outdated, and small pages. The current version can handle Web sites containing over 100,000 links—the total number of links is limited only by available system resources.

Web Link Validator 3.1 can access secure Web sites (via HTTPS) and password-protected pages, and supports a variety of proxy servers. Reports about the checked pages can be viewed on-screen and then saved in a variety of industry-standard formats. The program's filters enable research of any occurring problems. Web Link Validator also includes the ability to check the local copies of Web sites, individual settings for each site, and its customizable interface.

Web Link Validator 3.1 can perform thorough orphan analysis, helping to find files that are not linked from any Web page and just take up disk space. The new version can handle all kinds of hyperlinks, including dynamically created ones and those coded with JavaScript and Macromedia Flash. Other features of the new version include the ability to perform form-based authentication, new reports, filters, and profile settings, and better recognition of links embedded inside all kinds of image maps, scripts, applets, and forms.

Keep Up with Changing Web Standards

Bradbury Software LLC has released TopStyle Pro 3.1, a Windows HTML, XHTML, and Cascading Style Sheet (CSS) editor that allows Webmasters to comply with changing Web standards. TopStyle Pro was developed around the HTML 4.01, XHTML, and CSS standards. With its emphasis on ease of maintenance, TopStyle Pro offers support of CSS.

Using CSS, Webmasters can separate a site's design from its content, enabling them to change an entire site's colors, fonts, and other layout information by modifying a single style sheet. The program includes a Style Upgrade tool which upgrades outdated HTML, including FONT tags, to CSS. TopStyle Pro also converts HTML to XHTML. TopStyle Pro integrates with Watchfire's Bobby and with the W3C's HTML and CSS Validation Services, enabling Web authors to check their site for W3C compliance without leaving TopStyle Pro's editor.

In addition, TopStyle Pro integrates with many existing HTML editors, including Macromedia's Dreamweaver MX and HomeSite. Dreamweaver MX automatically detects the presence of TopStyle Pro and makes it the default external CSS editor.

TopStyle Pro helps you create cross-browser Web sites by including a split-screen preview which displays the same page in Microsoft Internet Explorer and Netscape Gecko (Mozilla). TopStyle Pro's Style Checker examines style sheets for cross-browser errors and identifies browser bugs that may affect their display.

Defect Tracking, Server Balancing, and Beta Management

TechExcel Inc. is shipping DevTrack 5.1, a Web-based defect- and project-tracking tool for large, distributed software development teams. DevTrack tracks and manages defects, change requests, feature enhancements, and provides workflow and process automation features, detailed searching, and reporting. This release includes a new Web portal for beta customers, support for Web server load-balancing, and automated installations and upgrades.

DevTrack 5.1's Web portal for Beta customers provides a secure and personalized Web interface to participate in the product development process. Beta customers can use the Web portal to submit and update issues, check the status of their issues, and communicate with the development and quality assurance teams.

DevTrack 5.1's multiple Web server support capability enables customers to deploy multiple Web servers, connected to the same DevTrack application server and database server, supporting hundreds of concurrent users.

Build Management

Kinook Software has released Visual Build Professional 5.0, which allows developers to create an automated, repeatable process for building and deploying software. Version 5.0 adds support for Borland Delphi and JBuilder, Perforce, SQL Server, Zip and Unzip files, generation of HTML log reports, integration with the Windows Task Scheduler and event log, script for build events, user creation of custom action screens and components, support for Unicode, as well as other new features and enhancements. Visual Build Professional runs on Windows NT 4.0, Windows 2000, Windows XP, and Windows Server 2003.

WSDL Editor

Cape Clear Software Inc. has released a new version of its WSDL Editor. The WSDL Editor provides a graphical environment for creating and editing Web Services Description Language (WSDL), a building block of Web Services. The editor has been designed for both experienced and novice Web Services developers and works with the .NET Framework, Java, J2EE, and CORBA. This latest release adds support for building Web Services from large XML Schema.

The Bookshelf

Addison-Wesley has released a series of books intended to help programmers upgrade their skills to Visual Studio .NET 2003. Database Access with Visual Basic .NET, Third Edition by Jeffrey P. McManus and Jackie Goldstein is a guide to developing database applications using Visual Basic .NET and ADO.NET. It offers real-world solutions to the data access issues developers using Visual Basic .NET face every day. Also included are in-depth explanations and numerous examples. Essential ASP.NET with Examples in C# and Essential ASP.NET with Examples in Visual Basic .NET by Fritz Onion teach developers how to most effectively use the features of ASP.NET. The books include working examples of how Web applications are built today. Also part of the series is .NET Web Services: Architecture and Implementation by Keith Ballinger. For more information about Addison-Wesley titles for programmers, visit http://www.awprofessional.com.

Contacts

AdeptSQL 4464 West 10th Avenue, Suite 302, Vancouver BC, V6R2H9 Canada, http://www.adeptsql.com

AntsSoft No 5, Lane 300, Room 402, South HongCao Road, Shanghai, 200233, PR China, +86 (21)54190679, http://www.antssoft.com

Bradbury Software LLC 203 Bellegrove Court, Franklin, TN 37069, http://www.bradsoft.com

Cape Clear Software Inc. 2929 Campus Drive, Suite 400, San Mateo, CA 94403, 888-227-3439, http://www.capeclear.com

Kinook Software Inc. P.O. Box 63413, Colorado Springs, CO 80962, 719-481-4128, http://www.kinook.com

LastBit Software 1133 Broadway, Suite 706, New York, NY 10010, 877-684-2996, http://www.lastbit.com

MCPCentral.com 351 West Vinedo Lane, Tempe, AZ 85284, 800-653-0024, http://www.MCPCentral.com

Nickellie LLC 2475 Harvest Moon Drive, Greenwood, IN 46143, 317-748-3232, http://www.nickellie.com

Olero Software Inc. 1738 Pearl Street, Suite 150, Boulder, CO 80302, 303-385-4908, http://www.olero.com

PrestoSoft P.O. Box 3611, Thousand Oaks, CA 91359, http://www.prestosoft.com

REL Software Corp. Ovchinnikova, 11-106 Perm, 614094, Russian Federation, +7(3422)246973, http://www.relsoftware.com

TechExcel Inc. 3675 Mt. Diablo Boulevard, Suite 200, Lafayette, CA 94549, 800-439-7782, http://www.techexcel.com

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Careyis a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: Font Sizing, Internationalization in JScript, and More

Web Q&A
Font Sizing, Internationalization in JScript, and More
Edited by Nancy Michell

Q I'm creating new Cascading Style Sheet (CSS) styles for my project. For the best compatibility between several display resolutions and browsers, is it better to use pixels (px) or points (pt)?

A Font sizes aren't usually specified in px in a production environment—instead, pt or em (a relative size) are generally used. Most sites that really want to control layout and have a target browser in mind (usually Microsoft® Internet Explorer) tend to use pt because they can get something like 8pt Verdana pretty easily and it looks the same way in Word. Sites that focus on accessibility and cross-browser, cross-platform compatibility tend to use em because it gives the user control over the size of the font in his browser (which won't work with either px or pt). The trade-off is that it's actually much harder to get font sizing to come out the way you want it to when you use em because the measurement is based on the current location in the Document Object Model (DOM). Figure 1 shows how this works.

Figure 1 Different Font Specs (Close Figure 1)
<html>
<head>
<style>
     .sizedPt { font-family:verdana; font-size:8pt; color:red; }
     .sizedEm { font-family:verdana; font-size:.68em; color:blue;}
</style>
</head>
<body>

<!-- Note here that with nesting, the TD with the style shows text
     smaller, but a table with the same style on an internal
     TD doesn't -->
<table class="sizedEm" border="1">
   <tr>
      <td class="sizedEm">
         <p>Some Text</p>
         <table border="1">
            <tr>
               <td class="sizedEm">
                  <p>Some more text</p>
               </td>
            </tr>
         </table>
      </td>
   </tr>
</table>

<p>&#160;</p>

<!-- Note here that even with nesting, everything is 8pt -->
<table class="sizedPt" border="1">
   <tr>
      <td class="sizedPt">
         <p>Some Text</p>
         <table border="1">
            <tr>
               <td class="sizedPt">
                 <p>Some more text</p>
               </td>
            </tr>
         </table>
      </td>
   </tr>
</table>
</body>
</html>
Figure 2 Large Font Browser Setting
Figure 2 Large Font Browser Setting

Figure 2 and Figure 3 show how the font size is affected by changing the font settings in the browser. As you see in the bottom table in each figure, the red text size does not change. Having a font size that can't change is good if you must have a degree of control over your layout, but remember, it's very convenient for users to be able to set the type to a size that's comfortable for them to read.

Figure 3 Small Font Browser Setting
Figure 3 Small Font Browser Setting

Q I've come to the conclusion that there isn't really any international support in JScript®. Is this correct, or am I missing something? For example, on a German locale where the decimal character is a comma instead of a period, the real and expected results of several operations are as follows:

var x = "123,45";
alert(parseFloat(x)); // shows 123, expected be 123,45
alert(parseInt(x));   // shows 123, correct
alert(Number(x));     // shows NaN, expected 123,45
alert(isNaN(x));      // shows true, expected false

Also, what if I want to convert from German to English?

A There is some international support in JScript. For example, there's the toLocaleString methods of Number and Date, the toLocaleDateString and toLocaleTimeString methods of the Date object, and the toLocaleLowerCase and toLocaleUpperCase methods of the String object. All are relatively recent additions to the language—with the exception of toLocaleString on Date, which has been around since the beginning, but did not actually output a localized date string! However, there is not much support for going the other way—turning localized strings into dates, numbers, currencies, and so on, which it looks like you want to do. This is unfortunate. For internationalization, use VBScript.

In Andrew Clinick's MSDN® Library article "It's All Greek to Me", the following code sets the locale to German, converts the value, multiplies it by the exchange rate, and converts it into a currency variant. After the conversion, the locale is set back to UK English and the currency is formatted in UK English format.

sub button2_onclick
   dim original
   original = setlocale("de")
   myvalue = CCur(GermanNumber.value * 2.89)
   original = setLocale("en-gb")
   USNumber.value = formatCurrency(myvalue)
end sub

Q How can I make the contents of a <td> element in an HTML table clickable? Currently, my code looks something like this:

<td><a href="http://somewhere"><img src=blah></a></td>
The problem with this is that the image inside the <td> is relatively small, while the cell is much bigger. Is it possible to make the whole <td> clickable, including the blank area around the small image? Here's my idea:
<td href="http://somewhere"><img src=blah></td>

A There are a couple of ways to do this, depending on your application. You could pad the image with a transparency or background color to fill the table cell. If the image is a GIF, it will add hardly anything to the size. But if you enlarge the image this way, the table containing it will not scale well with the user's changing font settings—it will be either too big or too small.

If you want to use an image and don't have any other content in your <td>, you can always scale it to the size of the cell:

<td><a href="http://somewhere"><img src="img.gif"
  width="100%" height="100%" /></a></td>

If you have other content in the table cell and you want it all to be clickable but still use an anchor tag <a>, then you can use that same code and just wrap all of the content in the anchor tag. If you want to make the <td> clickable and don't mind using JScript, you can do this:

<script language="javascript">
  function elementClicked(oElement) {
       if (oElement == tdClick) {
           location.href = "somewhere";
       }
   }
   </script>
   <td id="tdClick"
    onclick="elementClicked(this)">
    HTML content here</td>
You can also put the script in the tag:
<td id="tdClick"
 onclick="javascript:location.href='somewhere';">
 HTML content here</td>

However, the onclick event won't work in Netscape (because of limited DHTML support). The earlier suggestion

   <td><a href="http://somewhere"><img src="img.gif"
     width="100%" height="100%" /></a></td>
will work just fine in both Internet Explorer and Netscape, but you'll get those weird scaling issues if the image actually has content (rather than a blank GIF).

Q I have two ASP files: input1.asp and input2.asp. Input1.asp is:

<%
  Dim sInput
  sInput = Request.Form("txtFile")
%>

<HTML>
  <BODY>
    <form name="frmProcess" method="post" action="input1.asp">
      <input type="text" name="txtFile" value="<%=sInput%>">
      <input type="submit" value=Submit>
    </form>
  </BODY>
</HTML>

The content of input2.asp is the same except that action is now set equal to input2.asp and the input type is file. The problem is that when I enter something in input1.asp and hit submit, the value of the textbox is retained even after the page is refreshed.

As you can see, I am taking advantage of the "value" attribute on the element. Why can't I do that to the <input type="file"> object? In MSDN Library documents I don't see a "value" attribute available for type="file". Is this by design? I don't want the user to have to fill in the path every time the submit button is clicked. It would be good if the value could be retained in the input field.

A This question has come up before, so here's the overview. Security reasons dictate the fact that you cannot modify the value attribute of <input type="file" /> elements. If that were possible, then you could completely script the uploading of documents from a user's machine without their knowledge. For the value of the input to be modified, users are required to do it themselves.

Q What is considered correct behavior for generating UTF-8 files? Should HTML files have Byte Order Marks (BOMs)? If a file is marked as charset=utf-8 must it have a BOM or not? Is it acceptable to have one? What about UTF-8 text files and XML files? The problem that prompted this question was that some of my Japanese UTF-8 pages were not displaying in Internet Explorer unless the BOM was present. It turns out that the behavior was caused by the Title tag preceding the charset=utf-8 meta tag. Apparently, the initial scan for the meta tag stopped when it hit the non-ASCII characters and the auto-detect didn't identify the page as UTF-8. Moving the Title tag after the charset=utf-8 meta tag resolved it, but left me wondering about when to use a BOM.

A It used to be thought that placing a UTF-8 BOM at the beginning of a file was undesirable, but should be respected if it's present. However, this has been challenged recently. Browsing the Web to find discussions on this issue led to this: http://www.unicode.org/unicode/faq/utf_bom.html#29, which asks if a UTF-8 data stream can contain the BOM character (in UTF-8 form) and, if so, whether you can still assume that the remaining bytes are in big-endian order.

Yes, UTF-8 can contain a BOM. But it doesn't affect the order of the byte stream. UTF-8 always has the same byte order. So, UTF-8 BOMs are acceptable, but don't indicate byte-ordering.

One bad thing is that old browsers display garbage when the BOM is at the beginning of an HTML file. How old those browsers have to be for this to happen is uncertain, so if you have to support every old browser, test on your targets or leave it out.

Some used to think that placement of a BOM at the start of a UTF-8 file was a nonstandard Notepad-specific behavior. UTF-8 BOMs do, however, allow parsers to recognize the encoding of files more efficiently and unambiguously.

Q Is there a technique (either ASP.NET or plain DHTML) that can produce a multicolumn page, similar to a newspaper? Specifically, the effect I'm looking for is that content would be rendered in column1, and when the bottom of column1 is reached, it would continue at the top of column2. When the bottom of column2 is reached, it would continue on top of column3, and so on. I need to apply this effect to the way cells are rendered in a <TABLE>. An ASP.NET solution is preferred, but I'll take DHTML as well. The text for the columns is based on a SQL Server™ query result.

A There is no easy or effective way to do this with ASP.NET, HTML, or DHTML. You could do it with System.Drawing in ASP.NET, but that is probably not what you're looking for. The only way this would ever change is if the general HTML specs were enhanced. If you can figure out how many rows will fit on the client page, you should be able to data-bind a DataList or dynamically render an HTML table with just the rows you want.

Assuming you can grab the size of the viewable space on the client and can calculate the number of rows that would fit there, you should be able to render each column and just show the appropriate "window" of data for each row. The other hard part about doing this type of layout is changing the rows when the user resizes the page. An alternate method might be to return XML to the client and populate the page via the DHTML DOM and JavaScript on the client. This way, the page could re-render itself dynamically if the user resized. This will take some work to accomplish. However, if you really are considering formatting your pages this way, remember that reading columns on the screen is difficult for users.

Q When I set Internet security to high (disable Active Scripting) and also block all cookies from Internet Explorer, and then go to Expedia.com, I see that Expedia.com can detect that Active Scripting is disabled. I checked out the NetMon trace, but still can't figure out how the information is passed around.

A Take a look at http://www.w3schools.com/asp/asp_browser.asp. When a browser connects to a server, an HTTP User Agent Header is also sent to the server. This header contains information about the browser (such as browser type and version number). The BrowserType object then compares the information in the header with information in a file on the server called Browscap.ini.

While the User Agent identifies the browser, it does not indicate whether the user of the browser has disabled script. There are several, completely client-side ways you can detect it, though. Most common is the use of the <noscript> tag. For example:

<script language="javascript">
  document.write("You have script enabled");
</script>
<noscript>You do not have script enabled</noscript>
The contents of the noscript tag are only rendered when script is not available.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to to the following Microsoft developers for their technical expertise: Peter Averill, Dan Battagin, Mike Bizub, Daniel Boerner, Jerry Bryant, Andrew Clinick, Bryant Fong, Davide Garghentini, Pranav Kandula, Michael Kaplan, Michael Kung, Eric Lippert, Kirk Marple (Volt), Sajee Mathew, Harley Rosnow, Murray Sargent, Sean Shi, Chad Shurtz, Tommy Unger, Steve Van Dongen, Joel Varland


Data Points: Techniques in Filling ADO.NET DataTables: Performing Your Own Analysis

Data Points
Techniques in Filling ADO.NET DataTables: Performing Your Own Analysis
John Papa

Code download available at: DataPoints0306.exe (128KB)
Contents
Retrieval Methods
The Test Scenario
Test Results
Wrap-up

One of the most fundamental differences between ADO.NET and classic ADO is how the data is represented. In ADO.NET it's represented in related DataTable objects, whereas classic ADO represents data in Recordset objects. Using classic ADO, the typical app retrieves a single rowset and stores it in a single Recordset object. When using DataTable objects in ADO.NET, you can store several rowsets in several DataTable objects, all related to one another and contained within a single DataSet. For example, you can retrieve a customer rowset, order rowset, and an order detail rowset and store them in three DataTable objects within a single DataSet. This gives you great flexibility to manage your data.

I've recently received several inquires asking which technique is best for retrieving data and populating a DataSet. Since the Microsoft® .NET Framework offers so many choices on how to write the code, many developers are now taking a close look at the different options in an effort to make the best choice for their apps. One of these choices is how to retrieve data, relate it, display it, and represent it in an app. The concept of running scenarios before deciding how to architect an application isn't new by any means. Its importance is quite critical, especially when presented with a relatively new development model such as the .NET Framework. Because developers are faced with so many new ways to build apps, it could be argued that testing is more critical than ever.

So in this month's Data Points column I will take a look at one way to evaluate different architectural decisions using the retrieval of data as a focal point. I'll examine different ways to retrieve data into DataTables, and I'll discuss some performance tests I ran on them and when and where each of the techniques might be useful in applications. I'll also walk through scenarios that demonstrate how to retrieve relational data and manage it effectively. You can consider some of the points that follow to be best practices, but you should also run your own analysis.

Retrieval Methods

Here's a list of some techniques that can be used to load data into DataTable objects that are related to one another within a DataSet:

  • Retrieve all of the data from a single joined query into a single DataTable (see Figure 1).
  • Retrieve each rowset (three in all) from a query into its own DataTable using subqueries (see Figure 2).
  • Retrieve each rowset (three in all) from a query into its own DataTable using joins (see Figure 3).
  • Retrieve each rowset (three in all) from a query into its own DataTable using parameters.

Figure 3 Using Joins (Close Figure 3)
--- Retrieve Customer information
SELECT     CustomerID,
    CompanyName,
    City
FROM     Customers
WHERE    Country = 'USA'
ORDER BY
    CustomerID

--- Retrieve Order information
SELECT     o.CustomerID,
    o.OrderID,
    o.OrderDate
FROM     Customers c
    INNER JOIN Orders o ON c.CustomerID = o.CustomerID
WHERE     c.Country = 'USA'
ORDER BY
    o.CustomerID,
    o.OrderID

--- Retrieve Order Detail information
SELECT     od.OrderID,
    od.ProductID,
    od.UnitPrice,
    od.Quantity
FROM     Customers c
    INNER JOIN Orders o ON c.CustomerID = o.CustomerID
    INNER JOIN [Order Details] od ON o.OrderID = od.OrderID
WHERE     c.Country = 'USA'
ORDER BY
    o.CustomerID,
    o.OrderID,
    od.ProductID
Figure 2 Using Subqueries (Close Figure 2)
--- Retrieve Customer information
SELECT     CustomerID,
    CompanyName,
    City
FROM     Customers
WHERE    Country = 'USA'
ORDER BY
    CustomerID

--- Retrieve Order information
SELECT     CustomerID,
    OrderID,
    OrderDate
FROM     Orders
WHERE     CustomerID IN
    (
    SELECT CustomerID
    FROM Customers
    WHERE Country = 'USA'
    )
ORDER BY
    CustomerID,
    OrderID

--- Retrieve Order Detail information
SELECT     OrderID,
    ProductID,
    UnitPrice,
    Quantity
FROM [Order Details]
WHERE OrderID IN
    (
    SELECT OrderID
    FROM Orders
    WHERE CustomerID IN
        (
        SELECT CustomerID
        FROM Customers
        WHERE Country = 'USA'
        )
    )
ORDER BY
    OrderID,
    ProductID
Figure 1 Retrieve All Data in One Joined Query (Close Figure 1)
--- Retrieve Customer, Order and Order Detail information
SELECT    c.CustomerID,
    c.CompanyName,
    c.City,
    o.OrderID,
    o.OrderDate,
    od.ProductID,
    od.UnitPrice,
    od.Quantity
FROM    Customers c
    INNER JOIN Orders o ON c.CustomerID = o.CustomerID
    INNER JOIN [Order Details] od ON o.OrderID = od.OrderID
WHERE    (c.Country = 'USA')
ORDER BY
    o.CustomerID,
    o.OrderID,
    od.ProductID

Each of these techniques will retrieve all the customer, order, and order detail information for all customers in the U.S. The data will then be bound to an ASP DataGrid control in a Web Form to be displayed. First, I'll describe each of the retrieval methods, then I'll explain the tests I put them through, and finally I will share the results of my tests.

The first method will retrieve all of the data using a single joined query and populate a single DataTable. Thus there will be only one query and it will join the customer, order, and order detail tables together using a SQL query and return the fields from each of the required tables. The results are then stored in a DataTable within a DataSet and bound to the ASP DataGrid control.

The second method retrieves all the U.S. customer information from a SQL query and stores it in a DataTable. The order information is retrieved using a SQL query that narrows the orders down to only U.S. customers using a subquery in the WHERE clause. Then, the order detail information is retrieved using a SQL query that restricts those results to only U.S. customers using a nested subquery in the WHERE clause.

The third method retrieves all the U.S. customer information from a SQL query and stores it in a DataTable. Here the order information is retrieved using a SQL query and the orders are restricted to U.S. customers only by joining to the customer table and using the WHERE clause. Then, the order detail information is retrieved using a SQL query that gets U.S. customers only by joining to the customer and order tables and by using the WHERE clause to get order details for U.S. customers only.

The fourth method retrieves all the U.S. customer information from a SQL query and stores it in a DataTable. Then, a query is executed for each customer, one at a time, to retrieve all of the customer's orders. This SQL query has a parameter for the Orders.CustomerID column which is reset each time while looping through the customers. Finally, a query is executed for each order detail to retrieve all of the order's order detail records.

This SQL query has a parameter for the [Order Details].OrderID column which is reset each time while looping through the orders. The problem with this method of retrieval is that the number of queries that will be executed could be enormous compared to the other methods. The query to retrieve the customers is executed only once, but the query to retrieve the orders is executed once for every customer. Furthermore, the query to retrieve the order details is executed once for every order. So if there were 10 customers that have 10 orders each (thus 100 total orders), the query to retrieve the orders would be executed 10 times and the query to retrieve the order details would be executed 100 times. This technique would require 111 queries to be executed (1 to get the customers, 10 to get the orders, and 100 to get the order details).

Each query is specifically geared to getting a particular customer's orders or a particular order's details, thus the queries are all likely to be optimized. Because of the number of queries to be executed and the amount of data going back and forth between the database and the application in those 111 trips, the results from my test were not a surprise when this method took the longest by far to load the DataTables within a DataSet. However, the performance results I gathered from my tests on the other three retrieval methods were much better and not far apart from each other.

The Test Scenario

My goal was to find the most efficient way to load a DataTable from a database query and display the results in a Web Form. The test runs each of the first three methods of retrieval I just discussed and takes the results and loads them into a corresponding set of ASP DataGrids. The test executes the queries, loads the data into a series of DataTables within a DataSet, binds the data to a series of DataGrids, and then repeats itself for a total of 100 repetitions. The process is repeated 100 times to evoke any anomalies or rule them out with a good degree of certainty. The total time (in milliseconds) to perform the data retrieval and bind the data to the DataGrids is summed and then averaged over the set of 100 repetitions. Finally, the average time (in milliseconds) to perform the tasks is displayed on the Web Form, along with the data displayed in the DataGrids. The test isn't perfect, of course, but all three methods of retrieval undergo the exact same test in the same environment, so a comparison of the results from each method of retrieval is certainly valid.

Figure 4 shows the HTML that the Web Form uses to display the results of the test as well as the rowsets from each retrieval method in several ASP DataGrids. The ASP Label control is used to display the average time (in milliseconds) each test took to execute. The DataGrid controls are used to display the rowsets from the last iteration of each test of each retrieval method.

Figure 4 Displaying the Test Results (Close Figure 4)
<%@ Page language="c#"
    Codebehind="WebForm1.aspx.cs"
    Inherits="FillingDataTables.WebForm1" %>
<html><body>
    <form id="Form1" method="post" runat="server">
        <asp:Label id="lblMessage" runat="server" Height="332px"
            Width="516px"></asp:Label>
        <br>

        <!-- Separate queries using subqueries-->
        <asp:DataGrid id="grdCustomer1" runat="server"></asp:DataGrid>
        <br>
        <asp:DataGrid id="grdOrder1" runat="server"></asp:DataGrid>
        <br>
        <asp:DataGrid id="grdOrderDetail1" runat="server"></asp:DataGrid>
        <br>

        <!-- 1 joined query -->
        <asp:DataGrid id="grdCustomer2" runat="server"></asp:DataGrid>
        <br>

        <!-- Separate queries using joins-->
        <asp:DataGrid id="grdCustomer3" runat="server"></asp:DataGrid>
        <br>
        <asp:DataGrid id="grdOrder3" runat="server"></asp:DataGrid>
        <br>
        <asp:DataGrid id="grdOrderDetail3" runat="server"></asp:DataGrid>
    </form>
</body></html>

The structure of the code for the test is a traditional tiered approach slimmed down a bit for our purposes, but still very much in line with tiered applications. The WebForm1.aspx page handles the display of the data and the test results. It has a codebehind file called WebForm1.aspx.cs, which contains all of the code (for all of the events) that interacts with WebForm1.aspx. It is in the code-behind that the test is initiated, timed, and reported. The code-behind calls a custom class named Customer.cs which contains three public methods worth noting: GetData_Using1Query, GetData_UsingSeparateQueriesUsingJoins, and GetData_UsingSeparateQueriesWithSubqueries.

Each method represents one of the data retrieval techniques. They all accept the user name and password for the Northwind database and create an empty DataSet to store the results. The methods then instantiate a connection to the database using the SqlConnection object. From this point, the different methods diverge as each method sets its SQL statements accordingly. As shown in Figure 5, the method GetData_UsingSeparateQueriesUsingJoins sets the SQL statement to retrieve the CustomerID and CompanyName for the customers in the U.S. A new instance of a SqlCommand object is created and initialized with the SQL statement using the SqlCommand object's one-argument constructor. Next, the CommandType is set and the SqlConnection object is linked to the SqlCommand object.

Figure 5 Using Separate Join Queries (Close Figure 5)
//-----------------------------------------------------------------------
// public Method
// Overloaded:    No
// Parameters:    string sUserName
//        string sPassword
// Return Value:    DataSet
//-----------------------------------------------------------------------
public DataSet GetData_UsingSeparateQueriesUsingJoins(string sUserName,
   string sPassword)
{
    this.m_sMethodName = "[public DataSet
                         GetData_UsingSeparateQueriesUsingJoins"  +
                         " (string sUserName, string sPassword)]";

    DataSet oDS = new DataSet();
    string sSQL;

    try
    {
        //---------------------------------------------------------------
        //-- Set up the Connection
        //---------------------------------------------------------------
        string sConn = "Data Source=(local);Initial
            Catalog=northwind;User ID=" +
            "sUserName + ";Password=" + sPassword + ";";
        SqlConnection oCn = new SqlConnection(sConn);

        //---------------------------------------------------------------
        //-- Set up the SELECT Command for the Customer
        //---------------------------------------------------------------
        sSQL = "SELECT CustomerID, CompanyName, City FROM Customers " +
                " WHERE Country = 'USA' " +
                " ORDER BY CustomerID";
        SqlCommand oSelCmd_Customer = new SqlCommand(sSQL);
        oSelCmd_Customer.CommandType = CommandType.Text;
        oSelCmd_Customer.Connection = oCn;

        //---------------------------------------------------------------
        //-- Set up the SELECT Command for the Orders
        //---------------------------------------------------------------
        sSQL = "SELECT o.CustomerID, o.OrderID, o.OrderDate  " +
            " FROM Customers c INNER JOIN Orders o ON c.CustomerID =
                o.CustomerID  " +
            " WHERE c.Country = 'USA' " +
            " ORDER BY o.CustomerID, o.OrderID";
        SqlCommand oSelCmd_Order = new SqlCommand(sSQL);
        oSelCmd_Order.CommandType = CommandType.Text;
        oSelCmd_Order.Connection = oCn;
        //---------------------------------------------------------------
        //-- Set up the SELECT Command for the Order Details
        //---------------------------------------------------------------
        sSQL = "SELECT od.OrderID, od.ProductID, od.UnitPrice,
            od.Quantity  " +
            " FROM Customers c INNER JOIN Orders o ON c.CustomerID =
                o.CustomerID  " +                "    INNER JOIN [Order
                Details] od ON o.OrderID = od.OrderID  " +
            " WHERE c.Country = 'USA' " +
            " ORDER BY o.CustomerID, o.OrderID, od.ProductID";
        SqlCommand oSelCmd_OrderDetail = new SqlCommand(sSQL);
        oSelCmd_OrderDetail.CommandType = CommandType.Text;
        oSelCmd_OrderDetail.Connection = oCn;

        //---------------------------------------------------------------
        //-- Create and set up the DataAdapters
        //---------------------------------------------------------------
        SqlDataAdapter oDA_Customer = new SqlDataAdapter();
        SqlDataAdapter oDA_Order = new SqlDataAdapter();
        SqlDataAdapter oDA_OrderDetail = new SqlDataAdapter();
        oDA_Customer.SelectCommand = oSelCmd_Customer;
        oDA_Order.SelectCommand = oSelCmd_Order;
        oDA_OrderDetail.SelectCommand = oSelCmd_OrderDetail;

        //---------------------------------------------------------------
        //-- Get the data
        //---------------------------------------------------------------
        oDA_Customer.Fill(oDS, "Customer");
        oDA_Order.Fill(oDS, "Order");
        oDA_OrderDetail.Fill(oDS, "OrderDetail");

        //---------------------------------------------------------------
        //-- Dispose of objects
        //---------------------------------------------------------------
        oCn.Dispose();
        oSelCmd_Customer.Dispose();
        oSelCmd_Order.Dispose();
        oSelCmd_OrderDetail.Dispose();
        oDA_Customer.Dispose();
        oDA_Order.Dispose();
        oDA_OrderDetail.Dispose();
    }
    catch (Exception ex)
    {
        //---------------------------------------------------------------
        //-- Throw the exception
        //---------------------------------------------------------------
        LogAndThrow(ex, sUserName, this.m_sClassName, this.m_sMethodName,
            "my err description");
    }
    finally
    {
    }

    //--------------------------------------------------------------------
    //-- Return the DataSet
    //--------------------------------------------------------------------
    return oDS;
}

This process is repeated for both of the other SQL statements that retrieve the orders and order details. The method uses three SqlCommand objects and three SqlDataAdapter objects: one for the customers, one for the orders, and one for the order details. Once all of the SqlCommand objects have been instantiated, have had their SQL statements set, and have been linked to the SqlConnection, the three SqlDataAdapter objects are created. The SqlCommand objects are then set to the SelectCommand property of their corresponding SqlDataAdapter object. For example, the SqlCommand object that retrieves the customers (oSelCmd_Customer) is set to the SqlDataAdapter object's (oDA_Customer) SelectCommand property, and so on. The SqlDataAdapter has four command object properties to correspond to each of the four main data retrieval and manipulation SQL statements: InsertCommand, UpdateCommand, DeleteCommand, and SelectCommand. Since this code will only retrieve data, only the SelectCommand is required to be set.

Next, the Fill method for each of the three SqlDataAdapter objects is called, resulting in a trip to the database and the execution of each of the SQL statements separately. The data is then returned to the DataSet that was created at the beginning of the public method GetData_UsingSeparateQueriesUsingJoins and stored in the Customer, Order, and OrderDetail DataTables within the DataSet. The code then relates the DataTable objects to each other with DataRelation objects. This is only done in the two methods that retrieve three separate rowsets since the other method only retrieves a single rowset.

The code snippet shown in Figure 6 loops 100 times to retrieve the data from the Customer class. Each DataTable is bound to an ASP DataGrid and the elapsed time (in milleseconds) is tallied. Finally, the average elapsed time is calculated and displayed in an ASP Label on the Web Form.

Figure 6 Repeating the Data Retrieval 100 Times (Close Figure 6)
lblMessage.Text += "GetData_UsingSeparateQueriesUsingJoins<br>";
fAvg = 0;
for(i = 0; i < 100; i++)
{
    x = System.DateTime.Now;
    oDS = oCustomer.GetData_UsingSeparateQueriesUsingJoins("sa", "");
    grdCustomer3.DataSource = oDS.Tables["Customer"].DefaultView;
    grdCustomer3.DataBind();
    grdOrder3.DataSource = oDS.Tables["Order"].DefaultView;
    grdOrder3.DataBind();
    grdOrderDetail3.DataSource = oDS.Tables["OrderDetail"].DefaultView;
    grdOrderDetail3.DataBind();
    y = System.DateTime.Now;
    z = y - x;
    fAvg += z.Milliseconds;
}
lblMessage.Text += "Average Ms = " + (fAvg / i).ToString() ;
lblMessage.Text += "<hr>";

Test Results

After running this test several times over, the results showed that returning the data in a single joined query was consistently the poorest performing technique. The left side of Figure 7 shows the test results for the U.S. customers, their orders, and order details. Gathering the data using one joined query took three milleseconds longer than the other techniques, resulting in approximately a 16 percent slowdown over the next closest technique.

Figure 7 Test Results in Milliseconds (Close Figure 7)

Query | U.S. Customers | All Customers
Subquery | 18.52 | 123.47
Join | 18.81 | 138.56
One Query | 21.87 | 156.02

Consider that returning the data in a single joined query means gathering the data and retrieving what could amount to a significant redundancy. For example, the company name "Save-a-lot Markets" is repeated over 50 times in the rowset, once for every order and its order detail. That's a lot of data to be stored, transported, and rendered to a browser. It may not seem like a lot, but it is certainly more data than if the customer's company name was returned just once. While three milliseconds is not a ton of time by any means, keep in mind that when these tests are extrapolated into larger, real-world scenarios, the performance can really make a difference.

For example, while the Northwind database doesn't store as much information as you might have in a real-world scenario, let's take a look at what happens when all customers are retrieved—not just those from the U.S. (see the right side of Figure 7). Notice that the gaps still exist in the techniques, with the single query taking approximately 12 percent longer than the next fastest technique.

Keep in mind that the single query is only executing one query while the other techniques are executing three queries each. Despite running more queries, the other techniques consistently complete faster than the single joined query. This behavior demonstrates that running several optimized queries is better than running a single unoptimized query in most scenarios. This could also be interpreted to mean that the queries themselves are inconsequential and rendering and transporting the data is the real slowdown. This is because the single query returns more data than the three separate queries.

On the flip side, using a single joined query can be very useful at times. One such time is when the application requires the data to be shown in a single nonhierarchical grid. For the most part, I run a single query when I want to display data in a single venue like an ASP DataGrid. However, when an application requires reading and writing to several tables in an underlying database, I prefer using separate queries that load separate related DataTable objects due to the built-in flexibility ADO.NET provides.

The test could be altered in many ways, one of which could be to omit the DataRelation objects altogether. I ran a few tests without the DataRelation objects and found that using the DataRelations tacked on about one millisecond to the U.S. customers column results shown in Figure 7 for each of the methods that executed three queries. (Of course, the method that executes the single joined query does not require a DataRelation, so it was unaffected.) Thus, removing the DataRelation improved the performance of running the three separate queries that used either subqueries or joins because of how long ADO.NET takes to apply the relations.

You might conclude that if running a single joined query was slower than running separate queries, that the slowdown was mostly in joining the data. If you take a look at the code from the method that runs three queries using joins, you will notice that the same joins used in the single query are used. The biggest difference is actually in the fields being returned. The single query returns all fields in a single SQL statement, while the separate queries return only the fields required for each query. The customer query only goes to the customer table and gets the customer fields. The order query goes to the customer and order table and only gets the order fields. The order details query goes to all three tables but only gets the order detail fields. With all this in mind, this series of tests show that the joins did have an effect on performance but that the fields being returned (and the redundancy of them) likely had a greater impact on the performance.

The best performing method was the three queries that used subqueries. The code for this technique can get rather cumbersome since the third-level query has to run a subquery that runs a subquery to get its results. Since the results from running the three queries using subqueries was very close to the three queries using joins, I prefer the latter technique since the code is much easier to read and maintain. After all, performance is only one, albeit major, factor in proper application implementation. Maintainability, scalability, flexibility, reliability, and consistency are all very important as well.

Wrap-up

The code I've provided demonstrates how easy it is to retrieve data into ADO.NET DataSet objects using various techniques. Feel free to download the test code from this column and modify it when exploring more testing scenarios (see the link at the top of this article). The key point to remember is to consider carefully how you will architect your applications. Running these tests not only revealed the performance of the various methods, but it also revealed how easy the code was to write, modify, understand, and take to the next level. I highly recommend that when developing any application, you take time to run test scenarios for the various moving parts of your application including data retrieval, reporting, modifications, screen generation, event processing, and other major application aspects. If the application is tested well, you'll have fewer surprises and a more robust application.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive. You can reach him at mmdata@microsoft.com.


The XML Files: Advanced Type Mappings

XML Files
Advanced Type Mappings
Aaron Skonnard

Code download available at: XMLFiles0306.exe (157KB)

Q Can XmlSerializer deal with choice compositors?

A XmlSerializer can map XML Schema's sequence and all compositors quite logically to Microsoft® .NET Framework class definitions, but choice requires a bit of special attention. Xsd:choice indicates that a single choice of numerous particles is allowed at a given location within a complex type. For example, consider the XML Schema complex type definition shown in Figure 1. This definition states that an instance may contain one of three elements: commission, hourly, or salary, as illustrated here:

<paymentMethod xmlns="http://example.org/xmlserializer">
   <hourly>7.50</hourly>
</paymentMethod>

Figure 1 XML Schema Complex Type Definition (Close Figure 1)
<xs:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
   targetNamespace="http://example.org/xmlserializer"
   elementFormDefault="qualified"
>
<xs:complexType name="PaymentMethod">
   <xs:choice>
       <xs:element name="commission" type="xs:double" />
       <xs:element name="hourly" type="xs:double" />
       <xs:element name="salary" type="xs:double" />
    </xs:choice>
</xs:complexType>
<xs:element name="paymentMethod" type="PaymentMethod"/>
•••

Although this type of constraint is quite common in XML documents, it's not supported across many programming languages. It would be quite easy to map something like this to a language that supports C-style type unions since they define a structure that is a choice of several data members, as illustrated by the following:

union PaymentMethod
{
   double commission;
   double hourly;
   double salary;
};
PaymentMethod paymentMethod;

However, since C# doesn't support unions it requires an out-of-band technique. If you run the aforementioned schema definition through xsd.exe (with the /classes switch), it generates a class like the one shown in Figure 2.

Figure 2 PaymentMethod Class (Close Figure 2)
using System.Xml.Serialization;
using System.Xml.Schema;
•••
public class PaymentMethod {

   [XmlElement("commission", typeof(System.Double))]
   [XmlElement("hourly", typeof(System.Double)]
   [XmlElement("salary", typeof(System.Double)]
   [XmlChoiceIdentifier("ItemElementName")]
   public System.Double Item;

    [XmlIgnore]
    public ItemChoiceType1 ItemElementName;
}

Notice that the PaymentMethod class only contains two members: Item and ItemElementName. The combination of these two elements represents an instance of the choice type. Since all three of the particle choices were declared as type xsd:double, the Item member is defined as type System.Double, which seems reasonable since that's the only possible type in this case.

The Item member is annotated with four attributes, three of type XmlElementAttribute and one of type XmlChoiceIdentifier. The XmlElementAttribute declarations specify the different name/type possibilities for the member when serialized. The XmlChoiceIdentifier attribute indicates that this particular member represents a choice compositor. However, simply looking at the value doesn't enable the serialization engine to unambiguously determine which element should be used. It needs a little bit more information. Hence, the XmlChoiceIdentifier specifies the name of another class member that contains the name of the actual element to use during serialization.

The ItemElementName member is of type ItemChoiceType, which is an enumeration type defined as follows:

•••
[XmlTypeAttribute(
  Namespace="http://example.org/xmlserializer",
  IncludeInSchema=false)]
public enum ItemChoiceType {
    commission,
    hourly,
    salary,
}
•••
Notice that ItemChoiceType is marked with IncludeInSchema=false to prevent it from being exposed by xsd.exe, again, it's only used for dealing with the choice. So when you instantiate a new PaymentMethod object and assign the Item field a value, you also need to specify which member of the choice you're actually using. ItemElementName is annotated with XmlIgnoreAttribute, preventing it from being serialized into the instance by XmlSerializer. Again, it's simply there to help XmlSerializer figure out how to properly serialize the Item field. Check out the example that is shown in Figure 3.

Figure 3 Serialize (Close Figure 3)
using System;
using System.IO;
using System.Xml.Serialization;
•••
static void Serialize()
{
    PaymentMethod p = new PaymentMethod();
    p.Item = 7.5;
    // specify which member we're actually using
    p.ItemElementName = ItemChoiceType.hourly;

    XmlSerializer s = new XmlSerializer(typeof(PaymentMethod));
    FileStream fs = new FileStream("p.xml", FileMode.Create);
    s.Serialize(fs, p);
    fs.Close();
}

The code in Figure 3 generates an XML document named p.xml, which contains the following:

<?xml version="1.0"?>
<paymentMethod xmlns="http://example.org/xmlserializer">
  <hourly>7.5</hourly>
</paymentMethod>

Then, of course, when you're deserializing PaymentMethod objects, you can inspect the ItemElementName field to figure out which choice particle was used in the instance document:

static void Deserialize()
{
    XmlSerializer s = new XmlSerializer(typeof(PaymentMethod));
    FileStream fs = new FileStream("p.xml", FileMode.Open);
    PaymentMethod p = (PaymentMethod)s.Deserialize(fs);
    fs.Close();

    Console.WriteLine("{0}: {1}", p.ItemElementName, p.Item);
}

This turns out to be quite handy when working with choices of different types. In the earlier example I showed, each particle was of type xsd:double. Suppose, however, that you extend it to contain an additional particle of a different type:

<xs:complexType name="PaymentMethod">
   <xs:choice>
       <xs:element name="commission" type="xs:double" />
       <xs:element name="hourly" type="xs:double" />
       <xs:element name="salary" type="xs:double" />
       <xs:element name="other" type="xs:string" />
    </xs:choice>
</xs:complexType>
<xs:element name="paymentMethod" type="PaymentMethod"/>

If you regenerate the class definition using xsd.exe, you'll notice that the Item field is now declared as type object instead of a double, which makes sense since it can contain either a string or a double now. In situations like this, the ItemElementName method helps you figure out what type to expect in the Item field (see Figure 4).

Figure 4 Deserialize (Close Figure 4)
static void Deserialize()
{
    •••
    PaymentMethod p = (PaymentMethod)s.Deserialize(fs);
    switch(p.ItemElementName)
     {
       case ItemChoiceType.other:
          string items = (string)p.Item;
          •••
       default:
          double itemd = (double)p.Item;
          •••
      }
    •••
}

One last issue I'll touch on is how to deal with repeating choices. For example, let's consider the slightly modified PaymentMethod complex type definition code that contains a repeating choice, as illustrated in the following:

<xs:complexType name="PaymentMethod">
   <xs:choice maxOccurs="unbounded">
       <xs:element name="commission" type="xs:double" />
       <xs:element name="hourly" type="xs:double" />
This means that a repeating list of any of the particles (listed within choice) may be used within an instance of this type. If you regenerate the class definition with this change in place, you'll notice that the only difference is that the Item and ItemElementName fields are declared as arrays:
public class PaymentMethod {

•••
[XmlChoiceIdentifier("ItemsElementName")]
    public object[] Items;

    [XmlIgnore]
    public ItemsChoiceType[] ItemsElementName;
}

The fundamental method for dealing with choices remains the same. There is a separate object (in the Items array) for each instance of the choice and a corresponding ItemsChioceType value (in the ItemsElementname array) that indicates which particle was actually used. The arrays are associated with each other with respect to their index position.

As you can see, working with choices is more involved than dealing with the more common compositors, but it's workable if you understand the mapping and are willing to do a little extra work. You can download the complete sample illustrated here from the MSDN® Magazine Web site code page at the link at the top of this article.

Q Can XmlSerializer deal with mixed content models?

A Yes, XmlSerializer can also handle mixed content models to a degree. A mixed content model allows a combination of text and elements. For example, consider the following complex type with a mixed content model:

    <xs:complexType name="Employee" mixed="true">
        <xs:sequence>
            <xs:element name="id" type="xs:string" />
            <xs:element name="name" type="xs:string" />
        </xs:sequence>
    </xs:complexType>
    <xs:element name="employee" type="EmployeeType"/>

The mixed="true" attribute states that an instance of this type may contain text nodes between the allowed elements. For example, the following document contains a valid instance of the Employee type defined in the previous XML snippet:

<employee xmlns="http://example.org/xmlserializer">
  here is some text...
  <id>333-33-3333</id>
  here is some more...
  <name>Bob Smith</name>
  and here is even more...
</employee>

As with choice compositors, this type of XML structure doesn't map naturally to .NET Framework class definitions, but XmlSerializer does its best to deal with it. Running xsd.exe against the schema file generates the following class definition that attempts to handle mixed content:

public class EmployeeType {
    public string id;
    public string name;
    [XmlText]
    public string[] Text;
}

Notice that the element declarations map naturally to the id and name fields, but what about the extra text that may interleave with the elements? The only way to handle this is to provide a single bucket for holding all of the interleaving text nodes. That's the purpose of the Text field of type string[]. This field is annotated with XmlTextAttribute, indicating that it will contain the extra text nodes found within the document during deserialization. Note that you can only apply the XmlTextAttribute attribute to a single member of a class.

Interleaving text nodes encountered during deserialization are simply added to this array. For example, consider the deserialization code shown in this snippet:

XmlSerializer s2 = new XmlSerializer(typeof(Employee));
FileStream fs2 =
   new FileStream("employee.xml", FileMode.Open);
Employee e = (Employee)s2.Deserialize(fs2);
Console.WriteLine("id: {0}, name: {1}", e.id, e.name);
foreach (string t in e.Text)
    Console.WriteLine(t);
Running the mixed content example through the previous code produces the following console output:
id: 333-33-3333, name: Bob Smith
here is some text...
here is some more...
and here is even more...

As you can see, XmlSerializer enables you to get your hands on the interleaving text nodes, but there's no way to know exactly where they appeared in the original XML document. In many scenarios this can be a major drawback since the position of the text plays an important role in applications that commonly leverage mixed content models, such as document-oriented content management systems.

There are many XML structures that are perfectly reasonable in XML Schema but don't make much sense in the world of the .NET Framework (like choices or mixed content models). So if you're designing a system that relies heavily on the use of mixed content models, you're better off avoiding XmlSerializer altogether. Instead, you should use something like XSLT, which is well suited for processing document-oriented structures. You can download the complete sample from the MSDN Magazine Web site.

Q When using XmlSerializer, is it possible to force the value of a field to show up as the content of an XML element?

A Yes, through the XmlTextAttribute technique described in the previous question. For example, assume you have the following C# class definition

public class Employee
{
    public string id;
    public string name;
}
and you want to map it to the following XML document, where the id field maps to an XML attribute and the name field maps to the content of the element:
<Employee id="333-33-3333" xmlns="http://example.org/
    xmlserializer">Bob Smith</Employee>

To accomplish this you need to annotate the name field with XmlTextAttribute and the id field with XmlAttributeAttribute, as shown here:

public class Employee
{
    [XmlAttribute]
    public string id;
    [XmlText]
    public string name;
}

If you run the compiled assembly through xsd.exe to generate a schema definition for this class, it produces a complex type that derives by extension from xsd:string in order to attach some attributes to the value:

<xs:complexType name="Employee">
    <xs:simpleContent>
      <xs:extension base="xs:string">
        <xs:attribute name="id" type="xs:string" />
      </xs:extension>
    </xs:simpleContent>
  </xs:complexType>

This type definition basically states that the instance element will have an id attribute and a string value as its content. The infrastructure didn't define it as a mixed content model, like in the XmlTextAttribute examples from the previous question, because in this case the element's content is of the text-only variety. Had I included another field that wasn't mapped to an attribute, however, it would have had to create a complex type with a mixed content model like before.

Q I've noticed that XmlSerializer won't serialize objects that implement IDictionary by default. Is there any way around this?

A Unfortunately there's no way to make XmlSerializer function with IDictionary-derived objects since the infrastructure explicitly checks for IDictionary at run time and disables serialization. One way around this is to write a new class that wraps an IDictionary object and copies the values into an array of serializable objects. The XmlSerializer framework also has a hidden hook for writing custom serialization code. To take advantage of this hook you can implement an interface called IXmlSerializable for the object you want to serialize/deserialize.

When you instantiate an XmlSerializer object, the constructor usually generates a temporary assembly containing XmlReader and XmlWriter code for moving between object instances and XML documents. Before doing this, however, it first checks to see if the supplied type derives from IXmlSerializable and if so, it generates code to call into the IXmlSerializable members instead. In other words, if you implement IXmlSerializable, you completely bypass the automatic serialization process and have the opportunity to provide your own.

IXmlSerializable is covered in the official documentation, but the documentation states it's not intended for public use and provides no information beyond that. This indicates that the development team wanted to reserve the right to modify, disable, or even completely remove this extensibility hook down the road. However, as long as you're willing to accept this uncertainty and deal with possible changes in the future, there's no reason whatsoever you can't take advantage of it.

Let's walk through a complete example of custom serialization to illustrate how it all fits together. To begin, here's what the interface looks like:

interface IXmlSerializable
{
    XmlSchema GetSchema();
    void ReadXml(XmlReader reader);
    void WriteXml(XmlWriter writer);
}
ReadXml is called during deserialization, whereas WriteXml is called during serialization. The GetSchema method is called when the XML Schema is exported (either by using xsd.exe or via the .asmx WSDL generation process). To implement a class capable of serializing any IDictionary-derived object, you need to define a new class that implements IXmlSerializable and also wraps an IDictionary object, as illustrated in Figure 5.

Figure 5 IXmlSerializable Implementation (Close Figure 5)
public class DictionarySerializer : IXmlSerializable
{
    const string NS = "http://www.develop.com/xml/serialization";

    public IDictionary dictionary;

    public DictionarySerializer()
    {
        dictionary = new Hashtable();
    }
    public DictionarySerializer(IDictionary dictionary)
    {
        this.dictionary = dictionary;
    }

    public void WriteXml(XmlWriter w)
    {
        w.WriteStartElement("dictionary", NS);
        foreach (object key in dictionary.Keys)
        {
            object value = dictionary[key];
            w.WriteStartElement("item", NS);
            w.WriteElementString("key", NS, key.ToString());
            w.WriteElementString("value", NS, value.ToString());
            w.WriteEndElement();
        }
        w.WriteEndElement();
    }

    public void ReadXml(XmlReader r)
    {
        r.Read(); // move past container
        r.ReadStartElement("dictionary");
        while (r.NodeType != XmlNodeType.EndElement)
        {
            r.ReadStartElement("item", NS);
            string key = r.ReadElementString("key", NS);
            string value = r.ReadElementString("value", NS);
            r.ReadEndElement();
            r.MoveToContent();
            dictionary.Add(key, value);
        }
    }
    public System.Xml.Schema.XmlSchema GetSchema()
    {
        return LoadSchema();
    }
    •••
}

The WriteXml method generates an XML document capable of representing any dictionary object in the following form:

<dictionary
    xmlns="http://www.develop.com/xml/serialization">
    <item>
      <key>5</key>
      <value>Nathan</value>
    </item>
    <item>
      <key>4</key>
      <value>Michael</value>
    </item>
    •••
</dictionary>

The ReadXml method contains the necessary code for reading these documents and then converting the document back into a dictionary object form.

The following code illustrates how to serialize and deserialize a DictionarySerializer using XmlSerializer:

void Serialize(IDictionary d, Stream s)
{
  DictionarySerializer ds = new DictionarySerializer(d);
  XmlSerializer xs = new XmlSerializer(typeof(DictionarySerializer));
  xs.Serialize(s, ds);
}

IDictionary Deserialize(Stream s)
{
  XmlSerializer xs = new XmlSerializer(typeof(DictionarySerializer));
  return xs.Deserialize(s);
}

With this in place, it's now possible to serialize anything that derives from IDictionary such as Hashtable, SortedList, ListDictionary, or HybridDictionary, as shown in Figure 6.

Figure 6 IDictionary Serialization (Close Figure 6)
// serialize a Hashtable
FileStream fs = new FileStream("ht.xml", FileMode.Create);
Hashtable ht = new Hashtable();
ht.Add(1, "Aaron");
... // fill Hashtable
Serialize(ht, fs);
fs.Close();

// serialize a SortedList
FileStream fs2 = new FileStream("sl.xml", FileMode.Create);
SortedList sl = new SortedList();
sl.Add(1, "Aaron");
... // fill SortedList
Serialize(sl, fs2);
fs2.Close();

•••
// deserialize a Hashtable
FileStream fs3 = new FileStream("ht.xml", FileMode.Open);
Hashtable hash = Deserialize(fs3);
•••

Since the WebMethod framework also uses the XmlSerializer plumbing, it's now possible to return DictionarySerializer objects from WebMethods. For example, let's consider the WebMethod that returns a Hashtable through a DictionarySerializer wrapper as shown in the following code:

[WebMethod]
public DictionarySerializer GetHashTable()
{
    Hashtable ht = new Hashtable();
    ht.Add(1, "Aaron");
    ht.Add(2, "Monica");
    ht.Add(3, "Michelle");
    return new DictionarySerializer (h1);
}
If you execute this operation, the SOAP response looks something like the code shown here:
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetHashTableResponse
    xmlns="http://www.develop.com/xml/serialization/example">
      <GetHashTableResult>
        <dictionary
         xmlns="http://www.develop.com/xml/serialization">
          <item>
            <key>1</key>
            <value>Aaron</value>
          </item>
•••

As you can see, I was able to customize the serialization process by simply implementing IXmlSerializable's ReadXml/WriteXml members. The last thing I need to do is customize the schema generation functionality so that clients know what to expect ahead of time. The framework typically generates the schema automatically, but since I've customized the serialization process, only I know what the schema should look like. Hence, the infrastructure will call GetSchema to retrieve the appropriate schema from my code at run time. As long as I implement this correctly, clients will have access to everything they need through the normal schema generation hooks.

In my implementation of GetSchema, I returned an XML Schema definition loaded into a System.Xml.Schema.XmlSchema object (see Figure 7). You'll see these definitions if you use xsd.exe to export the schema from the assembly or if you browse to the .asmx file and inspect the documentation. Notice that even the documentation page for GetHashtable illustrates exactly what the client should expect to receive now (see Figure 8).

Figure 7 GetSchema (Close Figure 7)
<s:schema elementFormDefault="qualified"
  targetNamespace="http://www.develop.com/xml/serialization"
  xmlns:tns="http://www.develop.com/xml/serialization"
>
  <s:complexType name="DictionaryType">
    <s:sequence>
      <s:element maxOccurs="unbounded" name="item"
        type="tns:ItemType" />
    </s:sequence>
  </s:complexType>
  <s:complexType name="ItemType">
    <s:sequence>
      <s:element name="key" type="s:string" />
      <s:element name="value" type="s:string" />
    </s:sequence>
  </s:complexType>
  <s:element name="dictionary" type="tns:DictionaryType" />
</s:schema>
Figure 8 GetHashtable
Figure 8 GetHashtable

You can download a complete example from the MSDN Magazine Web page I cited earlier. Remember, this is an unsupported and undocumented area of the .NET Framework that may change in the future.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard is an instructor/researcher at DevelopMentor, where he develops the XML and Web Service-related curriculum. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000). Reach him at http://staff.develop.com/aarons.


Cutting Edge: ASP.NET Controls Templates

Cutting Edge
ASP.NET Controls Templates
Dino Esposito

Code download available at: CuttingEdge0306.exe (141KB)
Contents
The Common Ground of Web Controls
An HTML Chart Generator
The Rendering Engine
Designing a Data-bound Control
Resolving Data Sources
Designing a Templated Control

The simplest way to create a custom control in ASP.NET is to derive a new class from an existing control. Creating custom controls through inheritance this way is not really very different from creating any other derived class. Creating new ASP.NET controls from scratch, on the other hand, just may prove to be a more challenging undertaking.

When you need a Web server control and none of the existing ones meet your requirements, you can derive a new control from one of the base control classes—Control or WebControl. These classes provide all the basic properties and functionality of a Web server control and let you focus on the features you need.

From a high-level perspective, a Web server control is a software component that abstracts a piece of HTML markup and builds an object model on top of it. No matter how much programming power you have on the back end, remember that an ASP.NET control works within the boundaries of a Web page and, therefore can output only HTML and script code. If the functionality you have in mind cannot be obtained with a combination of HTML and client-side script, then I seriously doubt you will be able to make it work over the Web.

An ASP.NET custom control is a class with public methods, properties, and events that renders its state to an HTML text writer. The object model and the rendering engine are the key elements of any Web server control. In this column, I'll build an ASP.NET control from the ground up and then I'll expand it to support data binding and templates.

The Common Ground of Web Controls

The root class of all Web controls is System.Web.UI.Control. This class defines the properties, methods, and events common to all ASP.NET server controls. These include the methods and events that determine and govern the lifecycle of the control plus a few properties such as ID, UniqueID, Parent, ViewState, and the collection of child controls named Controls.

The WebControl class derives from Control and adds extra properties and methods, mostly concerning the user interface. These properties include ForeColor, BackColor, Font, Height, and Width. WebControl is the base class for the family of ASP.NET Web server controls, whereas Control serves as the base for all HTML controls.

If your control must render a rich user interface, then you should derive it from WebControl. If you are authoring a component that does not need to provide specific user interface features, your best bet for a base class is Control. Although these rules are appropriate in most cases, there might be situations in which other options are reasonable. For example, you can derive from Control if you want to provide a subset of the UI features, or when you combine multiple controls together to build a composite control. In this case, typical user interface properties such as BackColor or Font should be duplicated to let users address the background and the font of individual controls. In similar situations, you might want to get rid of the standard programming interface to roll your own.

Depending on the functionality of the control you're creating, you might have to implement some additional interfaces. Typically, a Web server control may implement any of the following three interfaces: INamingContainer, IPostBackDataHandler, and IPostBackEventHandler.

INamingContainer doesn't contain methods, but simply notifies the ASP.NET runtime that the control should be considered to be a naming container. Child controls contained within a naming container parent control have their UniqueID property prefixed with the ID of the container. The naming container, therefore, acts as a namespace and guarantees the uniqueness of the control within the specified naming scope. The use of the INamingContainer interface is essential if you're writing iterative or templated controls such as Repeater or DataList.

IPostBackDataHandler is needed whenever your control must examine postback data. If the user can execute actions that affect the state of the control, then you need to look into the data posted by the browser. For example, a TextBox control stores its configuration in the view state, but also needs to read what the user may have typed in. IPostBackDataHandler is also helpful if you need to raise events on the server based on changes to the posted data. Again, the TextBox is the perfect sample control; if the typed text changes between postbacks, the TextChanged event is raised.

IPostBackEventHandler serves to capture a client-side postback event (a click). Upon postback, the ASP.NET runtime first raises data change events, then looks for a server control whose UniqueID property matches the name of a posted value (for example, the name of the clicked button). If a match is found and the control implements IPostBackEventHandler, ASP.NET invokes the RaisePostBackEvent method on the control. RaisePostBackEvent is the only method defined on the IPostBackEventHandler interface. What a particular control does within the RaisePostBackEvent method can vary quite a bit. For example, the Button control—the typical control which would implement this interface—fires the Click event to the page.

Another important piece of the control's infrastructure is the HTML text writer, the companion class that generates HTML markup code. You seldom use such a writer if you derive the control from an existing one. However, whenever you build the control from the ground up you inevitably need to write the HTML output to the stream.

The HtmlTextWriter class writes HTML-specific characters and text on the logical canvas of an ASP.NET page. The class also provides formatting capabilities that ASP.NET server controls can take advantage of when rendering HTML content to clients. If you're familiar with XML writers (see my May 2003 article, "Real-World XML: Manipulate XML Data Easily with Integrated Readers and Writers in the .NET Framework"), then you won't find the programming model of HTML writers to be anything new.

An HTML Chart Generator

Several third-party controls provide highly specialized charting components for use with Web pages. In addition, the new features of the System.Drawing namespace make it possible for you to exploit GDI+ to create your own graphics and output them as images. If you don't need to create special types of charts (for example, pie or 3D charts), a lightweight but equally effective alternative exists—2D charts created with HTML tables. As long as you're content with a plain 2D bar chart, a pure HTML solution is possible and might even be considered somewhat snazzy and elegant.

An HTML bar chart control is made of a sequence of table elements with two rows and one cell. The height and colors of each row are calculated so that the output looks like a vertical gauge. Figure 1 lists the properties exposed by the control.

Figure 1 HTML Bar Chart Properties (Close Figure 1)

Property | Description
BackImageUrl | Indicates the URL of the image used to tile the background
BarCount | Gets the number of bars that form the chart
Caption | Title of the chart
CaptionTemplate | Template that represents the header of the chart; if not specified, Caption and SubTitle are used
ChartColor | Default color used to fill the bars in the chart
DataSource | Data source associated with the control
DataTextField | Field in the data source to use for the label of the bar
DataValueField | Field in the data source to use for the value of the bar
FooterTemplate | Template that represents the footer of the chart
Maximum | Maximum value the chart can represent; set to 100 by default
SubTitle | Subtitle of the caption; displays below the caption with a smaller font

In addition to the properties listed in Figure 1, the control will use ForeColor and BackColor as well as Width, Height, and Font. All of these are inherited directly from WebControl. Each bar in the chart is fully represented with a custom data structure called BarChartItem. Instances of this class are created during the data binding process. Chart items are stored in an internal collection (similar to the Items collection of many data-bound controls) that is persisted to the view state:

[Serializable]
protected class BarChartItem
{
public string Text;
    public float Value;
}

Note that the class must be marked as serializable if you want to cache it in the control's view state. More precisely, to be persisted in the view state, a type must either be serializable (marked with the Serializer attribute or implementing ISerializable) or have a TypeConverter object defined. Of the two possibilities, having a type converter is the most efficient from the view state's perspective. In the Microsoft® .NET Framework, type converters are a sort of lightweight and highly specialized serializer. Storing a serializable type that doesn't provide a converter in the view state results in slower code and may generate a significantly larger output. Type converters are also useful to give a design-time behavior to controls and to integrate them seamlessly with Visual Studio® .NET.

The view state is serialized using an object serialization format. This format is optimized for a few types, including primitive types, strings, arrays of primitives or strings, and HashTable types. As a result, you can either write a type converter or use any of these simple types. A control can also customize the way in which its property data is stored in the ViewState collection. To obtain this, the control must override the SaveViewState and LoadViewState methods on the Control class. These methods have the following signatures:

protected virtual object SaveViewState();
protected virtual void LoadViewState(object savedState);

The BarChart control persists its data source in the view state. I could have chosen another design, but a bar chart contains a few elements at most. After all, the DropDownList control works that way. The data source that you bind to a DropDownList control is saved to the view state, although not directly by the control. All items in the dropdown list are converted into ListItem objects and these objects are individually saved to the view state. If you think it over for a moment, you'll realize that in terms of internal structure and memory footprint, the DropDownList and BarChart controls are similar. Now that you understand my reasoning, take a look at parts of the source code for the new control, shown in Figure 2. Note that Figure 2 doesn't include the rendering code, nor does it contain data binding or templates.

Figure 2 BarChart (Close Figure 2)
public class BarChart : WebControl
{
    public BarChart() : base() {
        _barChartItems = new ArrayList();
        Items = _barChartData;

        Caption = "";
        SubTitle = "";
        BackImageUrl = "";
        BackColor = Color.White;
        ForeColor = Color.Black;
        ChartColor = Color.Orange;
        Maximum = 100;
        Font.Name = "verdana";
        Font.Size = FontUnit.Point(8);
    }

    public string BackImageUrl {
        get {return Convert.ToString(ViewState["BackImageUrl"]);}
        set {ViewState["BackImageUrl"] = value;}
    }

    public int BarCount {
        get {
            if (Items != null)
                return Items.Count;
            return 0;
        }
    }

    public Color ChartColor {
        get {return (Color) ViewState["ChartColor"];}
        set {ViewState["ChartColor"] = value;}
    }

    public string Caption {
        get {return Convert.ToString(ViewState["Caption"]);}
        set {ViewState["Caption"] = value;}
    }

    public string SubTitle {
        get {return Convert.ToString(ViewState["SubTitle"]);}
        set {ViewState["SubTitle"] = value;}
    }

    public float Maximum {
        get {return Convert.ToSingle(ViewState["Maximum"]);}
        set {ViewState["Maximum"] = value;}
    }

    private ArrayList _barChartItems;
    public void Add(string theLabel, float theValue) {
        BarChartItem bci = new BarChartItem();
        bci.Text = theLabel;
        bci.Value = theValue;

        // Copy data to the view state
        if (_barChartItems != null)
            _barChartItems.Add(bci);
        DataSource = _barChartItems;
    }

    protected virtual ArrayList Items {
        get {return (ArrayList) ViewState["Items"];}
        set {ViewState["Items"] = value;}
    }
}

The Add method lets you populate the control without resorting to complicated data binding techniques. You simply add values explicitly in much the same way you use <asp:listitem> elements with list controls, as shown here:

   BarChart1.Caption = "Total Sales";
   BarChart1.SubTitle = "(Far-East)";
   BarChart1.Add("Q1", 5.8f);
   BarChart1.Add("Q2", 12.9);
   BarChart1.Add("Q3", 10.1);
   BarChart1.Add("Q4", 8.4f);
   BarChart1.Maximum = 15;

This code produces the chart shown in Figure 3.

Figure 3 Bar Chart
Figure 3 Bar Chart

The Rendering Engine

The HTML structure of the BarChart control is a bit complex; it's comprised of a few nested tables (see Figure 4). The outermost table counts three rows—title, subtitle, and chart table. The chart table includes three rows—the chart, the separator, and the label. Each bar in the chart table is rendered as a standalone table with three rows and a single cell. The bottommost cell represents the value of the bar; the middle cell is the written value; the topmost cell is empty to denote what is left to the maximum.

Figure 4 HTML Structure of Bar Chart
Figure 4 HTML Structure of Bar Chart

The BarChart control renders its output by building a graph of Table objects and rendering the output to HTML using the RenderControl method. Note that from a performance perspective, building a graph of objects is not necessarily the most efficient way to generate the output of a Web control. However, a graph of objects results in much more manageable code. The outermost table object mirrors many of the base properties of the WebControl, including colors, borders, and font. The code in Figure 5 shows the main portion of the Render method.

Figure 5 Rendering the BarChart (Close Figure 5)
protected override void Render(HtmlTextWriter output)
{
    // Create the outermost table
    Table outer = new Table();
    outer.BorderColor = BorderColor;
    outer.BackColor = Color.White;
    outer.BorderStyle = BorderStyle;
    outer.BorderWidth = BorderWidth;
    outer.GridLines = GridLines.None;
    outer.CssClass = CssClass;
    outer.ForeColor = ForeColor;
    outer.Font.Name = Font.Name;
    outer.Font.Size = Font.Size;
    outer.BorderStyle = BorderStyle.Solid;

    // Create the caption row
    TableRow captionRow = new TableRow();
    if (!Caption.Equals(String.Empty))
    {
        TableCell captionCell = new TableCell();
        captionCell.ColumnSpan = BarCount;
        captionCell.HorizontalAlign = HorizontalAlign.Center;
        captionCell.Font.Bold = true;
        captionCell.Font.Size = FontUnit.Larger;
        captionCell.Text = Caption;
        captionRow.Cells.Add(captionCell);
    }
    outer.Rows.Add(captionRow);

    // Create the subtitle row
    TableRow subtitleRow = new TableRow();
    if (!SubTitle.Equals(String.Empty))
    {
        TableCell subtitleCell = new TableCell();
        subtitleCell.ColumnSpan = BarCount;
        subtitleCell.HorizontalAlign = HorizontalAlign.Center;
        subtitleCell.Font.Size = FontUnit.Smaller;
        subtitleCell.Text = SubTitle;
        subtitleRow.Cells.Add(subtitleCell);
    }
    outer.Rows.Add(subtitleRow);

    // Create the chart row
    if (Items != null)
    {
        TableRow chartRow = new TableRow();
        TableCell chartCell = new TableCell();

        CreateBarChart(chartCell);

        chartRow.Cells.Add(chartCell);
        outer.Rows.Add(chartRow);
    }

    // Render the output
    outer.RenderControl(output);
}

The values inherent in the individual bar are packed in the BarChartItem structure and are used to draw and configure the chart table. The ratio between the current and maximum values is conveniently rendered with percentages (0 to 100), which simplifies working with HTML tables. When finished, the new table is connected to the parent table's cell. Figure 6 contains the source code that creates the bars.

Figure 6 Creating the Bars (Close Figure 6)
protected virtual void CreateSingleChart(TableCell parent,
                                         BarChartItem bci)
{
    // Calculate the value to represent in a 0-100 scale
    float valueToRepresent = 100*bci.Value/Maximum;

    // Create the bar
    Table t = new Table();
    t.Font.Name = Font.Name;
    t.Font.Size = Font.Size;
    t.Width = Unit.Percentage(100);
    t.Height = Unit.Percentage(100);

    // The still-to-do area
    TableRow todoArea = new TableRow();
    todoArea.Height = Unit.Percentage(100-valueToRepresent);
    todoArea.Cells.Add(new TableCell());
    t.Rows.Add(todoArea);

    // The row with the value
    TableRow valueArea = new TableRow();
    valueArea.Height = 10;
    TableCell cell = new TableCell();
    cell.HorizontalAlign = HorizontalAlign.Center;
    cell.Text = bci.Value.ToString();
    valueArea.Cells.Add(cell);
    t.Rows.Add(valueArea);

    // The row with bar
    TableRow barArea = new TableRow();
    barArea.ToolTip = bci.ToolTip;
    barArea.Height = Unit.Percentage(valueToRepresent);
    barArea.BackColor = bci.BackColor;
    barArea.Cells.Add(new TableCell());
    t.Rows.Add(barArea);

    // Connect to the parent
    parent.Controls.Add(t);
}

The CreateSingleChart method is marked as protected and overridable to allow potential derived classes to modify how the control is rendered. Declaring protected and virtual methods is a good technique to allow customization from derived classes. As an alternative, you can consider firing events in much the same way the DataGrid and other iterative controls do. For example, you could fire a ChartTableCreated event when the chart table is created, or BarChartItemCreated when an individual bar has been created. Both events will pass references to tables and rows allowing users more specific customization, such as specifying more attractive styles or deciding chart colors on the fly according to runtime conditions.

To draw lines behind the bars to simulate a scale, you can specify a background image for the chart table. I used this trick to obtain the graphics in Figure 3. The picture doesn't have to be particularly large or complex; a simple transparent background and a one-pixel line would do the trick.

Designing a Data-bound Control

A data-bound control is primarily characterized by a data source property that references an enumerable object—that is, a type that implements ICollection or IEnumerable. A data-bound control also overrides the DataBind method and defines a few satellite properties to let developers better configure the mapping between data and control properties. The number and the type of these properties depend on the specific features you're implementing.

When overriding the DataBind method, a data-bound control must call the base implementation on the parent class, import data from the source, and prepare for rendering.

The typical name for the property that contains the data source is DataSource. Bear in mind that this is only a naming convention, not a requirement. The DataSource property represents the collection of data you want to display through the control. You can force it to be of any particular type that suits your needs; however, to comply with the .NET Framework standard, you should allow users to use an instance of a type that implements the IEnumerable or ICollection interface, an array, or a type that exposes the ITypedList or IListSource interface. DataTextField and DataValueField are string properties whose value is cached in the view state, as the following code shows:

public string DataTextField {
    get {return Convert.ToString(ViewState["DataTextField"]);}
    set {ViewState["DataTextField"] = value;}
}

public string DataValueField {
    get {return Convert.ToString(ViewState["DataValueField"]);}
    set {ViewState["DataValueField"] = value;}
}

The DataBind method represents the entry point in the binding mechanism of ASP.NET. The method is responsible for firing the DataBinding event and for preparing the control for rendering. To make sure that the DataBinding event regularly fires, you only have to issue a call to the base method:

public override void DataBind()
{
    // Call the base method
    base.DataBind ();

    // Import bound data into internal structures
    LoadBarChartData();
}
The LoadBarChartData method imports the bound data into the internal structure—the Items collection—where the Render method expects to find it (see Figure 7).

Figure 7 LoadBarChartData (Close Figure 7)
void LoadBarChartData() {
   IEnumerable data;
   data = ResolveDataSource(DataSource, "");
   if (data == null)
    return;

   // Enumerate the data items
   IEnumerator e = data.GetEnumerator();
   while (e.MoveNext()) {
      object dataItem = e.Current;
    string theLabel = (string) DataBinder.Eval(dataItem, DataTextField);
    float theValue = Convert.ToSingle(DataBinder.Eval(dataItem,
              DataValueField));
    Add(theLabel, theValue);
   }
}

The ResolveDataSource method is a helper routine that returns a generic object reference to let the code manage data through a common programming interface. The actual objects that you can use for binding can be quite varied indeed—arrays, collections, DataSet, DataTable, DataView. Not all of them, however, implement the required interfaces. The DataSet object, for instance, doesn't implement the IEnumerable interface, but contains data that can be enumerated and processed as a collection. The same can be said for the DataTable class. The DataView object, on the other hand, is a collection of data and can be referenced as an IEnumerable component.

The call to ResolveDataSource normalizes all possible differences between the available data sources and gives the data-bound control a chance to process data in a uniform manner. The code for the method gets an enumerator for the data source and walks its way through the data.

Information from individual records is extracted using the DataBinder.Eval method. The values of the DataTextField and DataValueField properties are used at this time to obtain the label and the value for the bar. The two objects are then passed to the Add method and copied into the bar chart items list.

Resolving Data Sources

All in all, as long as you're building only data-bindable controls and disregarding templates, the trickiest part of development is not how you bind the data, but how you resolve the data source type. The algorithm that resolves a data source object to an IEnumerable object reference can be written in many ways. The difference between implementations would be given by the order in which you check for types. The code in Figure 8 shows one possible way of proceeding.

Figure 8 Resolving the Data Source (Close Figure 8)
IEnumerable ResolveDataSource(object dataSource)
{
    IListSource listSource;
    IList list;

    if (dataSource == null)
        return null;

    // First try: IListSource (i.e., DataSet, DataTable)
    listSource = (dataSource as IListSource);
    if (listSource != null)
    {
        // Get an object that represent the list
        list = listSource.GetList();
        return list;
    }

    // Does the source implement IEnumerable?
    // (i.e., DataView, array, collections)
    if ((dataSource as IEnumerable) != null)
        return (IEnumerable) dataSource;

    return null;
}

The code first attempts to cast the data source object to IListSource. The interface is designed to allow objects that do not implement ICollection to return a bindable list of data for data-bound controls. In the entire .NET Framework, only two classes implement this interface: DataSet and DataTable. Objects that implement IListSource provide a method called GetList, which returns the internal list of data that makes the object a bindable source. The DataSet returns the contents of its DataViewManager property; the DataTable returns its default view, which is also contained by the DefaultView property.

In general, the GetList method can return a collection of IList objects. This is the case when a DataSet object is used as the data source and a data member property indicates which table is to be used. The ContainsListCollection Boolean property specifies whether the IList object is a simple list or a collection of lists. In the latter case, more work is needed to match the member name with the contained tables.

The DataView objects, as well as arrays and collections, are cast directly to IEnumerable. Notice the use of the C# as operator in Figure 8. The as operator is like a cast, except that it returns null if the conversion can't be accomplished. The operator never throws an exception. A similar operator doesn't exist in Visual Basic® .NET, but can be simulated using the TypeOf...Is construct.

Figure 9 Sample Bar Chart
Figure 9 Sample Bar Chart

You can now execute a query against a SQL Server database and bind the results to the new BarChart control. Let's consider the following SQL query:

SELECT e.lastname AS Employee, SUM(price) AS Sales FROM
(SELECT o.employeeid, od.orderid, SUM(od.quantity*od.unitprice) AS price
FROM Orders o, [Order Details] od
    WHERE Year(o.orderdate)=@TheYear AND od.orderid=o.orderid
    GROUP BY o.employeeid, od.orderid ) AS t1)
INNER JOIN Employees e ON t1.employeeid=e.employeeid
GROUP BY t1.employeeid, e.lastname
It returns a resultset of two columns—Employee and Sales. The Employees column contains the last name of the employee; the Sales column stores the employee's total sales for a given year. Here's how you bind that information to the control:
BarChart1.Maximum = 150000;
BarChart1.Caption = "Northwind Sales";
BarChart1.SubTitle = "(Year 1997)";
BarChart1.DataSource = ExecuteQuery(1997);
BarChart1.DataTextField = "Employee";
BarChart1.DataValueField = "Sales";
BarChart1.DataBind();
The DataSource property is set with the DataTable that results from the query execution. The DataTextField and DataValueField properties are set with the names of the fields to use for the label and the value. Make sure you set the Maximum property with an appropriate value. Figure 9 and Figure 10 show a sample in action.

Figure 10 Scale Factor 150,000
Figure 10 Scale Factor 150,000

Designing a Templated Control

In ASP.NET, you can import templates in two ways—through properties of type ITemplate or by dynamically loading Web user controls. If you employ user controls, the template is insulated in a separate file, which may or may not be an advantage. Having the template available right in the source code of the page greatly simplifies page authoring. Templates do not have to be assigned at design time, however. To load templates at run time you need an instance of a class that implements ITemplate.

Adding templates to the BarChart control allows for a more effective customization of those parts of the UI that the user controls. By default, the BarChart control contains a header and a footer with standard formats. The header is made of text displayed in two rows with slightly different font and size settings. The footer is limited to displaying labels at the base of each bar. If you want a richer caption bar, or want to add information that helps people reading the chart, the BarChart control isn't the answer.

What about using templated controls? A templated control must follow a few practical rules. First, it must implement the INamingContainer interface so that any child controls can be given a unique name. You should also pay attention to the ParseChildrenAttribute attribute, applying it to your control so that the page parser knows that all child tags are to be parsed:

[ParseChildren(ChildrenAsProperties = true)]
public class MyTemplatedControl : Control, INamingContainer
{
•••
}
However, if you derive the control from WebControl, using the ParseChildrenAttribute is unnecessary because the WebControl class is already marked with this attribute.

A templated control must have one or more properties of type ITemplate. A template property represents a collection of text and controls that is hosted within a container. The container is also responsible for exposing properties that page authors can use to create data-bound expressions. The following code snippet shows how to define a template property named CaptionTemplate:

[TemplateContainer(typeof(ChartTemplateContainer))]
public ITemplate CaptionTemplate {...}

The argument of the TemplateContainer attribute is the type of the container control in which you want the template to be instantiated. The container control is different from the templated control that you're building. A container that is independent from the main templated control lets you iterate the template many times using different data. The template container must be a naming container, too.

Finally, a templated control should override CreateChildControls and instantiate the template within the container. The container class is then added to the root control. A local event handler for the DataBinding event is also necessary to ensure that all child controls are created before the ASP.NET runtime attempts to process data-bound expressions. Again, note that overriding CreateChildControls is not mandatory, but certainly represents a programming best practice.

The <CaptionTemplate> property is aimed at providing a custom representation of the topmost part of the chart. If no template is specified, the caption is composed using the values of the Caption and the SubTitle properties. The CaptionTemplate property is declared and implemented as follows:

private ITemplate __captionTemplate = null;
[TemplateContainer(typeof(ChartTemplateContainer))]
public ITemplate CaptionTemplate
{
    get {return __captionTemplate;}
    set {__captionTemplate = value;}
}

The instantiation of the template takes place in the CreateChildControls overridden method:

private ChartTemplateContainer _captionTemplateContainer;
protected override void CreateChildControls()
{
   if (CaptionTemplate != null) {
      captionTemplateContainer = new ChartTemplateContainer(this);
      CaptionTemplate.InstantiateIn(_captionTemplateContainer);
      Controls.Add(_captionTemplateContainer);
   }
}

The method creates a new instance of the container and instantiates the template within. Next, the container is added to the Controls collection of the BarChart control. When the control is ready for rendering, the newly created instance of the template container class is linked to its physical container—the table cell that is located just above the bars:

if (CaptionTemplate != null)
    captionCell.Controls.Add(_captionTemplateContainer);

The template container is a wrapper control class used to decouple the root templated control—the BarChart in this case—from the classes behind template properties. Instead of instantiating the template directly within the root control, you use this intermediate class—the template container. In doing so, you gain the ability to repeat the template multiple times with different data.

The following is the source of the ChartTemplateContainer class that I'm using as a template for the CaptionTemplate property:

public class ChartTemplateContainer : WebControl, INamingContainer
{
   private BarChart __parent;
   public ChartTemplateContainer(BarChart parent) {
   _parent = parent;
   }

   public BarChart BarChart {
     get {return __parent;}
   }
}

You should derive the template container class from WebControl and make it implement the INamingContainer interface. (Note that being a naming container is necessary only if you're hosting multiple templates or controls of the same type.) The structure of the class is completely up to you—the one depicted earlier is reasonable. The constructor takes a reference to the parent control—the BarChart in this case—and another public property exposes that object as a whole or simply mirrors some of its properties. Note that only the public properties of the container class can be called from the data-bound expressions you may use within the template:

<CaptionTemplate>
<table width="100%"><tr><td>
<%# "<b>" + Container.BarChart.Caption + "</b>&nbsp;" +
          Container.BarChart.SubTitle %>
</td><td align="right">
<%# "Scale Factor <b>" + Container.BarChart.Maximum + "</b>"%>
</td></tr><table>
</CaptionTemplate>

The first cell displays the contents of the Caption and the SubTitle properties; the second cell is right-aligned and indicates the maximum displayable value—the Maximum property. You use the Container property of the template to obtain an instance of the ChartTemplateContainer class. Any public properties exposed by the class can be used in the data-bound expressions.

The output of the BarChart control may not be cool and attractive like a pie chart or a 3D chart, but it only requires HTML code, doesn't consume server resources to generate a picture and, more importantly, doesn't require a second round-trip to download the image. Try it out!

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. He is the author of Building Web Solutions with ASP.NET and ADO.NET and Applied XML Programming for .NET and is currently writing Programming ASP.NET—all from Microsoft Press. Get in touch with Dino at dinoe@wintellect.com.


Advanced Basics: Adding New Features with User Controls

Advanced Basics
Adding New Features with User Controls
Ken Spencer

Code download available at: AdvancedBasics0306.exe (156KB)

Q How can I draw a line on a Windows® Form that is selectable by the user and can respond to mouse events?

A In past versions of Visual Basic®, there were rudimentary graphics controls. In Visual Basic .NET you have the GDI+ library, which enables you to draw lines, circles, and most anything else. But how can you use the functionality of GDI+ to create lines and other graphics that respond to user mouse clicks and events?

To answer, I created a UserControl that implements a line drawn with GDI+. At first this seems like a trivial exercise, but it's fun because of the various properties you create to control a line and the testing you go through to make this work.

Figure 1 shows my test application using the new line control. You can see the line across the top of the form, and the status bar shows the results of clicking the line. You can fiddle with the line's size and other properties using this form by changing various options and clicking the Change Line Properties button.

Figure 1 Control Test App
Figure 1 Control Test App

Let's walk through the code used to implement this control. First, I created several enumerations to use for various line properties. The enumerations make it easy for users to enter a property value. They also strongly type the properties, making it very difficult to introduce errors by entering an incorrect value. The LineStyles enum controls the style of the line and the LineWeightStyles enum controls the weight (density) of the line:

Enum LineStyles
Solid = 0
    Hatched = 1
    TiledImage = 2
End Enum
Enum LineWeightStyles
    Small = 0
    Medium = 1
    Large = 2
    Graphic = 3
End Enum

Next, I added a set of private variables to handle the internals of size, line weight, color, and other such properties. These variables are shown in Figure 2.

Figure 2 Line Variables (Close Figure 2)
#Region "Private Variables"
    Private privateStart As Point = New Point(0, 0)
    Private privateEnd As Point = New Point(Me.width, 0)

    Private privateWeight As LineWeightStyles = _
     LineWeightStyles.Small
    Private privateWeightPixels As Integer = 2

    Private privateStyle As LineStyles = LineStyles.Solid
    Private privateColor As Color = Color.Black

    Private privateCurrentMouseX As Integer
    Private privateCurrentMouseY As Integer
#End Region

Then, I created several public properties that are used to control the line. These properties are shown in Figure 3. The LineWidth property is straightforward; it simply sets the width of the line in pixels. Notice that it also sets the width of the UserControl to make the control's size match the size of the line. LineColor determines the color of the line. The LineStyle property can be used to specify how the line is displayed. In this version, this property is not fully implemented since only the Solid and Hatched styles are supported. The LineWeight property controls the thickness of the line. This property is interesting because the weight controls not only the thickness of the line but also the height of the control. You can see how the value set using the enum is converted to a pixel value in the control. This allows you to present a meaningful value to the user such as Large and then you can internally set Large to, say, 15 pixels or so.

Figure 3 Public Properties (Close Figure 3)
#Region "Public Properties"
    Property LineWidth() As Integer
        Get
            Return privateEnd.X
        End Get
        Set(ByVal Value As Integer)
            privateEnd = New Point(Value, 0)
            If Me.Width <> privateEnd.X Then
                Me.Width = privateEnd.X
            End If
            If Me.Height <>
               privateEnd.Y + privateWeightPixels Then
                Me.Height = _
                  privateEnd.Y + privateWeightPixels
            End If
            RefreshMe()
            LogIt("LineWidth Set " & "end")
        End Set
    End Property

    Property LineColor() As Color
        Get
            Return privateColor
        End Get
        Set(ByVal Value As Color)
            LogIt("LineColor Set " & "start")
            privateColor = Value
            RefreshMe()
            LogIt("LineColor Set " & "end")
        End Set
    End Property

    Property LineStyle() As LineStyles
        Get
            Return privateStyle
        End Get
        Set(ByVal Value As LineStyles)
            LogIt("LineStyle Set " & "start")
            privateStyle = Value
            RefreshMe()
            LogIt("LineStyle Set " & "end")
        End Set
    End Property

    Property LineWeight() As LineWeightStyles
        Get
            Return privateWeight
        End Get
        Set(ByVal Value As LineWeightStyles)
            privateWeight = Value
            Select Case privateWeight
                Case LineWeightStyles.Small
                    privateWeightPixels = 2
                Case LineWeightStyles.Medium
                    privateWeightPixels = 6
                Case LineWeightStyles.Large
                    privateWeightPixels = 10
                Case LineWeightStyles.XLarge
                    privateWeightPixels = 15
                Case LineWeightStyles.XXLarge
                    privateWeightPixels = 20
                Case LineWeightStyles.Graphic
            End Select
            RefreshMe()
        End Set
    End Property

#End Region

The next bit of code in the class is the LineClick event. This event is fired when the user clicks the control. The definition for the event is shown in the following code:

#Region "Public Events"
    Public Event LineClick(ByVal ClickedLocation As Point)
#End Region

Now let's walk through creating the line in the Paint event (see Figure 4). The Paint event fires when the control is dirty and must be cleaned up—when another window covers the control or parent of the control. This event is a good place to add code to draw something on the control. The first line after the Try statement calls the ResetSize procedure, which sets the UserControl's size to the width and the height of the line.

Figure 4 Paint Event (Close Figure 4)
Private Sub MyLine_Paint(ByVal sender As Object, _
  ByVal e As System.Windows.Forms.PaintEventArgs) _
  Handles MyBase.Paint
    Dim LinePen As Pen
    Dim LineBrush As Brush
    Try
        ResetSize()
        Select Case privateStyle
            Case LineStyles.Solid
                Dim SolidBrush As New SolidBrush(privateColor)
                LinePen = New Pen(SolidBrush, privateWeightPixels)
                e.Graphics.DrawLine(LinePen, privateStart.X, _
                  privateStart.Y, privateEnd.X, privateEnd.Y)

            Case LineStyles.Hatched
                Dim HatchedBrush As New HatchBrush( _
                    HatchStyle.DiagonalCross, privateColor)
                LinePen = New Pen(HatchedBrush, privateWeightPixels)
                e.Graphics.DrawLine(LinePen, privateStart.X, _
                      privateStart.Y, privateEnd.X, privateEnd.Y)

                Case LineStyles.TiledImage
            End Select
        Catch exc As Exception
            Throw New _
              Exception("MyLine UserControl " & " - " & exc.Message)
        End Try
End Sub

Now let's look at the Select Case statement inside the Try block. The first Case statement draws the solid line:

Select Case privateStyle
Case LineStyles.Solid
This line creates a new solid brush in the color selected by the user:
Dim SolidBrush As New SolidBrush(privateColor)
The next line creates a new instance of a pen with the solid brush just created and the weight selected by the user:
LinePen = New Pen(SolidBrush, privateWeightPixels)
Next, draw the line by calling DrawLine:
e.Graphics.DrawLine(LinePen, privateStart.X, privateStart.Y, _
    privateEnd.X, privateEnd.Y)

Now let's look at the Hatched line. Think of it as a little lesson in using GDI+ and a few aspects of the Microsoft® .NET Framework. Adding support for the hatched line only requires a few tweaks. The enum for the hatched style is already in place, so you only need to add support for this style in the Paint event. The first step in creating the hatched style is to create a HatchBrush instead of a SolidBrush:

Dim HatchedBrush As New HatchBrush(HatchStyle.DiagonalCross, privateColor)

The HatchStyle allows you to select a variety of options. This example uses a hardcoded style, but of course you can easily add an enum and property for the hatched style and implement a property for it. Then you simply change the HatchStyle to point to that variable. The rest of the code is the same as the Solid line:

LinePen = New Pen(HatchedBrush, privateWeightPixels)
e.Graphics.DrawLine(LinePen, privateStart.X, privateStart.Y, _
    privateEnd.X, privateEnd.Y)

Now, let's look at the code that handles the click event. The first thing to do is track where the user clicks the mouse. This is handled in the MouseMove event, which picks up the x and y coordinates of the clicked location:

Private Sub MyLine_MouseMove(ByVal sender As Object, _
    ByVal e As System.Windows.Forms.MouseEventArgs) _
    Handles MyBase.MouseMove
 privateCurrentMouseX = e.X
 privateCurrentMouseY = e.Y
End Sub

When the user clicks the line, they are really just clicking the UserControl, so the actual click is processed by the Click event for the UserControl.

Private Sub MyLine_Click(ByVal sender As Object, _
    ByVal e As System.EventArgs) Handles MyBase.Click
RaiseEvent LineClick(New Point(privateCurrentMouseX, _
    privateCurrentMouseY))
End Sub
As you can see, the LineClick event is fired in the Click event for the UserControl. The already set mouse coordinates are then passed to this event.

Figure 5 shows the key event procedures from the form which uses the new line control, FormTwo.vb. The form load event checks several properties in the line control (MyLine1) and modifies the interface as necessary for those controls. The cmdChange_Click sets the control from properties set on the form. This will repaint the control with the new values. The exception to this is the color picker, which repaints the control by resetting the LineColor property directly.

Figure 5 Selected Events from FormTwo.vb (Close Figure 5)
Private Sub frmTwo_Load(ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles MyBase.Load

    numEnd.Value = MyLine1.LineWidth
    lblLineColor.Text = MyLine1.LineColor.ToString
    Select Case MyLine1.LineWeight
        Case LineWeightStyles.Small
            rdoWeightSmall.Checked = True
        Case LineWeightStyles.Medium
            rdoWeightMedium.Checked = True
        Case LineWeightStyles.Large
            rdoWeightLarge.Checked = True
        Case LineWeightStyles.XLarge
                rdoWeightXLarge.Checked = True
        Case LineWeightStyles.XXLarge
            rdoWeightXXLarge.Checked = True
        End Select

        If MyLine1.LineStyle = _LineStyles.Hatched Then
            chkStyleHatched.Checked = True
        Else
            chkStyleHatched.Checked = False
        End If
End Sub
Private Sub cmdChange_Click(ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles cmdChange.Click
    MyLine1.LineWidth = CInt(numEnd.Value)

    If rdoWeightSmall.Checked Then
        MyLine1.LineWeight = LineWeightStyles.Small
    ElseIf rdoWeightMedium.Checked Then
        MyLine1.LineWeight = LineWeightStyles.Medium
    ElseIf rdoWeightLarge.Checked Then
        MyLine1.LineWeight = LineWeightStyles.Large
    ElseIf rdoWeightXLarge.Checked Then
        MyLine1.LineWeight = LineWeightStyles.XLarge
    ElseIf rdoWeightXXLarge.Checked Then
        MyLine1.LineWeight = LineWeightStyles.XXLarge
    End If

    If chkStyleHatched.Checked Then
        MyLine1.LineStyle = LineStyles.Hatched
    Else
        MyLine1.LineStyle = LineStyles.Solid
    End If

    StatusBar1.Text = "Weight is " & MyLine1.LineWeight.ToString
End Sub

Private Sub cmdSelectColor_Click(ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles cmdSelectColor.Click
    Dim oDialog As New ColorDialog()
    oDialog.ShowHelp = True

    If (oDialog.ShowDialog() = DialogResult.OK) Then
        MyLine1.LineColor = oDialog.Color
        lblLineColor.Text = MyLine1.LineColor.ToString
    End If
End Sub

Private Sub MyLine1_LineClick( _
  ByVal ClickedLocation As System.Drawing.Point) _
    Handles MyLine1.LineClick
    StatusBar1.Text = "You clicked the line at " & _
      ClickedLocation.ToString
End Sub

The most important code in FormTwo that is related to your question is the MyLine1_LineClick shown in the following code:

Private Sub MyLine1_LineClick( _
ByVal ClickedLocation As System.Drawing.Point) Handles MyLine1.LineClick
    StatusBar1.Text = "You clicked the line at " & _
        ClickedLocation.ToString
End Sub
This implements the LineClick event from MyLine, which fires when the line is clicked. This event simply takes the location that's clicked and shows it in the status bar.

That's it. You now have a line control that can show either a solid or hatched horizontal line.

As mentioned earlier, you are not limited to a line control using this technique. You can build anything—such as a rectangle, circle, curve, sophisticated graphics, and so on. All of these controls can respond to any mouse event you want to implement because you can intercept the click to the UserControl just as this control does.

The only trick in implementing many other kinds of graphics will be how you handle non-rectangular images and other shapes. Let's say you need to implement a diagonal line. To do so you'll need to handle the non-rectangular shape in order to prevent the control from blocking other controls that are behind it and outside of the drawn area. The second issue concerns oddly shaped graphics. For non-horizontal and non-vertical shapes, you must calculate where on the control the user clicked and possibly check that a point is contained within a region. If you try this yourself, you'll see that it's pretty straightforward, but more difficult than implementing simple horizontal and vertical shapes.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencer works for 32X Tech (http://www.32X.com), where he provides training, software development, and consulting services on Microsoft technologies.


The ASP Column: Tree Controls with XSL

ASP Column
Tree Controls with XSL
George Shepherd

Code download available at: ASPColumn0306.exe (128KB)
Contents
Tree Controls
TreeView and the Designer
The TreeNode
Node Types
Dynamic TreeViews
TreeViews and XML
Building a Directory Tree
TreeView Events
Node Styles
Conclusion

In my last column I looked at a couple of the Microsoft® Internet Explorer WebControls available from MSDN®—specifically the TabStrip, the MultiView, and the PageView server-side controls. These three controls organize data using tabbed pages (see The ASP Column in the February 2003 issue of MSDN Magazine). This time I'll take a look at the TreeView server-side control for displaying hierarchical data.

Tree Controls

Tree-style controls appear all over the place on desktop applications. For example, the classic view of Windows® Explorer displays directory information using a tree control. Displaying data as a tree with collapsible nodes is useful for drilling down into data that's structured hierarchically.

To view the contents of a directory, click on the plus sign (+) next to the directory node and it expands. There's no reason this kind of user interface can't be provided on a Web page—and that's what the TreeView WebControl does.

To download the TreeView control (and the rest of the Internet Explorer server-side controls), go to Internet Explorer Web Controls Download Packages. Note that these controls are not supported by Microsoft, but full source code is available. Place the controls that you want in the toolbox by selecting Tools | Customize Toolbox in Visual Studio® .NET.

TreeView and the Designer

Like the other Internet Explorer controls, the TreeView includes full designer support. Manipulating the TreeView control from the designer is like most other designer-based controls. The designer offers a visual representation of the control as it might appear on the client browser. The designer also presents a list of properties to tweak for managing the tree control.

To use the TreeView control in your app, just get one from the toolbox and drop it into a form. Once a TreeView control is in the designer, the properties can be easily managed through the property window. Figure 1 shows a static TreeView control as it first appears in the designer.

Figure 1 TreeView on a Form
Figure 1 TreeView on a Form

By default, the TreeView has no nodes. To make the control work, you'll need to add some, either programmatically or through the designer. The Designer support for the TreeView includes a dialog box for defining node types. Adding node types and nodes is really quite straightforward. Figure 2 shows the source code produced by the designer after you have inserted several nodes into an ASPX page.

Figure 2 Displaying the TreeView (Close Figure 2)
<iewc:TreeView id="TreeView1"
      style=
      "Z-INDEX: 101; LEFT: 31px; POSITION: absolute; TOP: 22px"
      runat="server"
      Height="148px" Width="292px">
  <iewc:TreeNode Text="First Level" Expanded="True">
    <iewc:TreeNode Text="Second Level" Expanded="True">
      <iewc:TreeNode Text="Third Level"></iewc:TreeNode>
      <iewc:TreeNode Text="Third Level"></iewc:TreeNode>
    </iewc:TreeNode>
    <iewc:TreeNode Text="Second Level">
      <iewc:TreeNode Text="Third Level"></iewc:TreeNode>
      <iewc:TreeNode Text="Third Level"></iewc:TreeNode>
    </iewc:TreeNode>
  </iewc:TreeNode>
  <iewc:TreeNode Text="First Level">
    <iewc:TreeNode Text="Second Level">
      <iewc:TreeNode Text="Third Level"></iewc:TreeNode>
    </iewc:TreeNode>
  </iewc:TreeNode>
</iewc:TreeView>

The TreeNode

An empty TreeView by itself is pretty useless—any tree-based structure requires nodes. TreeView nodes are represented by the TreeNode class, which includes several standard and self-explanatory properties such as Text and Checkbox. In addition, TreeNode includes cascading style sheets (CSS) to define styles (I'll look at those later), image URLs for associating graphics with particular nodes (for example, directory nodes versus file nodes), a navigation URL (for redirecting when clicking on a node), and a data string to be associated with a node. The navigation URL is useful for managing hierarchical data that can be represented by URIs.

The TreeNode includes various methods for programmatic manipulation as well. For example, the TreeNode class includes methods for adding new nodes to a tree. I'll explain how that's done when creating a TreeView programmatically.

Node Types

The TreeNode class collects properties for representing an individual node. While it's possible to define each of the nodes with raw data, it's often useful to set up default values for different kinds of nodes. The TreeView lets you create your own TreeNode types for a TreeView. TreeNode types define default properties for TreeNode objects. For example, in setting up a TreeView for viewing a directory structure, you'll probably want to have the file and the directory nodes to be represented by different images. Defining TreeNode types lets you associate default images for the file nodes and one for the directory nodes.

Dynamic TreeViews

While building a TreeView using the designer is straightforward, using only the designer will produce a static TreeView. If you know what your data is going to be like ahead of time, that's fine. However, in most cases you know very little about the data you're representing until run time. For example, representing a directory structure through a TreeView should be done dynamically, as the contents of a directory change frequently. To see how to manage a TreeView dynamically, let's look at a Web-based directory explorer intended for viewing project source code through a browser.

Figure 3 Project Explorer
Figure 3 Project Explorer

The Project Explorer includes two TreeViews—one for categorizing project types and another for viewing the directory structure for each project. Finally, the application includes a read-only textbox for displaying the contents of source code files. Figure 3 shows the Project Explorer in action.

TreeViews and XML

One way to populate a TreeView at run time is to use XML to represent the hierarchical data. The format of the XML includes a TREENODES container and an arbitrarily deep hierarchy of elements. Figure 4 shows some XML for populating the TreeView of the current project. The top-level available project categories include several project nodes each—and each project node includes a string representing the physical project directory that contains all of the source files.

Figure 4 XML for Populating a TreeView (Close Figure 4)
<TREENODES>
 <treenode Text="ASP.NET Projects" >
  <treenode Text="DevelopMentor ASP Class"
  NodeData="c:\asp.net labs\work" />
  <treenode Text="DataGrid example"
  NodeData="c:\datagrid" />
 </treenode>
 <treenode Text = "WinForms Projects" >
  <treenode Text="C# Drawing Program"
  NodeData="c:\csharpdraw" />
  <treenode Text="C# Scribble Program"
  NodeData="c:\csharpscribble" />
 </treenode>
 <treenode Text="C++ Projects">
  <treenode Text="MFC Projects"
  NodeData="c:\samples\samples\vc7\mfc" />
  <treenode Text="ATLServer Online Address Book"
  NodeData="c:\samples\atlserver\onlineaddressbook"/>
  <treenode Text="Extensibility"
  NodeData="c:\samples\extensibility\managedcwinformswiz"/>
 </treenode>
</TREENODES>

Once the TreeView nodes are described in XML, they may be bound to the control using normal ASP.NET data binding, as shown in Figure 5. When the TreeView's TreeNodeSrc attribute points to the XML file, invoking DataBind connects the contents of the XML file to the TreeView. When working with the sample, modify the XML file to match directories on your machine.

Figure 5 Populating a TreeView Using XML (Close Figure 5)
void Page_Load(object sender, System.EventArgs e)
{
  if(!IsPostBack)
  {
    treeviewAvailableProjects.TreeNodeSrc =
     "availableprojects.xml";
    treeviewAvailableProjects.DataBind();
  }
  // CssCollection defined as a class field
  //  will be used later to define the hover
  //  style for the node.
  string s = @"color:black;background:white;
    border:black thin solid;";
  HoverCssCollection = new CssCollection(s);
}

Building a Directory Tree

The second way to build a TreeView at run time is by adding the nodes programmatically. When the user selects one of the available projects to view, the ASPX page examines the directory and builds a TreeView dynamically. Figure 6 shows how you can build a TreeView to represent a physical directory structure.

Figure 6 Building a TreeView for a Physical Directory (Close Figure 6)
TreeNode AddFileNode(FileInfo fi, TreeNode tnParent)
{
  TreeNode tn = new TreeNode();
  tn.Text = fi.Name;
  tn.NodeData = fi.FullName;
  tn.HoverStyle= HoverCssCollection;
  tn.ImageUrl = "File.bmp";
  tnParent.Nodes.Add(tn);
  return tn;
}

TreeNode AddDirNode(DirectoryInfo di, TreeNode tnParent)
{
  TreeNode tn = new TreeNode();
  tn.Text = di.Name;
  tn.ImageUrl = "Dir.bmp";
  tnParent.Nodes.Add(tn);
  return tn;
}

void BuildDirectoryTree(DirectoryInfo di, TreeNode tnParent)
{
  try
  {
    foreach(FileSystemInfo fsi in di.GetFileSystemInfos())
    {
      Trace.Write("BuildDirectoryTree", di.FullName);

      if(fsi.GetType() == typeof(FileInfo))
      {
        FileInfo fi = (FileInfo)fsi;
        TreeNode tn = AddFileNode(fi, tnParent);
      }
      else
      {
        DirectoryInfo subDi = (DirectoryInfo)fsi;
        TreeNode tn = AddDirNode(subDi, tnParent);
        BuildDirectoryTree(subDi, tn);
      }
    }
  }
  catch(Exception e)
  {
    Trace.Warn("Building dir list", e.ToString());
  }
}

The code in Figure 6 uses the .NET Framework Class Library directory support. The code starts with a top-level DirectoryInfo object representing a directory (like "c:\samples"). BuildDirectoryTree iterates through the directory entries. When a file node is found, BuildDirectoryTree adds a file node to the tree. When it finds an entry representing a directory, it adds a directory node and recursively calls itself to generate the next level of nodes. Notice that AddFileNode and AddDirectoryNode use different images for files and directories and that the AddFileNode attaches the physical file name to the NodeData property of the node. The Project Explorer needs the file name when responding to events generated by the TreeView.

TreeView Events

In reviewing the events generated by the TreeView, you'll see they're very similar to other selection-style controls (like the ListBox, the CheckBox, and the DropDownListBox). For example, there are events indicating that a node is checked and that a selected item index has changed. In addition, TreeView generates events indicating that nodes have been expanded or collapsed.

By default, the TreeView AutoPostBack property is false. This means that when you expand a node, contract a node, check a node, or select a node, no event is generated. When the target browser is an up-level browser, the controls use DHTML to modify the appearance of the page—and the server is none the wiser. When the browser is a down-level browser, events are generated for selection changes, checks, expansions, and contractions. Figure 7 shows code responding to the SelectedIndexChange event for the available project's TreeView, causing the TreeView representing the directory to be built. Notice how the directory accessing code is wrapped in a try block.

Figure 7 Responding to the SelectedIndexChange Event (Close Figure 7)
void treeviewAvailableProjects_SelectedIndexChange(
     object sender, TreeViewSelectEventArgs e)
{
  DirectoryInfo di;
  string strDir;
  string sSelectedNode;

  this.textboxFileContents.Text = "";
    treeviewProjectInfo.Nodes.Clear();

  sSelectedNode =
    treeviewAvailableProjects.SelectedNodeIndex;
  TreeNode tnAvailProjects =
    treeviewAvailableProjects.GetNodeFromIndex(
      sSelectedNode);

  strDir = tnAvailProjects.NodeData;
  if(strDir != "")
  {
    TreeNode tn;
    try
    {
      tn = new TreeNode();
      treeviewProjectInfo.Nodes.Add(tn);
      treeviewProjectInfo.Nodes[0].Text = strDir;

      di = new DirectoryInfo(strDir);

      BuildDirectoryTree(di, treeviewProjectInfo.Nodes[0]);
    }
    catch(Exception ex)
    {
      Trace.Warn(ex.ToString());
    }
  }
}

The TreeView uses a slightly different item-indexing scheme from other selection-style controls such as the ListBox and the DropDownList. When an item is selected from the DropDownList or the ListBox, you can easily retrieve the selected item's integer index and its text name by using the SelectedItemIndex and the SelectedItem.Text properties name directly. The TreeView is a bit more complicated in that the TreeView's SelectedNodeIndex is a string. You can't look up the node directly using an integer. The string index includes integers separated by periods representing the position of the node within the hierarchy. For example, the string "0.1.1.1" represents the highlighted node in the Project Info TreeView (the TreeView on the right-hand side) shown in Figure 3. Each number represents a position within each level. Both levels and nodes are zero-based. Other than the indexing issue, responding to TreeNode events is very much like responding to events generated by other controls.

Node Styles

Finally, the nodes within a TreeView may be customized easily using CSS style attributes. The TreeNode class includes properties for managing the default style (the DefaultStyle), the hover style (the HoverStyle property), and the selected style (SelectedStyle) for each node. You can add the styles manually into the page's code:

<SomeTreeView:treenode
  Text=" Node with Styles"
  DefaultStyle="background:white;border:solid
    1px;color:black "
  HoverStyle="color:gray"
  SelectedStyle=
    "color:blue;font-name:Arial;font-weight:bold"
/>

The designer also supports adding CSS styles. In addition, you may add the styles programmatically as shown in Figure 5 and Figure 6 (take a look at Page_Load and AddFileNode, respectively). The AddFileNode method assigns a CssCollection defining the hover style to the file node. When the mouse hovers over the node, a surrounding border appears. The HoverStyle behavior applies only when the target browser is an up-level browser that is capable of supporting DHTML.

Conclusion

Tree-based UI tools have been available to desktop developers for a long time now. For example, MFC includes a Tree class and a TreeView class for managing Internet Explorer and Windows Explorer-style user interfaces. The same sort of tree-based UI is now available to ASP.NET developers. Manipulating the TreeView server-side control is very much like programming any other ASP.NET server-side control. There are a number of properties, methods, and events that are available both programmatically and through the designer.

The TreeView supports hierarchical data representation using the ASP.NET server-side control architecture. If the target browser is an up-level browser smart enough to be using DHTML, the TreeView emits DHTML, and the TreeView nodes interact with the user without having to make round-trips. Down-level browsers work as well; however, the user experiences more postbacks. To see how the TreeView reacts using a down-level browser, set the ClientTarget property of the page to "downlevel."

Send your questions and comments for George to asp-net@microsoft.com.

George Shepherd writes .NET development tools with SyncFusion (http://www.syncfusion.com) and teaches at DevelopMentor. He is the author of a number of programming books, including Programming with Microsoft Visual C++.NET (Microsoft Press, 2002) and Applied .NET (Addison-Wesley, 2001). George may be reached at georges@syncfusion.com or georges@develop.com.


Bugslayer: SOS: It's Not Just an ABBA Song Anymore

Bugslayer
SOS: It's Not Just an ABBA Song Anymore
John Robbins

Code download available at: Bugslayer0306.exe (119KB)
Contents
Using SOS
Wrap-up
The Tips

Hidden deep inside the Microsoft® .NET Framework 1.1 is one of the more interesting features for extreme power users: Son of Strike (SOS). If your app is a pure managed code play, your development and debugging tasks are easily handled by existing Microsoft tools. However, if you're like most folks, you can't move over to the completely managed world instantly. You have a huge investment in COM or COM+ components that you need to leverage from your managed code. This border between managed and native code is where, as the old maps used to say, the dragons be.

Some of you might think that all you have to do is turn on mixed-mode debugging. Unfortunately, that doesn't help you see things like managed memory leaks, which can happen if you misuse the Framework Class Library (FCL). It also won't let you peer into a minidump file that you can create when the application crashes. Getting information from both the managed and native sides of your application used to be a supreme challenge consisting of long nights staring at enough x86 assembly language to drive you completely nuts.

Fortunately, Microsoft set out to solve the mixed programming issues with SOS. While they've done an admirable job, using SOS takes commitment because it's written as a WinDBG extension. Interestingly, starting with Visual Studio® .NET 2003, you can load SOS when doing mixed-mode debugging. However, I've found it easier to do from WinDBG. If you don't already have WinDBG, you can download it from Microsoft Debugging Tools. WinDBG is a hardcore debugger that handles everything from kernel mode to user-mode debugging all in a circa-1990 user interface. While the interface is a little hard to use, the power features of the debugger are outstanding. Since I don't have the space in this column to discuss WinDBG at length, I'll refer you to the WinDBG documentation or Chapter 8 of my book, Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press®, 2003). For the remainder of this column, I'll assume that you've read one of those aforementioned documents and therefore have become familiar with WinDBG.

The basic documentation for SOS is in the SOS.htm file in the \SDK\v1.1\Tool Developers Guide\Samples\SOS directory under your Visual Studio .NET installation path. If you open the documentation, you'll definitely see that "basic" is the operable term here. In essence, it consists of the list of commands in the SOS.DLL extension and a brief bit about their usage.

If you're dealing with larger systems based on the .NET Framework, especially heavy ASP.NET transactions, you'll also want to download the 170-page PDF file "Production Debugging for .NET Framework Applications,". If you want to know how to handle hung ASPNET_WP.EXE processes, deal with potential memory management issues in the .NET Framework, and take charge of other extreme-edge problems, this is an excellent document. The folks who wrote the document have definitely done their debugging of numerous live production systems, and their knowledge can save you quite a bit of hassle.

With these two documentation sources, you'll get a quick discussion of the SOS commands and an extreme hardcore tricks document, but there's still something missing: just how do you get started with SOS inside WinDBG? In this column, I want to help you get a leg up on SOS. My hope is that you'll get enough information out of this section to understand what's going on in the "Production Debugging for .NET Framework Applications" document. I won't be covering everything, such as all the garbage collector commands, because those are covered in detail in this longer document.

Before I start, I want to show you the easy way to get SOS.DLL loaded into WinDBG. SOS.DLL is part of the .NET Framework itself, so the trick involves getting the appropriate directories in your path so that WinDBG can easily load SOS.DLL. To do this you'll need to open an MS-DOS® command window and execute the file VSVARS32.BAT, which is located in the Visual Studio .NET\Common7\Tools directory. VSVARS32.BAT gets your environment set up so that all the appropriate directories for the .NET Framework are in your path.

Once you've executed VSVARS32.BAT, starting WinDBG from that MS-DOS command window allows you to load SOS.DLL simply by executing ".load sos" from the WinDBG Command window. WinDBG always puts the last loaded extension DLL onto the top of the chain, so executing "!help" from the command window shows you a quick listing of all the SOS.DLL commands.

Using SOS

Probably the best way to show SOS usage is with a live example. The ExceptApp program included with this column's sample files will show you how to get started with important commands (see the link at the top of this article). To keep things manageable, this code simply calls a few methods with local variables and finally throws an exception. I'll walk through an example of debugging EXCEPTAPP.EXE with SOS so that you can learn about the important commands for finding where you are when an application using managed code crashes or hangs. With that information, you'll be in a position to more easily apply SOS to the problems you'll encounter while being better able to understand the Production Debugging guide.

After you've compiled EXCEPTAPP.EXE and set up an MS-DOS command prompt as I described earlier, open up EXCEPTAPP.EXE in WinDBG and stop at the loader breakpoint. To make WinDBG stop when a common language runtime (CLR)-managed application throws an exception, you have to tell WinDBG about the exception number that the CLR throws. The easiest way to do this is to go into the Event Filters dialog box, click the Add button, and in the Exception Filter dialog box, enter 0xE0434F4D. Select "Enabled" in the Execution group box, and select "Not Handled" in the Continue group box. Once you click OK, you've successfully set WinDBG to stop whenever any EXCEPTAPP.EXE throws a .NET exception. If the value 0xE0434F4D looks somewhat familiar, you can always see what it stands for by using the .formats command.

After you have set the exception, run EXCEPTAPP.EXE until it stops on the .NET exception. WinDBG will report it as a first chance exception and stop the application on the actual Win32 API RaiseException call. After getting SOS loaded with a .load sos command, execute "!threads" (the first command you'll always want to execute in SOS) so that you can see which threads in the application or dump have CLR code in them. With EXCEPTAPP.EXE, the WinDBG thread command using the tilde character (~) indicates that three commands are running in the application. However, the all-important !threads command lists that only threads 0 and 2 have any .NET-compliant code in them, as shown in the output in Figure 1. (To get everything to fit in on the page, I show the individual thread information in a table. In WinDBG, you see it as a long horizontal display.)

Figure 1 !threads (Close Figure 1)
0:000>!threads
PDB symbol for mscorwks.dll not loaded
succeeded
Loaded Son of Strike data table version 5 from
         "e:\WINNT \Microsoft.NET \Framework \v1.1.4322 \mscorwks.dll"
ThreadCount: 2
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 0
Row Heading

WinDBG Thread ID  0                  2
Win32 Thread ID   884                9dc
ThreadObj         00147c60           001631c8
State             20                 1220
PreEmptive GC     Enabled            Enabled
GC Alloc Context  04a45f24:04a45ff4  00000000:00000000
Domain            00158300           00158300
Lock Count        0                  0
APT               Ukn                Ukn
Exception         System.ArgumentException (Finalizer)

The important information you'll encounter in the !threads display consists of the Domain field, which tells you which AppDomains the threads are running in, and the Exceptions field, which happens to be overloaded. In the EXCEPTAPP.EXE example, the first thread has thrown the System.ArgumentException, so you can see the current exception for any thread. The third thread in EXCEPTAPP.EXE shows the special value (Finalizer), which indicates that the thread is, as you can guess, the finalizer thread for the process. You'll also see the values (Threadpool Worker), (Threadpool Completion Port), or (GC) in the Exception field. When you see one of those special values, you'll know they represent runtime threads, not your threads.

Since you've determined that the WinDBG thread 0 contains the EXCEPTAPP.EXE exception, you'll want to take a look at the call stack with !clrstack -all to see the full details of the stack, including parameters and locals (see Figure 2). Although !clrstack has switches to see the locals (-l) and parameters (-p), if you specify them together, they seem to cancel each other out and you see neither. If you'd like to walk all thread call stacks at once, you can use the command ~*e !clrstack.

Figure 2 !clrstack -all (Close Figure 2)
** Note, I excised the registers from this display **
0:000> !clrstack -all
Thread 0
ESP       EIP
0012f5e0  77e73887 [FRAME: HelperMethodFrame]
0012f60c  06d3025f [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Doh
                                       (String,ValueClass ExceptApp.Days)
   at [+0x67] [+0x16] c:\junk\cruft\exceptapp\class1.cs:14
    PARAM: this: 0x04a41b5c (ExceptApp.DoSomething)
    PARAM: value class ExceptApp.Days StrParam
    PARAM: unsigned int8 ValueParam: 0x07
0012f630  06d301e2 [DEFAULT] [hasThis] Void ExceptApp.DoSomething.Reh
                                                   (I4,String)
  at [+0x6a] [+0x2b] c:\junk\cruft\exceptapp\class1.cs:23
    PARAM: this: 0x04a41b5c (ExceptApp.DoSomething)
    PARAM: class System.String i: 0x00000042
    PARAM: int8 StrParam: 77863812
    LOCAL: class System.String s: 0x04a45670 (System.String)
    LOCAL: value class ExceptApp.Days e: 0x003e5278 0x0012f63c
•••

In the parameter display, there seems to be a bug because !clrstack doesn't always display the parameter types correctly. In the DoSomething.Doh method, you can see it takes a String and a Days value enumeration in the prototype. However, the PARAM: information shows the StrParam parameter as value class ExceptApp.Days and ValueParam as an unsigned int8. Fortunately for value parameters, even when the type is wrong, the correct value displays next to the parameter name. In the ValueParam example, the value passed in is 7, which corresponds to the enumeration "Fri."

Before I jump into figuring out the values of the value classes and objects, I want to mention another stack-walking command you might find useful. If you're dealing heavily with cross-.NET Framework and native calls and you'd like to see a call stack that includes everything, the !dumpstack command is your friend. Overall, it does a good job, but having full PDB symbols for the .NET Framework might make it better. Occasionally, the !dumpstack command reports "Use alternate method which may not work," which seems to indicate that it's attempting to walk the stack when it's missing certain symbol information.

In the LOCAL: display under the call to DoSomething.Reh are two local variables: s (a String object) and e (a Days value class). After each comes the hexadecimal address describing the type. For the Days value, there are two numbers: 0x003E5278 and 0x0012F63C. The first number is the method table and the second is the location in memory where the value is stored. Seeing the value in memory is simple using one of WinDBG's memory dumping commands such as dd 0x0012F63C.

Seeing the method table that describes the method data, the module information, and the interface map, among other things, is done through SOS's !dumpmt command. Executing !dumpmt 0x003E5278 with the EXCEPTAPP.EXE example gives the information shown in the following code:

0:000> !dumpmt 0x003e5278
EEClass : 06c03b1c
Module : 001521a0
Name: ExceptApp.Days
mdToken: 02000002  (D:\Dev\ExceptApp\bin\Debug\ExceptApp.exe)
MethodTable Flags : 80000
Number of IFaces in IFaceMap : 3
Interface Map : 003e5380
Slots in VTable : 55

Looking at the method table, the first two numbers displayed show you which module a method comes from, as well as its execution engine class. For interfaces, the SOS documentation has an excellent example of how to walk the interface maps, and I would encourage you to look it over. If you have a burning desire to see all the methods in the vtable for a particular class or object along with their method descriptors, you can specify the -md option in front of the method table value. In the case of EXCEPTAPP.EXE's value class ExceptApp.Days, you'll see all 55 methods listed. As the SOS documentation mentions in the "How Do I... ?" section, getting the method descriptors is important to setting breakpoints on specific methods.

Since I'm looking at the class and module information for the ExceptApp.Days method table, I want to take a little detour. Once you have an execution engine class address, the !dumpclass command will tell you everything you ever wanted to know about a class, including all the data fields in the class. To see the information about a module, use the !dumpmodule command. The !dumpmodule output documentation has examples of how to walk through memory and find classes and method tables for a module.

Now that I've ground through the value class, let's make some sense out of the String local variable called "s", in DoSomething.Reh, which was displayed as follows:

LOCAL: class System.String s: 0x04a45670 (System.String)
Since s is an object, only one hexadecimal value is displayed after the variable name—the location of that object in memory. Using the !dumpobj command, you'll see all the information about that object (see Figure 3).

Figure 3 !dumpobj (Close Figure 3)
0:000> !dumpobj 0x04a45670
Name: System.String
MethodTable 0x79b7daf0
EEClass 0x79b7de3c
Size 92(0x5c) bytes
mdToken: 0200000f  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll)
String: Tommy can you see me? Can you see me?
FieldDesc*: 79b7dea0
      MT    Field   Offset         Type       Attr    Value Name
79b7daf0  4000013       4  System.Int32   instance       38 _arrayLength
79b7daf0  4000014       8  System.Int32   instance       37 m_stringLength
79b7daf0  4000015       c   System.Char   instance       54 m_firstChar
79b7daf0  4000016       0         CLASS     shared   static Empty
    >> Domain:Value 00158298:04a412f8 <<
79b7daf0  4000017       4         CLASS     shared   static WhitespaceChars
    >> Domain:Value 00158298:04a4130c <<

As you can see from the output, some of the fields—MethodTable, EEClass, and MT (Method Table)—can be used with commands I've previously discussed. For the field members, the !dumpobj command will show the values directly in the table for simple value types. In the String display in the output in Figure 3, the m_stringLength value is the 37 characters currently in the string. As you'll see in a moment, for object field members, the Value field will contain the object instance, and you can use the !dumpobj command to see the value.

The entries delineated by >> and << are showing you the domain instance and location in that domain for the static field prior to the >>. If I had multiple AppDomains in EXCEPTAPP.EXE, you'd see two domains and value information output for the static WhitespaceChars field.

Now that I've covered some of the basic commands, let's tie them together and see how you can look up useful data with them. With EXCEPTAPP.EXE stopped in WinDBG because of an exception, it would be nice to see what the exception is and what some of the fields are so you can see why EXCEPTAPP.EXE stopped in the middle of execution.

You know from executing the !threads command that the first thread is currently processing an exception, System.ArgumentException. If you look carefully at the output for !clrstack or !dumpstack, you'll notice that no locals or parameters that show any type of System.ArgumentException are displayed. The good news is that an excellent command, !dumpstackobjects, shows all the objects that are currently on the stack of the current thread, as you can see in the output in Figure 4.

Figure 4 !dumpstackobjects (Close Figure 4)
0:000> !dumpstackobjects
ESP/REG  Object   Name
ebx      04a45670 System.String    Tommy can you see me? Can you see me?
0012f50c 04a45f64 System.ArgumentException
0012f524 04a45f64 System.ArgumentException
0012f538 04a45f64 System.ArgumentException
0012f558 04a44bc4 System.String    Reh =
0012f55c 04a45f64 System.ArgumentException
0012f560 04a45670 System.String    Tommy can you see me? Can you see me?
0012f564 04a4431c System.Byte[]
0012f568 04a43a58 System.IO.__ConsoleStream
0012f5a0 04a45f64 System.ArgumentException
•••

Since the !dumpstackobjects command is wandering up the stack, you'll see some items multiple times as they are passed a parameter to multiple functions. In Figure 4, you can see multiple System.ArgumentException objects, but if you look at the object value next to each object, you'll notice they are all referring to the same object instance, 0x04A45F64.

To inspect the System.ArgumentException object, I'll use the !dumpobj command. Note that I had to wrap the Name column to get everything to fit on the page (see Figure 5).

Figure 5 System.ArgumentException Object (Close Figure 5)
0:000> !dumpobj 04a45f64
Name: System.ArgumentException
MethodTable 0x79b87b84
EEClass 0x79b87c0c
Size 68(0x44) bytes
mdToken: 02000038  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll)
FieldDesc*: 79b87c70
      MT    Field   Offset          Type       Attr    Value Name
79b7fcd4  400001d        4         CLASS   instance 00000000 _className
79b7fcd4  400001e        8         CLASS   instance 00000000
                                                        exceptionMethod
79b7fcd4  400001f        c         CLASS   instance 00000000
                                                 _exceptionMethodString
79b7fcd4  4000020       10         CLASS   instance 04a456cc _message
79b7fcd4  4000021       14         CLASS   instance 00000000
                                                        _innerException
79b7fcd4  4000022       18         CLASS   instance 00000000 _helpURL
79b7fcd4  4000023       1c         CLASS   instance 00000000
                                                            _stackTrace
79b7fcd4  4000024       20         CLASS   instance 00000000
                                                      _stackTraceString
79b7fcd4  4000025       24         CLASS   instance 00000000
                                                _remoteStackTraceString
79b7fcd4  4000026       2c  System.Int32   instance        0
                                                      _remoteStackIndex
79b7fcd4  4000027       30  System.Int32   instance -2147024809
                                                               _HResult
79b7fcd4  4000028       28         CLASS   instance 00000000 _source
79b7fcd4  4000029       34  System.Int32   instance        0 _xptrs
79b7fcd4  400002a       38  System.Int32   instance -532459699 _xcode
79b87b84  40000d7       3c         CLASS   instance 04a45708
                                                            m_paramName

Inside an exception, the Message property is the important property. Since I can't call a method directly from WinDBG to see its value, I'll have to look at the _message field, which is where the Message property stores the actual string. Since this field is marked with CLASS, the hexadecimal number in the Value column is the object instance. To look at the object, I'll do another !dumpobj command to view it. As you've seen, the String object will have a special field, so you can see its actual value, which turns out to be the innocuous "Throwing an exception".

Wrap-up

Given the complexity of SOS, I hope I was able to bridge the gap between the basic documentation and the "Production Debugging for .NET Framework Applications" document. SOS is one of those tools that you hope you'll never have to use, but if you do have to use it sometime, you want to make sure that you have played with it before you're faced with that showstopper bug. Trying to learn SOS and debug at the same time is a recipe for sleeping in your office for two straight weeks.

The Tips

Spring is the time of love and I'd love to have you send your debugging tips to john@wintellect.com.

Tip 55 To attach to a process in Visual Studio .NET for quick native debugging, hold down the control key when you click Attach in the Processes dialog. This will bypass the Attach To Process dialog and immediately attach for native debugging only.

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in programming for .NET and Windows. This column was adapted from a chapter in Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at http://www.wintellect.com.


.NET Column: The CLR's Thread Pool

.NET
The CLR's Thread Pool
Jeffrey Richter

Contents
The Birth of the Thread Pool
Capability 1: Calling a Method Asynchronously
Capability 2: Calling a Method at Timed Intervals
Capability 3: Calling a Method When a Single Kernel Object Becomes Signaled
Conclusion

Microsoft is always trying to improve the performance of its platforms and applications. Many years ago, Microsoft researched how threads were being used by application developers to see what could be done to improve their use. Out of this research came a very important discovery: developers frequently created a new thread to perform a single task and when the task was complete, the thread would die.

This pattern is extremely common in a server application. A client makes a request of the server, the server creates a thread to process the client's request, and then, when the client's request is complete, the server's thread would die. Compared to a process, creating and destroying a thread is fast and uses fewer OS resources. But creating and destroying threads is certainly not free.

To create a thread, a kernel object is allocated and initialized, the thread's stack memory is allocated and initialized, and Windows® sends every DLL in the process a DLL_THREAD_ATTACH notification, causing pages from disk to be faulted into memory so that code can execute. When a thread dies, every DLL is sent a DLL_THREAD_DETACH notification, the thread's stack memory is freed, and the kernel object is freed (if its usage count goes to 0). So, there is a lot of overhead associated with creating and destroying a thread that has nothing to do with the work that the thread was created to perform in the first place.

The Birth of the Thread Pool

The result of this study led Microsoft to implement a thread pool, which first appeared in Windows 2000. When a Microsoft® .NET Framework team was designing and building the common language runtime (CLR), they decided to implement a thread pool right into the CLR itself. This way, any managed application could take advantage of a thread pool even if the application was running on a version of Windows prior to Windows 2000 (such as Windows 98).

When the CLR initializes, its thread pool contains no threads. When the application wants to create a thread to perform a task, the application should request the task be performed by a thread pool thread. The thread pool knows that and will create an initial thread. This new thread will go through the same initialization as any other thread; but, when the task is complete, the thread will not destroy itself. Instead, the thread will return to the thread pool in a suspended state. If the application makes another request of the thread pool, then the suspended thread will just wake up and perform the task and a new thread will not be created. This saves a lot of overhead. As long as the application queues tasks to the thread pool no faster than the one thread can handle each task, the same thread gets reused over and over again saving an enormous amount of overhead over the app's lifetime.

Now, if the application queues up tasks for the thread pool faster than the one thread can handle it, then the thread pool will create additional threads. Of course, creating new threads does generate overhead, but it is very likely that the application will require just a few threads to handle all of the tasks thrown at it over the application's lifetime. So, overall, the application's performance improves by using the thread pool.

Now, you might be wondering what happens if the thread pool contains many threads and the workload on the application diminishes. In this case, the thread pool contains several threads that are sitting suspended for long periods of time, wasting OS resources. Microsoft thought about this, too. When a thread pool thread suspends itself, it waits for 40 seconds. If 40 seconds elapses and the thread is given nothing to do, then the thread wakes up and destroys itself, freeing all the OS resources (stack, kernel object, and so forth) that it was using. Also, it probably doesn't hurt performance to have the thread wake up and destroy itself because the application can't be doing too much anyway or the thread would have resumed execution. By the way though, I said that threads in the thread pool wake themselves up in 40 seconds, the actual amount of time is not documented and is subject to change.

The cool thing about a thread pool is that it is heuristic. If your application needs to perform many tasks, then the thread pool creates more threads. If your application's work load dies down, then the thread pool threads kill themselves. The thread pool's algorithms ensure that it contains as many threads as required by the workload placed on it!

So, hopefully, you now understand the general concept behind a thread pool and see the performance advantages that it can offer. At this time, I'd like to show you some code demonstrating how to use the thread pool. First, you should know that the thread pool offers four capabilities:

  • Calling a method asynchronously
  • Calling a method at a timed interval
  • Calling a method when a single kernel object is signaled
  • Calling a method when an asynchronous I/O request completes

The first three capabilities are quite useful and I will demonstrate them in this column. However, the fourth capability is very rarely used by application developers, so I will not demonstrate it here; perhaps I'll cover it in a future column.

Capability 1: Calling a Method Asynchronously

In your application, if you have code in which you create a new thread to perform a task, I recommend that you replace that code with new code that directs the thread pool to perform the task instead. In fact, you'll generally find that it is easier to have the thread pool perform a task then it is to have a new, dedicated thread.

To queue a task for the thread pool, you use the ThreadPool class defined in the System.Threading namespace. The ThreadPool class offers only static methods and no instance of it can be constructed. To have a thread pool thread call a method asynchronously, your code must call one of ThreadPool's overloaded QueueUserWorkItem methods, as shown here:

public static Boolean QueueUserWorkItem(WaitCallback wc, Object state);
public static Boolean QueueUserWorkItem(WaitCallback wc);

These methods queue a "work item" (and optional state data) to a thread in the thread pool and return immediately. A work item is simply a method (identified by the wc parameter) that is called and passed a single parameter, state (the state data). The version of QueueUserWorkItem without the state parameter passes null to the callback method. Eventually, some thread in the pool will process the work item, causing your method to be called. The callback method you write must match the System.Threading.WaitCallback delegate type, which is defined as follows:

public delegate void WaitCallback(Object state);

Notice that you never call any method that creates a thread yourself; the CLR's thread pool will automatically create a thread, if necessary, and reuse an exiting thread if possible. Also, this thread is not immediately destroyed after it processes the callback method; it goes back into the thread pool so that it is ready to handle any other work items in the queue. Using QueueUserWorkItem might make your application more efficient because you won't be creating and destroying threads for every single client request.

The code in Figure 1 demonstrates how to have the thread pool call a method asynchronously.

Figure 1 Thread Pool Calls a Method (Close Figure 1)
using System;
using System.Threading;

class App {
   static void Main() {
      Console.WriteLine("Main thread: Queuing an aynchronous
          operation.");
      ThreadPool.QueueUserWorkItem(new WaitCallback(MyAsyncOperation));

      Console.WriteLine("Main thread: Performing other operations.");
      // ...

      Console.WriteLine("Main thread: Pausing to simulate doing other
          operations.");
      Console.ReadLine();
   }

   // The callback method's signature MUST match that of a
   // System.Threading.WaitCallback delegate (it takes an
   // Object parameter and returns void)
   static void MyAsyncOperation(Object state) {
      Console.WriteLine("ThreadPool thread: Performing aynchronous
          operation.");
      // ...
      Thread.Sleep(5000);    // Sleep for 5 seconds to simulate doing
                             // work

      // Returning from this method causes the thread to
      // suspend itself waiting for another task
   }
}

Capability 2: Calling a Method at Timed Intervals

If your application needs to perform a certain task at a certain time or if your application needs to execute some method periodically, the thread pool is the perfect thing for you to use. The System.Threading namespace defines the Timer class. When you construct an instance of the Timer class, you are telling the thread pool that you want a method of yours called back at a particular time in the future. The Timer class offers four constructors:

public Timer(TimerCallback callback, Object state,
   Int32 dueTime, Int32 period);
public Timer(TimerCallback callback, Object state,
   UInt32 dueTime, UInt32 period);
public Timer(TimerCallback callback, Object state,
   Int64 dueTime, Int64 period);
public Timer(TimerCallback callback, Object state,
   Timespan dueTime, TimeSpan period);

All four constructors construct a Timer object identically. The callback parameter identifies the method that you want called back by a thread pool thread. Of course, the callback method you write must match the System.Threading.TimerCallback delegate type, which is defined as follows:

public delegate void TimerCallback(Object state);

The constructor's state parameter allows you to pass state data to the callback method; you can pass null if you have no state data to pass. You use the dueTime parameter to tell the thread pool how many milliseconds to wait before calling your callback method for the very first time. You can specify the number of milliseconds using a signed or unsigned 32-bit value, a signed 64-bit value, or a TimeSpan value. If you want the callback method called immediately, specify 0 for the dueTime parameter. The last parameter, period, allows you to specify how long, in milliseconds, to wait before each successive call. If you pass 0 for this parameter, then the thread pool will call the callback method just once.

After constructing a Timer object, the thread pool knows what to do and monitors the time automatically for you. However, the Timer class offers some additional methods allowing you to communicate with the thread pool to modify when (or if) the method should be called back. Specifically, the Timer class offers several Change and Dispose methods:

public Boolean Change(Int32    dueTime, Int32    period);
public Boolean Change(UInt32   dueTime, UInt32   period);
public Boolean Change(Int64    dueTime, Int64    period);
public Boolean Change(TimeSpan dueTime, TimeSpan period);
public Boolean Dispose();
public Boolean Dispose(WaitHandle notifyObject);

The Change method allows you to change the Timer object's due time and period. The Dispose method allows you to cancel the callback altogether and optionally signal the kernel object identified by the notifyObject parameter when all pending callbacks for the time have completed.

The code in Figure 2 demonstrates how to have a thread pool thread call a method immediately and every 2000 milliseconds (or two seconds) thereafter.

Figure 2 Using the Period Parameter (Close Figure 2)
using System;
using System.Threading;

class App {
   static void Main() {
      Console.WriteLine("Checking for status updates every 2 seconds.");
      Console.WriteLine("   (Hit Enter to terminate the sample)");
      Timer timer = new Timer(new TimerCallback(CheckStatus), null, 0,
                              2000);
      Console.ReadLine();
   }

   // The callback method's signature MUST match that of a
   // System.Threading.TimerCallback delegate (it takes an
   // Object parameter and returns void)
   static void CheckStatus(Object state) {
      Console.WriteLine("Checking Status.");
      // ...
   }
}

Capability 3: Calling a Method When a Single Kernel Object Becomes Signaled

While doing their performance studies, Microsoft researchers discovered that many applications spawn threads simply to wait for a single kernel object to become signaled. Once the object is signaled, the thread posts some sort of notification to another thread and then loops back, waiting for the object to signal again. Some developers even write code in which several threads each wait on a single object. This is a big waste of system resources. So, if you currently have threads in your application that wait for single kernel objects to become signaled, then the thread pool is, again, the perfect resource for you to increase your application's performance.

To have a thread pool thread call your callback method when a kernel object becomes signaled, you again use some static methods defined in the System.Threading.ThreadPool class. To have a thread pool thread call a method when a kernel object becomes signaled, your code must call one of the overloaded RegisterWaitHandle methods you see in Figure 3.

Figure 3 RegisterWaitHandle Methods (Close Figure 3)
public static RegisterWaitHandle RegisterWaitForSingleObject(
   WaitHandle h, WaitOrTimerCallback callback, Object state,
   UInt32 milliseconds, Boolean executeOnlyOnce);

public static RegisterWaitHandle RegisterWaitForSingleObject(
   WaitHandle h, WaitOrTimerCallback callback, Object state,
   Int32 milliseconds, Boolean executeOnlyOnce);

public static RegisterWaitHandle RegisterWaitForSingleObject(
   WaitHandle h, WaitOrTimerCallback callback, Object state,
   TimeSpan milliseconds, Boolean executeOnlyOnce);

public static RegisterWaitHandle RegisterWaitForSingleObject(
   WaitHandle h, WaitOrTimerCallback callback, Object state,
   Int64 milliseconds, Boolean executeOnlyOnce);

When you call one of these methods, the h parameter identifies the kernel object that you want the thread pool to wait on. Since this parameter is of the abstract base class System.Threading.WaitHandle, you can specify any class derived from this base class. Specifically, you can pass a reference to an AutoResetEvent, ManualResetEvent, or Mutex object. The second parameter, callback, identifies the method that you want the thread pool thread to call. The callback method that you implement must match the System.Threading.WaitOrTimerCallback delegate type, which is defined in the following line of code:

public delegate void WaitOrTimerCallback(Object state,
                                         Boolean timedOut);

The third parameter, state, allows you to specify some state data that should be passed to the callback method; pass null if you have no special state data to pass. The fourth parameter, milliseconds, allows you to tell the thread pool how long it should wait for the kernel object to become signaled. It is common to pass -1 here to indicate an infinite timeout. If the last parameter, executeOnlyOnce, is true, then a thread pool thread will execute the callback method just once. But, if executeOnlyOnce is false, then a thread pool thread will execute the callback method every time the kernel object is signaled. This is most useful with an AutoResetEvent object.

When the callback method is called, it is passed state data and a Boolean value, timedOut. If timedOut is false, then the method knows that it is being called because the kernel object became signaled. If timedOut is true, then the method knows it is being called because the kernel object did not become signaled in the time specified. The callback method should perform whatever action is necessary.

In the prototypes shown earlier, you'll notice that the RegisterWaitForSingleObject method returns a RegisteredWaitHandle object. This object identifies the kernel object that the thread pool is waiting on. If, for some reason, your application wants to tell the thread pool to stop watching the registered wait handle, your application can call RegisteredWaitHandle's Unregister method:

public Boolean Unregister(WaitHandle waitObject);

The waitObject parameter indicates how you want to be notified when all queued work items have executed. You should pass null for this parameter if you don't want a notification. If you pass a valid reference to a WaitHandle-derived object, then the thread pool will signal the object when all pending work items for the registered wait handle have executed.

The code in Figure 4 demonstrates how to have a thread pool thread call a method whenever a kernel object becomes signaled.

Figure 4 Method Called When Object Signaled (Close Figure 4)
using System;
using System.Threading;

class App {
   static void Main() {
      AutoResetEvent are = new AutoResetEvent(false);
      RegisteredWaitHandle rwh = ThreadPool.RegisterWaitForSingleObject(
         are, new WaitOrTimerCallback(EventSignalled), null, -1, false);
      for (Int32 x = 0 ; x < 5; x++) {
         Thread.Sleep(5000);
         are.Set();
      }

      rwh.Unregister(null);
      Console.WriteLine("Hit Enter to terminate the sample");
      Console.ReadLine();
   }

   // The callback method's signature MUST match that of a
   // System.Threading.WaitOrTimerCallback delegate (it takes an
   // Object parameter and a Boolean and returns void)
   static void EventSignalled(Object state, Boolean timedOut) {
      if (timedOut) {
         Console.WriteLine("Timed-out while waiting for the
                           AutoResetEvent.");
      } else {
         Console.WriteLine("The AutoResetEvent became signalled.");
      }
      // ...
   }
}

Conclusion

In this column, I've explained the need for thread pools and demonstrated how to use the various capabilities offered by the CLR's thread pool. By now you should see the value that a thread pool can bring to your own development efforts to improve your application's performance and simplify your own code.

Send your questions and comments for Jeff to dot-net@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect (http://www.Wintellect.com), a training, debugging, and consulting firm specializing in .NET and Windows technologies. He is the author of Applied Microsoft .NET Framework Programming (Microsoft Press, 2002) and several programming books on Windows.


C++ Q&amp;A: Locking Column Headers, Implementing Singleton Classes

C++ Q&A
Locking Column Headers, Implementing Singleton Classes
Paul DiLascia

Code download available at: CQA0306.exe (137KB)

Q I have developed an ActiveX® Template Library (ATL) composite control, containing a ListView and Buttons, specifically for use in an ASP Web application. Now I want to prevent the sizing of the column headers. I have tried to use messages such as HDN_TRACK, HDN_BEGINTRACK, and HDN_ENDTRACK, replacing the ID=0 in the message map notification handler with my ID. I even tried the ITEM_CHANGING message, but all to no avail.

Deryl Francis Dsouza

A You're on the right track with HDN_BEGINTRACK, but you may have missed a detail. When a user drags the mouse over the size bars in the column header, the header control sends its parent an HDN_BEGINTRACK notification. The way to prevent sizing is to eat this notification (without passing it to the header control), but as with many other Windows® messages and notifications, HDN_BEGINTRACK comes in two flavors: HDN_BEGINTRACKW (wide-character, Unicode) and HDN_BEGINTRACKA (ANSI). The "neuter" symbol is #defined to one or the other of these, based on the value of UNICODE defined in your project, as shown here:

// From commctrl.h
#ifdef UNICODE
#define HDN_BEGINTRACK HDN_BEGINTRACKW
#else
#define HDN_BEGINTRACK HDN_BEGINTRACKA
#endif

So when you implement a handler for HDN_BEGINTRACK, you're actually implementing it for HDN_BEGINTRACKA or HDN_BEGINTRACKW, depending on the value of UNICODE. But which message does the header control actually send? Remember, the header control is part of Windows, one of the common controls in comctl32.dll. Since the DLL is already compiled into executable code, changing the value of UNICODE in your project has absolutely no effect on its operation. How does the header control know which flavor of notification to send—A or W?

The answer lies in an oft-forgotten message, WM_NOTIFYFORMAT. When a control is first created, it sends a message to its parent, in effect asking, "do you want ANSI or Unicode notifications?" The parent responds with NFR_ANSI or NFR_UNICODE. If the parent doesn't handle WM_NOTIFYFORMAT, the Windows DefWindowProc responds based on the preference of the parent window or dialog itself. The default is Unicode. So I suspect the reason you had no luck trapping HDN_BEGINTRACK is that you compiled an ANSI program without handling WM_NOTIFYFORMAT. Your application is looking for HDN_BEGINTRACKA (ANSI) while the header control is sending HDN_BEGINTRACKW (Unicode).

One way to fix the problem is to implement a WM_NOTIFYFORMAT handler for your list control, one that returns NFR_ANSI. When I tried this, the header control did indeed send HDN_BEGINTRACKA and I was able to prevent sizing. But using NFR_ANSI broke other features. For example, the list control no longer repaints its columns while sizing.

A simpler, more reliable way to prevent sizing header columns is to implement handlers for both HDN_BEGINTRACKA and HDN_BEGINTRACKW. This not only obviates the need to process WM_NOTIFYFORMAT, it lets your code work in both ANSI and Unicode modes. I wrote a little program, LockHeader, that shows how to do it. LockHeader has a menu command View | Lock Columns that's similar to the Lock Toolbars command in Explorer (see Figure 1). When the user invokes View | Lock Columns, LockHeader prevents sizing the column headers. Invoking the command again unlocks the columns.

Figure 1 Lock Columns
Figure 1 Lock Columns

Figure 2 shows the source, which is relatively straightforward. The header control sends HDN_XXX notifications to the parent (list control) window, but when using MFC you can use message reflection to handle the notifications in the header itself. Since the "lockable columns" feature is more a property of the header than the list control, this is the approach I chose. If you're not using MFC, you'll have to handle these notifications in the list control. To do message reflection, you can use ON_NOTIFY_REFLECT in your header control's message map or simply override the virtual function OnChildNotify, as shown here:

BOOL CLockableHeader::OnChildNotify(
 UINT msg, WPARAM wp, LPARAM lp, LRESULT* pRes)
{
  NMHDR& nmh = *(NMHDR*)lp;
  if (nmh.code==HDN_BEGINTRACKW || nmg.code==HDN_BEGINTRACKA)
    return *pRes=TRUE;
  •••
}

Figure 2 CLockableHeader (Close Figure 2)

Header.h
#pragma once

//////////////////
// Header control with lockable columns.
// Call Lock to lock/unlock them.
//
class CLockableHeader : public CHeaderCtrl
{
public:
   CLockableHeader() : m_bLocked(FALSE) { }

   // turn locking on/off
   void Lock(BOOL bLock) {
      m_bLocked = bLock;
   }

   // find out if locked
   BOOL IsLocked() {
      return m_bLocked;
   }

protected:
   BOOL m_bLocked;   // columns are locked
   virtual BOOL OnChildNotify(UINT msg, WPARAM wp, LPARAM lp,
                              LRESULT* pRes);
   afx_msg BOOL OnSetCursor(CWnd* pWnd, UINT nHitTest, UINT message);
   DECLARE_MESSAGE_MAP();
};
Header.cpp
#include "stdafx.h"
#include "Header.h"

#ifdef _DEBUG
#define new DEBUG_NEW
#undef THIS_FILE
static char THIS_FILE[] = __FILE__;
#endif

BEGIN_MESSAGE_MAP(CLockableHeader, CHeaderCtrl)
   ON_WM_SETCURSOR()
END_MESSAGE_MAP()

//////////////////
// If columns are locked, don't display size cursor.
//
BOOL CLockableHeader::OnSetCursor(CWnd* pWnd, UINT nHitTest, UINT msg)
{
   return m_bLocked ? TRUE :
      CHeaderCtrl::OnSetCursor(pWnd, nHitTest, msg);
}

//////////////////
// The header will send either HDN_BEGINTRACKA or HDN_BEGINTRACKW
// depending on how your parent list control responds to
// WM_NOTIFYFORMAT (default=HDN_BEGINTRACKW, Unicode). But if you're
// implementing a control to be used in any app, you should handle both
// notifications if you want it to work in either situation.
//
// Since OnChildNotify is virtual, all you have to do is override
// it—no need for message map entries to handle HDN_BEGINTRACKA/W.
//
BOOL CLockableHeader::OnChildNotify(UINT msg, WPARAM wp, LPARAM lp,
   LRESULT* pRes)
{
   NMHDR& nmh = *(NMHDR*)lp;
   if (nmh.code == HDN_BEGINTRACKA || nmh.code == HDN_BEGINTRACKW) {
      if (m_bLocked)
         return *pRes=TRUE; // eat message to disallow sizing
   }
   // otherwise, pass to header control for default processing
   return CHeaderCtrl::OnChildNotify(msg, wp, lp, pRes);
}

Since OnChildNotify is virtual, there's no need for message map entries. All you have to do is implement it. In any given application, the header will send one or the other, not both. Either way, CLockableHeader eats the notification—that is, returns TRUE (handled) without passing on to the default header control. CLockableHeader controls locking through a flag m_bLocked which the app can set by calling CLockableHeader::Lock.

If you're going to prevent sizing, you should also disable the size cursor. Otherwise users may think your app is either broken or lame. Fortunately, it's trivial:

BOOL
CLockableHeader::OnSetCursor(
  CWnd* pWnd, UINT nHit, UINT msg)
{
  return m_bLocked ? TRUE :
    CHeaderCtrl::OnSetCursor(pWnd, nHit, msg);
}

In other words: if the columns are locked, OnSetCursor returns TRUE without setting the cursor; otherwise, let the header control do its size cursor thing. Now when the columns are locked, Windows displays its standard arrow cursor instead of displaying the left-right sizing cursor.

Once you've implemented your custom header control by deriving from CHeaderCtrl, how do you get Windows to use it? The same way you would for any dialog control, by subclassing. The right place is in the parent window's OnCreate handler:

// CMyView is derived from CListView
int CMyView::OnCreate(LPCREATESTRUCT lpcs)
{
  VERIFY(CListView::OnCreate(lpcs)==0);
  return m_header.SubclassDlgItem(0,this) ? 0 : -1;
}

This works because the header control always has ID = 0. With all this in place, the only thing left to do is implement the command and UI update handlers for the View | Lock Columns command. Figure 3 shows the relevant snippets. As always, you can download the full source from the link at the top of this article.

Figure 3 CMyView (Close Figure 3)

View.h
#pragma once
#include "Header.h"

//////////////////
// Generic list view uses lockable header
//
class CMyView : public CListView {

   •••

protected:
   CLockableHeader m_header; // header control with lockable columns

   afx_msg void OnLockHeader();
   afx_msg void OnUpdateLockHeader(CCmdUI* pCmdUI);
   afx_msg int  OnCreate(LPCREATESTRUCT lpCreateStruct);

   DECLARE_MESSAGE_MAP()
};
View.cpp
#include "stdafx.h"
#include "View.h"
#include "resource.h"

BEGIN_MESSAGE_MAP(CMyView, CListView)
   ON_WM_CREATE()
   ON_COMMAND(ID_VIEW_LOCKHEADER, OnLockHeader)
   ON_UPDATE_COMMAND_UI(ID_VIEW_LOCKHEADER, OnUpdateLockHeader)
END_MESSAGE_MAP()

•••

//////////////////
// View created: subclass the header control, which has ID=0.
//
int CMyView::OnCreate(LPCREATESTRUCT lpcs)
{
   VERIFY(CListView::OnCreate(lpcs)==0);
   return m_header.SubclassDlgItem(0,this) ? 0 : -1;
}

//////////////////
// Initial update: add some items to the list control.
//
void CMyView::OnInitialUpdate()
{

   •••  // Add items

}

//////////////////
// View | Lock Columns command: lock them.
//
void CMyView::OnLockHeader()
{
   m_header.Lock(!m_header.IsLocked());
}

//////////////////
// Set checked state of menu command based on current locked state.
//
void CMyView::OnUpdateLockHeader(CCmdUI* pCmdUI)
{
   pCmdUI->SetCheck(m_header.IsLocked());
}

Q I was reading your answers about singleton classes in the Microsoft® .NET Framework in the February 2003 issue. Your example works when the same application tries to create more than one instance, but fails when multiple applications access the singleton class. It creates one instance for each application. How can I implement a singleton that enforces the pattern across different instances of the same application?

Naveen Venkatraghavan

A If you're writing a distributed application that communicates over a channel across host boundaries, then you need .NET Framework Remoting (see "Design and Develop Seamless Distributed Applications for the Common Language Runtime" by Dino Esposito in the October 2002 issue). But if all you want to do is share a small amount of simple data among multiple processes running on the same machine, you can accomplish this without Remoting.

The .NET Framework has no notion of shared objects because managed objects are created in the managed heap (which is why they're called managed), and each process has its own. DLLs, however, provide a way to share data. Normally, a DLL shares only its code, not its data. When a program loads a DLL, Windows gives each instance a private copy of the DLL's static data. But you can share it by putting it in a shared segment:

// in DLL .cpp file
#pragma data_seg (".mydata")
long foo = 0;
int bletch = 17;
#pragma data_seg ()
#pragma comment(linker, "/SECTION:.mydata,RWS")
The data_seg pragmas tell the compiler/linker to put the variables foo and bletch in a segment called "mydata". The third pragma tells the linker to make it shared (RWS = Read, Write, Shared). Now all instances of the DLL will be accessing the same data. If process A sets foo = 3, process B will see the new value.

How can you take advantage of this in your .NET Framework-based app? By using managed C++, of course! At first you might be tempted to try something like this:

#pragma data_seg (".mydata")
Singleton* theInstance = NULL;
#pragma data_seg ()
// initialization code
if (!theInstance)
  theInstance = new Singleton();
But you'll quickly discover that managed C++ doesn't let you store global pointers to managed objects. Nor does using GCHandle circumvent the problem. If you think about it, you can see why: when you call new, the allocator creates the object in the managed heap of whatever process happens to be the one calling the initialization code, so how can other processes possibly see it? In general, the common language runtime (CLR) must be free to move objects as part of its garbage collection routine, without worrying that it will leave dangling C++ pointers. This explains why you can't store a global pointer to a managed object in managed C++. So what are you going to do?

To get around the problem, you can implement the singleton object as either raw data or an unmanaged C++ object. Either way, you can then wrap a managed class around your implementation. Figure 4 shows what I have in mind. GlobalCounter is a simple managed class that implements a global counter that's shared among all processes that use the class. To use GlobalCounter in your .NET Framework-based app, you can write:

// C#
using Counter;
•••
label1.Text = String.Format("Count={0}",
  GlobalCounter.Increment());

Figure 4 GlobalCounter (Close Figure 4)

Counter.h
#pragma once

using namespace System;

//////////////////
// Global counter class shows how to share data among multiple
// instances of an application running on the same machine.
// Each application using GlobalCounter accesses the same underlying
// counter through the static methods of a managed wrapper class,
// GlobalCounter.
//
namespace Counter
{
   public __gc class GlobalCounter {
   public:
      GlobalCounter() { }
      __property static int get_Count(); // Count property is read-only
      static int Increment();            // increment the global counter
      static int Decrement();            // decrement the global counter
   };
}
Counter.cpp
#include "stdafx.h"
#include "Counter.h"
#include "TraceWin.h"
#using <System.dll>
using namespace Counter;

// Put the counter in a shared data segment
#pragma data_seg (".mydata")
long g_count = 0;
#pragma data_seg ()
#pragma comment(linker, "/SECTION:.mydata,RWS") // tell linker: make it
                                                // shared

//////////////////
// Implement Count property: get counter's current value.
//
int GlobalCounter::get_Count()
{
   return g_count;
}

//////////////////
// Increment the counter. Important to use InterlockedIncrement to
// avoid contention.
//
int GlobalCounter::Increment()
{
   return InterlockedIncrement(&g_count);
}

//////////////////
// Decrement the counter. Important to use InterlockedDecrement to
// avoid contention.
//
int GlobalCounter::Decrement()
{
   return InterlockedDecrement(&g_count);
}

I put these lines in a generic Windows.Forms app, TestCounter.exe, to display the counter in a text label. There's no point printing the source here, since everything except the previous lines is boilerplate (curious-minded readers can download the full sample from the MSDN Magazine Web site), but Figure 5 shows the desired result of running TestCounter three times: all three instances increment the same global counter.

Figure 5 Running TestCounter
Figure 5 Running TestCounter

How does GlobalCounter work? Counter.cpp implements a variable g_count in a shared segment, as described earlier. Increment and Decrement methods increment and decrement g_count, and get_Count returns its value as a property. Since the methods are static, applications don't have to create an object instance. Using the class name as the scope qualifier makes the counter look like a global object:

int c = GlobalCounter.Count;
int d = GlobalCounter.Decrement();

Since the underlying variable g_count is shared, it's important to use InterlockedIncrement and InterlockedDecrement to avoid contention. If you use a shared segment to implement a more complex class, don't forget to lock your member data before manipulating it. If you like, you can implement your singleton as a global static instance of an unmanaged (__nogc) class, with a managed wrapper class that accesses it through static methods.

GlobalCounter is as simple as I could make it, but the basic approach of using a shared data segment works correctly in more complex situations—up to a point. Shared data works best with "raw" data and/or unmanaged objects. If your singleton needs to manipulate other managed objects, you're likely to run into trouble. The fundamental problem is that whereas C++ lets you override operator new to write custom allocators, the .NET Framework does not. This is one of the big trade-offs of garbage collection. Thus, for example, you can't use memory-mapped files to share managed objects because there's no way to create the objects in the mapped memory. (Although you could do it by serializing or deserializing objects to or from mapped memory or by using only blittable types.) To learn more about mixing managed and unmanaged objects, you should read Tomas Restrepo's article in the February 2002 issue of MSDN Magazine. Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at askpd@pobox.com or http://www.dilascia.com.


Resource File: MSDN Online Developer Centers

Resource File
MSDN Online Developer Centers

On this page we provide information on the most current Microsoft developer resources. This month we focus on the new developer centers at MSDN Online.

When you're looking for information about a specific technology, don't you wish you could rely on one site to point you to the most pertinent content? Now you can. Based on feedback from software developers, MSDN Online has centralized all developer information on specific technologies so you don't have to hunt in several locations. The goals of these new sites are:

  • Improved navigation to make content easier to find
  • Resources and information to increase productivity when using Visual Studio tools
  • MSDN content filtered through a specific product or technology-focused view
  • A rating feature that lets readers rate articles, ask questions, and talk to other users and to MSDN

MSDN has accomplished this by merging its data sources and created multiple views of it. In dev-speak, they're merging resources into one DLL and creating multiple UI elements that instantiate that same interface. The result is five new developer centers: Visual Studio, Visual Basic, Visual C#, Visual C++, and the .NET Framework, all created based on the successful .NET Architecture and Web Services Developer Centers.

The new developer centers share a common structure. In the Using section you'll find information previously found across a wide variety of resources, including the MSDN Library. The Getting Started section offers guidance to users just beginning to explore the possibilities in Visual Studio, while the Migrating section helps users who are moving to the latest and greatest tool set. For those who prefer to start with an overview of new products and technologies, there's the Understanding section. The Understanding section in the Visual Studio Developer Center, for example, includes an article on using Visual Studio .NET 2003 and Visual SourceSafe 6.0 in a team environment. The Building section delivers the information required to build any type of Windows-based application—components, Web apps, mobile apps, and client apps. The Additional Resources sections list books, documentation, and training and events.

Each developer center features a Deploying and Distributing section covering critical deployment issues, such as side-by-side execution of the .NET Framework—an important advancement. Also covered is no-touch deployment, a feature which allows system administrators to deploy both applications and updates to applications through a remote Web server.

Here are some of the highlights to look for on each site. The Visual Basic Developer Center will provide new content that will help you write mobile applications using Visual Basic .NET 2003. The Visual C++ site will include information on managed C++ libraries and guidance on taking advantage of this new set of features. You'll also learn how to utilize the new forms engine if you're using Visual C++ in Visual Studio 2003. The Visual C# site will contain information on how to use the Microsoft Java Language Conversion Assistant, a wizard that converts Java-language source code to code based on Visual C# .NET. The .NET Framework Developer Center has all new content for the .NET Framework 1.1, including coverage of the common language infrastructure (CLI), common language runtime (CLR), and .NET Remoting. You'll also find more cross-community resources on each of the developer centers, including columns, newsgroups, user groups, chats, and third-party communities.

MSDN has plans to offer more developer centers in the near future. The ASP.NET Developer Center will go live soon, covering all aspects of hosting the next generation of Windows-based applications. The developer center for Mobility, also scheduled to go live soon, will cover everything from the Microsoft .NET Compact Framework to the Smartphone SDK. As security has become an integral part of all development projects, there will also soon be a Security and Trustworthy Computing Developer Center.

MSDN would certainly like to hear what you think. What information do you want to see covered in these developer centers? Please contact Shawn Morrissey at shawnmor@microsoft.com with your suggestions.

The new developer centers can be found at:

  • http://msdn.microsoft.com/vstudio
  • http://msdn.microsoft.com/vbasic
  • http://msdn.microsoft.com/vcsharp
  • http://msdn.microsoft.com/visualc
  • http://msdn.microsoft.com/netframework