 MSDN Magazine May 2006

MSDN Magazine May 2006

Wrap It Up: Call Into The .NET Framework From Existing Visual Basic 6.0 Apps

Scott Swigart x64 Primer: Everything You Need To Know To Start Programming 64-Bit Windows Systems

Matt Pietrek Mix And Match: Integrate Windows Forms Into Your MFC Applications Through C++ Interop

Marcus Heege Bug Bash: Let The CLR Find Bugs For You With Managed Debugging Assistants

Managed Debugging Assistants are new to the .NET Framework 2.0 and help you to discover serious bugs quickly. Learn how to harness their power.

Stephen Toub Analyze This: Find New Meaning In Your Ink With Tablet PC APIs In Windows Vista

Markus Egger Editor's Note:

We know what you're thinking. Visual Studio 2005 has been out for a few months now. You're getting your head around it, discovering its vast inner beauty. But still, there's a nagging voice inside your head taunting you, asking why you're content to work with a released product.

Toolbox: Database job scheduling, Browser Analysis, and More

From a developer’s perspective, once a database’s schema has been defined and the tables, stored procedures, and other database objects have been created, it can be forgotten, abstracted away into the data access layer of the application’s architecture.

Scott Mitchell Basic Instincts: Resources and Localization

There are two ways you can utilize resources such as strings, images, and text-based files from your Microsoft® . NET Framework-based application. You can embed them directly in the app or you can load them from an external file.

Ted Pattison CLR Inside Out: The Performance Benefits of NGen.

T ypically, methods in managed executables are just-in-time (JIT) compiled. The machine code generated by the JIT compiler is thrown away once the process running that executable exits; therefore, the method must be recompiled when the application is run again.

Surupa Biswas Data Points: SQL Server 2005 XML Support, Exception Handling, and More

SQL Server 2005 includes several important improvements to the Transact-SQL (T-SQL) language. One added feature is a new kind of trigger that fires when data definition language (DDL) statements run.

John Papa Cutting Edge: Extending the GridView Control

Welcome to my100th consecutive installment of Cutting Edge. I've been writing this column since January 1998 in Microsoft Internet Developer. Looking back over the past eight years, I realize that I've touched on almost every subject in the Windows® SDK and the Microsoft® .

Dino Esposito Test Run: Stress Testing.

Stress testing is a fundamental quality assurance activity that should be part of every significant software testing effort. The key idea behind stress testing is simple: instead of running manual or automated tests under normal conditions, you run your tests under conditions of reduced machine or system resources.

Dr. James McCaffrey Extreme ASP.NET: Keeping secrets in ASP.NET 2.0.

Storing data securely in a configuration system is not an easy problem to solve. While I was on the ASP. NET team, this particular feature, secure connection string storage, looked as if it wouldn’t get done.

Rob Howard Security Briefs: Step-by-Step Guide to InfoCard

In my April 2006 column I began a discussion of InfoCard, the upcoming identity metasystem, which is being prepared for release in the Windows Vista™ timeframe. If you haven’t read that column, you should definitely start there because I’m going to assume you’re familiar with the basics I covered.

Keith Brown C++ at Work: Web Version Checking, Adding Sound to an App

This month: CWebVersion revisited using HTTP instead of FTP, and adding sounds to an MFC-based app.

Paul DiLascia { End Bracket }: The Pay as You Go Model

How would you feel if you were having your dream house built, but when you inspected the house you found some fairly obvious flaws in the work—cracks in the foundation, support beams missing, crooked and leaning walls.

Eric N. Bush


Wrap It Up: Call Into The .NET Framework From Existing Visual Basic 6.0 Apps

Wrap It Up

Call Into The .NET Framework From Existing Visual Basic 6.0 Apps

Scott Swigart
This article discusses:
  • Considerations for upgrading existing code
  • Setting up a wrapper in Visual Studio
  • A sample network connectivity wrapper
  • Using the FTP features of the .NET Framework
This article uses the following technologies:
Visual Basic 6.0, Visual Basic 2005 Express, .NET Framework
Code download available at: VBFusion05.exe (150KB)
Contents
The Cheap and Easy Approach
Calling the .NET Framework
Now You Have It; Now You Don’t
Something Old and Something New
Conclusion

M
any developers believe that if they are writing code in Visual Basic® 6.0, the vast world of the Microsoft® .NET Framework is off limits to them. But in this article, I’ll prove that isn’t true. I’ll show how you can easily leverage anything in the .NET Framework 2.0 from existing Visual Basic 6.0 applications through a technique known as Visual Basic Fusion. Make no mistake, I’m advocating nothing less than leaving some of your applications in Visual Basic 6.0. However, there’s no reason that the door to the .NET Framework class library must remain locked just because you choose not to upgrade to Visual Basic 2005.

So why not just upgrade? Tremendous resources are at your disposal if you choose to upgrade a Visual Basic 6.0 application to Visual Basic 2005. Visual Studio® provides a migration assistant that will automatically rewrite much of your Visual Basic 6.0 code. Microsoft has published a Visual Basic 6.0 to Visual Basic 2005 Migration Guide that includes an assessment tool to help determine the time and cost to upgrade your application. The Web is chock full of articles, how-tos, tips, and tricks on specific upgrading issues. There are even a few books on the topic. With so many resources available, some may question why any code should remain in Visual Basic 6.0. Why not just upgrade all of it?

In truth, upgrading code from Visual Basic 6.0 to Visual Basic 2005 can be a labor-intensive process, and the success of this endeavor hinges largely on two factors: code quality and developer experience with Visual Basic 2005 and the .NET Framework. Applications that are well architected and highly modular are easier to upgrade than applications that have become messy over time.

In the world of Visual Basic 6.0, many applications have been in production for years. Maybe these applications have needed feature enhancements on accelerated schedules. Maybe they have been programmed by numerous developers, some of whom did not have the time or patience to fully understand the code they were modifying. For whatever reason, programmers are often under pressure to get a project working and then move on. As a result, the code can get pretty convoluted over time. Sufficiently messy code is generally better off being rewritten rather than upgraded, and rewriting isn’t cheap or easy.

As for developer experience, upgrading a complex and mission-critical Visual Basic 6.0 application is a dreadful way to come up to speed on Visual Basic 2005. It’s important to have good experience with .NET before attempting to upgrade anything. There are several reasons for this. If you feed a Visual Basic 6.0 application down the chute of the migration wizard and turn the crank, what pops out the other side is a Visual Basic 2005 application that will not simply compile and run. The migration wizard will fling comments throughout the code marking locations where it was not able to automatically migrate. To fix the application and get it running, you need a solid understanding of Visual Basic 2005 and the .NET Framework. The code changes required range from simple and tedious to rewrites of entire application subsystems. So clearly, upgrading isn’t always easy.

With upgrading described as such a dismal prospect, you might wonder why anyone ever does it. Well, moving a code base to Visual Basic 2005 isn’t trivial, but it is ultimately the best way to get friction-free access to the full .NET Framework. I also find it much more productive to develop with the vast .NET Framework class libraries and with the superb productivity features provided by Visual Studio 2005. Visual Studio 2005 honestly makes Visual Studio 6.0 feel like something with a hand crank on the front. When it comes to upgrading, you’re going to have to make an application-by-application decision. Is it worth the effort? Is it feasible to do right now?

The Cheap and Easy Approach

Another option is to leave the application in Visual Basic 6.0 and just call into the .NET Framework as needed. Right in the box, the .NET Framework includes classes to ping IP addresses, detect changes to network connectivity, transmit files with FTP, access COM ports, play sounds, get operating system information such as physical memory and screen resolution, get user and domain information, read and write to the event log, access performance counters, start and stop services, and access the registry. The list goes on and on. The word "comprehensive" readily springs to mind when describing the .NET Framework 2.0 class library. By using the .NET Framework 2.0, you can likely eliminate dependencies on certain third-party controls and remove many Win32® API calls. Calling into the .NET Framework from Visual Basic 6.0 is easy and low risk. It’s also a good way to come up to speed on Visual Basic 2005 programming.

To call into the .NET Framework, as outlined in this article, you’ll need either Visual Studio 2005 or Visual Basic Express. Visual Basic Express includes all the toolbox controls, IntelliSense®, debugger, and many of the project items. It’s quite sufficient if you want to kick the tires and take Visual Basic 2005 for a spin. Plus, Visual Basic Express isn’t a trial version that stops working at some future date. Instead, it’s a pared down Visual Studio, designed to be used until the applications you’re building outgrow its capabilities.

You should also download the code associated with this article, particularly if you are using Visual Basic Express. Copy the included ComClass.zip file to your \My Documents\Visual Studio 2005\Templates\ItemTemplates\Visual Basic folder. As you’ll soon see, this project item is critical when calling into the .NET Framework from Visual Basic 6.0, and it’s not included with Visual Basic Express by default.

Calling the .NET Framework

Visual Basic 6.0 is great at calling into COM objects. However, the classes in the .NET Framework are not COM objects and are not directly callable from Visual Basic 6.0. Instead, the .NET Framework classes must be called through a wrapper. The wrappers are built using the COM Class item, and they behave like any other COM objects. The wrappers are callable from Visual Basic 6.0, Visual Basic Scripting Edition (VBScript), Visual Basic for Applications (VBA), or any other COM-compatible environment. The wrapper internals are constructed in Visual Basic 2005 and can call seamlessly into the .NET Framework (see Figure 1).

Figure 1 .NET Wrappers for Visual Basic 6.0
Figure 1 .NET Wrappers for Visual Basic 6.0

The construction of wrappers is a downright trivial task. These following steps will walk you through creating a COM callable wrapper for the Ping functionality in the .NET Framework 2.0.

  1. Start either Visual Studio 2005 or Visual Basic Express.
  2. Create a new Class Library project named FX20Wrapper.
  3. Select the Project menu, and then select Add New Item.
  4. Select the COM Class item template. For the name, enter NetworkWrapper, and click Add.
  5. Add the following function to the NetworkWrapper class:
    Public Function Ping(ByVal hostNameOrAddress As String, _
        Optional ByVal timeout As Integer = -1)
    If timeout >= 0 Then
        Return My.Computer.Network.Ping(hostNameOrAddress, timeout)
    Else
        Return My.Computer.Network.Ping(hostNameOrAddress)
    End If
End Function

That’s it! With just a few lines of code, you’ve created a COM object that will let you ping every IPv4 address in the known universe. This function lets you ping by host name or IP address. You can also specify an optional timeout value. Imagine the possibilities. The only remaining step is to use the Build menu to compile the Visual Basic Express project. The IDE compiles your project and emits a DLL, which Visual Studio registers as a COM object.

Next, you can construct a Visual Basic 6.0 project that uses this wrapper.

  1. Start Visual Basic 6.0 and create a new Standard EXE project.
  2. Create a user interface like the one shown in Figure 2.
    3. Figure 2 Ping Test
    Figure 2 Ping Test
  3. On the Project menu, select References to add a reference to FX20Wrapper. Notice that your .NET project shows up just like any other COM object.
  4. In the click event of the Ping button, add the following code:
    Private Sub Command1_Click()
    Dim network As FX20Wrapper.NetworkWrapper
    Set network = New FX20Wrapper.NetworkWrapper
    Text2 = Text2 & "Pinging " & Text1 & "..."
    DoEvents
    If network.ping(Text1) Then
        Text2 = Text2 & "Responded" & vbCrLf
    Else
        Text2 = Text2 & "Did not respond" & vbCrLf
    End If
End Sub
  5. 5.Run the application, enter an IP address, and click the Ping button. You should see a result like that shown in Figure 2.

Now You Have It; Now You Don’t

Applications are increasingly expected to handle unexpected changes in network connectivity. Laptops unplug. Users travel from one wireless oasis to another. The operating system notifies the user every time the network comes and goes. Wouldn’t it be nice if the operation system also notified your application? With Visual Basic 2005, it does. The My.Computer.Network object fires an event whenever the network connectivity status changes. The code in Figure 3 shows how to enhance your NetworkWrapper class to provide this notification.

Figure 3 Wrapper for Network Availability Notification (Close Figure 3)
Public Event NetworkAvailabilityChanged(ByVal IsAvailable As Boolean)

Public Sub New()
    AddHandler My.Computer.Network.NetworkAvailabilityChanged, _
        AddressOf My_NetworkAvailabilityChanged
End Sub

Public ReadOnly Property IsAvailable() As Boolean
    Get
        Return My.Computer.Network.IsAvailable
    End Get
End Property

Private Sub My_NetworkAvailabilityChanged(ByVal sender As Object, _
      ByVal e As Microsoft.VisualBasic.Devices.NetworkAvailableEventArgs)
    RaiseEvent NetworkAvailabilityChanged(e.IsNetworkAvailable)
End Sub

Notice that in the New subroutine (the constructor for the NetworkWrapper object), My.Computer.Network exposes a Net-workAvailabilityChanged event. This event is not directly COM-consumable. Instead, this event is hooked to a subroutine inside of the wrapper called My_NetworkAvailabilityChanged. From this routine, an event will be fired that is COM-consumable. This routine fires its own NetworkAvailabilityChanged event, which you can see contains a simple Boolean value to indicate the network status.

From the Visual Basic 6.0 side, this event is no harder to consume than a Button_Click event. The code for the Visual Basic 6.0 application is shown in Figure 4. With a tiny amount of code, your Visual Basic 6.0 applications can suddenly become network-aware. In an actual application that implements this wrapper, when the network is not available, the status bar might indicate its status as "Offline" and the ability to access network resources could be disabled.

Figure 4 Consuming NetworkAvailabilityChanged (Close Figure 4)
Dim WithEvents network As FX20Wrapper.NetworkWrapper

Private Sub network_NetworkAvailabilityChanged( _
        ByVal IsAvailable As Boolean)
    ShowAvailable IsAvailable
End Sub

Private Sub Form_Load()
    Set network = New FX20Wrapper.NetworkWrapper
    ShowAvailable network.IsAvailable
End Sub

Private Sub ShowAvailable(IsAvailable As Boolean)
    cmdUploadData.Enabled = IsAvailable
    StatusBar1.SimpleText = IIf(IsAvailable, "Online", "Offline")
End Sub

Something Old and Something New

The original RFC for FTP was published in 1973, and FTP remains one of the most popular protocols for transmitting files over the Internet. Microsoft has provided programmatic support for FTP in the past through the Microsoft Internet Transfer Control and the WinInet API. However, there has not been native support for FTP as part of the .NET Framework until recently. The .NET Framework 2.0 adds support for FTP through the WebClient, FtpWebRequest, and FtpWebResponse classes. Through these classes, you can perform a variety of FTP operations. The specific operations and the FTP commands that they map to are described by the WebRequestMethods.Ftp members, shown in Figure 5.

Figure 5 WebRequestMethods.Ftp Members (Close Figure 5)

Name | Description
AppendFile | Represents the FTP APPE protocol method that is used to append a file to an existing file on an FTP server.
DeleteFile | Represents the FTP DELE protocol method that is used to delete a file on an FTP server.
DownloadFile | Represents the FTP RETR protocol method that is used to download a file from an FTP server.
GetFileSize | Represents the FTP SIZE protocol method that is used to retrieve the size of a file on an FTP server.
ListDirectory | Represents the FTP NLIST protocol method that gets a short listing of the files on an FTP server.
ListDirectoryDetails | Represents the FTP LIST protocol method that gets a detailed listing of the files on an FTP server.
MakeDirectory | Represents the FTP MKD protocol method creates a directory on an FTP server.
PrintWorkingDirectory | Represents the FTP PWD protocol method that prints the name of the current working directory.
RemoveDirectory | Represents the FTP RMD protocol method that removes a directory.
Rename | Represents the FTP RENAME protocol method that renames a directory.
UploadFile | Represents the FTP STOR protocol method that uploads a file to an FTP server.
UploadFileWithUniqueName | Represents the FTP STOU protocol method that uploads a file with a unique name to an FTP server.

The time required to download a file can vary greatly and is affected by file size and network bandwidth. To accommodate this, the FTP classes support both synchronous and asynchronous operations. Synchronous operations are the simplest to program, with a download requiring only a couple lines of Visual Basic 2005 code:

Dim w As New WebClient
w.DownloadFile(address, fileName)

However, synchronous applications block (freeze) the application until the download operation has completed. While this behavior might be fine for command-line or batch operations, users rarely appreciate the experience of an application turning to stone without providing any indication of when it will reanimate. The FTP asynchronous operations perform the downloading on a separate thread and fire events notifying of the progress and download completion. Figure 6 shows a wrapper for the FTP classes that provides both synchronous and asynchronous operations.

Figure 6 Wrapper for FTP Functionality (Close Figure 6)
Imports System.Net

<ComClass(FTPWrapper.ClassId, FTPWrapper.InterfaceId, _
    FTPWrapper.EventsId)>
Public Class FTPWrapper

#Region "COM GUIDs"
‘ These GUIDs provide the COM identity for this class
‘ and its COM interfaces. If you change them, existing
‘ clients will no longer be able to access the class.
Public Const ClassId As String = "a9afe450-c10e-47cd-a004-f42dc975caaf"
Public Const InterfaceId As String = _
    "ec30e4cf-877c-4947-a093-715c2002aea0"
Public Const EventsId As String = "ddffd280-bff9-40dd-9238-215232b90b11"
#End Region

Dim WithEvents w As New WebClient
Dim totalBytes As Integer

Public Event DownloadProgressChanged( _
    ByVal bytesDownloaded As Integer, _
    ByVal totalBytes As Integer)

Public Event DownloadFileCompleted()

Public Sub New()
    MyBase.New()
End Sub

Public ReadOnly Property IsBusy() As Boolean
    Get
        Return w.IsBusy
    End Get
End Property

Public Sub DownloadFile(ByVal address As String, ByVal fileName As String)
    w.DownloadFile(address, fileName)
End Sub

Public Sub DownloadFileAsync( _
    ByVal address As String, _
    ByVal fileName As String)

    If w.IsBusy Then
        Throw New ApplicationException( _
          "FTPWrapper has not completed the previous operation. " & _
          "One operation must complete before another can be initiated.")
    Else
        Dim f As FtpWebRequest = _
            CType(WebRequest.Create(address), FtpWebRequest)
        f.Method = WebRequestMethods.Ftp.GetFileSize
        Using response As FtpWebResponse = _
                CType(f.GetResponse(), FtpWebResponse)
            totalBytes = response.ContentLength
        End Using
        w.DownloadFileAsync(New System.Uri(address), "c:\readme.txt")
    End If
End Sub

Private Sub w_DownloadFileCompleted(ByVal sender As Object, _
      ByVal e As System.ComponentModel.AsyncCompletedEventArgs) _
      Handles w.DownloadFileCompleted
    RaiseEvent DownloadFileCompleted()
End Sub

Private Sub w_DownloadProgressChanged(ByVal sender As Object, _
      ByVal e As System.Net.DownloadProgressChangedEventArgs) _
      Handles w.DownloadProgressChanged
    RaiseEvent DownloadProgressChanged(e.BytesReceived, totalBytes)
End Sub

End Class

The FTPWrapper class builds on concepts illustrated thus far. First, this class is built using the ComClass project item template. This ensures that FTPWrapper will contain the necessary GUIDs for registration as a COM object. An instance of WebClient is dimensioned WithEvents at the class level. This allows all methods within the class to access this single instance, and it allows the FTPWrapper class to handle the WebClient DownloadProgressChanged and DownloadFileCompleted events.

The DownloadFile subroutine contains a single line of code and performs a synchronous download of a specific file. As described previously, calling this subroutine turns your application to stone while the file downloads.

DownloadFileAsync is a more versatile and interesting way to download a file. This subroutine begins by ensuring that there isn’t another asynchronous operation in progress. If there is, the subroutine throws an exception, which is the Visual Basic 2005 approximation of Err.Raise. Next, DownloadFileAsync uses the FtpWebRequest class to execute a SIZE FTP command, which returns the size in bytes of the requested file. The file size is needed in order to accurately display download progress. Finally, the WebClient DownloadFileAsync function is called. This function, which initiates the file download process in the background, also begins the regular firing of the DownloadProgressChanged event.

The WebClient DownloadProgressChanged event is not directly COM consumable, so this event is also wrapped. The handler for DownloadProgressChanged fires a new event, which passes simple integer arguments for bytes downloaded and total file size. This event is easily consumable by Visual Basic 6.0. Likewise, the DownloadFileCompleted event is wrapped and made consumable for Visual Basic 6.0.

The primary goal of creating wrappers is to make .NET Framework classes appear as typical COM objects, with easily usable properties, methods, and events. The FTPWrapper achieves this goal, and Figure 7 shows the Visual Basic 6.0 code needed to download a file using FTP.

Figure 7 Downloading a File (Close Figure 7)
Dim WithEvents ftp As FX20Wrapper.FTPWrapper

Private Sub btnDownload_Click()
    lblProgress.Visible = True
    prgProgress.Visible = True
    Command1.Enabled = False
    ftp.DownloadFileAsync txtAddress.Text, txtFileName.Text
End Sub

Private Sub Form_Load()
    Set ftp = New FX20Wrapper.FTPWrapper
End Sub

Private Sub ftp_DownloadFileCompleted()
    lblProgress.Visible = False
    prgProgress.Visible = False
    Command1.Enabled = True
End Sub

Private Sub ftp_DownloadProgressChanged(ByVal bytesDownloaded As Long, _
        ByVal totalBytes As Long)
    prgProgress.Max = totalBytes
    prgProgress.Value = bytesDownloaded
End Sub

Starting a download is as simple as dimensioning the FTPWrapper class and calling the DownloadFile or DownloadFileAsync subroutine. If you call DownloadFileAsync, then the FTPWrapper will fire DownloadProgressChanged events as the download proceeds and a DownloadFileCompleted event when the download is completed. The DownloadProgressChanged event receives the number of bytes downloaded so far and the total file size. This information is all that’s needed to provide a progress indication of the download. Figure 8 shows the Visual Basic 6.0 FTP client while a download is in progress.

Figure 8 FTP Client Implemented in Visual Basic 6.0
Figure 8 FTP Client Implemented in Visual Basic 6.0

The code download for this article contains the wrapper class and sample applications shown in this article. To try out the samples, just download the code and execute the included install.bat file. This registers the wrapper class as a COM object. Then you can run any of the Visual Basic 6.0 samples.

Conclusion

Visual Basic 6.0 has become the new COBOL. This statement is made without malice toward either language. COBOL was enormously successful because it established itself as the language to use when writing mission critical mainframe applications. As the world went thick-client and GUI, Visual Basic assumed the mantle, showing itself to be arguably the best language for many business scenarios.

I can think of no mainstream business programming language that has truly gone extinct. Just as millions of lines of COBOL exist, and will for the foreseeable future, we will also be dealing with Visual Basic 6.0 for years to come. In fact, Microsoft has committed to shipping the Visual Basic 6.0 runtime on Windows Vista™, and running the Visual Basic 6.0 compatibility suite on Windows Vista to make a concerted effort to ensure that Visual Basic 6.0 applications just keep working.

But what is to be done with all this legacy code? Surely, it isn’t feasible to rewrite every line of it in Visual Basic 2005. Even if much of this code will eventually be migrated, it may not be possible to take that time and risk with a given application right now. Constraints dictate that many applications will continue their life on the Visual Basic 6.0 runtime. These applications need not ignore the .NET Framework and the productivity benefits that it brings.

You have seen how one tiny aspect of the .NET Framework, a few network classes, can be wrapped and exposed as COM objects, making their use from Visual Basic 6.0 trivial. Literally any portion of the .NET Framework can be wrapped and exposed in this way. Think of the .NET Framework as a gift from Microsoft of thousands upon thousands of classes. To use them, you need only supply the gift wrapping.

Scott Swigart spends his time consulting, writing, and speaking about converging and emerging technologies. Scott is the author of several books on .NET, a certified Microsoft trainer (MCT) and developer (MCSD), and a Microsoft MVP. You can contact Scott at scott@swigartconsulting.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


x64 Primer: Everything You Need To Know To Start Programming 64-Bit Windows Systems

New information has been added to this article since publication.
Refer to the Editor's Update below.

x64 Primer

Everything You Need To Know To Start Programming 64-Bit Windows Systems

Matt Pietrek
This article discusses:
  • Background on 64-bit versions of Windows
  • Just enough x64 architecture to get by
  • Developing for x64 with Visual C++ 2005
  • Debugging techniques for your x64 builds
This article uses the following technologies:
Windows, Win64, Visual Studio 2005
Contents
The x64 Operating System
Just Enough x64 to Get By
Developing for x64 with Visual C++
Making Your Code Win64-Compliant
Debugging
What About Managed Code?
Wrap Up

O
ne of the pleasures of working on the bleeding edge of Windows® is poking around in a new technology to see how it works. I don't really feel comfortable with an operating system until I have a little under-the-hood knowledge. So when the 64-bit editions of Windows XP and Windows Server™ 2003 appeared on the scene, I was all over them.

The nice thing about Win64 and the x64 CPU architecture is that they're different enough from their predecessors to be interesting, while not requiring a huge learning curve. While we developers would like to think that moving to x64 is just a recompile away, the reality is that we'll still spend far too much time in the debugger. A good working knowledge of the OS and CPU is invaluable.

In this article I'll boil down my experiences with Win64 and the x64 architecture to the essentials that a hotshot Win32® programmer needs for the move to x64. I'll assume that you know basic Win32 concepts, basic x86 concepts, and why your code should run on Win64. This frees me to focus on the good stuff. Think of this overview as a look at just the important differences relative to your knowledge of Win32 and the x86 architecture.

One nice thing about x64 systems is that you can use either Win32 or Win64 on the same machine without serious performance losses, unlike Itanium-based systems. And despite a few obscure differences between the Intel and AMD x64 implementations, the same x64-compatible build of Windows should run on either. You don't need one version of Windows for AMD x64 systems and another for Intel x64 systems.

I've divided the discussion into three broad areas: OS implementation details, just enough x64 CPU architecture to get by, and developing for x64 with Visual C++®.

The x64 Operating System

In any overview of the Windows architecture, I like to start with memory and the address space. Although a 64-bit processor could theoretically address 16 exabytes of memory (2^64), Win64 currently supports 16 terabytes, which is represented by 44 bits. Why can't you just load a machine up with 16 exabytes to use all 64 bits? There are a number of reasons.

For starters, current x64 CPUs typically only allow 40 bits (1 terabyte) of physical memory to be accessed. The architecture (but no current hardware) can extend this to up to 52 bits (4 petabytes) Even if that restriction was removed, the size of the page tables to map that much memory would be enormous.

Just as in Win32, the addressable range is divided into user and kernel mode areas. Each process gets its own unique 8TB at the bottom end, while kernel mode code lives in the upper 8 terabytes and is shared by all processes. The different versions of 64-bit Windows have different physical memory limits as shown in Figure 1 and Figure 2.

Figure 2 Physical Memory and CPU Limits (Close Figure 2)

Physical Memory and CPU Limits | 32-Bit Models | 64-Bit Models
Windows XP Professional | 4GB (1-2 CPUs) | 128GB (1-2 CPUs)
Windows Server 2003, Standard Edition | 4GB (1-4 CPUs) | 32GB (1-4 CPUs)
Windows Server 2003, Enterprise Edition | 64GB (1-8 CPUs) | 1TB (1-8 CPUs)
Windows Server 2003, Datacenter Edition | 64GB (8-32 CPUs) | 1TB (8-64 CPUs)
Figure 1 General Memory Limits (Close Figure 1)

 | 32-Bit Models | 64-Bit Models
Total virtual address space (based on a single process) | 4GB | 16TB
Virtual address space per 32-bit process | 2GB (3GB if system is booted with /3GB switch) | 4GB if compiled with /LARGEADDRESSAWARE (2GB otherwise)
Virtual address space per 64-bit process | Not applicable | 8TB
Paged pool | 470MB | 128GB
Non-paged pool | 256MB | 128GB
System Page Table Entry (PTE) | 660MB to 900MB | 128GB

Also just like in Win32, the x64 page size is 4KB. The first 64KB of address space is never mapped in, so the lowest valid address you'd expect to see is 0x10000. Unlike in Win32, system DLLs don't have a default load address near the top of the user mode address range. Instead, they're loaded above 4GB, typically at addresses around 0x7FF00000000.

A nice feature of many newer x64 processors is support for the CPU No Execute bit that Windows uses to implement hardware Data Execution Protection (DEP). On the x86 platform, many bugs and viruses exist because the CPU can execute data as if it were legal code bytes. A buffer overrun (intentional or not) can end up with the CPU executing through memory that was intended for data storage. With DEP, the OS can set much more clear boundaries around valid code regions, thus causing the CPU to trap if execution goes outside these expected boundaries. This helps in the continuing battle to make Windows less vulnerable to attack.

In a move designed to catch errors, the x64 linker assigns default load addresses for executables just above 32 bits (4GB). This helps you to quickly find these areas in existing code after the code has been ported to Win64. Specifically, if a pointer is stored in a 32-bit sized value (a DWORD, for example), it will effectively be truncated when running in a Win64 build, making the pointer invalid and thus triggering an access violation. This trick makes it much easier to find those nasty pointer bugs.

The subject of pointers and DWORDs segues nicely into the Win64 type system. How big is a pointer? How about a LONG? And what about handles, like HWNDs? Mercifully, when Microsoft made the messy transition from Win16 to Win32, they made the new type models easy to extend further to 64 bits. Generally speaking, and with a few exceptions, all types other than pointers and size_t are exactly the same in the new 64-bit world as in Win32. That is, a 64-bit pointer is 8 bytes, while int, long, DWORD, and HANDLE are still 4 bytes. I'll talk more about types later when I discuss developing for Win64. [ Editor's Update - 5/2/2006: Handles are defined as pointer values. Thus in Win64, a handle is 8 bytes, not 4 bytes.]

The file format for Win64 is called PE32+. From nearly every viewpoint, the format is structurally identical to the Win32 PE file. A very few fields such as the ImageBase in the header have been widened, one field was deleted, and one field was changed to reflect a different CPU type. Figure 3 shows the fields that have changed.

Figure 3 Changes to PE File FIelds (Close Figure 3)

Header Field | Change
Magic | Set to 0x20b instead of 0x10b
BaseOfData | Deleted
ImageBase | Widened to 64 bits
SizeOfStackReserve | Widened
SizeOfStackCommit | Widened
SizeOfHeapReserve | Widened
SizeOfHeapCommit | Widened

Beyond the PE header, there aren't many changes. A few structures such as IMAGE_LOAD_CONFIG and IMAGE_THUNK_DATA simply had some of their fields widened to 64 bits. The addition of the PDATA section is more interesting, as it highlights one of the major differences between the Win32 and Win64 implementation: exception handling.

In the x86 world, exception handling is stack-based. When a Win32 function contains try/catch or try/finally code, the compiler emits instructions that create a small data block on the stack. In addition, each try data block points to the previous try data structure, thus forming a linked list with the most recently added structure at the list head. As functions are called and exited, the head of the linked list keeps updating. When an exception occurs, the OS walks the linked list of blocks on the stack, looking for the appropriate handler. My January 1997 MSJ article describes this in much more detail, so I'll keep the description to a minimum here.

In contrast to the Win32 exception handling, Win64 (both x64 and Itanium versions) uses table-based exception handling. No linked list of try data blocks is built on the stack. Instead, each Win64 executable contains a runtime function table. Each function table entry contains both the starting and ending address for the function, as well as the location of a rich set of data about exception-handling code in the function and the function's stack frame layout. See the IMAGE_RUNTIME_FUNCTION_ENTRY structure in WINNT.H and in the x64 SDK for the nitty-gritty on these structures.

When an exception occurs, the OS walks the regular thread stack. As the stack walk encounters each frame and saved instruction pointer, the OS determines within which executable module the instruction pointer lies. The OS then searches the runtime function table in that module, locates the appropriate runtime function entry, and makes the appropriate exception-processing decisions from that data.

What if you're a rocket scientist and you've generated code directly in memory without an underlying PE32+ module? You're still covered in this case. Win64 has a RtlAddFunctionTable API that lets you tell the OS about your dynamically generated code.

The downside to table-based exception handling (relative to the x86 stack-based model) is that looking up function table entries from code addresses takes more time than just walking a linked list. The upside is that functions don't have the overhead of setting up a try data block every time the function executes.

Remember, this is just a quick introduction rather than a full fledged description of x64 exception processing, however exciting that might be! For a more in-depth overview of the x64 exception model, be sure to read Kevin Frei's blog entry.

x64-compatible versions of Windows don't contain dramatic numbers of truly new APIs; most new Win64 APIs were added to the Windows releases for Itanium processors. In the interest of keeping things brief, the main two existing APIs of importance are IsWow64Process and GetNativeSystemInfo. These allow Win32 apps to determine if they're running on Win64, and if so, the true capabilities of the system. Otherwise, a 32-bit process that calls GetSystemInfo only sees the system capabilities as if it's a 32-bit system. For instance, GetSystemInfo can only report the address range of 32-bit processes. Figure 4 shows the APIs that were not previously available on x86, but are available for x64.

Figure 4 New 64-Bit APIs (Close Figure 4)

Functionality | API
Exception Handling | RtlAddFunctionTable RtlDeleteFunctionTable RtlRestoreContext RtlLookupFunctionEntry RtlInstallFunctionTableCallback
Registry | RegDeleteKeyEx RegGetValue RegQueryReflectionKey
NUMA (Non-Uniform Memory Access) | GetNumaAvailableMemoryNode GetNumaHighestNodeNumber GetNumaNodeProcessorMask GetNumaProcessorNode
WOW64 Redirection | Wow64DisableWow64FsRedirection Wow64RevertWow64FsRedirection RegDisableReflectionKey RegEnableReflectionKey
Miscellaneous | GetLogicalProcessorInformation QueryWorkingSetEx SetThreadStackGuarantee GetSystemFileCacheSize SetSystemFileCacheSize EnumSystemFirmwareTables GetSystemFirmwareTable

While running a fully 64-bit Windows system sounds great, the reality is that you'll very likely need to run Win32 code for a while. Towards that end, x64 versions of Windows include the WOW64 subsystem that lets Win32 and Win64 processes run side-by-side on the same system. However, loading your 32-bit DLL into a 64-bit process, or vice versa, isn't supported. (It's a good thing, trust me.) And you can finally kiss good bye to 16-bit legacy code!

In x64 versions of Windows, a process that starts from a 64-bit executable such as Explorer.exe can only load Win64 DLLs, while a process started from a 32-bit executable can only load Win32 DLLs. When a Win32 process makes a call to kernel mode—to read a file, for instance—the WOW64 code intercepts the call silently and invokes the correct x64 equivalent code in its place.

Of course, processes of different lineages (32-bit versus 64-bit) need to be able to communicate with each other. Luckily, all the usual interprocess communication mechanisms that you know and love in Win32 also work in Win64, including shared memory, named pipes, and named synchronization objects.

You might be thinking, "But what about the system directory? The same directory can't hold both 32- and 64-bit versions of system DLLs such as KERNEL32 or USER32, right?" WOW64 magically takes care of this for you by the doing selective file system redirection. File activity from a Win32 process that would normally go to the System32 directory instead goes to a directory named SysWow64. Under the covers, WOW64 silently changes these requests to point at the SysWow64 directory. A Win64 system effectively has two \Windows\System32 directories—one with x64 binaries, the other with the Win32 equivalents.

Smooth as it may seem, this can be confusing. For instance, I was at one point using (unbeknownst to me) a 32-bit command-line prompt. When I ran DIR on Kernel32.dll in the System32 directory, I got the exact same results as when I did the same thing in the SysWow64 directory. It took a lot of head scratching before I figured out that the file system redirection was working just like it should. That is, even though I thought I was working in the \Windows\System32 directory, WOW64 was redirecting the calls to the SysWow64 directory. Incidentally, if you really do want to get at the 32-bit \Windows\System32 directory from an x64 app, the GetSystemWow64Directory API gives you the correct path. Be sure to read the MSDN® documentation for the complete story.

Beyond file system redirection, another bit of magic performed by WOW64 is registry redirection. Consider my earlier statement about Win32 DLLs not loading into Win64 processes, and then think about COM and its use of the registry to load in-process server DLLs. What if a 64-bit application uses CoCreateInstance to create an object that's implemented in a Win32 DLL? The DLL can't load, right? WOW64 saves the day again by redirecting access from 32-bit applications to the \Software\Classes (and related) registry nodes. The net effect is that Win32 applications have a different (but mostly parallel) view of the registry than x64 applications. As you'd expect, the OS provides an escape hatch for 32-bit applications to read the actual 64-bit registry value by specifying new flag values when calling RegOpenKey and friends.

Drilling down a bit, the last few OS differences near and dear to my heart concern thread local data. In x86 versions of Windows, the FS register is used to point at per-thread memory areas, including the "last error" and Thread Local Storage (GetLastError and TlsGetValue, respectively). On x64 versions of Windows, the FS register has been replaced by the GS register. Otherwise they work pretty much in the same manner.

Although this article focuses on x64 from the user mode perspective, there is one important kernel mode architectural addition to point out. New in Windows for x64 is a technology called PatchGuard, which is aimed at both security and robustness. In a nutshell, user mode programs or drivers that alter key kernel data structures such as the syscall tables and the interrupt dispatch table (IDT) create security holes and potential stability problems. For the x64 architecture, the Windows folks decided that modifying kernel memory in unsupported ways wouldn't be tolerated. The technology to enforce this is PatchGuard. It uses a kernel mode thread to monitor changes to critical kernel memory locations. If that memory is changed, the system stops via a bugcheck.

All things considered, if you're familiar with the Win32 architecture and how to write native code that runs on it, you won't find many surprises in the move to Win64. You can consider it to be mostly just a roomier environment.

Just Enough x64 to Get By

Now let's take a look at the CPU architecture itself, since a basic understanding of the CPU's instruction set makes development (especially debugging!) much easier. The first thing you'll notice in compiler-generated x64 code is how remarkably similar it is to the x86 code you know and love. This definitely wasn't the case for you folks who learned Intel IA64 coding.

The second thing you'll notice shortly thereafter is that the register names are slightly different than you're used to, and that there's a lot more of them. General-purpose x64 registers have names that begin with R, as in RAX, RBX, and so on. This is an evolution of the old E-based naming scheme for 32-bit x86 registers. Way back in the mists of time, the 16-bit AX register became the 32-bit EAX, the 16-bit BX became the 32-bit EBX, and so on. Transitioning from the 32-bit versions, all the E registers become R registers in their 64-bit incarnations. Thus, RAX is the successor to EAX, RBX succeeds EBX, RSI replaces ESI, and so on down the line.

In addition, eight new general-purpose registers (R8-R15) were added. The list of primary 64-bit general-purpose registers looks like Figure 5.

Figure 5 (Close Figure 5)

RAX
RBX
RCX
RDX
RSI
RDI
RSP
RBP
R8
R9
R10
R11
R12
R13
R14
R15

Also, the 32-bit EIP register becomes the RIP register. Of course 32-bit instructions must continue to run, so the original, smaller form factor versions of these registers (EAX, AX, AL, AH, and so on) are still available.

Lest you graphics and scientific programming gurus feel left out, the x64 CPU also has 16 128-bit SSE2 registers, which are named XMM0 through XMM15. The full set of x64 registers preserved by Windows can be found in the appropriately #ifdef'ed _CONTEXT structure defined in WINNT.H.

At any given time, an x64 CPU is operating in either legacy 32-bit mode or in 64-bit mode. In 32-bit mode, the CPU decodes and acts on instructions just like any other x86 class CPU. In 64-bit mode, the CPU has made some slight adjustment to certain instruction encodings to support the new registers and instructions.

If you're familiar with the CPU opcode encoding diagrams, you'll remember that the space for new instruction encodings was disappearing fast, and squeezing in eight new registers isn't an easy task. One way to do this was to eliminate a few rarely used instructions. So far, the only instructions I miss are 64-bit versions of PUSHAD and POPAD, which save and restore all the general purpose registers on the stack. Another way was to free up instruction encoding space was to eliminate segments entirely in 64-bit mode. Thus the life of CS, DS, ES, SS, FS, and GS come to an end. Not that many people will miss them.

With addresses being 64 bits, you might be wondering about code size. For instance, this is a common 32-bit instruction:

CALL DWORD PTR [XXXXXXXX]
Here, the X'ed portion is a 32-bit address. In 64-bit mode, does this become a 64-bit address, thereby turning a 5-byte instruction into 9 bytes? Luckily, the answer is no. The instruction remains the same size. In 64-bit mode, the 32-bit operand portion of the instruction is treated as a data offset relative to the current instruction. An example makes this clearer. In 32-bit mode, here's the instruction to call the 32-bit pointer value stored at address 00020000h:
00401000: CALL DWORD PTR [00020000h]
In 64-bit mode, the same opcodebytes call the 64-bit pointer value stored at address 00421000h (4010000h + 20000h). A little thought reveals that this relative addressing mode has important ramifications if you're generating code yourself. You can't just specify an 8-byte pointer value in an instruction. Instead, you need to specify a 32-bit relative address to a memory location where the actual 64-bit target address resides. Thus, there's an unspoken assumption that the 64-bit target pointer must lie within 2GB of the instruction that uses it. Not that big a deal for most folks, but if you do dynamic code generation or modify existing code in memory, it can byte you!

A key advantage of all the x64 registers is that compilers can finally generate code that passes most parameters in registers rather than on the stack. Pushing parameters on the stack incurs memory accesses. We've all had it drilled into our heads that a memory access that's not found in the CPU cache causes the CPU to stall for many cycles waiting for your regular RAM memory to catch up.

In designing the calling convention, the x64 architecture took advantage of the opportunity to clean up the mess of existing Win32 calling conventions such as __stdcall, __cdecl, __fastcall, _thiscall, and so on. In Win64, there's just one native calling convention, and modifiers like __cdecl are ignored by the compiler. The reduction in calling convention flavors is a wonderful boon for debuggability, among other things.

The primary thing to know about the x64 calling convention is its similarity to the x86 fastcall convention. Using the x64 convention, the first four integer arguments (from left to right) are passed in 64-bit registers designated for that purpose:

RCX: 1st integer argument
RDX: 2nd integer argument
R8: 3rd integer argument
R9: 4th integer argument

Integer arguments beyond the first four are passed on the stack. The this pointer is considered an integer argument, so can always be found in the RCX register. As for floating point parameters, the first four are passed in the XMM0 through XMM3 registers, with subsequent floating point parameters placed on the thread stack.

Drilling into the calling convention a bit, even though an argument can be passed in a register, the compiler still reserves space on the stack for it by decrementing the RSP register. At a minimum, each function must reserve 32 bytes (four 64-bit values) on the stack. This space allows registers passed into the function to be easily copied to a well-known stack location. The callee function isn't required to spill the input register params to the stack, but the stack space reservation ensures that it can if needed. Of course, if more than four integer parameters are passed, the appropriate additional amount of stack space must be reserved.

Let's look at an example. Consider a function passing two integer parameters to a child function. The compiler not only sticks the values in RCX and RDX, it also subtracts 32 bytes from the RSP stack pointer register. In the callee function, the parameters can be accessed in the registers (RCX and RDX). If the callee code needs the register for some other purpose, it can copy the registers into the reserved 32-byte stack region. Figure 6 shows the registers and stack after six integer parameters have been passed.

Figure 6 Passing Integers
Figure 6 Passing Integers

Parameter stack cleanup is a bit funny on x64 systems. Technically, the caller is responsible for cleaning up the stack, not the callee. However, you'll rarely see RSP adjusted anywhere other than in the prologue and epilogue code. Unlike the x86 compiler, which explicitly adds and removes parameters on to the stack with PUSH and POP instructions, the x64 code generator reserves enough stack space to call whatever the largest target function (parameter-wise) uses. It then uses the same stack region over and over to set up the parameters when calling child functions.

Put another way, the RSP rarely changes. This is quite different from x86 code, where the ESP value fluctuates as parameters are added and cleared from the stack.

An example helps here. Consider an x64 function that calls three other functions. The first function takes four params (0x20 bytes), the second takes 12 params (0x60 bytes), and the third takes eight params (0x40 bytes). In the prologue, the generated code simply reserves 0x60 bytes on the stack and copies parameter values into the appropriate spots within the 0x60 bytes so that the target functions can locate them.

A good description of the more detailed intricacies of the x64 calling convention can be found in Raymond Chen's blog. I won't belabor all the details, but here are some highlights. First, integer parameters that are less than 64-bits are sign extended, then still passed via the appropriate register, if among the first four integer parameters. Second, at no point should any parameter be in a stack location that's not a multiple of 8 bytes, thus preserving 64-bit alignment. Any argument that's not 1, 2, 4, or 8 bytes (including structs) is passed by reference. And finally, structs and unions of 8, 16, 32, or 64-bits are passed as if they were integers of the same size.

A function's return value is stored in the RAX register. The exception is for floating-point types, which are returned in XMM0. Across calls, these registers must be preserved: RBX, RBP, RDI, RSI, R12, R13, R14, and R15. These register are volatile and can be destroyed: RAX, RCX, RDX, R8, R9, R10, and R11.

Earlier I mentioned that the OS walks stack frames as part of the exception handling mechanism. If you've ever written stack-walking code, you know that the almost ad hoc nature of Win32 frame layout makes the process a tricky proposition. The situation is much better on x64 systems. If a function allocates stack space, calls other functions, preserves any registers, or uses exception handling, that function must use a well-defined set of instructions for generating standard prologues and epilogues.

Enforcing a standard way of creating a function's stack frame is one way the OS can guarantee (in theory) that the stack can always be walked. In addition to consistent, standardized prologues, the compiler and linker must also create an associated function table data entry. For the curious, all these function entries end up in table that's an array of IMAGE_FUNCTION_ENTRY64, defined in winnt.h. How do you find this table? It's pointed to by the IMAGE_DIRECTORY_ENTRY_EXCEPTION entry in the PE header's DataDirectory field.

I've covered a lot of architectural ground in a short amount of space. However, with an understanding of these big picture concepts and an existing knowledge of 32-bit assembly language, you should be able to understand x64 instruction in the debugger within a relatively short period of time. As always, practice makes perfect.

Developing for x64 with Visual C++

Although it was possible to write x64 code with the Microsoft® C++ compiler prior to Visual Studio® 2005, it was a clunky experience in the IDE. For this article I'll therefore assume that you're working with Visual Studio 2005 and that you've selected the x64 tools, which aren't enabled in a default installation. I'll also assume that you have an existing Win32 user mode project in C++ that you'd like to build for both x86 and x64 platforms.

The first step in building for x64 is to create the 64-bit build configuration. As a good Visual Studio user, you're already aware that your projects have two configurations by default: Debug and Retail. All you need to do here is create two more configurations: Debug and Retail in their x64 guises.

Begin with your existing project/solution loaded. On the Build menu, select Configuration Manager. In the Configuration Manager dialog box, from the Active solution platform dropdown menu, select New (see Figure 7). You should now see another dialog entitled New Solution Platform.

Figure 7 Creating a New Build Configuration
Figure 7 Creating a New Build Configuration

Select x64 as your new platform (see Figure 8) and leave the other settings in their default state; then click OK. That's it! You should now have four possible build configurations: Win32 Debug, Win32 Retail, x64 Debug, and x64 Retail. Using the Configuration Manager, you can easily switch between them.

Now let's see how x64-compliant your code is. Make the x64 Debug configuration the default, and then build the project. Unless the code is trivial, odds are that you'll get some compiler errors that don't occur in the Win32 configuration. Unless you've completely forsaken all principles of writing portable C++ code, it's relatively easy to fix these issues so that your code is truly Win32 and x64 ready, without requiring reams of conditionally compiled code.

Figure 8 Selecting the Build Platform
Figure 8 Selecting the Build Platform

Making Your Code Win64-Compliant

Probably the biggest effort in converting Win32 code to x64 is in getting your type definitions correct. Remember my earlier discussion of the Win64 type system? By using the Windows typedef types rather than the C++ compiler's native types (int, long, and so on) the Windows headers make it easy to write clean Win32 x64 code. You should continue this consistency in your own code. For instance, if Windows passes you an HWND, don't store it in a FARPROC just because it's handy and easy.

Having upgraded a lot of code, perhaps the most common and easy error I've seen is in assuming that a pointer value can be stored or transported in a 32-bit type such as an int, long, or even a DWORD. Pointers in Win32 and Win64 are different sizes by necessity, while integer types remain the same size. However, it's also not feasible for the compiler to disallow pointers from being stored in an integral type. It's a C++ habit that's just too ingrained.

To the rescue come the _PTR types defined in the Windows headers. Types such as DWORD_PTR, INT_PTR, and LONG_PTR let you declare variables that are of integral type, but that are always wide enough to store a pointer on the target platform. For instance, a variable defined as type DWORD_PTR is a 32-bit integer when compiled for Win32 and 64-bit when compiled for Win64. With practice, it became second nature for me when declaring types to ask, "Do I want a DWORD here, or do I really mean DWORD_PTR?"

As you'd expect, there might be occasions when you specify exactly how many bytes you need for an integer type. The same header file (Basetsd.h) that defines DWORD_PTR and friends also defines size specific integers such as INT32, INT64, INT16, UINT32, and DWORD64.

Another issue related to type size differences is printf and sprintf formatting. I'm certainly guilty of using %X or %08X to format pointer values in the past, and have been bitten when I ran that code on x64 systems. The correct way is to use %p, which automatically accounts for the pointer size on the target platform. In addition, printf and sprintf have the I prefix for size-dependent types. For instance, you might use %Iu to print out a UINT_PTR variable. Likewise, if you know the variable will always be a 64-bit signed value, you could use %I64d.

Having cleaned up errors caused by type definitions that aren't Win64 ready, you may still have code that can only run in x86 mode. Or perhaps you need to write two versions of a function, one for Win32 and the other for x64. This is where a set of preprocessor macros come in handy:

_M_IX86
_M_AMD64
_WIN64
Proper use of the preprocessor macros is essential to writing correct cross-platform code. _M_IX86 and _M_AMD64 are defined only when compiling for the specified processor. _WIN64 is defined when compiling for any 64-bit version of Windows, including the Itanium edition.

When using a preprocessor macro, think hard about what you want. For instance, is the code truly specific to the x64 processor and nothing else? Then use something like:

#ifdef _M_AMD64
On the other hand, if the same code could work on both x64 and Itanium, you might be better off with something like:
#ifdef _WIN64

A convention that I've found useful is that whenever I use one of these macros, I always explicitly create the #else cases so that I know early if I've forgotten something. Consider the following badly written code:

#ifdef _M_AMD64
// My x64 code here
#else
// My x86 code here
#endif
What happens if I now compile this for a third CPU architecture? My x86 code will unintentionally be compiled. A much better way to phrase the previous code is like this:
#ifdef _M_AMD64
// My x64 code here
#elif defined (_M_IX86)
// My x86 code here
#else
#error !!! Need to write code for this architecture
#endif

The one part of my Win32 code that didn't port easily to x64 was my inline assembler, which Visual C++ doesn't support for the x64 target. Fear not, assembler heads. A 64-bit MASM (ML64.exe) is provided and is documented via MSDN. ML64.exe and other x64 tools (including CL.EXE and LINK.EXE) are available from the command line. You can just run the VCVARS64.BAT file, which adds them to your path.

Debugging

You've finally gotten your code to compile cleanly on Win32 and x64 builds. The final piece of the puzzle is running and debugging it. Regardless of whether you build your x64 version on an x64 box, you'll need to use the Visual Studio remote debugging features to debug in x64 mode. Luckily, if you're running the Visual Studio IDE on the 64-bit machine, the IDE takes care of all of the following steps for you. If, for some reason, you can't use remote debugging, your other option is to use the x64 version of WinDbg. However, you'll be giving up many of the debugging niceties found in the Visual Studio debugger.

If you've never used remote debugging, there's not much cause for concern. Once you get it set up, remote debugging can be as seamless as local.

The first step is to install the 64-bit MSVSMON on the target machine. This is typically done by running the RdbgSetup program that comes with Visual Studio. Once MSVSMON is running, use the Tools menu to configure the appropriate security settings (or lack thereof) for the connection between your 32-bit Visual Studio and the MSVSMON instance.

Next, within Visual Studio you'll want to configure your project to use remote debugging for x64 code, rather than attempting local debugging. You can start this process by bringing up the project's properties (see Figure 9).

Figure 9 Debugging Properties
Figure 9 Debugging Properties

Make sure your 64-bit configuration is current and select Debugging under Configuration Properties. Near the top is a drop-down menu titled Debugger to launch. Normally this is set to Local Windows Debugger. Change this to Remote Windows Debugger. Below that, you can specify the remote command to execute when you start debugging (the program name, for example) as well as the remote machine name and connection type.

If you set up everything properly, it's possible to start debugging your x64 target application in the same way you start Win32 apps. You'll know you've successfully connected to the MSVSMON because its trace window emits a "connected" string each time the debugger successfully attaches. From here, it's mostly the same Visual Studio debugger that you know and love. Be sure to bring up the registers window and look at all those glorious 64-bit registers, and pop into the disassembly window to check out that oh-so-familiar-but-just-a-little-different x64 assembly code.

Note that a 64-bit minidump can't be loaded directly into Visual Studio like 32-bit dumps. Instead, you'll need to use the Remote Debugging. Also, interop debugging between native and managed 64-bit code isn't currently supported in Visual Studio 2005.

What About Managed Code?

One of the great things about coding with the Microsoft .NET Framework is that much of the underlying operating system is abstracted away for general-purpose code. In addition, the IL instruction format is CPU agnostic. As a result, at a theoretical level, it should possible for a .NET-based program binary built on a Win32 system to run unmodified on an x64 system. The reality is a little bit more complicated.

The .NET Framework 2.0 comes with an x64 version. After installing this on my x64 machine, I was able to run the same .NET executables that I'd previously run on my Win32 box. How cool is that? Of course there's no guarantee that every single .NET-based program will run equally well on Win32 and x64 without a recompile, but it does "just work" a reasonable percentage of the time.

If your managed code explicitly invokes native code (for instance, through P/Invoke in C# or Visual Basic®), you will very likely run into trouble if you try to run against the 64-bit CLR. However, there is a compiler switch (/platform) that allows you to be more specific about which platform your code should run on. For instance, you might want your managed code to run in WOW64, even though a 64-bit CLR is available.

Wrap Up

All things considered, moving to an x64 version of Windows was a relatively painless experience for me. Once you have a good grasp of the relatively minor differences in the OS architecture and tools, it's easy to keep one code base running on both platforms. Visual Studio 2005 makes the effort substantially easier. And more x64-specific versions of device drivers and tools such as Process Explorer from SysInternals.com are appearing every day, so there's no reason not to jump in!

Matt Pietrek has co-written several books on Windows system-level programming as well as the Under the Hood column for MSDN Magazine. Previously he was a primary architect for the NuMega/Compuware BoundsChecker series of products. He now works on the Visual Studio team at Microsoft.


Mix And Match: Integrate Windows Forms Into Your MFC Applications Through C++ Interop

Mix and Match

Integrate Windows Forms Into Your MFC Applications Through C++ Interop

Marcus Heege
This article discusses:
  • How C++ interop works
  • When to use C++ interop in your projects
  • Using Windows Forms controls in MFC apps
  • Moving from Windows Forms to WPF views
This article uses the following technologies:
C++, MFC, .NET Framework, Windows Presentation Foundation
Contents
C++ Interop Is Your Friend
When to Use C++ Interop
Windows Forms Controls in MFC Apps
Handling Control Events
Controls as Dialog Boxes
Windows Forms Views
Bridges to Avalon
Conclusion

A
lthough the Microsoft .NET Framework was first made public almost five years ago, and version 2.0 has just recently been released, many C++ applications are still pure unmanaged code. But interest in the .NET Framework is growing rapidly among C++ developers. And for good reason: many future Windows® APIs will be based on the .NET Framework. This is certainly true for the WinFX® components Windows Presentation Foundation, Windows Communication Foundation, and Windows Workflow Foundation.

Still, many C++ developers want to use native APIs instead of wrappers around existing APIs. Wrappers are often assumed to be buggy, slow, and less flexible. Apart from that, it would be really difficult to map huge APIs like WinFX so that they can be used from native code.

In most cases, extending C++ applications with features from the .NET Framework is easier than most developers believe. Visual C++® includes a feature called C++ interop, sometimes referred to as IJW, or "It Just Works." You can use this feature to seamlessly integrate .NET-based code into your existing C++ source code.

C++ interop is based on two major features. First, you can compile your existing C++ code to Microsoft® intermediate language (MSIL) by using the /clr switch of the C++ compiler. Your code can then use .NET features like garbage collection, sandboxed security, and types from the huge .NET Framework base class library.

The other feature is called mixed-mode, and it is equally important. It refers to the combination of both managed and unmanaged code. When compiling C++ code to MSIL, the compiler creates managed object files containing MSIL code instead of the normal, unmanaged object files containing native assembly code. The linker is able to take both managed and unmanaged object files as input. When the linker detects at least one managed input, it creates a .NET assembly, which can contain both managed code and unmanaged code. That"s why it is called mixed-mode (see Figure 1). This feature is extremely important for performance optimizations because it allows you to drastically reduce the number of transitions between the managed and unmanaged spaces.

Figure 1 MTS
Figure 1 MTS

C++ Interop Is Your Friend

My experience has shown that C++ interop is very reliable, and while it has some limitations, it"s a worthwhile addition to your developer toolbox. To understand how this feature is possible, you should be aware that C++ interop was a design feature of the .NET Framework. The Common Language Infrastructure (CLI), which is the base specification of the .NET Framework, has been designed to support this feature. In fact, there are many aspects of the CLI that were defined simply because C++ interop required them.

For example, managed metadata supports global functions. Yet C# and most other .NET-targeted languages require all methods to exist within the scope of a class. This feature exists because C++ code that should be mapped to .NET-based code can have global functions, and to map this C++ code to managed code, metadata needs an equivalent concept.

The MSIL instruction set has also been designed with C++ interop in mind. To map C++ code to MSIL code, MSIL must support all common data manipulation operations like the typical Boolean, arithmetic, and floating point operations. MSIL also supports pointer-based access to virtual memory. This feature is key to using native types in managed code. If your C++ code accesses a field of a native object, the C++/CLI compiler can emit MSIL code that pushes the address of the native object on the stack, adds the field"s offset to it, and uses the resulting address to access the field. All of this is possible because the MSIL instruction set has operations that can access virtual memory through addresses.

Another C++-specific feature of the IL language is the CALLI instruction. It can be used to call to native functions through function pointers. This feature is used to call virtual functions of native types. It is important because COM interfaces are native types with virtual functions. This method of COM interoperability has many advantages over the alternative COM interoperability techniques used by other languages.

Finally it"s worth mentioning the design impact that is most obvious to all developers who use .NET: the file format for components. While Java has defined its own file format (the .class file), the .NET Framework uses the standard PE file format. Therefore, .NET-based components have the comfortable file extensions DLL and EXE. The fact that PE files are still used is essential to supporting mixed-mode assemblies.

When to Use C++ Interop

To evaluate whether C++ interop is the way to go for your projects, you should also know about the impact that this feature has on the way you build your projects—compiler settings, for starters—as well as the impact on the output of this build process.

If you compile with the /clr option, your code implicitly depends on the multithreaded DLL version of the C runtime libraries (CRT). This means that all files in your project—even those compiled without /clr—must use the multithreaded DLL version of the CRT. This is also important for MFC projects. The static MFC library depends on the static CRT library, which is incompatible with /clr compilation. Therefore you have to make sure that your MFC projects link the DLL version of MFC.

There can also be issues regarding execution time. Initializing the CLR takes a certain amount of time, and JIT compilation has some overhead, too. Therefore you may experience a somewhat longer startup time. However, for most projects this should not be a big issue.

Method calls with managed-to-unmanaged transitions will take longer than normal method calls. As I have mentioned already, you can significantly reduce the number of transitions between managed and unmanaged code by deciding which specific parts of your app should be compiled to managed code. The calling convention __clrcall is another important optimization feature that can significantly reduce the number of transitions, but this feature is beyond the scope of this article.

In most cases, the just-in-time (JIT)-compiled code itself is not the cause of significant performance penalties. In fact, there are some optimizations that a JIT compiler can do that are not possible for the C++ compiler and the linker. For example, a JIT compiler can optimize code for the processor architecture on the target machine. In contrast to the C++ compiler, a JIT compiler can also provide cross-component inlining. This can be a very effective optimization, because code compiled with cross-component inlining can then be exposed to further optimizations.

Apart from the issues I"ve mentioned here, there are some other areas that will be impacted that I will not cover in detail here. Most of these can be easily resolved by modifying compiler and linker settings in the right way. The optimal compiler settings are not always straightforward. Some of these settings turn on features that are mandatory for C++ interop compilation, while some other settings may turn off features that are incompatible with /clr compilation or that are solved for Microsoft intermediate language in a different manner.

Windows Forms Controls in MFC Apps

All source files that are compiled with the /clr option can use types from the .NET Framework base class library. You can use most parts of the base class library as you have always used them. Some cases require special attention. Integrating Windows Forms controls in MFC applications is such an example.

You cannot pass System::Windows::Forms::Control^ when a CWnd* is expected. However, the Windows Forms API and MFC share a common heritage: good old User32.dll. Both APIs provide some backdoors to the HWNDs. While the MFC CWnd class provides a conversion operator to HWND, the Windows Forms control class implements the interface IWin32Window from the namespace System::Windows::Forms to expose the window handle:

public interface IWin32Window {
  property IntPtr Handle {
    IntPtr get ();
  }
};

However, getting a window handle from a window object is often not enough. MFC provides some classes with support for dealing with Windows Forms controls in different scenarios. These classes are declared in afxwinforms.h.

The most important of these classes is CWinFormsControl, which is defined in the namespace Microsoft::VisualC::MFC. This class allows you to host a Windows Forms control in an MFC Dialog or CDialog-derived class. It is a templated class with one template argument, used to pass the type of the Windows Forms control you want to host. The type can be either the concrete type you want to instantiate later or a base class of the concrete type. The CWinFormsControl template is quite simple. It has only two interesting features: it inherits CWnd and it provides several overloads of a self-explaining method, CreateManagedControl.

If you want to host a Windows Forms control in an MFC Dialog or CDialog-derived class, define a member variable of type CWinFormsControl<TWinFormsCtrl> in your class and pick one of the overloads of CreateManagedControl. If you implement a CDialog-derived class, a good candidate for creating the managed control is OnInitDialog. For most other cases, OnCreate (the message handler for WM_CREATE) is the best option. One of the overloads is especially interesting for dialog classes. This overload expects you to add a static placeholder control to a dialog resource. Place the static control so that it has the position and size that the Windows Forms control should have and give it a unique ID, as shown in Figure 4. To instantiate the hosted control, call CreateManagedControl as shown in Figure 2. As an alternative, you can even use a special variant of Dialog Data Exchange (DDX) like in Figure 3.

Figure 3 Hosting a Control with Dialog Data Exchange (Close Figure 3)
using Microsoft::VisualC::MFC;
using System::Windows::Forms;

class CMyDialog : public CDialog
{
public:
    CMyDialog(CWnd* pParent = NULL): CDialog(CMyDialog::IDD, pParent) {}

    enum { IDD = IDD_MYDIALOG };

private:
    CWinFormsControl<Button> m_wfBtn;

protected:
    virtual void DoDataExchange(CDataExchange* pDX) {
        DDX_ManagedControl(pDX, IDC_WFBTN, this->m_wfBtn);
    }
};
Figure 2 Creating a Hosted Control (Close Figure 2)
using Microsoft::VisualC::MFC;
using System::Windows::Forms;

class CMyDialog : public CDialog
{
public:
    CMyDialog() : CDialog(CMyDialog::IDD, NULL) {}

    enum { IDD = IDD_MYDIALOG };

private:
    CWinFormsControl<Button> m_wfBtn;

public:
    BOOL OnInitDialog() {
        CDialog::OnInitDialog();

        return this->m_wfBtn.CreateManagedControl(
            WS_VISIBLE | WS_CHILD, IDC_WFBTN, this));
    }
}
Figure 4 Using an ActiveX Control
Figure 4 Using an ActiveX Control

Notice that this way of hosting Windows Forms controls in MFC applications uses yet another common heritage of Windows Forms and MFC: ActiveX® controls. As you can see in the list in Figure 5, System::Windows::Forms::Control implements many interfaces that should be familiar to OLE and ActiveX control veterans. Inside of CreateManagedControl, the Windows Forms control is instantiated via the gcnew operator and in-place activated just like a normal ActiveX control.

Figure 5 Interfaces (Close Figure 5)

System::Windows::Forms::Control
IOleObject
IOleControl
IOleInPlaceObject
IOleInPlaceActiveObject
IOleWindow
IOleViewObject
IOleViewObject2
IPersist
IPersistStreamInit
IPersistPropertyBag
IPersistStorage
IQuickActivate

Also notice that CWinFormsControl overloads the -> operator so that the hosted control is returned. This allows you to initialize the button"s Text property through simple assignement:

m_wfBtn.Text = "Click me!";

This handling is not as clean as modifying a control"s properties in the properties window, but if you can live with the limited designer support that currently exists, the world of Windows Forms controls is open to your trusty old MFC dialog boxes.

Handling Control Events

There is another aspect that should be discussed here. So far the button"s click event is not handled. Since you have a Windows Forms control that is hosted like an ActiveX control, there are two options for event handling: connection points provided by the ActiveX control and event members of the Windows Forms control. The connection point alternative would only be successful if the Windows Forms control was prepared to support COM-based events. For most Windows Forms controls this is not the case, so handling events the .NET way is the only option here. Events of managed classes are a special kind of type member that allows the registration and deregistration of event handler delegates. The following line shows how to register the button"s click event:

m_wfBtn.Click += <an event handler delegate for the click event>

To provide such a delegate, a function with the following syntax is required:

void EventHandler(Object^ sender, EventArgs^ e);

Unfortunately, it is not enough to implement such a method in the CDialog-derived class. Delegate target functions must be member functions of managed classes, and the CDialog derived class is a native one. Visual C++ provides a header file, \Msclr\Event.h, that gives you a generic solution to this problem. The definitions in this header can be used whenever you want to handle a .NET event with a method of a native C++ class. As I will discuss later, you can also use this header when integrating the Visuals features of Windows Presentation Foundation. Since this header is helpful in so many different scenarios, it makes sense to discuss the solution it provides in depth.

To handle the events of the Windows Forms control, you could implement a managed class containing the message handler. C++ interop allows you to do this in an elegant way because this managed class can be nested into the CDialog-derived class:

class CMyDialog : public CDialog {
    ref class nested_managed_class {
        void OnWFBtnClick(Object^ sender, EventArgs^ e) {
            // handle the button"s click event here
        }
    };

    // other members
};

The most convenient way to handle the button"s click event is to forward this method call to a member function of the CDialog-derived class. The method that finally handles the click event can easily access other members of the derived class, just in the way you expect event handlers to work. The managed class containing the delegate target ends up being a simple proxy type. Although such a proxy type and its proxy event handlers can be created with macros defined in Event.h, let"s first have a look at a manual definition (see Figure 6).

Figure 6 CMyDialog (Close Figure 6)
class CMyDialog : public CDialog
{
public:
    ref class delegate_proxy_type;
    gcroot<delegate_proxy_type^> m_gc_managed_native_delegate_proxy;
    delegate_proxy_type^ get_proxy(CMyDialog* pNativeTarget) {
        if((delegate_proxy_type^)m_gc_managed_native_delegate_proxy ==
            nullptr)
        {
            m_gc_managed_native_delegate_proxy =
                gcnew delegate_proxy_type(pNativeTarget);
        }
        return (delegate_proxy_type^)m_gc_managed_native_delegate_proxy;
    }

    ref class delegate_proxy_type {
        CMyDialog* m_p_native_target;
    public:
        delegate_proxy_type(CMyDialog* pNativeTarget) :
            m_p_native_target(pNativeTarget) {}
        void OnWFBtnClick(Object^ sender, EventArgs^ e) {
            this->m_p_native_target->OnWFBtnClick(sender, e);
        }
    };

    void OnWFBtnClick(Object^ sender, EventArgs^ e) {
        this->m_wfBtn->Text = "Thanks for clicking";
    }

    // other members
};

As you can see, the delegate proxy has a constructor that takes a CMyDialog* argument. Such a constructor is necessary to forward method calls to the CDialog-derived class. This delegate proxy type could be used for all event handlers for all Windows Forms controls of the dialog class. It is not possible to add a member variable of the delegate_proxy_type to the derived class, since native classes cannot have handles to garbage-collected objects as fields. Using the gcroot template, you can bypass this restriction here. Therefore, a member variable of type gcroot<delegate_proxy_type> is necessary. However, there is still a remaining question: how could the gcrooted reference to the delegate proxy type be initialized properly? This can be handled by a separate method that is called whenever a handle to delegate_proxy_type is needed.

With the code from Figure 6, a delegate can easily be registered as in the following sample:

virtual BOOL OnInitDialog() {
    CDialog::OnInitDialog();

    this->m_wfBtn->Click +=
        gcnew System::EventHandler(get_proxy(this),
        &delegate_proxy_type::OnWFBtnClick);

    return TRUE;
}

Writing all this code would be a huge amount of work for just handling events of a Windows Forms control in a CDialog-derived class. Event.h contains macros and templates to achieve the same and even more with many fewer lines, as shown in Figure 7.

Figure 7 Simplified CMyDialog (Close Figure 7)
class CMyDialog : public CDialog
{
public:
    CMyDialog(CWnd* pParent = NULL): CDialog(CMyDialog::IDD, pParent) {}

    enum { IDD = IDD_MYDIALOG };

private:
    CWinFormsControl<Button> m_wfBtn;

protected:
    virtual void DoDataExchange(CDataExchange* pDX) {
        DDX_ManagedControl(pDX, IDC_WFBTN, this->m_wfBtn);
    }

public:
    BEGIN_DELEGATE_MAP(CMyDialog)
        EVENT_DELEGATE_ENTRY(OnWFBtnClick, Object^, EventArgs^)
    END_DELEGATE_MAP()

    void OnWFBtnClick(Object^ sender, EventArgs^ e) {
        this->m_wfBtn->Text = "Thanks for clicking";
    }

public:
    virtual BOOL OnInitDialog(){
        CDialog::OnInitDialog();

        this->m_wfBtn->Click += MAKE_DELEGATE(System::EventHandler,
            OnWFBtnClick);
        return TRUE;
    }
};

Every new version of MFC brings at least one new map. As you can see, the current version introduces the delegate map. The macro BEGIN_DELEGATE_MAP defines the delegate_proxy_type that contains the delegate targets. Every EVENT_DELEGATE_ENTRY line adds such a target method to the managed class. END_DELEGATE_MAP just finishes the managed class. Finally, MAKE_DELEGATE instantiates a delegate referring to the target method of delegate_proxy_type. In fact, the macros do even more than what I have just discussed. They are even prepared for scenarios where the managed class fires events to be sent to a native object that no longer exists.

Controls as Dialog Boxes

Another simple yet effective helper is the template CWinFormsDialog. You can use it to show a Windows Forms control as a modal or modeless dialog box. This class is fairly simple. It extends CDialog to provide the standard behavior of MFC dialog boxes. Apart from that, it uses a CWinFormsControl member variable to host the Windows Forms control, initializes the dialog box"s caption to the value of the hosted control"s Text property, sets the dialog"s initial size so that the control perfectly fits into it, and ensures that the size of the hosted control is adjusted whenever the size of the parent window changes.

This one-liner instantiates a temporary dialog box object hosting YourWinFormsDlgControl and shows it as a modal dialog box:

CWinFormsDialog<YourWinFormsDlgControl>().DoModal();

You should avoid this, however, since there"s no way to get the dialog"s return value using DoModal. For more realistic scenarios, consider deriving your own class from CWinFormsDialog<YourWinFormsDlgControl>. This allows you to override OnInitDialog to set properties of the nested Windows Forms control or to handle events of the hosted control. The macros from Event.h that I described earlier can be useful in this case, too. For more information, see msdn2.microsoft.com/ahdd1h97.aspx.

Windows Forms Views

Last, but not least, you can host Windows Forms controls as views. CWinFormsView is the little helper you need for this scenario. Unlike CWinFormsDialog, it is not a template class. Nevertheless, there are significant similarities. Both leave the actual hosting of the control up to the CWinFormsControl template and both handle the resizing of the hosted control.

To host a Windows Forms control as an MFC dialog box, you have to inherit your view class from CWinFormsView. The following code shows the declaration of a simple view class that is based on CWinFormsView:

class CMyWinFormsBasedView : public CWinFormsView
{
    CMyWinFormsBasedView ();
    DECLARE_DYNCREATE(CMyWinFormsBasedView)
};

If your view class was generated by a wizard, you should make sure to switch to CWinFormsView not only in the base class list of your view"s class declaration, but also in its implementation. This may cause you to change parameters of macros like IMPLEMENT_DYNAMIC and BEGIN_MESSAGE_MAP. To instantiate the hosted Windows Forms control, the constructor of CWinFormsView (your view"s base class) gets a handle to the System::Type object of the Windows Forms control. The implementation of the view class just shown can be very simple, too:

IMPLEMENT_DYNCREATE(CMyWinFormsBasedView, CWinFormsView)

CMyWinFormsBasedView:: CMyWinFormsBasedView ()
    : CWinFormsView(WinFormsViewControl::typeid) {}

At the first view, this code looks much too simplistic for a real-life scenario, but in many cases this is really all you need. The view class acts as a simple proxy for the real implementation that resides in the view"s control.

Many view implementations override virtual functions of the MFC CView base class. To keep the native view class acting as a simple proxy, you could override these methods in the native view class and forward the calls to equivalent methods in the view"s Windows Forms control. For the three most important overrides of CView, such forwarding has already been implemented in the base class CWinFormsView. These three methods are OnInitialUpdate, OnUpdate, and OnActivateView. The forwarding is based on a managed interface named Microsoft::VisualC::MFC::IView that is defined in the assembly Mfcmifc80.dll. To override these methods, implement this interface in the view"s Windows Forms control class as shown in Figure 8.

Figure 8 Overriding IView (Close Figure 8)
#using <mfcmifc80.dll>
using Microsoft::VisualC::MFC::IView;

public ref class MyWinFormsViewControl :
    public System::Windows::Forms::UserControl,
    public IView
{
    ...
protected: // IView implementation
    virtual void OnInitialUpdate() = IView::OnInitialUpdate {
        // implementation
    }

    virtual void OnUpdate() = IView::OnUpdate {
        // implementation
    }

    virtual void OnActivateView(bool activate) = IView::OnActivateView {
        // implementation
    }
};

In addition to overriding virtual functions, views can also implement command handlers. Again, it would be possible to implement such a command handler in the native view class and forward to the Windows Forms control, but since this is a common scenario, MFC provides a more convenient solution. The Mfcmifc80.dll assembly contains some more managed helper types for this scenario. To receive command messages from MFC"s command-routing infrastructure, Windows Forms controls must implement the interface Microsoft::VisualC::MFC::ICommandTarget:

interface class ICommandTarget
{
    void Initialize(ICommandSource^ commandSource);
};

When a view that implements ICommandTarget is created, ICommandTarget::Initialize is called on that view and a handle to a new command source object is passed as an argument. This argument is of the type ICommandSource^. The object that the argument refers to is a container for handlers. These handlers should sound familiar to MFC developers: they are command handlers that are called when a command is issued and command UI handlers that can control whether a control"s UI is enabled, checked, or adorned with a radio button, and what text is displayed in the command"s UI.

To register a handler, methods like AddCommandHandler and AddCommandUIHandler can be called on the command source. These methods expect two arguments: an unsigned integer for the command ID, and a delegate that is used to pass the control"s handler function. Since command handlers and command UI handlers have different signatures, two different delegate types are defined in the namespace Microsoft::VisualC::MFC:

delegate void CommandHandler(unsigned int);
delegate void CommandUIHandler(unsigned int,
    Microsoft::VisualC::MFC::ICommandUI^);

Figure 9 shows how to register a command handler for the ID_EDIT_PASTE command.

Figure 9 Registering Command Handlers (Close Figure 9)
public ref class MyWinFormsViewControl :
    public System::Windows::Forms::UserControl,
    public ICommandTarget
{
...
protected: // ICommandTarget implementation
    virtual void RegisterCmdHandlers(ICommandSource^ cmdSrc) =
        ICommandTarget::Initialize
    {
        cmdSrc->AddCommandHandler(ID_EDIT_PASTE,
            gcnew CommandHandler(this, &WinFormsView::OnEditPaste));
        cmdSrc->AddCommandUIHandler(ID_EDIT_PASTE,
            gcnew CommandUIHandler(this,
            &WinFormsView::OnUpdateEditPaste));

        // register further command handlers ...
    }

    void OnEditPaste(unsigned int)
    {
        // implement command handler
    }

    void OnUpdateEditPaste(unsigned int, ICommandUI^ cmdUI)
    {
        cmdUI->Enabled = ...; // your logic here
    }
};

When a view is supposed to handle a command, the delegate is used to call a member function of the view. The command"s ID is passed as an argument. If every command has its own private handler function, this argument is not important, but a single method can also be used as a handler for more than one command. AddCommand[UI]RangeHandler allows you to register one delegate for multiple commands.

The delegate for command UI handlers has an additional argument of type ICommandUI^. The handle that is passed using this parameter is a wrapper for the MFC CCmdUI class that allows you to control the appearance and usability of interface elements like menu items and toolbar buttons.

In addition to adding command handlers in ICommandSource::Initialize, it can also be useful to store the ICommandTarget handle in a member variable of the Windows Forms control. This gives you the option to add or remove further command handlers later, and you can even fire command events either synchronously or asynchronously through ICommandTarget::SendMessage or ICommandTarget::PostMessage.

Bridges to Avalon

Since Windows Presentation Foundation has not yet been released in final form, the current version of MFC does not have helper classes and templates for integrating its Visuals in CWnds and CDialogs. However, this does not mean that integrating Visuals in MFC applications is not possible. In fact, even without these little helper classes, it only takes a few lines of code to host a Visual in a CWnd or a CDialog.

The key feature for this integration comes from Windows Presentation Foundation itself. The assembly Windows.PresentationCore.dll provides a namespace System::Windows::Interop with very powerful class named HwndSource. This class provides the bridge between an HWND and a Visual. To the hosting USER32 window object, it looks like a child window with a normal HWND. Using its RootVisual property, you can switch into the new world of Windows Presentation Foundation. (Note that the information here is based on a prerelease version of Windows Presentation Foundation and may change with the final release.)

The HwndSource constructor expects a HwndSourceParameters value type as an argument. As you can see in Figure 10, this value type contains similar information that you would pass as arguments in a call to the Win32 function CreateWindowEx. Figure 11 shows how to use HwndSource in an OnInitDialog implementation.

Figure 11 Using HwndSource (Close Figure 11)
BOOL CWPFDemoDialog::OnInitDialog()
{
    __super::OnInitDialog();

    CRect rectClient;
    this->GetClientRect(&rectClient);

    System::Windows::Interop::HwndSourceParameters hwsPars;
    hwsPars.ParentWindow = System::IntPtr(this->m_hWnd);
    hwsPars.WindowStyle = WS_CHILD | WS_VISIBLE;
    hwsPars.PositionX = 0;
    hwsPars.PositionY = 0;
    hwsPars.Width = rectClient.Width();
    hwsPars.Height = rectClient.Height();
    System::Windows::Interop::HwndSource^ hws;
    hws = gcnew System::Windows::Interop::HwndSource(hwsPars);

    using System::Windows::Controls::MonthCalendar;
    MonthCalendar^ mc = gcnew MonthCalendar;
    hws->RootVisual = mc;

    return TRUE;
}
Figure 10 CreateWindowEx and HwndSourceParameters (Close Figure 10)

CreateWindowEx Arguments | HwndSourceParameters Properties
LPCTSTR lpClassName | N/A
N/A | WindowClassStyle
LPCTSTR lpWindowName | WindowName
DWORD dwStyle | WindowStyle
DWORD dwExStyle | ExtendedWindowStyle
int x | PositionX
int y | PositionY
int nWidth | Width
int nHeight | Height
HWND hWndParent | ParentWindow
HMENU hMenu | N/A
HINSTANCE hInstance | N/A

Conclusion

Using C++ interop, managed code can be seamlessly integrated into existing C++ sources. The MFC support for Windows Forms can be seen as strong evidence that this integration is an intended and supported feature of Visual C++. To simplify this integration, several helper classes and templates exist. This integration layer is neither complex nor difficult to use.

One thing that is missing is a seamless integration of these features into the wizards of Visual Studio. In most scenarios, this is not a big problem since much of the implementation will be left to the hosted Windows Forms control, which has great designer support in Visual Studio. When Windows Presentation Foundation ships, there"s a high probability that equivalent support for integrating Visuals into MFC applications will be provided. Even without such support, though, it is easy to host Visuals in MFC apps.

Marcus Heege works as a course author and trainer for DevelopMentor and provides consulting for different IT companies. He regularly posts his thoughts about C++/CLI and .NET in his blog at www.heege.net.


Bug Bash: Let The CLR Find Bugs For You With Managed Debugging Assistants

Bug Bash

Let The CLR Find Bugs For You With Managed Debugging Assistants

Stephen Toub
This article discusses:
  • What MDAs are and their usefulness in debugging
  • Common scenarios in which MDAs are beneficial
  • Enabling MDAs for managed and unmanaged debugging
This article uses the following technologies:
.NET Framework, Visual Studio 2005
Contents
PInvokeStackImbalance
LoadFromContext
CallbackOnCollectedDelegate
GcUnmanagedToManaged
InvalidFunctionPointerInDelegate
AsynchronousThreadAbort
StreamWriterBufferedDataLost
PInvokeLog
Enabling MDAs
Conclusion

A
sserts are a very valuable developer tool that allow you to test for specific conditions at run time. Often, you"re testing for situations that should never occur ("if this happens, something is wrong"), and sometimes you"re looking to verify that particular conditions do occur ("if this doesn"t happen, something is wrong"). Since these asserts aren"t meant for users" eyes, typically they"re only compiled into debug builds, or a log file is generated that maintains information about a series of such problems.

That"s all well and good for application developers, but it"s not great for developers of libraries. Developers of apps that consume libraries can benefit from the libraries providing these kinds of alerts, and yet frequently such functionality is intrusive. Instead of asserts, library vendors typically opt to throw exceptions which can bubble up to the application level to be dealt with. But exceptions aren"t appropriate substitutes for asserts in all cases.

For example, consider how the common language runtime (CLR) loads assemblies. There are two "contexts" into which an assembly can be loaded: the Load context and the LoadFrom context. Use of the LoadFrom context is frequently the cause of bugs and unexpected behavior related to serialization, casting, and assembly dependency resolution. You wouldn"t want an exception to be thrown any time an assembly is loaded into the LoadFrom context, but you might like to receive some sort of debug-time notification that the condition is occurring, and you"d like to be able to turn such notifications on and off easily. That would allow you to look into the situation and determine whether it"s a legitimate usage or an unintended side effect of some code on which you"re dependent. Enter Managed Debugging Assistants (MDA).

MDAs are asserts (probes) in the CLR and in the base class libraries (BCL) that can be turned on or off. When enabled, they provide information on the CLR"s current runtime state and on events that you as a developer could not otherwise access. Some even modify runtime behaviors to help expose otherwise very hard to find bugs. The Microsoft® .NET Framework 2.0 includes 42 of these MDAs, some more useful than others, but all of them very powerful debugging aids that can help you track down truly insidious problems at run time.

The MDAs currently available fall into three categories, as shown in the Type column in Figure 1. Informational MDAs provide data about the current state of the CLR and are not enabled by default. Behavioral MDAs are also not enabled by default. But rather than simply providing information to the developer, these MDAs modify the behavior of the CLR in an attempt to highlight particular bugs in the developer"s code. Detection MDAs directly identify some type of error condition that is occuring. Depending on the environment, some detection MDAs are easily enabled. As shown in Figure 2, the detection MDAs in the Managed Debugger category are available for selection from the Exceptions dialog in Visual Studio® 2005. A subset of the Managed Debugger category is checked by default in the Exceptions dialog and as a result will cause Visual Studio 2005 to halt debugging with an Exception Assistant dialog when encountered. When running under an unmanaged debugger, only two detection MDAs are enabled by default.

Figure 1 MDAs in the .NET Framework 2.0 (Close Figure 1)

 | Type | Managed Debugger | Unmanaged Debugger
Unmanaged Interop |  |  |
Reentrancy | Detection | (on by default) |
GcManagedToUnmanaged | Behavioral |  |
GcUnmanagedToManaged | Behavioral |  |
Unmanaged Interop - COM |  |  |
ContextSwitchDeadlock | Detection | (on by default) |
DisconnectedContext | Detection | (on by default) |
EarlyBoundCallOnAutoDispatchClassInterface | Detection |  |
ExceptionSwallowedOnCallFromCom | Detection |  |
FailedQI | Detection |  |
InvalidApartmentStateChange | Detection |  |
InvalidIUnknown | Detection |  |
InvalidMemberDeclaration | Detection | (on by default) |
InvalidVariant | Detection | (on by default) |
MarshalCleanupError | Detection |  |
Marshaling | Informational |  |
NonComVisibleBaseClass | Detection | (on by default) |
NotMarshalable | Detection |  |
RaceOnRCWCleanup | Detection | (on by default) |
ReportAvOnComRelease | Detection |  |
Unmanaged Interop - P/Invoke |  |  |
CallbackOnCollectedDelegate | Detection | (on by default) |
InvalidFunctionPointerInDelegate | Detection | (on by default) |
InvalidOverlappedToPinvoke | Detection |  |
OverlappedFreeError | Detection |  |
PInvokeLog | Informational |  |
PInvokeStackImbalance | Detection | (on by default) |
Loader |  |  |
BindingFailure | Detection |  |
DllMainReturnsFalse | Detection |  |
LoaderLock | Detection | (on by default) |
LoadFromContext | Detection |  |
Constrained Execution Regions |  |  |
IllegalPrepareConstrainedRegion | Detection |  |
InvalidCERCall | Detection |  |
OpenGenericCERCall | Informational |  |
VirtualCERCall | Informational |  |
Threading |  |  |
AsynchronousThreadAbort | Detection |  |
DangerousThreadingAPI | Detection |  |
Base Class Libraries |  |  |
DateTimeInvalidLocalFormat | Detection | (on by default) |
MemberInfoCacheCreation | Informational |  |
ModuloObjectHashcode | Behavioral |  |
ReleaseHandleFailed | Detection |  |
StreamWriterBufferedDataLost | Detection |  |
Miscellaneous |  |  |
FatalExecutionEngineError | Detection | (on by default) |
InvalidGCHandleCookie | Detection |  |
JitCompilationStart | Informational |  |
Figure 2 MDAs in the .NET Framework 2.0
Figure 2 MDAs in the .NET Framework 2.0

You might find certain MDAs more useful than others. My opinion is that you should start your development with all of the MDAs in the Managed Debugger category turned on in the Exceptions dialog, only disabling ones when you actively determine that they"re getting in the way. Especially when doing any sort of interop work, these MDAs can help bring to light problems that might otherwise go unnoticed for quite some time. In fact, as Mike Stall points out in his blog, the inclusion of MDAs in the .NET Framework 2.0 was, in a very real sense, a self-serving act by members of the .NET Framework team. Many problems can be caught by MDAs that would otherwise manifest as apparent bugs with the Framework, even though the symptom may be just one in a series of dominoes knocked over by something wrong much earlier in the developer"s code.

I do have a few favorite MDAs, which I"ll describe next.

PInvokeStackImbalance

If you do any P/Invoke work, this MDA is a must. It"s activated when the CLR detects that the stack depth after a P/Invoke call does not match the expected stack depth. Consider a simple method exported from kernel32.dll for which you might write an interop signature:

BOOL Beep(DWORD dwFreq, DWORD dwDuration);

A valid P/Invoke declaration could look like this:

[DllImport("kernel32.dll", SetLastError=true)]
[return: MarshalAs(UnmanagedType.Bool)]
internal static extern bool Beep(int frequency, int duration);

But what if you accidentally, due to haste or to a misunderstanding of how native and managed types relate, declare it as follows:

[DllImport("kernel32.dll", SetLastError=true)]
[return: MarshalAs(UnmanagedType.Bool)]
internal static extern bool Beep(long frequency, long duration);

By mistaking the parameters as longs instead of as ints, the caller will be passing into the target method 8 bytes more data than it actually expects (two 64-bit values instead of two 32-bit values). The Beep function utilizes the StdCall calling convention, which means that the callee is responsible for removing the parameters from the stack. Thus, the callee will remove 8 bytes from the stack (for the two 32-bit values it expected), leaving the stack unbalanced on return (the additional 8 bytes erroneously supplied). When this function returns to the CLR, the PInvokeStackImbalance MDA will be raised, resulting in a dialog like that shown in Figure 3.

Figure 3 PInvokeStackImbalance MDA Activation
Figure 3 PInvokeStackImbalance MDA Activation

Of course, the world of PInvokeStackImbalance isn"t entirely sunny. The just-in-time compiler (JIT) in the CLR implements several approaches to invoking native functions through P/Invoke, ranging from a very fast inlining technique down to a slow interpretation technique. The JIT chooses which technique to use based on a variety of factors, including the complexity of the P/Invoke declaration. Unfortunately, when the PInvokeStackImbalance MDA is enabled, P/Invoke calls are forced to use the slower techniques. In typical C# and Visual Basic®-based apps, P/Invoke is used relatively infrequently, so the performance hit is outweighed by the value provided by the MDA. However, applications that do a significant amount of interop (such as those written in C++ to take advantage of It Just Works interop) could experience noticeable performance degradation when run under the debugger with this MDA enabled. See Mike Stalls blog.

LoadFromContext

The scenario I described earlier for tracking assemblies loaded into the LoadFrom context was not hypothetical. The LoadFromContext MDA will alert you to any time any assembly is loaded into the LoadFrom context, making it possible to more easily identify bugs that manifest as invalid cast exceptions, deserialization failures, and the like. If at all possible, it"s best to install assemblies in the Global Assembly Cache or in the ApplicationBase directory, which then allows you to use Assembly.Load when explicitly loading assemblies.

CallbackOnCollectedDelegate

Frequently, interop work requires Windows® calling back into managed code, a process known as reverse P/Invoke. This is typically accomplished by passing a delegate to a P/Invoke method. Under the covers, the CLR creates a native function pointer to a thunk that, when invoked, in turn invokes the delegate. In the unmanaged world, that function pointer is a constant, and the unmanaged code that uses it rightfully expects the target to always exist. However, when the delegate is no longer needed by managed code, the garbage collector is free to reclaim it (reclaiming the delegate causes the associated thunk to be freed as well), which will result in the function pointer in unmanaged code becoming invalid. When the unmanaged code then attempts to call this invalid function pointer, access violations will typically ensue. Not good.

To prevent this, the delegate must be kept alive for as long as the unmanaged code might need to call it. Some of this work is done for you. For example, take the EnumWindows function exposed from kernel32.dll. EnumWindows accepts a callback function as a parameter, and that function will be called for each top-level window found. All of these callbacks happen during the invocation of EnumWindows. It is for situations like this that the CLR ensures that a delegate passed to a P/Invoke method will not be collected during the invocation of that method. But what if the unmanaged code caches away a copy of the function pointer supplied to it? At a later point in time, after the managed P/Invoke call finishes, the unmanaged code might attempt to call back on that function pointer, at which point the delegate might not be alive.

For a concrete example of this, consider the (buggy) code in Figure 4. Using a low-level keyboard windows hook, the code intercepts all WM_KEYDOWN messages sent to any window and prints out the corresponding key. When you run this app, it"ll most likely appear to work, at least for a little while. But at some point, the garbage collector will run and will see that the LowLevelKeyboardProc delegate instance passed to the SetWindowsHookEx P/Invoke method is no longer referenced anywhere, and the GC will reclaim it (if you don"t feel like waiting, insert a call to GC.Collect just before the call to Application.Run to force the delegate"s immediate removal). After that point, the next time Windows attempts to use this hook, the app will almost certainly die a horrible death, and it might not be at all apparent why.

Figure 4 Buggy Code for Setting a Low-Level Keyboard Hook (Close Figure 4)
using System;
using System.Diagnostics;
using System.Windows.Forms;
using System.Runtime.InteropServices;

public class InterceptKeys
{
    private const int WH_KEYBOARD_LL = 13;
    private const int WM_KEYDOWN = 0x0100;
    private static IntPtr _hookID = IntPtr.Zero;

    public static void Main()
    {
       _hookID = SetHook();
       Application.Run();
       UnhookWindowsHookEx(_hookID);
    }

    private static IntPtr SetHook()
    {
        using (Process curProcess = Process.GetCurrentProcess())
        using (ProcessModule curModule = curProcess.MainModule)
        {
            return SetWindowsHookEx(WH_KEYBOARD_LL,
                new LowLevelKeyboardProc(HookCallback), // BUG!!!
                GetModuleHandle(curModule.ModuleName), 0);
        }
    }

    private delegate IntPtr LowLevelKeyboardProc(
        int nCode, IntPtr wParam, IntPtr lParam);

    private static IntPtr HookCallback(
        int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode >= 0 && wParam == (IntPtr)WM_KEYDOWN)
        {
            int vkCode = Marshal.ReadInt32(lParam);
            Console.WriteLine((Keys)vkCode);
        }
        return CallNextHookEx(_hookID, nCode, wParam, lParam);
    }

    [DllImport("user32.dll", CharSet=CharSet.Auto, SetLastError = true)]
    private static extern IntPtr SetWindowsHookEx(int idHook,
        LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

    [DllImport("user32.dll", CharSet=CharSet.Auto, SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static extern bool UnhookWindowsHookEx(IntPtr hhk);

    [DllImport("user32.dll", CharSet=CharSet.Auto, SetLastError = true)]
    private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode,
        IntPtr wParam, IntPtr lParam);

    [DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError = true)]
    private static extern IntPtr GetModuleHandle(string lpModuleName);
}

CallbackOnCollectedDelegate to the rescue! When this MDA is enabled, thunks are not deleted when the delegate is garbage collected. Instead, the thunk is modified to activate the CallbackOnCollectedDelegate MDA so that you know exactly why your process is about to die (see Figure 5). Note that the CLR only keeps around a certain number of these modified thunks, and that number is configurable using an MDA configuration file (more on these shortly), though it"s rare you"d ever need or want to change this number.

Figure 5 PInvokeStackImbalance MDA Activation
Figure 5 PInvokeStackImbalance MDA Activation

After CallbackOnCollectedDelegate helps identify the problem, you need to fix it. Typical solutions are to use GC.KeepAlive to ensure a particular instance is kept alive until a certain time, or to make it a member variable of a class that itself will stay alive for a long enough period of time.

GcUnmanagedToManaged

In my discussion of CallbackOnCollectedDelegate, I mentioned that you could easily force the error by calling GC.Collect after setting up the hook (by this point all references to the delegate were gone). That"s certainly one solution, but it also requires modifications to your code, and one of the best benefits of MDAs is debugging help and error detection without code modification. This is where the GcUnmanagedToManaged MDA comes into play. Falling into the behavioral category, this MDA does not produce any output; instead, it causes the GC to perform a collection any time a thread transitions from unmanaged code to managed code. Thus, with this MDA enabled in addition to CallbackOnCollectedDelegate, any time the unmanaged code attempts to invoke the delegate, garbage collection will first be performed. This can come at a significant performance cost if there are lots of callbacks (resulting in lots of additional collections), but it also can significantly decrease the time between when the corruption happens (in this case, dropping all references to the delegate) and when the CallbackOnCollectedDelegate MDA alerts you to a problem. All that, with no modifications to your code, is a win in my book.

InvalidFunctionPointerInDelegate

A frequently asked for feature in the .NET Framework 1.x was the ability to make P/Invoke calls to dynamic targets, because the name, signature, or location of the target function was not known at compile time. This functionality is enabled in the .NET Framework 2.0 through a new method on the Marshal class, GetDelegateForFunctionPointer. It accepts two parameters: an IntPtr address of the target function and the Type of the delegate to instantiate around that function (the delegate"s signature should be compatible with the target function).

What happens if you pass a bad function pointer to Marshal.GetDelegateForFunctionPointer? It"ll happily create the delegate for you, but when you invoke that delegate, you could very well find yourself starting at an access violation, or worse. To help you find such problems, the InvalidFunctionPointerInDelegate MDA alerts you when an invalid function pointer is passed to GetDelegateForFunctionPointer. Consider the following code:

EventHandler ev = Marshal.GetDelegateForFunctionPointer(
    (IntPtr)0x12345678, typeof(EventHandler));
ev(null, EventArgs.Empty);

When run, an Exception Assistant alerts you at the first line with the following message:

Invalid function pointer 0x12345678 was passed into
the runtime to be converted to a delegate.
Passing in invalid function pointers to be converted
to delegates can cause crashes, corruption or data loss.

To top it all off, as shown in Figure 1, this MDA is enabled by default in Visual Studio, so you don"t even have to turn it on to take advantage of it. Automatic protection.

AsynchronousThreadAbort

Regular readers of the .NET Matters column may remember several previous columns in which I demonstrated how to use the Thread.Abort method. In one edition it was used to implement method timeouts (See ".NET Matters: StringStream, Methods with Timeouts"), while in another it helped to implement a wrapper for the ThreadPool that supports cancellation (See ".NET Matters: Abortable Thread Pool"). In both of those columns I elaborate on the dangers of using this technique, and in my article on reliability in the .NET Framework (See "High Availability: Keep Your Code Running with the Reliability Features of the .NET Framework"), I take a detailed look at the problems it can cause and on a variety of ways to improve the reliability of your managed applications.

In short, unless Thread.Abort is being used to abort the current thread (Thread.CurrentThread.Abort), avoid it whenever possible. To help alert you to situations in which your code is making use of Thread.Abort, the CLR provides the AsynchronousThreadAbort MDA. Like the LoadFromContext MDA, the AsynchronousThreadAbort MDA does not definitively indicate you have a problem in your code, but it does help you root out the causes of problems you might be experiencing by letting you know that you"re using something dangerous. Consider the following simple program:

class Program {
    public static void Main(string[] args) {
        Thread t = new Thread(delegate() {
            while (true) { Thread.Sleep(500); }
        });
        t.Start();
        t.Abort();
        Console.WriteLine("Done");
    }
}

This code spins up a new thread and immediately aborts it. If the AsynchronousThreadAbort MDA is enabled (a simple task, as it"s part of the Managed Debugger category and thus can be enabled from the Exceptions dialog in Visual Studio), Visual Studio will break at the line that aborts the thread, providing an Exception Assistant with the following text (though almost certainly with different thread IDs):

User code running on thread 4408 has attempted to abort thread 4956.
This may result in corrupt state or resource leaks if the thread being
aborted was in the middle of an operation that modifies global state
or uses native resources. Aborting threads other than the currently running
thread is strongly discouraged.

The MDA tells you exactly which thread was being interrupted and which thread was doing the interrupting, a very helpful feature when you, through the debugger, have access to all threads involved in the operation.

StreamWriterBufferedDataLost

Another gem of an MDA, StreamWriterBufferedDataLost alerts you to data loss that can result by not properly closing a StreamWriter. File I/O is expensive, so to minimize the number of I/O operations, StreamWriter buffers its data. When you call StreamWriter.Write, the data you pass may or may not be written immediately to the underlying file, depending on how much data you"re writing and how much you"ve already written. At some point, StreamWriter decides it"s received enough data and that it should be written to the file, at which time it does the actual I/O. Until then, it stores the data to be written in an internal buffer. If you forget to close the StreamWriter, the StreamWriter might contain some unflushed data in its buffers, and that data will never make it into the output file. If the StreamWriterBufferedDataLost MDA is enabled, the StreamWriter"s finalizer will activate the MDA if it detects unflushed data in its buffer, at which point the Exception Assistant will display information like that shown in Figure 6. Notice that the MDA not only tells you that the situation occurred, but it also gives you a full stack trace for where the StreamWriter whose data was abandoned was instantiated. How cool is that?

Figure 6 Message from StreamWriterBufferedDataLost MDA (Close Figure 6)
A StreamWriter was not closed and all buffered data
within that StreamWriter was not flushed to the underlying stream.
(This was detected when the StreamWriter was finalized with data
in its buffer.)  A portion of the data was lost.  Consider one of
calling Close(), Flush(), setting the StreamWriter’s AutoFlush property
to true, or allocating the StreamWriter with a "using" statement.  Stream type: System.IO.FileStream
File name: C:\test.txt
Allocated from:
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.IO.StreamWriter.Init(Stream stream, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamWriter..ctor(String path)
   at ConsoleApplication10.Program.Main(String[] args)
   at System.AppDomain.nExecuteAssembly(Assembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext,
   ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

PInvokeLog

Ever curious about what P/Invoke calls are being made by your application? When enabled, the PInvokeLog MDA fires when each P/Invoke declaration is invoked for the first time, allowing you to see exactly which unmanaged function is executed and which managed P/Invoke signature caused that to happen. Unfortunately, not all of that information is accessible to you in Visual Studio. Up until this point, I"ve glossed over how MDAs actually report information to a debugger. For the .NET Framework 2.0, the ICorDebug interfaces have been extended to support MDAs, specifically through ICorDebugManagedCallback2 and its MDANotification method. MDANotification provides the listener with an ICorDebugMDA instance, providing valuable information about the raised MDA, including an XML description. For CallbackOnCollectedDelegate, that XML output will look something like this:

<mda:msg xmlns:mda="http://schemas.microsoft.com/CLR/2004/10/mda">
  <!--
     A callback was made on a garbage collected delegate of type
     "test!InterceptKeys+LowLevelKeyboardProc::Invoke". This may
     cause application crashes, corruption and data loss. When
     passing delegates to unmanaged code, they must be kept alive
     by the managed application until it is guaranteed that
     they will never be called.
   -->
  <mda:callbackOnCollectedDelegateMsg break="true">
    <delegate name="test!InterceptKeys+LowLevelKeyboardProc::Invoke"/>
  </mda:callbackOnCollectedDelegateMsg>
</mda:msg>

If you compare this output with that from Figure 5, you"ll see that Visual Studio is displaying in the Exception Assistant the inner text comment from the MDA"s output. For PInvokeLog, the XML output from the MDA looks something like this:

<mda:msg xmlns:mda="http://schemas.microsoft.com/CLR/2004/10/mda">
  <mda:pInvokeLogMsg>
    <method name="test!InterceptKeys::CallNextHookEx"/>
    <dllImport dllName="C:\WINDOWS\system32\USER32.dll"
        entryPoint="CallNextHookEx"/>
  </mda:pInvokeLogMsg>
</mda:msg>

PInvokeLog will typically fire many times for even the simplest of .NET applications, and this sample is the output generated when the InterceptKeys.CallNextHookEx P/Invoke from Figure 4 was called. Unfortunately, Visual Studio doesn"t display the entire XML output, only the inner text as shown in the previous example. If you want all of this information, you"ll need to use a debugger that displays the full XML, such as Windbg. Of course, Windbg doesn"t have an Exceptions dialog as does Visual Studio, so you"ll need to enable the PInvokeLog MDA in a slightly different fashion.

Enabling MDAs

There are several ways to enable MDAs. If the MDA is in the Managed Debugger category (see Figure 1) and if you"re using Visual Studio, the Exceptions dialog is the way to go. However, this only covers a subset of the cases. MDAs can be enabled in two additional ways: through the registry and through an environment variable, both of which can be done in combination with a configuration file. Enabling MDAs through the registry involves setting the MDA value on the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework key to 1, as you can do with a .reg file containing the following:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework]
"MDA"="1"

This tells the CLR to look for an MDA configuration file in the same directory as the application with the name AppName.exe.mda.config, where AppName.exe is the name of the application being debugged. The XML contents of this file lists all of the MDAs you want enabled for this application. So, for example, a configuration file that enabled the eight MDAs I"ve described up to this point would look like the following (note that the XML is case-sensitive with respect to the names of the MDAs, and the MDAs must be listed in alphabetical order).

<?xml version="1.0" encoding="UTF-8" ?>
<mdaConfig>
  <assistants>
    <asynchronousThreadAbort />
    <callbackOnCollectedDelegate />
    <gcUnmanagedToManaged />
    <invalidFunctionPointerInDelegate />
    <loadFromContext />
    <pInvokeLog />
    <pInvokeStackImbalance />
    <streamWriterBufferedDataLost />
  </assistants>
</mdaConfig>

Some MDAs take additional parameters (you can find these parameters documented in the MSDN library), and in fact some must be configured in order to be worth anything. As an example of an MDA that can be configured but that doesn"t have to be, consider a configuration entry for the CallbackOnCollectedDelegate MDA that was discussed earlier:

    <mdaConfig>
      <assistants>
        <callbackOnCollectedDelegate listSize="1500" />
      </assistants>
    </mdaConfig>

The listSize attribute on the callbackOnCollectedDelegate element tells the CLR how many abandoned interop thunks to retain for MDA purposes, in this case 1500 (the value must be at least 50 and no more than 2000; the default is 1000). PInvokeLog, on the other hand, must be configured with additional information in order to be at all useful. PInvokeLog needs to be told which target DLLs should have output provided:

<mdaConfig>
  <assistants>
    <pInvokeLog>
      <filter>
        <match dllName="user32.dll"/>
        <match dllName="kernel32.dll"/>
      </filter>
    </pInvokeLog>
  </assistants>
</mdaConfig>

Enabling MDAs in the registry is global and will affect all managed applications you run. Instead, for many situations you"ll probably find it easier to enable MDAs using an environment variable. Like the registry key, the COMPLUS_MDA environment variable can be set to 1 to force executed applications to use an associated configuration file:

set COMPLUS_MDA=1

However, unless you need to configure specific MDAs with specific parameters, you can avoid a configuration file and instead set the environment variable to a semicolon-delimited list of the MDAs you want enabled. For example, the following command will enable just the GCUnmanagedToManaged and CallbackOnCollectedDelegate MDAs:

set COMPLUS_MDA=GCUnmanagedToManaged;CallbackOnCollectedDelegate

After setting this variable, from the command line you can compile and execute the code in Figure 4:

    csc /debug /o- BuggyCode.cs
    BuggyCode.exe

As soon as BuggyCode.exe intercepts your next key press, you"re presented with the standard debugger attach dialog (see Figure 7), stemming from the CallbackOnCollectedDelegate MDA"s invocation.

Figure 3 Just-In-Time Debugging for an MDA
Figure 3 Just-In-Time Debugging for an MDA

You can now use Windbg to view the output from PInvokeLog. Again, using the example in Figure 4 along with the PInvokeLog configuration file I showed earlier (which should be named BuggyCode.exe.mda.config and be in the same directory as BuggyCode.exe), you can execute it as follows:

    set COMPLUS_MDA=1
    Windbg BuggyCode.exe

As soon as Windbg loads and you continue execution, you"ll see the output shown in Figure 8, containing a plethora of XML snippets from the PInvokeLog MDA.

Figure 8 Using WinDbg to View PInvokeLog Output
Figure 8 Using WinDbg to View PInvokeLog Output

If you want to use Visual Studio with MDAs like PInvokeLog, you can, but it requires a bit of extra work, and while you"ll be notified when the MDA fires, you won"t get any of the additional information provided about the MDA (such as the DLL import information in the case of PInvokeLog). First go to the Exceptions dialog in Visual Studio and add an entry under Managed Debugging Assistants with the name of the MDA in which you"re interested. Make sure you"ve enabled MDAs in the registry and that you"ve properly created a configuration file that includes the MDA in which you"re interested. Additionally, some MDAs (like PInvokeLog) will only fire in Visual Studio when mixed-mode debugging is enabled. After taking all of those steps and executing my program, every time the PInvokeLog MDA fires, you"ll see a dialog like that in Figure 9.

Figure 9 PInvokeLog MDA in Visual Studio
Figure 9 PInvokeLog MDA in Visual Studio

Conclusion

A common question regarding MDAs is whether you as a developer can add your own. Unfortunately, no, you can"t, at least not with the .NET Framework 2.0. It"s possible future versions of the Framework will allow for this capability, but for now, it"s great to have the level of support offered thus far. Give MDAs a look; I think you"ll be happy with what you see.

Stephen Toub is the Technical Editor for MSDN Magazine.


Analyze This: Find New Meaning In Your Ink With Tablet PC APIs In Windows Vista

Analyze This

Find New Meaning In Your Ink With Tablet PC APIs In Windows Vista

Markus Egger
This article is based on a prerelease version of Windows Vista. All information herein is subject to change.
This article discusses:
  • Why ink analysis is needed
  • Implementing basic analysis with Windows Forms
  • Using analysis hints
  • Recognizing drawing elements
This article uses the following technologies:
Windows Vista, Windows XP Tablet PC
Code download available at: InkAnalysis05.exe (1,303KB)
Contents
Basic Ink Analysis
Paragraphs, Lines, and Words
Analysis Hints
Drawing Recognition
The Bigger Picture
Other Technical Aspects

W
ith Windows XP Tablet PC Edition, Microsoft introduced more than just a powerful platform for users. Through the associated SDKs, Microsoft has also empowered developers to create advanced ink-enabled applications. Recognizer objects provide the means to recognize handwriting as text. Similarly, gestures can be recognized and interpreted in any way the developer desires. Other objects allow for the division of digital ink into paragraphs, lines, and segments. The combination of these objects allow for the creation of very advanced applications. However, these options do not cover the complete range of features needed to create next-generation ink-enabled applications.

So what is missing? In short, the ability to add meaning to ink beyond the simple recognition of text. Users may add drawings, for instance. With ink recognizers, even a simple line is considered to be text and is thus recognized with very poor results. You need the ability to identify whether a part of a document is text or a drawing before recognition is applied.

Also, once a drawing element such as a line is identified, its meaning must also be interpreted. It could be meant as a true drawing, but it could be a line connecting an annotation with another part of a document, or it could be an underlined word. This sort of meaning can only be derived by using contextual analysis that looks at large sections of ink rather than just individual strokes that make up characters or words. And even if the line is ultimately identified as a true drawing, how can the system recognize drawing primitives such as circles, squares, or triangles?

The first-generation Ink SDK features do not make it easy to handle any of these scenarios. Furthermore, spatial analysis functionality is missing. Ink Divider objects can break digital ink into various segments, but logical context is limited. Segments are not the same as words, for instance. This makes it difficult to handle scenarios such as reflowing ink, which is necessary when ink-space gets resized. If a window containing ink changes its size, individual words of ink may need to move onto the next line and the rest of the ink has to move accordingly. Without the means to recognize concepts such as lines and words reliably, it is difficult to implement ink reflowing. Also, as ink moves, logically associated ink such as an underline or an annotation needs to move as well. To implement such a scenario successfully, relationships between different ink segments need to be recognized.

Another aspect of spatial analysis is the ability to assign names to certain sections of ink. In digital equivalents of paper forms, for instance, you would find different areas such as a space for a name or address.

The new InkAnalysis API (available in Windows Vista™ as well as downlevel to Windows XP through a redistributable) provides the means to handle all these scenarios with ease. This API is available as part of the new Windows® SDK, which contains the new WinFX® components (see msdn.microsoft.com/windowsvista/getthebeta). Note that the WinFX SDK has a dependency on the .NET Framework 2.0.

The InkAnalysis API supplements existing features that are still available. However, the API also supersedes the existing objects, especially the recognizer objects (specifically the Recognizer and RecognizerContext objects) and the ink Divider object. In essence, the InkAnalysis API combines the two different but related technologies that are required for handling digital ink: recognition and layout analysis and classification.

Ink recognition, often called handwriting recognition, is the task of analyzing strokes of ink in order to turn them into a text-based version of the ink that can be handled as standard strings. This is done based on specific language and culture assumptions for the ink that is to be recognized. In other words, ink recognition is the computer reading someone’s handwriting. Ink recognition can also be applied in a slightly different way. For instance, ink recognition could analyze sheet music for the purpose of turning it into some sort of standard data format, such as MIDI. Or ink recognition could be used to read Egyptian hieroglyphs. With the new InkAnalysis API, you can add drawing primitives and basic shapes to the list.

Ink layout analysis and classification, on the other hand, deals with the overall layout and spatial distribution of ink. Microsoft defines two areas of ink parsing: classification and layout analysis. Ink classification concerns itself with finding semantically meaningful parts such as paragraphs, words, or drawings. In many cases it is helpful to find such semantics before applying handwriting recognition. For instance, it makes no sense to apply handwriting recognition to drawings. The InkAnalysis API provides the means to do so using ink classification, a feature that was not available previously. Furthermore, the InkAnalysis API allows you to break ink into a hierarchical tree, which then directly provides the means to perform recognition on individual tree branches.

Layout analysis refers to the computational analysis of ink strokes and their positions relative to each other with the intent to find spatial and semantic meaning. This goes beyond the relatively simple task of finding lines and paragraphs. In simpler terms, layout analysis can find things such as annotations, bulleted lists, flow charts, and much more.

Basic Ink Analysis

To create a first ink analysis example, let’s start with a basic ink-enabled Windows Forms application just as if you were to create a conventional ink recognition example. If you are not familiar with ink development, here is the short version of how to create such a form.

Using the Microsoft® Tablet PC SDK, any form or control can be ink enabled. To do so, create a standard Windows Forms app and add an InkOverlay object. This object is part of the Microsoft.Ink namespace, exposed from Microsoft.Ink.dll, which needs to be added to the project references. (It is listed as Microsoft Tablet PC API in the Add References dialog box.) You can then link this object with any control to be ink-enabled by associating the ink overlay with the control’s window handle. When the InkOverlay’s Enabled property is set to true, ink collection and ink rendering are enabled on the control, including the ability to add, select, erase, and otherwise manipulate ink. The digital ink information is stored in an Ink object, accessible from the InkOverlay, where it can be accessed for recognition, programmatic manipulation, or storage.

To provide a good ink experience, it is important to create an inkable area of sufficient physical dimension. A good choice is a Panel control, perhaps with white background, that covers the majority of a window. Of course it is also possible to enable an entire window (Form) for inking, but in most scenarios other controls are needed in the interface, such as buttons. Therefore, the Panel approach is usually preferable.

To follow this Panel approach, create a new Form in the .NET-targeted language of your choice, drop in a Panel control, and name it inkPanel. To ink-enable the panel, add the following lines of code to the form:

private InkOverlay overlay;

public Form1()
{
    InitializeComponent();
    this.overlay = new InkOverlay(this.inkPanel);
    this.overlay.Enabled = true;
}

Figure 1 shows a panel, tied to an InkOverlay, in action.

Figure 1 A Simple Ink-Enabled App
Figure 1 A Simple Ink-Enabled App

The InkAnalysis API operates based on ink and stroke information, which is collected in the InkOverlay’s Ink object. Fundamentally, ink analysis is as simple as creating an instance of an InkAnalyzer object, pointing it at ink, and calling the Analyze method (to use InkAnalyzer, you must add a reference to Microsoft.Ink.Analysis.dll, which, as of the February 2006 Community Technology Preview, is available in %ProgramFiles%\ReferenceAssemblies\Microsoft\TabletPC\vl.7\).

Here is the simple code needed to trigger ink analysis:

InkAnalyzer analyzer = new InkAnalyzer(this.overlay.Ink, this);
analyzer.AddStrokes(this.overlay.Ink.Strokes);
analyzer.Analyze();

This analyzes all the strokes of ink added to the analyzer. This configures the analyzer and causes it to create analysis results internally that you can subsequently access. To access the root of the hierarchical ink representation, start with the RootNode property of the analyzer after executing Analyze:

ContextNode root = analyzer.RootNode;

To perform text recognition, you use the GetRecognizedString method of either the type-specific Node or the analyzer itself:

string text = analyzer.GetRecognizedString();
// ... or ...
string text = root.GetRecognizedString();

It is also possible to access all the hierarchical subelements of the root node individually. The individual elements found that way will vary with the exact ink the user writes. But, in general, immediate subnodes of the root node are likely to be paragraphs or drawings. The following code iterates over all child nodes and retrieves their handwriting recognition results one by one:

StringBuilder text = new StringBuilder();
foreach(ContextNode subNode in root.SubNodes)
{
    text.Append(subNode.GetRecognizedString());
}

It is important to realize the hierarchical nature of the analysis tree. Performing recognition on the root node returns the complete recognized text for all the ink data added to the analyzer. Performing recognition on the subnodes returns subsections of the entire document, most likely paragraph by paragraph (again, depending on the exact data). Each paragraph can then be broken down further by accessing its children, and so forth.

It is relatively easy to walk the entire tree in a recursive fashion. The code in Figure 2 performs such a tree walk and shows the recognition results of each node in the tree, as well as the node type. The complete code for the form shown in Figure 1 is included with the code download for this article.

Figure 2 Walking the Ink Analysis Tree (Close Figure 2)
private void analyzeTreeButton_Click(object sender, System.EventArgs e)
{
    StringBuilder sb = new StringBuilder();
    using(InkAnalyzer analyzer = new InkAnalyzer(this.overlay.Ink, this))
    {
        analyzer.AddStrokes(this.overlay.Ink.Strokes);
        analyzer.Analyze();
        this.ExploreNode(analyzer.RootNode, sb, 0);
    }
    MessageBox.Show(sb.ToString());
}

private void ExploreNode(ContextNode node, StringBuilder sb, int indent)
{
    sb.Append(new string(‘ ‘, indent*4));
    sb.AppendLine(node.GetType() +
        "  [" + node.GetRecognizedString() + "]");
    foreach(ContextNode subNode in node.SubNodes)
    {
        this.ExploreNode(subNode, sb, indent+1);
    }
}

The result of walking through the analysis tree can be seen in Figure 1. If you look closely, you can see the individual nodes on the tree and their types, as well as the recognition results.

You may see ink examples that use the ToString method of a Strokes collection to retrieve recognition results. For example, instead of using GetRecognizedString as I did in the previous sample, I could have used:

string text = node.Strokes.ToString();

This is not recommended as it is a legacy feature of the ink API. Using Strokes.ToString in this fashion does not take advantage of the processing and contextual analysis performed by InkAnalyzer. Thus, the recognition results could vary depending on whether the recognizer looks at strokes making up a whole paragraph or line of ink, or whether it just looks at the strokes for an individual word. This is due to how the recognizer rates recognition quality. If a whole sentence is recognized, grammatical and other language rules can be used to aid recognition; while in scenarios with individual word recognition, only spelling can be used to verify which recognition results are likely. You should avoid these issues by instead using GetRecognizedString. Additionally, I’ve wrapped usage of the InkAnalyzer with a C# using statement in order to ensure that InkAnalyzer.Dispose is called when I’m done with it. Calling InkAnalyzer.Dispose is mandatory to avoid resource leaks.

Paragraphs, Lines, and Words

An important part of ink analysis is the ability to find logical segments of text such as paragraphs, lines, and words. The previous example looks at the whole logical tree of ink nodes, which includes this information. Often it is preferable to specifically look for certain types of text. This can be achieved either through the Type property of each node or through the FindNodesOfType method on the analyzer object. The code in Figure 3 looks at all paragraphs and, subsequently, at all lines within a node. This makes it possible to recognize handwriting and preserve line breaks at the same time.

Figure 3 Finding Nodes by Type (Close Figure 3)
StringBuilder text = new StringBuilder();
using(InkAnalyzer analyzer = new InkAnalyzer(this.overlay.Ink,this))
{
    analyzer.AddStrokes(this.overlay.Ink.Strokes);
    analyzer.Analyze();

    ContextNodeCollection paras =
        analyzer.FindNodesOfType(ContextNodeType.Paragraph);
    foreach(ContextNode paragraph in paras)
    {
        foreach(ContextNode line in paragraph.SubNodes)
        {
            text.AppendLine(line.GetRecognizedString());
        }
        text.AppendLine();
    }
}
MessageBox.Show(text.ToString());

Of course, a similar result (which can be seen in Figure 4) could have been achieved using the older Divider object, but it would have been somewhat more difficult. Also, this approach is more powerful than the ink divider. For instance, this example would completely ignore drawings, since the FindNodesOfType method is used to return paragraphs only. The ink Divider is not capable of such distinction. Ink analysis is even capable of identifying more sophisticated logical elements of ink, such as bulleted lists.

Figure 4 Preserving Layout Information
Figure 4 Preserving Layout Information

Analysis of bulleted lists requires the identification of bullets of any kind. Bullets exist within paragraphs. Whenever a bullet is found, it is the first element within a paragraph. The bullet itself does not have any text beyond the label of the bullet (such as *, or 1., or A). The text that goes along with the bulleted list item is the first line element within the paragraph. This is not a subnode of the bullet, but a sibling. Unfortunately, there is no way to find siblings easily. Therefore, the way to find bulleted-list elements is to look at paragraphs and then check whether the first item within the paragraph is a bullet.

Another approach is the use of the FindNodesOfType method to retrieve all bullets. From there on, you can look at the parent (which is the paragraph) and then look at the paragraph’s second item, which is the text that goes along with the bullet. Better approaches would depend on the specifics of the scenario at hand. If the goal is to extract bulleted lists from a large amount of ink that may include non-bulleted text, then it is better to first retrieve a list of bullets using FindNodesOfType. If the goal is to process a large amount of ink-text and simply handle bulleted lists appropriately if encountered, then the paragraph-by-paragraph approach is preferable.

"

Figure 5 shows an example of a numbered list. Figure 6 provides the code that produces this result. Note that this example performs recognition on the line of ink associated with the bullet, as well as the bullet itself, to retrieve the bullet label.

Figure 6 Finding Bullet Labels and List-Item Text (Close Figure 6)
StringBuilder text = new StringBuilder();
using(InkAnalyzer analyzer = new InkAnalyzer(this.overlay.Ink,this))
{
    analyzer.AddStrokes(this.overlay.Ink.Strokes);
    analyzer.Analyze();

    ContextNodeCollection list =
        analyzer.FindNodesOfType(ContextNodeType.InkBullet);
    foreach(ContextNode listItem in list)
    {
        InkBulletNode bullet = (InkBulletNode)listItem;
        ParagraphNode para = (ParagraphNode)bullet.ParentNode;
        LineNode line = (LineNode)para.SubNodes[1];
        text.AppendFormat("{0} {1}{2}",
            bullet.Strokes, line.GetRecognizedString(),
            Environment.NewLine);
    }
}
MessageBox.Show(text.ToString());
Figure 5 Extracting a Bulleted List
Figure 5 Extracting a Bulleted List

Analysis Hints

The InkAnalysis API does an amazing job out of the box, but it is possible to improve results further and make it easier to work with the API at the same time by using analysis hints. Analysis hints allow the developer to provide additional information to the recognizer and the analysis engine. For instance, you can set "factoids" to specify information about the type of text that is expected. This drastically improves recognition results for hard-to-recognize text such as e-mail addresses.

Factoids have been available on simple recognition objects since the first versions of the ink API. However, what makes analysis hints special is that separate areas on an ink canvas can have different factoids. For instance, it is possible to create an area on a form that is dedicated to receiving a specific kind of ink input. The following defines a rectangle within the ink panel that is dedicated to writing a name:

   Rectangle nameRect = new Rectangle(
       100, 10, inkPanel.Width-110, 60);
   AnalysisHintNode hint =
      analyzer.CreateAnalysisHint(
          GetInkSpaceRectangle(nameRect));
   hint.Name = "Name";

In this example, a new node called Name is added to the analysis tree. The Name area is defined by a rectangle that begins 100 pixels from the left and 10 pixels from the top of the form, goes across almost the remaining width of the panel, and is 60 pixels tall. From this point on, the developer has the choice to refer to everything written in this area by name. It thus becomes easy to recognize ink entered in the Name area of the form. To do so, you simply access the Name node and retrieve all linked nodes in the following fashion:

ContextNodeCollection names = analyzer.GetAnalysisHints("Name");
if (names.Count > 0)
{
    AnalysisHintNode hint = (AnalysisHintNode)names[0];
    string name = hint.Links[0].DestinationNode.GetRecognizedString();
}

The linked node is generally a WritingRegionNode, which contains a paragraph and one or more lines. It is possible to either recognize the whole writing region or drill down further in a fashion identical to what you have seen in prior examples.

Note that the rectangle defining the input area has to be defined in ink space. Ink is collected and stored at a much higher resolution than it is displayed onscreen. Generally, developers are familiar with screen coordinates and thus like to define regions using that system. However, these coordinates have to be converted into ink coordinates before they can be used as analysis hints. A detailed discussion of ink versus screen coordinates is beyond the scope of this article, but here are two simple methods that convert screen X and Y coordinates to ink X and Y coordinates using a GDI+ Graphics object and the PixelToInkSpace method exposed by the Renderer object associated with the InkOverlay:

private int GetInkX(int screenX) { return GetInkPoint(screenX, 0).X; }

private int GetInkY(int screenY) { return GetInkPoint(0, screenY).Y; }

private Point GetInkPoint(int screenX, int screenY)
{
    using(Graphics g = this.CreateGraphics())
{
    Point p = new Point(screenX, screenY);
    this.overlay.Renderer.PixelToInkSpace(g, ref p);
        return p;
    }
}

The GetInkSpaceRectangle method seen previously uses these methods to convert a screen-space rectangle into an ink-space rectangle. The complete code used to convert the rectangle is included in the code download.

This analyzer hint is a rather basic one that serves the main purpose of creating a named, simple-to-access ink input area on a form. This is useful for the simulation of a paper form that has a number of distinct fields that need to be filled out. However, analysis hints offer a lot more than that. The following code example sets aside another part of the ink panel to allow for the entry of an e-mail address:

Rectangle emailRect = new Rectangle(100, 280, inkPanel.Width-110, 60);
AnalysisHintNode hint3 =
    analyzer.CreateAnalysisHint(GetInkSpaceRectangle(emailRect));
hint3.Name = "Email";
hint3.Factoid = "EMAIL";

E-mail addresses are difficult to recognize because they use odd characters like @ and they often use sequences of characters that cannot be found in recognizer dictionaries. However, they also follow a specific pattern. They must have one and only one @ sign. They cannot have spaces, and they must have at least one dot after the @ sign. This is valuable information for the recognizer, so setting this factoid will dramatically increase the ability to recognize e-mail addresses.

There are a large number of factoids, many of which are culture specific. For an exact list of factoids, refer to the Mobile PC and Tablet PC Development Guide. Note that it is possible to set a factoid for a complete ink area and not just a specific region by creating an analysis hint without specifying a limiting rectangle. Only one such global analysis hint can exist for an ink analyzer object.

Another valuable analysis hint feature is the ability to set analysis guides. Guides define an area of ink, the number of lines and columns, and the height of each line. Finding the baseline of each line of text in a free-form ink area is an inexact science. Also, finding character heights is difficult and thus the distinction between uppercase and lowercase characters can be inaccurate. Using guides, you can define the height of the midline, an imaginary line along the top edge of a half-height character like "u."

The following code snippet defines an address area within the ink panel that contains three lines and no columns. The entire space is 150 pixels tall, with a little bit of margin at the top and bottom so the user isn’t immediately penalized if they happen to write outside the defined area. This leaves 45 pixels per line. Considering that the midline is usually a little higher than half the height of an uppercase character, I define the midline to be 25 pixels. The last two parameters passed to the RecognizerGuide object define the writing rectangle (the ideal writing rectangle for each individual line without the margin of error granted to the user) and the overall rectangle the guide is for (measured relative to the area assigned to the overall hint). Of course, all these screen coordinates have to be converted to ink space:

Rectangle addrRect = new Rectangle(100,100, inkPanel.Width-110, 150);
AnalysisHintNode hint2 =
    analyzer.CreateAnalysisHint(GetInkSpaceRectangle(addrRect));
hint2.Name = "Address";
Rectangle drawnRect = new Rectangle(5, 10, addrRect.Width-20, 45);
Rectangle writingRect = new Rectangle(0, 0, addrRect.Width, 50);
hint2.Guide = new RecognizerGuide(3, 0, GetInkY(20),
    GetInkSpaceRectangle(writingRect), GetInkSpaceRectangle(drawnRect));

The trouble is that while you have now perfectly defined various subareas of the ink panel, the user does not know about these areas, since there is no visual indication of these areas in the user interface. It is therefore advisable to add such indicators. In general it is done with a few simple lines of GDI+ drawing code that is linked to the paint event of the ink-enabled control. Figure 7 shows one such paint event handler, and Figure 8 shows the result.

Figure 7 Drawing Area Indicators (Close Figure 7)
private void inkPanel_Paint(object sender, PaintEventArgs e)
{
    // Name
    e.Graphics.DrawRectangle(Pens.Blue, 100, 10, inkPanel.Width-110, 60);

    // Address with guide lines
    e.Graphics.DrawRectangle(Pens.Blue, 100, 100, inkPanel.Width-110,
        150);
    e.Graphics.DrawLine(Pens.Gray, 105, 150, inkPanel.Width-20, 150);
    e.Graphics.DrawLine(Pens.Gray, 105, 190, inkPanel.Width-20, 190);
    e.Graphics.DrawLine(Pens.Gray, 105, 230, inkPanel.Width-20, 230);

    // Email
    e.Graphics.DrawRectangle(Pens.Blue, 100, 280, inkPanel.Width-110,
        60);
}
Figure 8 Simulating a Form Using Ink-Enabled Areas
Figure 8 Simulating a Form Using Ink-Enabled Areas

Drawing Recognition

Recognizing drawings has been traditionally difficult to implement with the Tablet PC SDK since there was no predefined functionality for this purpose. The InkAnalysis API takes a first step towards solving the problem by providing the ability to recognize drawing primitives such as rectangles, circles, triangles, and the like. The following code snippet extracts all drawing nodes and retrieves the names of the individual shapes:

StringBuilder text = new StringBuilder();
using(InkAnalyzer analyzer = new InkAnalyzer(this.overlay.Ink, this))
{
    analyzer.AddStrokes(this.overlay.Ink.Strokes);
    analyzer.Analyze();

    ContextNodeCollection drawings =
        analyzer.FindNodesOfType(ContextNodeType.InkDrawing);
    foreach(InkDrawingNode drawing in drawings)
    {
        text.AppendLine(drawing.GetShapeName());
    }
}
MessageBox.Show(text.ToString());

Figure 9 Extracting Shape Information from Ink

Figure 9 also shows drawings analyzed for hot points. I created a GDI+ drawn overlay that renders red lines from hot point to hot point. The methods responsible for this behavior can be seen in Figure 10. Note that all hot points are defined in ink space coordinates and need to be converted to screen coordinates before they can be rendered with GDI+. The code for the coordinate conversion can also be found in Figure 10.

Figure 10 Redrawing Shapes Without Ink (Close Figure 10)
private void button6_Click(object sender, System.EventArgs e)
{
using(InkAnalyzer analyzer = new InkAnalyzer(this.overlay.Ink,this))
{
        analyzer.AddStrokes(this.overlay.Ink.Strokes);
        analyzer.Analyze();

    ContextNodeCollection drawings =
        analyzer.FindNodesOfType(ContextNodeType.InkDrawing);
    using(Graphics g = this.inkPanel.CreateGraphics())
    using(Pen pen = new Pen(Brushes.Red, 2f))
        {
            foreach(InkDrawingNode drawing in drawings)
            {
                for (int pcount = 0; pcount < drawing.HotPoints.Length;
                     pcount++)
                {
                    Point startPoint = drawing.HotPoints[pcount];
                    Point endPoint =
                        (pcount < drawing.HotPoints.Length - 1) ?
                            drawing.HotPoints[pcount+1] :
                            endPoint = drawing.HotPoints[0];
                    startPoint = this.InkPointToScreenPoint(startPoint);
                    endPoint = this.InkPointToScreenPoint(endPoint);
                    g.DrawLine(pen, startPoint, endPoint);
                }
            }
        }
    }
}

private Point InkPointToScreenPoint(Point inkPoint)
{
    using(Graphics g = this.CreateGraphics())
    {
        this.overlay.Renderer.InkSpaceToPixel(g, ref inkPoint);
    }
    return inkPoint;
}

Figure 11 Linking Drawing Objects

Instead of displaying the hierarchy in a messagebox, it is displayed in a treeview control. Each node in the treeview maintains a link to the ink context node created by the ink analyzer. The ink context node is assigned to the Tag property of each tree node, so it can be accessed at a later point in time. This enables the app to react to clicks on various TreeView nodes by selecting the ink and recognizing the handwritten text associated with each node:

private void treeView1_AfterSelect(object sender, TreeViewEventArgs e)
{
    ContextNode inkNode = (ContextNode)e.Node.Tag;
    // Recognize the text
    this.textBox1.Text = inkNode.GetRecognizedString();
    // Select the associated ink
    this.overlay.Selection = inkNode.Strokes;
}

Also, the Paint event of the ink panel can be used to draw bounding boxes around linked objects for the current selection. The code in Figure 12 accomplishes that task. Note that once again, the bounding box coordinates need to be converted from ink space to screen space.

Figure 12 Bounding Box for Linked Objects (Close Figure 12)
private void inkPanel_Paint(object sender, PaintEventArgs e)
{
    if (this.treeView1.SelectedNode != null)
    {
        ContextNode inkNode =
            (ContextNode)this.treeView1.SelectedNode.Tag;
        foreach (ContextLink link in inkNode.Links)
        {
            Rectangle box1 = link.SourceNode.Strokes.GetBoundingBox();
            g.DrawRectangle(Pens.Blue, InkRectToScreenRect(
                box1, e.Graphics));
            Rectangle box2 =
                link.DestinationNode.Strokes.GetBoundingBox();
            g.DrawRectangle(Pens.Red, InkRectToScreenRect(
                box2, e.Graphics));
        }
    }
}

A slightly more complex version of this example is available in the code download. In that example, ink analysis is triggered automatically 2 seconds after the user writes the last stroke of ink. If more ink is added later, analysis is triggered again. This is accomplished by listening for the Stroke event on the InkOverlay, which fires every time a stroke of ink has been completed. The event handler code associated with this event resets a timer object with an interval of 2000 milliseconds (2 seconds). If another stroke occurs within those 2 seconds, the timer is reset. Otherwise, the timer fires and triggers the analysis, which populates a treeview as in the example just shown. Whenever the user selects a node in the treeview, the associated ink is selected, and bounding boxes are drawn to show the links. The code used to do so is almost identical to the snippets you just saw, with the exception of some more sophisticated screen refresh code. Also, the code includes a method that converts ink space coordinates to screen space coordinates.

Figure 13 Shapes Linked by a Connector Line

Using link analysis, it is possible to create advanced behavior. For instance, linked objects can be moved automatically whenever one of the objects involved in the link scenario is moved around. This enables you to keep logical units, such as words and underlines, together. In other scenarios, ink may be manipulated in different ways. For instance, if the Step 2 shape from Figure 13 is moved down, the enclosed text has to move, but the line that connects the Yes/No shape with the Step 2 shape should not move, but stretch instead. With the basis laid by link analysis, this represents a relatively simple computational problem that can be resolved based on the number of linked objects and their positions.

Link analysis represents one of the biggest improvements in the Tablet PC SDK. Without ink analysis (and link analysis), such scenarios used to represent problems of such magnitude that it wasn’t feasible for the average developer to resolve them using old-style recognizers and ink dividers.

Other Technical Aspects

All the features introduced in this article so far are directly aimed at producing recognition and analysis results. However, there are a few other technical aspects that should be mentioned as well. For one, there is the aspect of performance. Any form of ink recognition and analysis is computationally expensive. This is especially true for large ink documents. For such scenarios, synchronous analysis may not be feasible. Instead, large ink documents need to be analyzed continuously in the background.

To perform background analysis, use the analyzer’s BackgroundAnalyze method instead of the Analyze method shown in this article. Since this method is executed asynchronously, it does not directly return a result. Instead, an event-driven model is used to indicate that analysis is complete, or new analysis results are available.

Another important new feature of ink analysis is the ability to store results to disk. Using conventional recognizer objects, recognition was performed in memory only. Of course, it was possible to retrieve recognition results and store them as text, but it was not possible to serialize the state of a recognizer object. If a large ink segment was reloaded later, all recognition had to be performed again, which can be a very time consuming task. Results may also vary depending on the configuration of the recognizer. Using ink analysis, the state of the analyzer can be serialized. The analyzer object provides SaveResults and Load methods for this purpose.

The ability to save the state of the analyzer is particularly useful since the analyzer can perform incremental analysis and recognition. This means that additional strokes of ink can be added for analysis and the analyzer will only have to analyze data associated with the new strokes. Depending on the exact scenario, only the added strokes may be analyzed, or a slightly larger set of strokes may be involved if the relationship between existing and new strokes needs to be analyzed. In almost all scenarios, the workload for the analyzer will be significantly reduced compared to re-analyzing a complete document. Performance reasons alone make InkAnalysis preferable over older ink objects.

If you are interested in Tablet PC development, then take some time and familiarize yourself with the new ink analysis services. This technology is currently available in beta, but it is at the very top of my "technologies I want to use as soon as possible" list, because it provides a number of key improvements over older Ink API components and technologies.

Markus Egger is an international speaker and the publisher of CoDe Magazine. Markus is also the President and Chief Software Architect of EPS Software Corp. consulting for object-oriented development, Internet development, B2B, and Web services. See his blog at www.markusegger.com/blog.


Editor's Note

We know what you're thinking. Visual Studio 2005 has been out for a few months now. You're getting your head around it, discovering its vast inner beauty. But still, there's a nagging voice inside your head taunting you, asking why you're content to work with a released product. "You used to be cool," it says. "Remember the good times we had back in 2000, running command-line compiles on early C# code?" You smile weakly, wondering if you're hallucinating.

Part of the thrill of being a programmer is feeling like you're the first of your friends to get your hands on the newest API, the most recent build of an upcoming language, that first can of Jolt in the morning. For you, the call of LINQ is starting to become audible, faintly at first. But soon, it will reach a crescendo. You will be inexorably drawn in to take a peek at LINQ. And you probably haven't even heard of it—until now!

LINQ is the Language Integrated Query framework, which will enable the .NET Framework to provide data through a native structured query language. It will be a key feature available through C# 3.0 and Visual Basic 9.0. Visual Basic, for example, will provide support for LINQ through XML literals and late binding over XML. In effect, you'll be able to add easy data access to your code without worrying about the underlying SQL.

Here's an example, shamelessly lifted from the overview that's accessible from msdn.microsoft.com/VBasic/Future. Suppose you have a database of countries that can be represented like this:

Class Country
    Public Property Name As String
    Public Property Area As Float
    Public Property Population As Integer
End Class

If you have a collection of these objects called Countries, you can query it in code quite simply:

Dim SmallCountries = Select Country _
                     From Country In Countries _
                     Where Country.Population < 1000000

If it looks like SQL, that's because it's meant to.

With XLinq, you will be able to perform in-memory XML queries. To skip a few steps, this code will create a valid XML representation within the variable CountryInfo:

Dim CountryInfo As XElement = _
  <Countries>
    <%= Select <Country Name=(Country.Name)
                        Density=(Country.Population/Country.Area)>
               </Country> _
        From Country In Countries _
  </Countries>

This code would create XML that looks something like:

<Countries>
    <Country Name="Palau" Density="0.037117903930131008" />
    <Country Name="Monaco" Density="16694.21052631579" />
    <Country Name="Belize" Density="9.5512195121951216" />
</Countries>

You can even use the It identifier instead of the SQL * selector:

Dim CountriesWithCapital = _
    Select It _
    From Country In Countries, City In Capitals _
    Where Country.Name = City.Country

This only scratches the surface of what LINQ will let you do. It's still very early. But if you want a look at the future of programming with data, you're in luck. The LINQ community preview and Visual Basic 9.0 preview are both available from the Visual Basic Future Versions page. Here you'll also find Channel 9 interviews with technical experts Paul Vick, Amanda Silver, and Erik Meijer. You can read more about the origins of LINQ here, and look at hundreds of samples on the MSDN site.

Whether to do your cutting-edge work or go bleeding edge and stay ahead of your peers is something that must be decided between you and that voice in your head. As LINQ, C# 3.0, and Visual Basic 9.0 approach their final releases, we'll have ample coverage in MSDN Magazine to help you decide. Just remember to tell your inner voice to get its own copy.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Joe Binder, Ryan Dawson, Kevin Frei, Matt Gibbs, Ken Henderson, Chris King, Ronald Laeremans, Martyn Lovell, Polita Paulus, Sebastian Poulose, Jose Reyes, Jay Roxe, Amanda Silver, Mike Stall, Kang Su Gatlin, Maura Van Der Linden, Manu Vasandani, and Jamie Wakeam.

ActiveX, Hotmail, IntelliSense, Internet Explorer, Microsoft, MSDN, MSN, Visual Basic, Visual C++, Visual Studio, Windows, Win32, WinFX, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN MAGAZINE DOES NOT MAKE ANY REPRESENTATION OR WARRANTY, EXPRESS OR IMPLIED WITH RESPECT TO ANY CODE OR OTHER INFORMATION HEREIN. MSDN MAGAZINE DISCLAIMS ANY LIABILITY WHATSOEVER FOR ANY USE OF SUCH CODE OR OTHER INFORMATION. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Media LLC. CMP Media LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Database job scheduling, Browser Analysis, and More

Toolbox
Database job scheduling, Browser Analysis, and More
Scott Mitchell

Contents
Database Job Scheduling Support
In-Depth Browser Analysis
Add Search to Your Web Site

Database Job Scheduling Support

From a developer’s perspective, once a database’s schema has been defined and the tables, stored procedures, and other database objects have been created, it can be forgotten, abstracted away into the data access layer of the application’s architecture. DBAs, however, know that databases require ongoing maintenance. Backups need to be made on a reasonably recurring basis; for heavily loaded servers, statistics need to be manually updated during off-peak hours, and so on. Old data might be moved on a weekly basis from the production database to a reporting database or deleted altogether if no longer needed. Even application-level background tasks may be implemented as jobs at the database level.

Microsoft® SQL Server™ has long offered the SQL Agent tool for defining and scheduling SQL Server jobs. With SQL Agent you can define a job as a series of transact SQL statements, operating system commands, replication commands, and so on, and then schedule the job to run on a schedule and optionally specify whether to notify any operators based on the job’s status. Unfortunately, SQL Agent does not ship with SQL Server 2005 Express Edition. It isn’t designed for large applications, but because it’s free to use and redistribute, it’s often a database of choice for small Web sites and micro-ISVs.

For those companies using the Express Edition or for DBAs in training, Express Agent, from Vale Software, brings SQL Agent-like support to the SQL Server 2005 Express Edition. Express Agent is composed of two applications: Job Builder, a client program for designing and scheduling jobs; and a server program that runs as a service and executes the scheduled jobs, notifying the operator(s) as needed.

Express Agent’s client user interface is quite similar to SQL Agent’s. After connecting to a database, you can review the current jobs (seeing their most recent execution date and status), create new jobs or modify existing ones, and execute existing jobs immediately.

Express Agent doesn’t offer the complete set of options found in SQL Agent. There’s no support for alerts, for example, and some of the available job step features are lacking due to the limited feature set in SQL Server 2005 Express Edition. In addition to the standard job, schedule, and notification support, Express Agent also offers the ability to export and import jobs, and a wizard for scheduling recurring backups.

Express Agent provides a clean, intuitive user experience that closely mimics the experience found in SQL Agent. For those using SQL Server 2005 Express Edition who need job scheduling support, or for database administrators interested in a cost-effective way of exploring job scheduling in SQL Server 2005, check out Express Agent.

Price: $79 for a single user license.

www.valesoftware.com

In-Depth Browser Analysis

The lack of control over the end user’s environment greatly complicates designing Web sites. Visitors to your Web site are likely to have different operating systems, different browsers, different screen resolutions, and different browser settings. A well-designed Web site should be able to determine whether a user’s browser supports the site’s minimum system requirements, and display a message indicating such rather than letting the user interact with a site that was not designed to support their setup.

When a browser requests a Web page, it sends along with the request a User-Agent string identifying the type and version of the browser and the operating system. This User-Agent string can then be examined by the server application to determine the baseline functionality for the particular browser. For example, from the User-Agent string you might determine that a user is visiting with Microsoft Internet Explorer® 5.0 on Windows® XP—a browser that supports cookies and JavaScript.

ASP.NET includes a list of User-Agent strings and their capabilities, which can be programmatically referenced through the Request.Browser property.While the Request.Browser property provides some usefulness, it suffers from two main drawbacks. First, the information that ASP.NET uses that ties a browser’s User-Agent to its baseline capabilities is only updated when a new version of ASP.NET ships. In the meantime Mozilla may have released a new browser, or Internet Explorer 7.0 may have come out of beta, thereby introducing new User-Agent strings that are not fully known by ASP.NET. Second, the browser capabilities in ASP.NET only list the baseline functionality for the browser and do not take into account any settings changes made by the user.

To perform more in-depth browser analysis, you’ll need to turn to a third-party component, such as BrowserHawk 9.0, by cyScape Inc. With just a couple of lines of code, BrowserHawk can determine not only the basic browser capabilities—the browser type, major and minor version, whether it supports JavaScript, and so on—but also more detailed, extended properties, such as whether the user has disabled JavaScript support, the browser’s width and height, the time zone difference between the client and server, whether the user is blocking popups, whether Flash, Acrobat, or other software is installed, and so on.

BrowserHawk uses Browser Definition Files (BDF) that define expressions that match against known User-Agent strings. These files include matches not only for browser User-Agents, but also User-Agents for common search engine spiders. The BrowserHawk Editor makes searching and modifying the BDF list as easy as pointing and clicking. Furthermore, to ensure your BDF information stays up to date, cyScape maintains this information on their server, and BrowserHawk can be configured to periodically download this data.

Other nice features include the ability to log browser details to a database without having to write a lick of code, and being able to determine a visitor’s connection speed and provide more accurate estimates as to how long it will take a user to download a particular file. While BrowserHawk’s gaggle of features is impressive, what’s most pleasing about working with this tool is its ease of use. Installation is a snap, working with the BrowserHawk Editor is intuitive, and determining a visitor’s capabilities can be accomplished with just a short few lines of code. To top it off, BrowserHawk is available for Web sites using classic ASP, ASP.NET, ColdFusion, and JSP.

If you need a foolproof way to determine or log your visitors’ browser capabilities, BrowserHawk may provide the support you’re looking for.

Price: $524 for the professional edition and one year of User-Agent list updates.

www.cyscape.com

Add Search to Your Web Site

As the quantity of content on the Internet has continued to increase exponentially, Web surfers, both novice and experienced, are quick to turn to search engines to find the needle in the haystack. With search becoming an ingrained reflex, it’s increasingly important that Web sites offer their visitors a fast and accurate way to search the site’s content. Without search, you may attract users to your site, but if they can’t find what they’re looking for they’ll turn to Google, MSN®, or Yahoo!, leaving your site in the dust.

Search features can be added to a Web site using any one of a number of techniques. If the bulk of your Web site content is in a database, you can enable full-text indexing and issue the user’s search queries directly against the table. For sites with mainly static, file-based content, Microsoft Index Server may be a viable option. For large, enterprise-level Web sites there are hardware-based solutions, such as Google’s Search Appliance. None of these options, however, are very attractive if you run a small- to medium-sized Web site and don’t have the interest or expertise to configure Index Server or create your own search solution.

Thankfully there are affordable, commercial products like UltimateSearch by Karamasoft that can add search capabilities to your Web site without any database setup, Index Server tuning, or coding. UltimateSearch indexes the directories and file types specified in a configuration file and stores the index both on disk and in memory.

A host of configuration options can be set including specifying files or directories to ignore and words or phrases to omit from the index. This index is automatically refreshed periodically (as specified by a parameter in a configuration file) and can be manually reindexed as well. Furthermore, the indexing performs file system-based scans for static content and crawls HTTP requests for dynamic content. That means your site will be correctly indexed whether it has hundreds of separate Web pages or just a single ASP.NET page that loads different content based on querystring parameters.

In addition to its indexing capabilities, UltimateSearch provides two ASP.NET server controls to assist in integrating the search functionality into your Web site. One control renders a user interface for the search, while the second displays the results. The first provides a bevy of properties for customizing the appearance, including a "suggest" option that, when enabled, displays a dynamic dropdown list of matching search terms that automatically updates as the user enters characters into the search box. There’s also a built-in administration page that lists pages in the index and the indexed terms.

UltimateSearch is clearly designed to be a quick and easy way to add search capabilities to your Web site, and stresses ease of use over in-depth functionality. Adding UltimateSearch to your Web site and adding a site-wide search box can be accomplished in a few seconds, but more involved tasks, such as configuring different index settings for different portions of the Web site or manually modifying the index, are beyond the reach of UltimateSearch.

Price: $249 for a developer license.

www.karamasoft.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

All prices were confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.
Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


Basic Instincts: Resources and Localization

Basic Instincts
Resources and Localization
Ted Pattison

Code download available at: BasicInstincts05.exe (146KB)
Contents
Embedding a Resource
Resource Files
Working with Resource Files
Resource Files in Visual Studio 2005
The Project-Level Resource File
Culture Settings and Localization
Satellite Assemblies
Loading Localized Resources
Localizing Form and Control Settings
Conclusion

There are two ways you can utilize resources such as strings, images, and text-based files from your Microsoft® .NET Framework-based application. You can embed them directly in the app or you can load them from an external file. When you choose to load from an external source rather than from an embedding resource, you must distribute the files along with the assembly. You must also make sure that code inside the application can determine the correct path and load the resource files at run time. This approach causes problems if the .exe ever becomes separated from the files it depends upon.

Taking the embedded option and compiling the resources you need directly into the assemblies that use them makes distribution more reliable and less error prone. This month I will discuss the use of resources, how and why you might embed a resource, and the role of resource files in .NET.

Embedding a Resource

Let's start with a simple example to see how embedding is accomplished. Assume you would like to embed a graphic image named LitwareLogo.png into your Windows® Forms-based application. You begin by adding the file to the Visual Studio® project. Then, from within the property sheet for the file, you set the Build Action to Embedded Resource, as shown in Figure 1. By doing this, you have instructed Visual Studio to embed the file into the physical image of the output assembly .exe file.

Figure 1 Set Build Action
Figure 1 Set Build Action

Once you have embedded a file as a resource, you must learn how to access it at run time. Examine the following code fragment, which obtains a reference to the current assembly object and then calls the GetManifestResourceStream method to acquire stream-based access to the embedded resource file. This code assumes that you have imported the System.Reflection namespace and the System.IO namespace:

'*** get current Assembly object.
Dim asm As Assembly = Assembly.GetExecutingAssembly()
'*** load embedded resource into stream
Dim ResourceName As String = "LitwareSmartClient.LitwareLogo.png"
Dim str As Stream = asm.GetManifestResourceStream(ResourceName)
    '*** convert stream into image and load in    '*** picture box
    Dim img As Image = Image.FromStream(str)
    PictureBox1.Image = img

As you can see, an Assembly object exposes the GetManifestResourceStream method, which allows you to pass a string name that identifies the embedded resource. Note that the resource name is case-sensitive even when you are using a case-insensitive language like Visual Basic®. In the example, the code calls the Image.FromStream method to convert the stream containing the image file into an Image object that can be loaded into a PictureBox control.

In addition to embedding image files, it can be convenient to embed text-based files containing XML, SQL, or JavaScript. This can make your life a lot easier if you find it tedious to concatenate large string fragments of XML, SQL, or JavaScript using Visual Basic.

For example, let's say you have large XML documents, SQL statements, or JavaScript functions your application needs. You can maintain these as standalone .xml files, .sql files, and .js files within a Visual Studio project. With this you get the benefit of Visual Studio color coding and statement completion. You can also take advantage of Visual Studio schema-driven IntelliSense® for XML files. All that's required is that you embed these source files into the output assembly and access them using the technique you have already seen. For example, if you have embedded a SQL file and an XML file in a Windows Forms-based application, you can access them using code like that in Figure 2.

Figure 2 Load Resources (Close Figure 2)
'*** get current Assembly object.
Dim asm As Assembly = Assembly.GetExecutingAssembly()

'*** load embedded SQL resources file
Dim SqlResourceName As String = "LitwareSmartClient.GetProducts.sql"
Dim strSQL As Stream = asm.GetManifestResourceStream(SqlResourceName)
Dim reader As New StreamReader(strSQL)
Dim sql As String = reader.ReadToEnd
reader.Close()

'*** load embedded XML resources file
Dim XmlResourceName As String = "LitwareSmartClient.Customers.xml"
Dim strXML As Stream = asm.GetManifestResourceStream(XmlResourceName)
Dim xmlDoc As New XmlDocument()
xmlDoc.Load(strXML)
strXML.Close()

Resource Files

The techniques you have just seen involve embedding resource files directly into an assembly and loading them using the GetManifestResourceStream method supplied by the Assembly class. But there is another alternative in .NET, resource files, which can make it even easier to handle resources in many scenarios. Plus, as you will see, Visual Studio 2005 provides some conveniences when it comes to working with resources and localizing apps.

Working with Resource Files

In .NET, resource files can be used to embed resources into assemblies. One of the key benefits of using resource files is that all the language and locale-specific elements in an application or class library DLL such as captions and user messages can be factored out of your application code. To do this you need to create a separate resource file for each spoken language that you need to support. The actual resource file is a text-based file containing XML with a .resx extension. Figure 3 shows a skimmed-down example of the XML data found inside a resource file.

Figure 3 Inside an XML Resource File (Close Figure 3)
<root>

 <data name="MainFormCaption">
   <value>Litware Customer Manager</value>
 </data>

 <data name="UserWelcome">
   <value>Good day</value>
 </data>

 <data name="ErrorMessage1">
   <value>Oh no, Something went wrong!</value>
 </data>

</root>

While the XML fragment in the figure isn't a complete resource file, it gives you a general idea of what's inside one. You can compile a resource file into a binary image using a .NET-based utility called the resource file generator (Resgen.exe). Compiled resource files typically have a .resources extension. For example, a developer at the company named Litware can create a resource file named LitwareStrings.resx and compile it into a binary image named LitwareStrings.resources by executing the following command within a batch file or from the Visual Studio 2005 Command Prompt:

RESGEN.EXE LitwareStrings.resx LitwareStrings.resources

After you compile a text-based .resx file into a binary .resource file, it is still not yet ready to use. Instead, you must further compile this binary image into a .NET assembly before you can use it from an application. This can be accomplished using another .NET tool called the Assembly Linker (Al.exe). For example, to compile LitwareStrings.resources into its own assembly DLL, you can run the following command-line instruction from a batch file or from the Visual Studio 2005 command prompt:

AL.EXE /t:library
 /out:LitwareStrings.resources.dll
 /link:LitwareStrings.resources

Once you have compiled a resource file into a .NET assembly, you can access the resources inside using the ResourceManager class that is defined in the System.Resources namespace. The following shows a simple example of code that accesses a string resource using the ResourceManager:

Dim asm As Assembly = Assembly.Load("LitwareStrings.resources")
Dim rm As New System.Resources.ResourceManager("LitwareStrings", asm)
Dim caption As String = rm.GetString("MainFormCaption")

Resgen.exe can also be used to generate a strongly typed resource class that exposes properties providing easy access to the resource inside. For example, to a generate a strongly typed resource class in Visual Basic, you can add the /str parameter and a value of "vb" to the command line when calling Resgen.exe:

RESGEN.EXE LitwareStrings.resx LitwareStrings.resources /str:vb

This command-line instruction generates a Visual Basic source file named LitwareStrings.vb. This source file contains a class named LitwareStrings. Inside the class, there is code that uses the ResourceManager to implement strongly typed properties that look like this:

Shared ReadOnly Property MainFormCaption() As String
  Get
    Return ResourceManager.GetString("MainFormCaption", resourceCulture)
  End Get
End Property

I have just quickly stepped through a high-level explanation of how resource files are compiled into assemblies and how they can be accessed using the ResourceManager class and strongly typed resource classes.This should give you a better idea of how the individual pieces fit together.

I don't want to spend any more time here on the low-level details of resource files because you will not be required to deal with them when you begin localizing applications and class library DLLs. That's because Visual Studio 2005 and Visual Basic supply many valuable conveniences behind the scenes. However, keep in mind that you might have to work directly with Resgen.exe, Al.exe, and some of the other .NET-based utilities and classes when you are localizing large-scale development projects.

Resource Files in Visual Studio 2005

Now I am going to focus on using resource files in a Windows Forms-based application. Most of the concepts I explain will apply to using resource files in a class library DLL as well. Visual Studio 2005 makes it easy to work with resource files by supplying a visual editor. You can create a new resource file using the Add New Item command and choosing Resources file, as shown in Figure 4.

Figure 4 New Item Template for Adding Resource Files
Figure 4 New Item Template for Adding Resource Files

Once you have added a resource file to a project, there's no need to work directly with the XML format required inside a .resx file. Instead, Visual Studio supplies the friendly visual resource designer that is shown in Figure 5. This resource file designer makes it easy to add and maintain strings as well as other types of file-based resources such as graphics and XML documents.

Figure 5 Visual Studio Resource Editor
Figure 5 Visual Studio Resource Editor

When you compile a project that contains a resource file, Visual Studio compiles this .resx file into a .resources file and then links it inside the physical image of the resulting output assembly. That means that all the details involved in compiling the resource file and embedding it into the image of the target assembly are handled for you by Visual Studio.

Visual Studio also builds in a strongly typed resource class and exposes it in Visual Basic projects using the My namespace features introduced in Visual Basic 2005. That means you get the benefit of having the .NET ResourceManager class loading your resources, yet you never have to program directly against this class. For example, if you want to access resource strings you have added to LitwareStrings.resx, you can simply write the following code:

Sub Main_Load(sender As Object, e As EventArgs)
  Handles MyBase.Load
  Me.Text = _
    My.Resources.LitwareStrings.MainFormCaption
  Me.lblUserWelcome.Text = _
    My.Resources.LitwareStrings.UserWelcome
End Sub

The Project-Level Resource File

While you can explicitly add one or more resource files to a Visual Studio project, that's often unnecessary because Visual Studio automatically includes a project-level resource file each time you create a new project. The project-level resource file can be accessed through the Project Properties dialog in the Visual Studio editor.

When you want to access resources such as strings from the project-level resource file, you can access them directly from within the My.Resources class:

Private Sub LoadResources()
  '*** load project-level resources
  Me.Text = My.Resources.MainFormCaption
  Me.lblWelcomeMessage.Text = My.Resources.UserWelcome
End Sub

As you can see, accessing strings from a resource file using a strongly typed resource class is pretty easy. You can take things a step further and get the same sort of strongly typed access to file-based resources when you have added graphics images and files containing things like XML, SQL, and JavaScript to a resource file. For example, assume you have added a graphics file named LitwareLogo.png and an XML file named Customers.xml to your project-level resource file. These resources will be automatically embedded into the project's output assembly and you can access them in a strongly typed fashion using the following code:

Me.picLogo.Image = My.Resources.LitwareLogo

Dim xmlDoc As New Xml.XmlDocument
xmlDoc.LoadXml(My.Resources.Customers)

You can observe that the strongly typed resource class automatically converts a.png file into an Image object that can be directly loaded into a PictureBox. The strongly typed resource class also converts the embedded XML file into a string that can be easily loaded into an XmlDocument object.

Culture Settings and Localization

It's a common requirement for software projects to be localized so they can be used by people who speak different languages. For example, imagine you are developing the Litware Customer Manager application with Visual Basic and Visual Studio 2005 and you are required to write localized versions of the application for users who speak English as well as users who speak French. Fortunately, the Microsoft .NET Framework and Visual Studio have many features geared toward localizing applications and class library DLLs.

When you first start designing and writing .NET-based software projects that need to support localization, you must quickly become familiar with the CultureInfo class defined inside the System.Globalization namespace. A CultureInfo object tracks a culture name that identifies a spoken language.

The culture name for English is "en" and the culture name for French is "fr". The culture name of a CultureInfo object can also carry additional information that identifies a particular region in the world such as "en-US" for US English, "en-GB" for British English, and "fr-BE" for Belgian French. Here's an example that creates and initializes CultureInfo objects with valid culture names:

Dim culture1 As CultureInfo = New CultureInfo("en-US")
Dim culture2 As CultureInfo = New CultureInfo("en-GB")
Dim culture3 As CultureInfo = New CultureInfo("fr")
Dim culture4 As CultureInfo = New CultureInfo("fr-BE")

There are actually two CultureInfo objects associated with the current thread that require your attention. The first CultureInfo object listed represents the current culture while the second CultureInfo object represents the current UI culture. You can determine the culture names for each of these two CultureInfo objects with the following code:

'*** determine current culture and current UI culture
Dim t As Thread = Thread.CurrentThread
Dim currentCulture As CultureInfo = t.CurrentCulture
Dim currentUICulture As CultureInfo = t.CurrentUICulture

'*** display cultures in console
Console.WriteLine("Current Culture: " & currentCulture.Name)
Console.WriteLine("Current UI Culture: " & currentUICulture.Name)

The first CultureInfo object known as the CurrentCulture is not used to localize strings in applications. Instead, it affects how the .NET Framework formats dates, numbers, and currency. For example, if you modify the CurrentCulture object back and forth between en-US and en-GB, it can lead to strange side effects such as a single currency value switching back and forth between US dollars ($100.00) and British Pounds ($100.00). As you can see, switching the currency formatting on a currency value can lead to incorrect results. For this reason, it's a common practice to keep the CurrentCulture static even when localizing your application for use with different languages.

If you determine that you do need to programmatically change the CurrentCulture, you should remember that you must use a culture name that includes a regional identifier. You can use a culture name of en-US, en-GB, or fr-BE. However, you will receive run-time errors if you try to change the CurrentCulture so that it has a culture name of en or fr because there is no regional information and formatting requirements become too ambiguous.

The second CultureInfo object known as the CurrentUICulture is far more important to the discussion of .NET-based application localization. Modifications to the CurrentUICulture object associated with the current thread influence how the .NET Framework and the ResourceManager class load assemblies that contain embedded resources. In particular, it can have the effect of loading the set of resources that have been localized to the language preferred by the current user.

To begin localization, you make multiple copies of resource files-one copy of each resource file for each language you want to support. For example, start by taking the project-wide resource file named Resources.resx and making a copy. You can do this with a simple copy-and-paste operation right inside the Visual Studio Solution Explorer.

Once you have copied a resource file such as Resources.resx, rename the copied resource file by adding the culture name just before the .resx extension, as shown in Figure 6. For example, you should have resource files named Resources.fr.resx for generic French localized strings and another resource file named Resources.fr-BE.resx for French strings localized specifically for Belgium.

Figure 6 Separate Resource Files for Each Language
Figure 6 Separate Resource Files for Each Language

Satellite Assemblies

When you compile the project with localized resource files like the one that is shown in Figure 6, Visual Studio does not compile all the resources into a single output assembly. Instead, it compiles each of these localized resource files into its own separate assembly. Each of these localized assemblies contains only resources and no code. This type of localized, resource-only assembly is known as a satellite assembly.

Each satellite assembly is associated with a master assembly known as the neutral assembly. The neutral assembly contains all the code and it loads whatever satellite assembly is necessary to get the localized resources required by the current user. In my example, LitwareSmartClient.exe is the neutral assembly which contains all the application code. Then there are several satellite assemblies associated with LitwareSmartClient.exe. Each of these has the same file name, LitwareSmartClient.resources.dll.

When satellite assemblies are deployed along with the neutral assembly in the AppBase directory, they must be deployed according to the rules of the .NET assembly loader. In particular,each satellite assembly must be deployed in a directory named after its localized culture name. For example, the AppBase directory that contains LitwareSmartClient.exe should contain a subdirectory named fr-BE which holds the satellite assembly localized for Belgian French named LitwareSmartClient.resources.dll. As long as these rules are followed, the .NET Framework assembly loader along with the assistance of the ResourceManager class will load the correct set of resources when required.

Fortunately, Visual Studio knows how to name the satellite assemblies correctly and how to deploy them in the correct directory structure expected by the .NET assembly loader. To gain a better understanding of how all of the various pieces fit together, you can simply compile your project and then examine the resulting structure of the AppBase directory and the subdirectories inside that hold the satellite assemblies.

Loading Localized Resources

Once you have created and edited all the required localized resource files and compiled your project, it's time to concentrate on how you are going to make your application load the correct set of localized strings that are preferred by the current user. One way you can accomplish this is to obtain a reference to the current thread and assign a new created CultureInfo object to the CurrentUICulture property. If you are writing a Windows Forms-based application in Visual Basic, you can also use the following code:

My.Application.ChangeUICulture("fr-BE")

In the sample app that accompanies this column, I added support for the users to select their preferred language and to have the application track user language preferences when the application is shut down and restarted. While it is possible to maintain these sorts of user preferences using registry keys, Visual Studio 2005 makes it possible to avoid using the registry by adding an application setting that is tracked on a user-by-user basis. The sample app tracks a user-scoped application setting named UserLanguagePreference for this purpose. The application also contains an application startup event (see Figure 7).

Figure 7 Application Startup Event (Close Figure 7)
'*** code in ApplicationEvents.vb
Namespace My
 Partial Friend Class MyApplication
   Private Sub MyApplication_Startup(ByVal sender As Object, _
       ByVal e As StartupEventArgs) Handles Me.Startup

     '*** initialize application by setting preferred language
     Dim lang As String = My.Settings.UserLanguagePreference
     My.Application.ChangeUICulture(lang)

   End Sub
 End Class
End Namespace

The sample application also provides the user with a group of radio buttons as a means to switch from one language to another. Figure 8 shows a fragment of code from the event handler that responds to a user's request to change languages.

Figure 8 Language Preferences (Close Figure 8)
'*** retrieve user's language preference
Dim lang As String = CType(sender, Control).Tag

'*** save user setting
My.Settings.UserLanguagePreference = lang
My.Settings.Save()

'*** change application's UI culture
My.Application.ChangeUICulture(My.Settings.UserLanguagePreference)

'*** call custom method to reload localized strings
LoadResources()

You have now seen all the fundamental code that allows the user to switch languages. The .NET Framework responds to the call to My.Application.ChangeUICulture by loading in the correct satellite assembly the next time the ResourceManager retrieves strings from the project-level resources. After the call to ChangeUICulture, the application can then requery the application-level resource strings and load them into controls on the form with the exact same code you saw earlier inside the custom LoadResources method:

Me.Text = My.Resources.MainFormCaption
Me.lblWelcomeMessage.Text = My.Resources.UserWelcome

Note that the .NET assembly loader will first attempt to find an exact match with both language and region between the requested culture name and the culture name of an available satellite assembly. If the .NET assembly loader cannot find an exact match, it will then look for an available satellite assembly with a matching language. For example, if the requested language is fr-CA for Canadian French, the .NET Framework would first look for a satellite assembly with that language. If it cannot locate a satellite assembly with fr-CA, it then looks for a satellite assembly with a culture name of fr. If the .NET Framework cannot locate a satellite assembly with a culture name of fr, it then resorts to using the resources found within the neutral assembly, which has a set of default culture resources embedded inside. As you just saw, the .NET Framework can always fall back on the default culture resources from the neutral assembly if it cannot find a more specific satellite assembly.

When you compile the neutral assembly, you can mark it with a special attribute to inform the .NET Framework that the default culture resources are sufficient for those users who require a specific language. For example, you can add an assembly-level attribute to the AssemblyInfo.vb file of a Windows Forms-based application project, as shown in the following line of code:

<Assembly: System.Resources.NeutralResourcesLanguage("en")>

As it is used here, the NeutralResourcesLanguage attribute informs the .NET assembly loader that it can use the default culture resources whenever the current user has requested that the application be localized for English.

Localizing Form and Control Settings

You have just seen how to localize string resources on a project-wide basis. This technique involves copying and maintaining localized resource files. Visual Studio 2005 provides some extra assistance when you need to localize strings specific to a form and the controls it contains.

Each form has a Localizable property that can be set to either true or false. If you set this property to true, as shown in Figure 9, Visual Studio 2005 will create and maintain a set of localized resource files for you behind the scenes.

When you set the Localizable property to true, the initial Language setting is default. As you add property values into the property sheet for the form and its controls, Visual Studio maintains them in a resource file behind the scenes that is compiled into the neutral assembly. When you change the Language property of the form to a specific language, such as "French (Belgium)," Visual Studio then creates a new localized resource file that will be compiled into a satellite assembly. Behind the scenes, things work exactly as they do with project-wide resources. It's just that Visual Studio eliminates your need to work directly with resource files and allows you to work with a standard property sheet as you add property values for things such as control Text properties.

Figure 9 Localizable Property
Figure 9 Localizable Property

Visual Studio is required to add some extra code into the form behind the scenes to support form localization. In particular, Visual Studio adds code to load the correct localized resources and assign their values to form and control properties. Instead of using the ResourceManager, the code generated by Visual Studio 2005 uses a more specialized class that derives from ResourceManager named ComponentResourceManager that is defined in the System.ComponentModel namespace.

When a localized form loads, everything required to load the localized resources is done for you by the code that is generated by Visual Studio. In particular, Visual Studio supplies the code to create an instance of the ComponentResourceManager, which loads the proper set of resources and assigns all the necessary control values.

However, if a form is already loaded and the user switches languages, you must supply some additional code to refresh the form with the resources of the requested language. The following shows a sample of code that uses the ComponentResourceManager to accomplish this goal:

Dim crm As ComponentResourceManager
crm = New ComponentResourceManager(GetType(Main))
crm.ApplyResources(cmdAddCustomer, cmdAddCustomer.Name)
crm.ApplyResources(mnuFile, mnuFile.Name)
crm.ApplyResources(mnuFileAddCustomer, mnuFileAddCustomer.Name)

As you can see, you can create and initialize an instance of the ComponentResourceManager by passing type information about the form, which in this case is named Main.

The completed sample application, which demonstrates all the localization techniques discussed in this column, is shown in Figure 10. The application now supports US English, British English, and Belgian French. Also note that since there is a satellite assembly localized to fr without any specific region, the application also supports a generic form of French for users around the world who speak French.

Figure 10 Supporting Localized Versions
Figure 10 Supporting Localized Versions

If you want to add support for additional languages in the future, it's not very complicated. It's simply a matter of creating and maintaining additional resource files. In fact, you can add support for new languages without ever having to recompile the neutral assembly that contains all the application code. This is one of the most valuable features of the .NET Framework strategy for localizing applications and class library DLLs.

Conclusion

This month's column covered the fundamentals of resources and localization in the .NET Framework by working through how to localize a simple Windows Forms-based application. In my next Basic Instincts column, I will build upon what I covered this month and move on to a discussion of resource and localization in ASP.NET 2.0, which has some valuable and unique features for utilizing resources and localizing applications.

Send you questions and comments for Ted to instinct@microsoft.com.

Ted Pattison is an author and trainer who delivers hands-on training through his company, the Ted Pattison Group. Ted is currently researching and writing a new book focusing on Windows SharePoint Services"V3" and Office 12 servers technology.


CLR Inside Out: The Performance Benefits of NGen.

CLR Inside Out
The Performance Benefits of NGen.
Surupa Biswas

Contents
Why Use NGen?
Using NGen
Assemblies and the GAC
Choosing Base Addresses
Avoid Rebasing
NGen and Multiple AppDomains
Using Hardbinding
Regenerating NGen Images
Final Notes

T ypically, methods in managed executables are just-in-time (JIT) compiled. The machine code generated by the JIT compiler is thrown away once the process running that executable exits; therefore, the method must be recompiled when the application is run again. Moreover, the generated code is tied to the process that created it and cannot be shared between processes that are running the same application.

These characteristics of .NET JIT compilation can have performance drawbacks. Fortunately, NGen (native image generation) offers some relief. NGen refers to the process of precompiling Microsoft® intermediate language (MSIL) executables into machine code prior to execution time. This results in two primary performance benefits. First, it reduces application startup time by avoiding the need to compile code at run time. Second, it improves memory usage by allowing for code pages to be shared across multiple processes.

Although at first glance NGen appears to be similar to traditional static back-end compilation, it is actually quite different. Unlike statically compiled binaries, NGen images are tied to the machine where they are created and hence cannot be deployed. Instead, the application's installer needs to issue commands to create native images for the specific assemblies on the client machine at setup time. Also unlike traditional binaries, NGen images merely form a cache-managed applications will continue to run correctly even if all the NGen images are deleted. Of course, there will be a performance hit if that happens, but there will be no correctness issues. On the other hand, managed applications may fail to run correctly if the MSIL assemblies are deleted once they have been compiled into NGen images.

Why Use NGen?

NGen typically improves the warm startup time of applications, and sometimes the cold startup time as well. Cold startup time refers to the time taken to start an application soon after a machine reboot, such that the application is being launched for the first time after that reboot. Warm startup time, on the other hand, is the time taken to start the application given that it was launched in the recent past (and there were no reboots in between).

Cold startup time is primarily dominated by the number of pages that need to be fetched from disk. The improvement in cold startup time while using NGen can be attributed largely to the fact that pages of MSIL that need to be touched during compilation no longer need to be accessed at execution time.

Improvements in warm startup time come from reusing pages of the NGen images that were brought in when the application had been running earlier. This is especially beneficial to large client-side UI applications where startup time is critical to the user experience.

NGen also improves the overall memory usage of the system by allowing different processes that use the same assembly to share the corresponding NGen image among them, as shown in Figure 1. This can be very useful in both client and server scenarios in which the total memory footprint must be minimized. A classic example is the Terminal Services scenario in which a large number of users might be logged in and running the same application at the same time. If you're building libraries or other reusable components, you may also want to use NGen so that applications using your components can share the generated code pages.

Figure 1 Sharing Code Among Processes
Figure 1 Sharing Code Among Processes

It is important to note that NGen does not always improve the cold startup time of applications, since NGen images are larger than MSIL assemblies. The only way to determine whether cold startup time and working set will improve or degrade with NGen for specific scenarios is to actually measure them. (More on performance measurement later.)

Using NGen

You can precompile your application with NGen by using the ngen.exe tool that ships with the Microsoft .NET Framework redistributable. The .NET Framework 2.0 has a completely reworked version of this tool and also has a new NGen service called the .NET Runtime Optimization service, which is capable of NGen-compiling assemblies in the background. Thus, NGen now supports both synchronous and asynchronous compilation. The complete syntax for ngen.exe can be found at .NET Framework Tools.

All NGen images are installed in the native image cache, which is located in the %WINDIR%\assembly directory. For example, "NGen install foo.exe" will create a native image called foo.ni.exe and install it in the native image cache. Figure 2 shows the location of the native image for System.Web.dll.

Figure 2 Native Image Cache
Figure 2 Native Image Cache

Using the NGen tool you can choose to compile your assemblies either synchronously (for example, "NGen install foo.exe") or asynchronously with one of three priority levels (for example, "NGen install foo.exe /queue:3"). The asynchronous commands return immediately and are processed in the background by the NGen service. Assemblies queued up with priority 1 or 2 are compiled as soon as possible by the NGen service (priority 1 assemblies are compiled before priority 2 assemblies). Priority 3 assemblies are compiled by the service when it detects that the machine is idle. You can also use the executeQueuedItems command-line switch to force all queued items to be executed, as shown in Figure 3.

Figure 3 Asynchronous NGen Compilation
Figure 3 Asynchronous NGen Compilation

Using synchronous NGen at install time guarantees that end users will never launch your application without all the assemblies having been precompiled by NGen. However, this causes setup to take longer because the compilation time is now added to the overall install time. As an alternative, the application's setup program can issue asynchronous NGen commands. Using asynchronous NGen with priority 1 or 2 ensures that the native images will be available soon after setup completes. There is, however, a risk of the compilation work adversely affecting the performance of other applications or the system's response time to user inputs.

The other option is to use asynchronous NGen with priority 3. Priority 3 compilation seems like it's "free" in the sense that it neither consumes machine cycles during setup nor disrupts system performance. However, since machines may not be idle for extended periods of time immediately after the install, the end user could launch and use the application before the NGen service gets an opportunity to compile the assemblies. Therefore you can finely tune this trade-off by dividing your application's assemblies into categories and precompiling each of these differently. Assemblies that are absolutely critical to startup time can be compiled synchronously, those that are important but not critical can be compiled by the service as priority 1 or 2, and those in less frequently used execution paths can be compiled by the service as priority 3.

Assemblies and the GAC

In order to get the optimal performance from NGen, all assemblies in the application need to be strong-named and installed into the Global Assembly Cache (GAC) prior to compilation. If an assembly is strong-named and not installed into the GAC, the loader will need to verify the assembly's signature at load time. Validating the strong-name signature requires the loader to parse the entire contents of the MSIL assembly, compute a hash, and then compare it to the assembly's signature. The verification process therefore fetches every page of the MSIL assembly from disk. This results in a spike in the application's working set, increases load time, and pollutes the OS disk cache.

If the assembly were installed in the GAC, however, the loader would not need to validate the strong-name signature at load time. This is because the signature of the assembly is verified at the time it is installed into the GAC. The loader assumes that since the GAC is a secure store that can only be modified by an administrator, and since the signature was verified at GAC installation time, it can skip revalidating the contents. When the loader attempts to load a pure MSIL assembly from the GAC and discovers a matching NGen image, it directly loads the NGen image. In that scenario, not even a single page of the MSIL assembly needs to be fetched into memory.

This difference in behavior usually causes a significant difference in the startup time of NGen-compiled applications. Unless you have a reason to not want to install the assemblies into the GAC, it is highly recommended that you do so in order to get the best performance from NGen.

Of course, the cost of validating the strong-name signature at run time in the non-GAC case is incurred regardless of whether the application is JIT-compiled or NGen-compiled. In the NGen case, it is just particularly painful since some or all of the MSIL pages that are brought into memory during signature verification may not be used during execution. Information on how to sign an assembly and then install it into the GAC can be found at "How to install an assembly into the Global Assembly Cache in Visual C#".

Choosing Base Addresses

The next important tip to remember is that in order to get good working set performance from NGen you must carefully choose base addresses for the application's assemblies such that the NGen images do not collide with each other. Note that if the application comprises only a single.exe and hence a single native image, choosing suitable base addresses, as described in this section, is not required.

When loading an executable file, managed or unmanaged, the operating system has to allocate a contiguous block of virtual memory whose size equals that of the executable. The contents of the file are then mapped to this memory block. Executable images can contain functions and methods, global variables and constants, and references to all of these. Many of these references are absolute memory addresses. They assume that the image will be loaded at a given address, which is called the preferred base address of the executable.

As long as the operating system manages to load the executable image at its preferred base address, all references contained in the image are valid and can be used right away. Furthermore, the image can be shared across multiple processes, which usually leads to substantial savings in the working set of shared components. However, if the loader cannot place the module at the desired address (because it overlaps with another module or piece of data already loaded or allocated), the module is rebased, meaning it is loaded at some other address. This implies that all addresses within the executable image need to be fixed up.

These fix ups are expensive from a throughput perspective, since they need to be applied at load time. They are also expensive from a working set perspective, since every page in the binary that contains a memory reference needs to be fetched into memory and updated, even though the reference might never actually be used during execution. Moreover, since all these pages have now been written to, they become private and can no longer be shared between processes. Thus, there is a substantial performance hit when native code is rebased.

In the case of managed applications precompiled with NGen, base address misses are even more expensive than they are for unmanaged binaries. This is because of hardbinding (more on hardbinding in a moment). If assembly A hardbinds to one of its dependencies B, and if B misses its preferred base address, not only do all pointers in B's NGen image need to be fixed up, but every pointer in A that refers to a type or a method in B needs to be fixed up, too. The fix ups required due to hardbinding can result in a large increase in the working set, to the extent that using NGen is no longer beneficial.

JIT-compiled code does not have a rebasing problem since the addresses are generated at run time based on where the code is placed in memory. Also, MSIL is rarely affected by base address misses since MSIL references are token-based, rather than address-based. Thus when the JIT compiler is used, the system is resilient to base address collisions.

Today there is no automatic support for base address management. When creating an assembly, if the developer does not explicitly specify a preferred base address, the compiler typically picks a default value. The unfortunate thing about letting the compiler assign base addresses is that all the binaries in the application end up with the same preferred load address.

Avoid Rebasing

The only way to avoid the rebasing scenario with NGen images that I've described is to carefully assign base addresses to the application's assemblies. When choosing base addresses, keep in mind that NGen images are significantly larger than MSIL assemblies, typically 2.5 to 3 times the size of their corresponding MSIL assemblies in the case of x86. They are even larger for 64-bit systems.

There is currently no way to directly specify the load address for an NGen image. What you can do is specify a base address for the MSIL assembly. For a pure MSIL assembly, NGen chooses the base address of the native image to be the same as that of the assembly. This works quite well, since at any point in time either the assembly or the NGen image must be loaded into memory, but not both. For a mixed-mode assembly (an assembly that contains both native and managed code), both the assembly and the NGen image might need to be loaded into memory together. Thus, for such assemblies NGen selects the base address of the NGen image to be equal to the base address of the MSIL assembly plus the size of the MSIL assembly plus some padding to round up to the nearest page boundary.

The 64-bit NGen images are larger than their corresponding 32-bit native images. Both of these can be generated from the same MSIL platform-neutral assembly. If NGen used the same base addresses for both 32-bit and 64-bit native images, selection of appropriate base addresses for 32-bit systems would lead to relocations on 64-bit systems; on the other hand, selection of appropriate base addresses for 64-bit systems would lead to inefficient address space usage on 32-bit systems. NGen solves this problem by choosing base addresses for the 64-bit images based on the formula (Offset + N * Base address of MSIL assembly). For the .NET Framework 2.0, Offset is set to 0x64180000000 and N is set to 2 on x64 systems (these values might change in future releases of the .NET Framework). Correct selection of base addresses for 32-bit images typically should just work for 64-bit images, thanks to this algorithm.

The challenge for the developer then is to choose the appropriate addresses for the MSIL assemblies. Since these native images are approximately three times the size of the corresponding assemblies, one method is to make sure that two adjacent assemblies A1 and A2 are separated by at least three times the size of A1. In the case of mixed-mode assemblies, since both the MSIL and the NGen image will need to be loaded together, A1 and A2 will need to be separated by at least four times the size of A1. It is generally a good idea to leave some additional space, since assemblies (and hence the NGen images) might grow with time because of servicing or localization. Also, the factor of three or four is merely a heuristic, although it works for most assemblies. The only way to figure out the exact multiplier for a particular assembly would be to actually NGen-compile the assembly and compare the size of the image generated to that of the MSIL assembly. Since NGen itself multiplies the base address of the MSIL by N, choosing addresses that ensure that the 32-bit NGen images do not overlap with each other will typically ensure that 64-bit native images do not either.

Base addresses can be assigned to assemblies from within Visual Studio® 2005 (as shown in Figure 4) or by using the /baseaddress switch in the C# compiler (see "How to: Specify a Base Address for a DLL" and "/baseaddress (Specify Base Address of DLL) (C# Compiler Options)" for more information). Once base addresses have been assigned to each assembly, it is important to install the assemblies in the GAC, NGen-compile them, and then run the application to make sure that there are indeed no collisions. A tool that can be used to view base address collisions is Process Explorer from Sysinternals. This tool displays the preferred and actual addresses of all binaries that get loaded into memory and highlights the ones that get relocated.

Figure 4 Setting a Base Address in Visual Studio 2005
Figure 4 Setting a Base Address in Visual Studio 2005

NGen and Multiple AppDomains

When using multiple AppDomains, assemblies are loaded by default into AppDomains as domain-specific. This implies that each assembly can only be used in the context of the particular AppDomain that it was loaded into. If two AppDomains load the same assembly as domain-specific, both get their own copy of the JIT-compiled code. Alternatively, you can specify assemblies to be loaded as domain-neutral. When that happens, the assemblies are loaded into a special AppDomain called the shared domain. An assembly in the shared domain does not belong to the AppDomain which attempted to load it; instead, it belongs to all AppDomains in the process. Such assemblies are not unloaded until the process exits. Thus, another AppDomain that needs to use the same domain-neutral assembly can simply use the copy already loaded into the shared domain.

In order to get good performance when using NGen on applications with multiple AppDomains, all assemblies should be loaded as domain-neutral. When an assembly with an NGen image is loaded into an AppDomain as domain-specific, the NGen image can only be used the first time the assembly is loaded. Subsequent attempts to load the same assembly into different AppDomains will result in code being JIT-compiled. This happens because when the NGen image is loaded as domain-specific, the loader initializes data structures in the NGen image resulting in the image being hardwired to that AppDomain. Once that happens, the code essentially becomes nonshareable.

An alternative approach to falling back to JIT compilation is to load different copies of the native image into every AppDomain that loads the corresponding assembly as domain-specific. However, this approach would require every copy of the NGen image, except for the first, to be loaded at a different address from its preferred base address. Thus all of these copies of the NGen image would need their addresses fixed up.

Loading an assembly with generic instantiations in it as domain-specific can result in JIT compilation even when the code for the instantiation is stored in the NGen image and even though this might be the first time the assembly is being loaded into an AppDomain. This happens because the system requires that domain-neutral generic types be instantiated as domain-neutral so that the instantiation can be used across application domains. Consider an example where assembly A contains a generic instantiation called GI that uses value types T1 and T2. If both T1 and T2 are defined in assemblies that are loaded as domain-neutral, GI is treated as a domain-neutral generic instantiation. Now if an AppDomain attempts to load assembly A as domain-specific, the system needs to JIT-compile this instantiation, even though the code for it was available in A's NGen image.

As I mentioned earlier, JIT-compiling code in the scenarios I just described can be avoided by loading the assemblies as domain-neutral. There are some trade-offs involved here, however. Domain neutrality does not come for free. First, a domain-neutral assembly cannot be unloaded even when all the AppDomains that were using it have been unloaded. Second, NGen-compiled code is slower than JIT-compiled code when it comes to accessing static variables. This is because each AppDomain has its own copy of the static variable stored at a different location in memory. Since the same NGen-compiled code is executed by every AppDomain, the code first needs to detect the AppDomain whose context it is running in and then load or store the corresponding static variable. This extra code may result in some decrease in throughput performance in the case of C++-style code that makes heavy use of statics. However, the effect on real application code (as opposed to ported old-style benchmarks) is typically minimal. You should measure the performance in the domain-specific and domain-neutral cases and check whether there is any degradation.

An assembly can be loaded as domain-neutral by setting the LoaderOptimizationAttribute on the assembly's main method to either MultiDomain or MultiDomainHost. The default behavior of the system is to only share mscorlib.dll across all AppDomains. Setting this attribute to MultiDomain in assembly A will enable A to be shared; setting it to MultiDomainHost will enable A to be shared, but only when A is loaded from the GAC. For the latter case, assemblies not loaded from the GAC will not be shared, and hence they can be unloaded at any time. For the performance reasons I outlined previously, most assemblies are likely to be installed in the GAC, and so there should be little difference between MultiDomain and MultiDomainHost.

Using Hardbinding

Throughput performance of NGen-compiled code is worse than that of JIT-compiled code. This is one of the most common reasons cited in the literature about why you might not want to use NGen. The throughput of NGen-compiled code was on an average 5-10 percent worse than that of its JIT-compiled counterpart in versions 1.0 and 1.1 of the .NET Framework. In the .NET Framework 2.0, improvements have been made to NGen to narrow the gap in throughput performance. The improvement is due to a new feature called hardbinding. In the .NET Framework 2.0, all NGen images, by default, hardbind to the NGen images of mscorlib.dll and system.dll. This change by itself has made the throughput of NGen-compiled code comparable to that of JIT-compiled code. You may also choose to hardbind some of your assemblies to certain dependencies to further improve performance.

Throughput of NGen-compiled code is lower than that of JIT-compiled code primarily for one reason: cross-assembly references. In JIT-compiled code, cross-assembly references can be implemented as direct calls or jumps since the exact addresses of these references are known at run time. For statically compiled code, however, cross-assembly references need to go through a jump slot that gets populated with the correct address at run time by executing a method pre-stub. The method pre-stub ensures, among other things, that the native images for assemblies referenced by that method are loaded into memory before the method is executed. The pre-stub only needs to be executed the first time the method is called; it is short-circuited out for subsequent calls. However, every time the method is called, cross-assembly references do need to go through a level of indirection. This is principally what accounted for the 5-10 percent drop in throughput for NGen-compiled code when compared to JIT-compiled code.

In the .NET Framework 2.0, a new concept has been introduced-that of hardbound and softbound dependencies. Softbound dependencies are the regular kind of dependencies for which the behavior is the same as what I just outlined. Hardbound dependencies are those that are "high confidence" dependencies in the sense that these are either always or almost always loaded. If assembly A hardbinds to one of its dependencies B, code in A's NGen image will directly refer to types and methods in B's NGen image by assuming that B's image will get loaded at its preferred base address. Thus hardbinding has the advantage of avoiding the extra level of indirection that is typically required in NGen-compiled code while making cross-assembly references. This can substantially improve throughput performance. In most cases, the working set improves, too, since the fix up table pages are no longer required.

Hardbinding, however, has a few disadvantages. First, all hardbound dependencies of an assembly need to be loaded into memory before the assembly itself can be loaded. Many of these assemblies might not be required until much later (or they may not be used at all), and it can be more economic from a working set perspective to have these loaded lazily. Second, if a hardbound dependency gets relocated, NGen images for both assemblies need to be fixed up. Thus, if there are a lot of assemblies that hardbind to a particular dependency, and if that assembly happens to get rebased, the cost of performing the fix ups typically nullifies the working set savings from having used NGen.

Because of this second reason, you need to carefully choose the preferred load address of an assembly before deciding to let other assemblies hardbind to it. For instance, the base address of mscorlib.dll has been carefully chosen to minimize the chances of it being rebased. Hence for the majority of the scenarios, hardbinding all assemblies to mscorlib.dll improves throughput but does not increase the working set size. To further improve throughput, you can set attributes on your assemblies in order to hardbind them to their .NET Framework or non-.NET Framework dependencies. For example, assembly A can be hardbound to its dependency by either setting the DependencyAttribute on A or the DefaultDependencyAttribute on B. Thus if assembly A frequently calls assembly B, the developer can set the DependencyAttribute in the code for A such that it hardbinds to B:

[assembly: DependencyAttribute("AssemblyB", LoadHint.Always)]

On the other hand, if all the assemblies in the application make a lot of calls to assembly B, under all the interesting scenarios, the developer can instead just set the DefaultDependencyAttribute in the code for B such that all assemblies hardbind to B:

[assembly: DefaultDependencyAttribute(LoadHint.Always)]

Regenerating NGen Images

What I'm about to say is neither about getting good memory utilization from NGen, nor about getting good throughput. It is about making sure that the NGen images are available at all times. In order to make sure that NGen images are always available, they need to be regenerated when they become invalid.

There are a number of reasons why NGen images might get invalidated after they have been created. For example, the native image of assembly A gets invalidated by updates made to A, updates made to any of A's dependencies, changes to security policy between NGen time and run time (such as changing the trust policy from full trust to partial trust), changes to binding decisions (such as adding an application configuration file with binding redirects), and so on. These invalidations are due to the way NGen works today. For instance, native images contain fix ups for cross-assembly references (described earlier) and these are expressed in terms of metadata tokens. Once assembly A is recompiled, its metadata tokens change, and all native images in the system that had used assembly A's tokens earlier are now rendered invalid. For hardbound dependencies, there are no fix ups, and instead direct addresses are used. These direct addresses, of course, are also rendered invalid when the dependency is recompiled.

Changing the security policy occurs because while generating native images, NGen uses the security policy effective at that time to resolve link and inheritance demands. If the security policy is changed at a later time, these demands need to be reevaluated and hence the code in the NGen image is no longer valid.

The most common reason for an NGen image to become invalid is that its corresponding assembly or one of its dependencies has been serviced. Therefore, whenever an assembly is patched, the NGen images for all assemblies that depend on it and the NGen image for the assembly itself (in case it exists) must be regenerated.

NGen images can be regenerated by using the update command in ngen.exe. Issuing this command computes the list of all native images on the machine that need to be recompiled, compiles them, installs them into the native image cache, and then deletes the invalidated images. Thus, every patch to the .NET Framework 2.0 will issue the update command at the end to ensure that all native images in the system, including those generated from third-party assemblies, are kept up to date. In the future, the update command will also be issued whenever a new version of the .NET Framework is installed on the machine or whenever an existing version is uninstalled. Thus, NGen images will be available even though the install/uninstall may result in applications being rolled back or floated forward to a different version of the Framework. In that sense, the ngen.exe tool is compatible with multiple versions. For example, running version N of the ngen.exe tool on an application that is configured to run against version N-1 of the .NET Framework will actually generate version N-1 native images.

You should also remember to regenerate the NGen images whenever you issue patches to your applications. The update command has a very simple syntax-"NGen update" or "NGen update /queue"-for synchronous and asynchronous NGen, respectively. In fact, if the end user runs into issues with NGen images, most of the time these can be resolved by simply typing "NGen update" on the command line. Typically, however, end users should not run ngen.exe on random managed assemblies installed on their machines, unless they have a specific reason for doing so. Adding new native images to the native image cache results in extra disk space usage and extra machine cycles spent recompiling the images after servicing events.

Final Notes

Before I wrap up this discussion of NGen, it is important to mention that creating NGen images for all assemblies in an application and the transitive closure of its dependencies does not guarantee that the JIT compiler (mscorjit.dll) will not be loaded at run time. When mscorjit.dll is loaded into memory, it adds another 200KB or so to the overall memory footprint on an x86 system (the JIT compiler is shared across all managed applications). More importantly, whenever the JIT compiler is loaded, it implies that code is being compiled on the fly, which in turn implies that MSIL pages are being accessed, private code pages are being added to the application's working set, and so forth. It is therefore a good idea to check to see whether mscorjit.dll actually gets loaded into your process when you run your application after having created the NGen images. You can do that by simply attaching a debugger or by using the Assembly Binding Log Viewer.

The latter, shown in Figure 5, enables you to log all binds to disk and also view the list of NGen images being loaded (each log entry details the loading and verification process, as shown in Figure 6). If the list of native images is empty or if certain dependencies are missing, MSIL was being JIT-compiled. There is also a new JitCompilationStart Managed Debugging Assistant that can be used to detect whether JIT-compilation happens when the application is executed. For more information on Managed Debugging Assistants, see Stephen Toub's article in this issue.

Figure 6 Log for the SampleApp1.exe Native Image (Close Figure 6)
*** Assembly Binder Log Entry  (2/16/2006 @ 1:23:28 PM) ***

The operation was successful.
Bind result: hr = 0x0. The operation completed successfully.

Assembly manager loaded from:  C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
Running under executable  C:\SampleApp1\SampleApp1.exe
--- A detailed error log follows.

LOG: Start binding of native image SampleApp1, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=736435799a064253.
LOG: IL assembly loaded from C:\SampleApp1\SampleApp1.exe.
LOG: Start validating native image SampleApp1, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=736435799a064253.
LOG: Start validating all the dependencies.
LOG: [Level 1]Start validating native image dependency System,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 2]Start validating native image dependency mscorlib,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 2]Start validating IL dependency System.Xml,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 2]Start validating IL dependency System.Configuration,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: Dependency evaluation succeeded.
LOG: [Level 1]Start validating native image dependency mscorlib,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: Dependency evaluation succeeded.
LOG: [Level 1]Start validating IL dependency SampleLibrary1,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=a2b6b55ab9a3daa5.
LOG: Dependency evaluation succeeded.
LOG: [Level 1]Start validating IL dependency System.Drawing,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: Dependency evaluation succeeded.
LOG: [Level 1]Start validating IL dependency System.Windows.Forms,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: Dependency evaluation succeeded.
LOG: Validation of dependencies succeeded.
LOG: Start loading all the dependencies into load context.
LOG: Loading of dependencies succeeded.
LOG: Bind to native image succeeded.
Native image has correct version information.
Attempting to use native image
C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\SampleApp1\
05113aeb2745b64999e6b9959b8bd844\SampleApp1.ni.exe.
Native image successfully used.
Figure 5 Log Viewer
Figure 5 Log Viewer

When JIT-compilation is detected, you should first check to see whether there were dependencies that could not be NGen-compiled because they could not be found. A good way to check whether there were any missing dependencies is to look at the output from ngen.exe and see whether there were any errors or warnings generated. However, even when there are no errors or warnings, the JIT might still be invoked due to use of certain APIs. For example, APIs that create MSIL or source code on the fly, such as Reflection.Emit, System.Text.RegularExpressions.RegexOptions.Compiled, System.CodeDom, System.Xml.Serialization, Type.MakeGenericType, MethodInfo.MakeGenericMethod will cause JIT-compilation. Also, the use of certain loader APIs, such as Assembly.LoadFrom, require JIT-compilation, since Assembly.LoadFrom may return a different assembly each time it is called.

When considering the use of NGen, measure your application's working set size with and without having used NGen. To get an accurate reflection of the memory improvements from NGen, first build the application's setup program that installs the appropriate assemblies into the GAC and the native image cache. Then compare this to the configuration where the assemblies are added to the GAC, but not to the native image cache. Simply NGen-compiling assemblies and looking at the working set with and without having an NGen image for a particular assembly may not provide an accurate reflection.

Finally, it is good to weigh the benefits against the costs before committing to the use of NGen. The benefits include better sharing, faster startup time, and smaller memory footprint. The costs include extra disk space requirements and servicing overhead to keep the images up to date.

Send your questions and comments to clrinout@microsoft.com.

Surupa Biswas is a program manager on the CLR team at Microsoft. She works on the runtime's back-end compiler and focuses primarily on pre-compilation technologies.


Data Points: SQL Server 2005 XML Support, Exception Handling, and More

Data Points
SQL Server 2005 XML Support, Exception Handling, and More
John Papa

Code download available at: DataPoints05.exe (116KB)

SQL Server 2005 includes several important improvements to the Transact-SQL (T-SQL) language. One added feature is a new kind of trigger that fires when data definition language (DDL) statements run. DDL triggers are useful when tracking or securing the creation and modification of database objects or changes to the database server. Another of these new features involves exception handling, which has made a huge leap forward with the inclusion of TRY/CATCH blocks. Another set of new features surround the new XML datatype, which represents a huge step forward in managing XML data within enterprise applications that interact with SQL Server. Now XML documents or fragments can be stored within an XML column, bound to a schema, and queried using the XQuery syntax. This month I'll answer three interesting questions readers have sent me on these topics.

Q What are some of the advantages of both typed and untyped XML columns?

A Untyped XML columns store any well-formed XML fragment or document, and typed XML columns can be bound to an XML Schema. Untyped XML is useful when you aren't sure what schema the XML will follow. For example, if you must consume some XML from another app but you can't determine what XML schema the data will follow, untyped XML columns can do the job. Of course, untyped XML brings up other issues. For example, since you don't have a schema to program against, it is difficult to consume the XML effectively. But sometimes you can't avoid untyped XML. Creating an untyped XML column is as simple as this:

CREATE TABLE Foo(FooID INT, someXml XML)

Another consequence of not binding a schema to an XML column is that SQL Server™ 2005 stores untyped XML as a string. Is this a good thing or a bad thing? Actually, both. Storing the XML as a string gives you greater flexibility to store any XML fragment or document. You might need this based on a business rule of your application. On the other hand, storing the XML as a string means that the XML data can be neither efficiently stored nor efficiently searched when compared to typed XML. SQL Server is not told what the XML will contain, nor is it told the hierarchical structure, or the XML nodes' datatypes. However, keep in mind that just because these are untyped XML columns does not mean that they can take any format you can invent. Both untyped and typed XML columns still only accept well-formed XML.

Typed XML columns are ideal if you want to store an XML document whose schema you know. A schema can define the elements, the attributes, their datatypes, which fields are required, and the overall hierarchical structure of the data. Because this detailed information is known about the XML column's data, the XML data is stored more efficiently internally by SQL Server 2005. If you try to store XML data that does not match the schema, SQL Server will detect this and stop you.

Creating a typed XML column is as simple as adding the name of the schema inside parentheses, as you see here:

CREATE TABLE Foo(FooID INT, someXml XML(CONTENT FooSchema))
This statement indicates that the someXml column must adhere to the XML Schema Collection named FooSchema. You can specify that the XML must be a document or that it can contain a fragment by including the appropriate keyword, DOCUMENT or CONTENT, respectively. If omitted, the default is CONTENT.

You can add an XML schema collection to a database using the T-SQL command as follows:

CREATE XML SCHEMA COLLECTION [FooSchema] AS N 'put your schema here'
You can also use SQL Server Management Studio (SSMS) to create an XML Schema Collection template. Open the Template Explorer from the View menu and navigate to the XML Schema Collections node and expand it. Then you can double-click on the CREATE template to open a template that gives you a head start on the syntax to create the XML Schema Collection (see Figure 1).

Figure 1 XML Schema Collection Template is SSMS
Figure 1 XML Schema Collection Template is SSMS

The AdventureWorks database that ships with SQL Server 2005 has a table named HumanResources.JobCandidate. This table has a column named Resume which is an XML column bound to the schema collection HumanResources.HRResumeSchemaCollection. Any XML data that is stored in this column must adhere to this schema. Figure 2 shows a diagram of this schema.

Figure 2 Resume Schema
Figure 2 Resume Schema

Both typed and untyped XML columns can be indexed, but XML columns that are bound to a schema have some advantages over untyped XML columns when it comes to indexing. When an XML index is applied to an untyped XML column, much of the XML structure must be parsed to locate matching nodes. However, when an XML index is applied to a typed XML column, the specific nodes are identifiable and can be located based on the schema. So, the indexes can function more efficiently with typed XML since it knows where to look. Also, if a search for a range of a numeric value is required, the index would perform a datatype conversion with untyped XML since the datatypes are unknown. Typed XML defines its datatypes, so the expense of conversion is avoided.

Q I just migrated a SQL Server 2000 database to SQL Server 2005. I've heard that SQL Server 2005 introduces TRY/CATCH exception handling to T-SQL. How does the new exception handling work versus the existing IF @@ERROR <> 0 technique?

A One of the common complaints about writing code in T-SQL has always been its lack of a robust exception handling construct. Up through SQL Server 2000 you could write a T-SQL batch of code that checked for and even raised errors when needed. But the techniques to do so are fairly rudimentary when compared to the TRY/CATCH techniques. First I will define the problems associated with exception handling using the @@ERROR function and then I will show how the new TRY/CATCH technique compares to this.

SQL Server exposes the built-in @@ERROR function, which returns the error number for the last T-SQL command that was executed. The problem with this function is that it always returns the error returned from the statement that was executed immediately previously. This means that if you execute an INSERT statement that causes an error, then execute another SQL statement of any kind that does not cause an error, and then check to see what @@ERROR returns, the function will return a 0 because the previous statement did not return an error. You have to be very careful to make sure you check the value of @@ERROR after every single statement.

Figure 3 shows some sample code that will run inside a transaction to delete the customer's order details, its orders, and then the customer itself. Following each DELETE statement the code checks the value of the @@ERROR function to see if the DELETE statement causes an error. If it does, then I roll back the transaction, stop executing the code, and return from the batch.

Figure 3 Crude Exception Handling with @@ERROR (Close Figure 3)
BEGIN TRANSACTION

DELETE [Order Details] WHERE OrderID IN
(SELECT OrderID FROM Orders WHERE CustomerID = 'ALFKI')
IF @@ERROR <> 0
BEGIN
    ROLLBACK TRANSACTION
    RETURN
END
DELETE Orders WHERE CustomerID = 'ALFKI'
IF @@ERROR <> 0
BEGIN
    ROLLBACK TRANSACTION
    RETURN
END
DELETE Customers WHERE CustomerID = 'ALFKI'
IF @@ERROR <> 0
BEGIN
    ROLLBACK TRANSACTION
    RETURN
END
PRINT 'I got here'
-- Normally do a COMMIT TRANSACTION here.
-- But I do a ROLLBACK so I don't truly delete my test data.
ROLLBACK TRANSACTION

This is standard transaction management that aborts a transaction and returns the state of the data to its initial state if something goes wrong. Notice that if all goes well and no errors occur, I print a success message and then roll back the transaction. Normally I would not roll it back on success but because this is a test transaction, I don't actually want to delete my data.

Notice that I had to follow every statement with the @@ERROR function and then follow it up with a ROLLBACK and a RETURN. This approach can get really ugly. Imagine writing a transaction that contains dozens of queries that all need to be checked. The process could be simplified somewhat by using GOTO statements, but I am certainly not an advocate of them and they still don't solve the problem of needing to check the @@ERROR function immediately following the action query.

SQL Server 2005 still supports the @@ERROR function but it also includes the widely known TRY/CATCH paradigm. The TRY/CATCH construct is similar to the C# try/catch construct in that errors are trapped within the try block and execution is then transferred to the catch block. (The T-SQL version of TRY/CATCH has no FINALLY clause option as does the C# version. You can simulate one by catching the exceptions, not returning, and then following the catch block with appropriate cleanup code.) Therefore you need to watch out for this in certain circumstances, like when you have a CURSOR allocated and open at the time an error is thrown in the T-SQL TRY block. In this case the cursor should be checked in the CATCH block to see if it is open; if it is open, it should be closed and deallocated.

Right away what likely strikes you about the TRY/CATCH construct in Figure 4 is how compact it is versus the @@ERROR technique. Notice that the repetitive checks to see if there is an error are not necessary with the TRY/CATCH block, thus reducing the number of lines of code, which also reduces the number of places you could make a coding mistake.

Figure 4 Exception Handling with TRY/CATCH (Close Figure 4)
BEGIN TRY
    BEGIN TRANSACTION
    DELETE [Order Details] WHERE OrderID IN
        (SELECT OrderID FROM Orders WHERE CustomerID = 'ALFKI')
    DELETE Orders WHERE CustomerID = 'ALFKI'
    DELETE Customers WHERE CustomerID = 'ALFKI'
    PRINT 'committing deletes'
    COMMIT TRANSACTION
END TRY
BEGIN CATCH
    ROLLBACK TRANSACTION
    RETURN
END CATCH

The code in Figure 4 attempts to begin a transaction, execute a series of DELETE statements, and then commit the transaction. (Again, notice that in my example I am not committing the transaction but instead I am rolling it back so I don't actually delete my data. In reality this would be a COMMIT TRAN statement.)

There are a few built-in functions that you can access from within a TRY/CATCH construct that can help you determine what caused the code to enter the CATCH block. For example, you could add the following statement inside of the CATCH block to return information about that error:

 SELECT
    ERROR_LINE()         AS ErrorLine,
    ERROR_MESSAGE()      AS ErrorMessage,
    ERROR_NUMBER()       AS ErrorNumber,
    ERROR_PROCEDURE()    AS ErrorProcedure,
    ERROR_SEVERITY()     AS ErrorSeverity,
    ERROR_STATE()        AS ErrorState

These functions (where applicable) will return the line number in which the error occurred, the error message, the error number, the stored procedure, or trigger in which the error occurred, the severity level of the error, and the state level of the error.

The TRY/CATCH construct traps errors that have a severity of 11 through 19, inclusive. Severity levels below 11 are warnings and are not considered errors. Errors with a severity level of 20 and higher are considered severe errors. However, if these 20+ severity level errors do not cause the database engine to stop, they will be caught by the TRY/CATCH. Overall the TRY/CATCH statement is much easier to read, much easier to maintain, and less prone to copy and paste mistakes. Additionally, the @@ERROR technique is not always reliable. For example, some errors abort the plan or the entire batch.

Q I know that I can create triggers to audit changes to data, but how can I perform an action when a table's schema is changed?

A As I mentioned in the introduction, there's a new trigger in town: the DDL trigger in SQL Server 2005. While data manipulation language (DML) triggers fire based on data manipulation, DDL triggers fire when a change is made to the schema of a database or server. DDL triggers can only be created to fire AFTER the triggering event has occurred, as opposed to DML triggers, which can fire AFTER the event or INSTEAD OF the event. The following is the syntax for creating a DDL trigger, outlined by the SQL Server 2005 documentation:

CREATE TRIGGER trigger_name
ON { ALL SERVER | DATABASE }
[ WITH ENCRYPTION ]
{ FOR | AFTER } { event_type [ ,...n ] | DDL_DATABASE_LEVEL_EVENTS }
  [ WITH APPEND ]
  [ NOT FOR REPLICATION ]
{ AS
  { sql_statement [ ...n ] | EXTERNAL NAME < method specifier > }
}
< method_specifier > ::= assembly_name:class_name[::method_name]

Notice that you can create the trigger so that it fires when changes are made to the schema of a database or when changes are made to the database server. You can also define the events that will cause the trigger to fire by specifying the event types, which are all listed in the SQL Server 2005 documentation. Alternatively, you can have the DDL trigger fire after all events by specifying the DDL_DATABASE_LEVEL_EVENTS indicator.

With a DDL trigger you can log changes made to the schema of a database. Instead of preventing a change from being made, you can audit the change using a DDL trigger. Alternatively, you could log the attempt and then roll it back, all via the DDL trigger. You might also want to enforce naming conventions on your objects when they are created. For example, you might want all stored procedures prefixed with the characters pr. With a DDL trigger, you could enforce this naming convention.

DDL triggers are similar to DML triggers in that they both can roll back a transaction. However, DDL triggers have no inserted or deleted tables. DDL triggers have access to a built-in function called eventdata that returns XML within an XML datatype that contains information regarding the event that fired the trigger. Every event causes slightly different information to appear in the event data. However the eventdata function always returns the following information for all events:

  • The time of the event
  • The type of event
  • The SPID of the connection that caused the trigger to fire
  • The login name and user name of the user context in which the event occurred

Notice that the DDL trigger in Figure 5 is defined to fire whenever a stored procedure is created or altered on the current database. When the trigger fires, the event's data is captured and stored in a local XML variable. While this is not necessary, it can help make the trigger more efficient if you intend to access the eventdata function more than once. The trigger then checks to see if the name of the object begins with pr. If it does, I allow the transaction to complete. If it does not, then I roll back the transaction and the stored procedure would remain unchanged. In this case I know that the ObjectName element will contain the name of the stored procedure since the trigger is only being invoked when a stored procedure is created or altered. Figure 6 shows the event data for this trigger when I executed the following command:

create proc test1 as select getdate

Figure 6 Event Data for Creating a Procedure (Close Figure 6)
<EVENT_INSTANCE>
  <EventType>CREATE_PROCEDURE</EventType>
  <PostTime>2005-10-20T00:52:16.160</PostTime>
  <SPID>51</SPID>
  <ServerName>MYDBSERVER</ServerName>
  <LoginName>CAMELOT\jpapa</LoginName>
  <UserName>dbo</UserName>
  <DatabaseName>AdventureWorks</DatabaseName>
  <SchemaName>dbo</SchemaName>
  <ObjectName>test1</ObjectName>
  <ObjectType>PROCEDURE</ObjectType>
  <TSQLCommand>
<SetOptions ANSI_NULLS="ON" ANSI_NULL_DEFAULT="ON" ANSI_PADDING="ON"
      QUOTED_IDENTIFIER="ON" ENCRYPTED="FALSE" />
    <CommandText>create proc test1 as select getdate()</CommandText>
  </TSQLCommand>
</EVENT_INSTANCE>
Figure 5 DDL Trigger Fires Based on Stored Procedure (Close Figure 5)
CREATE TRIGGER ddlTrigger_ProcNamingConvention
    ON DATABASE
    AFTER CREATE_PROCEDURE, ALTER_PROCEDURE
AS
    DECLARE @eventDataXml XML
    SET @eventDataXml = EVENTDATA()
IF SUBSTRING(@eventDataXml.value('(//ObjectName)[1]',
        'VARCHAR(200)'), 1, 2) = 'pr'
        PRINT 'starts with pr'
    ELSE
    BEGIN
        PRINT 'does not start with pr'
        ROLLBACK TRANSACTION
    END
GO

You can also take the event data and store it in its entirety in an auditing table's XML column. Another option would be to pull out the specific node's contents and store them individually. Obviously the implementation depends on the situation, but your options are numerous with DDL triggers. One thing to keep in mind when using DDL triggers is that they operate synchronously. Because of the synchronous execution, a trigger should not execute time-consuming operations if at all possible. Otherwise they can adversely affect performance of the database server. Just like DML triggers, DDL triggers should be defined to do what they need to do and end quickly. If you need asynchronous execution, you can use the Service Broker and Event Notification models.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a Senior .NET Consultant with ASPSOFT (aspsoft.com) and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored several books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences such as VSLive or blogging at codebetter.com/blogs/john.papa.


Cutting Edge: Extending the GridView Control

Cutting Edge
Extending the GridView Control
Dino Esposito

Code download available at: CuttingEdge05.exe (132KB)
Contents
The GridView Difference
A New GridView Control
Adding a Checkbox Column
The New Grid In Action
Styling Selected Rows with Script Code
Script Code Injection
Wrap-Up

Welcome to my100th consecutive installment of Cutting Edge. I've been writing this column since January 1998 in Microsoft Internet Developer. Looking back over the past eight years, I realize that I've touched on almost every subject in the Windows® SDK and the Microsoft® .NET Framework. In all that time, one topic has really stood out and kept you coming back for more: the DataGrid. So to celebrate my 100th column, I'm going to talk about the successor of the ASP.NET DataGrid control—the GridView. I've already provided some details in my August 2004 article "GridView: Move Over DataGrid, There's a New Grid in Town!". Now that ASP.NET 2.0 is released, I'll continue the discussion.

The GridView Difference

Both DataGrid and GridView are rich, sophisticated controls that provide a full bag of goodies on their own. In addition, both controls expose a rich, customizable infrastructure of properties, styles, templates, and a good number of overridable members. Both the DataGrid and GridView controls are designed from the ground up for extensibility. They feature a number of protected virtual methods that control developers can utilize to architect and build even more complex components specific to their applications.

In this installment of the column, I'll show how to implement a GridView-derived control that supports multiple selection. Along the way, you'll see internal members overridden, new members added, script code silently injected into the page, and a number of state management techniques.

I think it will be much easier for you to follow the development of this column if you know exactly where I'm taking you. The final destination will be a grid like the Hotmail® Inbox grid that is shown in Figure 1.

Figure 1 Hotmail Inbox Grid
Figure 1 Hotmail Inbox Grid

As you can see the page displays e-mail messages in a grid that has sorting and filtering capabilities, along with links to images in some rows, which is not very difficult to accomplish. The checkbox column, however, fascinates me every time I check my Hotmail account.

In the grid, the checkbox column allows users to select multiple messages and execute on all of them any function that is available on the toolbar. As you probably know, DataGrid and GridView controls support selection, but only one item at a time can be selected. So how does Hotmail allow the selection of multiple messages?

To have a checkbox column in a grid, you don't need a custom control. In ASP.NET 2.0, you add a bound CheckBoxField to the GridView. If more flexibility is required, you add a templated column that includes a CheckBox control. As you'll see in a moment, the CheckBoxField control requires binding to a data source field and doesn't display any user interface elements if not bound. A templated column is the way to go if you plan to use the checkbox as a selection element. But my goal is different—I want multiple selection. When multiple selection is enabled the grid should automatically add a checkbox column whose position can be configured at design time as well as at run time.

As in the Hotmail grid, when the user checks an element, the whole row is rendered with a different style. The row returns to its original style when the element is unchecked. The operation is accomplished entirely on the client through script and doesn't require a postback. Finally, the checkbox column also contains a checkbox in the header cell. When the top element is checked or unchecked, all rows in the grid are set accordingly. This capability is also achieved through script and without round-trips to the server.

A New GridView Control

The new control will be named GridView but it will clearly belong to a different namespace—for example, MsdnMag.CuttingEdge.Controls, as shown here:

Namespace MsdnMag.CuttingEdge
Public Partial Class GridView
      System.Web.UI.WebControls.GridView
   ...
End Class
End Namespace

Note the partial modifier on the class declaration. It is not strictly required from a syntax point of view, but it does come in handy to split the source code of large and complex controls into multiple files. As you can see from the companion code for this column, the new GridView control is divided into two source files—gridview.cs and gridview.helpers.cs. Both contribute a different part of the code, thus making the whole library project more manageable. Partial classes are a hot new feature of some of the compilers in the .NET Framework 2.0 and are not specific to ASP.NET.

The object model of the base GridView control is extended with two new properties: AutoGenerateCheckBoxColumn and CheckBoxColumnIndex (see Figure 2). AutoGenerateCheckBoxColumn is a Boolean property and indicates whether a checkbox column should be generated automatically at run time. CheckBoxColumnIndex indicates the (0-based) index of this new column.

Figure 2 Properties of the New GridView Control (Close Figure 2)
<Category("Behavior")> _
<Description( _
  "Whether a checkbox column is generated automatically at runtime")> _
<DefaultValue(False)> _
Public Property AutoGenerateCheckBoxColumn As Boolean
      Get
            Dim o As Object = ViewState("AutoGenerateCheckBoxColumn")
            If o Is Nothing Then Return False
            Return DirectCast(o, Boolean)
      End Get
      Set(ByVal value As Boolean)
            ViewState("AutoGenerateCheckBoxColumn") = value
      End Set
End Property

<Category("Behavior")> _
<Description("Indicates the 0-based position of the checkbox column")> _
<DefaultValue(0)> _
Public Property CheckBoxColumnIndex As Integer
      Get
            Dim o As Object = ViewState("CheckBoxColumnIndex")
            If o Is Nothing Then Return 0
            Return DirectCast(o, Integer)
      End Get
      Set(ByVal value As Integer)
            ViewState("CheckBoxColumnIndex") = IIf(value < 0, 0, value)
      End Set
End Property

When you create a custom control, the naming convention you use for properties is more important than you may realize. Using names that are like names of similar elements in other ASP.NET controls significantly helps other developers using your code.

In ASP.NET 2.0, the ListBox supports multiple selection (the other control to do so is the CheckBoxList). It features a SelectionMode property that takes values out of the ListSelectionMode enumeration—Single, Multiple. What's the best choice? Should you dovetail with the ListBox style or name properties after the GridView standard naming? I believe neither option is wrong here. My first impulse was to use a SelectionMode property. Later, I realized that the GridView, which has good support for single selection, doesn't have a property to explicitly enable selection. It does that by adding a select button column. In other words, for the GridView, the implementation detail—the button column—counts more than the functionality. Since I'm extending the GridView, I thought that an AutoGenerateCheckBoxColumn property would have been better.

Of course, coupling the auto-generation of the column with the concept of multiple selection may limit users. For instance, what if I want to enable multiple selection but I want to set styles on my checkbox column? Or what if I want to use radio buttons instead of checkboxes? I'd have to duplicate the multiple selection logic. All of the code for this column is available for download from the MSDN®Magazine Web site, and if you disagree with my decisions, you're welcome to edit the code to your heart's content.

Adding a Checkbox Column

The columns displayed by a grid control can be specified in two ways. You can statically indicate which columns you want to display through the <columns> tag, or you can charge the grid with the task of adding as many columns as there are fields in the bound data source. This latter approach is called autogeneration and is controlled by the AutoGenerateColumns property. The two approaches can be combined and you can have, for example, a grid that first autogenerates its column set and then adds a number of statically defined columns.

When it comes to adding columns to a grid dynamically, a key fact to bear in mind is that the Columns property of a GridView and the Columns property of the DataGrid don't contain the complete list of columns being displayed. If the controls' columns are autogenerated, Columns returns an empty collections. Otherwise, both properties only list statically defined columns.

The real column set for a grid control is determined by a method that is invoked during the rendering process. The method, named CreateColumns, is marked as protected and overridable. The method takes a couple of parameters, as you see here:

Protected Overridable Function CreateColumns( _
    ByVal dataSource As PagedDataSource, _
    ByVal useDataSource As Boolean) As ICollection

The first argument is an object of type PagedDataSource. PagedDataSource is a public type that wraps the bound data source. The second argument, useDataSource, is a Boolean value that indicates whether the column set should be created from the data source (that is, through autogeneration). The CreateColumns method returns a collection formed by DataControlField objects. The useDataSource argument is only looked at if AutoGenerateColumns is true. If it is, if useDataSource is also true, then it uses the data source to create the autogenerated columns. Otherwise, it uses view state properties to create the autogenerated columns.

To add a new column, you override the CreateColumns method, as shown in Figure 3. The overridden version of CreateColumns first calls the base method and gets the default collection of columns. Next, if the checkbox column is required, it adds a new DataControlField to the collection and returns. Note that the base CreateColumns method returns an object that implements ICollection. What's ICollection exactly? It is an interface defined in the following manner:

Public Interface ICollection
       Implements IEnumerable
   Sub CopyTo(ByVal array As Array, ByVal index As Integer)
   ReadOnly Property Count As Integer
   ReadOnly Property IsSynchronized As Boolean
   ReadOnly Property SyncRoot As Object
End Interface
As you can see, the ICollection interface doesn't provide any methods to add new items. To insert a new column into the list, you need to copy the ICollection object into an ArrayList, or a similar ICollection-based object that allows you to add items to the collection, and then proceed with required changes. Figure 4 shows the source code of the AddCheckBoxColumn method where the ICollection object is copied into an ArrayList and then extended with a newly created field.

Figure 4 Adding a New Checkbox Column to the Grid (Close Figure 4)
Protected Overridable Function AddCheckBoxColumn( _
        ByVal columnList As ICollection) As ArrayList

    ' Get a new container of type ArrayList that contains the
    ' given collection. This is required because ICollection
    ' doesn't include Add methods.
    Dim list As New ArrayList(columnList)

    ' Create a new custom CheckBoxField object
    Dim field As New InputCheckBoxField
    field.HeaderText = "<input type=checkbox>"
    field.ReadOnly = True

    ' Insert the checkbox field into the list at the specified position
    If CheckBoxColumnIndex > list.Count Then
        list.Add(field)
        CheckBoxColumnIndex = (list.Count - 1)
        Return list
    End If

    list.Insert(CheckBoxColumnIndex, field)
    Return list
End Function
Figure 3 Override the CreateColumns Method (Close Figure 3)
Protected Overridable Function CreateColumns( _
        ByVal dataSource As PagedDataSource, _
        ByVal useDataSource As Boolean) As ICollection

    ' Let GridView create the default set of columns
    Dim list as ICollection = _
        MyBase.CreateColumns(dataSource, useDataSource)
    If Not AutoGenerateCheckBoxColumn Then Return list

    ' Add checkbox column if required
    Return AddCheckBoxColumn(list)
End Function

The new field will be inserted in the position specified by the CheckBoxColumnIndex property. When you set the property, any negative value is automatically evaluated as zero, as implemented in Figure 2. The CheckBoxColumnIndex property setter, though, can't check the number of available columns to set an upper bound for the value. This check is performed when the column is physically added to the list, that is, as soon as the real number of columns being displayed is known.

As you see in Figure 4, a custom data control class is used to display a checkbox in the column. ASP.NET 2.0 has a brand new CheckBoxField class that seems perfect, so why not use it instead? The problem is that CheckBoxField puts a checkbox it each cell only if it is bound to a valid data source field. In this case, you do want a checkbox in each column, but you also need to keep the column unbound and use it only to collect some input. For this task, a CheckBoxField component is simply not appropriate. In Figure 5 you see the source code of a replacement class—InputCheckBoxField. The class is marked internal and sealed because its usefulness is limited to the implementation of this enhanced GridView control.

Figure 5 InputCheckBoxField Class (Close Figure 5)
Namespace MsdnMag.CuttingEdge.Controls
  Friend NotInheritable Class InputCheckBoxField : Inherits CheckBoxField

      Public Sub New()
      End Sub

      Public Const CheckBoxID As String = "CheckBoxButton"

      Protected Overrides Sub InitializeDataCell( _
              ByVal cell As DataControlFieldCell, _
              ByVal rowState As DataControlRowState)

          MyBase.InitializeDataCell(cell, rowState)

          ' Add a checkbox anyway, if not done already
          If (cell.Controls.Count = 0) Then
              Dim chk As New CheckBox
              chk.ID = InputCheckBoxField.CheckBoxID
              cell.Controls.Add(chk)
          End If
      End Sub
  End Class
End Namespace

The InputCheckBoxField class overrides the InitializeDataCell method, which is the nerve centre of a grid column class where the UI of a cell is determined. The overridden method calls the base version and then, if no controls have been created, adds an unbound checkbox with a predefined ID. You're pretty much finished with the core functionality—checking multiple rows simultaneously. Before putting a sample page through its paces, I'll briefly discuss the reasons why the ASP.NET team decided to expose the column set through a collection class instead of a list class.

When writing a public API that includes a list of objects as input or output, should you use a collection or a list? Here a collection is a class that implements ICollection, whereas a list is a class that implements IList. In general, you should use a collection for class properties and for return values of methods. A list is more appropriate for implementing internal properties or for specifying the input value of a property or the argument of a method. The CreateColumns method returns ICollection in accordance with this guideline.

IList has a richer set of methods than ICollection and, more importantly, has members that can add or remove items. Because IList implements ICollection, you can use a list as an input value wherever an ICollection is accepted.

What if you really need the extra methods of a list like in the override of the CreateColumns method that you saw in Figure 3? In this case, it is recommended that you copy your nonextensible collection into a list object such as an ArrayList or a List(Of T) and proceed from there. An interesting blog that covers these topics is written by Krzysztof Cwalina, a program manager on the common language runtime (CLR) team at Microsoft.

The New Grid In Action

Let's create a page that contains my custom GridView. It will provide a simple and effective toolbar to expose any functions to call on the selected items. The markup for the grid is nearly identical to the markup of a regular GridView, except for the new properties AutoGenerateCheckBoxColumn and CheckBoxColumnIndex. Figure 6 shows the page. Users can click on the checkboxes and thus mark the row for selection. I'll discuss in a moment how to enhance the code to have each selected row rendered with a different style.

Figure 6 New GridView Control
Figure 6 New GridView Control

So you now have a multiselection control. How do you know which rows are currently selected? Imagine the user clicks on a page button or a menu item, and the page posts back needing to execute some special code for each selected item. How would you work that out?

The ListBox control, another ASP.NET control to support multiple selection, exposes a public method named GetSelectedIndices. This method returns an array of integers where each integer represents the 0-based index of a selected element:

Public Overridable Function GetSelectedIndices() As Integer()
    Return CType(SelectedIndices.ToArray( _
        GetType(Integer)), Integer())
End Function

The method wraps an internal property that contains all selected indices. The heart of the function is just inside the implementation of the property in Figure 7.

Figure 7 SelectedIndices (Close Figure 7)
Friend Overridable ReadOnly Property SelectedIndices As ArrayList
    Get
        Dim indices as New ArrayList
        For i As Integer = 0 To Rows.Count - 1
           Dim cb As CheckBox
           cb = CType(Rows(i).FindControl(
                    InputCheckBoxField.CheckBoxID), CheckBox)
           If cb Is Nothing Then Return indices
           If cb.Checked Then indices.Add(i)
        Next
        Return indices
    End Get
End Property

The property is protected and as such doesn't show up in the Visual Studio 2005 IDE. It is a read-only property that works by copying the indices of the selected row into a local array list object. The get accessor of the SelectedIndices internal property loops through the Rows collection of the grid and retrieves the instance of the checkbox. If the checkbox is checked, the index is added to the array.

From a codebehind class, you consume the list of selected indices as follows:

   For Each index As Integer in _
       GridView1.SelectedIndices
    Dim key As String = _
       GridView1.DataKeys(index).Value.ToString()
       ...
   Next

As long as you're interested in the plain index, the array returned by SelectedIndices is perfect. Unfortunately, you usually need a bit more than the index. Typically, you need a key value for the selected row that allows you to easily and effectively retrieve the data item associated with it. For example, if the grid displays a list of employees, the key value is typically the employee ID. The GridView allows you to define multiple data key fields through the DataKeyNames property.

Upon postback, the values that are effective for the specified fields are available in the DataKeys array. By using a returned index as a selector in this array, you easily obtain key information for the selected rows:

For Each index As Integer in GridView1.SelectedIndices
   Dim key As String = GridView1.DataKeys[index].Value.ToString()
   ...
Next

Styling Selected Rows with Script Code

As I mentioned, the GridView control fully supports single-row selection. It does that through the SelectedIndex and the SelectedRowStyle properties. Once you set the SelectedIndex property to an index, the control renders the corresponding row according to the SelectedRowStyle style attributes. Another property, called AutoGenerateSelectButton, allows you to add a column of select buttons. Clicking any button automatically sets the SelectedIndex property to the index of the clicked row. Note that this clicking causes a postback.

When you click on a checkbox element instead, no round-trip occurs. Would it be possible to update the style of the clicked row on the client and persist these changes to the server on the next postback? Fortunately, ASP.NET takes care of most of the burden by automatically persisting the checked state of each checkbox in the column. In Figure 6, in fact, you see a message obtained by processing the selection. This fact clearly indicates that a postback occurred. At the same time, the checkboxes mentioned in the message are still checked.

You need to add some script code to each checkbox so that the style of the row changes in accordance with the checked state of the checkbox. To keep client-side events in sync with server-side events, you might want to use the same visual attributes of SelectedRowStyle on the client. Figure 8 shows a modified version of the control that uses client-side script to style selected rows. As you can guess from the figure, if you write good, cross-browser script code, the feature will work on most browsers available today, including Firefox.

Figure 8 Cross-Browser Compatibility
Figure 8 Cross-Browser Compatibility

You need a JavaScript onclick event to be associated with each checkbox displayed in the column. The onclick event handler will simply set some style attributes. Figure 9 details the JavaScript code injected by the GridView control into each page that uses it.

Figure 9 Client-Side Styling of the GridView (Close Figure 9)
// Invoked to check/uncheck all when header checkbox is clicked
// ---------------------------------------------------------------
function CheckAll(me)
{
    // Assume the ID of this element is controlID_HeaderButton
    // (i.e. GridView1_HeaderButton)

    var index = me.name.indexOf('_');
    var prefix = me.name.substr(0,index);

    // Looks for the right checkbox
    for(i=0; i<document.forms[0].length; i++)
    {
        var o = document.forms[0][i];
        if (o.type == 'checkbox')
        {
            if (me.name != o.name)
            {
                if (o.name.substring(0, prefix.length) == prefix)
                {
                    // Needs to set the state of the header to any child
                    // checkbox. To avoid passing too many parameters,
                    // sets state by simulating the click. To ensure that
                    // the click sets the checkbox to the "me.checked"
                    // state, it must first the current value.
                    o.checked = !me.checked;
                    o.click();
                }
            }
        }
    }
}

// Invoked to properly style the checkbox on a row
// ---------------------------------------------------------
function ApplyStyle(me, selectedForeColor, selectedBackColor, foreColor, backColor, bold, checkBoxHeaderId)
{
    var td = me.parentNode;
    if (td == null)
        return;

    var tr = td.parentNode;
    if (me.checked)
    {
       tr.style.fontWeight = 700;
       tr.style.color = selectedForeColor;
       tr.style.backgroundColor = selectedBackColor;
    }
    else
    {
       document.getElementById(checkBoxHeaderId).checked = false;
       tr.style.fontWeight = bold;
       tr.style.color = foreColor;
       tr.style.backgroundColor = backColor;
    }
}

Figure 10 illustrates the server code that runs in the pre-render phase and injects any needed onclick handlers. The ApplyStyle function is associated with each checkbox in the column and sets font and color on the row. The function takes six parameters—foreground and background colors to use when the row is selected and unselected, plus a value that indicates whether a boldface font is required. The final argument is the ID of the checkbox in the header. (I'll return to this in a moment.)

Figure 10 Supporting Client Scripting (Close Figure 10)
Protected Overrides Sub OnPreRender(ByVal e As EventArgs)

    ' First do as usual
    MyBase.OnPreRender(e)

    ' Adjust each data row
    Dim row As GridViewRow
    For Each row In Row

        ' Get the appropriate style object for the row
        Dim style As TableItemStyle = GetRowStyleFromState(row.RowState)

        ' Retrieve the reference to the checkbox
        Dim cb As CheckBox = CType(row.FindControl( _
            InputCheckBoxField.CheckBoxID), CheckBox)

        ' Build the ID of the checkbox in the header
        Dim headerCheckBoxID As String = _
            String.Format("{0}_HeaderButton", ClientID)

        ' Add script code to enable selection
        cb.Attributes.Item("onclick") = String.Format( _
            "ApplyStyle(this, '{0}', '{1}', '{2}', '{3}', '{4}', '{5}')",
            ColorTranslator.ToHtml(SelectedRowStyle.ForeColor),
            ColorTranslator.ToHtml(SelectedRowStyle.BackColor),
            ColorTranslator.ToHtml(style.ForeColor),
            ColorTranslator.ToHtml(style.BackColor),
            IIf(style1.Font.Bold, 700, 400),
            headerCheckBoxID)
        If cb.Checked Then
            row.BackColor = SelectedRowStyle.BackColor
            row.ForeColor = SelectedRowStyle.ForeColor
            row.Font.Bold = SelectedRowStyle.Font.Bold
        Else
            row.BackColor = style.BackColor
            row.ForeColor = style.ForeColor
            row.Font.Bold = style.Font.Bold
        End If
    Next
End Sub

The colors for the selected state are taken directly from the SelectedRowStyle object. The SelectedRowStyle is just part of the style applied to a selected row. RowStyle and AlternatingRowStyle will also apply for style properties that aren't set in SelectedRowStyle. You can think of the applied style as a layering of RowStyle, those that apply of AlternatingRowStyle and EditRowStyle, and finally SelectedRowStyle. This means that a classic select button row will look different than a new multiselect row.

The header of the checkbox column also shows a checkbox. The idea is to use that checkbox to check or uncheck all rows. To begin with, the header text of the checkbox column should be changed to a more sophisticated template, like this:

<input type='checkbox' id='{0}' name='{0}' {1} onclick='CheckAll(this)'>

You must give the checkbox a unique ID that must be known to other portions of code in the control. This ID can be considered a private constant. I arbitrarily named it XXX_HeaderButton where XXX stands for the ID of the grid control.

When the user clicks the header checkbox, the CheckAll script function simulates the click on all checkboxes in the column. Given this expected behavior, it follows that anytime you uncheck an element in the column, you must also ensure that the header checkbox is unchecked. For this reason, the ApplyStyle function needs to know the ID of the header checkbox (see Figure 9).

There's one little feature left. The state of the header checkbox is not automatically persisted across postbacks. This is due to the fact that you are forced to add the checkbox via text. In ASP.NET, in fact, you can only use a string to set the header of a column unless you use a template field.

In any case, the browser posts a field with the client-side state of the header checkbox. Because there's no server-side control in the page where ID matches the XXX_HeaderButton name, that value is lost. How can you retrieve that? Simply by using the old faithful Page.Request object. By adding the following code to the AddCheckBoxColumn method you saw earlier, you insert a key "checked" attribute in the HTML string that renders out the checkbox in the header:

Dim shouldCheck As String = String.Empty
Dim checkBoxID As String = String.Format("{0}_HeaderButton", ClientID)
If Not DesignMode Then
Dim o As Object = Page.Request(checkBoxID)
If o IsNot Nothing Then shouldCheck = "checked='checked'"
End If

Note that the custom GridView control takes full advantage of the design-time infrastructure of the parent GridView control. You don't need to add any design-time specific feature, except perhaps for some property descriptions. However, the AddCheckBoxColumn method is invoked when the control renders out. This means you will get an error in Visual Studio 2005 because Page.Request is not defined at design time. On the other hand, at design time you don't need to know about the state of the header checkbox and can simply render it always unchecked. The new DesignMode property that is found in all ASP.NET 2.0 controls lets you fix issues like these throughout the control code.

An alternative solution is to have InputCheckBoxField override InitializeCell to look for header cells, put a checkbox in each, and give each checkbox in a header the ID you want, making it easier to find later. That would avoid this whole issue.

Script Code Injection

So the multiselection grid requires some script code to be injected in the host page. Pasting the code from Figure 9 in every page where you use the grid control is clearly not an option. A better option would be to save the code to a .js file to include in each host page. In both cases, though, you have a tight coupling between the script and the page. It would be much better if the control itself could automatically inject the code in each page where it is hosted.

ASP.NET provides a number of methods for registering code with the .NET Framework that will be emitted in the markup served to the browser. The set of methods in ASP.NET 2.0 is rooted in the ClientScript object. RegisterClientScriptInclude, for example, emits a <script> tag with a src attribute. The great news is that in ASP.NET 2.0 you can embed a .js file in the assembly and expose it as a Web resource.

You take the .js file that contains the source code to inject and add it to a control project as an embedded resource. It is important that you change the build action of Visual Studio 2005 for this file from Content (the default) to Embedded Resource. In this way, the .js file becomes part of the assembly and is deployed along with the control. The following code shows how to register a <script> block bound to an autogenerated URL that refers to the script in the assembly:

Dim t As Type = Me.GetType()
Dim js As String = "MsdnMag.CuttingEdge.Controls.HotGridView.js"
Dim url As String = Page.ClientScript.GetWebResourceUrl(t, js)
Page.ClientScript.RegisterClientScriptInclude(t, js, url)

GetWebResourceUrl returns a URL for the assembly resource. The returned URL is based on webresource.axd, a new system HTTP handler defined in ASP.NET 2.0. It is important that the resource name includes namespace information and that the resource is decorated with the WebResourceAttribute. You do that in the AssemblyInfo.vb file through the following code:

<assembly: WebResource("MsdnMag.CuttingEdge.Controls.HotGridView.js", _
                       "application/x-javascript")>

Wrap-Up

Although the GridView is a rich and powerful control, there are still changes you can make to improve it. The new grid control I created here, for example, works like the parent GridView control and supports multiple selection through a checkbox column with client-side capabilities.

When it comes to extending the feature set of an existing control, what you can do mostly depends on how the extendee control was designed. Without a virtual CreateColumns method, for example, it would have been much harder, if not impossible, to create a new grid control as I did here. When you create your own controls, think about ways that other developers might want to customize and extend them.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino by e-mailing him at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Test Run: Stress Testing.

Test Run
Stress Testing.
James McCaffrey

Code download available at: TestRun05.exe (116KB)
Contents
Overall Program Structure
The Tool in Detail
Cool Console Output
Extending the Tool
Wrapping It Up

Stress testing is a fundamental quality assurance activity that should be part of every significant software testing effort. The key idea behind stress testing is simple: instead of running manual or automated tests under normal conditions, you run your tests under conditions of reduced machine or system resources. The resources to be stressed generally include internal memory, CPU availability, disk space, and network bandwith. To reduce these resources for testing you can run a tool called a stressor.

The image in Figure 1 shows one such stress tool, EatMem, in action. It is common to name stressors with an "Eat" prefix followed by the type of resource being consumed, so stressors that reduce memory, CPU availability, and disk space might be named EatMem, EatCpu, and EatDisk, respectively. The EatMem stressor is a command-line program that accepts a single argument specifying how long, in minutes, to run. Approximately every three seconds EatMem attempts to allocate a random amount of available memory. As you see in Figure 1, it is possible for the memory request to fail. In that case EatMem keeps trying until successful. Of course, the stressor tool by itself does not perform any actual testing; it simply prepares a machine state for testing. By running this program at the same time as the actual program under test, you can simulate conditions for your application that may otherwise be difficult to test for under normal testing circumstances.

Figure 1 EatMem Stressor in Action
Figure 1 EatMem Stressor in Action

Overall Program Structure

The overall structure of the EatMem stressor tool, written in C#, is presented in Figure 2. I begin by adding a using statement to the InteropServices namespace so I can call native Win32® functions. In particular I call the GlobalAlloc and GlobalFree functions to allocate and free memory, and the GlobalMemoryStatus function to determine how much memory to allocate. I also use some Win32 API functions to manipulate the console text colors.

Figure 2 EatMem Program Structure (Close Figure 2)
using System;
using System.Runtime.InteropServices;

namespace EatMem
{
  class Class1
  {
    [DllImport("kernel32.dll")]
    extern static IntPtr GlobalAlloc(uint uFlags, uint dwBytes);

    [DllImport("kernel32.dll")]
    extern static IntPtr GlobalFree(IntPtr hMem);

    [DllImport("kernel32.dll")]
    extern static void GlobalMemoryStatus(ref MEMORYSTATUS lpBuffer);

    public struct MEMORYSTATUS
    {
      public  uint dwLength;
      public  uint dwMemoryLoad;
      public  uint dwTotalPhys;
      public  uint dwAvailPhys;
      public  uint dwTotalPageFile;
      public  uint dwAvailPageFile;
      public  uint dwTotalVirtual;
      public  uint dwAvailVirtual;
    }

    static MEMORYSTATUS memStatusBuffer = new MEMORYSTATUS();

    [STAThread]
    static void Main(string[] args)
    {
      try
      {
        // get number of minutes to run from command line
        // determine stopping time
        // print output header

        while (currentTime < stopTime) // main loop
        {
          for (int cycle = 0; cycle < 8; ++cycle)
          {
            // get the elapsed time so far

            // compute a random percentage
            // get current memory status
            // compute number bytes to eat

            // allocate memory
            // get current memory status
            // print current time, memory status

            // pause to give app under test stress time

            // free memory
          } // for loop
        } // main while loop
      }
      catch(Exception ex)
      {
        Console.WriteLine(«Fatal error: « + ex.Message);
        Console.ReadLine();
      }
    } // Main()
  } // class
} // ns

The GlobalAlloc and GlobalFree functions are defined in the kernel32.dll library. By using the DllImport attribute I can call them from my C# stressor program. The original C++ signature for GlobalFree is:

   HGLOBAL GlobalAlloc(UINT uFlags,
       SIZE_T dwBytes);

The return type, HGLOBAL, is one of the thousands of Win32 symbolic constants. It is essentially a pointer to the first address of memory allocated by GlobalAlloc, so I translate that type into a System.IntPtr type. IntPtr is a platform-specific type that is used to represent either a pointer or a handle. I will explain GlobalAlloc and its partner GlobalFree more fully in the next section. I use the GlobalMemoryStatus function to retrieve the amount of currently available memory so I can determine how much memory to attempt to allocate with the GlobalAlloc function. The Win32 signature for GlobalMemoryStatus is:

void GlobalMemoryStatus(LPMEMORYSTATUS lpBuffer);

You pass a pointer/reference to a struct into GlobalMemoryStatus and it populates the struct with memory information. The struct is defined in Figure 2. The most important field in the struct is the dwAvailVirtual field, which will hold the amount, in bytes, of available virtual memory, after the function GlobalMemoryStatus has been called.

The EatMem tool is organized primarily as a while loop that will iterate until the current system date/time exceeds the date/time determined by adding the duration, in minutes, specified on the command line to the time when the tool started. Inside the main while loop, there is a secondary for loop that executes eight times. This secondary loop is not functionally necessary; it is there to provide a nice way to display progress output every so often.

The Tool in Detail

Now let's go through each section of the EatMem tool in detail. I will discuss fancy console output in the next section of this column, and I will discuss alternatives and modifications to the basic code later. Figure 3 shows the code that will get EatMem going.

Figure 3 Initialize EatMem (Close Figure 3)
const uint basicUnit = 1024; // 1 KB
Random r = new Random();

if (args.Length != 1 || !args[0].StartsWith("-d"))
{
  Console.WriteLine("\nExample usage: eatmem -d100");
  Console.Write("Randomly eats between 10% and 70%
  Console.WriteLine(" of virtual memory for 100 minutes");
}

// determine stopping time
int duration = int.Parse(args[0].Substring(2, args[0].Length-2));
TimeSpan ts = new TimeSpan(0, duration, 0); // hr, min, sec
DateTime startTime = DateTime.UtcNow;
DateTime stopTime = startTime + ts;

I begin by declaring a constant of 1024 because I will do all EatMem display in kilobytes, while the calculations will be performed in bytes. Next I instantiate a Random object so I can generate pseudorandom numbers later.

Then I use a very crude technique to grab the duration (in minutes) that EatMem should run. My code assumes there is exactly one command-line argument, in the exact form of -dN where N is a number. I can get that number as shown because args[0] will represent a string like "-d123", and Substring(2) will be the "123" part. I use a standard technique to determine the stopping time: I create a TimeSpan object of the desired number of minutes, then use that TimeSpan with the overloaded "+" operator applied to the time when EatMem started.

The last thing that I need to do before entering the main while control loop is to display a brief header for the output along the lines of the following code:

// print output headers
WriteLine("Begin EatMem for " + duration + " minutes");
WriteLine("Consuming random 10%-70% of virtual memory");
WriteLine("Launch test harness while EatMem stressor runs");

WriteLine("      Available  Available    Current");
WriteLine("       Physical    Virtual     Memory");
WriteLine(" Time    Memory     Memory  Allocation   Pct.");
WriteLine("---------------------------------------------");
Note that I've changed the actual statements slightly for clarity. You can get the exact statements in the code download that accompanies this column.

As you'll see later in this column there is a lot of additional information available to you. Because stressor tools are intended to modify the state of the machine on which they run, all output from the stressors is to some extent unnecessary. The most important data to monitor is the amount of virtual memory available at any given time.

Once inside the main while loop and the auxiliary for loop, I get the elapsed time into a TimeSpan object and compute a random percent, as you see here:

while (DateTime.UtcNow < stopTime) // main loop
{
  for (int cycle = 0; cycle < 8; ++cycle)
  {
    // get the elapsed time so far
    TimeSpan elapsedTime = DateTime.UtcNow - startTime;

    // compute a random percentage
    double randomPct = (r.NextDouble() * (0.70 - 0.10)) + 0.10;
    if (randomPct < 0.10 || randomPct > 0.70)
      throw new Exception("Bad random pct");

Here I've hardcoded EatMem so that it generates a random number between 0.10 and 0.70 using a standard mapping technique. The method call Random.NextDouble will return a number in the range [0.0, 1.0], which means greater than or equal to 0.0 and strictly less than 1.0. If I multiply that interval by (0.70 - 0.10) = 0.60, the interval will map to a random number in the range [0.0, 0.6]. Then if I add 0.10, the final interval will map to [0.10, 0.70].

The next step is to compute the number of bytes to eat:

// compute number of bytes to eat
GlobalMemoryStatus(ref memStatusBuffer);
uint numBytesToEatInBytes =
  (uint)(randomPct * memStatusBuffer.dwAvailVirtual);
uint numBytesToEatInKB =
  numBytesToEatInBytes / basicUnit;

I call the Win32 GlobalMemoryStatus method using the C# interoperability mechanism to get the number of bytes of virtual memory currently available. Notice I pass in a ref to the MEMORYSTATUS struct because the original C++ signature requires a pointer to the struct argument. Or, to put in managed code terms, I pass in a reference to a struct because the struct will change. After the memory status information is stored into the memStatusBuffer object, I can pull it out. I determine the number of bytes to allocate by multiplying the number of available bytes by the random percentage computed in the previous step. I also convert that result to kilobytes for display purposes.

I am now ready to attempt to allocate memory using the GlobalAlloc function (see Figure 4).The GlobalAlloc function accepts two arguments. The first argument is a flag that specifies one of several ways to allocate:

GMEM_FIXED    = 0x0000 Allocates fixed memory.
GMEM_MOVEABLE = 0x0002 Allocates movable memory.
GMEM_ZEROINIT = 0x0040 Initializes memory contents to 0.

GPTR = 0x0040 Combines GMEM_FIXED and GMEM_ZEROINIT.
GHND = 0x0042 Combines GMEM_MOVEABLE and GMEM_ZEROINIT.
Here I use GPTR = 0x0040 in order to allocate fixed memory that is initialized to zero.

Figure 4 Allocate Memory (Close Figure 4)
// allocate memory
uint GPTR = 0x0040; // Combines GMEM_FIXED and GMEM_ZEROINIT
IntPtr p = GlobalAlloc(GPTR, numBytesToEatInBytes);
if (p == IntPtr.Zero)
{
  string time = elapsedTime.Hours.ToString("00") + ":" +
         elapsedTime.Minutes.ToString("00") + ":" +
         elapsedTime.Seconds.ToString("00");
  Console.Write("{0,8}", time);

  Console.WriteLine("   GlobalAlloc() failed.  Trying again");
  --cycle;
  continue;
}

The second argument to GlobalAlloc is the number of bytes to allocate. If the allocation is successful, GlobalAlloc returns an IntPtr that points to the first address of the allocated memory. If the allocation fails, the return value is IntPtr.Zero (roughly equivalent to unmanaged code null). The allocation can fail for several reasons; in fact, I would expect the allocation to fail sometimes under conditions of stress. So I do not throw an exception, I merely print a warning message to try again. The GlobalAlloc function allocates the specified number of bytes from the heap.

When dealing with memory allocation there are three components to be aware of: physical memory, the page file, and virtual memory. A program can execute an instruction only when the instruction is in RAM. RAM has some fixed size; 512MB is a typical amount on a fairly recent desktop machine. If a program is smaller than the amount of RAM, then the entire program may be loaded into memory. The idea of virtual memory is to allow programs that are larger than RAM to run. A page file is used by the operating system to swap code in and out of memory when necessary. The default original page file size is 1.5 times the size of RAM, but the page file can increase if necessary. The maximum size of virtual memory varies, but 2GB is typical on a 32-bit version of Windows®.

After allocating memory, I need to get an updated memory status so I can display various diagnostic data:

// get current memory status
GlobalMemoryStatus(ref memStatusBuffer);
uint availablePhysicalInKB =
 memStatusBuffer.dwAvailPhys / (uint)basicUnit;
uint availableVirtualInKB =
 memStatusBuffer.dwAvailVirtual / (uint)basicUnit;
numBytesToEatInBytes =
 (uint)(randomPct * memStatusBuffer.dwAvailVirtual);
numBytesToEatInKB = numBytesToEatInBytes / basicUnit;
I call GlobalMemoryStatus again. This time I compute the available amount of physical memory in kilobytes, as I do for virtual memory. Then I compute the memory to eat in both bytes and kilobytes, just for display purposes.

Most of the lines of code in EatMem are dedicated to output display. I begin by printing the current elapsed time:

// print time
string time = elapsedTime.Hours.ToString("00") + ":" +
       elapsedTime.Minutes.ToString("00") + ":" +
       elapsedTime.Seconds.ToString("00");
Console.Write("{0,8}", time);
Then I print the available amounts of physical and virtual memory, and the number of bytes to allocate:
// print the memory numbers
Console.Write("{0,12} {1,12} {2,12}",
              availablePhysicalInKB + " KB",
              availableVirtualInKB + " KB",
              numBytesToEatInKB + " KB");

Notice that for simplicity, the amount of allocated memory I display is based on the current memory status rather than on the memory status when GlobalAlloc was actually called. Next I display the percentage of available memory eaten and a pipe character to act as the left margin for the status bar display:

// print the current alooc percent number
Console.Write("   " +  (int)(randomPct*100.0));
Console.Write("|");

Now to display the status bars I use an old trick:

// do the bars
uint totalNumBars = 20;
uint eachBarIs = availableVirtualInKB / totalNumBars;
uint barsToPrint = numBytesToEatInKB / eachBarIs;

string bars = new string('#', (int)barsToPrint);
string spaces = new string('_', 20-(int)barsToPrint);

Console.Write(bars);
Console.Write(spaces);
Console.WriteLine("|");

Here I have decided to display the status bars as 20 characters. If I have 20 characters, I can figure the width that each character represents by dividing the amount of available virtual memory by 20. The status bar will consist of some characters plus some blanks. The number of characters will be the number of kilobytes allocated divided by the width of each bar. The number of spaces will be 20 minus the number of characters. In the previous code, I use the '#' character. I will explain how to print a fancy colored bar in the next section.

After all the display work is done, I need to halt the thread of execution for a while so that the system under test can be exercised by the tests that I assume are running at this time. (Remember, the whole idea of a stressor is to set up a machine state where tests can run under conditions of reduced resources.) Here I pause for an arbitrary three seconds and then free up the allocated memory:

// pause to give app under test stress time
System.Threading.Thread.Sleep(3000);

// free memory
IntPtr freeResult = GlobalFree(p);
if (freeResult != IntPtr.Zero)
  throw new Exception("GlobalFree() failed");

The GlobalFree function accepts a pointer to memory that was returned by a call to the GlobalAlloc function. It attempts to free up that memory. If the free operation succeeds, GlobalFree returns null/IntPtr.Zero. If the operation fails, GlobalFree returns the input argument. In terms of the stressor, if GlobalFree fails I have a serious problem and need to throw a fatal exception.

The EatMem tool finishes up by displaying an information bar at the end of each of the eight auxiliary for loop cycles, and a "done" message after the main while loop terminates when the current time exceeds the specified run time.

Cool Console Output

When writing command shell testing tools, appropriate use of text color can make your output easier to read and interpret. For example, EatMem uses green for memory numbers, red for error messages, and white blank space characters to create the memory allocation status bars. My colleagues tell me I went a bit overboard with the additional colors here. The trick to printing console text in color in a Microsoft® .NET Framework 1.1 environment is to use the Win32 functions GetStdHndle and SetConsoleTextAttribute. You declare them like this:

[DllImport("kernel32.dll")]
extern static IntPtr GetStdHandle (int nStdHandle);

[DllImport("kernel32.dll")]
extern static bool SetConsoleTextAttribute(
    IntPtr hConsoleOutput, int wAttributes);

To set a text color, you first call GetStdHandle, which will return a handle to Standard Output. Then you call SetConsoleTextAttribute to specify the text color. This code will print "Hello" in pale yellow followed by "Bye" in pale red:

const int STD_OUTPUT_HANDLE = -11;
IntPtr hStdOut = GetStdHandle(STD_OUTPUT_HANDLE);

SetConsoleTextAttribute(hStdOut, 0x0006);
Console.WriteLine("Hello");
SetConsoleTextAttribute(hStdOut, 0x0004);
Console.WriteLine("Bye");

The value -11 is a magic constant that specifies Standard Output. The constants 0x0006 and 0x0004 are pale yellow and red, respectively. The specified text color stays in effect until a new call to SetConsoleTextAtribute is made. With console output, the "pale" versions of color constants are usually too dull for text. You can intensify any text color constant by logical OR'ing it with the value 0x0008. So this code will print "Howdy Dowdy" in bright yellow:

SetConsoleTextAttribute(hStdOut, 0x0006 | 0x0008);
Console.Write("Howdy Dowdy");

Because dealing with hex constants is a bit annoying, it is common to place symbolic constants in your tool. Here are the ones I like to use with command shell programs:

const int BLACK = 0x0000;
const int BLUE = 0x0001 | 0x0008; // bright blue
const int GREEN = 0x0002 | 0x0008; // bright green
const int CYAN = 0x0003 | 0x0008; // etc.
const int RED = 0x0004 | 0x0008;
const int MAGENTA = 0x0005 | 0x0008;
const int YELLOW = 0x0006 | 0x0008;
const int WHITE = 0x0007 | 0x0008;

In addition to specifying text colors, you can specify background colors. To do this you logical OR the text color with another constant representing the background color. For example, this code will set the text context to bright yellow (0x0006 | 0x0008) on a pale red background (0x0040):

SetConsoleTextAttribute(hStdOut, 0x0006 | 0x0008 | 0x0040);

In general, I like to use only bright white, pale white (gray), and black as my backgrounds. The following shows the constants I used in EatMem:

const int BGWHITE = 0x0070 | 0x0080;
const int BGBLACK = 0x0000 | 0x0000;
const int BGGRAY = 0x0070;

With this code you can do a lot of clever color formatting. Here is how I print the memory status bars in EatMem:

string bars = new string('_', (int)barsToPrint);
string spaces = new string('_', 20-(int)barsToPrint);
SetConsoleTextAttribute(hStdOut, BLACK | BGWHITE);
Console.Write(bars);

SetConsoleTextAttribute(hStdOut, WHITE | BGBLACK);
Console.Write(spaces);

SetConsoleTextAttribute(hStdOut, GREEN | BGBLACK);
Console.WriteLine("|");

By printing blank spaces as black text against a white background, you can produce what appears to be a white bar. Then if you switch to printing white text against a black background, you will produce a black background. I use the underscore character to produce a line under each bar.

The Console class in the .NET Framework 2.0 has greatly enhanced support for fancy output. If you are working in a .NET Framework 2.0 environment, using the new Console class is a much better approach than using the P/Invoke mechanism. The Console class contains methods and properties to get or set the size of the screen buffer, console window, and cursor; to change the position of the console window and cursor; to move or clear data in the screen buffer; to change foreground and background colors; to change the text displayed in the console title bar; and to play the sound of a beep.

Extending the Tool

The EatMem tool I've presented here can be used as is, but was specifically designed so you can easily modify it. Let's look at a few of the ways you might want to extend or enhance the basic version of EatMem.

The most significant possibility for modification involves the technique used to allocate memory. EatMem uses the Win32 GlobalAlloc and GlobalFree functions, but there are several other memory allocation functions you can use. One possibility is to use the Marshal.AllocHGlobal and Marshal.FreeHGlobal managed methods; they are managed wrappers around LocalAlloc and LocalFree. The AllocHGlobal method is somewhat easier to use than GlobalFree though not quite as flexible. But if you want to avoid using the P/Invoke mechanism directly, using AllocHGlobal is your solution.

Another alternative to using GlobalAlloc and GlobalFree is to use the VirtualAlloc and VirtualFree Win32 functions. The VirtualAlloc function reserves or commits a region of pages in the virtual address space of the calling process. For an example of VirtualAlloc, see Figure 5.

Figure 5 Using VirtualAlloc (Close Figure 5)

P/Invoke Declarations
[DllImport("kernel32.dll")]
extern static  IntPtr VirtualAlloc(IntPtr lpAddress,
  uint dwSize, uint flAllocationType, uint flProtect);
// LPVOID VirtualAlloc(LPVOID lpAddress, SIZE_T dwSize,
//   DWORD flAllocationType, DWORD flProtect);

[DllImport("kernel32.dll")]
extern static int VirtualFree(IntPtr lpAddress,
  uint dwSize, uint dwFreeType);
// BOOL VirtualFree(LPVOID lpAddress, SIZE_T dwSize,
//   DWORD dwFreeType);

Allocating Memory
// VirtualAlloc
uint MEM_RESERVE = 0x1000;
uint MEM_COMMIT = 0x2000;
uint PAGE_READWRITE = 0x04;
IntPtr p = VirtualAlloc(IntPtr.Zero, numBytesToEatInBytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
if (p == IntPtr.Zero)
{
  string time = elapsedTime.Hours.ToString("00") + ":" +
         elapsedTime.Minutes.ToString("00") + ":" +
         elapsedTime.Seconds.ToString("00");
  SetConsoleTextAttribute(hStdOut, CYAN | BGBLACK);
  Console.Write("{0,8}", time);
  SetConsoleTextAttribute(hStdOut, RED | BGBLACK);
  Console.WriteLine("   VirtualAlloc() failed.  Trying again");
  --cycle;
  continue;
}

Freeing Memory
uint MEM_RELEASE = 0x8000;
int freeResult = VirtualFree(p, 0, MEM_RELEASE);
if (freeResult == 0)
  throw new Exception("VirtualFree() failed !");

An enhancement you might want to consider adding to the basic version of EatMem is to protect yourself against failed attempts to allocate memory. As written, if a memory allocation attempt fails, EatMem simply prints an error message and tries again. This could lead to a series of failed attempts that might go unnoticed. You could easily add a counter that tracks the number of consecutive failed allocation attempts, and if the counter exceeds some threshold value, you could throw an exception or dynamically reduce the random maximum percentage (currently hardcoded at 70 percent) of memory attempted to allocate. A possible variation on this enhancement is to track the percentage of failed memory allocation attempts and adjust stressor parameters accordingly.

If you are in a .NET Framework 2.0 environment and do not want to use the managed AllocHGlobal because you need the flexibility of the Win32 GlobalAlloc, you can take advantage of the new SafeHandles namespace, which contains classes that provide functionality supporting file and operating system handles. You could create a SafeHandle-derived class for GlobalAlloc, which would look something like Figure 6.

Figure 6 Using the SafeHandles Namespace (Close Figure 6)
[SecurityPermission(SecurityAction.LinkDemand, UnmanagedCode=true)]
public sealed class SafeGlobalAllocHandle :
  SafeHandleZeroOrMinusOneIsInvalid
{
  [DllImport("kernel32.dll")]
  public static extern SafeGlobalAllocHandle GlobalAlloc(uint uFlags,
    IntPtr dwBytes);

  private SafeGlobalAllocHandle() : base(true) { }

  protected override bool ReleaseHandle()
  {
    return GlobalFree(handle) == IntPtr.Zero;
  }

  [SuppressUnmanagedCodeSecurity]
  [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
  [DllImport("kernel32.dll", SetLastError=true)]
  private static extern IntPtr GlobalFree(IntPtr handle);
}

This approach provides several benefits. You can use a using statement to make it easy to clean up the memory in the event of an exception being thrown. For example:

using(SafeGlobalAllocHandle mem =
   SafeGlobalAllocHandle.GlobalAlloc(0, numBytes))
{
  if (mem.IsInvalid) { ... }
  else { ... }
}

This approach makes the system much more reliable, which can be particularly important in situations where EatMem stressor code is copied directly into an application or test harness that is prone to asynchronous failures.

Some of my colleagues have randomized the sleep time and memory allocation pattern. As written, EatMem sleeps a constant 3000 ms each time through the auxiliary for loop. Specifying a random sleep time gives you a more representative background memory consumption in some testing situations. Instead of EatMem simply asking for a block of memory, you can ask for the memory in increments. The key to this approach is to maintain an ArrayList collection that holds the incremental allocation IntPtr objects so you can incrementally free memory as you need it by iterating through the collection.

Aside from these relatively major changes, I'm sure you can think of many modifications, such as parameterizing the random range of memory allocation, fetching additional memory status information using the GlobalMemoryStatus function, and adding additional error-checking. Note that I decided to strip away most of my error handling for brevity in this column.

Wrapping It Up

In addition to using the memory stressor presented here, you should also stress your system by reducing CPU availability and reducing disk space. Those stressors are similar to memory reduction, but there are a few additional tricks involved that I'll investigate in a future Test Run column.

In addition, other types of stress testing you might want to explore include running test automation over an extended period of time under normal conditions. Stress analysis, or subjecting your application under test to increasingly large reductions in memory, CPU availability, or disk space until the system fails, is another variation. The idea in this case is to determine functional boundaries of the system under test. Still another type of stress test, especially suited to Web apps, is called "load testing." This usually refers to subjecting the Web application and/or Web server to heavy user loads. So as you see, there's a lot to discover in the area of stress testing. Stay tuned to future columns for more.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com. Thanks to Lutz Roeder for his help with this column.


Extreme ASP.NET: Keeping secrets in ASP.NET 2.0.

Extreme ASP.NET
Keeping secrets in ASP.NET 2.0.
Rob Howard

Code download available at: ExtremeASPNET05.exe (118KB)
Contents
Wait, You Can’t Do That
ASP.NET 1.1—A Better Confidant
ASP.NET 2.0 Secrets
Tell Smarter Secrets
In Closing

Storing data securely in a configuration system is not an easy problem to solve. While I was on the ASP.NET team, this particular feature, secure connection string storage, looked as if it wouldn’t get done. A whole host of problems surrounding it, such as key storage, were in the way. The good news is that not only was it eventually completed, but it has become part of the powerful new set of APIs in ASP.NET 2.0 that allow you to manage the ASP.NET configuration file programmatically.

Before diving into ASP.NET 2.0, though, let’s take a look at the problem and various solutions in ASP.NET 1.x. If you have used ASP.NET for any time at all, you are no doubt aware of the recommendation to store shared settings in the web.config file. For example, rather than specifying your connection string each time you create a new database connection, you store the string in the ASP.NET configuration file’s <appSettings /> section. The connection string is then accessible through the ConfigurationSettings.AppSettings property. Here’s an example <appSettings/> section:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="ConnectionString"
         value="server=.;database=demo;uid=db;pwd=*u%a" />
  </appSettings>
</configuration>
Then, whenever you need to change the connection string you simply open the file, make your changes, and you’re finished.

This feature struck a chord with many developers moving from classic ASP to ASP.NET where most global values were stored as application variables. In fact, for ASP.NET 1.x, it is a recommended practice to store connection strings in <appSettings/>. It’s also worth noting that you can store other common data in <appSettings/>, too, including LDAP paths, common application settings, and other miscellaneous data your application needs. The goal of <appSettings/> is to simplify the writing of custom configuration section handlers—a more advanced technique of interacting with the ASP.NET configuration system. Custom configuration section handlers allow you to author and process your own XML sections within the configuration system.

You may have noticed that the contents stored in <appSettings/> are not encrypted but rather are stored as plain text. The same is true of the <sessionState/> section, which enables the out-of-process storage of Session data. One of the storage options is to use SQL Server™ and have the credentials stored in plain text in the <sessionState/> configuration slot.

Wait, You Can’t Do That

A shortcoming of storing the connection string in <appSettings/> was that this was not secure because the file was neither encrypted nor compiled. Not that compiling the configuration information would do anything to improve security; DVD manufacturers used key-based encryption to protect intellectual property, only to have DVD player software vendors store the decryption key in their compiled code. A little code spelunking by some hackers easily turned up the decryption key. It’s also common to read blog posts about individuals trying to figure out how something works in the Microsoft® .NET Framework, when someone makes the suggestion: "Just use Reflector."

ASP.NET 1.1—A Better Confidant

In ASP.NET 1.0, it was not possible to securely store the connection string in the configuration file without extra custom code. The ASP.NET team addressed this problem in ASP.NET 1.1 by enabling encryption for several configuration entries. The solution was accomplished through the Windows Data Protection APIs (DPAPI), resulting in the ability to encrypt the following configuration entries:

<identity/> Used to store the Windows® identity of the ASP.NET worker process for impersonation.

<processModel/> Used to control the Windows account that the ASP.NET worker process executed under. Not used in IIS 6.0 (see the following note).

<sessionState/> Specifically contains the stateConnectionString and sqlConnectionString attributes used to control how ASP.NET authenticated to the out-of-process state servers.

It’s important to note that IIS 6.0, the Web server included with Windows Server™ 2003, provides its own worker process management subsystem to which ASP.NET defers. Thus some worker process settings found in the configuration system are not used when ASP.NET is hosted by IIS 6.0.

A tool called aspnet_setreg.exe was provided with ASP.NET 1.1 to encrypt data in the configuration files and store the decryption key in a Windows registry entry. The resulting registry key had an Access Control List (ACL) configured to limit the Windows accounts that could access the key. This technique is explained, in detail, in Knowledge Base article Q329290, "How to use the ASP.NET Utility to Encrypt Credentials and Session State Connection Strings".

However, this solution had some shortcomings, too. It broke what the ASP.NET team lovingly referred to as xcopy deployment, which allowed an ASP.NET application to be deployed without access to the server. With the technique outlined earlier, the developer or system administrator had to have local machine access to run the command-line tool to encrypt the configuration data and store the key in the registry.

ASP.NET 2.0 Secrets

This brings me to all of the work the team did to address this problem for ASP.NET 2.0. Yet again, there is a command-line tool for managing the encryption of configuration data: aspnet_regiis.exe. Aspnet_regiis.exe existed in previous versions of ASP.NET and was used primarily for manually registering ASP.NET with IIS. For example, it was used to add the aspnet_isapi.dll to IIS as well as configure the script directories that ASP.NET applications used. You can find this tool in the \Windows\Microsoft.NET\Framework\version#\ directory.

Using aspnet_regiis.exe for encrypting configuration sections is as cryptic as the results it generates! Take a look at the results of executing aspnet_regiis.exe /help in Figure 1 and you’ll see what I mean.

Figure 1 Encrypting Configuration Data
Figure 1 Encrypting Configuration Data

As you can see, unless you are intimately familiar with security terminology, you’ll be quickly overwhelmed by the number of options and various settings. Unfortunately this is a case of powerful capabilities and confusing tools. Fortunately, the Microsoft patterns & practices team wrote a great in-depth article, "How To: Encrypt Configuration Sections in ASP.NET 2.0 Using DPAPI," that walks you through aspnet_regiis.exe.

Instead of examining how to use aspnet_regiis.exe, let’s look at an example ASP.NET page that uses the new configuration APIs to accomplish the encryption of configuration sections. The single page, ConnectionEncryption.aspx (available from the MSDN®Magazine Web site), contains a GridView which is populated with a list of all configuration sections. It looks like Figure 2.

Figure 2 Using the New Configuration APIs
Figure 2 Using the New Configuration APIs

Before examining how the internals of ConnectionEncryption.aspx work, I’ll look at the results from this page. But first a warning: use of this tool requires that the process hosting ASP.NET has the permission to write to the web.config file of the current application. By default, an ASP.NET application running in IIS will not have the necessary permissions. However, an application hosted in the ASP.NET Development Web Server runs with the permission set of the logged-on user. All uses of this tool you see here are shown within the ASP.NET Development Web Server. It is recommended that you do not change permission settings for IIS without a thorough understanding of the effects.

Following is a sample entry found in the new <connectionStrings> section of web.config used to store connection strings. The <connectionStrings> section is nearly identical to <appSettings> and is now the recommended place to store connection string data as there are new APIs specifically for working with connection strings sprinkled throughout ASP.NET:

<connectionStrings>
  <add name="Northwind" providerName="System.Data.SqlClient"
      connectionString="Server=localhost;Integrated
          Security=True;Database=Northwind" />
</connectionStrings>
Note, in this case Windows authentication is still used for connecting to the database.

Clicking on the Encrypt link in ConnectionEncryption.aspx changes the connection string value in web.config to something like that shown in Figure 3. Once encrypted, the ConnectionEncryption.aspx page reports the state of the entry as encrypted (the link changes to "Decrypt", as you see in Figure 4).

Figure 3 New Encrypted Connection String (Close Figure 3)
<connectionStrings>
  <EncryptedData>
   <CipherData>
<CipherValue>AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAALJZXjMW9c0G4r/2N3hAQAAAACAAAAAAADZgAAqAAAABAAAAD7l2fNIXMFeuAohZ1FjRP+AAAAAASAAACgAAAAEA
AAALK0hmPPfWgWJRVr69wbkSWAAQAAr+idwrf3qkg5NPzXa4HnpbeMy1ngAQyn5GlIO99JT40
FC8bY3keUqVkDlUhwj1WrHV/i7+tMbkj5MJPvRV4z33tp4d6H1PVsMVpjsbmRb/9YdY50kRCN
tWxWye/chGbDDoNePcQhdidPmdH5Yf2lDdpMT3hX1K8JCiWXTXa7sk0/Vu6oQARZpqWMxXJAU
GGY4G/WNAUchQ0Rw3U09ygn3aykMb4loqBwvhdlW7EuuGB3OXCrZ2SfrGGwiw0CD3BWHoOdKV
2yH6TQYUED7yLYHPATEUyj42T2H1J0B6SaQLIA9N61lThZQWdLbeWCpBAkOcUpivO9jhPJtB2
zNs3RnCaPOKyhpX650nBH/qRzO8D5B7PmJ4RHFN97ePoIZ0nz5LnOWUGWra+kQBD3/F9Dl9XO
al6pVvO/EEXbJZFhIobAzynjoec1msAPcIWxzx3PypoLUQEjMHeEkk1TtKiZbmznnPu8lPzlJ
SECfbyajQLTQOtetxCmpOKdhEFgsnHRFAAAAEcgWPEHOEF32JkblJNOzL/Sh4ux
</CipherValue>
   </CipherData>
  </EncryptedData>
</connectionStrings>
Figure 4 The New Page
Figure 4 The New Page

Now that you have seen the page in action, let’s take a look at the code. On Page_Load, the GridView is populated when the new ASP.NET 2.0 WebConfigurationManager class retrieves an instance of the Configuration class for the local path on line 18 in ConnectionEncryption.aspx.cs:

Configuration config =
  WebConfigurationManager.OpenWebConfiguration(Request.ApplicationPath);

An ArrayList is populated with data retrieved from the configuration variable and is then bound to the GridView. A majority of the other methods in the source (see the code download) are used for business logic rules around the actual data binding operation, such as determining the scope and state of a section (encrypted or unencrypted). The magic happens in the GridView1_RowCommand event when the link for Encrypt or Decrypt is clicked. When the GridViewCommandEvent.CommandName value is "Encrypt", the following code is executed:

section.SectionInformation.ProtectSection(
    "DataProtectionConfigurationProvider");
config.Save();
When the GridViewCommandEvent.CommandName value is "Decrypt", the following code is executed:
section.SectionInformation.UnprotectSection();
config.Save();

Thus the actual work of encrypting and decrypting the data is handed off to a provider. One built-in provider is the DataProtectionConfigurationProvider. It uses the built-in DPAPI to store secure data, which is the same one used by the ASP.NET command-line tool.

You should note that when encrypting data, a provider can be named, but when decrypting it’s not required (if null or an empty string is provided to ProtectSection, the default provider specified in the configProtectedData section of the configuration file will be used). This is because the APIs write another entry into the configuration file that names the encryption protection provider used:

   <connectionStrings configProtectionProvider=
           "DataProtectionConfigurationProvider">
       <EncryptedData>...</EncryptedData>
   </connectionStrings>
This configuration entry is used not only by APIs to determine how to decrypt sections, but also by ASP.NET internally when it needs to read values, such as the connection string, into memory which first must be decrypted.

The great news about the new encryption functionality of ASP.NET 2.0 is that not only can you encrypt many of the built-in configuration sections, but you can also write custom encryption providers! Providers are an awesome new extensibility model in ASP.NET 2.0 that enable developers to provide their own implementation of core functionality such as membership, personalization, and so forth. Additionally, due to the way Configuration Encryption was implemented, custom configuration sections can easily be encrypted as well, so protection of data is not limited to a few configuration sections as was the case in ASP.NET 1.1.

Tell Smarter Secrets

A column on encrypting secrets wouldn’t be complete without several warnings. First, if you can avoid secrets, do so. If you’re using SQL Server and you don’t want to store sensitive connection string information in the configuration system, use integrated authentication with SQL Server. Windows authentication will be used to connect from your application server to the database. With this technique, the connection to SQL Server is authenticated and authorized through Windows directly. Using Windows authentication, SQL Server requests a token from the authenticating server (whether local or remote), which contains a security identifier (SID) for the user along with other information and is compared with a list maintained by SQL Server to determine if access is granted or denied. No passwords or usernames are stored in the configuration file.

Another thing to keep in mind is that just because data in the configuration file is stored securely doesn’t mean you’re safe from all kinds of attacks. Any attacker worth his salt (security pun) could do plenty of harm if he got far enough to access your web.config file on your server. An attacker who obtained access to the system to manipulate the web.config could perform other actions to manipulate the database without requiring knowledge of the connection string. Thus, securely storing your connection string or other application data is simply one line of defense. Another effective defense is using stored procedures and placing more fine-grained control over what the account accessing your database has permissions to. Case in point, an encrypted connection string is worthless if you have SQL injection vulnerabilities (for more on SQL injection attacks, see msdn.microsoft.com/msdnmag/issues/04/09/SQLInjection).

Having built and run a number of high-visibility sites, such as www.asp.net, I’ve seen my fair share of attack attempts. One common attribute is that the attack is often completely unrelated to the initial vulnerability. By not attacking through the original vulnerability, the attacker is attempting to protect the backdoor created through the original vulnerability.

In Closing

Protecting secrets in your ASP.NET applications is possible in any version of ASP.NET, though it’s made easier in ASP.NET 1.1, and much more so in ASP.NET 2.0. With ASP.NET 2.0, rather than configuration encryption being an afterthought, it was something that was built into the new configuration APIs. With ASP.NET 2.0, not only can you use the aspnet_regiis.exe tool to encrypt configuration sections, but you can also write your own custom code—along with custom providers if you wish—to encrypt and decrypt configuration data.

For more information on storing secrets in ASP.NET 1.1, see "Security Considerations for Hosting ASP.NET version 1.1."

Send your questions and comments for Rob to xtrmasp@microsoft.com.

Rob Howard is a founder of Telligent Systems, specializing in high-performance Web apps, knowledge management, and collaboration systems. Previously, Rob was employed by Microsoft where he helped design the infrastructure features of ASP.NET 1.0, 1.1, and 2.0. You can contact Rob at rhoward@telligentsystems.com.


Security Briefs: Step-by-Step Guide to InfoCard

Security Briefs
Step-by-Step Guide to InfoCard
Keith Brown

Code download available at: SecurityBriefs05.exe (125KB)
Contents
InfoCard Interactions and Protocols
Considerations for Relying Parties
The Notepad Service
InfoCard-Enabling the Notepad Service
Exploring InfoCard Without WCF
Winding Down

In my April 2006 column I began a discussion of InfoCard, the upcoming identity metasystem, which is being prepared for release in the Windows Vista™ timeframe. If you haven’t read that column, you should definitely start there because I’m going to assume you’re familiar with the basics I covered. This time around, I’ll go deeper into how the identity metasystem works, and write a service that relies on InfoCard for authentication.

A number of you sent me questions after reading the April column, so I’d like to clarify some things before drilling deeper. Technically, InfoCard is only one part of the identity metasystem: it’s the identity selector—the front-end GUI that’s displayed to the user. I tend to use the term InfoCard to refer to the metasystem as a whole simply because it’s a heck of a lot easier to say, and I think it sounds more concrete.

The identity selector being implemented by Microsoft will first surface in the Windows Vista operating system. If you saw any PDC 2005 talks, you most likely saw the identity selector in action. But InfoCard will be available on earlier versions of Windows® as well, including Windows XP. And Microsoft has made it clear that it wants to work with other operating system vendors to provide tips on designing secure identity selectors for their platforms as well. If InfoCard is to be successful, it needs to be very widely deployed, and not just on Microsoft® operating systems.

Some of you also asked for clarification about Security Assertion Markup Language (SAML), which is a standard that specifies the shape of a credential (or token, in WS-* terminology), as well as a protocol for passing these credentials around. InfoCard can use many types of tokens, including SAML tokens; self-issued cards happen to use SAML 1.1 tokens, for example. While SAML tokens are fairly common in InfoCard, WS-Trust is the protocol that is used to obtain and exchange tokens. The SAML protocol is not used by InfoCard.

What about folks using the Java language to build Web services? Back in August 2005, Ping Identity, a major player in the SAML space, released a Java toolkit for writing server-side applications that interact with InfoCard. It was released as a work in progress, but when it is finished, according to the toolkit documentation, it should be possible to build Web services (on top of Axis) and servlet-based Web applications that can request and handle all kinds of InfoCards from InfoCard-enabled clients.

And finally, InfoCard is not the "Son of Passport" that some in the media seem to fear. Passport is a single identity system and, as such, naturally fits into the metasystem as one of many technologies available to InfoCard users. But unlike the old days when Passport was being positioned as the only identity system you’d ever need on the Internet, today Microsoft is proposing a metasystem that allows the user to choose the identity technologies that make sense for her. Passport works well for accessing Microsoft properties like MSN®, but as an InfoCard user, that’s the only context in which I’ll be using Passport. My friends will tell you that I was never a big fan of Passport, but I am really excited about InfoCard, and you should be too!

InfoCard Interactions and Protocols

In the April column I talked about three InfoCard players—the subject, relying party, and identity provider. Most Web developers who need to add support for InfoCard to their Web services and applications will be acting as a relying party, so I’ll focus on that role here.

Figure 1 is an interaction diagram from "A Guide to Integrating with InfoCard," which you can get from the Microsoft download center. It illustrates how a user can authenticate with a service using InfoCard. In the diagram, you can see the identity provider (the relying party), which is a Web service, and the subject, which is the user. The client app the user is running is also shown for clarity.

Figure 1 InfoCard Interaction
Figure 1 InfoCard Interaction

The first thing to notice is that in order to use InfoCard the user must first enroll with an identity provider. That’s the out-of-band step shown by a dashed line in the figure. It could be as simple as creating a self-issued card on her own machine using the InfoCard GUI, in which case the identity provider is right on the user’s machine. Alternatively, the user might obtain an InfoCard from a standalone identity provider. A newly issued InfoCard is an XML document that can be transmitted to the user via e-mail, Web download, and so forth. There is no required protocol for how the identity provider authenticates the user or issues the InfoCard.

Once the user has one or more cards, the required protocols move into action. When the user runs an InfoCard-enabled application that makes a request of an InfoCard-enabled Web service, the first step is for the app to determine what type of identity and channel security the service requires. This is shown as the first round-trip to the service (Steps 2 and 3 in Figure 1). The protocol used here is WS-MetadataExchange, often referred to as WS-MEX, or just MEX.

The application now invokes the identity selector (Step 4), passing in the policy retrieved from the service. The identity selector pops up a secure GUI showing the user a collection of cards that would work for the given policy (Step 5). This is primarily governed by the type of token and the type of claims required by the service.

If the user has never used this service before, InfoCard will display the service’s and issuer’s certificate logos, giving the user a chance to visually verify that he’s talking to the right Web service so he doesn’t give away identity details to an imposter.

Once the user selects a card to send (Step 6), the identity selector initiates a behind-the-scenes conversation with the identity provider that issued the card. First there’s a MEX exchange (Steps 7 and 8), where the provider’s security policy is discovered so that the exchange can be properly secured. This also tells InfoCard how the user is supposed to prove his identity to the identity provider. For example, if the provider requires a password, the identity selector will prompt the user for his password. Now a WS-Trust request is sent to the identity provider (Step 9). This request includes the user’s proof, such as the password as in the example. It also includes the ID of the card, which allows the provider to figure out which account is being accessed. Also included is a list of requested claims—the identity details that the Web service wants.

Assuming the identity provider can validate the user’s password that was sent, it will now construct a security token for the user to send to the service. This token will contain all the claims the service requested. If the service requested the user’s street address, that address will be included in the token. Remember that the original list of cards shown to the user included only those that support the claims being requested. Note that a password is just one of several ways an identity provider can authenticate the user. Other techniques include using a certificate, Kerberos credentials, or a self-issued InfoCard.

WS-Trust is the core protocol around which InfoCard is built. It is basically a credential exchange (or more technically, a security token exchange) protocol, and is implemented by what is called a security token service (STS). The most common type of STS is an identity provider like the one shown in Figure 1. In Steps 9 and 10, the user’s password is being exchanged for a signed set of claims from the identity provider’s STS.

Now the identity provider signs and encrypts the token and sends it back to the InfoCard identity selector (Step 10) along with some key material that can be used by the client application to prove that it obtained a legitimate token from the identity provider. This key material is known as proof of possession.

When the identity selector receives the token and proof of possession, it decrypts the token and shows the user one final GUI. This allows the user to see exactly what identity data is going to be sent to the service. If the service requested the user’s street address, the actual street address will be displayed to the user. Figure 1 doesn’t show this particular interaction, but it’s an important one for the user. Once the user gives his final blessing by pressing Send, the identity selector returns to the application (finally!) giving it the token and proof of possession it needs to authenticate with the target service. As you can see, there’s a lot going on under the hood when you select an InfoCard. Fortunately, if you’re a client or service developer, you don’t have to worry about building any of this infrastructure.

At this point the application makes its request to the target service using WS-Security to send the token along with a signature formed via the proof-of-possession key material (Step 11). The user has now been authenticated and the service has the claims it requested and can use those claims to make authorization decisions and carry out the user’s requests.

It’s possible and even likely that WS-SecureConversation might be used at this point to set up an efficient secure channel for an ongoing conversation between the application and the service, but that’s beyond the scope of the InfoCard specification.

Considerations for Relying Parties

As a relying party, the first thing you’re going to need is a certificate to identify your organization. This is the beacon type identity that I discussed in my previous column. A large percentage of secure Web services today already use certificates to enable secure sockets layer (SSL), so this shouldn’t be a significant barrier to adoption of the technology.

What is a bit new is that this certificate needs a name that represents your organization, not your URL (as in an SSL certificate). It also should include logo images for both the issuer and subject, following the guidelines in RFC 3709. Each image isn’t physically embedded in the certificate; rather, it’s linked to avoid making certificates even bigger than they already are. The logotype extension includes a URL that points to an image on the Web. It also includes a cryptographic hash of that image. The hash is embedded in the certificate to allow the program displaying the logo to ensure that the image hasn’t been modified since the certificate was issued.

Another major consideration is the type of identity information (claims) you’ll request of your users. InfoCard has already defined several claim types that will cover the needs of most applications, but this list can be extended by third-party identity providers that serve other types of claims. The current list is shown in Figure 2. To form the full Uniform Resource Identifier (URI) for each claim, append its suffix to schemas.microsoft.com/ws/2005/05/identity/claims. You’ll see these URIs in policies exposed by InfoCard-enabled services. This is the way you specify the identity expectations your service places on its users.

Figure 2 Claim Types Defined by InfoCard Profile (Close Figure 2)

Claim | URI Suffix
First name | givenname
Last name | surname
E-mail address | emailaddress
Street address | streetaddress
Locality name or city | locality
State or province | stateorprovince
Postal code | postalcode
Country | country
Primary/home telephone number | homephone
Secondary/work telephone number | otherphone
Mobile telephone number | mobilephone
Date of birth | dateofbirth
Gender | gender
Private personal identifier | privatepersonalidentifier

Many Web services don’t need any personally identifiable information at all, and you shouldn’t ask for anything you don’t absolutely need. With InfoCard, a great way to start is by simply identifying the user with a Private Personal Identifier (PPID). This is communicated as a base64-encoded byte array. In my own tests this value has been 32 bytes long, but its definition doesn’t include a length, so don’t rely on this. As an example, a self-issued InfoCard will generate a unique PPID for each target service based on the card’s unique ID, the service’s public key, and a random salt value. The PPID alone is not considered personally identifiable information, so it’s a great way to associate your own data with a user while preserving her anonymity. I’ll show an example shortly.

The Notepad Service

Microsoft support for InfoCard rests entirely within Windows Communication Foundation today. It is surprisingly easy to add InfoCard support to a Web service, making that service into a relying party. It’s really just about selecting the appropriate binding, and you can do it all with a configuration file if you want.

Figure 3 shows a Windows Communication Foundation Web service with a contract called INotepad. This allows a user to leave herself a note by sending the SetNote message, and lets her retrieve that note later via the GetNote message.

Figure 3 Notepad Service (Close Figure 3)

The Contract
[ServiceContract]
public interface INotepad {
    [OperationContract] string GetNote();
    [OperationContract] void SetNote(string data);
}
The Service Code
public class Notepad : INotepad {
    static object _lock = new object();
    static string note = "";
    public string GetNote() {
        lock (_lock) return note;
    }
    public void SetNote(string newNote) {
        lock (_lock) note = newNote;
    }
}

class Program {
    static void Main(string[] args) {
        try {
            ServiceHost sh = new ServiceHost(typeof(Notepad),
                        new Uri("http://localhost:8080/notepadService"));
            sh.Open();
            Console.WriteLine(
                "Listening. Press <enter> to quit the service.");
            Console.ReadLine();
            sh.Close();
        }
        catch (Exception x) {
            Console.WriteLine(x);
        }
    }
}
The Configuration File
<configuration>
  <system.serviceModel>
    <services>
      <service type="InfoCardSample.Notepad">
        <endpoint address="http://localhost:4123/notepadService"
          contract="InfoCardSample.INotepad"
      binding="wsHttpBinding"
      bindingConfiguration="myBinding"/>
      </service>
    </services>
    <bindings>
      <wsHttpBinding>
        <binding name="myBinding"/>
      </wsHttpBinding>
    </bindings>
  </system.serviceModel>
</configuration>

Figure 4 shows the corresponding Windows Communication Foundation client code. Because the client dynamically discovers the service’s binding requirements via WS-MEX, there’s no need for a client-side configuration file in this app. If you build and run this code, you’ll quickly see a flaw in the design. It works fine as a scratchpad for a single user, but as soon as two or more users invoke the service, they begin stepping on each other’s notes. By adding InfoCard support, I’ll ensure that each user gets a personal storage area for notes that no other user can see. I’ll do this without requiring user registration or any personal information.

Figure 4 Client Code (Close Figure 4)
using System;
using System.ServiceModel;
using System.ServiceModel.Design;
using Microsoft.InfoCards;
using InfoCardSample;

class Client {
  static void Main(string[] args) {
    try {
        if (1 != args.Length) {
            Console.WriteLine("Usage: client \"string to send\"");
            return;
        }
        string url = "http://localhost:8080/notepadService/mex";
        ChannelFactory<INotepad> factory = findService(url);
        INotepad chn = factory.CreateChannel();

        displayNote(chn.GetNote(), "Old");
        chn.SetNote(args[0]);
        string newNote = chn.GetNote();
        displayNote(chn.GetNote(), "New");

        // Clean up
        factory.Close(); // close the client’s channel
    }
    catch (Exception x) {
        Console.WriteLine(x);
    }
}

  static ChannelFactory<INotepad> findService(string addr) {
    MetadataResolver mexProxy = new MetadataResolver(
                                new EndpointAddress(addr));
    foreach (ServiceEndpoint serviceEndpoint in
                             mexProxy.RetrieveEndpoints()) {
        if (serviceEndpoint.Contract.Name.Equals(
                             typeof(INotepad).Name)) {
            return new ChannelFactory<INotepad>(
                serviceEndpoint.Binding,
                serviceEndpoint.Address);
        }
    }
    throw new Exception(
        string.Format("Couldn’t find INotepad at address {0}", addr));
  }

  private static void displayNote(string note, string desc) {
    Console.WriteLine("{0} note: {1}",
        desc, null == note ? "<<NULL>>" : note);
  }
}

The majority of typing you’ll do will be in the service configuration file. You’ll need to tell Windows Communication Foundation where to find its certificate. You’ll also need to advertise what sort of security token you want from your clients and what claims you expect to see in that token. For this application, I allow my clients to remain anonymous, choosing to rely on a PPID claim. I let clients use the self-issued identity provider, which issues SAML tokens.

It may seem odd to authenticate clients that will remain anonymous, but it’s really no different than the way a typical bulletin board works today. When you create an account, you supply a user name and password. The user name is pretty meaningless; keith3425 doesn’t give the board a clue as to how to get in touch with me. But the password allows the site to ensure that it’s the same person logging in each time.

InfoCard makes this type of service very easy for users. There’s zero registration required; because PPIDs are unique for each site, the user can simply create a single card that she uses at all anonymous sites like this. She doesn’t have to go into that familiar loop of attempting user names until she finds one that hasn’t been used already. You can think of her user name as being replaced by a PPID, which is generated automatically by InfoCard. She also doesn’t have to generate and manage yet another password. A strong key will be generated dynamically to use with the service.

The biggest drawback to using self-issued tokens is that they can be a hassle if the user works on multiple machines. Eventually, a third-party identity provider will be able to centralize the user’s identity so that it can be used from any machine. In the future there may even be portable STS hardware devices that you can carry around and plug into your computer’s USB port. This would make self-issued cards more portable and secure.

InfoCard-Enabling the Notepad Service

Enough talk; let’s do it. The first step in InfoCard-enabling the service is to obtain and install a certificate. I’m going to use the one that came with the InfoCard Resource Kit that shipped back in September 2005. The logos in it are configured to point to GIF files in a virtual directory at localhost/ServiceModelSamples. I’ll supply a batch file to get the certificates and logos set up for you if you’re following along with the code sample.

Figure 5 shows the updated service configuration file. Note that the references to the Fabrikam certificate are required because it’s used to authenticate the service to the client application, allowing the client app and service to negotiate security for their communication channel. Also, it’s used to authenticate the service to the identity provider.

Figure 5 Updated Service Configuration File (Close Figure 5)
<configuration>
  <system.serviceModel>

    <services>
      <service type="InfoCardSample.Notepad"
               behaviorConfiguration="MyServiceBehavior">
        <endpoint address="http://localhost:4123/notepadService"
               contract="InfoCardSample.INotepad"
           binding="wsHttpBinding"
           bindingConfiguration="simpleInfoCardBinding">
          <identity>
            <certificateReference findValue="Fabrikam"
                      storeLocation="LocalMachine"
                      storeName="My"
                      x509FindType="FindBySubjectName" />
          </identity>
        </endpoint>
      </service>
    </services>

    <bindings>
      <wsHttpBinding>
        <binding name="simpleInfoCardBinding">
          <security mode="Message">
            <message clientCredentialType="InfoCard"/>
          </security>
        </binding>
      </wsHttpBinding>
    </bindings>

    <behaviors>
      <behavior name="MyServiceBehavior"
              returnUnknownExceptionsAsFaults="true" >
        <serviceCredentials>
          <serviceCertificate findValue="Fabrikam"
                    storeLocation="LocalMachine"
                    storeName="My"
                    x509FindType="FindBySubjectName" />
        </serviceCredentials>
      </behavior>
    </behaviors>

  </system.serviceModel>
</configuration>

Also note the binding’s security mode. I’m now specifying message-level security, which indicates the need for the WS-Security suite of protocols. The clientCredentialType is specified as InfoCard, which is really just shorthand for a self-issued SAML token that has no personally identifiable information in it, only a PPID. If you want to specify other claims, use the wsFederationBinding instead of wsHttpBinding; this allows you to specify the exact claim types you’re looking for.

Figure 6 shows the updated service code, with only the guts of the Notepad class that changed. You’ll see here that I’m now using a dictionary instead of a single string to store notes. The dictionary is keyed off the base64-encoded PPID. There’s really no reason to bother decoding the PPID into a byte array if a unique string is good enough to identify the user, so my dictionary uses a string PPID as its key.

Figure 6 Updated Service Code (Close Figure 6)
const string CLAIMTYPE_PPID = "http://schemas.microsoft.com/ws/"
    + "2005/05/identity/claims:PrivatePersonalIdentifier";
static Dictionary<string, string> ppidToNotes =
    new Dictionary<string, string>();

string getCallerPPID() {
    AuthorizationContext ctx =
    OperationContext.Current.ServiceSecurityContext.AuthorizationContext;
    foreach (ClaimSet cs in ctx.ClaimSets) {
        foreach (Claim c in cs) {
            if (CLAIMTYPE_PPID == c.ClaimType) {
                return (string)c.Resource;
            }
        }
    }
    throw new Exception("Expected a PPID claim");
}

public string GetNote() {
    string base64EncodedPPID = getCallerPPID();
    lock (ppidToNotes) {
        if (ppidToNotes.ContainsKey(base64EncodedPPID))
            return ppidToNotes[base64EncodedPPID];
        else return null; // new users start with null note
    }
}

public void SetNote(string data) {
    lock (ppidToNotes) {
        string base64EncodedPPID = getCallerPPID();
        ppidToNotes[base64EncodedPPID] = data;
    }
}

Besides the synchronization code, the major new addition is the getCallerPPID function, which shows you how to use the new claims-based authorization feature in Windows Communication Foundation to discover the claims that were sent in the token. This code simply searches for a PPID claim. If there are other claims you requested, such as a user name or e-mail address, you can look for those as well using the URI for the claim types you are interested in.

The client code that looks up the service via WS-MEX also changed a bit. The MEX code doesn’t change, but once the channel factory is created, a behavior is dynamically added that tells the client-side Windows Communication Foundation plumbing to use InfoCard credentials with the service, as well as what certificate to expect for the service. This extra piece of code is here:

ChannelFactory<INotepad> factory = new ChannelFactory<INotepad>(
    serviceEndpoint.Binding, serviceEndpoint.Address);

// indicate that we’re going to use InfoCard
InfoCardClientCredentials creds = new InfoCardClientCredentials();
X509CertificateIdentity cid = (X509CertificateIdentity)
    serviceEndpoint.Address.Identity;
creds.ServiceCertificate.Certificate = cid.Certificates[0];
factory.Description.Behaviors.Add((IChannelBehavior)creds);
return factory;

If you build and run the new service, you’ll notice that on the very first call to the Web service, the identity selector pops up! Once you select a card and submit it, you’ll see the second GUI that shows the actual information that’s going to be sent; click the Information tab to see the personal information that will be sent. In this case, you will see an empty list since none of your personal information is requested. The PPID isn’t really personal information; it’s just an anonymous identifier that doesn’t mean anything to the user anyway, so it’s not shown here. After pressing the Send button, the rest of the client code runs, making a couple more calls to the Web service. You shouldn’t be prompted again while the client application is running though since the Windows Communication Foundation proxy is caching your credentials and doesn’t need to bug you anymore.

It’s pretty amazing to realize what you’re getting for about 10 lines of configuration code and maybe 20 lines of C#. You now have a login system that works over the Web with state-of-the-art authentication protocols. Windows Communication Foundation handles the heavy crypto required for authenticating the user’s credentials. There’s no worry about weak passwords, because you’re not relying on them! The user can create a single InfoCard that works at your site and at other sites; each site no longer needs to ask for a user name and password.

Exploring InfoCard Without WCF

You can play with InfoCard without using Windows Communication Foundation. Earlier I explained how the client application must invoke the InfoCard identity selector, handing it the server’s policy so the selector can display appropriate cards to the user. Well, if you’re using Windows Communication Foundation, the client-side plumbing handles this for you transparently. You don’t have to call InfoCard yourself, but you can if you want to.

If you look in the %WINDIR%\system32 directory, you’ll find a file called infocardapi.dll. A quick dump of its exports shows a function called GetToken. There’s also the managed assembly, Microsoft.Infocards.dll, which has a class called InfoCardClient with a static GetToken function that wraps the unmanaged version. You can build an array of policy elements and call this method directly if you want to pop up the identity selector and let the user choose a card. When the function returns, you’ll have the dereferenced InfoCard in your hands in the form of a security token, which is a great idea if you want to learn more about how the identity selector works.

If you want to simply pop the identity selector so the user can manage his cards, you can do this via the ManageInfoCard method. And if you want to import an InfoCard, you can do so by calling ImportInfoCard. There are no APIs for enumerating or otherwise programmatically manipulating InfoCards; you need to remember that this is something that the user must manage, since it’s his own personal information.

Oh, and one final note. If you’re wondering where the InfoCard identity selector itself lives, you can find it in a service called Microsoft Digital Identity Service. The code behind this service lives in a file called, appropriately enough, infocard.exe.

Winding Down

In this issue, I drilled deeper into InfoCard and showed the code for a relying party and a client. If it’s not already apparent, I’m really excited about the possibilities that InfoCard and the underlying identity metasystem present. Kim Cameron (the chief identity architect at Microsoft I told you about last time) has often said that if we can succeed in building an identity metasystem that respects the laws of identity, we will reach what he calls the "identity big bang," a new era of software which is identity-aware. I’m hopeful that InfoCard will help lead us there.

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is a cofounder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight’s Applied .NET Security course, as well as several books, including The .NET Developer’s Guide to Windows Security, which is available both in print and on the Web. Learn more at www.pluralsight.com/keith.


C++ at Work: Web Version Checking, Adding Sound to an App

C++ At Work
Web Version Checking, Adding Sound to an App
Paul DiLascia

Code download available at: CAtWork05.exe (185KB)

Q In your April 2003 column you described how to implement a class called CWebVersion that checks the software version against a file stored on the Web, in order to prompt the user to upgrade the program if the version is out of date. Your implementation uses FTP to download the file, but the ISP that hosts my site doesn't allow FTP connections unless I log in with a username and password. Can I download the version file as a simple Web page using HTTP instead of FTP?

Mark Simpson

A In case you didn't read the April 2003 issue, CWebVersion is a class I wrote to let you compare your program's version number against a version number stored on the Web. I use it in my TraceWin program to notify users when there's a new version to download.

Yes, you can use HTTP; and no, you don't have to convert your file. I probably should have used HTTP in my original implementation, since HTTP is more widely supported than FTP. Many Web hosting providers disallow anonymous FTP for security reasons. While FTP is more efficient for transferring files (that's what it was designed for), HTTP is fine for fetching a simple text file.

CWebVersion reads a text file that has four comma-separated numbers: the high- and low-order 32-bit major and minor version numbers. To use it, you can write:

if (CWebVersion::Online()) {
  CWebVersion webver("www.mysite.com");
  if (webver.ReadVersion("myversion.txt")) {
    // dwVersionMS and dwVersionLS now
    // hold the version numbers
  }
}

The static member function CWebVersion::Online calls ::InternetQueryOption with INTERNET_OPTION_CONNECTED_STATE to see if the computer is currently connected to the Internet. If so, CWebVersion::ReadVersion reads the version file from your Web site. Once you've done that, you can compare the numbers with the ones compiled in your program, usually in your VERSIONINFO or DllGetVersion resource. (See my April 1998 column at microsoft.com/msj/0498/c0498.aspx for details on this.) The original CWebVersion used FTP to fetch the file; this month I've modified it to use HTTP. Reading an HTTP file over the Web is easy with MFC's WinInet classes:

// in CWebVersion::ReadVersion
CInternetSession session(_T("MySession"));
CHttpConnection* pConn =
  session.GetHttpConnection("www.dilascia.com",
  INTERNET_DEFAULT_HTTP_PORT);
CHttpFile* pFile =
  pConn->OpenRequest(CHttpConnection::HTTP_VERB_GET, "TraceWinVer.txt");
pFile->SendRequest();

This sequence attempts to download the file www.dilascia.com/TraceWinVer.txt. After calling SendRequest you can call CHttpFile::QueryInfoStatusCode to get the status code—for example, 404 for file not found or 200 for status OK (see wininet.h for a complete list of codes)—then call CHttpFile::Read to read the file into your buffer. CWebVersion::ReadVersion does this, then calls scanf to parse the contents from the format "Mhi,Mlo,mhi,mlo" where Mhi, Mlo, mhi, and mlo are the high- and low-order WORDs of the Major and minor version numbers. CWebVersion stores these in CWebVersion::dwMajorVersion and CWebVersion::dwMinorVersion. Figure 1 shows the full source.

Figure 1 WebVersion (Close Figure 1)

WebVersion.h
WebVersion.h
//////////////////
// This class encapsulates over-the-Web version checking. It expects a
// text version file that contains four numbers separated by commas, the
// same format for FILEVERSION and PRODUCTVERSION in VS_VERSION_INFO.
// ReadVersion reads these numbers into dwVersionMS and dwVersionLS.
// Uses HTTP, not FTP, to fetch the file.
//
class CWebVersion {
protected:
   enum { BUFSIZE = 64 };
   char    m_version[BUFSIZ];           // version number as text
   CString m_sServer;                   // server name (www.mysite.com)
   DWORD   m_dwStatus;                  // status code
   CString m_sStatus;                   // status text

public:
   DWORD dwMajorVersion;      // version number: most-sig 32 bits
   DWORD dwMinorVersion;      // version number: least-sig 32 bits

   CWebVersion(LPCTSTR server) : m_sServer(server) { }
   ~CWebVersion() { }

   static BOOL Online();
   BOOL     ReadVersion(LPCTSTR lpFileName);
   LPCSTR   GetVersionText()      { return m_version; }
   DWORD    GetStatus()           { return m_dwStatus; }
   CString  GetStatusText()       { return m_sStatus; }
};
WebVersion.cpp
WebVersion.cpp
#include „stdafx.h"
#include „WebVersion.h"

//////////////////
// Check if connected to Internet.
//
BOOL CWebVersion::Online()
{
    DWORD dwState = 0;
    DWORD dwSize = sizeof(DWORD);
    return InternetQueryOption(NULL,
       INTERNET_OPTION_CONNECTED_STATE, &dwState, &dwSize)
       && (dwState & INTERNET_STATE_CONNECTED);
}
//////////////////
// Read version number as string into buffer. Use MFC CHttpFile.
//
BOOL CWebVersion::ReadVersion(LPCTSTR lpFileName)
{
   ASSERT(lpFileName);
   BOOL bRet = FALSE;                   // assume failure
   m_dwStatus=0;
   m_sStatus.Empty();

   // Read version file using MFC Inet classes.
   INTERNET_PORT nPort = INTERNET_DEFAULT_HTTP_PORT;
   CInternetSession session(_T("MySession"));
   CHttpConnection* pConn = NULL;
   CHttpFile* pFile = NULL;
   try {
      pConn = session.GetHttpConnection(m_sServer, nPort);
      pFile = pConn->OpenRequest(
         CHttpConnection::HTTP_VERB_GET, lpFileName);
      pFile->SendRequest();
      pFile->QueryInfoStatusCode(m_dwStatus);
      if (m_dwStatus==HTTP_STATUS_OK) {
         const UINT BUFSIZE = 128;
         UINT nRead = pFile->Read(m_version, BUFSIZE);
         if (nRead>0) {
            // read version number in the form Mhi,Mlo,mhi,mlo
            m_version[nRead] = 0;
            int Mhi,Mlo,mhi,mlo;
            _stscanf(m_version, (_T(«%x,%x,%x,%x»)),
               &Mhi, &Mlo, &mhi, &mlo);
            dwMajorVersion = MAKELONG(Mlo,Mhi);
            dwMinorVersion = MAKELONG(mlo,mhi);
            bRet = TRUE;                // success!
         }
      } else {
         pFile->QueryInfo(HTTP_QUERY_STATUS_TEXT, m_sStatus);
      }

   } catch (CInternetException* pEx) { ... }
   session.Close();

   delete pFile;
   delete pConn;

   return bRet;
}

To test CWebVersion, I wrote a program called GetVersion.exe (see Figure 2). When I first tested CWebVersion on my own Web site, I named my version file TraceWinVer.dat. When I tried to download it, I got error 404 (file not found) even though the file was there. At first I thought perhaps I should add .dat to the list of accepted file types in the request header:

static LPCTSTR MyHeaders = _T("Accept: text/dat\r\n");
...
pHttpFile->AddRequestHeaders(MyHeaders);

Figure 2 A Little Test Program
Figure 2 A Little Test Program

However, this didn't fix the problem. I still got error 404. Well, it turns out my hosting provider had their server configured to prevent browsing .dat extensions. They were happy to change this, but I opted instead to keep the security and rename my version file TraceWinVer.txt. After all, it's text.

If you're programming with the Microsoft® .NET Framework, you can use HttpWebRequest and HttpWebResponse to fetch a file using HTTP, instead of MFC. With the .NET Framework, you create an HttpWebRequest with the full URL, then call GetResponse to send the request and fetch the response:

HttpWebRequest* req = dynamic_cast<HttpWebRequest*>(
    WebRequest::Create(S"http://www.dilascia.com/TraceWinVer.txt"));
req->Timeout = 5000; // 5 sec
HttpWebResponse* resp =
  dynamic_cast<HttpWebResponse*>(req->GetResponse());

The dynamic casts are kind of clunky, but necessary to access the HTTP-specific properties and methods of HttpWebRequest and HttpWebResponse. If you're using Visual Studio® 2005 with C++/CLI, replace the pointers with tracking handles (^) and you don't need the S for managed string literals. To read the contents of the file in .NET, create a StreamReader on the response stream and read it. In code, it looks like this:

StreamReader* strm =
  new StreamReader(resp->GetResponseStream(), encoding);
String* content = strm->ReadToEnd();
strm->Close();

For .NET-heads, I wrote a full GetVersion program in managed C++. You can download it from the MSDN®Magazine Web site.

Q How can I add any sound (not only with the MessageBeep function) to an MFC-based app?

Alexander Potapenko

A It's not hard to add sounds to your MFC-based app, but before I show you how, let me remind you that, especially in software, silence is golden. While there are many situations where sound is appropriate (your build failed, e-mail arrived, It's time to buy groceries), in most cases, It's better to keep quiet. Software sounds are the programming equivalent of belching. Have you ever visited one of those Web sites with the home page that plays a little jingle? The first thing most people do is press the Back button. If you must add sounds, please don't forget to provide a Mute option in your program's settings, under Tools | Options.

OK, now that I'm off my soapbox I can tell you about a Windows® function called PlaySound that does what you want. It's defined in <mmsystem.h> and you have to link with winmm.lib. PlaySound takes the name of a sound file or resource and plays the sound. The following shows an example:

PlaySound("woofwoof.wav",NULL,SND_NODEFAULT);

Here the special flag SND_NODEFAULT tells Windows: don't play the default sound (MessageBeep) if you can't find the file. Figure 3 shows the other flags. As with so many Windows functions, there are many different ways to use PlaySound, many of them poorly documented. In fact, some flags continue to mystify me. Never fear, I'll shine what light I can into this fog.

Figure 3 PlaySound Flags (Close Figure 3)

Flag | Description
SND_APPLICATION | Play app-specific sound. (Read from registry key HKCU\AppEvents\Schemes\Apps\progname).
SND_ALIAS | The pszSound parameter is a system-event alias in the registry or the WIN.INI file.
SND_ALIAS_ID | The pszSound parameter is a resource ID.
SND_ASYNC | Play sound asynchronously. PlaySound returns immediately after launching the sound. Call PlaySound(NULL) to stop.
SND_FILENAME | Play a sound file; pszSound is the file name.
SND_LOOP | Play sound repeatedly until PlaySound is called again with the pszSound parameter set to NULL. You must also use SND_ASYNC to indicate an asynchronous sound event.
SND_MEMORY | Play sound image in memory.
SND_NODEFAULT | Don't play default sound if the requested sound can't be found.
SND_NOSTOP | Don't stop the current sound already playing. If this flag is not specified, PlaySound attempts to stop the currently playing sound so that the device can be used to play the new sound.
SND_NOWAIT | If the driver is busy, return immediately without playing the sound.
SND_RESOURCE | Play a sound (WAVE) resource.

One of the most useful ways to play a sound is with the SND_APPLICATION flag, which plays an application-associated sound. For example:

PlaySound("AppExit",NULL, SND_APPLICATION|SND_NODEFAULT);

Figure 4 Happy Sounds
Figure 4 Happy Sounds

This plays your AppExit sound. And what exactly is that? Windows looks in the registry for HKCU\AppEvents\Schemes\Apps\AppExit and reads the value for .current. If the .current value is a file name like exit.wav, Windows plays it. Windows searches the current directory, the Windows directory, the Windows system directory, and your PATH environment variable. Why are app-associated sounds cool? Because users can customize them with the Control Panel sound applet (see Figure 4). PlaySound also has a SND_RESOURCE flag that lets you play sounds from your resource file. To play a resource sound, you must first add the sound to your resource (.rc) file:

AppExit WAVE "res\\STExit.wav"

Note the resource must be a WAVE resource—a critical detail that isn't documented, as far as I can tell. The resource compiler embeds the WAV file in your EXE so now you can use SND_RESOURCE to play it, like so:

PlaySound("AppExit", AfxGetResourceHandle(), SND_RESOURCE);

PlaySound needs the module handle for the module that contains the resource, which you can get by calling AfxGetResourceHandle (which for most apps is the same as AfxGetInstanceHandle). In the previous snippets, the resource identifier is a string ("AppExit"), but you can also use an integer ID if you specify SND_ALIAS_ID.

To simplify your life, I wrote a little class, CSoundMgr, that makes implementing app sounds as easy as eating cherries. CSoundMgr lets you define logical sounds and play them by ID. It has functions to register your sounds for you, and lets you mute your app in one fell swoop by changing a flag. CSoundManager even searches for default sounds in your resource file. To show how it works in practice, I wrote a test program called—what else?—SoundTest. SoundTest is a typical MFC dialog-based app. It displays the current values for five application sounds (see Figure 5).

Figure 5 SoundTest
Figure 5 SoundTest

The first step to define sounds for SoundTest is to create IDs for the sounds. I used an enum with five values: MYSND_HAPPY, MYSND_UNHAPPY, and so on. Don't use zero for any sound ID; CSoundMgr::PlaySound(0) stops the current sound, equivalent to ::PlaySound(NULL, NULL, 0). With IDs defined, you can use the macros defined in SoundMgr.h to build a sound table, like so:

BEGIN_SOUND_MAP(MySounds)
DEFINE_SOUND(MYSND_HAPPY, _T("ST_Happy"), _T("SoundTest Happy"))
...
END_SOUND_MAP()

Each table entry holds three items: an ID, a logical name, and a GUI name. The ID is used to play the sound. The logical name (for example, ST_Happy) is the name used internally for the registry keys and default sound resources. The GUI name is the human-readable name users see when they use the Sounds control panel applet—for example, "SoundTest Happy" as in Figure 4. Once you've defined your table, the next step is to create a CSoundMgr initialized from your table:

// THE sound manager for this app
CSoundMgr SoundMgr(MySounds);

You only need one CSoundMgr for the entire app. Finally, if you want default built-in sounds, you must add a WAVE resource for each logical sound name that has a default. For example:

ST_HAPPY   WAVE "res\\STHappy.wav"
ST_UNHAPPY WAVE "res\\STUnhappy.wav"

Now with the sounds defined, all you have to do to play a sound is write the following:

SoundMgr.PlaySound(MYSND_HAPPY)

And CSoundMgr plays the happy sound. CSoundMgr looks first in the registry for HKCU\AppEvents\Schemes\Apps\SoundTest\ST_Happy\.current; if there's no such registry key/value, CSoundMgr::PlaySound plays the sound resource with the same logical name, ST_Happy. You can call CSoundMgr::IsRegistered to see if your sounds are registered—if not, call CSoundMgr::Register to register them, as shown here:

    if (!SoundMgr.IsRegistered())
      SoundMgr.Register();

CSoundMgr::Register creates all the registry keys required for users to customize sounds in the Sounds control panel applet. It doesn't actually set any of the key values, but rather leaves them empty so CSoundMgr::PlaySound will use the default resource sounds. If you don't want default sounds, don't create resources for them, or use the following:

  CSoundMgr::m_bUseResourceSounds =
    FALSE;

Implementing CSoundMgr is fairly straightforward. Most of the code is spent groveling through the registry, perhaps one of the most onerous chores in all of Windows-based programming (see Figure 6). Lucky for you I've already done the work. The subkeys for each logical sound are in HKCU\AppEvents\Schemes\Apps\progname, where progname is your program's name, the same string you use for program Settings, the one returned by ::AfxGetAppName. Each subkey holds the sound file name in the .current value. CSoundMgr doesn't create the values, only the keys, since the Sounds Control Panel applet creates a .current value when the user changes a sound. The default value for each logical sound key is the human-readable GUI name for the sound, but Windows ignores this value as far as I can tell; it looks for GUI names in a different key, HKCU\AppEvents\EventNames. Go figure.

Figure 6 SoundMgr (Close Figure 6)

SoundMgr.h
#pragma once

// one of these for each application sound
struct APPSOUND {
   UINT id;                             // ID for programming
   LPCTSTR log_name;                    // logical event name for registry
   LPCTSTR gui_name;                    // user-visible event name
};

//////////////////
// Class to manage app sounds. Use macros to create a static table, then
// construct one of these initialized from the table.
//
class CSoundMgr {
protected:
   APPSOUND* m_soundtab;                // table of APPSOUNDs

   // helper fns
   BOOL GetSoundFileName(LPCTSTR logname, CString& s);
   BOOL GetLogicalName(UINT id, CString& logname);

public:
   BOOL m_bUseResourceSounds;           // look for default sounds in res
                                           file?
   BOOL m_bEnableSounds;                // enable/disable sound

   CSoundMgr(APPSOUND* tab);
   ~CSoundMgr() { }

   CString GetSoundSchemeKey(LPCTSTR logname=NULL);

   BOOL PlaySound(UINT id, DWORD dwAddFlags=0);
   BOOL PlaySound(LPCTSTR logname, DWORD dwAddFlags=0);

   BOOL IsRegistered();                 // are sounds registered?
   BOOL Register();                     // register sounds
};

//////////////////
// Macros to build sounds table.
//
#define BEGIN_SOUND_MAP(name) APPSOUND name[] = {
#define DEFINE_SOUND(id, regname, guiname) { id, regname, guiname },
#define END_SOUND_MAP() { 0, NULL, NULL } };
SoundMgr.cpp
#include "stdafx.h"
#include "SoundMgr.h"

//////////////////
// Construct sound manager from table.
//
CSoundMgr::CSoundMgr(APPSOUND* tab) : m_soundtab(tab)
{
   m_bUseResourceSounds = TRUE;   // default: use default resource sounds
   m_bEnableSounds = TRUE;        // default: sounds enabled
}

//////////////////
// Get name of registry key for sound events:
// HKCU\AppEvents\Schemes\Apps\logname, where logname is optional.
//
CString CSoundMgr::GetSoundSchemeKey(LPCTSTR logname)
{
   CString key;
   key.Format(_T("AppEvents\\Schemes\\Apps\\%s"), AfxGetAppName());
   if (logname) {
      key += _T("\\");
      key += logname;
   }
   return key;
}

SoundMgr.h
#pragma once

// one of these for each application sound
struct APPSOUND {
   UINT id;                             // ID for programming
   LPCTSTR log_name;                    // logical event name for registry
   LPCTSTR gui_name;                    // user-visible event name
};

//////////////////
// Class to manage app sounds. Use macros to create a static table, then
// construct one of these initialized from the table.
//
class CSoundMgr {
protected:
   APPSOUND* m_soundtab;                // table of APPSOUNDs

   // helper fns
   BOOL GetSoundFileName(LPCTSTR logname, CString& s);
   BOOL GetLogicalName(UINT id, CString& logname);

public:
   BOOL m_bUseResourceSounds;           // look for default sounds in res
                                           file?
   BOOL m_bEnableSounds;                // enable/disable sound

   CSoundMgr(APPSOUND* tab);
   ~CSoundMgr() { }

   CString GetSoundSchemeKey(LPCTSTR logname=NULL);

   BOOL PlaySound(UINT id, DWORD dwAddFlags=0);
   BOOL PlaySound(LPCTSTR logname, DWORD dwAddFlags=0);

   BOOL IsRegistered();                 // are sounds registered?
   BOOL Register();                     // register sounds
};

//////////////////
// Macros to build sounds table.
//
#define BEGIN_SOUND_MAP(name) APPSOUND name[] = {
#define DEFINE_SOUND(id, regname, guiname) { id, regname, guiname },
#define END_SOUND_MAP() { 0, NULL, NULL } };

SoundMgr.cpp
#include "stdafx.h"
#include "SoundMgr.h"

//////////////////
// Construct sound manager from table.
//
CSoundMgr::CSoundMgr(APPSOUND* tab) : m_soundtab(tab)
{
   m_bUseResourceSounds = TRUE;   // default: use default resource sounds
   m_bEnableSounds = TRUE;        // default: sounds enabled
}

//////////////////
// Get name of registry key for sound events:
// HKCU\AppEvents\Schemes\Apps\logname, where logname is optional.
//
CString CSoundMgr::GetSoundSchemeKey(LPCTSTR logname)
{
   CString key;
   key.Format(_T("AppEvents\\Schemes\\Apps\\%s"), AfxGetAppName());
   if (logname) {
      key += _T("\\");
      key += logname;
   }
   return key;
}
//////////////////
// Play logical sound from ID.
//
BOOL CSoundMgr::PlaySound(UINT id, DWORD dwAddFlags)
{
   if (id==0) {
      ::PlaySound(NULL,NULL,0); // ID 0 = stop playing async sound
      return TRUE;
   }
   CString logname;
   return GetLogicalName(id, logname) ?
      PlaySound(logname, dwAddFlags) : FALSE;
}

//////////////////
// Play logical sound using logical name.
//
BOOL CSoundMgr::PlaySound(LPCTSTR logname, DWORD dwAddFlags)
{
   if (m_bEnableSounds) {
      CString soundname;
      if (GetSoundFileName(logname, soundname)  && !soundname.IsEmpty()) {
         return ::PlaySound(soundname, NULL,
            dwAddFlags|SND_APPLICATION|SND_NODEFAULT);
      } else if (m_bUseResourceSounds) {
         // play sound resource w/same logical name
         return ::PlaySound(logname, AfxGetResourceHandle(),
            dwAddFlags|SND_RESOURCE|SND_NODEFAULT);
      }
   }
   return FALSE;
}

//////////////////
// Get logical sound name from ID. Search table for sound that matches.
//
BOOL CSoundMgr::GetLogicalName(UINT id, CString& logname)
{
   if (m_soundtab) {
      for (int i=0; m_soundtab[i].id; i++) {
         if (m_soundtab[i].id==id) {
            logname = m_soundtab[i].log_name;
            return TRUE;
         }
      }
   }
   return FALSE;
}

//////////////////
// Read current sound file name from registry.
//
BOOL CSoundMgr::GetSoundFileName(LPCTSTR logname, CString& s)
{
   CString key = GetSoundSchemeKey(logname); // get registry key

   // open key and read the ".current" value.
   BOOL bRet = FALSE;
   HKEY hkey;
   if (RegOpenKey(HKEY_CURRENT_USER, key, &hkey)==ERROR_SUCCESS) {
      TCHAR val[255]={0};
      LONG  vlen=sizeof(val)/sizeof(TCHAR);
      if (RegQueryValue(hkey, _T(".current"), val, &vlen)==ERROR_SUCCESS)
      {
         s = val;                       // return in caller’s string
         bRet = TRUE;                   // success
      }
      RegCloseKey(hkey);
   }
   return bRet; // failed :(
}
//////////////////
// Test if sounds are registered. Algorithm: see if first logical sound is
// registered. Note this is not foolproof if someone deletes other keys.
//
BOOL CSoundMgr::IsRegistered()
{
   ASSERT(m_soundtab);
   CString key = GetSoundSchemeKey(m_soundtab[1].log_name);
   HKEY hkey;
   if (RegOpenKey(HKEY_CURRENT_USER, key, &hkey)==ERROR_SUCCESS)
      return TRUE;
   RegCloseKey(hkey);
   return FALSE;
}

// Helper function to open a registry key, creating if necessary
static LONG RegCreateOpenKey(HKEY hkey, LPCTSTR lpSubKey, PHKEY phkRes)
{
   return RegOpenKey(hkey, lpSubKey, phkRes)==ERROR_SUCCESS ?
      ERROR_SUCCESS : RegCreateKey(hkey, lpSubKey, phkRes);
}

//////////////////
// Register app-defined sounds. Note this only creates all the keys,
// it does NOT set the values (file names) since the app will play default
// sounds from the resource file. Creating the keys is all that’s required
// to let users change the sounds.
//
BOOL CSoundMgr::Register()
{
   ASSERT(m_soundtab);
   BOOL bRet = FALSE;

   static LPCTSTR APPS = _T("AppEvents\\Schemes\\Apps");
   static LPCTSTR LABELS = _T("AppEvents\\EventLabels");
   HKEY hkApps,hkLabels;

   if (RegOpenKey(HKEY_CURRENT_USER, APPS, &hkApps)==ERROR_SUCCESS) {
      if (RegOpenKey(HKEY_CURRENT_USER, LABELS, &hkLabels)==ERROR_SUCCESS)
   {
         HKEY hkApp;
         if (RegCreateOpenKey(hkApps, AfxGetAppName(),
               &hkApp)==ERROR_SUCCESS) {
            // opened event key, now add all the sounds
            for (int i=0; m_soundtab[i].id; i++) {
               HKEY hk;
               LPCTSTR lpName = m_soundtab[i].gui_name;
               DWORD dwLen = (DWORD)_tcslen(m_soundtab[i].gui_name);

               // add app event
               RegCreateKey(hkApp, m_soundtab[i].log_name, &hk);
               RegSetValue(hk, NULL, REG_SZ, lpName, dwLen);
               RegCloseKey(hk);

               // add event name
               RegCreateKey(hkLabels, m_soundtab[i].log_name, &hk);
               RegSetValue(hk,NULL,REG_SZ, lpName, dwLen);
               RegCloseKey(hk);
            }
            RegCloseKey(hkApp);
            bRet = TRUE; // success! (maybe)
         }
         RegCloseKey(hkLabels);
      }
      RegCloseKey(hkApps);
   }
   return bRet;
}

So you have to create another set of keys, one for each logical sound name, whose default value is the human-readable GUI name. Of course, if you use CSoundMgr, you can forget all about registry keys. Just define your sounds and call CSoundMgr::Register. If the user has customized sounds, calling Register won't clobber them since it only creates those keys that don't already exist. If you want to implement a reset command to restore sounds to their defaults, you should delete HKCU\AppEvents\Schemes\Apps\progname and then call Register again.

One last bit of advice: in choosing your logical names, be careful to avoid collisions with other apps. Unfortunately, all the event names live in the same namespace within HKCU\AppEvents\EventNames, so I recommend using a prefix like I did for SoundTest, where all the logical names begin with ST_. It makes them easier to find too, since REGEDT32 lists the registry keys in alphabetical order.

SoundTest has an Enable Sounds checkbox that lets users turn sound on or off. The command handler for this button toggles the variable CSoundMgr::m_bEnableSounds. I leave you to download and ponder the source for full details. Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: The Pay as You Go Model

{ END BRACKET }
The Pay as You Go Model
Eric N. Bush

How would you feel if you were having your dream house built, but when you inspected the house you found some fairly obvious flaws in the work—cracks in the foundation, support beams missing, crooked and leaning walls. Then to make matters worse, the construction foreman walks up to you and says, "Isn’t it beautiful? And we’re even right on schedule. And by the way, don’t worry about those apparent flaws, we’re just about to move into our quality phase and rip everything apart to set things right."

Not only would it take much longer to build a house this way, but the foreman would also have a high degree of customer dissatisfaction. As software engineers, one thing we must insist upon is quality right from the start. If we don’t pay the price to build in quality as part of a continuous process, then we must pay a much higher price attempting to test it in later.

In some companies, developers are given a very short coding phase followed by what is termed a very long "stabilization" phase. Most of the time in "stabilization" is spent trying to clean up the mess they got themselves into during the insanely compressed coding phase. In order to meet this compressed timeline, code inspections, unit testing, and other essential practices are tossed aside. The definition of "code complete" under these circumstances is extremely lacking. Nothing works for a stretch of time until features start to stabilize little by little. The project eventually reaches some measure of quality, but it is not as high as it should be. We can call this the "test in quality" model. Working under these conditions is very demoralizing to a team and causes high levels of worker stress and burnout.

Teams need to follow a "pay as you go" model. The battle cry should be "start stable and stay stable." Features must go in completely tested and working over a greater expanded coding phase. Slowing down and doing things the right way is really the key to speeding things up in the long run. It is much better to have 80 percent of development time spent in coding (in a continuous quality manner) and leave 20 percent of your time at the end for verification. "Code complete" in this model really means something.

One way or another there is a price to be paid for quality. You can spread that cost evenly across the complete development cycle, or pay a much higher cost later and attempt to test in the quality at the end. It makes much more sense to build in the quality right from the start. The benefits are incredible—higher final quality, shorter overall schedule, higher team morale, consistent working builds, realistic estimates, and, of course, increased customer satisfaction. You can calculate the fixed cost of the engineering overhead needed to follow a strict process for building in quality. You can’t, however, predict the cost of searching for and fixing all the issues with a "test in quality" model. In those cases, development is only putting in enough hacks to get the product out the door and is not really fixing the code to make it solid. It is best to establish expected good behavior right from the start of the project.

Get started by creating a project plan that gives the details of your quality goals and how you will reach them. This plan will explain how you are going to address the performance, reliability, instrumentation, security, and maintainability of the code. Then, as you sit down to write your code each day, you should have the following process checklist handy:

  • Keep in mind the quality goals
  • Follow team coding standards
  • Use defensive coding practices, including input validation and proper error handling
  • Run static code analysis tools
  • Run dynamic code analysis tools, such as those for reliability and performance profiling
  • Create and run associated unit tests for all your code
  • Hold regular code inspections/reviews
  • Do buddy builds and simple build verification tests
  • Refactor code for maintainability, simplicity, consistency

Never make a check-in without meeting that standard. The goal is to have a stable, ready-to-ship product every day. When features go in, there will be no question as to their readiness. This model is much more efficient and it fits in well with other agile ideas, such as having shorter milestones along with having better defined quality checkpoints. Make it a practice to build your house the right way, with quality built in right from the start. In following the "pay as you go" model, your company will be in a greater position to not only innovate but to grow as well.

Eric N. Bush is a software development lead in the Engineering Excellence group at Microsoft. He works as an internal consultant and classroom instructor driving engineering and organizational improvements in the areas of people skills, process, and technology. Reach him at ericbush@microsoft.com.