 MSDN Magazine April 2004

MSDN Magazine April 2004

Testing: Perform Code Coverage Analysis with .NET to Ensure Thorough Application Testing

When running your tests how do you know which parts of your product code are actually executed and tested? This article presents a complete system called Fundamental Function code coverage that operates at the method level. The author gives an overview of the system so you can understand code coverage principles, explains the key parts of the underlying code, and discusses how code coverage fits into the overall product development cycle. After reading this article you will be able to perform code coverage analysis on any .NET software system.

James McCaffrey Test-Driven C#: Improve the Design and Flexibility of Your Project with Extreme Programming Techniques

Test-driven development (TDD) should be on every developer's radar screen because a comprehensive set of tests makes for maintainable code and frees you from having to create a perfect design up-front. This article explains how to perform TDD and takes you step-by-step through a number examples to get you started.

Will Stott and James Newkirk Instrumentation: Powerful Instrumentation Options in .NET Let You Build Manageable Apps with Confidence

As systems grow and become more heterogeneous, so their complexity increases. The more code you write, the more that can go wrong. The more that can go wrong, the more you need a good instrumentation policy. In this article, the author looks at the various technologies available in the .NET Framework, such as tracing, logging, WMI, EIF, which are designed to help you. He will also look at the pitfalls you should avoid and provide you with the fundamentals from both a technical and managerial perspective so that you can instrument your code effectively.

Jon Fancey Stress Testing: Custom LoadGenerator Tool Identifies the Issues Your Application Faces Under Stress

It's easy to postpone stress testing when developing an application, and it's easy to forgo it altogether. Having an easy-to-use framework at your fingertips to conduct these tests can make the task far less painful. This article walks you through an application that eases the task of generating load for a variety of layers within an application.

Brian Otto Mobility: Add Keyboard Support to Compact Framework Apps by Trapping Windows Messages

The Compact Framework Control class doesn't provide direct access to Windows messages. However, with P/Invoke, a few lines of native code, and the Compact Framework MessageWindow class, it's still possible to access underlying Windows messages. This can be used to work around any .NET Framework features, including keyboard support, that are not included in the Compact Framework.

Alan Pulliam Editor's Note: Testing, Testing 1 2 3

Each month dozens of readers turn to this page for the latest news on what the editors of MSDN Magazine are thinking. And we do our best to oblige with our hard-hitting yet pithy observations on the state of the world around us, the latest in XML fashions, what's coming up in the current issue, and more.

New Stuff: Resources for Your Developer Toolbox

Infragistics is shipping NetAdvantage 2004 Volume 1, a toolset for developing commercial-quality user interfaces for the Microsoft® . NET Framework, ASP. NET, Tablet PC, and COM. NetAdvantage 2004 Volume 1 features Windows® Forms and ASP.

Theresa W. Carey Web Q&A: JScript Leaks, Getting the XmlDataDocument, and ASPX Includes

Edited by Nancy Michell Data Points: Creating Audit Tables, Invoking COM Objects, and More

Dealing with error handling between T-SQL and a calling application, evaluating when a field's value has changed, and creating auditing tables in SQL Server™ are all common issues that developers must tackle.

John Papa Cutting Edge: Image Generation Service for ASP.NET 1.1

Dino Esposito The XML Files: All About Blogs and RSS

Aaron Skonnard Advanced Basics: Synchronizing Multiple Windows Forms

Ken Spencer .NET Matters: Const in C#, Exception Filters, IWin32Window, and More

Welcome to . NET Matters. This new column will delve into the ins and outs of the Microsoft® . NET Framework, answering readers' questions on various topics related to its extensive libraries, languages, and the common language runtime.

Stephen Toub Security Briefs: Beware of Fully Trusted Code

The vast majority of managed applications run with full trust, but based on my experience teaching . NET security to developers with a broad range of experience, most really don't understand the implications of fully trusted code.

Keith Brown Bugslayer: .NET Internationalization Utilities

As you saw in last month's column, . NET internationalization support is excellent and allows you to move your application to a world audience quite easily. Before you jump into this month's discussion, you may want to go back and read the March column.

John Robbins C++ Q&A: CD Burning and Device Discovery with IMAPI

Paul DiLascia Talking To…: Robert Green Talks About Communities for Visual Basic .NET

MSDN Magazine recently talked to Robert Green, the Community Lead Program Manager for Visual Basic, about what to expect in the next version and how the Visual Basic team hopes to work more closely with the Visual Basic community.


Testing: Perform Code Coverage Analysis with .NET to Ensure Thorough Application Testing

Testing

Perform Code Coverage Analysis with .NET to Ensure Thorough Application Testing

James McCaffrey
This article discusses:
  • The importance of code coverage analysis
  • A custom code coverage tool
  • The role of code coverage in the development process
  • Profiling in .NET
This article uses the following technologies:
C++, C#, Microsoft .NET Framework
Code download available at: CodeCoverageAnalysis.exe (174KB)
Contents
Overview of Code Coverage Analysis
The Profiling DLL
Building ProfilerFF.dll
Preparing for Code Coverage
Running Tests Under Code Coverage
Reporting Code Coverage
Before and After .NET
Conclusion

T
he basic idea behind code coverage is straightforward. During product development, a large number of test cases are created and run to ferret out bugs in the system. Code coverage analysis monitors which parts of the product's code are exercised by the collection of test cases. For it to be effective, 100 percent of the product's code should be executed by the test cases. If there are segments of product code that are never run during testing, then the product has not been thoroughly tested.

Although the idea of code coverage is simple enough, actually performing code coverage analysis in a non-.NET environment can be very time consuming, difficult, and expensive. However, the Microsoft® .NET environment provides software developers and testers with a simple and effective new way to perform code coverage analysis. In the days before .NET, code coverage was one of the most frustrating parts of my job as a tester. Recently, though, I discovered that the .NET environment provides a much improved process for performing code coverage. In this article, I will provide you with all the code you need to perform .NET code coverage analysis. I call this system Fundamental Function code coverage. Adding code coverage analysis skills to your toolset will help you keep a watchful eye on your code, whether you are a developer, tester, or program manager.

In the sections that follow, I will walk through an example of code coverage at a high level so you can understand the principles involved. Then I will discuss the details of the implementation. I will conclude with a discussion of how code coverage analysis fits into the software product development cycle.

Overview of Code Coverage Analysis

Because traditional code coverage analysis is so tricky, I'll walk through a concrete example at a high level. Code coverage analysis has three phases: preparing the coverage environment, enabling coverage profiling and running tests, and then disabling coverage profiling and generating a report.

The product I will analyze is a Windows®-based application that references a dummy class library that contains several dummy constructors and methods. Figure 1 illustrates how my code coverage environment is prepared.

Figure 1 Methods Under Coverage
Figure 1 Methods Under Coverage

First, you build a special profiling DLL which, when enabled later, will watch the execution of code through the common language runtime (CLR) and log all entries to specified methods in the product being tested. After creating the profiler, you identify which .NET assemblies you want to include under coverage. In this case, I'm looking at the DummyApp.exe and DummyLib.dll assemblies.

Next you determine which constructors, methods, and properties are in the assemblies under coverage. As you can see, there are 12 of these under coverage in Figure 1.

Figure 2 shows the second phase of code coverage: enabling the profiling DLL. Now any .NET managed code that runs in the environment will be monitored and recorded.

Figure 2 Enabling and Using a Profiler
Figure 2 Enabling and Using a Profiler

Next, you run either a single test case or a suite of tests. In this example, I just clicked the Methods button of the dummy application to simulate testing it. When performing code coverage analysis, I'm usually not concerned with whether tests pass or fail, but rather how much of the codebase of the product under test is touched by the tests. In other words, even if every test fails, but code coverage is 100 percent, I know I'll need to fix my code but I'm happy that all the methods in the product are being tested. That said, you need to be careful about test failures and their affects on code coverage. If you don't do coverage using the tests as they're supposed to run, you can't be sure that you covered the right code. After running the test, close the dummy application to flush to a text file all coverage information that the profiler gathered.

The third phase of code coverage is illustrated in Figure 3. I disabled coverage profiling so that it wouldn't continue gathering information. Then I ran the report generator, which collects the information about all methods that could potentially be touched (which was generated in Phase 1) and gathers the information about the methods that were actually touched (generated in Phase 2). It then compares these two lists to determine the resulting code coverage percentage. In this example, 8 of the potential 12 methods were exercised by the test, for a 67 percent coverage rate. I can use the report data to determine which methods have not been exercised and create new test cases.

Figure 3 Report Generation
Figure 3 Report Generation

In other systems the preparation of code coverage, during which time you determine which methods you will monitor, and the reporting part, where you analyze the results, are often performed by two different programs. In the Fundamental Function system presented here, I put both of these functionalities into a single program named FFcover.exe and controlled them by -m and -r command-line arguments.

The system described in this article monitors code coverage at the method level. With it I determine if a particular constructor, method, or property has been entered, but I do not learn about the path of execution inside the method. More granular code coverage systems operate at lower levels. For example, one system called Basic Block coverage operates at the block level (testing whether execution enters a particular block of statements).

The Profiling DLL

As mentioned in the previous section, at the foundation of code coverage is a profiling DLL that can monitor all the activity that passes through the CLR. As you might expect, this is very sophisticated code. Fortunately, the .NET Framework comes with two complete example profiler source code sets. I was able to use one of them with very few changes to create my profiler—ProfilerFF.dll.

In the example shown in Figure 2, the profiler works behind the scenes to capture method names that were entered when the dummy test application ran in the coverage-enabled environment. That data is logged to a text file named ffcoverXXX.log, where the Xs represent a time-stamp value. One line of the resulting log file looks like the following:

0x00000d38;void DummyLib.Dummy::Bar(int32&,float64&,String[])
Here the data following the thread ID indicates that the Bar method in the Dummy class, which is part of the DummyLib namespace and which accepts three parameters and returns void, was entered during the test.

Let's examine the profiler. The base profiler source code from which I derived my code coverage profiler is located by default in the C:\Program Files\Microsoft Visual Studio .NET\FrameworkSDK\Tool Developers Guide\Samples\profiler\hst_profiler folder. HST stands for hot spot tracker to indicate that a DLL built from that code can monitor how much time is spent in methods.

I first created a C:\Coverage root folder on my machine and then copied the hst_profiler and Include folders to it. The build process will require that these two folders have the same parent folder. Next, I renamed the hst_profiler folder to FFprofiler to reflect its new functionality.

Before editing, I renamed three files in the FFprofiler folder: ProfilerHST.cpp to ProfilerFF.cpp, ProfilerHST.def to ProfilerFF.def, and ProfilerHST.mak to ProfilerFF.mak. It's clear that these three files ending with HST are specific to the timing profiler, so renaming them makes sense. The files ProfilerInfo.cpp, ProfilerInfo.h, ProfilerCallback.cpp, and ProfilerCallback.h are, for the most part, generic to any profiling DLL, so it is reasonable to leave their names alone. I also renamed EnableProfiler.bat to enable.bat to make it easier to type on a shell command line.

The first file I edited is ProfilerCallback.h. At the beginning of the file, I made a few minor changes:

extern const GUID __declspec( selectany ) CLSID_PROFILER =
{ 0x5ac86959, 0x7927, 0x4177, { 0x98, 0x02, 0xa5, 0x40, 0xf1, 0x96,
                                0x78, 0x74 } };

#define THREADING_MODEL     "Both"
#define PROGID_PREFIX       "ProfilerFF"
#define COCLASS_DESCRIPTION "Fundamental Function Code Coverage"
#define PROFILER_GUID       "{5AC86959-7927-4177-9802-A540F1967874}"
Every profiler has a GUID identifier so that different profilers can be active in the same environment, monitoring different CLR activity. I used the Visual Studio® .NET Create GUID tool to generate a new GUID and pasted it at the two places shown. I also changed the values of PROGID_PREFIX and COCLASS_DESCRIPTION to reflect code coverage functionality.

The second change I made was in the ProfilerFF.def definition file. Again, this was just a minor change to make it conform to the other edits, so that the first line below is changed to the second:

LIBRARY         ProfilerHST
LIBRARY         ProfilerFF

The third set of edits was in the ProfilerFF.mak makefile. I will cover those changes in the next section when I build the profiler DLL. The fourth and last file in folder ProfilerFF that I edited is ProfilerInfo.cpp. Most of the changes were in the FunctionTimingInfo::Dump function. The original Dump function outputs a semicolon-delimited text file containing the name of the function entered, the period of time the execution engine spent there, the number of times the function was entered, and other useful information. For most types of code coverage, you only need to know which methods were entered. In fact, additional information just makes parsing more difficult later. The portion of the code that records which methods were actually entered during a test in a coverage-enabled environment is shown in Figure 4.

Figure 4 Profiler Output Code (Close Figure 4)
if ( pFunctionInfo->m_functionName[0] != NULL )
{
  if ( wcsstr(pFunctionInfo->m_functionName,L"DummyApp") != NULL ||
       wcsstr(pFunctionInfo->m_functionName,L"DummyLib") != NULL )
  {
    LOG_TO_FILE( ("0x%08x;", m_win32ThreadID) )
    if ( pFunctionInfo->m_returnTypeStr[0] != NULL )
      LOG_TO_FILE( ("%S ", pFunctionInfo->m_returnTypeStr) )
    else
      LOG_TO_FILE( ("No return value") )

    LOG_TO_FILE( ("%S(", pFunctionInfo->m_functionName) )

    if ( pFunctionInfo->m_argCount > 0 )
    {
      WCHAR *parameter;
      WCHAR *separator = L"+";

      parameter = wcstok( pFunctionInfo->m_functionParameters,
                          separator );
      while ( parameter != NULL )
      {
        LOG_TO_FILE( ("%S", parameter) )
        parameter = wcstok( NULL, separator );
        if ( parameter != NULL )
          LOG_TO_FILE( (",") )
      }  // while

    }  // if m_argCount > 0
    LOG_TO_FILE( (")") )
    LOG_TO_FILE( ("\n") )
  }
}

else
  LOG_TO_FILE( ("Unable to Retrieve Function Information \n") )

The Dump code first checks to make sure that the pointer to the function name is not NULL, or in other words that there is a function name to log:

if ( pFunctionInfo->m_functionName[0] != NULL )

The next if conditional acts as a filter to specify which methods to log. The following code returns true only for functions (such as .NET methods, constructors, and properties) whose names begin with DummyApp or DummyLib:

wcsstr(pFunctionInfo->m_functionName,L"DummyApp") != NULL ||
     wcsstr(pFunctionInfo->m_functionName,L"DummyLib") != NULL )
Because the full name of a .NET method begins with its namespace, this filter will capture all methods in the DummyApp.exe and DummyLib.dll assemblies. If you don't add such a filter, the profiler will capture all system-related methods too, and there are a lot of them. This can, of course, be changed to suit your needs.

All actual logging is done using a LOG_TO_FILE macro defined in the Include\basehdr.h file. The following code writes an ID associated with the thread of execution to a text file specified in the file Include\basehlp.hpp (which I will examine in a moment):

LOG_TO_FILE( ("0x%08x;", m_win32ThreadID) )
Having a thread ID is useful when the application under test is multithreaded. It also gives you an easy way to determine if a line of output is actual data or other information such as comments.

It would be wonderful if you could identify a method using only its namespace prefix and its name, but because of overloading you need return type and parameter information to differentiate methods with the same name. Here's the code that logs the method return type and method name:

if ( pFunctionInfo->m_returnTypeStr[0] != NULL )
    LOG_TO_FILE( ("%S ", pFunctionInfo->m_returnTypeStr) )

LOG_TO_FILE( ("%S(", pFunctionInfo->m_functionName) )

Dealing with data type names is a major issue. The problem is that there are three sets of type names: those used by the profiler (for example, int32), those used by the CLR and the .NET Framework (System.Int32), and those used by C# and other .NET-compliant languages (int). For now it is enough for you to realize that the profiler has one set of type names that will later have to be mapped to .NET type names.

The code that logs the function parameters uses an old C/C++ parsing technique, as shown here:

parameter = wcstok( pFunctionInfo->m_functionParameters, separator );
while ( parameter != NULL )
{
  LOG_TO_FILE( ("%S", parameter) )
  parameter = wcstok( NULL, separator );
  if ( parameter != NULL )
    LOG_TO_FILE( (",") )
}
The wcstok function is the wide-character version of the venerable strtok function. The code logs each parameter to a file, separated by the comma character. There is nothing particularly special about the comma—any character could have been used, but commas are standard delimiters.

Of the 11 files in the Include folder, only one needs to be edited. The basehlp.hpp file specifies the path and file name of the text file that the profiler writes the log data to. The original line was:

strcpy( logfile, "output.log" );
I replaced it with this:
time_t ltime;
time( &ltime );
char buffer[20];
_i64toa( ltime, buffer, 10 );

strcpy( logfile, "C:\\Coverage\\FFdataFiles\\ffcover" );
strcat( logfile, buffer );
strcat( logfile, ".log" );

The new code gets a Unix-style timestamp and uses it to create a file name like ffcover0123456789.log so that multiple log files can reside in the same folder. The location of the files is hardcoded, which is usually a bad idea but in this case leads to a simpler design.

Building ProfilerFF.dll

Building a DLL of any size is a nontrivial task. The system in this article is predominantly command-line oriented, so I will build my profiling DLL from the command line. Because the Visual Studio .NET GUI environment is so powerful, it is quite possible that you have never built a program or DLL this way, so I will explain the process in detail. As you'll see, it is quite easy once you've seen an example.

Although it is possible to invoke the compiler (cl.exe) and linker (link.exe) directly from the command line, there are so many options available that developers frequently employ a utility program to manage all of them. The nmake.exe program uses a data file (usually with a .mak extension) to determine the recipe for creating your project using build tools. Again, it is possible to call the nmake.exe program directly but there are many user environment variables, such as PATH and INCLUDE, that need to be set, so it is common to write a .bat file to manage them. In short, this .bat file sets up environment variables and calls nmake.exe, which in turn reads compiler options from a .mak file then calls the compiler and linker to produce the resulting DLL.

I created the short .bat file named build.bat with the key statements shown in Figure 5. The first two SET statements tell the shell environment where required files such as mspdb70.dll and executables such as nmake.exe reside. You will probably have to modify the part of the .bat file that sets user environment variables, depending on the PATH variable your shell inherits from the system environment variables. Try to build the DLL, and if you get an error message about a missing file, find the file and add another SET path statement.

Figure 5 Build Script (Close Figure 5)
rem ** build.bat

set path=%path%;C:\Program Files\Microsoft Visual Studio .NET\Common7\IDE
set path=%path%;C:\Program Files\Microsoft Visual Studio .NET\VC7\bin

set include=%include%;C:\Program Files\Microsoft Visual Studio .NET\VC7\include
set include=%include%;C:\Program Files\Microsoft Visual Studio .NET\VC7\PlatformSDK\Include
set include=%include%;C:\Program Files\Microsoft Visual Studio .NET\FrameworkSDK\include

set lib=%lib%;C:\Program Files\Microsoft Visual Studio .NET\VC7\PlatformSDK\lib
set lib=%lib%;C:\Program Files\Microsoft Visual Studio .NET\VC7\lib

nmake /nologo /a /f profilerFF.mak
echo Profiler DLL for code coverage has been built

The next three SET statements tell the shell where to look for files such as cor.h that are found in the various C++ #include statements. The final two SET statements tell the shell where library files used by the linker are located. Again, you may have to modify or add statements depending on your environment.

Finally, nmake.exe is called with the compiler and linker options specified in ProfilerFF.mak. The ProfilerFF.mak data file consists of 14 file references such as $(INTDIR)\ProfilerFF.obj, which specifies the location and name of the intermediate .obj file created during compilation.

With everything ready to go, you can build the code coverage profiling DLL from a command shell using this command:

C:\Coverage\FFprofiler>build.bat
You can see the shell command in Figure 1. Because nmake.exe looks for its makefile in the current directory, you need to invoke build.bat from the FFprofiler folder. It is unlikely that the build will work on your first attempt, but a few edits to the build.bat file as I've described will lead to a successful build.

Preparing for Code Coverage

Before you enable the code coverage profiler, you need to specify which .NET assemblies to include in the coverage monitoring, and determine which methods in those assemblies you want to watch. The profiling DLL will create a list of all methods that were actually entered, and you can compare it to the list you provided to determine your code coverage percentage.

My system manages these lists by storing data in simple text files. The first step is to manually create an ffcoverAssemblies.txt file and save it in the Coverage\FFdataFiles folder. For the example shown in Figure 1, Figure 2, and Figure 3, the contents of this file are:

C:\Coverage\AppsToTest\DummyApp\bin\Debug\DummyApp.exe
C:\Coverage\AppsToTest\DummyApp\DummyLib\bin\Debug\DummyLib.dll
These are the .NET assemblies that make up my system under test. Determining which assemblies are referenced in a large product can be a difficult task. You can use the .NET Framework Configuration tool, mscorcfg.msc, to determine assembly dependencies. You can also write a little utility program that uses the System.Assembly.GetReferencedAssemblies method to find dependencies.

After you have created the ffcoverAssemblies.txt file, the next step is to find all the methods, constructors, and properties in the assemblies. I put this code, along with code to do the reporting, in one project named FFcover. Here's where the powerful methods in the System.Reflection namespace come into play. The algorithm is shown in Figure 6 in C#-like pseudocode.

Figure 6 Find Existing Classes, Methods, Properties (Close Figure 6)
open file ffcoverAssemblies.txt for reading
open file ffcoverMethods.txt for writing
for each assembly in ffcoverAssemblies.txt loop
{
  get all classes in the assembly
  for each class loop
  {
    get all constructors
    for each constructor
    {
      get all parameters
      write assembly.class::constructor(params) to output
    }

    get all methods
    for each method
    {
      get the return type
      get all parameters
      write returnType assembly.class::method(params)
    }

    get all properties
    for each property
    {
      get the return type
      get all parameters
      write returnType assembly.class::property(params)
    }

  } // each class
}   // each assembly

In principle this is easy, but there are a few tricks along the way. I decided to read all of the assemblies into an ArrayList container and work from it, rather than use a file-oriented approach:

ArrayList al = new ArrayList();  // assemblies under coverage
StreamReader sr = new
    StreamReader("C:\\Coverage\\FFdataFiles\\ffcoverAssemblies.txt");

string line;
while ( (line = sr.ReadLine()) != null )  // read assemblies to test
{
  Assembly a = Assembly.LoadFrom(line.Trim());
  al.Add(a);  // store assembly in array list al
}
Notice that information such as file paths is hardcoded and error checking has been omitted; this has been done for simplicity and clarity only. In a production system, plenty of error checking would need to be added and the hardcoded strings should instead be passed as parameters.

I get the types (classes, enumerations, and arrays) in each assembly using the following loop:

foreach (Assembly a in al)
{
  Type[] types = a.GetTypes();
•••
Then for each class I get the constructors, methods, and properties that belong to the class. Because the code for getting constructor, method, and property information from a class is similar, let's look at the code for methods only. You start by setting up a BindingFlags variable that acts as a filter for the kinds of methods you want to catch, as you can see in the following code:
foreach (Type type in types)
{
  BindingFlags flags =
  BindingFlags.Public |
  BindingFlags.NonPublic |
  BindingFlags.Instance |
  BindingFlags.Static |
  BindingFlags.DeclaredOnly;
•••
Depending on your system, you may want to modify these flags to include or exclude different kinds of methods. Next, you get all methods in the current class into an array:
MethodInfo[] methodinfos = type.GetMethods(flags);
You then iterate through each method, building up its signature in the following format:
return-type Assembly.Class::MethodName(param1,param2,. . . )
Starting with an empty string, you append the return type, a blank space, the full name of the Type (for example, the class name preceded by namespace), and the method name:
foreach (MethodInfo method in methodinfos)
{
  string s = "" + method.ReturnType.FullName + " " + type.FullName +
      "::" + method.Name + "(";
•••

After you get the method return type and name, you must get the parameter types. Remember that this is necessary because with overloading you can have multiple methods with the same name and return types, but with different parameter lists:

ParameterInfo[] parameterinfos = method.GetParameters();
foreach (ParameterInfo p in parameterinfos)
{
  if (p.ParameterType.FullName != "System.Decimal" &&
      p.ParameterType.FullName != "System.EventArgs")
  {
    s += p.ParameterType.FullName + ",";
  }
}

During the development of this code coverage system I noticed that the coverage profiler does not capture the System.Decimal data type or System.EventArgs parameters. This required a design decision: I could either modify the profiler source code to catch these or modify the C# analysis code. I chose the second approach, mostly because I wanted to change as little as possible in the profiler. However, given more time it would have been better to change the profiler source code itself.

After the method string has been built up, there is a critical processing step:

s = Mapped(s);  // map C# type names to Profiler type names
The entire string is modified using a Mapped function that replaces the .NET return type representation (System.Int32, for example) with the return type representation used by the profiler (int32). The two representations must match because the profiler outputs its own representation of data types for methods actually hit during testing. Here's the Mapped function:
static string Mapped(string s)
{
    string t = s;

    while (t.IndexOf("System.Int32") >= 0)
        t = t.Replace("System.Int32", "int32");

  // etc.

    while (t.IndexOf("System.String") >= 0)
        t = t.Replace("System.String", "String");

    return t;

}

The mappings I used handle the C# data types because the application under test was written in C#. Depending on the system you're analyzing, you may have to add mapping statements to map other data types. The technique I used here—a while loop—isn't very elegant. If you're a fan of regular expressions, you can recast the Mapped function using them instead. A table lookup would also be appropriate.

Finally, you strip off any trailing comma characters, append a closing parenthesis character, and write the result to the ffcoverMethods.txt file in the FFdataFiles folder:

if (s[s.Length-1] == ',') s = s.Remove(s.Length-1,1);
s += ")";

Console.WriteLine(s);
sw.WriteLine(s);
sw.Flush();

Running Tests Under Code Coverage

After the code coverage profiling DLL has been built, and information has been gathered about which methods in the product should be hit while testing, you can run a test or a suite of tests under code coverage. The steps are easy: enable the code coverage environment, run a test or a suite of tests, and disable the code coverage environment.

The code coverage environment is activated with a simple enable.bat file that registers the profiler and sets three shell variables. The key statements in enable.bat are:

@set DBG_PRF_LOG=0x1
@set Cor_Enable_Profiling=0x1
@regsvr32 /s debug\ProfilerFF.dll
@set COR_PROFILER={5AC86959-7927-4177-9802-A540F1967874}
The first set statement tells ProfilerFF.dll to send output to the log file specified in the ProfilerInfo.cpp file. Recall that this file is named ffcoverXXX.log, where the Xs are a timestamp. The second statement sets the value of the environment variable that the profiler checks to determine if it should profile or not (0x1 represents true). The third statement registers the profiling DLL. The last statement sets an ID value for the profiler that is run-ning in the current environment. It is the GUID specified in the ProfilerCallback.h file.

After calling enable.bat, you simply run a test or a suite of tests just as you would normally. Your tests can be manual or automated. All .NET activity is monitored and when the application terminates, the information about which methods were actually entered are saved as an ffcoverXXX.log file in folder FFdataFiles. Here are a few lines of code from the example shown at the beginning of this article:

0x00000c58;void DummyApp.Form1::Main()
0x00000c58;void
    DummyApp.Form1::button2_Click(Object)
0x00000c58;void
    DummyLib.Dummy::.ctor(int32,int64)
0x00000c58;void
    DummyLib.Dummy::Bar(int32&,float64&,String[])
Each line of data is preceded by an arbitrary thread ID in case your application is multithreaded. Notice that parameters such as int32 are in profiler format and not in .NET format (System.Int32).

If you run a second test, a second log file with a different timestamp will be saved. This is a common scenario in code coverage—you run a sequence of tests to obtain the cumulative coverage ratio. After you are finished running your tests, you want to turn off code coverage. You do this with a disable.bat file that has just one important statement:

@set Cor_Enable_Profiling=0x0
This statement effectively instructs the profiler not to monitor activity anymore. The relationship between the various parts of the Fundamental Function code coverage system are illustrated in the diagram in Figure 7.

Figure 7 Code Coverage System
Figure 7 Code Coverage System

Reporting Code Coverage

After you have built a profiling DLL, created a list of all methods in the system under test that can potentially be entered during testing, and produced a list that contains all the methods that were actually entered, you are ready to create a results report. I organized my reporting code algorithm around three hash tables:

Hashtable ht1 = new Hashtable();  // holds all potential methods
Hashtable ht2 = new Hashtable();  // holds methods actually hit
Hashtable ht3 = new Hashtable();  // holds methods missed
In pseudocode, the algorithm looks like this:
load ht1 from ffcoverMethods.txt with methods that could be hit
load ht2 from all *.log files with methods actually hit

for each method in ht1 loop
{
  if method is not in ht2, add method to ht3
}

percent coverage = ht2.Count / ht1.Count

Loading the first hash table with names of methods that could be entered is easy because I already have this data in the ffcoverMethods.txt file. I simply open the file for reading, traverse through it, and add them like so:

if (!ht1.ContainsKey(line)
  ht1.Add(line, line);

Loading the second hash table with the methods that were actually entered during the test requires two steps. First I examine the FFdataFiles folder to find all files with names in the form of ffcoverXXX.log form, as shown here:

DirectoryInfo dir = new
    DirectoryInfo("C:\\Coverage\\FFdataFiles\\");
FileInfo[] files = dir.GetFiles("ff*.log");
foreach (FileInfo file in files)
{
  al1.Add(file.FullName);
}

I then take each file, open it for reading, and add data lines to the hash table. The main processing loop is shown in Figure 8.

Figure 8 Adding Methods Touched (Close Figure 8)
foreach (string fn in al1)  // for each log file
{
  StreamReader sr2 = new StreamReader(fn);
  while ((line = sr2.ReadLine()) != null)  // add methods hit to ht2
  {
    if (!line.StartsWith("0x"))
      continue;

    string[] tokens = line.Split(';');
    if (!ht2.ContainsKey(tokens[1]))
      ht2.Add(tokens[1], tokens[1]);
  }
  sr2.Close();
}  // for each log file

Because I stored a thread ID along with method data, the StartsWith method makes it easy to tell when I've hit a line of data. When I'm at a data line, I use the String.Split method to parse out the method data from the thread ID, check to make sure that method hit is not already in the hash table, and then store it into the hash table.

Although not absolutely necessary, I like to determine which methods were not hit. This gives me feedback to help design new tests that should be added to increase coverage. To do this, I iterate through the first hash table of methods that could be hit and if any method is not in the hash table of methods actually hit, it must have been missed. Because hash tables are designed for quick data lookups, you might not have seen the code to traverse all data in a hash table before:

foreach(DictionaryEntry e in ht1)
{
  if (!ht2.ContainsValue(e.Value))  // if not in hit table
    ht3.Add(e.Value, e.Value);      // it was missed
}

Finally, I do a little math to determine the percentage of methods hit, as shown in the following code:

int numtotal = ht1.Count;
int numhit = ht2.Count;
int nummissed = ht3.Count;
double percent = (double)numhit/(double)numtotal * 100;
percent = Math.Round(percent, 0);
I now have all the information I need to produce a code coverage analysis report.

Before and After .NET

If you know a little bit about code coverage in environments other than .NET, it's easier to understand how the Fundamental Function code coverage technique presented in this article fits into the overall production cycle. The main difficulty in traditional environments is that you cannot work with normal product code. You must first obtain a special build of the product you want to analyze. Then you must process that release, inserting code hooks to notify the profiler when methods or blocks of statements have been entered. This process is usually called instrumentation. This is not only a serious technical challenge, but on large products the management issues are difficult too.

Because all .NET code is executed in the CLR, you can trap method entries there using code built in the usual way, as demonstrated in this article. The overall result is that .NET code coverage is much easier to perform. In large projects before .NET, you would typically assign one full-time engineer just to manage the code coverage process, and even then results were inconsistent at best. The code coverage system presented in this article has been used successfully on several large .NET projects and is simple enough to be used by team members on an ad hoc basis.

In terms of the overall product development cycle, because Fundamental Function code coverage is so easy to perform, you can employ it earlier and more often in the development cycle. Pre-.NET code coverage is normally so time-consuming that it isn't feasible in early stages of development when the product is unstable (just when you need code coverage the most). Additionally, the simplicity allows daily code coverage analysis as part of the code check-in and build processes. In many product groups, before checking in new code, developers run a small set of developer regression tests to ensure that the new code has not broken old functionality. It is also very common for each new build of a product to be subjected to a set of build verification tests to ensure basic functionality. In both cases, these sets of tests can be analyzed with code coverage analysis to determine their effectiveness, even in a rapidly changing environment.

Compared with code coverage techniques before .NET, code coverage with Fundamental Function coverage does have some disadvantages. First, the coverage is not as granular as other techniques. I have found that this is not a serious drawback. By breaking up large methods into smaller methods (which is good design anyway), relatively little coverage information is lost. Additionally, I have noticed that the typical method in my .NET project tends to have approximately 25 percent fewer lines of code because of various efficiencies. Second, the ad hoc nature of the technique presented here means you don't get detailed reports. In my experience, this is a small price to pay for the greatly increased flexibility that this system provides. It's better to get very good code coverage today than very, very good code coverage tomorrow.

Conclusion

Even though code coverage analysis is a critical part of testing any significant software product, it is often so time-consuming in development environments before .NET that it is simply not done. Without a code coverage analysis, you can never be sure that your test cases touch all the product code. It is not unusual for a test team to believe they have full coverage, then run code coverage analysis and discover that their initial suite of tests hits less than 75 percent of the code base. In the old days, it may have been acceptable to push the product out the door and worry about additional testing later, but with the increased focus on security, this is no longer a viable option.

James McCaffreyworks for Volt Information Sciences Inc., where he manages technical training for software engineers working at the Microsoft Redmond, WA campus. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Test-Driven C#: Improve the Design and Flexibility of Your Project with Extreme Programming Techniques

Test-Driven C#

Improve the Design and Flexibility of Your Project with Extreme Programming Techniques

Will Stott and James Newkirk
This article discusses:
  • A comprehensive description of test-driven development
  • The benefits of TDD and extreme programming
  • An introduction to NUnit
  • A comparison with traditional development techniques
This article uses the following technologies:
Visual Studio, C#, Testing, Development Techniques
Code download available at: ExtremeProgramming.exe (127KB)
Contents
Why Designs Degrade
A Simple Introduction
Lessons Learned
Automating Your Testing with NUnit
Taking NUnit for a Test Spin
Test Cases
Test Fixtures and Suites
Using TDD in the Real World
Developing a Test and Creating a Library
Creating the GUI
Conclusion

W
ouldn't you just love to create code that becomes easier to work with as the project matures rather than more difficult? It seems that no matter how much care you take, sooner or later, your code will become a maze. The bigger the project, the worse this problem gets. How many times have you started a project with a near-perfect design only to see it unravel once coding starts?

Test-driven development (TDD) alters the process of writing code so that change is not only possible, but desirable. Development revolves around three basic activities: writing a test, writing code to pass the test, and refactoring the code to banish duplication to make it simpler, more flexible, and easier to understand.

This cycle is repeated frequently, each time running all the tests in order to ensure that the product is kept in a working state. The long gaps between the design, coding, and testing phases are gone, thus making for a much better learning environment. Therefore, your design (and code) actually improves rather than degrades as the project matures.

What makes TDD so effective is the automation of your programmer (unit) tests, and the great news is that the only tools you need are available for free on the Internet. These are not reduced functionality versions of a commercial product, but high-quality software made available by fellow developers. This article explains how you can obtain and use NUnit to practice TDD for development with C# (or any Microsoft® .NET Framework-based language). Note that similar tools are available for C++ and Java developers, as they are for most other languages and operating systems. The ready availability of such tools gives TDD a universal appeal which, combined with its close association with extreme programming, has done much to encourage its use.

Why Designs Degrade

Most traditional software processes are based on the assumption that you can get the design right in the beginning and then pass it through the development machine to generate a perfect product. This is a production-line mentality that values uniformity and minimizes variation. Such processes don't value communication and feedback in the same way that TDD does, so they are less effective at both generating information (failing tests) and allowing people to learn from it (fixing the design).

Why can't you get the design right from the beginning? Because at the start of the project you have imperfect knowledge about the software you're developing. Iterative development acknowledges this fact and helps you identify significant problems earlier in the project rather than leaving them all to be discovered at the end. However, the iteration will not be canceled to let you return to the design phase to address a simple problem like a badly named public class. Nobody wants to hear such minor concerns and the process is unfortunately often designed to suppress them because the cost of moving between phases is too high.

The accumulation of these small issues causes the real difficulties in many traditional development processes. You will always have more important priorities than making changes that offer no functional advantage. However, the longer the badly named class remains in the code base the more prevalent its usage becomes, and the bigger the task of changing it. After a while, the team will begin fixing such problems unofficially while coding, so soon you will be planning an entire release that does little else but attempt to make the code correspond to the design documentation, or vice versa. At this stage, your up-front design is worthless—it is the code that speaks.

Test-driven development allows you to defer decisions until you understand the problem better. You don't need to produce a perfect architectural design up front at a time when you have the least knowledge about how the product will develop. This challenges most established ideas about software development and is to some degree counterintuitive, so we suggest you try TDD with an open mind and discover its power for yourself.

There are also benefits to TDD that are much easier to grasp. It encourages communication by making your programs self-documenting—the set of tests you develop show how the code works in a way that a manual can't. It encourages feedback by asking you to view your code from the perspective of someone writing a test for it, which helps you create objects and components that are more loosely coupled and cohesive. It encourages simplicity by letting you defer big decisions so you can concentrate on getting the little ones right. Finally, it gives you the courage to push the boundaries of your code by giving you a set of tests that tell you immediately when something breaks.

A Simple Introduction

The only way to really understand TDD is to do it, so let's start with a very simple example that doesn't require any special tools. We're going write a small program to help us plan a rectangular lawn for a home, but before we start there are two things we will want to test: calculate the area as 6 units when the length is 3 units and the width is 2 units, and also calculate the perimeter as 10 units for a length of 3 units and width of 2 units.

Writing down these tests helps focus our attention on what is important in our program. In this case, it seems reasonable to develop an object that will help model the problem domain, so we'll create a class called Quad, named for the quadrangular lawn we're trying to construct, and instantiate it as an object in a simple console application. Here are the steps to follow along:

  1. Start up Visual Studio® on your desktop, select File | New | Project, then choose "Console Application" from the list of C# project types. Type "QuadApp" into the Project Name box and press OK.
  2. Type into the member function Main a few lines of code to create an instance of Quad and then assert that when the object Area method is given a length of 3 and width of 2, it returns the value 6:
     static void Main(string[] args)
 {
     Quad q = new Quad();
     System.Diagnostics.Debug.Assert(q.Area(3,2) == 6);
 }
  3. Select Project | Add Class, type the name "Quad.cs" into the dialog box, and press Open to create the class.
  4. Type the code required to give Quad a member function Area as required by its use in Main:
     public class Quad
 {
     public int Area(int length, int width)
     {
         return 0;
     }
 }
  5. Select Build | Build QuadApp from the menu and this time the build should succeed. However, when you run the program (Debug | Start) it asserts because Area doesn't return 6.
  6. Hardcode a return value of 6 for the function Area, rebuild the program, and run it again. This time the program runs without asserting.
  7. Improve the implementation and design of Area by returning the product of its input parameters rather than hardcoding a return value. Check that you haven't broken anything by rebuilding the program and rerunning the test.

The second test (calculate the perimeter as 10 units) works in a similar way. Start by writing the test using assert in Main and then follow Steps 4 to 7, but for Perimeter rather than Area. When you're trying to improve the design (Step 7), you will probably reach the conclusion that perhaps you should be passing the length and width to the constructor and storing them as properties of the object rather passing them as parameters to the member functions. Change Quad so that it stores length and width as properties (shown in Figure 1), rebuild the program, and then run it to make sure that you haven't broken anything.

Figure 1 Quad After Some Refactoring (Close Figure 1)
public class Quad
{
    private int m_length;
    private int m_width;

    public Quad( int length, int width)
    {
        m_length = length;
        m_width = width;
    }
    public int Area()
    {
        return m_length * m_width;
    }
}

We have just completed an exercise that demonstrates TDD in its very simplest form. Here's a summary of each step involved:

Write a test that fails We selected the test that seemed the easiest to implement (although in this case they were all simple) and wrote it. Next, we wrote the simplest implementation of Quad in order to get the program to compile. However, when we executed the program, it asserted because Area didn't return the value 6.

Fix the code so you pass the test To fix the code, we performed the simplest task and hardcoded a return value of 6. The objective was just to get the program working so it passed the test. However, the next test might use different parameter values, forcing us to implement a proper algorithm in order to pass both tests, even if we didn't refactor as described next.

Refactor the code Since we achieved the correct observable behavior in our program, we tried to make the code more maintainable by removing duplication, making it simpler, more flexible, and easier to understand. We checked that this change hadn't altered the desired behavior of the program by rerunning the test.

The idea of reworking your code to improve its maintainability without changing its observable behavior is not new. However, there is more to refactoring than just tidying up your code from time to time. Refactoring must become a key part of your software development activity, done in a methodical way using appropriate tools so as to progressively improve the quality of your code. The aim is to achieve perfectly maintainable software by performing many small refactorings of the code throughout the product's life.

Refactoring is an essential step in test-driven development, providing the necessary feedback that allows your design (and code) to improve as the product develops. You can, however, refactor without performing TDD and still banish duplication while making the code simpler, easier to understand, and more flexible. You just need to make refactoring a regular activity in whatever development process you are following and ensure that your changes are validated with appropriate regression testing.

The power of refactoring lies in its ability to reduce the danger inherent in making changes to working code. Good tools can help address some of the potential perils, but equally important is taking a series of small steps and having a structured and disciplined approach. Martin Fowler's book Refactoring: Improving the Design of Existing Code (Addison-Wesley, 1999) gives a good introduction to these issues by providing a useful catalogue of patterns for refactoring and guides you through the refactoring process by offering several relevant examples.

At present, there is not much support for refactoring in Visual Studio .NET, apart from using Find and Replace. However, more powerful tools are forthcoming so that you can do smart things like symbolic renaming, and changing the name of a class at the level of the compiler-generated parse tree rather than just performing textual substitution in your source files. You can look forward to the day when you can select a piece of code, apply a refactoring pattern from a menu provided by Visual Studio, and then continue with your work—confident that you've just made the program more maintainable without introducing any bugs or changing its observable behavior.

Lessons Learned

The most important lesson you should have learned so far is that TDD is simple. In fact, TDD is a great for obtaining a better understanding of any new language, technology, or component that comes your way. It allows you to decide the size of steps you need to take. An expert might have taken bigger steps and, using his experience, avoided some of the intermediate steps that we took. When things start to go wrong, however, you always have the option of going back and doing it again, taking smaller steps. There are number of other things you may have noticed about TDD:

  • The tests document the code. We can see exactly what Quad does by looking at the tests.
  • We can measure progress by the tests passed. Each piece of functionality is proved by a test, so anyone can run the tests to see that claims of progress are well founded.
  • The tests make everyone confident about changing the code. Even though you might be a novice C# programmer working on a class written by an expert, running all of the tests will reassure you that you haven't broken anything.
  • Mistakes, like passing the length and width as parameters to Area and Perimeter rather than to the constructor, are corrected by refactoring.
  • When you apply TDD in the real world, you're going to generate lots of tests, so you need to organize your testing using a tool like NUnit.

Automating Your Testing with NUnit

A TDD project might generate thousands, or even hundreds of thousands, of tests. Given that everyone on the team will be going through the "write a test, fix the code, refactor" cycle frequently, it's essential that you can both write and run tests efficiently. A test framework like NUnit is designed to help you do this. It allows you to arrange your test cases into individual projects that can be loaded into NUnit in the same way that you create projects in Visual Studio. You can also display all the project's test cases in a hierarchy, run tests individually or as a suite, and see the result of their execution as pass (green) or fail (red) with detailed information about each failure (see Figure 2).

Figure 2 Test Status in NUnit
Figure 2 Test Status in NUnit

Additionally, NUnit provides special functions you can execute at the start of a test suite and upon its completion to initialize and clean up "test fixtures" (such as log file, database connection, comm link) shared by various test cases. You can also define a Setup function to be run before a test case is executed and a TearDown function to be run after it completes. This helps you isolate test cases from each other by resetting the system between each one.

NUnit lets you write a test case in the same language and environment as your application. With no special test language to learn, developing tests is quick and straightforward. There is also a fully featured Assert class that will provide information about why a particular test failed.

Finally, NUnit can be run as a console application that simply writes its results to the command prompt. This allows you to automate a formal build process so you can, for example, rebuild the application overnight and run the full suite of tests. It is also possible for the NUnit console application to produce an XML log of the test results.

People with a background in software testing will appreciate the value of these features, but even if you don't have this sort of experience, it should be clear that NUnit has much to offer you as a developer (see the sidebar "Testing Terminology"). Nevertheless, expert or novice, the best way to find out how NUnit helps you organize your testing is to try it out. You can download NUnit from http://www.nunit.org. Precise details of what you are permitted to do with the product are given in its license file.

I used the NUnit version 2.1 beta release for this article, but you should download the MSI file for the latest production release (approximately 1.5MB). Once this file is on your PC, you just need to double-click it to start the Windows® Installer. It works with both version 1.0 and version 1.1 of the Microsoft .NET Framework, though you should refer to the NUnit documentation for more information on system requirements.

After installing NUnit, select the submenu called Test in order to run the suite of tests that confirm that the product is working properly. Just click the menu option, wait for NUnit to open on your desktop, and then press the Run button.

After less than a minute, all the nodes in the treeview should be green, thus indicating that the tests have been executed and passed. The number of tests run is shown in the status bar, together with the time taken to execute the tests, and the number of failures. In the unlikely event that you get some red nodes, indicating failed tests, you may need to reinstall the product or get some advice from the NUnit.org site.

Taking NUnit for a Test Spin

NUnit comes with a number of samples to get you started, so let's walk through a development episode using one of them, namely, the Money project.

Start up Visual Studio on your desktop, select File | Open | Project, and select the file Money.csproj, which is in the NUnit installation directory (Program Files\NUnit) in the subdirectory src\samples\money. Press Open and wait for the project to load in Visual Studio. In the Solution Explorer window, look in the References folder and check for nunit.framework. Build the project Money by choosing Build | Build Money. The build should succeed without any errors or warnings.Testing Terminology

Test case (Test) A unit of test in the same way that a function is a unit of code. A test case usually contains the test object, the various fixtures it needs, and the statements and assertions that make up the test.

Test environment The (known) state of the system upon which you are running the tests. It is usually a simpler and more controllable version of the production environment in which the product will eventually operate.

Test fixture (test scaffolding) An object shared by one or more test cases concerned with initializing the test object or providing it with some resource.

Test framework A library that facilitates the writing and running of test cases, such as the various classes provided by nunit.framework.dll.

Test object The object being tested, typically an instance of the class you're developing.

Test script A file containing one or more test cases and any other information that may be required for creating a test program, such as the source code files of the test program.

Test suite A collection of test cases (or suites) that are associated with a particular component, assembly, or area of the product. When you run a full test suite, you are running all the application's test suites (the test cases).

Test tool An application concerned with running test cases and suites, such as the NUnit application.

Start up NUnit on your desktop. The treeview should be empty—if not, close any existing project by selecting File | Close. Open the file Money.csproj by choosing File | Open in much the same way you opened it in Visual Studio. The treeview should now contain a collection of test cases.

Select the root node in the treeview (the file Money.csproj) and click the Run button in NUnit's right-hand pane. The full suite of test cases will then be executed and you should have just one failing test—MoneyBag Equals.

Select the test case MoneyBag Equals (the red leaf node) and locate the problem line in each file. This information is given in the lower part of NUnit's right hand pane, shown in Figure 3.

Figure 3 NUnit Error Tracing
Figure 3 NUnit Error Tracing

Switch to Visual Studio and open the offending file at the appropriate line. Comment out the line (insert // at the start of the line) and rebuild the program (Build | Rebuild Money). The build should succeed without any errors or warnings.

Now switch to NUnit and note that the treeview's nodes are all grey, indicating that the test cases have not yet been run for the program you've just updated. Then select the root node in the treeview and click the Run button again. You should observe that all the tests are running without failure.

This is just about all there is to working with NUnit. You are now ready to start writing test cases for your own projects.

Test Cases

The test case applies a defined set of inputs and also measures a de-fined set of outputs in order to validate that some program characteristic has been implemented without introducing any bugs. This is true for any kind of test that you might want to write.

First of all, a test case should concern itself with validating just one aspect of the program's behavior and it should either pass or fail. A test may fail because of a bug in the product, but it can also fail due to a bug in the test itself—a false negative. Similarly, a test may pass due to a bug in the test rather than because the product has behaved correctly—a false positive. Good tests should obviously avoid false negatives and should never yield false positives. In addition, a good test suite will not only test valid scenarios but will also test error cases.

The test object and its environment should always be in the same state before a test case is executed so that it can be repeated as often as necessary and give consistent results. When a test case fails you should be given adequate information about the location of the failure and its reason so that the problem can be corrected, which is why you want each test case to validate just one thing.

Now let's apply this theory by writing a few simple test cases for the Quad example. Open the Quad project in both Visual Studio and NUnit the way you did for the Money project—select File | Open and locate QuadApp.csproj. Add to the project the reference to nunit.framework by selecting Project | Add Reference, browsing for the file nunit.framework.dll (located in the NUnit installation directory within the subdirectory bin), and pressing OK.

Use Visual Studio to create the class QuadTest: select Project | Add Class, enter the name QuadTest.cs, and press Open. Enter the code shown in Figure 4 into QuadTest, build it (Build | Build QuadApp), and then run the tests using NUnit (press Run). Click the Console.Out tab in the right-hand pane to see the execution order.

Figure 4 QuadTest Class (Close Figure 4)
using System;
using NUnit.Framework;

namespace QuadApp
{
    [TestFixture]
    public class QuadTest
    {
        [TestFixtureSetUp]
        public void DoTestSuiteSetUp()
        {
            Console.WriteLine("TestSuiteSetUp");
        }

        [TestFixtureTearDown]
        public void DoTestSuiteTearDown()
        {
            Console.WriteLine("TestSuiteTearDown");
        }

        [TearDown]
        public void DoTestCaseTearDown()
        {
            Console.WriteLine("TestCaseTearDown");
        }

        [SetUp]
        public void DoTestCaseSetup()
        {
            Console.WriteLine("TestCaseSetup");
        }

        [Test]
        public void Area()
        {
            Quad q = new Quad();

            Assert.IsTrue( q.Area(2, 3) == 6);

            Console.WriteLine("Area");
        }
        [Test]
        public void Perimeter()
        {
            Quad q = new Quad();

            Assert.IsTrue( q.Perimeter(2, 3) == 10);

            Console.WriteLine("Perimeter");
        }
    }
}

You can now see how the various Setup and TearDown functions might allow you to control the test environment so that your test cases can be repeated as often as necessary and give consistent results. These functions help you ensure that the running of one test case doesn't affect another (test case isolation). A good proof of test isolation is the ability to run your test cases in any order. Try changing the order of your test cases to confirm their isolation.

Test Fixtures and Suites

An NUnit test fixture is an object shared by one or more test cases concerned with initializing the test object or providing it with some resource. In terms of NUnit, a test fixture is a class with the attribute [TestFixture] whose methods provide:

  • [Test] functions that form the various test cases. These test cases should be atomic operations and should have no dependencies on other tests.
  • [SetUp] and [TearDown] functions needed to reset the environment between test cases. They are each run once at the beginning and end of each test.
  • [TestFixtureSetUp] and [TestFixtureTearDown] functions required for objects shared by test cases. They are each run once at the beginning and end of the test fixture.

Any objects that you need to run the tests can be created as instance variables of the [TestFixture] class (shared by all test cases) or local variables of the methods (private to an individual test case), and this is why it is described as a test fixture. However, you can also think of NUnit's [TestFixture] as a way of organizing a test suite as each class forms an individual branch in the treeview under the name of the project.

You should now be able to start practicing TDD as you can write suites of tests for your own projects and can use NUnit to run them. Let's look now at some of the issues you might encounter doing more serious development work.

Using TDD in the Real World

One of the first things to consider when using TDD since a commercial project is deciding how to organize your programs so that the production code can be easily separated from the code used to test it. You want to run your programmer tests throughout the product's development and yet be able to easily remove them for the purposes of releasing the code.

Another issue you might encounter is the difficulty of testing GUI applications driven by mouse and keyboard input. For example, how do you write a test that simulates a user clicking a dropdown list and then verifies that it is populated with a given list of country names?

The answer to both of these problems lies in dividing the code into appropriate components that can be built, tested, and deployed separately. For example, rather than building Quad as a class contained in the same executable file as the main application, it could have been contained in a separate library (.dll). This would have allowed us to develop both the test program and the domain program as separate executables (.exe) that shared a common library (.dll) containing Quad. Note that if you're only building a library, NUnit and your test suite can be used as the interface, rather than requiring a separate harness.

The idea of keeping the main program very simple and putting the business complexity into classes contained in a library can also help solve the problem of testing GUI applications. One of the rules of TDD is that you don't test third-party code, so there is no requirement for you to test the GUI framework classes, although it's sometimes useful to test interfaces, particularly if they're a bit flaky. This means you can catch the user event in a class that you know works and then pass it through for processing to the class you're developing. Again, you can separate the test program and the domain program into separate executables sharing the common libraries that form the bulk of your development effort.

Let's see how this might work in practice by using TDD to develop a combobox list for a simple forms application.

Developing a Test and Creating a Library

Create a new console application called CountryTest by selecting File | New | Project in Visual Studio. Add to the project the reference to nunit.framework as you did when writing the first test case. Create the class CountryBoxTest by selecting Project | Add Class, entering the name CountryBoxTest.cs, and pressing Open. Enter the code shown in Figure 5 into CountryBoxTest and attempt to build it. The build fails because the class CountryList hasn't been written yet.

Figure 5 CountryBoxTest Class (Close Figure 5)
[TestFixture]
public class CountryBoxTest
{
    [Test]
    public void CheckContent()
    {
        CountryLib.CountryList list = new
            CountryLib.CountryList();

        Assert.AreEqual("UK", list.GetCountry(1));
        Assert.AreEqual("US", list.GetCountry(2));
        Assert.AreEqual("CH", list.GetCountry(3));
        Assert.AreEqual(null, list.GetCountry(4));
    }
}

Use Visual Studio to create a new class library called CountryLib by selecting File | New | Project (add to the existing solution). Create the class CountryList by selecting Project | Add Class, entering the name CountryList.cs, and pressing Open. Enter the following code into CountryList and build the library:

public class CountryList
{
    public String GetCountry(int No)
    {
        return null;
    }
}

First add to the project the reference to CountryLib.dll and rebuild the console application, CountryTest. This time it works because the CountryList has now been written.

Open the project CountryTest.csproj in NUnit ( File | Open) and run the test. It fails because CountryList.GetCountry returns null. Switch to Visual Studio and in CountryList fix GetCountry so it returns the values required by the test—just add some initialized strings as instance variables, as shown in Figure 6. Do the simplest possible thing to get the program working. Rebuild the CountryList library, switch to NUnit, and then rerun the test—it should now run successfully.

Figure 6 CountryList After Some Refactoring (Close Figure 6)
public class CountryList
{
    string one = "UK";
    string two = "US";
    string three = "CH";

    public string GetCountry(int No)
    {
        if ( No == 1 )
            return one;
        if ( No == 2 )
            return two;
        if ( No == 3 )
            return three;
        else
            return null;
    }
}

Do any refactoring of CountryList that you think is necessary and rerun the test to check that you haven't broken anything. You might think that storing the strings in an ArrayList would make the code simpler, easier to understand, and avoid duplication, but you might also choose to defer this decision for now.

Creating the GUI

Use Visual Studio to create a new Windows-based application called CountryApp by selecting File | New | Project (add to the existing solution). Add to the project the reference to CountryList.dll. Next, add a combobox to the application by dragging one from the toolbox onto the main form. Enter the code shown in Figure 7 into CountryApp and build the application.

Figure 7 CountryApp Class (Close Figure 7)
private void InitializeComponent()
{
    ...

    CountryLib.CountryList list = new CountryLib.CountryList();

    String country;
    int index = 1;

    while ((country = list.GetCountry(index++)) != null)
       this.comboBox1.Items.Add(country);

    ...

}

You now have the basic organization for your project that will allow you to create the GUI application by developing your CountryList library using TDD techniques. Think about how CountryList could process user interface events (mouse click, keyboard) in a way that could be both tested by CountryTest and used by CountryApp. You will probably come up with a much better solution for CountryList than the one presented here, which is precisely the point we want to make: test-driven development enables the design to improve as it matures. You don't need to produce a perfect design up front, which is good news for developers.

Conclusion

Most designs are created from the top down and are essentially an exercise in understanding and solving a problem by classifying things into categories based on observable traits—in other words, we try to create a hierarchy of objects that model the problem domain. In contrast, TDD is performed from the bottom up by sequentially applying a series of simple solutions to small problems that eventually evolves into a design. Refactoring ensures that the design converges toward a good solution rather than diverging away from it. If you think this is radical, you're right. TDD has the potential to change the way software is developed in this decade in much the same way object-oriented techniques did in the last.

A note on NUnit: when using Visual Studio to open some of the samples and pre-built tests that are distributed with NUnit, you may discover that the reference to nunit.framework.dll is shown as "not found," so look in the Solution Explorer window, References folder. In this case you need to delete the existing reference (select the file, right-click | Remove) and then add it again (select References, right-click | Add Reference, browse to locate the file, and press OK). The file nunit.framework.dll can be found in the NUnit installation directory within the subdirectory bin.

Will Stott is a freelance consultant living in Zurich, Switzerland. He is also an associate of Exoftware, a Dublin-based software company specializing in the introduction of extreme programming into organizations. Reach him at wstott@exoftware.com.

James Newkirk is the development lead for the Microsoft Platform Architecture Group team. He is the co-author of Test-Driven Development in Microsoft .NET and Extreme Programming in Practice. He can be reached at jamesnew@microsoft.com.


Instrumentation: Powerful Instrumentation Options in .NET Let You Build Manageable Apps with Confidence

New information has been added to this article since publication.
Refer to the Editor's Update below.

Instrumentation

Powerful Instrumentation Options in .NET Let You Build Manageable Apps with Confidence

Jon Fancey
This article discusses:
  • Instrumentation in the managed world
  • What instrumentation can help you measure
  • Performance counters, WMI, tracing, and other measuring techniques
  • Choosing the best instrumentation for your purposes
  • Security considerations
This article uses the following technologies:
WMI, .NET, Visual Basic .NET
Contents
The Landscape
What to Measure and How to Plan
New Possibilities with .NET
Event Log
Performance Counters
Tracing
Structured Exceptions
WMI
The Good, the Bad, and the Ugly
Managed World
A Strategy
Security
Conclusion

I
nstrumentation allows you to determine the running state of a system once it has been deployed. This is crucially important today since the people supporting systems are unlikely to be the same people who developed them. Like good error handling, instrumentation is best done at development time. A sound instrumentation policy must be established at the beginning of the development process to determine what should be instrumented, why, where, and how.

Instrumentation can be performed using a wide variety of available technologies that provide high-quality support and diagnostic information from within an application itself. The technology I will focus on here is a mixture of core traditional Windows® facilities—event logging, debug tracing, performance counters, and Windows Management Instrumentation (WMI). I will also cover the Enterprise Instrumentation Framework (EIF) and features provided by the Microsoft® .NET Framework such as tracing and structured exception handling (SEH). The .NET Framework minimizes the overhead and makes all of this easy to use, which is a key point since business users often consider instrumentation to be a nonessential process.

The Landscape

Distributed applications are getting more and more complex. When you start looking at systems in their entirety, the complexity can be daunting. I'm going to focus on the challenges faced by enterprise designers and developers who are building complex distributed apps. Whether you're building Web sites, services, or traditional client-server applications, you'll find the task of managing these applications to be quite a challenge due to the huge invisible architectures.

Instrumentation can help by allowing you to answer questions such as: what went wrong where and how often does that happen? How do I know if an application is alive? How many business transactions are performed hourly? What's the maximum throughput that can be handled? What's the busiest time for various activities? How do the applications scale up and out? How do I know who's doing (or trying to do) what?

These questions fall into four distinct categories: performance, reliability, scalability, and security. Performance is crucial to support service-level agreements (SLAs) and reliability for Quality of Service (QoS) contracts, especially with third parties. Scalability is important for growth, including sales and marketing opportunities. Security, of course, is always critical.

One of the first things you must recognize is that errors should be anticipated. Deciding how to handle them is not a runtime-only decision, but a design decision. Leaving it until the coding stage will often lead to a weak and incoherent solution that does little to provide good instrumentation.

What to Measure and How to Plan

After deciding to instrument while you're writing your application logic, the second most important thing is to decide exactly what needs instrumenting. User details, date/time, and thread/process information (crucial for providing sequencing) are absolute essentials. You will also want to include your application name and where the event was logged (class/method names). Other common useful information includes state information, such as parameter values and current context values. You should also be careful to avoid calling information that is of little use, such as logging disk I/O activity, total system threads, and so on. You should include the essentials plus anything specific to the task your code is performing, such as database connection details (but be careful since exposing sensitive data in this way can create a security risk).

You must decide what the monitoring and support requirements are for your project and document them. This task involves the development team as well as members of your testing and support teams. You can then determine where to include instrumentation on the requirements that are determined by these three groups. You have to take a holistic approach so that instrumentation is present throughout your design. Any missed areas will be excluded from your monitoring and can adversely affect your project.

New Possibilities with .NET

If you're coming from a Visual Basic® 6.0 background, the .NET Framework probably looks like an oasis in the desert. The challenge used to be getting all the core functional requirements into an application. Instrumentation was impossible to do well without a great deal of time and effort. The solutions tended to be either too complex (tricky C++ DLLs) or overly simplistic (App.LogEvent). It was easy to introduce an unacceptable performance overhead which often led to incomplete instrumentation.

The irony of this is that you can't really tell how fast parts of an application run without instrumentation. Capacity planning (the analysis of anticipated load versus actual capability) is a key activity that is still relatively marginalized when it comes to Web-based distributed applications. It tends to be performed on the surface of the application, typically from virtual clients such as Mercury Interactive's LoadRunner or the Microsoft ACT tool. If the app starts getting slow, where is it slow? Is everything slow? That's highly unlikely, but you need to be able to prove it. Good instrumentation can answer questions like this and remove much of the guesswork.

Software development is going through a fundamental change, and developers are still trying to catch up with it. The typical development approach to Web-based applications is to divide up tasks for individual developers to build a small part of a new system and test it on individual machines. They then deploy it to a testing environment along with the rest of the team's work. This process is fundamentally flawed. Web-based applications are developed and tested on single workstations where everything performs fine. Often no regard is given to the fact that thousands of users could be hitting the application once it's live, rather than just a single user. Build verification testing can help a great deal to address this problem by prescribing early integration.

The bottom line is that complex distributed applications are unmanageable without good instrumentation. So let's take a look at what you can accomplish with instrumentation. First I'll review the traditional options and then examine the new possibilities that are available in the .NET Framework.

Event Log

The event log (eventvwr.exe) has always been part of Windows NT®. It provides a central place to log information and is divided into three main categories: application, system, and security. You can write to any of these categories, but usually you should stick to the application group since this is the most relevant. It's as easy as executing the following code in Visual Basic .NET:

Imports System.Diagnostics

EventLog.WriteEntry("MyAppSource",
                    "Details",
                    System.Diagnostics.EventLogEntryType.Information)

There are a couple of things to point out here. If you've used the Visual Basic 6.0 App.Logevent, you'll notice that you can define your own event sources at will (no long list of VBRuntime entries in the event viewer anymore). Also, it used to be a real pain to create the descriptions for events because they involved building a message compiler resource file and linking it into your application DLLs. The .NET Framework now performs this task for you. Behind the scenes, the mechanism is the same. Look in the registry under the following key:

HKLM\System\CurrentControlSet\Services\EventLog\Application\MyAppSource

If you put that previous code snippet in a Windows Application project this key will be there. It is read by eventvwr.exe (or anything else) to locate the descriptions for the events you raise when they are displayed. This points to a generic event description handler provided by the .NET Framework located at:

%WINDOWS%\Microsoft.NET\Framework\%version%\EventLogMessages.dll

This is a resource-only DLL which has a load of replaceable %1 parameters to wire up the descriptions you specify in your code to the event viewer. The registry entry is added at run time when the statement is actually executed, not at compile time. You may find, though, that you are quickly building up junk in the registry under the Application key because each time you change the source argument, you'll add a new subkey. These subkeys will remain in the registry permanently unless you deliberately remove them (another reason to think about instrumentation early in development).

Performance Counters

Performance counters were difficult to implement in the days of Visual Basic 6.0. The performance counters API was impossible to use directly from Visual Basic 6.0 due to the entry points that need to be defined in your DLLs—everything in Visual Basic was emitted as COM CoClass with four standard DLL entry points and no way to add your own. The typical setup was to use a helper DLL that the performance monitor (perfmon.exe) would load and talk to in order to get your application statistics. Your app would be instrumented to make calls to update the counters at the appropriate points. This was nontrivial, as some kind of shared memory was required between object instances to maintain the counter values. This is because Perfmon runs in a separate process, calling a central collection function (residing in the Perfmon process) in your application that is independent of your individual objects. The .NET Framework makes these headaches go away. Figure 1 shows the current performance counter architecture. The .NET Framework component, netfxperf.dll, runs in the Microsoft Management Console process mmc.exe that hosts the performance counter snap-in. This Process (B) loads the CLR and talks to your application in Process A to gather the statistics.

Figure 1 Performance Counter Architecture
Figure 1 Performance Counter Architecture

In order for the Perfmon snap-in (or anything else) to find your counters, a number of registry settings must be added. The .NET Framework conforms to this automatically, as with event logging, when it hits the actual line of code. There is one main key that is created:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet    \Services\category\
This key usually contains a single subkey named Performance.

By using a generic DLL for this, the only issue to solve is how your assembly is associated with the key so that netfxperf.dll knows where to get your counters (and their values) when loaded by the snap-in. The solution uses the Wbem values that exist under this key. These values are used to obtain your counters using WMI. The .NET Framework registers your counters with the underlying WMI repository when it executes your code. Later, when Perfmon comes along, it retrieves your counter names from the registry, and using WMI, makes a request to be connected to them. If your application is running, WMI will provide the connection to the actual values. Incidentally, this is accomplished using the automatic discovery and purge feature of WMI, which is why some of the registry values are prepended with WbemAdap (WMI is an implementation of Web-Based Enterprise Management, also known as WBEM). The other registry counter values, First Counter/Help and Last Counter/Help, have to be strung together across all the counters on your system. In the past this could be accomplished by calling the lodctr/unlodctr API functions. If you get this wrong it can be a real problem because you will affect the ability of Perfmon to enumerate any of the counters on your machine. Again, the .NET Framework takes care of all of this for you.

The .NET Framework supports the full range of available Windows counter types. The most useful ones are the absolute, average, and timing groups. These allow you to represent such things as the number of SQL commands executed by your application, the average time it takes to compile a certain value, or the number of Web service requests per second.

Creating a performance counter and incrementing its value in .NET is as easy as the following:

Imports System.Diagnostics

Dim objCat as PerformanceCounterCategory
Dim objCtr as PerformanceCounter

objCat = PerformanceCounterCategory.Create("MyApp", _
                                           "MyApp help", _
                                           "counter 1", _
                                           "counter 1 help")

objCtr = new PerformanceCounter("MyApp","counter 1",false)
objCtr.Increment()
Again, you need to be careful that during development you don't make too much of a mess of your registry. When the category's Create method is hit, the MyApp key will appear under the registry path just shown. When the new PerformanceCounter is hit, the Wbem values are created during the WMI ADAP processing.

Tracing

The Win32® debugging API has always been a handy tool. The OutputDebugString function can be added to applications to provide dynamic tracing so that you can actually see an application's debugging information using a tool like DbgView.exe (http://www.sysinternals.com) or DBMON (provided with the Platform SDK). You can even connect to it from a remote machine. The .NET Framework still makes use of this API in its tracing implementation, which is what I'll discuss next.

Tracing is a useful idea. You want to see which methods are touched in what circumstances: ASP.NET provides the Trace method mechanism in the System.Web.TraceContext class while the base class library (BCL) provides the Trace and Debug classes in the System.Diagnostics namespace. To use System.Diagnostics tracing from your applications you must add tracing statements to your code and ensure that tracing is enabled at compile time. Tracing statements take one of several similar forms. The most commonly used form looks like this:

System.Diagnostic.Trace.WriteLine("something useful")

Both System.Diagnostics.Trace and System.Diagnostics.Debug are sealed classes. The main difference between them is the conditional compilation string used to include calls made on these classes in the build. The Debug class methods will only be compiled into your builds when the DEBUG compilation constant is set, whereas the Trace methods will be compiled into any build that has the TRACE compilation constant set (see the SDK documentation for full details). You can also set the tracing level in your application .config file so that you have some measure of control over the level of detail actually produced.

With the System.Diagnostics namespace you can also write your own trace listener classes and add them to the Trace or Debug Listeners collection. When the CLR hits a Trace or Debug statement, all of the listeners in the collection will be called in turn. The DefaultTraceListener class, which provides debug tracing, is by default always in the collection. The Framework also comes with the EventLogTraceListener to dump these messages into the current log, and a TextWriterTraceListener to write them to streams.

Structured Exceptions

SEH is one of the best features of the .NET managed environment. From the perspective of Visual Basic 6.0, this is a fantastic improvement over On Error statements. You either catch an exception explicitly because you know what to do with it, or you don't bother and let someone else catch it higher up the chain.

Figure 2 Log It
Figure 2 Log It

But if you use SEH just for this, you're missing out. You can implement your own base exception class from which all your other custom exceptions derive. In it you can capture data about the environment (such as the current user name or the thread ID) so that you have a complete log of everything that happened, allowing exceptional situations to be better evaluated and management reports to be generated as needed. Figure 2 illustrates the process where an exception is propagated up the call chain to a generic handler that can then record it. You can log these any way you want, but a database usually makes the most sense (unfortunately, it's also the most work for you). If the exception logging mechanism is time intensive, it can also be done asynchronously to allow the application to continue execution while the exception information gets stored in the log.

WMI

WMI is by far the most powerful and sophisticated instrumentation mechanism included in Windows. Prior to the .NET Framework, it had a reputation for being difficult to use. This was largely due to the fact that the WMI API was practically impossible to use from RAD tools like Visual Basic 6.0. WMI is big—just take a look at the SDK documentation. Fortunately, you don't need to understand it all to make use of its best features. For example, add a reference to System.Management.dll. This will allow you to write the following code:

Imports System.Management.Instrumentation

<System.ComponentModel.RunInstaller(True), _
InstrumentationClass(InstrumentationType.Event)> _
Public Class MyEvent

    ' event "property"
    Public EventProperty As String

End Class

Even better is this code, which is more efficient than the declarative approach just shown:

<System.ComponentModel.RunInstaller(True)> _
Public Class MyEvent
             Inherits BaseEvent

    ' event "property"
    Public EventProp As String

End Class
You can raise the event just as easily:
Dim objEvent as MyEvent

objEvent = new MyEvent()
objEvent.EventProp = "Hello world!"
Instrumentation.Fire(objEvent)
This publishes an event to WMI which supports a loosely coupled publish/subscribe model. WMI allows you to raise events that will be delivered to their subscribers or, if none are running, that can be picked up later when one subscribes. This means that you don't have the overhead of actually publishing events if you know no one is listening.

One good reason to write your own listeners is to make use of logging facilities your company may already have, such as existing COM components that log to a central database which is used by system support personnel for all applications.

WMI is standards based; it's an implementation of the Distributed Management Task Force (DMTF), WBEM initiative, and the DMTF Common Information Model (CIM). Browse http://www.dmtf.org for more details. You'll see all of these acronyms as you dig around. WMI can interoperate with SNMP applications, making the enterprise vision of centralized monitoring an actual possibility. In the heterogeneous Web services world, a heterogeneous instrumentation capability is essential.

To start consuming these events, create a System.Management.ManagementEventWatcher with the appropriate WMI scope and query, and register an event handler for its EventArrived event. A simple event handler is shown in the code in Figure 3.

Figure 3 Event Handler (Close Figure 3)
Imports System.Management

Private Sub MyEventArrivedHandler(
        ByVal sender As Object, ByVal evt As EventArrivedEventArgs)

    Console.WriteLine("Event received with the following properties:")

    ' Loop through all of the ManagementBaseObject's properties
    For Each eventProperty In evt.NewEvent.Properties
        Console.WriteLine("Property {0}: {1}",
            eventProperty.Name, eventProperty.Value)
    Next eventProperty

EndSub

You can make this even easier using the WMI server explorer extensions (see "Managed WMI Extensions for Visual Studio .NET 2003 Server Explorer" in the Microsoft Download Center). You can just set up a query (in WQL, a subset of SQL) to your event, and drag and drop it onto a form. All the code I've shown is then generated for you.

The Good, the Bad, and the Ugly

Let's look now at when you should or shouldn't use each option by examining their strengths and limitations.

Event logging has a number of benefits. It's easy to use, it's durable, and you can view events from or log events to remote machines. You can even save off logs periodically for later investigation and reporting. Unfortunately, the disadvantages outweigh the benefits in most real-world situations. The event log is finite. You can set its size when the log is created, but then it's fixed. At that point, either your logging attempts throw an exception or you can configure the log to wrap, overwriting the earliest-written entries first. Because of this limitation, the event log is only useful for logging small amounts of information such as critical events like your application's start and stop messages or other high-level notifications. Additionally, it's also only available on Windows NT. If you still target Windows 98 or Windows Me, this facility isn't available.

Performance counters can be extremely useful. They provide excellent aggregated statistics on the health of your application. The number of clients connected, the number of database connections, and average page-turn times are ideal measurements for investigating load-related issues. Again, as with event logging, performance counters are not good for large amounts of specific details. Performance counters work using either average running values or absolute values. Counters are helpful because Perfmon provides remote machine access and file logging. It even provides alerts, for example, running a program to send an e-mail message or Short Message Service (SMS) if your average response time exceeds an SLA.

Tracing is a real boon in the common language runtime (CLR) world. Built-in support provides out-of-the-box functionality and the .NET Framework allows you to use its supplied tracing handlers or to implement your own.

It should be noted that ASP.NET provides its own tracing facility separate from that provided by the System.Diagnostics namespace in the Framework. In ASP.NET, this is accessible through HttpContext.Current.Trace or through Page.Trace if you're code is running in a Web Form. The TraceContext class returned by these properties exposes two main methods for writing output messages: Write and Warn. All output is visible on the trace.axd page in your application's virtual directory.

The only difference between the two is that using the Warn method causes the message to appear on trace.axd in the log in red. You can also have ASP.NET dump the tracing output to the response stream after the current output has been rendered, causing the messages to be appended to the bottom of each page. This is only good for debugging, not for routine production instrumentation. The ASP.NET tracing system also logs additional useful information about each request. For example the current session ID, time, HTTP status code and headers, and cookies are shown.

By default, the System.Diagnostics tracing system and the ASP.NET tracing system are not linked in any way. Messages written to one do not get forwarded to the other. However, it is fairly simple to write a custom trace listener that forwards all trace and debugging messages to the ASP.NET trace. If you do this, just make sure to check that a current HttpContext exists.

There are a few other issues to be aware of when writing your own custom listeners. For example, if you make use of the performance counter or the event log and need to create a counter or an event source, you're going to need write access to the registry (as mentioned previously). This isn't practical or desirable for security reasons under the ASP.NET default account, so you have to be sure that your applications are installed and set up correctly prior to use. If possible, you should also configure your trace listeners using web.config rather than programmatically so that you can add and remove the listeners after deployment by changing your web.config or <appname>.config as follows:

<system.diagnostics>
    <trace autoflush="true" indentsize="0">
        <listeners>
            <add name="CustomListener",
                       type="MyLibrary.Class1,MyLibrary"/>
        </listeners>
    </trace>
</system.diagnostics>

The type attribute on the add element must be set to the listener's fully qualified type name. When using ASP.NET tracing, all trace calls are logged internally and can be displayed using a built-in IHttpHandler that exposes the tracing data of http://yourservername/yourapp/trace.axd.

The maximum number of requests logged here defaults to 10, but can be changed to another value in your web.config, like so:

<trace
        enabled="true"
        requestLimit="15"
        pageOutput="false"
        traceMode="SortByTime"
        localOnly="true"
/>

Unfortunately, with ASP.NET tracing you can't use your own tracing handler to process the output messages. The other, perhaps more significant, problem is that you can't assume you've got an HttpContext object available in the class libraries you develop. If you're developing generic class libraries that can be deployed on application server configurations, you always need to check whether HttpContext.Current is a valid reference before attempting to get the current TraceContext.

As soon as the requests exceed the RequestLimit attribute, you won't be logging your valuable tracing information anymore. This renders tracing for live problem diagnosis useless as there will likely be thousands of requests before the one you want to view, and then you still have no tools other than the provided page to track that request. The technique is really much more useful for debugging situations in which you want to check the correct flow control through your code.

A more general problem with ASP.NET tracing is that you need to explicitly turn it on in your configuration file. When you do this, however, you cause the application to be restarted. This is a bit heavy-handed just to configure your application's instrumentation. It is also a real issue if you're investigating a transient problem that actually requires the current application state.

Too much information can be a problem here too—when using diagnostics tracing, filtering isn't provided unless you code for it by using granular trace levels and trace switches. The amount of information you're likely to generate is overwhelming. This is especially true without a durable logging store, such as a file, and without a tool to process it usefully.

I've already talked about using SEH effectively. However, it is not meant to be an instrumentation tool. You'll end up bending your application code to throw exceptions for informational-type logging, which reduces the clarity that exceptions were designed to bring to CLR-based code. In addition, this hurts performance.

WMI is a mature instrumentation technology built into Windows 2000 and all subsequent versions. Since WMI is COM-based, you will be using interop, which can affect performance if used improperly. It also relies on DCOM for cross-machine logging and event correlation. WMI does, however, provide enterprise-scale facilities to perform many kinds of instrumentation. In fact, much of Windows itself is instrumented using WMI. For an effective instrumentation solution you need to be able to use the best of what's available and to make it configurable from your application. That way you can emit and consume the instrumentation you require according to the situation that presents itself. WMI brings you closer to this ultimate goal by unifying access to the instrumentation provided in your code.

A powerful feature of WMI is the ability to define an instrumentation contract for your application, called a schema. This schema defines the instrumentation you provide. It is specified in terms of event and data classes. .NET provides simple control over the schema content generated for your application. WMI is complementary to the logging schemes discussed already. For example, Event Log information and performance counters can actually be surfaced by WMI. This means that if you're already using these, you can make use of WMI now. The WMI SDK provides various tools, such as the browser-based CIM studio, so that you can examine your application's instrumentation (and query the schema). Many industrial-strength tools are based on WMI, such as Microsoft Operations Manager (MOM), HP OpenView, and IBM's Tivoli products. The WMI classes provided in the .NET System.Management namespace allow you to raise your own events and provide management facilities.

The System.Management namespace provides natural mappings onto WMI constructs. WMI classes become .NET Framework classes. Properties and attributes become fields and qualifiers. The data types in the .NET Framework map cleanly onto WMI data types, including strings and arrays.

Managed World

.NET now brings much greater power to all of these facilities with the EIF. This builds on WMI, making it easier to use. Although EIF dramatically reduces the overhead of WMI, it isn't included with the .NET Framework or Visual Studio®. Until now it hasn't been available unless you were an MSDN subscriber, so it hasn't exactly entered the mainstream. This is likely to change soon, especially with the new Logging Application Block released by the Microsoft Patterns and Practices group. [ Editor's Update - 3/26/2004: EIF is now available for download at Microsoft Enterprise Instrumentation Framework.] WMI is the foundation for both of these and because it's been designed for scalability and doesn't require stopping your app for reconfiguration, it is a better choice than web.config/app.config-based solutions or #define TRACE and DEBUG.

EIF provides the Microsoft.EnterpriseInstrumentation namespace. There are two central constructs: event sources and sinks. An event source is the creator or publisher of an event. This is decoupled through configuration to the associated event sinks. Sinks consume and deal with events. This could be done, for example, by logging to the Windows Event Log or a database. Events in EIF are either implicit or explicit. Four implicit event sources are provided for you as classes: TraceMessageEvent, ErrorMessageEvent, AuditMessageEvent, and AdminMessageEvent.

To consume events, all event sinks must derive from the abstract EventSink class, which has a single static method, Raise. EIF provides three event sinks: Windows Trace, Event Log, and WMI. Windows tracing is implemented as a service, which runs in kernel mode to optimize performance. The Event Log sink directs events to the Windows Application Event Log, and the WMI sink uses WMI for logging events. The power of EIF is that the style of code you write to surface events is always the same. This makes it as easy as possible for you to start instrumenting your code. All you need to do is call the event's static Raise method, like so:

Imports Microsoft.EnterpriseInstrumentation
TraceMessageEvent.Raise("Hello World")
You can also write your own custom sinks to extend or add to the supplied ones. EIF is installer-aware and creates an EnterpriseInstrumentation.config file which you can edit manually or use the EIF API configuration classes.

Explicit events are those you define yourself, as follows:

<System.ComponentModel.RunInstaller(true)>

Public Class EIFEvent
Inherits Microsoft.EnterpriseInstrumentation.Schema.BaseEvent
    Public MyProperty as string  ' field
End Class
As you can see, it's almost identical to the WMI example shown earlier. This is because the class derives from the BaseEvent class which effectively replaces the BaseEvent you would use in WMI when developing your own event sources, as I've already discussed. The attribute defined ensures your event is installed in WMI when you run InstallUtil.exe against your .exe or.dll.

There are two configuration files needed to wire everything up. The EnterpriseInstrumentation.config is generated for you during installation and enables you to specify the relationships between sources and sinks and also filter events based on object type using categories. The TraceSessions.config file is needed to define tracing details only when you are using the Windows trace sink. I'll cover the use of TraceSessions.config in a minute, but let's first look at the salient features in the other one, shown in Figure 4.

Figure 4 Config File (Close Figure 4)
<eventCategories>
    <eventCategory name="All Events" description="A category that contains
    all events.">
        <event type="System.Object" />
    </eventCategory>
</eventCategories>

<filters>
    <filter name="defaultSoftwareElementFilter" description=" A default filter
    for the Software Element event sources.">
        <eventCategoryRef name="All Events">
            <eventSinkRef name="wmiSink" />
            <eventSinkRef name="traceSink" />
            <eventSinkRef name="logSink" />
        </eventCategoryRef>
    </filter>
</filters>

<eventSources>
    <eventSource name="Application" type="softwareElement"
    internalExceptionHandler="ignore" description="Application level event
    source." />
</eventSources>

<filterBindings>
    <eventSourceRef name="Application">
        <filterRef name="defaultSoftwareElementFilter" />
    </eventSourceRef>
</filterBindings>

The code in Figure 4 has four sections. First, there is an <eventCategory> called All Events. As the <event> element specifies System.Object as the type, it includes all tag events raised by the System.Object type or any derived class—which means everything. The second element is the <filter>. This defines the event sinks that are wired up to the All Events category. An event raised will be dealt with by all three of the provided event sinks. The <eventSources> element is where you define the name of your instrumented application. This enables you to connect your application's events to one or more sinks through the <filterBindings> element. It defines the relationship between your sources and sinks. This example uses the implicit Application event source that EIF provides for your whole application. It is the same source name used in the earlier code samples.

The default TraceSessions.config file resides in the folder [EI_Dir]\Bin\Trace Service, where [EI_Dir] is the location in which you installed EIF. You can use the default session name, TraceSession, or you can add your own session name. See the EIF documentation for further details.

WMI and EIF applications can be a bit cumbersome to configure and get working properly. You may also find you get a few fatal exceptions during development due to the managed/unmanaged interaction that takes place.

Logging Application Block builds on WMI and EIF by providing common facilities that developers need, such as database and queue-based logging. See the Patterns and Practices site for more details.

A Strategy

WMI and its dependent technologies offer abstraction by decoupling the incidence of instrumentation from the action of handling it. EIF goes much further than WMI, though. Not only are the source and sinks decoupled, but standard sinks are provided for you for common requirements. Without EIF, you are deciding what types of instrumentation you want to use and coding specifically for them. If you want event logging, for example, you use System.Diagnostics.EventLog as detailed earlier. But what if your application grows in size and you want to move your instrumentation logging to a database? You've got a big find-and-replace operation on your hands. With EIF, you can just change the EnterpriseInstrumentation.config to use a different event sink and you're done—if you use Logging Application Block you don't even need to write this sink yourself since there's one provided.

You need to think about how best to make use of each of the available facilities. Each of the available features in Windows is very specific and you need them all in order to be effective. This might seem like a lot of work even with the .NET Framework, but it's not. Judicious use of each will yield the best results and require the least amount of work. Figure 5 details a few guidelines regarding when to use each one.

Figure 5 Instrumentation Options (Close Figure 5)

Feature | Positive Attributes | Negative Attributes
Event log | High-level application events (for example start and stop) | Limited in number of events
Performance counters | Use when you have rollup summary information (number of active users) or absolute (simple) values you want to measure (total requests) | Not suitable for complex data
Debug output | Single user code pathing and debugging | Overwhelming if used everywhere
SEH | Good for unexpected exceptions | Can bend your design
System.Diagnostics.Trace | Easy to use | Limited to Web apps
ASP.NET trace | Great for debugging | Fixed functionality
WMI | Widely supported, complementary to other schemes (for example, your event log code can be surfaced through WMI) | More complex to use and install applications with
EIF | Easier to program than WMI | Doesn't expose all of WMI's power
LAB | Common requirements provided: MSMQ logging, DBMS logging | A lot to learn

The first thing here is order. Sprinkling code for exceptions and tracing is never effective. You must think about what instrumentation to add as you are coding. Error and exception handling are exactly the same in that you can't retrofit this stuff effectively. When you write code, you know exactly what your intentions are—the semantics. Everything's fresh in your mind, the task is at hand. This is the best time to add your exceptions, error handling, and instrumentation because you are thinking about what can go wrong with each line of code. You are also thinking about what would be useful to measure. Doing this at the design and modeling phase would be even better, but often this level of detail is hard to attain before starting to code so you should always be pragmatic.

Once you have added instrumentation code, you find that it actually gives you more than just simple text logging to various places. Its flexibility gives you an extensibility point. The implementation of your trace code is able to change as you add new features without affecting any of the actual application code. Just as with writing a good base, SEH is critical before you start coding. Doing the same with instrumentation will yield benefits that will be reusable for future projects as well.

For example, you might want to determine method timings. If in each function you have a trace in/out and exception path, you can measure the time between them. Note this is a different scheme from aspect-oriented programming (AOP), which is limited to intercepting calls and has no knowledge of what those calls do. Here you are explicitly adding that information to help you later. Or you might want to filter the instrumentation. You might only want to record the fact that you are receiving SOAP messages greater than 100KB in size. Although not an error, this is something you want to track. So you can add a specific log point with this information, which your generic handler can track for you. You can achieve this easily by implementing your own custom event sink.

Security

Logging information brings its own security risks. If you start instrumenting sensitive data, such as database connection strings, you risk sending it in the clear across your network. With distributed applications this can mean a big security hole.

As mentioned earlier, a certain level of registry access is required in order to use performance counters, Windows Event Logging, and WMI from your code. A more general problem, though, is accessing the instrumentation data once it's been recorded. If your production applications run on an isolated network with demilitarized zones (DMZs), firewalls, and the like, chances are you won't be able to remove the logs from your corporate network. This makes it even more important that you log all your information to a database so that you can be granted read privileges when necessary for problem investigation.

Conclusion

When asked why an application isn't better instrumented, many designers and developers cite performance as the key concern. Never use performance as an excuse for not providing adequate instrumentation. This argument just doesn't hold up in most cases when you account for the processing time of business logic. Good instrumentation can save many hours of late-night frustration and keep the wheels of your enterprise applications running smoothly. I've only covered the fundamentals of instrumentation here. I'm sure you can think of ways to expand on the ideas I've presented. Gaining an understanding of the new features of the .NET Framework is well worth the effort. Spend the time to take a closer look and see how you can apply what I've shown in your next project.

Jon Fanceyruns an IT consultancy company in the United Kingdom that specializes in enterprise-class software design and implementation. You can reach him at instrumentation@jonfancey.com.


Stress Testing: Custom LoadGenerator Tool Identifies the Issues Your Application Faces Under Stress

Stress Testing

Custom LoadGenerator Tool Identifies the Issues Your Application Faces Under Stress

Brian Otto
This article discusses:
  • The importance of stress testing
  • Building a load generator
  • Managing threads
  • Recording performance data
This article uses the following technologies:
C#, Multithreading, and Reflection
Code download available at: StressTestingTool.exe (164KB)
Contents
Getting Started
Creating a Worker
Inside the LoadGenerator App
Initializing a Simulation
Running the Simulation
Completing the Simulation
Conclusion

I
n order to create a reliable application you must understand how various components will perform under heavy loads and ensure that the system behaves appropriately. Server components for popular Web sites need to respond to millions of requests per day. Corporate intranets and desktop applications may only need to handle a few hundred or a few thousand requests per day. Many smaller development organizations feel they don't have the time or money to devote resources to building out environments to conduct rigorous stress tests, even though much of the success of an application can be attributed to how consistently it responds to users. If you don't take the time to understand how your application will respond to extreme use and failures, it could lead to your application's demise.

Stress testing should be conducted at all stages of development, including the stage before individual components are incorporated into a larger application to prevent issues later in the development cycle. Having the right tools available to perform this initial set of tests can make the task quick and easy.

Getting Started

The Microsoft® .NET Framework provides a great foundation to build a reusable and extensible stress tool that makes it easier to test your application. The application in this article uses reflection to dynamically load a custom class in an assembly that you create. The class performs a set of tasks representing a desired test. You control how many instances of the class are loaded and how many times the application calls into each instance during the simulation. Each instance of the class is instantiated on a separate thread. Since the actual tasks being performed are in a custom assembly that you create, you have the flexibility to create worker components that stress just about any application layer, be it a Web service, a database, or a COM+ serviced component. This stress-testing tool can help you verify that your application meets both its performance and concurrency requirements, as well as help you determine how it responds to stressful loads.

Figure 1 Load Generator
Figure 1 Load Generator

Figure 1 shows the main form of the application. In the Test Setup frame, you can configure how the simulation will be conducted. The Workers setting identifies how many instances (threads) will be created for executing the test. The number of iterations for each instance is set through the Cmds/Worker value. Think time represents the delay between successive iterations for each instance. When conducting a stress test, it is also important to control the warm-up time. Most systems have required initialization that must be performed before any work can be done (for example, ASP.NET pages are compiled the first time they're accessed). During the warm-up period, no performance data is collected so as to minimize the distortion of the test results. The information in the Statistics frame provides a means to monitor the simulation. When the test is complete, a status form appears showing some statistics for each instance (see Figure 2).

Figure 2 Performance Stats
Figure 2 Performance Stats

If you checked the Show Detailed Progress box during the simulation on the main form, this status form will be visible during the test. Because updating this form during the test will consume resources, it should be done sparingly when simulating a heavy load.

Creating a Worker

To help maintain better control over the execution of the simulation, the application does not use the thread pool when spawning worker threads. Each thread is created and managed manually. If I had used the thread pool, new requests might have been queued if CPU activity was high or no free threads existed in the thread pool. Since this is a stress-testing tool, I wanted to maintain direct control over how much activity is generated based on the test setup options.

Creating a worker thread involves creating a class in an assembly that implements the LoadGenerator.Interfaces.IWorker interface, as shown here:

public interface IWorker: IDisposable
{
    void Initialize(long instanceID);
    void Process();
}
As you can see from the definition, your class needs to implement the IDisposable interface. The Dispose method will be called at the end of the simulation to provide your class an opportunity to perform cleanup tasks. The Initialize method is called only once, before any tests are executed. The instanceID parameter represents the unique instance number of the worker relative to all the workers being created. The Process method is where the code that performs the work for the simulation should reside. This method will be called repeatedly as defined by the Test Setup settings on the main form.

For this example, I will create a worker component that stresses a stored procedure in the Northwind database. SQL Server™ and Northwind are assumed to be on the same machine as the LoadGenerator application. In a real test, however, SQL Server and Northwind would reside on a separate machine. On a single CPU system, running the client load generator and SQL Server on the same machine can quickly overtax the computer and defeat the intent of the test. In addition, it is perfectly viable to have a test client running on multiple machines.

Start by creating a new C# library assembly called DBWorker in Visual Studio® .NET. Add a reference to the LoadGenerator.Interfaces assembly and change the default class declaration to the following line of code:

public class MyWorker : LoadGenerator.Interfaces.IWorker
At the end of the line, Visual Studio should prompt you to press the Tab key, which will cause the stubs for all the required methods of the IWorker interface to be created for you automatically.

Figure 3 shows the Initialize, Process, and Dispose method implementations. A class-level Command object is created in the Initialize method and reused across calls to the Process method. Reducing the unnecessary overhead within each execution of the Process method helps improve the efficiency and effectiveness of the test. The Initialize method configures the Command object to call the [employee sales by country] procedure in the Northwind database with two dates as arguments. The Process method opens a pooled database connection, executes the command, and releases the connection back into the pool. A more realistic test would probably have the Process method change the arguments to the procedure on subsequent executions.

Figure 3 DBWorker Class Implementing IWorker (Close Figure 3)
public void Initialize(long lInstance)
{
    //Initialize a command object
    _cmd = new SqlCommand();
    _cmd.CommandText="[employee sales by country]";
    _cmd.CommandType = CommandType.StoredProcedure;
    _cmd.Connection = new SqlConnection("Integrated
        Security=SSPI;Persist Security Info=False;Initial
        Catalog=Northwind;Data Source=(local)");
    _cmd.Parameters.Add("@Beginning_Date", "1/1/1997");
    _cmd.Parameters.Add("@Ending_Date", "1/1/1998");
}

public void Process()
{
    try
    {
        _cmd.Connection.Open();
        _cmd.ExecuteNonQuery();
    }
    finally
    {
        _cmd.Connection.Close();
    }
}

public void Dispose()
{
    if (_cmd != null)
    {
        _cmd.Connection.Close();
        _cmd.Dispose();
        _cmd = null;
    }
}

The stored procedure is executed without regard for the resultset the procedure returns. If you wanted to test the performance impact of loading the resultset into a DataTable, you could compare this test to one that filled a DataTable. Just be aware of how much client work is being performed for each test since it may consume CPU time and limit the client's ability to execute additional tests.

Now you can compile the assembly and copy it to the application directory. The application directory contains an application configuration file that tells the application what assembly and class to load when conducting a test. Open the LoadGenerator.exe.config file and add the name of your assembly and class:

<configuration>
    <configSections>
       <section name="Worker"
        type="System.Configuration.SingleTagSectionHandler"/>
    </configSections>
    <Worker assembly="DBWorker" type="DBWorker.MyWorker"/>
</configuration>

After updating the configuration file, you are ready to run the simulation. Start up the LoadGenerator application and change the number of workers to one so you can verify that everything is working; then click the Start button. If there are any errors when activating your component or executing the tests, they will be displayed in the Status field of the status form. If everything worked, you are ready to run more instances and change the think time and warm-up periods to vary the stress on the system. That's all there is to making this tool work. To get the most out of this example, the Northwind database should reside on a different computer. More sophisticated worker components can perform a variety of tasks to stress your applications.

Inside the LoadGenerator App

The application divides most of the simulation work into two classes. The WorkerState class houses the code that creates an individual thread and provides access to start and stop its execution as well as to report state information. This class also loads the custom class and calls to the IWorker interface methods. For every instance of a worker defined in the test setup, one instance of the WorkerState class is created. The WorkerCollection class is the container for all of the WorkerState objects created for a given simulation. This class controls the starting and stopping of the simulation and exposes one event called TestComplete that provides notification to the forms when a simulation has completed.

To conduct a simulation, the application first initializes the WorkerCollection object and then calls WorkerCollection.Start. Much of the initialization is performed asynchronously from the main application thread from which the WorkerCollection is accessed. Because of this, the Start method blocks until initialization is complete to ensure that no tests are conducted while some threads are still initializing. The initialization of the WorkerCollection object includes the creation and initialization of WorkerState objects for each worker defined by the simulation. Initialization of each WorkerState object creates a new thread and loads the custom assembly. Once all threads have loaded the custom assembly, the WorkerCollection.Start method stops blocking and each thread is allowed to proceed to the warm-up cycle and begin making calls into the custom assembly to conduct tests.

The following sections step through the initialization, running, and completion tasks.

Initializing a Simulation

When a simulation is started by clicking the Start button on the main form, the form hooks up to the TestComplete event of the WorkerCollection object and calls the WorkerCollection.Initialize and WorkerCollection.Start methods. As you will see a little later, the Start method may take up to 30 seconds to return. To keep the application more responsive to the user, the Start method is called asynchronously:

_workerCollection = new WorkerCollection();
_workerCollection.TestComplete +=new
    TestCompleteEventHandler(_workerCollection_TestComplete);
_workerCollection.Initialize(Convert.ToInt32(txtInstances.Text),
    Convert.ToInt32(txtWarmup.Text),
    Convert.ToInt32(txtInterval.Text),
    Convert.ToInt64(txtMaxCmds.Text));
StartDelegate _startSimulation = new
    StartDelegate (_workerCollection.Start);
StartDelegate.BeginInvoke(null,null);

The WorkerCollection.Initialize method takes as arguments all the test setup parameters specified on the main form:

void Initialize(int totalWorkers, int warmUpTime, int thinkTime,
                long maxCommands)
This method then looks into the application configuration file to retrieve the class that will be doing all the actual stress tests and passes it on to each of the WorkerState objects. The WorkerState objects are created to control execution of each thread and retrieve state information (see Figure 4).

Figure 4 Get State Information (Close Figure 4)
WorkerState workerState = new WorkerState();
workerState.InstanceID = index;
workerState.IWorkerAssemblyName = assemblyName;
workerState.IWorkerTypeName=typeName;
workerState.ThinkTime = thinkTime;
workerState.MaxCommands = maxCommands;
workerState.WarmUpDelay = (int)(((double)_warmUpTime/_totalCount) *
    (double)index * 1000);
workerState.Initialize(_syncStart,
    new WorkerCallBack(WorkerInitialized),
    new WorkerCallBack(WorkerStarted),
    new WorkerCallBack(WorkerComplete));
_workerCollection[index] = workerState;

After setting the running parameters on the WorkerState object, the WorkerState.Initialize method is called. The thread created by this method will wait to be signaled by the application before conducting any stress tests. The _syncStart argument is a ManualResetEvent that will be signaled after all WorkerState object instances have completed their initialization tasks. The remaining arguments to the Initialize method are callback delegates from the WorkerState object to the WorkerCollection object. The calling of these delegates will indicate to the WorkerCollection when each worker has completed initialization, started conducting tests, and finished conducting tests. As you'll see, the WorkerCollection object uses these callbacks to track the progress of the simulation.

The WorkerState.Initialize method creates and starts the worker thread, specifying which method the thread should execute:

ThreadStart ts = new ThreadStart(ProcessCmd);
_thread = new Thread(ts);
SetActionState(ActionEnum.Initialize);
_syncStart = startWaitHandle;
_thread.Start();

The actions of the worker threads are controlled through the _actionState variable, whose type is an enumeration called ActionEnum. This enumeration helps identify the task the thread is performing and what task the thread is supposed to perform. If the simulation is stopped by clicking the Stop button on the main form, the WorkerCollection object will set the variable to ActionEnum.Stop. As the thread proceeds though its tasks, it regularly checks the value of this variable. If the enumerated value is Stop, the thread will exit gracefully. Since this value is updated by both the worker thread and the main application thread, all updates to this variable are synchronized and performed by one method called SetActionState.

Once Thread.Start is called, the WorkerState.ProcessCmd method begins executing. This method creates an instance of the custom class specified in the configuration file. The ProcessCmd method then indicates its completion of initialization tasks by issuing a callback to the WorkerCollection and waiting on a signal from the _syncStart ManualResetEvent:

Assembly workerAssembly = Assembly.Load(_IWorkerAssemblyName);
objWork = (IWorker)workerAssembly.CreateInstance(_IWorkerTypeName, true);
_initCallBack(_instanceID);
_syncStart.WaitOne();

The method invoked in the WorkerCollection object by the _initCallBack delegate will signal the _syncInitialized wait handle when all workers have completed their initialization work:

lock(this)
{
    if (++_initCount == _workerCollection.Length)
        _syncInitialized.Set();
}

This completes the initialization phase of the simulation. At this point, each thread has created an instance of the worker component and is now waiting on _syncStart.WaitOne. The signaling of this wait handle, which happens in the WorkerCollection.Start method, will start the simulation.

Running the Simulation

The WorkerCollection.Start method is called from the main form after the WorkerCollection.Initialize method returns. Since many of the initialization steps are performed by the individual worker threads asynchronous to the Initialize method, the Start method waits until all threads have finished their initialization tasks. This method waits up to 30 seconds on the _syncInitialized wait handle. As you saw earlier, this wait handle is signaled after the last thread has invoked the _initCallBack delegate. If the WorkerCollection times out while waiting for all the workers to initialize, the simulation is stopped and all threads will exit, as you can see in the code listed in Figure 5.

Figure 5 Waiting (Close Figure 5)
public bool Start()
{
    try
    {
        if (_syncInitialized.WaitOne(30000,false) == false)
            throw new Exception("Timeout waiting for initialization.");
        _syncStart.Set();
        return true;
    }
    catch (Exception err)
    {
        Stop();
        return false;
    }
}

If _syncInitialized is signaled before the timeout, the Start method will in turn signal the _syncStart ManualResetEvent. This is the wait handle that each of the worker threads are waiting on. Once signaled, the worker threads proceed beyond the Initialized state and begin the warm-up cycle:

_syncStart.WaitOne();
if (_actionState == ActionEnum.Stop) return;
DelayForWarmUp();
objWork.Initialize(_instanceID);
_startCallBack(_instanceID);
SetActionState(ActionEnum.Process);

After being signaled to start the warm-up delay, the worker thread first checks to verify that the simulation wasn't stopped and then proceeds to sleep the appropriate amount of time to accommodate the warm-up period. Each thread has a different warm-up period (delay) that it needs to spend sleeping. These delays are assigned to ensure that threads begin stress tests at regular intervals with all threads active at the end of the warm-up period. As I mentioned, no metrics on performance are captured during this warm-up time. If the warm-up period is 10 seconds and there are 10 threads, 1 thread will begin a test every second. The first thread will have a _warmUpDelay of 0, and the tenth thread will have a _warmUpDelay of 9 seconds. The threads do not stay asleep for the entire warm-up delay period. The DelayForWarmUp method runs in a loop, sleeping for short intervals of no more than 500 milliseconds until the warm-up time has elapsed:

private void DelayForWarmUp()
{
    DateTime startTime = DateTime.Now;
    TimeSpan totalDelay = TimeSpan.Zero;
    int remainingTime = _warmUpDelay;
    while(remainingTime > 0 && _actionState != ActionEnum.Stop){
        Thread.Sleep(remainingTime > 500 ? 500 : remainingTime);
        totalDelay = DateTime.Now.Subtract(startTime);
        remainingTime = (int)(_warmUpDelay -
            totalDelay.TotalMilliseconds);
    }
}

Sleeping in this manner allows the thread to stay responsive to a stop request from the user without having to resort to the use of the Thread.Interrupt method. The Thread.Interrupt method injects an exception into the sleeping thread and wakes it from its sleep. This approach is a bit costly when the simulation has a few hundred threads. If the _actionState variable is updated to Stop, DelayForWarmUp exits the loop and returns to ProcessCmd, which will allow the thread to exit gracefully.

After the warm-up delay is over, the IWorker.Initialize method is called to allow the worker class to prepare for the tests. The _startCallBack delegate is then invoked, informing the WorkerCollection that the thread is beginning its tests. The _actionState variable is updated accordingly.

In the WorkerCollection object, the callback method invoked by the _startCallBack delegate only needs to track the number of threads that have moved beyond their respective warm-up delay and have begun running stress tests. The Interlocked class in the System.Threading namespace provides a very simple and efficient mechanism to synchronize variable incrementing. The method calls Interlocked.Increment(ref _activeCount) to increment the number of active threads. This value will be queried by the forms to show progress.

With the custom class initialized, the worker thread proceeds with the execution of the stress tests:

while(_iterations < _maxCmds && _actionState == ActionEnum.Process){
    _iterations++;
    objWork.Process();
    if(_thinkTime > 0) Thread.Sleep(_thinkTime);
}
As long as the current action state is Process and there are more iterations left to go, the IWorker.Process method is called, and the thread sleeps for the required period of think time between each call. When the loop exits, the code in the finally block executes to clean up the instances, as shown here:
finally
{
    _completeCallBack(_instanceID);
    if (objWork != null) objWork.Dispose();
    objWork = null;
}
This invokes a delegate that calls back into the WorkerCollection object to notify completion of the tests. IWorker.Dispose is then called to allow the custom component to perform any clean-up tasks before being released.

Completing the Simulation

When the last thread has called the _completeCallBack delegate, the TestComplete event is invoked in the WorkerCollection object (see Figure 6). The TestComplete event is invoked on whichever thread happens to be the last to complete. The forms are hooked into this event to allow them to display any final information and reset themselves for the next simulation. For the main form and status form, the completion code needs to be invoked on the thread that owns the form:

private void _workerCollection_TestComplete(object sender,
    TestCompleteEventArgs e)
{
    testCompleteEventHandler d = new
        testCompleteEventHandler(this.TestComplete);
    this.Invoke(d,new object[] {e});
}

Figure 6 Is Test Complete? (Close Figure 6)
bool bComplete=false;
WorkerState workerState = (WorkerState) sender;
lock(this)
{
    if (workerState.ThreadError) _errorCount++;
    if (_activeCount > 0) _activeCount—;
    if (++_completeCount == _totalCount) bComplete = true;
}
if (bComplete)
{
    TestCompleteEventArgs args = new TestCompleteEventArgs("",
        _duration.TotalSeconds);
    TestComplete(this, args);
}

The form event handlers for the WorkerCollection.TestComplete event create a new delegate that points to a private method in the TestComplete form class, which is then invoked on the same thread that controls the form. The main form resets itself for a new test and the status form is displayed with the individual running times and number of iterations for each thread.

Conclusion

This article would not be complete without a mention of the importance of security when loading late-bound assemblies. Although you won't likely find third-party assemblies that implement my IWorker interface, it is important to understand how to load late-bound assemblies in a secure manner. Jason Clark has a great article in the October 2003 issue of MSDN® Magazine covering this topic in the context of writing plug-in frameworks for your .NET-based applications. If you plan to use this functionality in your applications, I highly recommend you read his article.

The LoadGenerator application simplifies the tasks of conducting basic stress tests by providing the core plumbing needed to simulate high concurrency. The components you create that implement the IWorker interface need not worry about this plumbing, allowing you to focus on understanding the behavior of your application under stress. Your components can test any number of scenarios, including the efficiency of a new algorithm, the throughput of a Web service, or the concurrency of a database procedure. What you make the component do is up to you.

Brian Otto is a Senior Development Consultant at Microsoft, specializing in .NET and SQL Server development. He has created a variety of software solutions in the financial, automotive, and media industries.


Mobility: Add Keyboard Support to Compact Framework Apps by Trapping Windows Messages

Mobility

Add Keyboard Support to Compact Framework Apps by Trapping Windows Messages

Alan Pulliam
This article discusses:
  • How the .NET Framework and the .NET Compact Framework handle Windows messages
  • P/Invoke and calls between managed and unmanaged code
  • The MessageWindow class and native Windows messages
  • Capturing keyboard events
This article uses the following technologies:
.NET Compact Framework, C#, eMbedded Visual C++
Code download available at: KeyboardSupport.exe (147KB)
Contents
Features Supported by the Compact Framework
Pocket PCs Don't Have Keyboards...Or Do They?
The Mysterious MessageWindow Class
Native Window Handles for My .NET Controls
Intercepting Windows Messages
Managing Keyboard Focus
Implementing a KeyMessage Event Handler
Putting It All Together
Building and Deploying the Native DLL
Conclusion

T
he Microsoft® .NET Framework provides access to almost all of the features of the Windows® operating system from managed code. For situations in which the .NET Framework doesn't provide the desired functionality, the P/Invoke mechanism can be used to access unmanaged code, including the native Windows API. For UI code, the .NET Framework Control class provides the Handle and WndProc members to expose the underlying native window handle and messages. With access to the underlying windows messages and to the Windows API, any capability of the Windows OS can be accessed from within a .NET Framework-based app, providing an easy workaround for any features not provided by the .NET Framework.

The .NET Compact Framework is a trimmed-down version of the .NET Framework, designed to run on resource-limited devices. As a result, many features available in the Framework had to be excluded from the compact version in order to maintain its small footprint and performance. While the Compact Framework supports P/Invoke, providing access to the Windows CE platform API, the Compact Framework Control class does not expose underlying native window handles or messages as does its bigger brother. Nevertheless, by using the .NET Compact Framework-only MessageWindow class, P/Invoke, and a few lines of native code, it is possible to use Windows subclassing to access native window handles and messages. This technique can be used in place of the Control class's Handle and WndProc members. The example that this article is based on uses this technique to add standard Tab key navigation to a Compact Framework-based application that runs on a Pocket PC.

Features Supported by the Compact Framework

If you've done much development with the .NET Compact Framework, you've probably tried at some point to use a feature from the .NET Framework and found that your code won't compile. When you check the documentation, you find that feature is not supported by the .NET Compact Framework. The Compact Framework was designed for resource and processor-limited Pocket PC devices running Windows CE, which has similar size constraints. The designers of the .NET Compact Framework have done a great job of anticipating developers' needs and only removing features that aren't appropriate for mobile devices. Nevertheless, it's impossible to anticipate every developer's needs. As I said, in the .NET Framework, you always have access to all the capabilities of the underlying Windows API using P/Invoke and to the underlying Windows messages via the Control class's window procedure (WndProc member). The Compact Framework includes P/Invoke; however, it does not allow direct access to the Control's underlying Windows CE messages. That said, it is still possible to access all the capabilities of the Windows CE operating system; it just requires a little more custom code.

By using P/Invoke and a few Windows CE API calls, it is possible to find the corresponding native window handles for Compact Framework controls. With the native handles, the windows can then be subclassed in a few lines of native code, allowing all Windows messages to be intercepted. Finally, using the special Compact Framework MessageWindow class, these messages can be passed to managed code, allowing a Compact Framework-based application to respond directly to Windows CE message events. As with the Control class's WndProc in the .NET Framework, this can be used to add functionality to Compact Framework-based apps—an ability that's normally only available in native Windows CE code. One feature I personally miss is keyboard support. While most Pocket PCs don't have keyboards, it's still helpful to have keyboard support for applications targeting the Pocket PC when using an external keyboard. In this article, I'll show you how to use subclassing to trap WM_KEYDOWN and other keyboard messages from child controls on a form to add standard Tab key navigation to .NET Compact Framework form-based applications.

Pocket PCs Don't Have Keyboards...Or Do They?

First, let me start by explaining that I'm not the type who creates solutions in search of a problem. While this subclassing technique turned out to be an ideal solution for providing access to Windows messages in Compact Framework-based apps, I was actually trying to solve a very specific problem. I was developing an inventory application for a handheld barcode scanner running on a Windows Mobile™ 2002-based Pocket PC. This device was equipped with the standard stylus and touch screen, but also had an alphanumeric keypad. The client wanted to have custom navigation on the application forms, with the Enter key used in addition to the Tab key to advance focus to the next control on each form, and soft keys functioning like Back and Forward browser buttons to move to and from previous and next forms.

At first, this didn't seem to be much of a challenge. For the most part, it sounded like a normal forms application with an additional key to advance focus to the next control. .NET Controls have TabStop and TabIndex properties that specify whether a control can receive focus and in what order. The .NET Framework automatically sets focus to the first enabled control in the tab order, and then advances focus to the next enabled control in tab order when the user presses the Tab key. I thought it was simply a matter of checking the KeyDown event on the form and providing the same behavior for the Enter key. Indeed, when I took a close look at the .NET Framework docs, I found that the KeyPreview property allowed me to capture control messages at the form level and the GetNextControl method (that's shown in Figure 1) allowed me to find the next control in the tab order.

Figure 1 Using KeyPreview (Close Figure 1)
private void InitializeComponent()
{
       •••
    // this sends all child controls' KeyDown events to the form
    this.KeyPreview = true;
       •••
    this.KeyDown += new
        System.Windows.Forms.KeyEventHandler(this.Form1_KeyDown);
}

private void SelectNextControl(Control current, bool forward)
{
    Control next = GetNextControl(current,forward);
    if (next != null && next.CanFocus && next.TabStop)
        next.Focus();
    else
        SelectNextControl(next,forward);
}

private void Form1_KeyDown(object sender,
   System.Windows.Forms.KeyEventArgs e)
{
    if (e.KeyCode == Keys.Enter)
    {
        SelectNextControl(this.ActiveControl,true);
    }
}

I thought I had gotten off easy, until I discovered that the KeyPreview property didn't appear with IntelliSense® in my Visual Studio® SmartDevice project. When I looked again at the docs, the key phrase "Supported by the .NET Compact Framework" was missing. In fact, the infrastructure for keyboard focus on forms, including TabStop, TabIndex, and GetNextControl, is not included in the Compact Framework. For an application running on a Pocket PC that uses the stylus, this isn't an issue. Each control gets focus when the user touches its client area with the stylus. Since most Pocket PCs don't even have a keyboard, the keyboard focus code isn't needed. However, I had a Pocket PC with a keyboard.

While Compact Framework forms don't have the KeyPreview form property, individual Compact Framework controls can still capture key events. By subscribing to the Control.KeyDown event for each focusable control and using a single event handler, a form can conveniently receive key events in a central location. Like KeyPreview, this also allows keyboard handling at the form level. In fact, with a little code to manage tab order and focus, which I'll cover in a later section, this simple technique can be used to add basic keyboard support to a Compact Framework-based application. This technique is demonstrated in the PPCKeyboardSample1 application accompanying this article.

Unfortunately, this basic key handling technique didn't solve my problem. It worked fine for adding Enter key navigation to my application, but it didn't work with the Tab key. When I dug into the .NET Framework documentation, I found that the Tab key is considered a special command key and is handled internally by the .NET Framework, so the KeyDown event is never fired. This is not specific to the Compact Framework. In the .NET Framework, the Tab key is first handled by the ProcessCmdKey key method and then routed to the ProcessTabKey method. Both of these are protected virtual methods which can be overridden in derived controls, but unfortunately neither of these methods is available in the Compact Framework. Even if they were available, the Compact Framework doesn't support direct visual control inheritance, so creating derived controls would pose additional problems.

My next inclination was to use the safety net provided by the .NET Framework and simply access the WM_KEYDOWN message using the WndProc member of the Control class. When I checked the docs, I found that neither the Handle property nor the WndProc procedure was supported in the Compact Framework. I looked into setting up a Windows systems keyboard hook with the SetWindowsHookEx system call to intercept WM_KEYDOWN messages, but hooks aren't supported in the Windows CE operating system. Finally, I decided to use SetWindowLong to subclass control windows and to intercept all window messages, including WM_KEYDOWN. Unfortunately, both SetWindowsHookEx and SetWindowLong work by accepting a pointer to a function, which Windows calls back on message events. Unlike the .NET Framework, P/Invoke in the Compact Framework does not allow callbacks from Windows CE to .NET code. At this point I was stumped, so I dug out my copy of the Microsoft .NET Compact Framework (Core Reference) and my MSDN® CD for some research.

The Mysterious MessageWindow Class

Of course I knew that the designers of the Compact Framework wouldn't remove all those helpful interoperability features. Fortunately, I found the missing link, or at least the missing link between Windows CE and the Compact Framework in Chapter 22 ("Interoperating with Native Code") of the Microsoft .NET Compact Framework (Core Reference).

To allow managed code to call unmanaged code, including the Windows API, the .NET Framework provides the P/Invoke mechanism. For a good description of P/Invoke on the Compact Framework, see the MSDN article, "An Introduction to P/Invoke and Marshaling on the Microsoft .NET Compact Framework" by Jon Box and Dan Fox. The basic P/Invoke is pretty straightforward. You simply declare the unmanaged function you want to call and map the parameters to .NET types in the import signature:

[DllImport("coredll.dll, EntryPoint="GetFocus")]
private extern static IntPtr GetFocus();
•••
control.Focus();
IntPtr handle = GetFocus();
This is a bit of a simplification, especially when dealing with complex types and objects, but Windows API calls, for the most part, use primitive types that map directly to a corresponding .NET type. These are known as blittable types. So that takes care of calls from managed code to unmanaged code, but how does unmanaged code call managed code?

Why would unmanaged code need to call managed code? The answer is asynchronous event notifications. Many asynchronous Windows API calls, such as SetWindowLong, take a function pointer as a parameter. Windows calls back to the function when certain events occur. P/Invoke in the .NET Framework allows .NET delegates to be passed as function pointers, so unmanaged code can directly call managed code. The .NET Framework Control class's Handle and WndProc members allow Windows messages to be posted from unmanaged code to Control objects. Unfortunately, P/Invoke in the Compact Framework doesn't provide any support for function pointers for callbacks and the Compact Framework Control class provides no access whatsoever to underlying windows, so Windows CE messages can't be posted directly to Compact Framework Control objects.

This is where the Compact Framework-only MessageWindow class comes into the picture. This class is included in a special library assembly, Microsoft.WindowsCE.Forms.dll, which must be added to the references list of a SmartDevice project. The MessageWindow class is provided specifically to allow access to Compact Framework managed code from unmanaged code. Like the NativeWindow class in the .NET Framework, the MessageWindow class creates a native window specifically for receiving Windows messages. The MessageWindow class provides access to its native window handle through the Hwnd property and receives messages through the overridden WndProc method.

Now you have a Compact Framework class that can receive Windows messages from Windows CE, but how does that help with asynchronous event notifications and intercepting messages, such as WM_KEYDOWN, from your user interface controls? This is where that tiny bit of native code is required. I'll go into the details of creating the native code DLL with Microsoft eMbedded Visual C++® a little later. Basically, you can write some unmanaged, native code to handle callbacks from Windows CE. The native code then posts a Windows message to the handle of a MessageWindow-derived class. Then you have the ability to intercept the message and handle it in the MessageWindow's overridden WndProc method in managed code.

The Microsoft .NET Compact Framework (Core Reference) and the MSDN article "Using the Microsoft .NET Compact Framework MessageWindow Class" provide examples on the use of the MessageWindow class. One example shows how to find a form's native window handle with the FindWindow call, subclass the window with SetWindowLong, intercept the WM_ACTIVATE message in native code, and finally post a user-defined message to a MessageWindow class in managed code (see the sidebar "Getting A Window Handle" for more information). I realized I could use the same technique to intercept the WM_KEYDOWN messages for controls on a form and handle them before the .NET Framework processed them. This would allow me to use the same focus management code as in my first sample, but with notification of keyboard events for Tab or any other key.

The old adage "the devil is in the details" seems to fit software development perfectly. For this issue, the devil first proved to be figuring out how to get the native window handles for my .NET controls and then ensuring that my subclassing didn't interfere or conflict with the .NET Compact Framework.

Native Window Handles for My .NET Controls

Once again, I started with the subclassing example in the Microsoft .NET Compact Framework (Core Reference). This example uses the FindWindow system call to get the handle of a Compact Framework form. The FindWindow call only returns the handles of top-level windows, which works for .NET-based forms in the .NET Framework. Other system calls must be used to find the handles on child controls.

With a little experimentation, I found that GetFocus and WindowFromPoint work well for controls on a form. GetFocus simply returns the control with focus. In order to use this call, each control must be given focus with the Control.Focus method prior to using this call. Obviously, you wouldn't want to do this when a form is visible, but it can be done inconspicuously in the form's Load event handler before the form is displayed for the first time, as shown in the code you saw a few paragraphs ago.

WindowFromPoint, as the name implies, simply returns the handle of the window containing the given point. Since the .NET Control class provides access to its bounds rectangle, WindowFromPoint can also be used to return the handle of the controls. This call must be used with care, however, because it returns the topmost window when controls are stacked, and doesn't recognize nonvisible controls. There are also a couple of issues with calling WindowFromPoint through P/Invoke. The WindowFromPoint system call expects absolute screen coordinates, not taking into account the Windows CE taskbar at the top of the screen. The .NET Control coordinates, on the other hand, are relative to the parent form, which is positioned below the taskbar. Therefore, the relative control coordinates have to be adjusted to absolute coordinates, which are expected by WindowFromPoint:

int x = control.Top +  form.Top;
int y = control.Left +  form.Left;
IntPtr handle = WindowFromPointYX(y,x);

You'll also notice that I'm calling WindowFromPointYX, which has the X and Y coordinates reversed, instead of WindowFromPoint. This is because the WindowFromPoint system call actually expects a POINT structure passed by value. P/Invoke only allows structures to be passed by reference, but as long as they contain blittable types, structures can be passed by value by placing the individual members directly on the stack. The POINT structure consists of two DWORD coordinate parameters, so WindowFromPoint can be called with two .NET integer parameters. Structures are laid out sequentially, but since P/Invoke uses the C calling convention, which places parameters on the stack right to left, the parameters must be reversed. To make it clear what's going on, it's a good idea to give the imported method an alias:

[DllImport("coredll.dll", EntryPoint="WindowFromPoint")]
private extern static IntPtr WindowFromPointYX(int y, int x);

As you can imagine, when working with these system calls, especially WindowFromPoint, it's easy to get the wrong window handle. During development, I needed to be able to determine if I was getting the correct handles for my Compact Framework controls. This is where some of the tools that come with eMbedded Visual C++ come in handy.

Microsoft eMbedded Visual C++ includes several Windows CE versions of standard SDK tools, which run remotely on the desktop over an ActiveSync connection. I used Windows CE Remote Spy extensively to view window handles, window class attributes, and Windows messages. Windows CE Remote Spy is more basic than the Spy in the Windows Platform SDK. Windows CE Remote Spy doesn't decode the window style flags, so you will have to use the Winuser.h file from eMbedded Visual C++ to manually decode style attribute flags, some Windows messages IDs, and the wParam and lParam values. Despite a little extra work, this tool is essential for checking window handles and is invaluable for observing how the .NET Compact Framework interacts with Windows CE and responds to Windows messages.

This brings me to my last point about working with a control's native handles. To keep from interfering with the .NET Framework, window handles should not be retrieved until after the .NET Framework has finished creating the native windows for all controls on a form. The form's Load event works well for this. Unlike many other form events, the Load event doesn't correspond directly to a Windows message. It is simply fired once by the .NET Framework prior to the first time a form is displayed, after all initialization is complete.

Intercepting Windows Messages

Once you have the native window handle for a control on a form, the next step is to use subclassing to capture the Windows messages; in my case it's the WM_KEYDOWN message. The MSDN library defines subclassing as "a technique that allows an application to intercept and process messages sent or posted to a particular window before the window has a chance to process them." For a good overview of subclassing, you can refer to the MSDN article, "Safe Subclassing in Win32," by Kyle Marsh.

All native windows interact with the Windows operating system using window procedures, but developers don't typically use window procedures directly in their applications. Instead, application frameworks, including the .NET Framework, take care of this. When you subclass the native windows for .NET controls, you're telling Windows to send messages to your window procedure instead of the .NET Framework-provided window procedure. Obviously, you want your controls' native windows to still work with the .NET Framework, so you have be sure to forward all the messages you don't handle to the original .NET window procedure.

Once again, since the Compact Framework does not allow callbacks from Windows to managed code, the actual subclassing is done in native code. The good news is there's not much code involved and you don't have to worry about purchasing another development environment (eMbedded Visual C++ can be downloaded from the MSDN Mobility Web site). At the end of this article, I will tell you how to download eMbedded Visual C++ and create a native DLL project.

Strictly speaking, to add keyboard support to a Compact Framework-based application, only the WM_KEYDOWN message needs to be intercepted, so I could have evaluated the message type in native code and forwarded just the WM_KEYDOWN message or a custom message to my Compact Framework-based application code as demonstrated in the .NET Compact Framework (Core Reference) sample. Instead, I opted for a more generic architecture and decided to forward all Windows messages to managed code. While this incurs some extra overhead, I decided that the combined benefit of keeping all my application logic together in managed code and the flexibility of reusable subclassing code was well worth a slight performance trade-off.

Therefore, I created a Subclasser utility class, shown in Figure 2, which encapsulates the subclassing of a control's native window and exposes the window's messages to managed code through a .NET event. This class effectively replaces the missing functionality of the full .NET Framework Control class.

Figure 2 Subclasser.cs (Close Figure 2)
using System;
using System.Runtime.InteropServices;
using System.Collections;
using Microsoft.WindowsCE.Forms;

namespace PPCTools
{

    /// <summary>
    /// Subclasser subclasses native windows and fires the MessageEvent
    /// on all messages.
    /// </summary>
    public class Subclasser : IDisposable
    {
        private class SubclasserMessageWindow : MessageWindow
        {
            private Subclasser subclasser;

            public SubclasserMessageWindow(Subclasser subclasser)
            {
                this.subclasser = subclasser;
            }

            protected override void WndProc(ref Message msg)
            {
                if (subclasser.Message != null)
                {
                    MessageEventArgs msgEventArgs = new
                        MessageEventArgs(msg);
                    subclasser.Message(subclasser, msgEventArgs);
                    // allow event handler to modify message parameters
                    msg.Msg = msgEventArgs.Msg;
                    msg.WParam = msgEventArgs.WParam;
                    msg.LParam = msgEventArgs.LParam;
                    msg.Result = msgEventArgs.Result;
                }
            }
        }

        [DllImport("native.dll")]
        private extern static void Subclass(IntPtr subclassWindow,
                                            IntPtr messageWindow);

        [DllImport("native.dll")]
        private extern static void Unsubclass(IntPtr subclassWindow);

        private SubclasserMessageWindow messageWindow = null;
        private ArrayList windowHandles = new ArrayList();

        /// <summary>
        /// Fired on messages from all subclassed native windows
        /// </summary>
        public event MessageEventHandler Message;

        /// <summary>
        /// Default constructor
        /// </summary>
        public Subclasser()
        {
            messageWindow = new SubclasserMessageWindow(this);
        }

        /// <summary>
        /// Constructs Subclasser and subclasses specified native
        /// window.  Useful when Subclasser is used for single window
        /// </summary>
        /// <param name="hWindow">Native window handle</param>
        public Subclasser(IntPtr hWindow) : this()
        {
            AddWindow(hWindow);
        }

        public IntPtr Hwnd
        {
            get
            {
                return messageWindow.Hwnd;
            }
        }

        /// <summary>
        /// Subclasses specified native window
        /// </summary>
        /// <param name="hWindow">Native window handle</param>
        public void AddWindow(IntPtr hWindow)
        {
            windowHandles.Add(hWindow);
            Subclass(hWindow,messageWindow.Hwnd);
        }

        /// <summary>
        /// Unsubclasses specified native window
        /// </summary>
        /// <param name="hWindow">Native window handle</param>
        public void RemoveWindow(IntPtr hWindow)
        {
            windowHandles.Remove(hWindow);
            Unsubclass(hWindow);
        }

        /// <summary>
        /// Unsubclasses all native windows
        /// </summary>
        public void Dispose()
        {
            while (windowHandles.Count > 0)
            {
                RemoveWindow((IntPtr)windowHandles[0]);
            }
        }

        ~Subclasser()
        {
            Dispose();
        }

    }
}

Since finding the correct native window handle for a control depends on whether you're working with a form or child control, native handles must be obtained before using this class. I have separate focus management code, described in the next section, which takes care of this.

I designed Subclasser to intercept messages for one or more native windows. A single window handle can be passed in a constructor or multiple windows can be added with the AddWindow method. Subclasser contains only one MessageWindow, so it's not possible to tell which message was meant for which window when there are multiple subclassed windows. This isn't a problem for keyboard handling code, where I'm already keeping tracking of the active focus control. However, if you need to keep track of messages per window, you can simply create a separate Subclasser instance for each of your windows.

When Subclasser receives a message for any of its subclassed windows, it fires its Message event using the MessageEventHandler delegate, as you can see in the following code:

public delegate void MessageEventHandler(object source,
    MessageEventArgs args);
The event handler can then process or change the message. But to keep from interfering with .NET, messages should be ignored by default. Unless the message handler sets the Result field of the MessageEventArgs to a nonzero value, the native code forwards the message to the .NET message queue for normal processing.Getting A Window Handle

The book Microsoft .NET Compact Framework (Core Reference) shows how the FindWindow system can be used to locate the handle for a Compact Framework form as long as it has a uniquely identifiable title. Even when your forms don't have a unique title or, for that matter, any title at all, you can still use FindWindow to find a form handle by temporarily setting the form's Text property to a unique string, just before calling FindWindow, and restoring the original title afterward:

string oldTitle = form.Text;
form.Text = "F5486279-64FC-4f9a-B1ED-EDC941DD05D0";
IntPtr hWnd = FindWindow(form.Text,null);
form.Text = oldTitle;

Also, GetActiveWindow can be used as an alternative to FindWindow for forms. The active window is the topmost window which contains the current focus control. GetActiveWindow can be used to return the form's native window handle in form's Load event, which is fired by the .NET Framework just before the form is displayed for the first time.

Subclasser provides a convenient .NET event interface for messages received by the MessageWindow class, but the real work to intercept messages is done in the native Subclass function, which is called by the Subclasser.AddWindow method:

[DllImport("CFNative.dll")]
private extern static void Subclass(IntPtr subclassWindow,
    IntPtr messageWindow);
•••
public void AddWindow(IntPtr hWindow)
{
    windowHandles.Add(hWindow);
    Subclass(hWindow,messageWindow.Hwnd);
}

Actually, even the real work is only a few lines of C++ code. Figure 3 shows the header and source file for CSubclasser. CSubclasser.cpp provides the implementation of CSubclasser, a C++ class that encapsulates the subclassing of native windows. It also includes the two C-style entry points, Subclass and Unsubclass, which are required for the P/Invoke.

Figure 3 CSubclasser (Close Figure 3)

CSubclasser.h
class CSubclasser
{
    WNDPROC oldWndProc_;  // original .NET window procedure
    HWND messageWindow_;  // Compact Framework MesssageWindow class
    HWND subclassWindow_; // control's native window

    static LRESULT APIENTRY SubclassWndProc(HWND hwnd,
           UINT msg, WPARAM wparam, LPARAM lparam);

public:
    CSubclasser(HWND subclassWindow,HWND messageWindow);
    ~CSubclasser(void);
};
CSubclasser.cpp
#include "stdafx.h"
#include "CSubclasser.h"

// Use MFC map collection to keep track of CSubclasser objects
CMapPtrToPtr Subclassers;

// Keep track of the Compact Framework control's native window handle
// (subclassWindow),the MessageWindow's handle, and the original .NET
// window procedure address.
CSubclasser::CSubclasser(HWND subclassWindow,HWND messageWindow)
{
    subclassWindow_ = subclassWindow;
    messageWindow_ = messageWindow;
    oldWndProc_ = (WNDPROC) SetWindowLong(subclassWindow_,
                  GWL_WNDPROC, (LONG)SubclassWndProc);
}

// Clean up and restore the original .NET window procedure when the object
// is deleted
CSubclasser::~CSubclasser()
{
    SetWindowLong(subclassWindow_,GWL_WNDPROC,(LONG)oldWndProc_);
}

// The window procedure is a static member shared by all CSubclasser
// instances. It uses a global map to look up the CSubclasser object by
// the hwnd of the message.

LRESULT APIENTRY CSubclasser::SubclassWndProc(HWND hwnd,
        UINT msg, WPARAM wparam, LPARAM lparam)
{
    LRESULT res = 0;
    CSubclasser* subclasser = NULL;
    if (Subclassers.Lookup(hwnd,(void*&)subclasser))
    {
        LRESULT res = SendMessage(subclasser->messageWindow_,
                                  msg,
                                  wparam,
                                  lparam);
        if (!res)
            res = CallWindowProc(subclasser->oldWndProc_,
                      hwnd, msg, wparam, lparam);
    }
    return res;
}

// Entry points for .NET Framework P/Invoke

EXTERN_C _declspec(dllexport) void Subclass(HWND subclassWindow,
                                            HWND messageWindow)
{
    CSubclasser* subclasser = new CSubclasser(subclassWindow,
                                              messageWindow);
    Subclassers[subclassWindow] = subclasser;
}

EXTERN_C _declspec(dllexport) void Unsubclass(HWND subclassWindow)
{
    CSubclasser* subclasser = NULL;
    if (Subclassers.Lookup(subclassWindow,(void*&)subclasser))
    {
        Subclassers.RemoveKey(subclassWindow);
        delete subclasser;
    }
}

The Subclass method takes two parameters: the handle of a native window to subclass and the handle of Subclasser's MessageWindow to forward messages. For each native window, a new CSubclasser instance is created to keep track of the native window handle, the Compact Framework MessageWindow handle, and the original .NET window procedure address. The CSubclasser constructor calls SetWindowLong with the address of its shared SubclassWndProc member. After-ward, Windows calls SubclassWndProc with all messages for the window. For each subclassed window, SubclassWndProc simply sends the messages to the Compact Framework MessageWindow first, and then forwards the unhandled messages to the original .NET window procedure. Note that an MFC map collection, CMapPtrToPtr, is used to look up the CSubclasser for a particular native window since the SubclassWndProc member of CSubclasser is static and is shared among all CSubclasser instances for all subclassed windows.

Managing Keyboard Focus

.NET Framework controls have a TabStop property, which dictates whether the control can accept focus, and a TabIndex property, which dictates in what order it receives focus relative to other focusable controls. While it would be nice to add TabStop and TabIndex properties to a set of derived Compact Frameworks, I chose to do things a little differently. First of all, I wanted to work with the stock controls in the toolbox. Even if I had been inclined to create my own version of all the controls, there is another limitation of the Compact Framework that makes this a little more difficult. In the .NET Framework, it is easy to create custom controls to extend the behavior of an existing control through visual inheritance in the designer. The Compact Framework doesn't support visual inheritance—at least not without some extra work.

The process is a bit complicated and is well beyond the scope of this article. See Chapter 18 (Custom Control) of the Microsoft .NET Compact Framework (Core Reference) or the MSDN article, "Creating Custom Controls for the .NET Compact Framework" for the details involved in creating your own custom Compact Framework controls.

For my purposes, I decided to use a special collection to implement the functionality of TabStop and TabIndex. I created a FocusControls collection class similar to the Form's Controls collection. While the Controls collection contains all the controls on a form, my FocusControls collection contains only the focusable controls on the form. All of the controls in this collection implicitly have TabStop set to true, and the order of controls in this collection determines the TabOrder.

In the .NET Framework, the first control in the tab order receives focus the first time the form is activated. The user can then advance focus to the next control in the tab order using the Tab key. Afterward, when a form is deactivated and reactivated, the last control to have focus automatically regains it. In the Compact Framework, when a form is activated, only the form gets focus. For each form, the FocusControls class remembers which control on a form has focus, and assigns focus whenever the form is activated. Like the Form class (which implements the IControlContainer interface) in the .NET Framework, the FocusControls class has an ActiveControl property, which stores the control with focus:

public Control ActiveControl
 {
     get
     {
         return activeControl;
     }
     set
     {
         if (this.Contains(value))
             value.Focus();
     }
}

As with the .NET Framework, the first control in the tab order, or in my case, the first control in the collection, receives focus initially and is assigned to the activeControl. By subscribing to the Form.GotFocus event, the FocusControls class forwards focus to the activeControl whenever the form receives focus:

private void form_GotFocus(object sender, System.EventArgs e)
{
    if (activeControl != null) activeControl.Focus();
}

In addition to advancing focus with the Tab or Enter key, focus can be changed with other input devices. On the desktop, this is usually a left mouse click, while on the Pocket PC it is a stylus touch. The last detail of handling focus is making sure that my activeControl member stays in sync with the UI. By also subscribing to the GotFocus event on all the controls in my collection, I can simply update my activeControl member in the GotFocus handler:

private void control_GotFocus(object sender, EventArgs e)
{
    activeControl = sender as Control;
}

The check to determine if activeControl is not null in the form's form_GotFocus event handler is necessary because the GotFocus event is fired before the Load event. The Load event is fired one time just before a form is displayed for the first time. This occurs after the form has been constructed and all its native windows have been created, so this is the ideal point at which to subclass controls on the form. The FocusControls collection subscribes to the form's Load event and implements an event handler, form_Load, where it subclasses all the controls in the collection. Unless the user has explicitly set the ActiveControl property, the activeControl member will not have been set the first time the form is displayed. The code in the form_Load event handler, shown in Figure 4, mimics standard focus behavior and sets the activeControl to the first focusable control, if it hasn't already been set, and finally sets focus to the activeControl, assuming one is available.

Figure 4 form_Load Event Handler (Close Figure 4)
private void form_Load(object sender, System.EventArgs e)
{
    if (!initialized)
    {
        initialized = true;

        foreach (Control control in controls)
        {
            subclassControl(control);
            control.GotFocus += new EventHandler(control_GotFocus);
        }

        // If there are focusable controls and none has focus,
        // set focus to first control
        if (activeControl == null && controls.Count > 0)
            activeControl = this[0];

        if (activeControl != null)
            activeControl.Focus();
    }
}

The private subclassControl method makes use of Windows system calls to get the native window handle for each control and then sets up subclassing for the native window using an instance of the Subclasser class (see Figure 5).

Figure 5 subclassControl (Close Figure 5)
private void subclassControl(Control control)
{
    // Get handle of native window by coordinates of top,left position of
    // control. Control position is relative to form, which is located
    // under Window CE task bar.  WindowFromPoint is expects absolute
    // screen position, so adjust coordinates by position of form.
    int x = control.Top +  form.Top;
    int y = control.Left +  form.Left;
    IntPtr handle = WindowFromPointYX(y,x);

    // Alternate way to get handle of control
    //control.Focus();
    //IntPtr handle = GetFocus();

    subclasser.Add(handle);
}

Finally, FocusControls implements an event handler, subclasser_MessageEvent, for the Subclasser Message event, which is fired on every Windows message for subclassed native windows, as shown in Figure 6. The subclass_MessageEvent handler ignores all events except for keyboard messages, which it forwards to any listeners on its own KeyMessage event.

Figure 6 Implementing subclasser_MessageEvent (Close Figure 6)
public void subclasser_MessageEvent(object source, MessageEventArgs args)
{
    switch(args.Msg)
    {
        case WindowsMessages.WM_KEYDOWN:
        case WindowsMessages.WM_CHAR:
        case WindowsMessages.WM_KEYUP:
        case WindowsMessages.WM_SYSKEYDOWN:
        case WindowsMessages.WM_SYSCHAR:
        case WindowsMessages.WM_SYSKEYUP:
            // capture all keyboard messages
            if (KeyMessage != null)
                KeyMessage(this,args);
            break;
    }
}

While I could have handled the Tab and Enter keys internally in subclasser_MessageEvent and advanced focus internally, I chose to simply expose keyboard events with the KeyMessage event and let the form decide what to do in its KeyMessage event handler. This design allows forms to implement validation logic similar to the Validating and Validated events found in the .NET Framework before advancing focus.

Implementing a KeyMessage Event Handler

Sometimes you may want to check for a special key, such as a Control-C (break), Alt-F (file menu accelerator), or a capital letter. These are generated by simultaneously holding down the Control, Alt, or Shift keys and pressing another key. The Control, Alt (or Menu), and Shift keys are known as modifiers because they modify the state of other regular keys. FocusControls fires its KeyMessage event for each raw key message from Windows, so you could keep track of the modifier key and regular keys' KEY_DOWN messages.

Luckily there's an easier approach. Windows keeps track of the state of Modifier keys for you. You can retrieve the state of these keys with the GetKeyState system call. Figure 7 shows a sample event handler, focusControls_KeyMessage, for the FocusControls KeyMessage event. For each WM_KEYDOWN and WM_SYSKEYDOWN message from a control, the handler checks the Modifier state with GetKeyState, constructs a KeyEventArgs object, and fires the form's KeyDown event.

Figure 7 focusControls_KeyMessage (Close Figure 7)
private void focusControls_KeyMessage(object sender,
                      MessageEventArgs args)
{
    try
    {
        // Create KeyEventArgs to fire OnKeyDown event
        int keyCode = (int)args.WParam;
        int keyModifier = 0;
        if (GetKeyState((int)Keys.ShiftKey) < 0)
            keyModifier |= (int)Keys.Shift;
        if (GetKeyState((int)Keys.ControlKey) < 0)
            keyModifier |= (int)Keys.Control;
        if (GetKeyState((int)Keys.Menu) < 0)
            keyModifier |= (int)Keys.Alt;

        if (args.Msg == WindowsMessages.WM_KEYDOWN ||
            args.Msg == WindowsMessages.WM_SYSKEYDOWN)
        {
            KeyEventArgs keyEventArgs =
              new KeyEventArgs((Keys)(keyCode | keyModifier));
            OnKeyDown(keyEventArgs);
            args.Result = keyEventArgs.Handled ? (IntPtr)1 : IntPtr.Zero;
        }
    }
    catch (Exception e)
    {
        // marshal exception back to main UI message pump
        exception = e;
        timer1.Interval = 1;
        timer1.Enabled = true;
    }

}

You'll also notice that this KeyMessage handler code has a try-catch block with some unusual timer code. If you have a try-catch exception-handling block around the main thread of your application, you'll quickly discover that, by default, it doesn't catch exceptions that are thrown while handling subclassed messages. While these messages are handled by the same Windows CE thread that is running the main message queue of the application, they are running in a different .NET stack context. This is because the native code window procedure is calling SendMessage, which directly calls the MessageWindow window procedure rather than going through the application's message queue.

You can still catch exceptions thrown while handling subclassed messages by marshaling the exceptions back to the stack context of the application's main message queue. This is where the timer is used. The System.Windows.Forms.Timer class performs the act of firing events by posting messages back to the message queue of the window on which it was created, regardless of where the forms timer was started. To marshal an exception in the subclassed message handler code, you should catch the exception locally, save it to a member variable, and then fire the forms timer. The forms timer will be signaled by a message in the main message queue and the timer event handler can then re-throw the exception in the main UI stack context:

private void timer1_Tick(object sender, System.EventArgs e)
{
    // get timer message and re-throw exception
    timer1.Enabled = false;
    throw exception;
}

Putting It All Together

Figure 8 shows how both the Subclasser and FocusControls classes work with Windows messages and the .NET Compact Framework. After obtaining native window handles for .NET Controls with Windows systems calls, the windows are subclassed in native code with SetWindowLong, which causes their messages to be passed to CSubclasser's SubclassWndProc. Then using SendMessage, SubclassWndProc sends the messages to the MessageWindow-derived class owned by the managed Subclasser class. Subclasser exposes Windows messages in managed code via its Message event. FocusControls, the class responsible for managing keyboard focus of controls on a form, implements an event handler for its Subclasser instance. The handler ignores all messages for form controls, except keyboard messages, which are exposed through the FocusControl's KeyMessage event.

Figure 8 Window Subclassing and Message Flow
Figure 8 Window Subclassing and Message Flow

The Subclasser and FocusControls event handlers can indicate that the message event has been handled, which prevents the message from being sent to its corresponding .NET Control class. However, to keep from interfering with the normal operation of the .NET Framework, all Windows messages are typically forwarded by CSubclasser's SubclassWndProc to the .NET Compact Framework Control's internal WndProc method. This allows the .NET Framework to interact with native windows and triggers message-related .NET events, such as KeyDown, Paint, GotFocus, LostFocus, and so on.

Building and Deploying the Native DLL

To write Windows CE native code for the Pocket PC, you need to use eMbedded Visual C++, which can be downloaded from http://msdn.microsoft.com/mobility/downloads/sdks. Be aware, however, that different versions of eMbedded Visual C++ are targeted to specific versions of Windows CE and Pocket PC, so it's tricky to figure out exactly what to download. Windows Mobile 2002 for Pocket PC runs on Windows CE 3.0 and requires eMbedded Visual C++ 3.0, which is included in the Pocket PC SDK and eMbedded Visual Tools 3.0. The newer Windows Mobile 2003 for Pocket PC runs on Windows CE .NET (Windows CE 4.2) and requires eMbedded Visual C++ 4.0, which is packaged in the SDK for Windows Mobile 2003-based Pocket PCs.

Once you have downloaded the appropriate version, the project wizards make creating the native DLL project very easy. P/Invoke can only call into a DLL, so choose WCE Dynamic-link Library or WCE MFC AppWizard (DLL) for the project type. I chose an MFC DLL simply for the collection classes. If you use MFC, be sure to choose the shared MFC DLL project and build a release DLL target, since the release versions of the MFC shared libraries are already included on the Pocket PC. This will result in the smallest footprint for the native DLL.

While you can use C++ to write your native code, you will need to export simple C-style functions for P/Invoke. Each exported function should be prefixed with:

EXTERN_C _declspec(dllexport)
The EXTERN_C specifies the C calling convention, which is required by the Compact Framework P/Invoke and _declspec(dllexport) indicates that the method is to be exported. For complete details on how to format functions for P/Invoke, see the MSDN article "Writing Unmanaged Functions for Microsoft .NET Compact Framework-based Applications".

Unlike Compact Framework SmartDevice projects in Visual Studio, in eMbedded Visual C++ you'll have to specify the platform that you're targeting and build a separate binary for each. If you've been doing much Compact Framework development, you may have found the Pocket PC emulator (which runs on the desktop) useful. You may be surprised to learn that subclassing will work with the emulator as well. In addition to your target Pocket PC platform, you'll need to create an x86 version of the native DLL for use in the emulator.

Since the native code is in a traditional DLL rather than a .NET assembly, Visual Studio .NET debugging and deployment tools are unaware of the dependency on the native DLL. The DLL must either be in the same directory as the Compact Framework-based application or on the device path. If P/Invoke cannot locate a DLL, you will get a "MissingMethodException" whenever you try to make a native call.

When you build the native DLL in eMbedded Visual C++, by default it automatically deploys the DLLs to the Windows directory of a Pocket PC, which is always in the device system path. You can also use the file explorer in ActiveSync® to place DLLs manually.

Copying the DLL to the emulator isn't quite as straightforward. You have to set up a network share between the emulator and your desktop PC to transfer the native DLL to the emulator image. This process is described in the MSDN article "Using the Emulator in Smart Device Projects".

This takes care of the development environment, but you'll want to include the native DLL in your application installation as well. Visual Studio .NET includes the CabWiz application installation tool, "Build Cab File" on the Build menu, that generates self-installing CAB archives for the Pocket PC. When you run CabWiz, it creates a CAB directory under the project containing self-installing CAB files for all supported Pocket PC platforms. Additionally, it creates a BuildCab batch file and a CabWiz INF file in the project's obj directory. To be included in the CAB, the native DLL needs to be manually added to the INF file. The MSDN article, "Generating Custom CAB files for Device Projects", walks through the process of using CabWiz and modifying the INF files.This can be pretty tricky, so an easier alternative is to add the native DLL to your .NET assembly project as a content file. Keep in mind, this will only work if you're dealing with one platform and build type.

Conclusion

After reading this article, you'll have a good idea of how the Compact Framework handles, or rather doesn't handle, keyboard focus. As I've shown, with the Compact Framework MessageWindow class and a little helper native code, you can subclass Compact Framework controls, and intercept Windows CE messages. Then, by intercepting the WM_KEYDOWN message, you can add standard keyboard navigation to Compact Framework-based apps that run on the Pocket PC.

If you're like me, however, what you really want is some code to help make your life easier. I've provided four sample projects accompanying this article: PPCKeyboardSample1, PPCKeyboardSample2, CFTools and CFNative. PPCKeyboardSample1 shows how to add basic keyboard support (without tab key support) to Compact Framework-based applications using only the built-in Control.KeyDown event. PPCKeyboardSample2 shows how to add full keyboard support to a Compact Framework-based application using subclassing, and demonstrates the processes of form-level keyboard handling and exception marshaling. PPCKeyboardSample2 includes the library project CFTools, which in turn uses the native DLL project called CFNative.

The CFTools library project contains both the Subclasser and FocusControls classes discussed in this article. These classes can be used as-is in any Compact Framework-based application. To use these classes, you simply have to build the CFTools assembly and the supporting CFNative DLL. Then, add a reference to the CFTools DLL and make sure the native CFNative.DLL is in the path on the target device.

Alan Pulliam is a software architect with over 14 years of industry experience. He resides in Atlanta and is currently working as an independent consultant. He can be contacted at apulliam@iname.com.


Editor's Note: Testing, Testing 1 2 3

Editor's Note
Testing, Testing 1 2 3

Each month dozens of readers turn to this page for the latest news on what the editors of MSDN Magazine are thinking. And we do our best to oblige with our hard-hitting yet pithy observations on the state of the world around us, the latest in XML fashions, what's coming up in the current issue, and more. Well, not this month. For a delightful change of pace, we've got some actual news to discuss in this month's Editor's Note, so let's dive right in.

This month, we've put together a special issue on testing with four feature articles on some advanced topics. Leading off, we'll explain to you the fundamentals of code coverage in .NET. Code coverage is a way to make sure that all parts of your application are tested properly—it doesn't determine that they work properly, just that they're covered by your test suite.

Following that, we discuss extreme programming—a discipline that's rapidly growing in popularity. Through the test-driven development that is the hallmark of extreme programming, you can build more reliable software ecosystems before the design gets too complex and becomes unwieldy to test.

Instrumentation is a key element of testing and monitoring an application's health, both during development and after deployment. In this month's issue, we take a look at several options you have for instrumenting your code using the .NET Framework.

Finally, we wrap up our look at testing and software quality by walking you through a test application that generates load tests for your applications, letting you see more clearly where your project is likely to fail in real-world situations.

With all of these features, the message is simple. Design your programs so that they can be properly tested. While it may be just a dream to create zero-defect software, modern development techniques can help us approach that ideal.

We're also pleased to announce the launch of a new column this month—.NET Matters. Love answers, but hate having to come up with them yourself? Curious about some of the inner workings of the .NET Framework? Stephen Toub, our own technical editor and resident .NET expert, capitalizes on our popular Q&A format to answer some of the trickiest questions about .NET, asked by real developers both inside and outside Microsoft.

As you flip through this month's issue (and we strongly encourage you to purchase it instead of just flipping through someone else's copy), you may notice that we've redesigned the first page of each feature article. It's more than just some new fonts and colors; the summary box for each article has been revamped based on valuable reader input.

We've simplified the summary boxes to make them easier to scan for information. We'll tell you the technologies used in each article. To help you make your decision on whether to read the article now or save it for later, we'll provide a bulleted list of topics that are discussed. If an article has a downloadable code project associated with it, we'll tell you how to get a hold of it. Finally, readers asked us to put information about the author at the beginning of each piece, so we've moved the bio from the end of each feature to the summary box on the first page.

All the content in MSDN Magazine is reviewed for accuracy by Microsoft experts in their particular fields. This helps us bring you accurate content with the most up-to-date information about the topics we cover. Starting this month, we're thanking these important contributors by featuring them on the Editor's Note page. Thanks for helping to make MSDN Magazine a success.

—J.T.

Thanks to the following Microsoft technical experts: Jay Bazuzi, Scott Berry, Subash Bhamidipati, Karim Elias, Dave Fetterman, Cindy Gross, Eric Gunnerson, Mario Hewardt, Jesper Holmberg, Rob Howard, John Iwasz, Raja Krishnaswamy, Ronald Laeremans, Rico Mariani, Ivan Medvedev, Craig Neable, Dare Obasanjo, Matt Powell, Elliot Rapp, Geoff Schwab, Yasser Shohoud, and Jonathan Wells

ActiveSync, ActiveX, IntelliSense, JScript, Microsoft, MSDN, MS-DOS, Outlook, Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Mobile and Windows Server are trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Theresa W. Carey

Contents
Presentation Layer Toolset
Bidirectional Data Synchronization
Plot, Compute, Interpret, and Embed
Design and Distribute Reports
Web-based Collaboration Server
Serial Comm Toolkit for .NET
Transition Legacy Apps in .NET
Database Designer
The Bookshelf

Presentation Layer Toolset

Infragistics is shipping NetAdvantage 2004 Volume 1, a toolset for developing commercial-quality user interfaces for the Microsoft® .NET Framework, ASP.NET, Tablet PC, and COM. NetAdvantage 2004 Volume 1 features Windows® Forms and ASP.NET components that resemble the Outlook® 2003 interface for both the Microsoft .NET Framework explorer bar and scheduling elements.

NetAdvantage 2004 Volume 1 also features data input elements for ASP.NET. Several elements are compliant with Section 508 of the Rehabilitation Act for the visually impaired, including the ASP.NET menu, tree and grid, and the Windows Forms grid, toolbars, and editors.

Additional features include new chart types for .NET and ASP.NET charting, and enhancements to the ASP.NET presentation layer framework. Subscribers receive all source code for .NET and ASP.NET designers and elements. You'll also find tabs, explorer bars, menus, listbars, trees, a dock manager, UI and editing elements, and more.

http://www.infragistics.com

Bidirectional Data Synchronization

MaXware International AS has released MaXware ExpresSync, a one-to-one data synchronization tool that enables enterprises to manage ongoing information integration demands by securely moving data between any two repositories or applications, independent of platform, protocol, type, or location. The product offers bidirectional synchronization capabilities and the ability to manipulate data in transfer. The MaXware ExpresSync data processing engine is designed to efficiently process large amounts of data, which results in more rapid and efficient data integration across the enterprise. Additional features include a graphical user interface, support for JavaScript, VBScript, and PERL, and the ability to create audit trails to keep track of changes down to the attribute level.

ExpresSync also includes a proprietary delta mechanism that is intended to ensure that updated data is written to the target repositories, thus eliminating the need to update all the data every time synchronization takes place. The result is a reduction in the write load of the target system. To reduce the read load on the source system, ExpresSync can utilize existing change-log features of directory servers or add triggers to SQL Server™ databases to achieve the same functionality. Note that ExpresSync provides support for SSL/TLS for encryption.

http://www.maxware.com

Plot, Compute, Interpret, and Embed

SoftIntegration Inc. recently announced the availability of Ch 4.5 and Embedded Ch 4.5. Ch is a C/C++ interpreter for cross-platform scripting, 2D/3D plotting, numerical computing, shell programming, and embedded scripting. Product enhancements include C++ features, file name completion using a tab key in command shell, easier interface with binary C/C++ libraries and embedded scripting, and bug fixes.

Ch supports 1999 ISO C Standard (C99) and C++ classes. Ch also supports many industry standards with over 8,000 functions. Functions in static or dynamical binary C/C++ libraries can be executed interpretively in scripting without recompilation using the Ch SDK.

Ch shell is a C-compatible shell while C shell (csh) is a C-like shell. Ch is an interpretive implementation of C, similar to Unix and MS-DOS® shells. Ch in Windows includes over 100 Unix commands for portable shell programming. Ch can also be used as a login shell just like sh, csh, and ksh. Ch has built-in 2D/3D plotting support, generic mathematical functions, computational arrays for linear algebra and matrix computations, and advanced high-level numerical functions for linear systems, differential equation solving, Fourier analysis, and curve fitting. Ch SDK 4.5 can integrate the existing C/C++ libraries as part of the Ch scripting module. Functions and classes in C/C++ static or dynamic libraries can be invoked either in a Ch script file or in a Ch command shell.

Embedded Ch allows users to embed Ch into other C/C++ applications and hardware. With Embedded Ch, C/C++ applications can be extended with all features of Ch. It allows execution of C/C++ scripts generated dynamically online. Ch is available for Windows, Linux, Mac OS X, Solaris, and HP-UX.

http://www.softintegration.com

Design and Distribute Reports

Business Objects, which recently acquired Crystal Decisions, has announced its newest suite of products, Crystal Version 10. The suite combines Crystal Enterprise, an enterprise reporting platform, with the report design capabilities of Crystal Analysis and Crystal Reports.

The product suite includes greater user interactivity and tighter integration with Microsoft Office, application integration options with a new Java report engine, advanced controls for developers using Visual Studio® .NET, increased performance and scalability, as well as large-scale administration capabilities, and a new architecture for data access.

In conjunction with the launch, Business Objects is also introducing two new professional accreditation programs, Crystal Enterprise Certified Professional (CECP) program and the Crystal Reports Certified Professional (CRCP) program, to maximize an organization's effectiveness when deploying Crystal Version 10.

http://www.businessobjects.com

Web-based Collaboration Server

Akiva has announced the availability of WebBoard 7.0, a Web-based collaboration server. WebBoard 7.0 includes features that track recent trends in online collaboration including blogging, RSS/XML feeds (inbound and outbound), calendaring, polling, message ratings, and a SOAP (Web services) interface.

Other WebBoard features include Web-based discussion forums, e-mail discussion lists, News (NNTP) discussions, instant messaging, chat rooms, and whiteboards. WebBoard has integrated all of these tools into one Windows-based server, which provides access control, moderation, reporting, searching, and customization.

WebBoard is used within education and corporate environments to create virtual classrooms, for internal project collaboration, for establishing standards and best practices, and for customer support applications. The new XML and SOAP interfaces enable integration of the product into other enterprise applications.

http://www.akiva.com

Serial Comm Toolkit for .NET

ComponentScience Inc. has released TransPort, a serial communications toolkit for the .NET Framework. TransPort, written in C#, handles situations as simple as reading a serial port or as complex as handling a file transfer through protocols such as XModem, YModem, and ZModem.

Features include multithreaded access to any of the computer's physical and virtual COM ports on any bus (including USB), a drop and go Winsock port for multithreaded, asynchronous access to the TCP/IP layer, and real-time, regex-based DataPackets which provide flexible handling of information received over a serial or Winsock port.

TransPort also includes RAS client support to establish, manage, and control remote network connections, TTY emulation for real-time, user-driven interaction, and the ability to monitor and manage telephony devices via TAPI.

http://www.componentscience.net

Transition Legacy Apps in .NET

Visible Systems Corporation recently unveiled the next iteration of its tool, Visible Developer 3.3, designed to help companies build enterprise-wide applications using Visual Studio .NET 2003.

Visible Developer 3.3 works within the Visual Studio .NET IDE to automatically generate more than 90 percent of the business logic and database access code in Visual Basic .NET and Visual C#® .NET directly from the application model. You can extract database schema information (tables, fields, data types, keys, and so on) from your existing databases through an interface to Visible Analyst or using an XML document interface.

Databases including Microsoft Access, SQL Server, Oracle, and others are supported. Software design and implementation details are contained in code patterns. You design the business application and then select the code pattern. Visible Developer generates code matching the requirements expressed in your model. Patterns exist for C#, Visual Basic .NET, Visual Basic 6.0 three-tier business objects, Visual Basic forms, SQL Server and Oracle stored procedures, ASP, and ASP.NET.

http://www.visible.com

Database Designer

Datanamic has released DeZign for Databases 3.1, a database development tool using an entity relationship diagram. DeZign for Databases visually supports the layout of entities and relationships and automatically generates most SQL and desktop databases.

Some of the new features in version 3.1 include the support of views, which can be created using a query builder, adding options to an index, and exporting SQL Server comments on tables and columns to the SQL script. Version 3.1 of the application now uses the new DBISAM components so that DBISAM V4 tables can be created.

DeZign for Databases runs on Win32® platforms including Windows NT® (versions 3.51 and 4.0) and Windows 95, Windows 98, Windows Me, Windows 2000, and Windows XP. DeZign for Databases supports SQL Server, Access, Oracle, MySQL, Interbase, Sybase, DB2, Paradox, FoxPro, Clipper, PostgreSQL, Firebird, DBISAM, Advantage DB, and Pervasive SQL.

http://www.datanamic.com/dezign

The Bookshelf

Addison-Wesley has published Extending MFC Applications with the .NET Framework by Tom Archer and Nishant Sivakumar, aimed at experienced MFC programmers. This book teaches traditional Windows-based developers how to incorporate .NET functionality into their applications. Instead of publishing another .NET book that attempts to move MFC developers to .NET, this book focuses on showing how to combine MFC with .NET to improve productivity. Highlights include practical examples that show how to incorporate .NET functionality into legacy applications as well as help in preparing developers for "Longhorn," the code name for the next major release of Windows. An accompanying CD-ROM includes the code from all the samples in the book.

http://www.aw-bc.com

InstallShield Software Corporation recently released Practical Windows Installer Solutions for Building InstallShield Setup Applications, providing expert techniques for creating customized software installations. Written by Windows Installer expert Bob Baker, the book provides a real-world guide to help developers overcome obstacles that can crop up during installation authoring and application deployment.

Topics include delving into how the Windows Installer service manipulates Windows security to provide elevated privileges to packages targeting locked-down desktops, and how to overcome the access issues involved in deploying software to locked-down environments. You'll also learn how to use Windows Installer custom actions to extend installations, how to create installations that are easy to update, and how to upgrade applications using full installation packages. There's a section detailing how to generate and apply patches, and information regarding the installation of Plug and Play device drivers.

http://www.installshield.com/ispress

Send your New Stuff to Theresa at newstuff@microsoft.com.

Theresa W. Carey is a freelance writer who lives in California. Her byline has appeared in Barron's and PC World.


Web Q&amp;A: JScript Leaks, Getting the XmlDataDocument, and ASPX Includes

Web Q&A
JScript Leaks, Getting the XmlDataDocument, and ASPX Includes
Edited by Nancy Michell

Q I have built a Web application that has a rich GUI and functionality to match. The application building blocks are XML, XSLT, and JScript®. I have had huge problems with the application consuming lots of memory, but I have no idea why this occurs. I use closures to implement event handlers and am wondering if this is contributing to leaks.

A The use of closures as event handlers is a very common cause of memory leaks. Closures allow inner functions to always have access to the variables and parameters of its outer function.

There's a blog entry by Eric Lippert about closures and memory leaks at What are closures?. Note, however, that this article contains an error that has not been corrected as of press time. It is not true that the circular reference is passivated when Microsoft® Internet Explorer navigates the page. That was briefly the case when this issue first cropped up a few years ago, but the Internet Explorer team discovered that this passivation broke too many existing pages, so they turned it off.

As far as building a Web app that has rich GUI functionality, as you stated, that's a bad idea in the first place— it leads to bloated software that leaks memory, for one thing (see Thin To My Chagrin).

The prudent use of closure semantics depends on whether the function is dynamically generated or not. If the function is dynamically generated, say via the new operator, you should find a way not to dynamically generate it. The expression new Function is just another way of saying eval, and eval is bad for performance. Eval starts a compiler. If the function needs to be dynamically generated, and it does not use closure semantics, then move the generator to a helper function that does not take arguments that form a circular reference. That way the generated function will be closed over the scope of something that does not refer back to the Internet Explorer object model. If the function is static and does not need closure semantics, then just don't nest it. If the function is static and does need closure semantics—well, rearchitect your application until you're out of this situation.

With all that said, remember that all of this is basically "enabling" advice—and not a good idea. Thin clients are thin for a reason—they are designed for running thin programs, not rich programs. In the long run it would be better to get away from this development paradigm altogether, and into the rich-client world of the .NET common language runtime (CLR).

Q How can I determine the XmlDataDocument associated with a DataSet? If I create an XmlDataDocument from the DataSet in the following line of code

XmlDataDocument xmlData = new XmlDataDocument(dataset);
it is easy to get the DataSet corresponding to the XmlDataDocument that is shown here:
DataSet dataset = xmlData.DataSet;

However, once the association has been made between a DataSet and XmlDataDocument, how can I determine the XmlDataDocument from the DataSet? This is important because if I try to create a second XmlDataDocument from the Dataset, an exception is thrown. I am relying on the DataSet. In many processes, I operate on the data in XML format. When the DataSet is passed between methods I now need implicit knowledge about the calling method, whether an XmlDataDocument has been created, and what its reference is.

A There is no simple way out. Unfortunately, the DataSet does not provide any public API to check if an XmlDataDocument is attached or provide a reference to one. However, as a workaround, you can choose to do one of the following:

  1. Always instantiate XmlDataDocument in a try block and catch the argument exception that gets thrown if the DataSet is already attached to an XmlDataDocument.
  2. Programmatically keep track of whether the DataSet instance has an associated XmlDataDocument.

Then you can use DataSet.ExtendedProperty in order to create a user-defined property for keeping track of an attached/unattached XmlDataDocument.

For instance, register an attach operation using this code:

DataSet.ExtendedProperty.Add("XmlDataDocumentAttached", "1")
This extended property can be checked prior to instantiating the XmlDataDocument instance.

If you can ensure that the DataSet would not be serialized using Web services or remoting, then you can even register the attached XmlDataDocument instance as the property value (serialization would break if an object is attached to DataSet's ExtendedProperty). Otherwise, an alternative would be to construct an XmlDocument from the return of GetXml, but then you should make all changes through the DataSet.

Q I have a two-year-old Web app that has a standard menu header used by all pages. It's invoked in each ASP page, like so:

#include file="../Include/UDBMenuAccess.asp"
I'm now adding a new .aspx page to the site, and I want to include this menu. However, my new page either won't recognize the include, or produces errors later in the code. Isn't it possible to include an .asp page in an .aspx file? What's the workaround?

A Convert the old include file to an .ascx file and then include that file in your .aspx page. Includes haven't gone away; they are simply called custom controls now and they are quite powerful. Bear in mind that these controls must be written in the same language as your project. If you choose to go the C# route and have some old Visual Basic® includes, they will need to be converted to C#. See an example, the copy control, in Figure 1.

Figure 1 A Custom Control (Close Figure 1)
<%@ Control Language="c#" AutoEventWireup="false" Codebehind="cr.ascx.cs"
Inherits="thisSite.includes.cr_B"
TargetSchema="http://schemas.microsoft.com/intellisense/ie5"%>
<img height="16" src="http://thisSite/images/bar2003.gif" width="748"
border="0"><br>
<table class="betaBorder" width="748" bgColor="#eef1f4">
  <tr>
    <td class="CenterSmall">© thisSite 2003 -
    <a tabIndex="40" href="http:// thisSite /default.aspx"> Home</a> -
    <a tabIndex="41" href="http://thisSite/info/contacts.aspx"
    target="_parent"> Contacts</a> -
    <a tabIndex="42"
    href="http://thisSite/info/disclaimer.aspx" target="_blank">
    Disclaimer</a>
    </td>
  </tr>
</table>

Now in the .aspx you should include it in the following manner. At the top of the .aspx page declare it using:

<%@ Register TagPrefix="thisSite" TagName="copyright"
 src="includes/cr.ascx" %>
Then when you want to place your control in your page, use the following element to call it:
<thisSite:copyright id="copyright1" runat="server">
</thisSite:copyright>

Q I have a piece of code that looks like this:

SqlConnection sqlConnection;
CloseConnection()
{
    sqlConnection.Close();
    sqlConnection.Dispose();
    sqlConnection = null;
}
Does calling Dispose on the SqlConnection object remove it from the connection pool? Also, must I call both Dispose and Close? I am using the .NET Framework 1.1.

I read an article which said that calling Dispose in the .NET Framework 1.0 removes the connection from the pool, whereas the .NET Framework 1.1 does not. Is this true?

A Dispose does not remove the connection from the pool, it puts it back into the pool if that has not already been done. For the .NET Framework versions 1.0 and 1.1, you can basically consider Close and Dispose to be equivalent.

In response to your other question, Dispose is the same for .NET Framework versions 1.0 and 1.1. Always Close or Dispose a SqlConnection and you will avoid many SqlConnection finalizer/garbage collection-based problems.

If you want to be even smarter, you can also use the using keyword, as shown in the following code:

using ( SqlConnection conn = new SqlConnection(CONNECT_STRING) )
{
    try
    {
        conn.Open();
        // Use conn in here...
    }
    catch ( SqlException sqlEX )
    {
        // Log errors here...
    }
} // Once the scope of this brace is complete, Dispose will be
  // called on SqlConnection object.

For more programming best practices for SqlConnection, see Data Access Application Block for .NET.

Q Is there a built-in way of extracting XML schema namespace URI information from an XSD? Or do I need to parse through a schema file manually? I want to ask a user to select a whole bunch of XSD files and load the schema namespace URIs of all the files automatically without asking that user for further input.

A To achieve what you're asking, you have at least three choices. First, you can load the schema into a DOM and use XPath to select the /xs:schema/@targetNamespace node (where xs maps to http://www.w3.org/2001/XMLSchema). Second, you can load the schema into a schema object model (SOM) by calling System.Xml.Schema.XmlSchema.Read and read the TargetNamespace property. Your final choice is to use an XmlTextReader to parse the first element in the schema, check that it is {http://www.w3.org/2001/XMLSchema}schema, and read the targetNamespace property.

All of these options are pretty simple to implement. However, the last one is by far the most efficient in terms of both space and time, since the first and second options require performing the third option anyway, plus a lot more overhead.

Q How does System.Data.SqlClient handle semicolon-delimited statements in a batch? Such batches appear to work just fine; however, I haven't seen any documentation on the semicolon in the .NET Framework documents.

Is a normally formatted code block like this

DECLARE @val int
SET @val=10

UPDATE dbo.Foo
SET val=@val
IF (@@ERROR <> 0) GOTO EXIT_SCRIPT

SET @val=20
UPDATE b
SET b.val=@val
FROM dbo.Foo AS f INNER JOIN dbo.Bar AS b ON f.id = b.id

IF (@@ERROR <> 0) GOTO EXIT_SCRIPT

EXIT_SCRIPT:
processed differently than this?
DECLARE @val int;SET @val=10;UPDATE dbo.Foo SET val=@val;IF (@@ERROR <> 0)
GOTO EXIT_SCRIPT;SET @val=20;UPDATE b SET b.val=@val FROM dbo.Foo AS f
INNER JOIN dbo.Bar AS b ON f.id = b.id;IF (@@ERROR <> 0)
GOTO EXIT_SCRIPT;EXIT_SCRIPT:;
Does it make any difference what kind of statements are called in such a batch (transaction statements, stored procedures)?

A SqlClient passes all the statements directly to SQL Server™ without making any modifications on the client side. SQL Server accepts semicolons as valid delimiters, so it just works. As far as SQL Server is concerned, there is no difference between the two batches.

Q I have a question regarding ADO.NET relationships and deleted rows. When I do something like this

parentRow.GetChildRows( "relationship name" ) ;
I get no DataRows returned where the RowState is Deleted.

However, the problem is that I really need that information. I need to be able to get all the related rows that have been added, modified, or deleted. GetChildRows returns only the added and modified rows, but doesn't return any of the deleted ones.

I tried passing the optional DataRowVersion parameter

parentRow.GetChildRows("relationship name", DataRowVersion.Original);
to see if that would return the deleted rows, but it doesn't pass the deleted rows either. How can I accomplish this?

A The second parameter on GetChildRows controls the version of the data you'll receive (current, original, or proposed). It is not a row state (added, modified, deleted, unchanged) filter.

To access the child rows that contain pending changes, you could use DataTable.Select, then create a filter that looks for the desired rows (such as FKColumn = Value) and supply the desired row states to search (Added, ModifiedCurrent, Deleted). There's some sample code shown in Figure 2.

If you don't mind using DataViews, you could also use DataRowView.CreateChildView and set the RowStateFilter accordingly, as shown in the following code:

DataView vueCustomers = ds.Tables["Customers"].DefaultView;
    vueCustomers.Sort = "CustomerID";
DataView vuePendingOrders =
    vueCustomers[vueCustomers.Find("ALFKI")].CreateChildView
    ("Customers_Orders");
vuePendingOrders.RowStateFilter = DesiredRowStates;
Console.WriteLine("ALFKI has {0} pending modified orders",
    vuePendingOrders.Count);

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Jeff Abrams, Kawarjit Bedi, Jeffrey Brendecke, Lale Divringi, Blaine Dockter, Yves Dolce, Tim Ewald, Michael Hestness, Eric Lippert, Matt Neerincx, Carl Nolan, Thomas Rahm, David Sceppa, Larry Sun, Vinayak Tadas


Data Points: Creating Audit Tables, Invoking COM Objects, and More

Data Points
Creating Audit Tables, Invoking COM Objects, and More
John Papa

Code download available at: DataPoints0404.exe (111KB)

Dealing with error handling between T-SQL and a calling application, evaluating when a field's value has changed, and creating auditing tables in SQL Server™ are all common issues that developers must tackle. This month's column will address these and other development scenarios based on some of my most frequently received T-SQL questions. Sample code is included in the download file for many of the following questions including code to send an e-mail from a stored procedure by invoking a COM object.

Q I have a table for which I want to track both the date and time each record was created and the last date and time that any field in the row was updated. I often need to update several rows at once, so my solution has to work for one or more rows. How can I accomplish this using features in SQL Server?

A This issue can be resolved using a handful of features in SQL Server including a DEFAULT constraint, the built-in GETDATE function, and an AFTER UPDATE trigger. To set a value indicating when a record was created, a DEFAULT constraint can be applied to the appropriate column using the default value from the GETDATE function. A column that has a DEFAULT constraint does not need to be included in an INSERT statement since it can use the default value. This means that if you add a column to represent the date and time a record was created, you can do so without modifying any existing INSERT statements written against the table (as long as those statements include a column list). Using a DEFAULT constraint to assign the current date and time using the GETDATE function works whether 1 row or 100 rows are inserted. So to accomplish the first part of this task a column named dtCreated could be added to an existing table named tblCustomer. The code in Figure 1 shows the SQL Data Definition Language (DDL) used to create tblCustomer, its primary key constraint, and the new column called dtCreated. Theoretically, however, someone could still pass an explicit value for the dtCreated column. This would mean that the default value would be overwritten and the creation data might be incorrect. If you want to force this time stamp to always be the current value, an insert trigger can be used that always sets the value to GETDATE even if a specific value was in the insert list.

Figure 1 Creating tblCustomer and dtCreated (Close Figure 1)
CREATE TABLE tblCustomer
(
    nCustomerID INT IDENTITY,
    sCustomerName VARCHAR(50),
    sAddress VARCHAR(50),
    sCity VARCHAR(50),
    sState CHAR(2),
    sZip CHAR(5)
)
GO

ALTER TABLE tblCustomer ADD PRIMARY KEY (nCustomerID)
GO

ALTER TABLE tblCustomer ADD dtCreated DATETIME DEFAULT GETDATE()
GO

The second part of this task cannot be addressed with a DEFAULT constraint, but triggers will do the trick. First, another column should be added to the tblCustomer table to store the last date and time a record was modified. The following code will add the column named dtLastUpdated to the table:

ALTER TABLE tblCustomer
    ADD dtLastUpdated DATETIME NULL
GO

The key here is to make sure that for every row that is updated via a SQL UPDATE statement, its corresponding dtLastUpdated column's value is set to the current date and time. An AFTER UPDATE trigger could be created on the tblCustomer table to trap the date and time the modifications are made to the table. Then, a cursor could be created to traverse the rows that were modified. These rows are stored in the special table called INSERTED. (For more information on the inserted and deleted tables, refer to my December 2003 column.) The cursor could be looped through and each record could be matched to the corresponding record in the tblCustomer table and then updated. While this technique will work, it does cause an UPDATE statement to be executed for every row that was originally updated. Thus, if there were 100 rows in the tblCustomer table modified by an UPDATE statement, then the cursor would loop through all 100 rows that were modified and issue 100 individual UPDATE statements to set the value in the dtLastUpdated column. Not a good thing to do.

There is a better and easier way to tackle this problem still using an AFTER UPDATE trigger. Instead of using a cursor, which can be inefficient, you can use a variation of an UPDATE statement. The following UPDATE statement will set the dtLastUpdated column's value for each modified row. It does this by joining the base table to the INSERTED table, causing a single UPDATE statement to execute inside of the trigger:

CREATE TRIGGER trUpd_tblCustomer ON tblCustomer AFTER UPDATE
AS
    UPDATE  tblCustomer SET dtLastUpdated = GETDATE()
    FROM    inserted i INNER JOIN tblCustomer c ON i.nCustomerID =
            c.nCustomerID
GO

You might wonder if, by updating the tblCustomer table inside of an AFTER UPDATE trigger on the same table, it will call itself recursively. Triggers don't do that unless the RECURSIVE_TRIGGERS option is turned on. When this setting is off, direct recursion (for example, when tblCustomer updates tblCustomer in its own AFTER UPDATE trigger) will not occur. You can use the sample code in Figure 2 to test out this solution.

Figure 2 Some Inserts (Close Figure 2)
INSERT INTO tblCustomer
    (sCustomerName, sAddress, sCity, sState, sZip)
    VALUES ( 'Company X', '1 Lois Lane', 'Metropolis', 'NY', '55555')
INSERT INTO tblCustomer
    (sCustomerName, sAddress, sCity, sState, sZip)
    VALUES ( 'Company Y', '2 Somewhere Drive', 'Anytown', 'NY', '44444')
INSERT INTO tblCustomer
    (sCustomerName, sAddress, sCity, sState, sZip)
    VALUES ( 'Company Z', '3 Nowhere Street', 'Who Knows', 'NC', '33333')

UPDATE    tblCustomer SET sState = 'FL' WHERE sState = 'NY'

SELECT * FROM tblCustomer

Q When using the UPDATE function in a trigger to evaluate if a column's value has changed, how does the function compare values to determine if the field was indeed updated? Does the UPDATE function return true if the value updated merely exists in the originating UPDATE statement or does it truly check to see if the value has changed? Is there any other way I can detect if a column's value has actually changed?

A The UPDATE function does not actually check to see if a column value is different. Specifically, the UPDATE function can only be used inside a trigger. It accepts a single parameter that represents a column in a table and it returns a bit value to indicate whether the column's value has been involved in an update (1 for modified, 0 otherwise). However, the UPDATE function evaluates to 1 even if the column was updated but its new value is not different. For example, the following UPDATE statement issued against Northwind sets the FirstName field from Nancy to Nancy:

UPDATE employees SET FirstName = 'Nancy' WHERE EmployeeID = 1

While this statement effectively does not change the net result of what is stored in the row, it does change the value from Nancy to Nancy. Assuming there is an AFTER UPDATE trigger on this table, it would then execute:

CREATE TRIGGER trU_Employees_UpdateTest  ON Employees AFTER UPDATE
AS
IF UPDATE(FirstName)
    SELECT 'Update Evaluated to True'
GO

This AFTER UPDATE trigger would fire and the UPDATE(FirstName) expression would evaluate to 1, even though the name has the same value before and after. You can test this out easily enough by running the previous UPDATE statement. It should return the string value of Update Evaluated to True in this case.

But what if you want to compare the old and new values? Well, in that case the UPDATE function alone just won't cut it. However, you could compare the old and new values by using the special INSERTED table and DELETED tables inside the trigger. The trigger I just showed could be modified to accomplish this fairly easily. I would still use the UPDATE function to check if the FirstName column was in the UPDATE statement or not. It would also add some logic to compare the values in the inserted and deleted tables, as follows:

ALTER TRIGGER trU_Employees_UpdateTest  ON Employees AFTER UPDATE
AS
IF UPDATE(FirstName)
    IF EXISTS(
    SELECT  i.EmployeeID
    FROM    inserted i INNER JOIN deleted d ON i.EmployeeID =
            d.EmployeeID
    WHERE   i.FirstName <> d.FirstName
    )
        SELECT 'Update Evaluated to True'
GO

This is less efficient than just using the UPDATE function, so you should think about whether or not the extra precision is necessary. This code makes sure that at least one of the rows that was updated had its FirstName column updated to a new value. Note that in a production environment, you would not want to put SELECT statements in a trigger. Rather, you would only want to perform "silent" actions, such as updating another table.

Q I have several ActiveX® components running on my SQL Server database server. These components are very extensive and contain many business rules that some of my stored procedures need to invoke. I know that you can call a stored procedure from an ActiveX DLL, but is there any way I can call an ActiveX component from my stored procedure?

A There are a few system stored procedures in SQL Server that give developers the ability to instantiate an ActiveX component/COM object, call its methods and properties, destroy it, and even retrieve information about any exceptions that might occur. Calling a COM object from T-SQL starts with sp_OACreate. This stored procedure creates an instance of the object specified by the progid parameter and returns an integer value via the objecttoken output parameter. The objecttoken returns a handle to the COM object that can be used in the remaining sp_OA stored procedures to reference the COM object. The following code sample shows how to instantiate a COM object with sp_OACreate:

EXECUTE @nOLEResult = sp_OACreate 'CDO.Message', @nCDO OUT

The sp_OACreate stored procedure also returns an integer value, which represents whether an error occurred within the COM object. A nonzero value indicates that an error has occurred. The error information can be retrieved via the sp_OAGetErrorInfo stored procedure after each call. The following code sample executes the sp_OAGetErrorInfo stored procedure, passing to it the reference to the COM object (@nCDO) and retrieving the error's source and description information via output parameters:

EXECUTE sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
SET @sMsg = 'Error creating object ' + @sProgID + '. Error # = ' +
    CAST(@nOLEResult AS VARCHAR(20)) +
    '. Description = ' + @sDescription
RAISERROR (@sMsg, 16, 1)
RETURN
This information can be used to format an error message that could be bubbled up to an application or to a user. Error handling is crucial in most applications and while T-SQL does provide for error-handling techniques, they are less sophisticated than what more robust languages such as Visual Basic®, Visual Basic .NET, or C# offer. It is important to provide for and thoroughly test the error conditions inside of a T-SQL batch. The previous code sample demonstrates how to trap the errors from the COM object and raise them using the RAISERROR statement.

As a test, I intentionally caused an error by trying to instantiate a nonexistent COM object called CDO.Message123. The preceding error handling T-SQL logic returned the following error:

Server: Msg 50000, Level 16, State 1, Procedure prSendMail, Line 25
Error creating object CDO.Message123. Error # = -2147221005.
Description =  Invalid class string

Just as their names suggest, the sp_OASetProperty and sp_OAGetProperty stored procedures provide the means to retrieve and set public properties of a COM object. Further, the sp_OAMethod stored procedure provides a way to invoke a method of a COM object. Once you're finished using an object, you call sp_OADestroy to release it. Objects that aren't explicitly freed are automatically released at the end of the T-SQL batch. The sample code in Figure 3 demonstrates these techniques through a sample stored procedure called prSendMail. This stored procedure invokes the CDO.Message object and sets its To, From, Subject, and TextBody properties before ultimately sending an e-mail message using the CDO component.

Figure 3 Invoking COM Objects and Sending E-mail Using T-SQL (Close Figure 3)
ALTER PROCEDURE prSendMail
    @sTo VARCHAR(500),
    @sBody VARCHAR(8000),
    @sSubject VARCHAR(500),
    @sFrom VARCHAR(500)
AS

    DECLARE @nCDO INT,
        @nOLEResult INT,
        @nOutput INT,
        @sSource VARCHAR(255),
        @sDescription VARCHAR(255),
        @sProgID VARCHAR(50),
        @sMsg VARCHAR(250)

    SET @sProgID = 'CDO.Message'

    EXECUTE @nOLEResult = sp_OACreate @sProgID, @nCDO OUT
    IF @nOLEResult <> 0
    BEGIN
        EXEC sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
        SET @sMsg = 'Error creating object ' + @sProgID +
            '. Error # = ' + CAST(@nOLEResult AS VARCHAR(20)) +
            '. Description = ' + @sDescription
        RAISERROR (@sMsg, 16, 1)
        RETURN
    END

    EXECUTE @nOLEResult = sp_OASetProperty @nCDO, 'To', @sTo
    IF @nOLEResult <> 0
    BEGIN
        EXEC sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
        SET @sMsg = 'Error executing CDO.To property. Error # = ' +
            CAST(@nOLEResult AS VARCHAR(20)) +
            '. Description = ' + @sDescription
        RAISERROR (@sMsg, 16, 1)
        RETURN
    END

    EXECUTE @nOLEResult = sp_OASetProperty @nCDO, 'From', @sFrom
    IF @nOLEResult <> 0
    BEGIN
        EXEC sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
        SET @sMsg = 'Error executing CDO.From property. Error # = ' +
            CAST(@nOLEResult AS VARCHAR(20)) +
            '. Description = ' + @sDescription
        RAISERROR (@sMsg, 16, 1)
        RETURN
    END

    EXECUTE @nOLEResult = sp_OASetProperty @nCDO, 'Subject', @sSubject
    IF @nOLEResult <> 0
    BEGIN
        EXEC sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
        SET @sMsg = 'Error executing CDO.Subject property. Error # = ' +
            CAST(@nOLEResult AS VARCHAR(20)) +
            '. Description = ' + @sDescription
        RAISERROR (@sMsg, 16, 1)
        RETURN
    END

    EXECUTE @nOLEResult = sp_OASetProperty @nCDO, 'TextBody', @sBody
    IF @nOLEResult <> 0
    BEGIN
        EXEC sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
        SET @sMsg = 'Error executing CDO.TextBody property. Error # = ' +
            CAST(@nOLEResult AS VARCHAR(20)) +
            '. Description = ' + @sDescription
        RAISERROR (@sMsg, 16, 1)
        RETURN
    END

    —Call Send method of the object
    EXECUTE @nOLEResult = sp_OAMethod @nCDO, 'Send', Null
    IF @nOLEResult <> 0
    BEGIN
        EXEC sp_OAGetErrorInfo @nCDO, @sSource OUT, @sDescription OUT
        SET @sMsg = 'Error executing CDO.Send method . Error # = ' +
            CAST(@nOLEResult AS VARCHAR(20)) +
            '. Description = ' + @sDescription
        RAISERROR (@sMsg, 16, 1)
        RETURN
    END

    —Destroy CDO
    EXECUTE @nOLEResult = sp_OADestroy @nCDO

    RETURN @nOLEResult

GO

As you can see, these system stored procedures are pretty straightforward to use. But calling COM objects from T-SQL is not reserved merely for third-party or Microsoft components such as CDOSYS, ADODB, and SQLDMO. You can also use these techniques to instantiate valid custom COM components. However, while the sp-OA* procedures are easy to use, you need to be very careful with them. Since sp-OA* procedures execute within the SQL Server memory space, improper usage could result in memory problems in your database. There are several Knowledge Base articles on the subject that go into more detail.

Q I have created an audit table to track changes to a primary table. After every change to the primary table I want to insert a row to the audit table to track the change. I could accomplish this using triggers, stored procedures, or even a business application written in Visual Basic. What are the pros and cons of these methods and which do you recommend?

A There are several techniques for creating tables that exactly or closely mirror a primary table including those that use audit tables, shadow tables, archive tables, and history tables. Basically, in these situations the goal is that for every row inserted into a primary table, a corresponding row is inserted into a secondary audit table. You'll create these types of tables when you want to audit changes to the primary table, keep a permanent record of the original state of the primary table, or create a mirror table that can be used for special calculations or processing, to name a few scenarios.

Since this process entails taking a row of data and copying it in some form to another table, this is a task that belongs in the database tier (if you do this in the application layer, then inserts/updates made through other apps, batch processes, or ad hoc queries will not move rows to the audit table). One way to accomplish this is to modify the stored procedure that inserts into, updates, or deletes from the primary table to also insert or update a record in the secondary table. One consequence of using a single stored procedure to do this is that an explicit transaction should be created to wrap both INSERT statements inside of an atomic transaction. If one of the INSERT statements fails, it could raise an error and roll back the transaction. Another consequence of this solution is that only inserts made through this stored procedure will be made to both the primary and secondary tables. So if any INSERT statements are issued directly against the primary table or even through another stored procedure, they will not be copied to the secondary table. Using this type of solution leaves open the possibility that the primary and secondary tables could get out of sync.

Another way to tackle this situation is to use triggers. AFTER INSERT triggers are commonly used when actions (such as populating shadow tables) must be taken following data modifications on a primary table. For this example, I will mirror all rows inserted into the Region table of the Northwind database to a secondary table called Region_Audit:

CREATE TABLE Region_Audit
(
    RegionID INT NOT NULL,
    RegionDescription NCHAR(50) NULL
)

The following trigger will insert a copy of the new region row into the audit/shadow Region_Audit table:

CREATE TRIGGER trRegion_Audit_Insert ON [Region] FOR INSERT
AS
INSERT INTO Region_Audit (RegionID, RegionDescription)
    SELECT r.RegionID, r.RegionDescription
    FROM Region r INNER JOIN inserted i ON i.RegionID = r.RegionID
GO

Using a trigger in this type of situation covers the bases more thoroughly than the first technique. The trigger makes sure that no matter how the data is inserted into the primary table, it will also be inserted into the audit table.

Q I have a trigger that must check for an error condition and report it back to whatever application caused the trigger to fire in the first place. How can a trigger raise an error using the RAISERROR statement so that it can be accessed from a Visual Basic-based program? I want the error message that is raised in the trigger to be displayed in the user interface.

A To answer this question, let me set some basic assumptions up front for this scenario. First, let us assume that there is a program using Visual Basic 6.0 that issues an UPDATE statement against the Northwind database table called [Order Details]. The [Order Details] table has an AFTER UPDATE trigger that contains business rules logic that prevents a line item's quantity from being modified to be larger than what is currently in inventory. When an UPDATE statement is issued from the Visual Basic-based application that violates this rule, the trigger will raise an error that needs to bubble up to the application.

RAISERROR ('Inventory is out of stock', 16, 1)

In this case, the T-SQL RAISERROR statement accepts a specific error message, severity, and state. For more details on the syntax and parameters of RAISERROR, refer to the MSDN® Library (RAISERROR).

When this error is raised inside of the trigger, the calling Visual Basic-based application should have an error handling routine that traps the error. In Visual Basic 6.0, the code that could trap the error and strip the error's description from it could look something like the following:

MyErrHandler:
    Dim sMsg As String
    Dim lErr As Long
    Dim sSrc As String

    sMsg = Err.Description
    lErr = Err.Number
    sSrc = Err.Source

The error could then be raised again to eventually be thrown back to the user or it could be formatted into a friendlier message that can be presented to the user. The RAISERROR statement in the trigger throws the error back to the Visual Basic-based application which uses standard error-handling code to trap it and then deal with it appropriately. Nothing special needs to be done here to tell the error-handling routine in Visual Basic that this error originated inside of a trigger. In the Microsoft® .NET Framework, error handling is performed slightly differently than in Visual Basic 6.0 since .NET makes the more robust try-catch-finally statement block available. Some sample code to handle an error in C# is shown in Figure 4.

Figure 4 Error Handling in C# (Close Figure 4)
try
{
    SomeMethodThatCallsMyStoredProc();
}
catch(SqlException sqlEx)
{
    lblMyErrorDescLabel.Text = sqlEx.Message;
}
catch(Exception ex)
{
    lblMyErrorDescLabel.Text = ex.Message;
}

Notice how the multiple catch blocks can catch different types of exceptions. Since code has been written to look for the more specific SqlException, the exception from the trigger would be handled in its catch block. Any other exception type would be caught by the generic Exception catch block.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Cutting Edge: Image Generation Service for ASP.NET 1.1

Cutting Edge
Image Generation Service for ASP.NET 1.1
Dino Esposito

Code download available at: CuttingEdge0404.exe (134KB)
Contents
What's IGS Anyway?
Comparing Image and DynamicImage
Comparing Pages and HTTP Handlers
Designing a ASP.NET 1.x Image Service
The CachedImageService.axd Handler
The DynamicImage Control In Action
Adding Support for Image Types
Conclusion

Sometimes you need more than just static images on your Web site—you may want to create images dynamically. Thanks to GDI+, ASP.NET 1.x makes it easy to generate images programmatically. The ASP.NET infrastructure can output images read from a source file, and can also retouch them in memory before outputting them. While this gives programmers enough power to build effective one-off solutions, it requires them to reinvent the wheel every time a new system is needed.

Aware of this limitation, the ASP.NET team designed more powerful image support into ASP.NET 2.0 and called it the Image Generation Service (IGS). With IGS, working with images requires much less code than was required in ASP.NET 1.x and lets you reference dynamically generated images in your pages more easily. You can read image data stored in a local or remote file, in database fields, or in memory. Whatever the source of the image, a new server control in IGS lets you link the image to the page.

In this column, I'll build a version of IGS targeted at ASP.NET 1.x and keep it as close as possible to the ASP.NET 2.0 implementation. This way you get tools to improve the manipulation of images in your current applications and can get ready for the new ASP.NET 2.0 features. Note that ASP.NET 2.0 is still in development and some of these APIs might change before release.

What's IGS Anyway?

No matter what kind of abstraction layer you build, to display images over the Web you still need an <img> tag pointing to a public URL. IGS is composed of two main elements—a server control and an HTTP handler. The server control is an enhanced version of the Image control currently available in ASP.NET 1.x. It's a server-side wrapper around the HTML <img> element. The HTTP handler is a helper component that provides the data behind the URL of the <img> tag that is sent out to the browser. The HTTP handler represents a common piece of server code used to generate and manipulate the bytes of the image to display.

Before I write my own version of the IGS for ASP.NET 1.x, let's briefly review the ASP.NET 2.0 DynamicImage control. In ASP.NET 2.0, the server control is named DynamicImage and, at least at first glance, looks like a close relative of the version 1.x <asp:Image> control. But the DynamicImage control has more features. Run a page that contains the following code:

<asp:DynamicImage runat="server" id="MyFileImage"
     ImageFile="images/tools.jpg" />
The result of running the code is that the following markup is generated for displaying an image in a browser:
<img id="MyFileImage"
     src="CachedImageService.axd?data={auto-generated guid}"
     alt=""
     style="border-width:0px;" />

Notice that the Src attribute is not set to the specified image file but points to an auto-generated URL. When instantiated, the control loads the image from the specified resource and stores it in the ASP.NET cache. The key used to store and retrieve the image is a GUID managed by ASP.NET.

The CachedImageService.axd URL corresponds to the public name of a built-in HTTP handler. The handler retrieves the image from the ASP.NET cache and serializes its bytes to the browser using the proper content type. The data parameter in the handler's query string indicates the key used to select the image from the ASP.NET cache. The DynamicImage control also features a read-only ImageUrl property that returns the exact URL of the image being generated.

In ASP.NET 2.0, the CachedImageService.axd service provides an additional capability. It can adjust the size and format of the image based on the detected capabilities of the browser. If the browser cannot render that type of image, the service will convert the image to a type supported on that browser, thanks to the ASP.NET 2.0 adaptive rendering engine.

The DynamicImage class inherits from the DynamicImageBase class. Figure 1 lists the properties of the DynamicImage control. As mentioned, ImageUrl is a read-only property and as such can't be used to set the URL of an image to be displayed. This may be a source of confusion because ImageUrl is just the name of the main property of the Image control in ASP.NET 1.x. (Note that dynamic images don't support width and height expressed as percentages; they can only be expressed in pixels.)

Figure 1 ASP.NET 2.0 DynamicImage Control Properties (Close Figure 1)

Property | Description
AutoSize | True by default. Indicates whether to modify the size of the image for browsers with limited graphics capabilities.
Height | Inherited from DynamicImageBase. Gets and sets the height of the image in pixels.
Image | Object of type System.Drawing.Image. Gets and sets the source of the dynamic image.
ImageBytes | Gets and sets a reference to a byte array that contains the dynamic image.
ImageFile | Gets and sets the name of the file that contains the image.
ImageGeneratorUrl | Gets and sets the URL of the HTTP handler image-generation service to use.
ImageType | Inherited from DynamicImageBase. Gets and sets the preferred image type for the dynamic image.
ImageUrl | Inherited from DynamicImageBase. Gets the URL used to generate the dynamic image. This property is read-only.
Parameters | Gets a collection of parameters to pass to the image-generation service. This property is important when an HTTP handler is used to generate the image.
PercentScreenCover | Gets and sets the percentage of the browser screen that the image will occupy. This property only applies to mobile devices and has no effect on desktop browsers.
ScaleBasedOnWidth | Gets and sets a value indicating how to scale dynamic images to make them fit into the specified width and height with respect to the image aspect ratio. If true, the image is scaled based on its width, otherwise by its height.
Width | Inherited from DynamicImageBase. Gets and sets the width of the image in pixels.

The DynamicImage control can acquire the image bytes from a variety of sources and render them. Four of the properties listed in Figure 1 are used to define usable image sources. They are: Image, ImageBytes, ImageFile, and ImageGeneratorUrl. Image gets and sets the image as an instance of a GDI+ image class. The property is of type System.Drawing.Image. This property is ideal for images dynamically created in memory using GDI+ capabilities. ImageBytes represents the image as an array of bytes and is suitable for images stored in databases. ImageFile gets and sets the URL to the image, whereas ImageGeneratorUrl returns the URL of a new image-specific HTTP handler having the .asix extension.

Figure 2 The Control's Architecture
Figure 2 The Control's Architecture

Whatever the format of the image, it is internally normalized to an instance of the System.Drawing.Image object, assigned a random generated key, and cached in the ASP.NET Cache object. The diagram in Figure 2 shows an approximation of the internal architecture of the control.

Comparing Image and DynamicImage

Overall, in ASP.NET 2.0 you have two image server controls—Image and DynamicImage. Image is ideal for referencing a static image as a server-side file or URL. If you can reference an image using a file path, then use the Image control, which would produce the following familiar markup:

<img src="url" />

Referencing static images via a URL remains the fastest way to have an image downloaded to a client. No extra code is required and the browser and the Web server would handle it automatically without involving the ASP.NET runtime. IIS, for example, is optimized to serve static images such as JPG and GIF files on parallel threads without involving any ISAPI code.

In DynamicImage, the image is cached in memory rather than being read from disk or downloaded from a remote Web site on every access. However, consider that the vast majority of browsers are configured to cache images locally, which is better for performance than server-side caching anyway. The real added value of ImageFile becomes clear when you write applications designed to work with both desktop and mobile browsers. In this case the IGS can automatically adapt the image to the device, at least in the most common scenarios.

Comparing Pages and HTTP Handlers

Many developers still reference dynamic images using a URL that points to an .aspx page. Here's an example:

<img src="photo.aspx?id=9" />

If it weren't for the .aspx extension, you might think this code came from an old ASP application. In ASP.NET, however, this is a bad approach. The reason is simple: pages typically incur the most overhead of any HTTP handler you can use to serve a request. In ASP.NET, each HTTP request is served by a special component known simply as the HTTP handler. An HTTP handler is a class that implements the IHttpHandler interface—in particular, it implements the ProcessRequest method.

The Page class is an HTTP handler with a really complex and sophisticated ProcessRequest method. The method loops through the collection of child controls and instantiates them all. Next, it manages the viewstate and performs postback events. None of this is really needed to manage images. Therefore, the most efficient way to handle dynamic images in ASP.NET is to write a custom HTTP handler and optimize it to merely retrieve, serve, and possibly cache images.

In ASP.NET 1.x, you have to write such a handler yourself, and sometimes you'll find you're doing it repeatedly, one project after the next. In ASP.NET 2.0, on the other hand, you not only have the CachedImageService.axd handler available to you for immediate use, but you can afford to know practically nothing about how it works under the covers because the DynamicImage control shields you from all of the nitty-gritty details.

Designing a ASP.NET 1.x Image Service

This month's source code contains two projects—a DynamicImage custom server control and a sample application. The control assembly also includes an ImageService HTTP handler. The control mimics the functionality of the ASP.NET 2.0 DynamicImage class, while the handler presents some of the functionality of the CachedImageService.axd handler. The code in Figure 3 shows the structure of the DynamicImage control.

Figure 3 DynamicImage Control (Close Figure 3)
namespace MsdnMag
{
    public class DynamicImage : System.Web.UI.WebControls.Image
    {
        private const string IgsBaseUrl = "cachedimageservice.axd?data={0}";

        public DynamicImage()
        {
            m_imageBytes = null;
            m_image = null;

            this.PreRender += new EventHandler(DynamicImage_PreRender);
        }

        // PRIVATE members
        private System.Drawing.Image m_image;
        private byte[] m_imageBytes;

        // PROPERTY: StorageKey
        protected string StorageKey
        {
            get {return Convert.ToString(ViewState["StorageKey"]);}
            set {ViewState["StorageKey"] = value;}
        }

        // PROPERTY: ImageUrl (override)
        public override string ImageUrl
        {
            get {return GetImageUrl();}
            set {throw new NotSupportedException();}
        }

        // PROPERTY: ImageFile
        public string ImageFile
        {
            get {return Convert.ToString(ViewState["ImageFile"]);}
            set {ViewState["ImageFile"] = value;}
        }

        [Browsable(false)]
        public byte[] ImageBytes
        {
            get {return m_imageBytes;}
            set {m_imageBytes = value;}
        }

        [Browsable(false)]
        public System.Drawing.Image Image
        {
            get {return m_image;}
            set {m_image = value;}
        }

        // More code here...
    }
}

The DynamicImage control inherits Image and adds three new properties—ImageFile, Image, and ImageBytes. Deriving from the Image control, it also inherits the ImageUrl property that the base class exposes to let developers define the source of the resulting <img> tag. Just like its ASP.NET 2.0 counterpart, the ASP.NET 1.x DynamicImage control overrides the ImageUrl property to make it read-only. This is done by simply throwing an exception in the set accessor of the property. Not very elegant, but effective nonetheless:

public override string ImageUrl
{
   get {return GetImageUrl();}
   set {throw new NotSupportedException();}
}

In DynamicImage, ImageUrl plays a radically different role from the one it plays in Image. The actual URL of the image as returned by ImageUrl is determined by looking at the contents of the properties listed in Figure 4. Of course, only one of the source properties is used at a time. The control doesn't raise an error if more than one property is set, but only one is taken into account (the current ASP.NET 2.0 implementation of this control only allows one of these to be set and throws an exception otherwise). In my implementation, if set, ImageFile takes precedence over the other two. Otherwise, the control first checks ImageBytes and then Image.

Figure 4 (Close Figure 4)

Property | Type | Design-time Support | Description
Image | System.Drawing.Image | No | References an image held in memory. The image is represented with a managed object. The content of the image is loaded from a disk file, a stream, or created using GDI+ interfaces.
ImageBytes | Byte[] | No | References the image as an array of bytes. Typically, you use this member when you want to reference an image stored in a database field.
ImageFile | String | Yes | References the image through its file name. This is equivalent to using a static file with the <img> tag.

ImageFile fully replaces ImageUrl as implemented in the Image control, though if you want to reference an image through its physical or virtual file name, this is the property to use. ImageFile is also the only property you can set at design time. The Browsable attribute, in fact, prevents Image and ImageBytes from displaying in the Properties box, as shown here:

[Browsable(false)]
public byte[] ImageBytes {...}
[Browsable(false)]
public System.Drawing.Image Image {...}

When you set the ImageFile property declaratively or programmatically, the markup sent to the browser looks like this:

<img src="file" ... />
The DynamicImage control outputs an <img> tag even when you set the image through the ImageBytes or Image property. HTML provides just one way to call an image: through the <img> tag. That tag can only accept input expressed as a publicly accessible URL. This means that the bytes of the image read from a database table, or the in-memory image, must be referenced using a URL. This is where the image service HTTP handler comes in:
<img src="cachedimageservice.axd?data=..." />

The constructor of the DynamicImage control registers a handler for the PreRender event. If the event fires, that means at least one source property has been set. The control figures out the highest priority source for the image content and determines the URL to use. The source code that demonstrates this feature is the GetImageUrl method in Figure 5. GetImageUrl is invoked from within the get accessor of the ImageUrl property (see Figure 3).

Figure 5 GetImageURL Method (Close Figure 5)
void DynamicImage_PreRender(object sender, EventArgs e)
{
    if (ImageUrl.Length == 0)
        return;

    // Cache bytes or image
    if (ImageBytes != null)
        StoreImageBytes();
    else
        if (Image != null)
            StoreImage();

    return;
}

string GetImageUrl()
{
    string url = "";

    // Check ImageFile
    if (ImageFile.Length >0)
    {
        url = ImageFile;
    }
    else // Check ImageBytes and Image
    {
        if (ImageBytes != null || Image != null)
        {
            if (StorageKey.Length == 0)
            {
                Guid g = Guid.NewGuid();
                StorageKey = g.ToString();
            }
            return GetCachedImageUrl();
        }
    }

    return url;
}

string GetCachedImageUrl()
{
    return String.Format(IgsBaseUrl, StorageKey);
}

The PreRender event fires just before any ASP.NET 1.x control enters rendering mode. If the image is not referenced through a plain URL—the content is set through either ImageBytes or Image—the control ensures that any image data is available to the built-in HTTP handler. Each image referenced as an object or an array of bytes is given a unique key. This key is merely a GUID:

Guid g = Guid.NewGuid();
string storageKey = g.ToString();

The GUID is stored in the viewstate and preserved across page requests. This ensures that the same key always identifies the image displayed by a given control:

protected string StorageKey
{
   get {return Convert.ToString(ViewState["StorageKey"]);}
   set {ViewState["StorageKey"] = value;}
}
The GUID is also used as the key to cache the image object or bytes in the ASP.NET cache.

Except when the ImageFile property is set, the output of the DynamicImage control is retrieved from the cache and served to the browser by an HTTP handler. Figure 6 shows the code that stores the image content to the ASP.NET cache.

Figure 6 Store Image in Cache (Close Figure 6)
private void StoreImage()
{
    StoreData(m_image);
}

private void StoreImageBytes()
{
    StoreData(m_imageBytes);
}

private void StoreData(object data)
{
    if (Page.Cache[StorageKey] == null)
    {
        Page.Cache.Add(StorageKey,
                       data,
                       null,
                       Cache.NoAbsoluteExpiration,
                       TimeSpan.FromSeconds(60),
                       CacheItemPriority.High,
                       null);
    }
}

The cache entry is created only if it doesn't exist. The Add method, in fact, fails if the specified entry is found to already exist. The current storage key is used to uniquely identify the cache entry. The image is given a sliding expiration policy along with a high priority. These settings match the settings of the DynamicImage control in ASP.NET 2.0.

Sliding expiration indicates that the cached item will expire after a certain period of inactivity. If the cached item is not read or written for the specified duration, it will be removed. If the item is accessed before its natural expiration time, then the sliding period is automatically renewed.In this example, I've defined a one-minute interval—it is a longer interval in ASP.NET 2.0, but configurable through the web.config file. You indicate a sliding expiration through a TimeSpan object—a class that expresses a time interval. The FromSeconds static method creates an interval in seconds.

Each item stored in the ASP.NET cache is given a priority that is used when the system runs low on memory. Items with a lower priority are removed first to keep the system healthy.

The CachedImageService.axd Handler

As I mentioned, an HTTP handler is a piece of code used to service requests of a certain type. Typically, an HTTP handler processes requests that have a certain extension or that match a certain URL. Figure 7 shows the code of the HTTP handler used to retrieve image data from the cache.

Figure 7 Retrieve Image from Cache (Close Figure 7)
using System;
using System.Web;
using System.Drawing.Imaging;

namespace MsdnMag
{
    public class CachedImageService : IHttpHandler
    {
        // Override the IsReusable property
        public bool IsReusable
        {
            get { return true; }
        }

        // Override the ProcessRequest method
        public void ProcessRequest(HttpContext context)
        {
            string storageKey = "";

            // Retrieve the DATA query string parameter
            if (context.Request["data"] == null)
            {
                WriteError();
                return;
            }
            else
                storageKey = context.Request["data"].ToString();

            // Grab data from the cache
            object o = HttpContext.Current.Cache[storageKey];
            if (o == null)
            {
                WriteError();
                return;
            }

            if ((o as byte[]) != null)
                WriteImageBytes((byte[]) o);
            else
                if ((o as System.Drawing.Image) != null)
                    WriteImage((System.Drawing.Image) o);
        }

        private void WriteImageBytes(byte[] img)
        {
            HttpContext.Current.Response.ContentType = "image/jpeg";
            HttpContext.Current.Response.OutputStream.Write(img, 0,
                img.Length);
        }

        private void WriteImage(System.Drawing.Image img)
        {
            HttpContext.Current.Response.ContentType = "image/jpeg";
            img.Save(HttpContext.Current.Response.OutputStream,
                ImageFormat.Jpeg);
        }

        private void WriteError()
        {
            HttpContext.Current.Response.Write("No image specified");
        }
    }
}

The IHttpHandler interface has only two methods of its own, one of which (IsReusable) just returns false in most common implementations. The method value indicates whether another request can reuse the instance of the handler.

The core of the handler is in the ProcessRequest method. This method contains the logic of the handler and governs the processing of the request. The image service handler expects to find a data parameter in the query string. If the parameter is missing, the method outputs an error message:

if (context.Request["data"] == null)
{
   WriteError();
   return;
}
void WriteError()
{
   HttpContext.Current.Response.Write("No image specified");
}

The data parameter represents the key of the item in the ASP.NET cache that contains the image to display. The handler grabs the image from the cache and outputs it to the browser. The image is rendered according to the storage format. If the image is stored as an array of bytes, the array is flushed to the OutputStream property of the Response object. Otherwise, if the image is a GDI+ object, the Save method of the Bitmap class targets the same stream.

The code in Figure 7 defaults to JPEG as the image format. This is arbitrary and using another image type is certainly possible. Later, I'll return to this point to design a more general solution that includes the image type as a variable argument.

The HTTP handler is a class and gets compiled in the same assembly as the DynamicImage control. How would you invoke it? ASP.NET 2.0 uses the CachedImageService.axd URL to link to it. Can the same be done in ASP.NET 1.x? Yes, of course. The mapping between an HTTP handler and a URL name (or extension) can be set declaratively in the application's web.config file. If that is going to be a system setting, then you can move the following script to the machine.config file. (This happens to be exactly what ASP.NET 2.0 does.)

<configuration>
  <system.web>
    <httpHandlers>

      <add verb="GET"
           path="cachedimageservice.axd"
           type="MsdnMag.CachedImageService,DynamicImage" />

 </httpHandlers>
 </system.web>
</configuration>
The <httpHandlers> section instructs the ASP.NET runtime to use the MsdnMag.CachedImageService handler class to service any incoming GET request targeted to the cachedimageservice.axd URL. The .axd URL doesn't correspond to a real server file or resource. It is just a fake name used to bind a given functionality (coded into the HTTP handler class) to a URL.

The DynamicImage Control In Action

Now that the ASP.NET 1.x image generation service is up and running, let's see how to use it in a sample application. Create your own ASP.NET project, tweak the web.config file as described previously, and reference the DynamicImage control in your toolbox. The sample page has three instances of the DynamicImage control, each set with a different source property—ImageFile, Image, and ImageBytes:

<cc1:DynamicImage id="diUrl" runat="server"
     ImageFile="images/msdnmag.gif" />
<cc1:DynamicImage id="diGdi" runat="server"
     Image='<%# GetRandomNumber() %>' />
<cc1:DynamicImage id="diDb" runat="server"
     ImageBytes='<%# GetEmployeePhoto(3) %>'/>

Both the Image and ImageBytes properties are bindable and can be set declaratively with the return value of any function that generates the proper data type. Figure 8 shows the source code of the two helper functions that provide the contents of the images to display. GetRandomNumber creates and returns an in-memory image that represents a randomly generated number. GetEmployeePhoto retrieves the picture of an employee from the Northwind database and attaches those bytes to the ImageBytes property of a DynamicImage control. Figure 9 shows the page in action. The grid at the bottom of the page shows the current content of the Cache object. The output is generated by a custom user control that iterates through the cache and populates a grid (see the source code in this month's download).

Figure 8 GetRandomNumber and GetEmployeePhoto (Close Figure 8)
public System.Drawing.Image GetRandomNumber()
{
    // Store the number
    Random gen = new Random();
    int number = gen.Next();

    // Generate the image
    Bitmap bmp = new Bitmap(200, 100);
    using (Graphics g = Graphics.FromImage(bmp))
    {
        g.Clear(Color.LightCyan);
        using (Font f = new Font("Impact", 20))
        {
            g.DrawString(number.ToString(), f, Brushes.Blue, 10, 10);
        }
    }
    return bmp;
}

public byte[] GetEmployeePhoto (int empID)
{
    using (SqlConnection conn = new SqlConnection(
        "SERVER=localhost;DATABASE=northwind;Integrated Security=SSPI"))
    {
        using (SqlCommand cmd = new SqlCommand(
               "SELECT photo FROM employees WHERE employeeid=@id", conn))
        {
            cmd.Parameters.Add("@id", empID);
            conn.Open();
            byte[] img = (byte[]) cmd.ExecuteScalar();
            conn.Close();
            MemoryStream ms = new MemoryStream (img, 78, img.Length - 78);
            return ms.ToArray();
        }
    }
}
Figure 9 The Page in Action
Figure 9 The Page in Action

The image in the top-left corner comes from a server-side image file. The URL of the corresponding <img> tag served to the browser matches the URL assigned to the ImageFile property of the server control. The other two images are generated dynamically. Their URLs look like this:

cachedimageservice.axd?data=2a1612b8-0d4a-4757-ab91-7a29f25c1aea

If you make a separate request for this URL you'll get the image, as long as the cached item has not been removed. The cache lifetime depends on the sliding expiration set in the DynamicImage control (see Figure 6). As mentioned, this parameter is configurable in ASP.NET 2.0 and set to five minutes by default. If the image is removed from the cache, a direct call would return an error. A call made through the control, on the other hand, would trap the error, generate a new GUID, then reload and cache the image.

Adding Support for Image Types

In the example developed thus far, the image type was assumed to be a JPEG. There's no reason you can't change it to any other type that the Microsoft® .NET Framework supports (GIF, PNG, BMP, and the like). The image type can sometimes be inferred from the image bytes and the in-memory representation of the image. More precisely, if you hold the image bytes you can apply some patterns to the array and recognize the format—all image formats begin with a well-known signature and header; however, if the image is created as a GDI+ object, then no information about the image type is stored. A GDI+ image is rendered as a format-independent object—the Bitmap class. In spite of the name, though, the .NET Framework Bitmap class simply represents an image as a matrix of RGB pixels (plus the alpha channel, for a total of 32 bits per pixel) with no relationship to a common image file format. Only when the Bitmap object is serialized does the image format become an essential parameter:

img.Save(stream, imageType)

The internal serializer looks at the desired type and converts the generic RGB format to a layout typical of JPG or GIF files.

In ASP.NET 2.0, the image type is expressed as an attribute to the DynamicImage control and stored with the cached image object. This parameter is then retrieved and used when it is time to output the image stream. To add this capability to the ASP.NET 1.x image generation service, you start by adding a new public property, called ImageType, to the DynamicImage control:

public ImageType ImageType
{
   get {return (ImageType) ViewState["ImageType"];}
   set {return ViewState["ImageType"] = value;}
}

The type of the property is an enumeration named ImageType. The ImageType enumeration indicates the preferred image type for dynamically generated images. It is worth noting that in ASP.NET 2.0, if the requesting browser does not support the image type you specify, the DynamicImage control automatically provides a different type of image based on the display capabilities of the particular browser in question:

public enum ImageType
{
   Jpeg,
   Gif,
   Bmp,
   Png
}

The ImageType property indicates the desired type of the output image. You can also build into the DynamicImage control the ability to convert the image from the original format to the desired one. If the ImageType property is not specified, you can make it default to a fixed type (JPEG) or infer it from the file name extension if the ImageFile property is set.

In any case, once the desired image type has been determined, you pack the image to cache into a new data structure, like so:

class ImageInfo
{
    object imageObject;
    ImageType imageType;
}
The imageObject field contains the bytes of the image if the ImageBytes property has been set, or it contains the GDI+ object if the Image property is used instead. The imageType property sets the desired output type.

It is important to note that setting the ImageType property doesn't automatically ensure that the image sent to the browser is of the right type. This is easy to code for GDI+ objects, but not if the image is read out of a database field or a file:

void WriteImage(System.Drawing.Image img)
{
   string mimeType = ImageTypeToMimeType();
   HttpContext.Current.Response.ContentType = mimeType;
   img.Save(HttpContext.Current.Response.OutputStream,
      m_imageType);
}

If the image is stored as a format-independent object, then you simply need to express its type as a MIME type (for example, image/jpeg). The conversion from the RGB format to the desired one is necessary and cannot be avoided. Things are a bit different if the image is an array of bytes or a file. If it's a file, the image type is well-defined and implicit in the bytes but may not match the value of the ImageType property. The simplest way to code a control that always outputs the image as specified by ImageType, irrespective of the original format, is to convert the source format to a GDI+ object. This code shows how to load an image from a file:

FileStream fs = new FileStream(url, ...);
Bitmap img = Image.FromStream(fs);
Similarly, you can load a GDI+ image from an array of bytes.
MemoryStream ms = new MemoryStream(m_imageBytes);
Bitmap img = Image.FromStream(ms);

At this point, regardless of the original format, you cache the image as a generic Bitmap object and can output it in the desired format with no problem—the approach taken in ASP.NET 2.0.

Conclusion

The DynamicImage control in ASP.NET 2.0, and in the implementation provided here for ASP.NET 1.x, achieves two main goals. First, it gives you an easy way to display images that are dynamically generated and not simply read from a static file. As I demonstrated, you can also bind the properties of the new control to data-bound expressions for maximum flexibility. Just remember to place a call to Page.DataBind somewhere in the code to trigger the evaluation of any data-bound expressions throughout the page.

Second, the DynamicImage control provides a smart way to cache generated images. When static files are used, they are downloaded the first time (unless cache-specific settings prevent that) and loaded from the local browser cache when needed. One of the issues with dynamic images accessed through .aspx pages or .ashx handlers in ASP.NET 1.x, is that the image is regenerated each time it is accessed. The DynamicImage control reduces this overhead significantly by storing the generated image in the ASP.NET cache and retrieving it from there. A configurable parameter lets you decide how long the images can be stored. Each image is given a unique GUID and when the image is no longer available from the cache, it is regenerated.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an instructor and consultant based in Rome, Italy. Author of Programming ASP.NET (Microsoft Press, 2003), he spends most of his time teaching classes on ADO.NET and ASP.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


The XML Files: All About Blogs and RSS

The XML Files
All About Blogs and RSS
Aaron Skonnard

Q What is blogging all about?

A First, "blog" is short for Web log. It's a medium in which an author writes a journal-style Web site with provisions for readers to respond. These Web logs are becoming quite valuable in the software community for sharing ideas. Check out blogging on MSDN® at http://blogs.msdn.com.

Q What's the easiest way to set up a Web log?

A The easiest way to set up your own Web log is to go to a site like blogger.com and register as a new user. It provides a Web interface for creating a customized Web log that you can use immediately. There are many other sites like blogging.com that provide support for Web log features.

If you'd like more control over the blogging infrastructure or would like to host your blog on your own server, you can also use one of many blogging applications available today including Radio Userland, Manila, and Movable Type—some of the most popular commercial products. There are also free .NET blogging applications that are easy to use. The most popular are .Text and dasBlog. To set these up, simply download the bits and follow the instructions. You'll be up and running in minutes.

Functionally, both .NET-based applications are fairly equivalent. However, one major difference is that .Text requires a database, either SQL Server™ or MSDE, while dasBlog stores everything in XML files (it's based on the original BlogX framework created by some Microsoft developers). Another difference is that .Text is capable of hosting multiple blogs on a single installation (for example, it's what drives http://blogs.msdn.com today) while dasBlog requires multiple installations. dasBlog has one feature that really stands out called "Mail to Weblog", which allows you to post new entries via e-mail.

The new MSDN blogging site and PDC Bloggers are both good starting places for finding Web logs on any software development topic. Simply browse to one of these sites and read their aggregated feeds. Their feeds will expose you to many individual Web logs and over time you'll naturally find some that you like to read more than others. Then, you can subscribe directly to the individual feeds you enjoy most.

For blogs that specifically cover XML and Web services, check out the list on the MSDN Web Services Developer Center. I personally spend a lot of time on some of these Web logs.

Q What's a feed and how can I subscribe to it?

A A Web log can provide a feed to its content by producing an RSS document available via a well-known URL. An RSS document is an XML file that contains a number of discrete news items, such as entries in a Web log (see Figure 1 for a sample RSS feed). As an XML format, RSS is easily consumed by other programs.

Figure 1 Sample RSS 1.0 Document (Close Figure 1)
<rdf:RDF
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns="http://purl.org/rss/1.0/"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
>
  <channel rdf:about="http://skonnard.com/blog/rss.xml">
    <title>The XML Files</title>
    <link>http://skonnard.com/blog</link>
    <description>by Aaron Skonnard</description>
    <image rdf:resource="http://skonnard.com/blog/images/image.gif" />
    <items>
      <rdf:Seq>
        <rdf:li resource=" http://skonnard.com/blog/entry1" />
        <rdf:li resource=" http://skonnard.com/blog/entry2" />
      </rdf:Seq>
    </items>
  </channel>
  <image rdf:about=" http://skonnard.com/blog/images/image.gif">
    <title>skonnard.com</title>
    <link>http://skonnard.com/blog</link>
    <url>http://skonnard.com/blog/images/image.gif</url>
  </image>
  <item rdf:about="http://skonnard.com/blog/entry1">
    <title>1st blog entry</title>
    <link>http://skonnard.com/blog/entry1</link>
    <description>This is my first blog entry.</description>
    <dc:date>2004-01-13T17:16:44.9803903-07:00</dc:date>
  </item>
  <item rdf:about="http://skonnard.com/blog/entry1">
    <title>2nd Blog Entry</title>
    <link>http://skonnard.com/blog/entry1</link>
    <description>This is my second blog entry.</description>
    <dc:date>2004-01-13T17:16:45.9803903-07:00</dc:date>
  </item>
</rdf:RDF>

An RSS aggregator is a program that reads RSS documents and displays new items. Most aggregators make it possible to subscribe to a feed by simply entering the URL to the RSS document.

RSS makes reading Web logs easy. Most developers who frequently read Web logs use an aggregator of some sort to help them sift through their subscriptions efficiently. An aggregator makes reading Web logs feel a lot like reading e-mail since they highlight new items and cache items for offline reading (see Figure 2).

Figure 2 Sample RSS 2.0 Document (Close Figure 2)
<rss version="2.0">
  <channel>
    <title>The XML Files</title>
    <link>http://Skonnard.com/blog</link>
    <description>by Aaron Skonnard</description>
    <image>
      <url>http://skonnard.com/blog/images/image.gif</url>
      <title>skonnard.com</title>
      <link>http://skonnard.com/blog/</link>
    </image>
    <item>
      <title>1st blog entry</title>
      <link>http://skonnard.com/blog/entry1</link>
      <description>This is my first blog entry.</description>
      <pubDate>Wed, 14 Jan 2004 17:16:44 GMT</pubDate>
    </item>
    <item>
      <title>2nd blog entry</title>
      <link>http://skonnard.com/blog/entry1</link>
      <description>This is my second blog entry</description>
      <pubDate>Wed, 14 Jan 2004 17:16:45 GMT</pubDate>
    </item>
  </channel>
</rss>

There are also some online RSS aggregators that consolidate your RSS subscriptions on a separate Web site. This approach has the advantage of being easy to set up and you can access your subscriptions from any computer. The downside, of course, is that you have to be connected to do any reading.

RSS is ultimately what's made blogging such a powerful new form of communication. Before blogging, most developers spent a lot of time sifting through boring, irrelevant posts in order to discover the rare gems that would occasionally appear from people they respect. Blogging puts readers back in control by allowing them to choose which feeds to read, effectively building their own personalized content streams.

Other types of sites can also take advantage of RSS to syndicate content. For example, most of the major news sites including Wired, CNet, Yahoo!, and NPR News provide RSS feeds. Check out Blogdigger and Syndic8 to find sites that support RSS.

At Microsoft, MSDN provides RSS feeds to syndicate new technical content as it's added to the site. The MSDN Just Published feed is a great way to keep up with new MSDN articles and downloads. Even MSDN Magazine has its own RSS feed! Subscribe to http://msdn.microsoft.com/msdnmag/rss/rss.aspx to receive a monthly update on what's in the current issue. There are many RSS aggegators to choose from today. You can find a fairly complete list at http://blogs.law.harvard.edu/tech/directory/5/aggregators. Some of these are online aggregators while others are desktop applications. Some are free while others charge a fee.

Q Which RSS version is the most current?

A The answer depends on who you ask. There have been several versions of RSS including 0.90, 0.91, 0.92, 0.93, 0.94, 1.0, and 2.0. Making sense of these different versions has been one of the biggest challenges. Understanding them requires a bit of history.

Netscape created the original version of RSS, 0.90, which stood for "RDF Site Summary" or "Rich Site Summary" (the spec says the former was the official name). Netscape invented RSS 0.90 for use in their Web portal activities, but others latched onto the concept and saw more potential uses. Userland Software was one of the first to begin using RSS commercially in their Web log products.

Version 0.90 was heavily based on the W3C's Resource Description Framework (RDF). Many considered the RDF approach overly complex, so a simplified RDF-free version was proposed and labeled 0.91. It was around this time that control of 0.91 passed to Userland Software. Userland Software continued to evolve the simplified spec with several new versions including 0.92, 0.93, and 0.94. To emphasize their focus on simplicity, it wanted RSS to stand for "Really Simple Syndication."

As Userland Software continued with their focus on simplicity, another group of developers resurrected the original RDF version (0.90) because RDF promised them more flexibility. They eventually published RSS 1.0, which officially stands for "RDF Site Summary" again. This version is fundamentally different from those controlled by Userland Software because it uses RDF while the others don't. Userland Software didn't like the fact that RSS 1.0 seemed to displace RSS 0.94, so it shipped a new version and bumped the version number up to 2.0.

And that's where it stands today. The split that occurred left two major competing versions: one that's based on RDF (1.0) and one that isn't (2.0), but they both share the same name. This is terribly confusing since the version numbers lead you to believe that 2.0 is an improvement on 1.0 when in reality they're completely different specifications with different goals. Another group of developers has been working to resolve this confusion once and for all by defining a new syndication specification that breaks free from the RSS name. They're calling it Atom, a project I'll discuss in more detail later in this column.

It doesn't matter much which version you use. Most RSS aggregators support all RSS versions (and some even support Atom) without a glitch. The decision mostly comes down to whether you want to use RDF, which is typically fueled by one's belief in the concept of the Semantic Web.

Q What do RSS 1.0 and 2.0 look like?

A The RSS 1.0 and 2.0 formats contain the same core information, but they're structured differently. I've provided a sample RSS 1.0 document (see Figure 1) and the equivalent RSS 2.0 document (see Figure 2) for you to look over.

You'll notice the differences start right at the top with the root element. In RSS 1.0, the root element is rdf:RDF, and in RSS 2.0 it's rss. The rss element also contains a mandatory version attribute to indicate the precise RSS format in use (possible values include 0.91, 0.94, and so forth). Another major difference is that RSS 1.0 documents are namespace-qualified, while RSS 2.0 documents are not. The information contained in both documents, however, is essentially the same.

Both versions contain channel elements. A channel element contains three required elements: title, description, and link, as illustrated in the following code:

<channel>
   <title><!-- the channel's title --></title>
   <description><!-- a brief description --></description>
   <link><!-- the channel's URL --></link>
   <!-- optional/extensibility elements go here -->
</channel>

In addition to these required elements, RSS 1.0 defines three additional elements: image, items, and textinput, where image and textinput are optional. RSS 2.0, on the other hand, provides 16 additional elements including image, items, and textinput. Examples of these include language, copyright, managingEditor, pubDate, and category. RSS 1.0 allows for making this type of metadata available through extensibility elements defined in separate XML namespaces.

The main structural difference between the two formats has to do with the representation of item, image, and textinput nodes. In RSS 1.0, the channel element contains references to item, image, and textinput nodes that exist outside of the channel itself. This establishes an RDF association between the channel and the referenced node. In Figure 1, the channel element is associated with an image element and two item elements. In RSS 2.0, the item elements are simply serialized in the channel element (see Figure 2).

The item element contains the actual news item information. The structure of item is similar across both versions. The item element usually contains title, link, and description elements, as shown in the following code:

<item>
   <title><!-- the item's title --></title>
   <link><!-- the item's URL --></link>
   <description><!-- a brief description --></description>
   <!-- optional/extensibility elements go here -->
</item>

In RSS 1.0, title and link are required, while description is optional. In RSS 2.0, either title or description must be present; everything else is optional. These are the only item elements defined in RSS 1.0, while RSS 2.0 provides several other optional elements including author, category, comments, enclosure, guid, pubDate, and source. RSS 1.0 makes such metadata available through extensibility elements defined in separate XML namespaces known as RSS modules. For example, in Figure 1 the item's date is represented using the Dublic Core module's <dc:date> element.

Check out the RSS 1.0 and 2.0 specifications for complete details on the different formats.

Q So, what is Atom anyway?

A As I mentioned earlier, Atom is the name of a project for developing a new Web log syndication format to address what many feel are the main problems with RSS today (a soup of confusing version numbers, not a truly open standard, inconsistent, poorly defined, and so on). Atom hopes to offer a clean version that addresses everyone's needs. It is designed to be completely vendor neutral, freely extensible by anybody, and thoroughly specified.

Many of today's blogging engines already support the current Atom syndication format. Figure 3 shows a sample Atom 0.3 feed that is equivalent to the RSS feeds shown in Figure 1 and Figure 2. Notice that the Atom feed is namespace qualified but it doesn't use RDF. This gives Atom something in common with both RSS 1.0 and RSS 2.0. It will be interesting to see how Atom's acceptance plays out in the years to come.

Figure 3 Sample Atom 0.3 Feed (Close Figure 3)
<feed version="0.3" xml:lang="en-us"
  xmlns="http://purl.org/atom/ns#"
>
  <title>The XML Files</title>
  <link>http://skonnard.com/blog/</link>
  <modified>2004-01-13T17:16:45.0004199-07:00</modified>
  <tagline>by Aaron Skonnard</tagline>
  <author>
    <name>Aaron Skonnard</name>
  </author>
  <entry>
    <title>1st blog entry</title>
    <link>http://skonnard.com/blog/entry1</link>
    <created>2004-01-13T17:16:44.9803903-07:00</created>
    <content type="text/html" mode="xml">
      <body xmlns="http://www.w3.org/1999/xhtml">
        <p>This is my first blog entry</p>
      </body>
    </content>
  </entry>
  <entry>
    <title>2nd blog entry</title>
    <link>http://skonnard.com/blog/entry2</link>
    <created>2004-01-13T17:16:45.9803903-07:00</created>
    <content type="text/html" mode="xml">
      <body xmlns="http://www.w3.org/1999/xhtml">
        <p>This is my second blog entry</p>
      </body>
    </content>
  </entry>
</feed>

In addition to defining a new syndication format, also hopes to define a standard archiving format and a standard Web log editing API (the Atom API). Check out The Atom Project to peruse the specifications and other Atom resources.

Q What's a blogroll?

A A blogroll is simply a collection of Web log feeds. Most bloggers provide a blogroll on their personal Web log. This allows their readers to connect with others who share similar interests or writing styles. Blogrolls facilitate building networks of respect. A blogroll can be exchanged in XML format using the Outline Processor Markup Language (OPML). Figure 4 shows a sample blogroll.

Figure 4 Sample Blogroll (OPML) (Close Figure 4)
<opml>
  <head> <title>Aaron's Favorite Blogs</title> </head>
  <body>
    <outline type="rss" title="PDC Bloggers"
      description="PDC Bloggers website"
      xmlUrl="http://pdcbloggers.net/Feed.rss"
      htmlUrl="http://PDCBloggers.net" />
    <outline type="rss" title="MSDN Magazine: Current Issue"
      description="The Microsoft Journal for Developers"
      xmlUrl="http://msdn.microsoft.com/msdnmag/rss/recent.xml"
      htmlUrl="http://msdn.microsoft.com/msdnmag/" />
    <outline type="rss" title="MSDN Just Published"
      description="Keep current ..."
      xmlUrl="http://msdn.microsoft.com/rss.xml"
      htmlUrl="http://msdn.microsoft.com/" />
  </body>
</opml>

Most blogging engines will manage blogrolls for you and generate the proper XML format when readers request it. Likewise, most aggregators make it possible to import a blogroll and automatically subscribe to the contained feeds. See http://opml.scripting.com for more information on OPML.

Q Can you explain what referrers, trackbacks, and pingbacks are?

A Most blogging software makes it possible for readers to add comments to a Web log. It's actually more common, however, for readers to add an entry to their own Web log that links back to the original post. Bloggers like to keep track of when this happens so that new readers can follow the entire conversation.

A referrer is an external site from which a user clicked on a hyperlink to reach your site. Many blogging engines will automatically keep track of referrers as readers navigate to an entry on your Web log. Most engines will display the list of referrers at the bottom of the Web log entry so readers can navigate back to the referrer's site and see what they have to say about the entry, based on the assumption that they probably wrote something about it if they linked to it. The problem with referrers has to do with this assumption—there isn't enough information to tell if the referring page actually contains additional relevant information. In fact, spammers have already taken advantage of this loophole to redirect readers for marketing purposes.

Trackback and pingback are similar specifications developed to remedy this situation. Using trackback or pingback, other bloggers can automatically send a ping to your Web log indicating explicitly that they have written an entry that references a specific post. This type of reverse linking allows your Web log to display a list of all entries that have actually commented on your post in a more explict manner. Most of today's blogging software supports all of these techniques. See TrackBack Technical Specification and Pingback 1.0.

Q How can I generate an RSS feed for my Web site?

A Figure 5 illustrates how to generate an RSS 2.0 feed in an .aspx page using an asp:Repeater control. This page assumes that you'll set the control's DataSource property in the codebehind file to the appropriate database resultset.

Figure 5 Generating an RSS 2.0 Feed in ASP.NET (Close Figure 5)
<%@ Page language="c#" Codebehind="rss.aspx.cs"
    AutoEventWireup="false" Inherits="SimpleBlog.rss" %>
<rss version="2.0">
  <channel>
    <title>My Blog</title>
    <link>http://localhost/simpleblog/default.aspx</link>
    <description>A weblog about nothing...</description>
    <language>en-us</language>
    <asp:Repeater id="Items" runat="server">
      <ItemTemplate>
        <item>
        <title><%#DataBinder.Eval(Container.DataItem,
            "title")%></title>
        <description><%#DataBinder.Eval(
            Container.DataItem,"description")%></description>
        <pubDate><%#DataBinder.Eval(Container.DataItem,
            "pubdate") %></pubDate>
        <link><%# DataBinder.Eval(Container.DataItem, "link") %></link>
        </item>
      </ItemTemplate>
    </asp:Repeater>
  </channel>
</rss>

Q I'd like to aggregate several RSS feeds and display the information on my personal Web site. Can you explain how to do this?

A Since RSS feeds are XML files, accomplishing this is an exercise in using your favorite XML API, such as System.Xml in the Microsoft .NET Framework. Figure 6 contains the code for an ASP.NET Web user control that I wrote to aggregate the RSS feeds listed in a blogroll file (.opml). The code assumes that the opml element will contain a numberToDisplay attribute to indicate how many items from each feed you want to display.

Figure 6 RSS Aggregator Web User Control (Close Figure 6)
<%@ Control Language="c#" AutoEventWireup="true"
    EnableViewState="false" Debug="true"%>
<%@ Import namespace="System.Xml" %>
<%@ OutputCache Duration="1800" VaryByParam="none" %>
<script runat="server" language="C#">
private void Page_Load(object sender, System.EventArgs e)
{
  StringBuilder sb = new StringBuilder();
  XmlDocument doc = new XmlDocument();
  doc.Load(Server.MapPath("blogroll.opml"));
  int numToDisp = int.Parse(doc.SelectSingleNode(
    "/opml/@numberToDisplay").InnerText);
  XmlNodeList rss = doc.SelectNodes("//outline/@xmlUrl");
  foreach (XmlNode r in rss)
  {
    XmlDocument blogdoc = new XmlDocument();
    blogdoc.Load(r.Value);
    XmlNodeList items = blogdoc.SelectNodes("//item");
    for (int i=0; i<items.Count && i<numToDisp; i++)
    {
      string author="";
      XmlNode authorNode = items[i].SelectSingleNode(
       "*[local-name()='author' or local-name()='creator']");
      if (authorNode != null) author = authorNode.InnerText;
      sb.Append(String.Format(
        "&#149;&nbsp;<a href={0}>{1} ({2})</a><br/>",
        items[i].SelectSingleNode("link").InnerText,
        items[i].SelectSingleNode("title").InnerText, author));
    }
  }
  EntriesHTML.Text = sb.ToString();
}
</script>
<style> <!-- styles omitted for brevity --> ...  </style>

<div class="title">UNUG Blogs</div>
<asp:Literal id="EntriesHTML" runat="server"></asp:Literal>
Figure 7 ASP.NET Web User Control
Figure 7 ASP.NET Web User Control

You can drop this control into any .aspx page and it will display items from the various feeds listed in the blogroll. Figure 7 shows this control in action on the Utah .NET User Group Web site.

Q Are there any Web service APIs for interacting with Web logs?

A Many blogging engines provide their own proprietary Web service interface for interacting with a Web log programmatically, but I wouldn't say a standard has emerged yet.

Both .Text and dasBlog provide some .asmx endpoints that provide editing functionality via SOAP, but their interfaces are different. Blogger.com provides an interactive API (Blogger API) based on XML-RPC. Userland Software enhanced the Blogger API and called it the MetaWeblog API. These are probably the most widely recognized Web log APIs today, but still not all Web log engines support them. There is also a separate API for adding comments called the Comment API, but again, it's not universally supported.

The Atom group is currently working to resolve this mess. The Atom API defines a standard Web log API for publishing and editing all Web log content. You can check out their work at The Atom Project.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard teaches at Northface University in Salt Lake City. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000), and frequently speaks at conferences. Reach him at http://www.skonnard.com.


Advanced Basics: Synchronizing Multiple Windows Forms

New information has been added to this article since publication.
Refer to the Editor's Update below.

Advanced Basics
Synchronizing Multiple Windows Forms
Ken Spencer

Code download available at: AdvancedBasics0404.exe (135KB)

Q I'm creating a Windows® Forms application with multiple instances of a single form. I'd like to write it so that any operation performed on any one instance of the form will be reflected on all other instances. How can I do this?

A This is an interesting question. I guarantee that some smart code jockey is going to suggest that I use delegates. Before that happens, let's explore several solutions to this problem.

Say I have two forms, each with two textbox controls: txt1stData and txt2ndData. How can I keep those controls in sync on each of the two forms? For the purposes of this discussion, it does not matter whether I have two forms or ten, the problem is the same.

The first approach is rather simple. In fact, it's even simpler than using delegates directly, which I think is sometimes overkill. First, I created a class that contains properties that I want to share with all the forms in the application (see Figure 1). For instance, MyData and MoreData hold data that each form can display. I will come back to this class in a minute.

Figure 1 Class with Shared Properties (Close Figure 1)
Public Class DataClass
    Private privateMyData As String = ""
    Private privateMoreData As String = ""

    Public Event MyDataChanged As EventHandler
    Public Event MoreDataChanged As EventHandler
    Public Property MyData() As String
        Get
            Return privateMyData
        End Get
        Set(ByVal Value As String)
            privateMyData = Value
            RaiseEvent MyDataChanged(Me, New EventArgs)
        End Set
    End Property
    Public Property MoreData() As String
        Get
            Return privateMoreData
        End Get
        Set(ByVal Value As String)
            privateMoreData = Value
            RaiseEvent MoreDataChanged(Me, New EventArgs)
        End Set
    End Property
    Public Sub Reset()
        MyData = ""
        MoreData = ""
    End Sub
End Class

Second, I created two forms with the same controls (txt1stData and txt2ndData) as mentioned earlier. You can see the layout of the form in Figure 2. Both forms have exactly the same data, and I'll explain why in a moment.

Figure 2 Form Layout
Figure 2 Form Layout

Next, I created a module named modGeneral and added the following line of code:

Friend DataStuff As DataClass
This line creates a friend variable for my new DataClass that you can access throughout the assembly, which for this simple example is the entire application. Then I added the following code to the Form1 Load event:
DataStuff = New DataClass

Me.txt1stData.DataBindings.Add("Text", DataStuff, "MyData")
Me.txt2ndData.DataBindings.Add("Text", DataStuff, "MoreData")
The first line creates a new instance of DataClass. The second two lines of code set up the data binding to the textbox controls. That's it for this form!

Now, how do you synchronize them with the data on Form2 and all other forms? Add the following two lines to the form load event of Form 2:

Me.txt1stData.DataBindings.Add("Text", _
    DataStuff, "MyData")
Me.txt2ndData.DataBindings.Add("Text", _
    DataStuff, "Moredata")

That's one easy way to make sure all forms are in sync with most any type of data. You can simply data bind the controls to the same instance of a class and you're finished.

Now, for another approach. I created a new form named frmBase. Then I placed a textbox (txtNextData) and label on it. I wanted each form in my application to share the textbox and label and I wanted them all to stay in sync with each other, so I rebuilt the project. I created Form1 and Form2 by inheriting from the new frmBase so they could inherit all the new controls. But how can I keep the controls in sync? This time a little coding is required to make it work, but the code is in a single class where it's reused by simply calling a function.

The code sample in Figure 3 shows the module called modGeneral. Its first task is to define two variables: MyForms and localNextData. MyForms is a collection that will contain a list of all the forms I want to synchronize. The variable that I called localNextData will store all the data that I want to display in the forms. Keep in mind that both of these variables could reside in a class instead of a module.

Figure 3 modGeneral (Close Figure 3)
Module modGeneral
    Friend DataStuff As DataClass
    Friend MyForms As New Collection
    Friend localNextData As String
    Sub AddForm(ByVal ThisForm As Form)
        MyForms.Add(ThisForm)
        UpdateControlsNextData(localNextData)
    End Sub

    Friend Property NextData() As String
        Get
            Return localNextData
        End Get
        Set(ByVal Value As String)
            localNextData = Value
            UpdateControlsNextData(localNextData)
        End Set
    End Property

    Private Sub UpdateControlsNextData(ByVal Value As String)
        Dim frm As frmBase
        Dim obj As Object

        For Each obj In MyForms
            frm = CType(obj, frmBase)
            If Not IsNothing(frm.txtNextData) Then
                frm.txtNextData.Text = Value
            End If
        Next
    End Sub
End Module

The AddForm procedure from modGeneral takes a form instance and adds it to the MyForms collection. [ Editor's Update - 12/6/2004: The code in Figure 3 has been updated to prevent multiple form instances from being added to the MyForms collection with the same key.] I will use this collection in the UpdateControlsNextData procedure to determine which forms to update. AddForm also calls UpdateControlsNextData to ensure a new form is updated with the correct data.

The other code in modGeneral is the NextData property. This property's set accessor updates localNextData and also calls UpdateControlsNextData to synchronize all of the forms. Now all I need to do is set NextData whenever I want to change it and all forms will be updated by the call to UpdateControlsNextData.

The third approach, custom linking, is a refinement of the second one. I created it to get a bit more flexibility when processing controls on a form. For instance, I only want to track and process certain forms, those with the controls that must be in sync. This method also lets me define which controls I want to synchronize and then process only those forms.

I added another module for this approach (modGeneralv2) which is shown in Figure 4. This module includes a collection (MyFormsToUpdate) which will hold all the forms I want to sync. The module also has a new array (ControlsToUpdate) which provides a list of the controls I want to sync. The definition for the array looks like this:

Private ControlsToUpdate() As String = _
    {"txtCustomer", "txtAddress", "txtName"}

Figure 4 modGeneralv2 (Close Figure 4)
Friend MyFormsToUpdate As New Collection
Private ControlsToUpdate() As String = _
  {"txtCustomer", "txtAddress", "txtName"}

Friend Sub AddFormToUpdate(ByVal ThisForm As Form)
  Dim ctrl As Control
  Dim i As Integer
  Dim AddThisForm As Boolean = False

  Try
    For Each ctrl In ThisForm.Controls
      For i = 0 To ControlsToUpdate.GetUpperBound(0)
          If ctrl.Name = ControlsToUpdate(i) Then
            AddThisForm = True
           Exit For
         End If
         If AddThisForm Then
           MyFormsToUpdate.Add(ThisForm, ThisForm.Name)
           Exit For
         End If
       Next
     Next
     If AddThisForm Then
       If MyFormsToUpdate.Count > 1 Then
         UpdateControlsOnAllForms( CType(MyFormsToUpdate(1), Form))
       End If
     End If
   Catch ex As Exception
     Throw New Exception( _
       "AddForm generated this error: " & ex.Message, ex)
   End Try
 End Sub

 Friend Sub UpdateControlsOnAllForms(ByVal MasterForm As Form)
   Dim frm As Form
   Dim ctrlMaster, ctrlClient As Control
   Dim i As Integer
   Try
   For Each frm In MyFormsToUpdate
     For Each ctrlMaster In MasterForm.Controls
       For i = 0 To ControlsToUpdate.GetUpperBound(0)
         If ctrlMaster.Name = ControlsToUpdate(i) Then
           For Each ctrlClient In frm.Controls
             If ctrlClient.Name = ControlsToUpdate(i) Then
               ctrlClient.Text = ctrlMaster.Text
                 Exit For
              End If
            Next
          End If
        Next
     Next
   Next
   Catch ex As Exception
     Throw New Exception( _
       "UpdateControlsNextData generated this error: " & ex.Message, ex)
     End Try
 End Sub

Instead of AddForm, this module has a new and improved version called AddFormToUpdate. This method works in a fashion similar to AddForm but now it only adds the forms that actually contain one or more of the controls in my ControlsToUpdate array, so only those forms with the specific controls are in the update collection. This allows me to call this function from every form. If I decide to add one of the special controls later, it will automatically be added to the forms list. I only need to make one small change to the forms code to hook it up.

This module also contains the UpdateControlsOnAllForms procedure, which performs the update. Instead of using an application-level variable as shown in the last method, I now use the concept of a master form. Thus I can copy the values from that form to all other forms in the collection. UpdateControlsOnAllForms is really a set of simple For...Nexts that loop through the controls on a form, find the ones to update, and update them.

To implement this functionality in my forms, I add this line to the form's Load event:

AddFormToUpdate(Me)
Alternately, I can add it into the constructor. This line will add the current form instance to the collection to update.

Now let's hook up a single event procedure:

Private Sub txt_Leave(ByVal sender As Object, _
    ByVal e As System.EventArgs) Handles txtAddress.Leave, _
    txtCustomer.Leave, txtName.Leave
        UpdateControlsOnAllForms(Me)
End Sub
This wires up the Leave event of all three controls (txtAddress, txtCustomer, and txtName) that I want to synchronize to one event handler. Then I can add the single line to call UpdateControlsOnAllForms. Me is passed to this procedure call to cause the other forms to be synchronized with this form.

Now I have three versions of code that can synchronize controls on forms, so I have a choice. I could have used custom events, defining an event in the DataClass and having each form subscribe to it. Then when the event fires, the forms could pick up the new data from each event handler and set the appropriate controls. But that's at least as much code as just data binding the controls to the class like I did in the first method. I could create a single procedure that performs the update and place this procedure in a module. I would need to pass the form instance to the procedure for it to perform the update. I could fire this procedure with an event handler from the class. The procedure would look like this one:

Sub UpdateControls(ByVal ThisForm As frmBase)
    With ThisForm
        .txtNextData.Text = localNextData
    End With
End Sub
The parameter to ThisForm is typed as frmBase so that it can have access to IntelliSense® and pick up properties that are custom to the form. Simply typing it as a Form will not cause the properties that are in frmBase and its derived forms to be shown.

Another option is to use delegates. A delegate would, of course, let me redirect an invocation of the delegate to a method in each form. If I used multicasting I could let each form handle the event and update the appropriate controls. It sounds really simple to set up this functionality with delegates, but to me it's more trouble than it's worth in practice. Plus, other than the nested For...Next loops in the third method, the code is not difficult to understand. And after all, the most costly part of an app is still its maintenance.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Spencer works for 32X Tech, where he provides training, software development, and consulting services on Microsoft technologies.


.NET Matters: Const in C#, Exception Filters, IWin32Window, and More

New information has been added to this article since publication.
Refer to the Editor's Update below.

.NET Matters
Const in C#, Exception Filters, IWin32Window, and More
Stephen Toub

Welcome to .NET Matters. This new column will delve into the ins and outs of the Microsoft® .NET Framework, answering readers' questions on various topics related to its extensive libraries, languages, and the common language runtime. If you have a question you'd like answered, feel free to contact us at netqa@microsoft.com. In the meantime, sit back, relax, learn, and enjoy.

Q Are const variables in C# stored in a global location? Can you tell me what are the performance and memory advantages of using a const in a class?

A Very similar to a #define in C, const "variables" in the .NET Framework don't have their own actual storage location but rather are substituted into the Microsoft Intermediate Language (MSIL) instructions at compile time where they are referenced. For example, if I have the following C# code

int myInt = 5;
Console.WriteLine(myInt);
csc.exe will generate this MSIL:
ldc.i4.5
stloc.0
ldloc.0
call void [mscorlib]System.Console::WriteLine(int32)
In other words, it loads the value 5, stores it to the myInt variable, then loads this value back onto the execution stack, and prints it out using Console.WriteLine.

If you change the myInt variable to be a constant, like so

const int myInt = 5;
Console.WriteLine(myInt);
the MSIL generated will be:
ldc.i4.5
call void [mscorlib]System.Console::WriteLine(int32)

Now, wherever myInt is referenced in the code, instead of having to do a "ldloc.0" to get the value from the variable, the MSIL just loads the constant value which is hardcoded into the MSIL. As such, there's usually a small performance and memory advantage to using constants. However, in order to use them you must have the value of the variable at compile time, and any references to this constant at compile time, even if they're in a different assembly, will have this substitution made.

Constants are certainly a useful tool if you know the value at compile time. If you don't, but want to ensure that your variable is set only once, you can use the readonly keyword in C# (which maps to initonly in MSIL) to indicate that the value of the variable can only be set in the constructor; after that, it's an error to change it. This is often used when a field helps to determine the identity of a class, and is often set equal to a constructor parameter.

Q In Visual C++® .NET and Visual Basic® .NET, I can use exception filters with my catch blocks to determine which exceptions should be caught by a catch block (using more than just the type of the exception). How can I do this in C#?

A While MSIL exposes exception filter functionality (and both managed C++ and Visual Basic .NET take advantage of this), not all languages that compile down to MSIL expose the complete feature set available in it. C# does not include support for exception filters. [ Editor's Update - 3/21/2004: This answer concerns user filters, the kind of filter generally implied by the term "exception filter". C# does not support user filters. As shown in the example, it does support type filters.] In all honesty, though, most applications that you write will not require exception filters. And if you find yourself in a situation where you think you need them, it's certainly possible to get close with a little bit more code. For example, let's say you have the following code in Visual Basic .NET:

Try
    ' do something that could throw an exception
Catch e As FileNotFoundException When MyFilterMethod(e)
    ' •••
Catch e As DivideByZeroException
    ' •••
End Try

This code has two catch blocks. The first one only catches FileNotFoundExceptions and only when the method MyFilterMethod returns true when the caught exception is passed to it. If the first catch block cannot handle the exception, the second one, which handles all DivideByZeroExceptions, will be given a chance.

C# doesn't support exception filters, so you'd need to write equivalent code that resembles Figure 1. Here, I combine all of the catch blocks I had in the previous snippet into a single catch block in my C# code. Then I manually test the exception against the exception types and filters and act accordingly. If none of my virtual catch blocks match, I rethrow the exception (note that I use "throw" rather than "throw e" so as not to rewrite the StackTrace contained in the exception).

Figure 1 Simulating Exception Filters in C# (Close Figure 1)
try
{
    // do something that could throw an exception
}
catch(Exception e)
{
    if (e is FileNotFoundException &&
        MyFilterMethod((FileNotFoundException)e))
    {
        // •••
    }
    else if (e is DivideByZeroException)
    {
        // •••
    }
    else throw;
}

// or alternatively using specific type filters, as
// the caught exception types are unrelated by inheritance
try
{
    // do something that could throw an exception
}
catch(FileNotFoundException e)
{
    if (MyFilterMethod(e))
    {
        // •••
    }
    else throw;
}
catch(DivideByZeroException e)
{
    // •••
}

It is true that C# does not support exception handlers, but as shown in Figure 1 it doesn't mean it is any less capable. This just happens to be one case in which Visual Basic .NET provides a simpler syntax to express a concept than does C#.

Q I'm writing a Windows® Forms application that runs in the system tray and displays message boxes to the user every now and then. Unfortunately, these message boxes are often covered up by other windows on the desktop, so I need to find a way to display my message box parented by whatever is the foreground window. In Win32®, I know I can use the GetForegroundWindow function to get a handle to the window with which the user is currently working. I can use interop to call this function, but it gives me back a handle and I have no idea how to use that in Windows Forms. Do you have any suggestions?

A Windows Forms provides the IWin32Window interface for just this purpose. This interface is implemented by objects that expose Win32 HWND handles, such as System.Windows.Forms.Control, from which System.Windows.Forms.Form is derived. As such, you need to wrap the P/Invoke to GetForegroundWindow with a class that implements IWin32Window. A complete sample implementation is shown in Figure 2.

Figure 2 Implementing IWin32Window (Close Figure 2)
public class ForegroundWindow : IWin32Window
{
    private static ForegroundWindow _window = new ForegroundWindow();
    private ForegroundWindow(){}

    public static IWin32Window Instance
    {
        get { return _window; }
    }

    [DllImport("user32.dll")]
    private static extern IntPtr GetForegroundWindow();

    IntPtr IWin32Window.Handle
    {
        get
        {
            return GetForegroundWindow();
        }
    }
}

Now you can display a message box that uses the foreground window as a parent using code like the following:

MessageBox.Show(ForegroundWindow.Instance, "Displayed on top!");
The message box retrieves the handle from ForegroundWindow using the IWin32Window.Handle property, which in turn calls to GetForegroundWindow to retrieve the appropriate handle. This technique could also be used with functions like GetActiveWindow, GetDesktopWindow, FindWindow, or any other piece of code that results in a valid HWND.

Note, too, that I've used a private constructor to prevent external instantiation of this class. The next version of the Framework will also add static class support, making this construct unnecessary. [ Editor's Update - 3/21/2004: Static classes prevent all instantiations. One would not be helpful in this particular scenario as ForegroundWindow needs to privately instantiate itself.]

Q Does every keyword in C# have an equivalent keyword in MSIL? For example, to what does the using keyword map? I wasn't able to find anything that matched.

A Everything written in C# needs to be expressable in MSIL as the C# compiler targets it, but that doesn't mean that there is a one-to-one mapping between C# keywords and MSIL keywords. For example, C# provides high-level looping constructs such as the for loop and the while loop. In MSIL, these concepts are expressed using conditional checks and jumps. The using keyword you mention is syntactic sugar provided in C# that makes it very easy to clean up resources that require disposal (using the IDisposable interface). The following C# code

using (StreamReader reader = new StreamReader(@"C:\test.txt"))
{
    string line = reader.ReadLine();
}
is compiled down to the MSIL shown in Figure 3. If you convert the MSIL back to C#, you get:
StreamReader reader = new StreamReader(@"C:\test.txt");
try
{
    string line = reader.ReadLine();
}
finally
{
    if (reader != null) ((IDisposable)reader).Dispose();
}
In other words, the using keyword simply maps to a try/finally construct that ensures the used object is always disposed. The code snippet just shown represents the typical expansion for the using keyword. You should note, however, that if a value type is used instead of a reference type, the compiler will not generate the check for a null value.

Figure 3 'using' Compiled to MSIL (Close Figure 3)
IL_0000:  ldstr      "C:\\test.txt"
IL_0005:  newobj     instance void
                 [mscorlib]System.IO.StreamReader::.ctor(string)
IL_000a:  stloc.0
.try
{
    IL_000b:  ldloc.0
    IL_000c:  callvirt   instance string
                 [mscorlib]System.IO.TextReader::ReadLine()
    IL_0011:  stloc.1
    IL_0012:  leave.s    IL_001e
}  // end .try
finally
{
    IL_0014:  ldloc.0
    IL_0015:  brfalse.s  IL_001d
    IL_0017:  ldloc.0
    IL_0018:  callvirt   instance void
                 [mscorlib]System.IDisposable::Dispose()
    IL_001d:  endfinally
}  // end handler
IL_001e: ...

Q What is the proper way to raise an event in C#? I've heard that I need to be concerned about multithreading issues when doing so.

A If an event in C# has no delegates registered with it, attempting to raise the event will cause a NullReferenceException. As a result, given an event declared as

public event EventHandler MyEvent;
you'll often see it raised with code such as:
if (MyEvent != null) MyEvent(this, EventArgs.Empty);

This works fine in a single-threaded environment, but consider the scenario in which multiple threads are accessing MyEvent simultaneously. In such a case, one thread could check whether MyEvent is null and determine that it isn't. Just after doing so, another thread could remove the last registered delegate from MyEvent. When the first thread attempts to raise MyEvent, an exception will be thrown. A better way to avoid this scenario is shown in the following code snippet:

void MyEventInvoke(object sender, EventArgs args) {
    EventHandler ev = MyEvent;
    if (ev != null) ev(sender, args);
}

Whenever a delegate is added to or removed from an event using the default implementations of the add and remove accessors, the Delegate.Combine and Delegate.Remove static methods are used. These methods return a new instance of a delegate, rather than modifying the ones passed to it. In addition, assignments of object references in .NET are thread-safe, and the default implementations of the add and remove event accessors are synchronized. As such, the previous code succeeds by first copying the multicast delegate from the event to a temporary variable. Any changes to MyEvent after this point will not affect the copy you've made and stored. You can now safely test whether any delegates were registered and subsequently invoke them.

As an aside, the RaiseEvent keyword in Visual Basic handles the null check for you and, as a result, developers using Visual Basic don't first need to test whether any delegates are registered with an event before raising it in a single-threaded scenario.

Q I'm trying to use the System.Diagnostics.Process class to run the C# compiler csc.exe, but I need to know the full path to the application, which is located in the .NET Framework's installation directory. The System.Environment.SpecialFolder enumeration allows me to get the path to a variety of folders on my system, but not this particular one. Using C#, how can I retrieve the runtime's installation directory? On my system, this is c:\WINDOWS\Microsoft.NET\Framework\v1.1.4322.

A Mscoree.dll exposes a GetCORSystemDirectory method which can be used to retrieve the installation directory for the version of the common language runtime (CLR) that is loaded into the current process, as shown here:

HRESULT GetCORSystemDirectory(
    LPWSTR pbuffer,   DWORD cchBuffer, DWORD* dwlength);
If the CLR is not loaded, it will return the installation directory for the latest version installed.

The first parameter, pbuffer, takes a pointer to the string that will hold the path upon the function's return. cchBuffer is the number of characters that can be safely stored in pbuffer, and dwlength is a pointer to an integer that holds the number of characters actually in the path. To use it from C#, you need to use interop:

[DllImport("mscoree.dll")]
private static extern int GetCORSystemDirectory(
    [MarshalAs(UnmanagedType.LPWStr)]StringBuilder pbuffer,
    int cchBuffer, ref int dwlength);

In the .NET Framework, strings are immutable, meaning that once created, they cannot be changed (some counter this assertion stating that strings can be changed using unsafe code and direct memory access, but doing so is a very bad idea). As such, instead of a String, the first parameter in the P/Invoke declaration is a mutable StringBuilder, marshaled as an unmanaged LPWSTR. You also need to ensure that StringBuilder is large enough to hold the path of the folder, which on a Win32 system is at most 260 characters. The resulting C# method looks like the following:

private static string GetClrInstallationDirectory()
{
    int MAX_PATH = 260;
    StringBuilder sb = new StringBuilder(MAX_PATH);
    GetCORSystemDirectory(sb, MAX_PATH, ref MAX_PATH);
    return sb.ToString();
}

All that said, if you're doing this to access the C# compiler as you say, you should investigate the Microsoft.CSharp.CSharpCodeProvider class, which provides access to instances of the C# code generator and code compiler.

Q While I was using remote desktop to access my machine at home, I accidentally ejected the CD tray. While searching for a tool to close the tray, it occurred to me that this should be possible programmatically (since all of these tools are able to do so). How can I do this from C#?

A Believe it or not, I've done the same thing multiple times, and while I'm sure nothing would happen to my computer with the CD tray sticking out, it nags at me. There are no .NET libraries in the Framework that provide this functionality, but that doesn't mean you're at an impasse. You can use interop to access the necessary functionality provided by Win32.

The Windows Multimedia API, available in winmm.dll, exposes a method mciSendString which can be used to send a command to a Media Control Interface (MCI) device, such as your computer's CD drive. To use this method in C#, I'll define an interop signature that looks like the following:

[DllImport("winmm.dll", EntryPoint="mciSendStringA",
    CharSet=CharSet.Ansi)]
public static extern int MciSendString(string lpszCommand,
    StringBuilder lpszReturnString, int cchReturn, IntPtr hwndCallback);
I can then call this with the appropriate command that tells the CD drive to close the tray:
private static void CloseCdTray()
{
    MciSendString("Set cdaudio door closed wait", null, 0, IntPtr.Zero);
}

If you have more than one CD drive, you can change the argument to MciSendString to target one in particular. For more information on the Windows multimedia API, see About MCI.

Q I'm writing a C# application to monitor various happenings on my system and I'd like to query for the current CPU consumption. Is this possible?

A There are probably a few ways to retrieve the data you need, but one easy way is to use performance counters. The System.Diagnostics.PerformanceCounter class allows easy access to system counters as well as to counters you've created for your own applications. The "% Processor Time" counter in the "Processor" category allows you to retrieve the current CPU consumption percentage as a float ranging between 0 and 100.

If you're going to be retrieving the usage multiple times, it makes sense to store the PerformanceCounter object so that it doesn't need to be recreated every time you want to retrieve the value, as shown in the code listed in Figure 4.

Figure 4 Querying CPU Usage (Close Figure 4)
private static PerformanceCounter _cpuCounter;
private static float CpuUsage
{
    get
    {
        if (_cpuCounter == null)
        {
            _cpuCounter = new PerformanceCounter(
                "Processor", "% Processor Time", "_Total", true);
        }
        return _cpuCounter.NextValue();
    }
}

This code delay-loads the PerformanceCounter (so that it isn't instantiated if it's never used), caches it so that you only need to create it once, and then on each request uses the cached counter object to retrieve the next value of the performance counter—that is, the current CPU usage value.

Q I've heard that XmlSerializer generates code dynamically to serialize and deserialize my classes. Is there any way to take a look at the code that it generates?

A By default, XmlSerializer deletes the code it generates as soon as it is compiled into an assembly. However, the nice developers at Microsoft used a trace switch to allow you to configure this behavior. If you add the following code to your machine.config

<system.diagnostics>
    <switches>
        <add name="XmlSerialization.Compilation" value="4"/>
    </switches>
</system.diagnostics>
XmlSerializer will leave behind in your temp directory key files that it generated, including the source code for the custom serializer class. On a machine running Windows XP, the temp directory is available at %TEMP%. After running your app that uses XmlSerializer, inspect the temporary files for a recently created .cs file.

Send your questions and comments to netqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Jay Bazuzi, Eric Gunnerson, Gerrard Lindsay, and Peter Wieland


Security Briefs: Beware of Fully Trusted Code

Security Briefs
Beware of Fully Trusted Code
Keith Brown

Contents
Does the CLR Verify All Code Before Running It?
Are Private Methods Really Private?
Does IStackWalk.Deny Have Teeth?
Interlude: AppDomain Security Policy
Do AppDomains Have Teeth?
Does Fully Trusted Code Have Any Limitations?
Conclusion

The vast majority of managed applications run with full trust, but based on my experience teaching .NET security to developers with a broad range of experience, most really don't understand the implications of fully trusted code. So I've pulled together a number of examples where fully trusted code can skirt around common language runtime (CLR) security features, starting each with a question that seems to have an obvious answer.

Does the CLR Verify All Code Before Running It?

Lots of security features in the CLR depend on type safety. As a simple example, consider private members of a class. The CLR restricts access to them at run time, but it can only do so if the caller uses legitimate means to try to access those members. Consider the following unmanaged C++ class definition:

class DiskQuota {
private:
  long MinBytes;
  long MaxBytes;
};

I've chosen unmanaged C++ here because it shows the types of fraud that can be perpetrated when type systems are violated:

void EvilCode(DiskQuota* pdq) {
   // use pointer arithmetic to index
   // into the object wherever we like!
   ((long*)pdq)[1] = MAX_LONG;
}

By using pointers and unsafe casts, the attacker is able to access and even modify private members of a class. There is no runtime protection against this in unmanaged code. However, the CLR was designed to detect this sort of type system abuse during just-in-time (JIT) compilation (unsafe pointer arithmetic in this particular case) and raise an exception. A buffer overflow is another example of a type system violation, one that can lead to nasty security holes. The CLR closes all of these holes by ensuring the type safety of all code, right?

Yes, unless you shoot yourself in the foot by granting the assembly too much trust. Any assembly can mark itself with a flag, technically called a permission request, which tells the CLR to skip verification. Here's an example in C# that does just this:

// evilcode.cs
using System.Security.Permissions;

[assembly: SecurityPermission(
   SecurityAction.RequestMinimum,
   Flags=SecurityPermissionFlag.
            SkipVerification)]

// your evil code goes here

You may even end up with this permission request on your assemblies without knowing it. If you use the /unsafe switch to compile C# code that uses pointers, the compiler silently adds this permission request. If you build an assembly in managed C++, the same thing happens by default, although this behavior can be changed with new compiler flags in the next version of C#, code-named "Whidbey." When the CLR successfully loads an assembly with a SkipVerification permission request, it will skip the type safety verification that normally happens during JIT compilation. The Framework Class Library uses this feature; even MSCORLIB.DLL, the core .NET Framework assembly, makes this permission request. When making an omelet, some eggs must be broken, or so the saying goes.

What you have to decide is which assemblies you trust with the power to ignore the type system and which you don't. Your preference is expressed via .NET security policy. Any assemblies that run with FullTrust, which by default includes the .NET Framework itself as well as assemblies installed on the local hard drive, will be allowed to skip verification upon request. Assemblies downloaded from the network will be denied this request by the default policy.

I don't know about you, but knowing this makes me want to run every assembly I can with partial trust.

Are Private Methods Really Private?

In my last example, I showed how fully trusted code could skip verification and use pointer arithmetic to read or write private member variables of any class. Well, it turns out that you don't need pointers to do this sort of thing. Using reflection is another way to achieve the same result (see Figure 1).

Figure 1 Using Reflection to Read/Write Variables (Close Figure 1)
using System;
using System.Reflection;

class EvilCodeWithFullTrust
{
  static void CallPrivateMethod(object o, string methodName) {
    Type t = o.GetType();
    MethodInfo mi = t.GetMethod(methodName,
       BindingFlags.NonPublic |
       BindingFlags.Instance);
    mi.Invoke(o, null);
  }
  static void Main() {
    CallPrivateMethod(new NuclearReactor(), "Meltdown");
    }
}

Surely the common language runtime will protect your private methods by throwing an exception when another class tries to call them, even via reflection, right? Once again, the only way that you can protect your classes against this sort of funny business is via .NET security policy. The important permission in this case is ReflectionPermission. Fully trusted code has this permission in the dose required to reflect on private members.

Think about what this means. Say you ship an assembly that includes sensitive private methods that are for your internal use only. What's to keep your client from calling those methods directly, perhaps passing malformed input that could cause your methods to malfunction and behave dangerously? Nothing in this case, because you have no control over your client's security policy. But wait! Perhaps you've heard of a feature called a strong name LinkDemand. The Framework Class Library uses it, so it's got to be good. Here it is:

public class NuclearReactor {
  [StrongNameIdentityPermission(
    SecurityAction.LinkDemand,
    PublicKey="002400000...")]
  private void Meltdown() {
    // calling assembly must have specified public key!
  }
}

Now the previous trick using reflection won't work. Nobody can call Meltdown except assemblies that have the specified public key. A link demand directs the CLR to check the assembly linking to the method at JIT time to ensure that it can satisfy the demand. By using a StrongNameIdentityPermission, I am requiring that the caller has a particular public key.

Unfortunately, fully trusted code can get around this permission demand as well, in more than one way. The simplest option is to turn off code access security (CAS) entirely with the following command-line option:

caspol -s off

You have to be an administrator to do this, thank goodness (and even if you are one, never do it!) In fact, there are so few good reasons for this to exist that the Whidbey team is considering removing it from the next release. But even a non-administrator can write the following program to effectively do the same thing if the application has a high level of trust:

using System.Security;
class EvilCodeWithFullTrust {
  static void Main() {
    SecurityManager.SecurityEnabled = false;
    // now call Meltdown via reflection!
  }
}

These mechanisms for turning off CAS have the same effect, although the former turns it off on the entire machine while the latter only affects a single process—although there are no guarantees that it will work reliably. See the documentation for more information. The result is that all demands succeed, even link demands like the one shown earlier. The second example demonstrates yet another very dangerous code access permission, namely a SecurityPermission called ControlPolicy. Any code that's been granted this permission can turn off code access security for the process with a single line of code. Fully trusted code has all permissions, including this one.

But maybe you don't want to stoop to this level. It turns out that you don't have to. Just delay-sign your assembly with the public key being demanded and turn off strong name verification for that key. See my MSDN® article on strong names for more details (Security Briefs: Strong Names and Security in the .NET Framework). Microsoft uses the strong name LinkDemand in core framework libraries like MSCORLIB.DLL, but it's never stopped me from calling one of their private methods in my investigations of the CLR.

I hope I'm making it clear that it's impossible to give someone an assembly and restrict them from calling certain methods or accessing certain variables. Unless, of course, you control the machine on which they are running your code, in which case you can force them to run in a partially trusted environment. Lots of security guarantees disappear in a fully trusted environment.

Does IStackWalk.Deny Have Teeth?

It's a little-known fact that you can choke down the permissions on your call stack by using a deny modifier:

using System.Security;
using System.Security.Permissions;

class WellMeaningCode {
  public void CallPlugIn(EvilCode plugin) {
    // put a CAS modifier on the stack that denies all file system access
    new FileIOPermission(
      PermissionState.Unrestricted).Deny();
    plugin.DoWork();
    CodeAccessPermission.RevertDeny();
  }
}

The idea here is that even though your WellMeaningCode may be fully trusted, you may not trust the plug-in or some other third-party extension, so you reduce your permissions temporarily before making calls to that plug-in. But if the plug-in is granted full trust by security policy, this will only stop a terribly naive author of evil plug-in code. Here's an implementation that gets around the deny:

class EvilCodeWithFullTrust {
  void DoWork() {
    new PermissionSet(
      PermissionState.Unrestricted).Assert();
    // happily access the file system
    // regardless of the caller's deny!
      }
}

Fully trusted code is very slippery! Assert is another stack modifier that effectively cancels out the Deny. But what code is allowed to Assert? Any code that's been granted a SecurityPermission called Assertion. And, of course, that includes all fully trusted code. So trying to use a Deny (or a PermitOnly for that matter) in assembly A to constrain assembly B is pointless if B has been granted FullTrust. To give Deny some teeth, run it with partial trust.

Interlude: AppDomain Security Policy

If you are familiar with the .NET Framework Configuration Tool, you have probably seen three security policy levels: Enterprise, Machine, and User. But there's a fourth level that's quite useful for dynamically sandboxing code like the plug-in that I mentioned earlier. I don't have enough space in this column to drill into this topic in much depth, but I wanted to point it out to you in case you've never seen it before.

The fourth level is associated with the AppDomain, and by default it adds no restrictions to the set of permissions granted by the Enterprise, Machine, and User policy levels. But if you want to sandbox a piece of code, say a plug-in that might be installed on the local machine, it's possible to do this by creating a second AppDomain and calling AppDomain.SetAppDomainPolicy. The policy itself can be loaded from a file via SecurityManager.LoadPolicyLevelFromFile. If you think you might want to use this technique, the best reference I've seen on the topic so far is in a book written by Brian LaMacchia, Sebastian Lange, and some other folks at Microsoft called .NET Framework Security (Addison-Wesley, 2002). Pay special attention to Chapter 10 on hosting.

Do AppDomains Have Teeth?

It wouldn't surprise me if the entire notion of an AppDomain was necessitated by ASP.NET, which uses them to isolate Web applications within a process. If you're not familiar with the AppDomain, it is conceptually similar to a process, but has a much lighter weight. When a process hosts the CLR, it always has a default AppDomain where all managed types are loaded. But by creating multiple AppDomains, a process can play host to several different applications, with a high degree of confidence that types in one AppDomain cannot wreak havoc on types in any other AppDomains in the process. Sharing object references between AppDomains requires the use of remoting, and it doesn't happen automatically. If you and I are in separate AppDomains and I want to touch your objects, you've got to pass a reference to me. I can't poke around without being invited. That's the theory at least.

In practice, assemblies that run with full trust within an AppDomain are really not constrained by the AppDomain boundary. Fully trusted assemblies don't have to be type safe. Heck, if I wanted to, I could simply take a pointer and scrape through the entire virtual address space of the process looking for interesting data.

Another thing to keep in mind is that fully trusted assemblies are allowed to call to native code via a SecurityPermission called UnmanagedCode. This makes dipping into someone else's AppDomain really easy. Say you and I are running in separate AppDomains in an ASP.NET worker process. If I want to inject my own code into your AppDomain, it's pretty easy if I'm fully trusted. All I need is a reference to your AppDomain and I can use something as simple as AppDomain.DoCallBack to send some attack code into your domain. The code in Figure 2 shows a somewhat contrived example where I cheat a bit by handing the attacker a reference to the victim's AppDomain.

Figure 2 Executing Code in Another AppDomain (Close Figure 2)
// YourCode.cs --> YourCode.dll
using System;

public class Init : MarshalByRefObject  {
    // entry point for victim's AppDomain
    public Init() {
        Console.WriteLine("YourCode is running in {0}",
            AppDomain.CurrentDomain.FriendlyName);
    }
}

public class NuclearReactor {
    // some function we don't want an attacker to call
    private static void Meltdown() {
        Console.WriteLine("Reactor meltdown!");
    }
}

public class SecretData {
    private static string TheData = "555-55-5555";
}

// MyCode.cs --> MyCode.dll
using System;
using System.Reflection;

public class Init : MarshalByRefObject {
    // entry point for attacker's AppDomain
    public Init(AppDomain target) {
        Console.WriteLine("MyCode is running in {0}",
            AppDomain.CurrentDomain.FriendlyName);
        Console.WriteLine("Injecting code into {0}...",
            target.FriendlyName);

        // here's how we inject the code
        target.DoCallBack(
            new CrossAppDomainDelegate(InjectedAttackCode));
    }
    public static void InjectedAttackCode() {
        Console.WriteLine("InjectedAttackCode in {0}",
            AppDomain.CurrentDomain.FriendlyName);

        // time to melt down the nuclear reactor using reflection
        Type t = Type.GetType("NuclearReactor, YourCode");
        MethodInfo mi = t.GetMethod("Meltdown",
            BindingFlags.Static | BindingFlags.NonPublic);
        mi.Invoke(null, null);

        // steal secret data from the victim
        t = Type.GetType("SecretData, YourCode");
        FieldInfo fi = t.GetField("TheData",
            BindingFlags.Static | BindingFlags.NonPublic);
        Console.WriteLine("Found a secret: {0}", fi.GetValue(null));
    }
}

// host.cs --> host.exe
using System;
using System.Reflection;

class Host {
    static void Main() {
        AppDomain victim = AppDomain.CreateDomain("Victim's Domain");
        AppDomain attacker = AppDomain.CreateDomain("Attacker's Domain");

        victim.CreateInstance("YourCode", "Init");
        attacker.CreateInstance("MyCode", "Init", false,
          BindingFlags.Public | BindingFlags.Instance,
          null, new object[]{victim}, null, null, null);
    }
}

How can one AppDomain pilfer a reference to another AppDomain in order to play these sorts of games? This can be accomplished by calling out to unmanaged code. The CLR provides an unmanaged hosting API, and by writing a DLL in C++ and calling out to it from managed code, you can obtain references to all AppDomains in the process.

What it comes down to is this: AppDomains can provide no security when all the code is fully trusted. You need a partial-trust environment to give AppDomains any real teeth. Partial trust requires verification of managed code and restricts access to native code. By default, ASP.NET server applications run with full trust because all the code is installed right there on the server. In fact, prior to version 1.1 of the .NET Framework, it wasn't even possible to run ASP.NET applications with anything less than full trust. Nowadays you can easily force Web applications on the server to run with partial trust via machine.config:

<configuration>
  <system.web>
    <trust level='Medium'/>
  </system.web>
</configuration>

But frankly I doubt you'll find many ISPs doing this because not many developers have the knowledge and the patience necessary to write partially trusted Web apps. I'm hopeful this will change though. For now, if you decide to host your ASP.NET app in a shared environment, you should ask lots of questions about how your Web app will be isolated. You should prefer Windows Server™ 2003 to Windows® 2000 because ASP.NET only supports a single worker process in the latter case. Be sure the ISP gives you your own private worker process with its own dedicated Windows user account. Some ISPs running Windows 2000 will isolate Web apps from one another using virtual machines. Just be sure to ask so you know what you're getting into. Of course, your best bet is to host your app on your own dedicated Web server.

Does Fully Trusted Code Have Any Limitations?

After all this doom and gloom you might be getting the impression that there's nothing that fully trusted code can't do. But remember, the CLR still runs on top of an operating system that should have its own security constraints. When I talk about fully trusted code, what I am really saying is that the code can do anything the user running it is allowed to do. Say Bob runs a managed application that's installed locally on his machine. As far as the CLR is concerned, by default that application runs with full trust. But if Bob's logon to Windows prevents him from accessing a particular file, the managed application will have the same constraints. Trust in the CLR is a sliding scale whose upper bound, what's called full trust, reaches the level of privilege of the user running the application. For server applications like ASP.NET, this upper bound depends on the security account chosen for the server process, which is one reason that I prefer Windows Server 2003; it allows me to run each ASP.NET application in separate processes, each with any level of privilege I want.

Conclusion

The goal of this column was to demonstrate that many of the security features of the CLR can only be enforced in a partial-trust environment. While the notion of full trust might seem obvious to some, I've reviewed plenty of designs that make assumptions about CLR security that simply don't fly in a full trust scenario. If you compare the CLR's built-in security to Windows built-in security, running with full trust is akin to running as SYSTEM. Fully trusted code can get around all of the CLR's built-in security features. That's why it's called fully trusted—it must be trusted to do the right thing. SYSTEM can get around any security constraint in Windows, which is why code running as SYSTEM must be trusted.

My challenge to you is to learn to write code that runs in a partial-trust environment. Ivan Medvedev has some good advice at Writing managed code for semi-trusted environment. If what I saw at the PDC is any indication, this will be an important skill for the next version of Windows, code-named "Longhorn," where even locally installed code may not run with full trust. Remember the principle of least privilege, and design and code with it in mind.

Send your questions or comments for Keith at briefs@microsoft.com.

Keith Brown is an independent consultant specializing in application security. He authored the book Programming Windows Security (Addison Wesley, 2000) and is writing a new security book for .NET programmers. Read the new book online at http://www.develop.com/kbrown.


Bugslayer: .NET Internationalization Utilities

Bugslayer
.NET Internationalization Utilities
John Robbins

Code download available at: Bugslayer0404.exe (244KB)
Contents
.NET Resource Handling
Building Real-world Internationalization DLLs
Windows Forms Internationalization
Internationalization Helper Programs
Tips

As you saw in last month's column, .NET internationalization support is excellent and allows you to move your application to a world audience quite easily. Before you jump into this month's discussion, you may want to go back and read the March column.

This month, I'll discuss numerous utilities to make real-world .NET internationalization easier to achieve in a team development environment. The first is a set of batch files that can ease the pain of building your internationalized resources. The second utility is used to translate just the string resources so you don't have to distribute your entire source code tree. The final utility, Pseudoizer, is a great testing tool that allows you to do internationalized resource testing even if you are a monolingual American. Before I jump into the tools, I need to discuss .NET resource handling because it sometimes confuses developers the first time they see it.

.NET Resource Handling

Back in the days of Win32®, you were on your own to devise a resource-handling scheme that automatically loaded internationalized resources. With the .NET Framework, Microsoft built a very nice scheme, called hub-and-spoke, directly into the Framework itself. The hub represents the parent assembly that contains the code, non-localizable resources, and the neutral or default language localizable resources. There is only one hub. The spokes (there's one for each language) are satellite assemblies that have no code and contain only the localized resources for a language.

Describing the exact steps the Framework takes to load the appropriate satellite assembly could take an entire column on its own. Fortunately, there is a good explanation of what happens when you request a satellite DLL in the "Resource Fallback Process" section in the MSDN® Library. You should also read the section on "Versioning Support for Satellite Assemblies". I found it extremely helpful to draw out the steps on paper as I understood them for the Resource Fallback Process section to ensure I could see how the Framework accomplished the loading. The beauty of the resource fallback process scheme is that the Framework will go to heroic efforts to find each resource you request. This means that if you forget to translate a particular string, the user will see at least the default language of the hub assembly.

The documentation does not make clear the physical disk layout for loading satellite assemblies from directories below the parent assembly. Figure 1 shows the directory structure for the InternationalizationDemo program. The main assembly directory contains InternationalizationDemo.exe with code and default resources for the application. Under the main directory is a directory for each supported locale.

Figure 1 Directory Layout for Resource Loading
Figure 1 Directory Layout for Resource Loading

The one interesting directory is ES, which contains the Spanish language resources that are not culture specific. If the locale is Spanish as spoken in Mexico (es-MX), the ResourceManager will look in the .\es-MX directory for the resources from the satellite DLL. If .NET does not find a requested resource, it looks in the satellite DLL in the .\es directory, which contains the Spanish language, culture-independent resources. Falling back to the base language in this manner allows you to reuse common language resources. As you would expect, if the resource is not found in the satellite DLL in .\es, the system takes a look in the main assembly, InternationalizationDemo.exe, for the resource as that is considered to be the default resource.

Before I get too far into the discussion of resources, I want to talk about the different types of files you can put your raw resources into and how to convert them into resource files. I'll also add a few words of wisdom. You can use two file formats for your resources. The first is the text file in which all the strings in the file are in the form of <value>=<string> with one per line. The format I prefer is the XML-based ResX file.

You've probably always seen ResX files as part of Windows® Forms applications, but there's nothing stopping you from using ResX files for any of your resource needs. What makes ResX files so much better than text files is that they let you store any type of resource—strings, graphics, or any custom type you want. Since they are XML files, you can easily edit them with the built-in XML editor in Visual Studio® .NET.

Using either format, it's a two-step process to turn the source files into an assembly. RESGEN.EXE, the resource generator, turns the source into a .RESOURCES file. The second step is to embed the .RESOURCES file into an assembly with AL.EXE, the Microsoft® intermediate language (MSIL) assembler. The good news is that the work of embedding your resources is automatically handled by Visual Studio .NET for your main assembly. I'll talk about building those specific cultural resources in a moment.

If you've read the documentation, you might be wondering why I haven't talked about using .RESOURCES files directly by your applications. While it's possible, you're strongly encouraged not to do so. The main problem is that the .RESOURCES file is just a binary file, so there is no version information associated with the file, and you can't strongly sign the file. .RESOURCES files will cause file locking issues with ASP.NET applications and can make XCOPY deployment more difficult.

Adding a ResX file to your projects is as simple as right-clicking on the project, selecting Add New Item from the context menu, and selecting Assembly Resource File as the item type. The name of the file is important, as that will form part of the name you'll need to pass to the ResourceManager class to load resources. Once you add the ResX file, you'll see that its Build Action is set to Embedded Resource. Any resources you put in the new ResX file will be included in the main assembly in your project. One problem with ResX files added this way into Visual Studio .NET is that the internal dependency checking doesn't always notice that the edited ResX file has changed for a normal build. If you're wondering what happened to the resources you just added, you can do a rebuild to see your changes.

Once you've got the resources embedded into your assembly, it's a simple matter of allocating an instance of the ResourceManager class to do the loading. Figure 2 shows part of the InternationalizationDemo program—the part that loads internationalized greetings based on the UI culture. If you need to load types other than strings, you can use the ResourceManager.GetObject method. Note that in Figure 2 the StartUp.ApplicationUICultureInfo is an internal object, a property that retrieves the CultureInfo class that the whole application is supposed to use. This is a trick to allow an application to display and use a different culture than the default. Keep in mind that ResourceManager instances are not inexpensive to create, so you'll probably want to create one instance that you can share or batch load the resources you'll need.

Figure 2 ResourceManager in Action (Close Figure 2)
private void MainForm_Load(object sender, System.EventArgs e)
{
    // Start up the timer.
    timerTicker.Tick += new EventHandler ( TimerTickEvent ) ;
    // Force the first display data in the date and time.
    TimerTickEvent ( null , null ) ;

    // Fill out all the data on the form using the UI culture
    // selected at startup.
    labelUILanguage.Text = StartUp.ApplicationUICultureInfo.NativeName ;

    labelCultureLanguage.Text = StartUp.ApplicationCultureInfo.NativeName ;

    // Get the string resources out of the internationalized greetings
    // string table.
    ResourceManager rm =
        new ResourceManager ( "InternationalizationDemo.Greetings" ,
                              Assembly.GetExecutingAssembly ( )     ) ;

    // Technically, if you've set the Thread.CurrentThread.CurrentCulture,
    // you don't need to pass the IFormatProvider derived class, but I
    // do it here to make FxCop happy.  Notice that since these values
    // are being displayed in the listbox, I'm using the *UI* culture.
    String str = rm.GetString ( "FormalMan"                      ,
                                StartUp.ApplicationUICultureInfo  ) ;
    listboxGreetings.Items.Add ( str ) ;

    str = rm.GetString ( "FormalWoman"                     ,
                          StartUp.ApplicationUICultureInfo  ) ;
    listboxGreetings.Items.Add ( str ) ;

    str = rm.GetString ( "InformalMan"                    ,
                         StartUp.ApplicationUICultureInfo  ) ;
    listboxGreetings.Items.Add ( str ) ;

    str = rm.GetString ( "InformalWoman"                   ,
                         StartUp.ApplicationUICultureInfo   ) ;
    listboxGreetings.Items.Add ( str ) ;
}

Building Real-world Internationalization DLLs

While I've gone through the gyrations necessary to get ResX files added to the projects and the strings loaded, now I'll describe the steps necessary to build separate satellite assemblies that will contain the internationalized resources in a clean and reproducible manner. While you could create a separate project for each internationalized satellite assembly, that would be a little tedious. I wanted a way to build these assemblies that would work for real-world build systems. In thinking about how I wanted to tackle the problem, I ran across an article by Satya Komatineni where he presented a batch file to call RESGEN and AL for ResX files. I took Satya's idea, added lots of error handling, generalized some approaches, and added new features to arrive at the BuildIntl.CMD, BuildIntlHelper.CMD, and IntlRes.CMD batch files.

Integrating BuildIntl.CMD into your build process should be simple. Put your internationalized ResX files for a particular language/region into a directory that follows the RFC 1766 naming convention. For example, my InternationalizationDemo program in the .\InternationalizedDemo directory has four directories, .\InternationalizedDemo\ES, .\InternationalizedDemo\ES-MX, .\InternationalizedDemo\EN-AU, and .\InternationalizedDemo\EN-GB, for the four internationalized sets of resources. The three batch files that make up BuildIntl.CMD go into the directory above the named language/region directories. In the case of InternationalizedDemo, they would go into the .\InternationalizedDemo directory.

Once you've created the international directories and the appropriate ResX files are set, copy the three batch files into the root directory. You'll only have to edit the file BuildIntl.CMD to put in your particular options (see Figure 3 for an example). The first items you may want to change are the output directories for debug and release builds. Search for OUTDIR and you'll see that they are set depending on the command-line option passed to BuildIntl.CMD. The second items to change are the internationalized directories to process for your project. Simply add them to the FOR statement's list, each separated by a space. The last step is to put the name of your assembly between the %%d and the %OUTDIR% on the FOR statement line. Once you've saved your changed BuildIntl.CMD, pop open a Command Prompt and ensure that the RESGEN.EXE and AL.EXE files are both in the path by executing <Visual Studio .NET Install Dir>\Common7 \Tools\VSVARS32.BAT.

Figure 3 BuildIntl.CMD (Close Figure 3)
@ECHO OFF

@REM ------------------------------------------------------------------
@REM Takes care of compiling all resources for the
@REM Internationalization Demo program
@REM
@REM The only valid parameters are debug or release.
@REM ------------------------------------------------------------------

@REM Any environment variables are local to this batch file.
SETLOCAL

@REM - You may want to change the OUTDIR settings depending on your
@REM - build.
@REM Check the required command-line option.
SET OUTDIR=
IF /i "%1"=="debug" SET OUTDIR=..\bin\debug\
IF /i "%1"=="release" SET OUTDIR=..\bin\release\
IF "%OUTDIR%"=="" GOTO InvalidArg

@REM - Add your RFC1766 directories in parenthesis
@REM - Change InternationalizationDemo to the name of your assembly
@FOR %%d IN (en-AU es es-MX en-GB) DO CALL BuildIntlHelper.CMD %%d
InternationalizationDemo %OUTDIR%
GOTO End

:InvalidArg
@ECHO BuildIntl.CMD - John Robbins - (john@wintellect.com)
@ECHO Usage: BuildIntl [debug][release]
@ECHO   debug   - Build resources into debug directory
@ECHO   release - Build resources into release directory

:End

As you can see from Figure 3, BuildIntl.CMD simply loops through your internationalized directories and calls the BuildIntlHelper.CMD batch file. BuildIntlHelper.CMD will change to the appropriate directory and call the real workhorse batch file, IntlRes.CMD, shown in Figure 4. The main work of IntlRes.CMD is to wrap the call to RESGEN.EXE and AL.EXE to produce the internationalized satellite assembly. The first parts of the processing involve figuring out if the output directory exists and deleting any existing .RESOURCES files that might be lying around. After RESGEN.EXE is called on all the ResX files found in the directory, it's time to build up the command line to AL.EXE.

Figure 4 IntlRes.CMD (Close Figure 4)
@ECHO OFF
@REM ------------------------------------------------------------------
@REM A batch file that takes care of compiling a directory of .RESX
@REM files into an internationalized .NET satellite DLL.
@REM
@REM This batch file idea is from Satya Komatineni's article:
@REM http://www.ondotnet.com/pub/a/dotnet/2002/10/14/local2.htm?page=1
@REM John Robbins (john@wintellect.com) added all the error handling,
@REM file cleanup, ALLOPTS.TXT processing, and optional output code.
@REM
@REM RESGEN.EXE and AL.EXE are assumed to be in your path.
@REM If not run VSVARS32.BAT
@REM ------------------------------------------------------------------

@REM Any environment variables are local to this batch file.
SETLOCAL

@REM Clear out any previous error files
IF EXIST ErrorOut.txt DEL /Q ErrorOut.txt > nul

@REM Check the required command-line options.
IF "%1"=="" GOTO InvalidArg
IF "%2"=="" GOTO InvalidArg
SET LANG=%1
SET NAMESPACE=%2

@REM Check the optional command-line arg.  If it's present, add the
@REM language onto the directory and build the directory variable.
IF NOT "%3"=="" (
    IF NOT EXIST %3%LANG% (
        MKDIR %3%LANG% >> ErrorOut.txt
        IF ERRORLEVEL==2 GOTO Error
    )
    SET OUTDIR=%3%LANG%\
) ELSE (
    SET OUTDIR=.\
)

@REM Delete any .RESOURCES files in the current directory.
IF EXIST *.RESOURCES DEL /Q *.RESOURCES > nul

@REM Echo out the file names we're about to do as RESGEN does not report
@REM the file name with the error in it.
@FOR %%f IN (*.RESX) do ECHO %%f >> ErrorOut.txt

@REM Convert all the .RESX files in this directory to .RESOURCE files.
@FOR %%f IN (*.RESX) do RESGEN %%f >> ErrorOut.txt
IF NOT %ERRORLEVEL%==0 GOTO Error

@REM Get rid of any existing RESP.TXT file.
IF EXIST RESP.TXT DEL /Q RESP.TXT
@REM Ensure the type is set to LIB.
ECHO /t:lib > RESP.TXT
@REM Poke in the resources in the directory.
FOR %%f IN (*.resources) DO ECHO /embed:%%f,%NAMESPACE%.%%f >> RESP.TXT
@REM If ALOPTS.TXT is there, add that to the mix.
IF EXIST ALOPTS.TXT TYPE ALOPTS.TXT >> RESP.TXT
@REM Ensure there's a CR/LF after the ALOPTS.TXT just in
@REM case the user forgot to add it.
ECHO. >> RESP.TXT
@REM Specify the culture.
ECHO /culture:%LANG% >> RESP.TXT
@REM Add the output name.
ECHO /out:%OUTDIR%%NAMESPACE%.resources.dll >> RESP.TXT

@REM Delete the resources DLL I'm about to create.
IF EXIST %OUTDIR%%NAMESPACE%.resources.dll DEL /Q
%OUTDIR%%NAMESPACE%.resources.dll > nul

@REM Create the resource DLL.
AL /nologo @RESP.TXT >> ErrorOut.TXT
IF NOT %ERRORLEVEL%==0 GOTO Error

@REM Get rid of files no longer needed.
IF EXIST ErrorOut.TXT DEL /Q ErrorOut.TXT > nul
IF EXIST RESP.TXT DEL /Q RESP.TXT > nul
IF EXIST *.RESOURCES DEL /Q *.RESOURCES > nul
GOTO End

@REM Spit out any error output file.
:Error
IF EXIST ErrorOut.TXT TYPE ErrorOut.txt
IF EXIST ErrorOut.TXT DEL /Q ErrorOut.TXT > nul
GOTO End

@REM Show the usage.
:InvalidArg
@ECHO IntlRes.CMD - John Robbins - (john@wintellect.com)
@ECHO Invalid command-line arguments
@ECHO Usage: IntlRes [language] [Application Namespace] [Version] [Output
@ECHO Dir]
@ECHO   [language]              - The full language for the resources
@ECHO                             See the list in the CultureInfo class
@ECHO                             overview. (Required)
@ECHO   [Application Namespace] - The application namespace for all the
@ECHO                             resources. (Required)
@ECHO   [Output Dir]            - The optional output directory for the
@ECHO                             output DLL.  If specified, the output
@ECHO                             DLL is placed in
@ECHO                             [Output Dir]\[language]
@ECHO                             Trailing backslash IS REQUIRED!
@ECHO If you want to set additional command-line options to AL, (such as
@ECHO version, signing, and so on) put the options in a file called
@ECHO ALOPTS.TXT in the same directory as the resources.  If the file is
@ECHO present, this batch file picks them up and ensures they are
@ECHO included.

@REM The end of the program.
:End

Since AL.EXE can take a response file listing the options, I chose that option because I wanted the user to be able to specify their own options to AL.EXE. You can do that by including a file called ALOPTS.TXT in the same directory as the ResX files. For the most part, you'll want to include the version number information as well as the strong name key file in your own ALOPTS.TXT files. See the InternationalizationDemo for examples of using ALOPTS.TXT files (available in the download at the link at the top of this article). Always remember to set the version information in satellite assemblies to the same as the main assembly so all satellite assemblies will share the main assembly's strongly named key file. After creating the full response, AL.EXE builds the satellite assembly that will have the appropriate output name of <Assembly Name>.resources.DLL. The rest of the processing in IntlRes.CMD takes care of cleaning up any temporary files used by the build.

Windows Forms Internationalization

Now that you have a way to cleanly build your international resources for both ASP.NET and Windows Forms applications, I want to spend a little time going over Windows Forms internationalization. The most important issue with Windows Forms internationalization is setting a form's Localizable property to true in the Properties window. Setting Localizable to true causes two important additions to the form. The first is to reflect any changes to the code generated in the InitializeComponent method. When Localizable is set to false, here is the code that's generated. You can see the hardcoded strings in the code:

private void InitializeComponent()
{
    this.AutoScaleBaseSize = new System.Drawing.Size(5, 13);
    this.ClientSize = new System.Drawing.Size(292, 266);
    this.Name = "Form1";
    this.Text = "Example Form";
}

Setting the Localizable property to true causes the code in Figure 5 to be generated. The code is quite different and uses the ResourceManager class to do all the work of loading anything about the form. As you can probably guess, the second action performed when setting Localizable to true is to put all hardcoded values into the ResX file associated with the form.

Figure 5 Localizable Set to True (Close Figure 5)
private void InitializeComponent()
{
    System.Resources.ResourceManager resources = new
    System.Resources.ResourceManager(typeof(Form1));
    //
    // Form1
    //
    this.AccessibleDescription =
         resources.GetString("$this.AccessibleDescription");
    this.AccessibleName = resources.GetString("$this.AccessibleName");
    this.AutoScaleBaseSize =
         ((System.Drawing.Size)(resources.GetObject
         ("$this.AutoScaleBaseSize")));
    this.AutoScroll = ((bool)(resources.GetObject("$this.AutoScroll")));
    this.AutoScrollMargin =
         ((System.Drawing.Size)(resources.GetObject
         ("$this.AutoScrollMargin")));
    this.AutoScrollMinSize =
         ((System.Drawing.Size)(resources.GetObject
         ("$this.AutoScrollMinSize")));
    this.BackgroundImage =
         ((System.Drawing.Image)(resources.GetObject
         ("$this.BackgroundImage")));
    this.ClientSize =
         ((System.Drawing.Size)(resources.GetObject("$this.ClientSize")));
    this.Enabled = ((bool)(resources.GetObject("$this.Enabled")));
         •••
}

To work on a different language version of a particular form, you just need to set its Language property to that particular language. That will copy the default ResX file to two separate ResX files. For example, if you choose "Spanish (Mexico)" in the Language property for Form1, that will create Form1.es.ResX and Form1.es-MX.ResX. These two files will be added to the project, though you won't see them by default. You'll need to click the Show All Files button in Solution Explorer to see them under the Form1.CS entry. When you go to build your application, you'll see two new directories created under the output directory. In the case of the example using "Spanish (Mexico)" you'll see .\ES and .\ES-MX directories.

While it's nice that Visual Studio .NET does provide for internationalization in your Windows Forms, it has a few drawbacks. First, because Visual Studio .NET takes over the building of the internationalized satellite DLLs, you have no way to add your own resources to the mix. Consequently, this makes general internationalization much more difficult. Second, because Visual Studio .NET produces the ResX files, you'll have to give all your source code to the translators so they can translate the UI. While that might be fine if you have the translators on staff to do the work, most development shops outsource the translation work and won't want to expose their intellectual property.

These issues, in my opinion, make the use of Visual Studio .NET for internationalization less than ideal. Fortunately, working around these issues is relatively simple. When you undertake the process, you should always set the Localizable property to true on every one of the Windows Forms that you want to internationalize. This makes the code load the resources in the ResX file. Once you have the form working and the UI is set, you can then hand off just the form's ResX file to the translator because that file will contain all the information about the form. Your translators must have the Framework SDK installed, for they will use WinRes.EXE to open the ResX file. When you run WinRes.EXE, you'll see that it is essentially the Windows Forms editor from Visual Studio. Your translators can translate all the strings and save the file for the appropriate language/region, which will add the RFC 1766 name between the form name and the ResX extension. Once the translation is complete, they'll ship the translated ResX file back to you.

There are a couple of caveats about using WinRes.EXE. The first issue with WinRes.EXE is that the ResX files it produces are no longer editable by Visual Studio .NET, but there's nothing you can do about that. Second, WinRes.EXE does not just limit changes to strings in the form. The translator could reposition or resize controls on the form. For languages such as Chinese that require special fonts and resizing, that might be necessary. However, most development teams don't want their translators rearranging their forms. The ideal solution would be for WinRes.EXE to support an editing mode that only allowed changes to string data. It should be noted, however, that unless great care has been taken to design a UI that can withstand localization, the quality of the localized software will suffer if localizers aren't given the ability to adjust sizing and positioning. Doing so could force localizers to compromise on language or could result in a slew of sizing bugs which will then need to be found and fixed. That said, there is still some value in such a feature.

Thinking about the problem, I came up with a program called ResXInternational that runs through a ResX file and produces an output ResX file that contains just the core string and a few other international values. By running ResXInternational across the files returned from the translator, you'll have just the core translated strings and you'll avoid accidental UI changes. Since your default resource set will have all the other pieces of the form, everything will work just great. You can find ResXInternational in this month's source code distribution. Now that all the dealings with ResX files are out of the way, I want to turn to some implementation highlights of ResXInternational and some of the other programs that I wrote or enhanced for this column.

Internationalization Helper Programs

As you've already seen, there are a couple of places where I found that writing a program can make your internationalization chores easier. The first program I tackled was ResXInternational. Originally, I thought I was going to have to do quite a bit of work to read in and process the XML-based ResX format. A quick search of the MSDN Library yielded the ResXResourceReader and ResXResourceWriter classes for reading and writing ResX files. As you look over the source code to ResXInternational, you'll see just how simple it really is to manipulate resource files. In all, it took me about 30 minutes to hammer out ResXInternational.

In working on my internationalized application, which needed a huge number of internationalized string resources, I found that the Visual Studio .NET XML editor worked pretty well. Its only downside is that it does not provide a way to add icon and graphic objects to the ResX file. I was all set to start working on a small resource editor when I ran across the ResEdit sample from the Framework SDK. When I fired it up, I realized it was an excellent ResX file handler all on its own. I was using it for a while and ran into a bug in which it had trouble opening certain files. I tracked it down, fixed the bug, and created a full Visual Studio .NET solution for the project, which it did not have originally. The problem turned out to be that ResEdit did not handle entries that had a null value. As those types were empty strings, I added a check in ResHolder.cs to change the value from null to String.Empty. While I was in the code, I also updated the File Open dialog to look for ResX files first instead of .RESOURCES. ResEdit isn't perfect; it groups by type (such as string and anchor), instead of by control or form values. Nevertheless, in all the resource files I've edited, that hasn't been much of a problem.

The Pseudoizer program is something I've wanted for a very long time. When it comes to testing localization, it's generally quite difficult because unless the testers and developers are multilingual and are paying very close attention, it's easy for untranslated strings to slide through. One way to find internationalization problems is to use a technique called pseudo-localization. It involves translating the resources and other UI pieces into something readable, but drastically different from normal text. For example, you could replace every "a" with an "â". Pseudo-localization also adds extra padding characters to the ends of strings. The types of errors that pseudo-localization helps you find are: hardcoded strings that need internationalization, strings that shouldn't be translated, non-Latin characters, and errors handling longer language strings.

While you could manually run through all your string resources and convert each character by hand, the word tedious doesn't even begin to describe the agony. What I wanted was a program that would read in my resources and allow me to choose which strings to translate. This was important as some strings included in translated resources, such as the menu separator string, do not need to be pseudo-translated.

At this point, it's probably easiest if I show you a pseudo-translated UI. Figure 6 shows the InternationalizationDemo UI with all resource strings in pseudo-English form. All strings are surrounded by "[]" so that you can see where each begins and ends. As you can see in Figure 6, it's relatively simple to read the strings that have be pseudo-translated and it makes finding the non-translated strings a snap. The non-translated strings in the figure are from the .NET Framework itself, so they cannot be pseudo-translated.

Figure 6 Pseudo-translated UI
Figure 6 Pseudo-translated UI

I poked around in .NET a bit to see if I could create my own language/region, but only the ones listed in the CultureInfo class documentation are usable. That meant I needed to pick a culture that I was going to use as the pseudo-translated culture. As you can see in Figure 6, I decided on English as spoken in the United Kingdom (en-GB) as my pseudo-translated culture. This also meant I would get date, time, and currency translations that were different from my normal US-based English. Whatever language/region you choose to override for your pseudo translation, just make sure to avoid shipping the translated resources with your application or you're going to get some angry tech support calls!

While it's now relatively easy in Windows XP to set the current language/region through the Regional and Language Options control panel applet, I set the culture manually in my debug builds. In the Pseudoizer program itself, I have debug-only code that looks at the command-line options and sets the language/region appropriately. If you have a complex multithreaded application, you may want to set an environment variable and add debug-only code that looks for it and sets the language/region in each of the threads.

To be thorough I just had to run Pseudoizer on itself. I was quite surprised to see an internationalization bug pop right out! In the About box I was using a LinkLabel control to link to my Web site. When I ran the pseudo-translated version, I noticed that the LinkArea didn't extend to the end of the control as I expected. I found that when you use a LinkLabel, the Form editor hardcodes the length of the string into the LinkArea at design time, which is a bug. I'm happy when a utility can find a bug in itself!

The last program in the source code distribution is the InternationalizationDemo I've been mentioning through this column. As the name implies, it shows off the techniques of internationalization. I translated the UI into Spanish and supported two region areas, Spanish (Mexico) and English (Australia). If you pass a valid RFC 1766 language/region on the command line, you'll see the various language-specific items change appropriately. Additionally, in InternationalizationDemo I show a technique for overriding the user locale by providing the static Startup.ApplicationCultureInfo and Startup.ApplicationUICulture, which I use to set all thread cultures and pass to all ResourceManager loads.

The tools and information provided in these two columns on internationalization should get you started and make real-world internationalization easier to achieve. As I mentioned last time, Microsoft earns 60 percent of its revenues from internationalized software so that should provide you with plenty of motivation.

Tips

Tip 61 If you're looking to see how .NET works or want to see how someone implemented a cool program, but you don't have the source, look at Lutz Roeder's Reflector. Not only does it disassemble, it's a complete decompiler that will translate MSIL into C# or Visual Basic .NET.

Tip 62 While Reflector is a standalone program, Jamie Cansdale has written a cool article in which he takes Reflector and, with some magic, converts it into a Visual Studio .NET AddIn. Not only does he make a phenomenal utility even more useful, Jamie's code is a great example of .NET Framework programming.

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in programming for .NET and Windows. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at http://www.wintellect.com.


C++ Q&amp;A: CD Burning and Device Discovery with IMAPI

C++ Q&A
CD Burning and Device Discovery with IMAPI
Paul DiLascia

Code download available at: CQA0404.exe (149KB)

Q I read your January 2004 column about getting the drive letter and about using the GetDriveType function to find out if a disk drive is a CD (DRIVE_CDROM), but how can I know if the CD drive is a recording device? Also, how do I write files to a recordable CD? Windows® XP lets you drag and drop files using Windows Explorer to write files to a CD. Can you tell me if there is a way I can do this from my program without requiring my users to purchase third-party software such as Nero or Roxio? Are there any MFC classes that do this?

Bill Fenton

A MFC doesn't have any CD-burning classes, but Windows XP provides built-in support for writing to CD. If all you want to do is copy some files and folders, you can use the shell's ICDBurn interface. If you want to record audio or have more detailed control over recorders, there's a special API, which I'll discuss shortly.

ICDBurn has three methods. HasRecordableDrive scans the system for a CD drive with write capability, returning TRUE if one is found. GetRecorderDriveLetter retrieves the drive letter of a CD drive which has been marked as write enabled. And finally, Burn instructs Windows to copy data from the "staging area" to a writable CD. The staging area is a special folder that's usually "%userprofile%\Local Settings\Application Data\Microsoft\CD Burning", but you should query for the exact name by calling SHGetFolderPath with the special code CSIDL_CDBURN_AREA. You should always use SHGetFolderPath because users can manually change the location of their burn folder in the registry or by using a tool like PowerTools/TweakUI. For example, on my machine, I set my burn path to "C:\TEMP\Burning".

I wrote a little class called CCDBurn to encapsulate ICDBurn. The constructor calls CoCreateInstance with CLSID_CDBurn, so you can just instantiate and go:

CCDBurn burner;
if (!burner.HasRecordableDrive()) {
   printf("Oops—No recordable drive!\n");
} else {
   CString dl =
       burner.GetRecorderDriveLetter();
   printf("Default Recorder drive letter =
       %s\n", (LPCTSTR)dl);
}

The drive letter is the one that has Enable CD recording on this drive checked in the drive's Recording Properties (see Figure 1). Only one drive can have this property set at any given time. Assuming HasRecordableDrive returns TRUE—that is, assuming the computer has at least one recordable CD drive—all you have to do to write files to it is copy the files to the staging area, then call Burn. Since I'm so nice, I added another method, GetBurnFolderPath, that calls SHGetSpecialFolderPath to get the burn folder path in a CString, as shown in the following line of code:

CString path = burner.GetBurnFolderPath();

Figure 1 CD Drive Recording Properties
Figure 1 CD Drive Recording Properties

What if you want to record music, find other recorders (there can be more than one), or get detailed information such as the make and model or whether it's a CD-R or CD-RW? For that, Windows XP has IMAPI. That's Image Mastering API, not to be confused with MAPI, the Messaging API used for e-mail. Gosh, it's hard to keep up with all the acronyms! IMAPI provides COM interfaces to query recorders and write data or the latest Radiohead tunes to your shiny plastic disc. Figure 2 gives the big picture overview.

Figure 2 CD Burning/IMAPI Interfaces (Close Figure 2)

Interface | System | Description
ICDBurn | shell | Simple interface for writing files to CD. Burn method copies staging area to CD.
IDiscMaster | IMAPI | Controls an IMAPI session. Open, close, enumerate, and select recorders. Burn data or music.
IDiscRecorder | IMAPI | Provides access to a single CD recorder connected to the system.
IRedbookDiscMaster | IMAPI | Provides access to audio (redbook) data format.
IJolietDiscMaster | IMAPI | Provides access to file system (Joliet) data format.
IDiscMasterProgressEvents | IMAPI | Interface you can implement to receive feedback about burn progress.

Since coping with COM can cause great consternation, I went all out and wrote a little class library, IMAPITools, to encapsulate most of the nasties. I also wrote a program, CDINFO, that shows how to use it. CDINFO displays information about CD recorders in a console window. You can download the source code from the link at the top of this article. Figure 3 shows it running. CDINFO shows that my home computer has two recorders, a Sony and a Pioneer, and my default drive is at K:. CDINFO shows how to get information about recorders, but it doesn't actually burn data. Recording isn't hard once you get the hang of IMAPI, but it's still too much for one little C++ column so I'll touch on this subject only briefly after describing the basics.

Figure 3 CDINFO
Figure 3 CDINFO

First, CDINFO creates a CCDBurn object to display the drive letter and burn path. Next, it creates a CDiscMaster to open an IMAPI session:

  CDiscMaster dm; // create IDiscMaster
  if (!dm.Open()) {
     printf("Oops: ...");
     return;
  }

CDiscMaster encapsulates the first and main IMAPI interface, IDiscMaster. Its constructor calls CoCreateInstance to create the Microsoft MSDiscMasterObj object and get its IDiscMaster interface. IDiscMaster (see Figure 4) has methods to enumerate formats and recorders, select the active recorder, and so on. The underlying object also implements two other interfaces: IJolietDiscMaster and IRedbookDiscMaster (see Figure 5).

Figure 4 IDiscMaster and IDiscRecorder (Close Figure 4)

IDiscMaster Method | Description
Open | Opens an IMAPI object
EnumDiscMasterFormats | Retrieves a format enumerator
GetActiveDiscMasterFormat | Retrieves the currently selected recorder format
SetActiveDiscMasterFormat | Sets a new active recorder format
EnumDiscRecorders | Retrieves a recorder enumerator
GetActiveDiscRecorder | Retrieves the active disc recorder format
SetActiveDiscRecorder | Selects a new active disc recorder
ClearFormatContent | Clears the contents of an unburned image
ProgressAdvise | Registers for progress notifications
ProgressUnadvise | Cancels progress notifications
RecordDisc | Burns the staged image to the active recorder
Close | Closes the interface
IDiscRecorder Method | Description
Init | Initializes the object for an underlying device
GetRecorderGUID | Retrieves the underlying device GUID
GetRecorderType | Identifies the device as CD-R or CD-RW
GetDisplayNames | Retrieves a name suitable for GUI display
GetBasePnPID | Returns an identifier unique to the device class
GetPath | Returns an OS path to the device
GetRecorderProperties | Retrieves a pointer to the IPropertyStorage interface for the recorder
SetRecorderProperties | Sets properties for the recorder
GetRecorderState | Checks if the recorder is ready to burn
OpenExclusive | Opens a device for exclusive use
QueryMediaType | Identifies the type of media in the recorder
QueryMediaInfo | Retrieves media properties
Eject | Ejects a recorder's tray, if possible
Erase | Erases CD-RW media, if possible
Close | Closes a recorder after exclusive access
Figure 5 MSDiscMasterObj Interfaces
Figure 5 MSDiscMasterObj Interfaces

CDs come in two basic flavors: CD-Audio for audio and CD-ROM for files. The respective formats are known as Redbook and Joliet. Redbook is the original music format designed by Philips and Sony in 1980 as the "red book" standard; Joliet is a CD-ROM file format designed by Microsoft as an extension of the ISO-9660 format. Microsoft designed Joliet about the time of Windows 95 in order to extend ISO-9660 to allow long file names and deeper directory hierarchies.

The history of CDs is actually quite fascinating. CD-ROMs became successful on the coattails of the music industry. The volume of music sales drove the manufacturing cost down to a point where it became affordable to use CDs for data. The key factor was that CDs, like LPs, can be stamped. An entire disc is recorded in a single instant, which makes it cheap to produce in large quantities. The fixed cost of a stamping facility was quite high (in the millions) but justified by this new profitability. Once the facilities were already built, it cost nothing extra to stamp a few CD-ROMs now and then, so CD-ROMs piggybacked on music sales. Now, of course, anyone can buy a recorder for $100—but CD-ROMs still suffer from their audio heritage in the form of slow performance because data is stored in a single linear groove (like LPs) that winds around the disc. A different optical technology, called magneto-optical disc, uses a track/sector format to provide much faster access.

OK, so much for the history lesson. Back to reality: to find out which formats your recorder supports, you can call IDiscMaster::EnumDiscMasterFormats. This requires all the suffering of COM enumerators, so I encapsulated it in a simple method, CDiscMaster::GetSupportedFormats, that returns the formats as an array of IID, as shown here:

const MAXNFORMATS = 2;
IID fmts[MAXNFORMATS];
int nFormats = dm.GetSupportedFormats(fmts,MAXNFORMATS);

The fmts array will contain the IIDs identifying the formats, IID_IRedbookDiscMaster and IID_IJoiletDiscMaster, and you'll never have to deal with IEnumDiscMasterFormats. You can use the interface IDs to QueryInterface your IDiscMaster or call Get/SetActiveDiscMasterFormat. Practical programmers may rightly wonder why the designers of IMAPI chose such a complex API to obtain the supported formats when there are only two. A single enum code provides more than adequate bandwidth to pass this information. Only the friendly Redmondtonians know the answer. Perhaps they were expecting someone to write an audio recorder in Visual Basic®? In any case, if you use IMAPITools, you can forgo COM enumerators.

Once you've opened a session, you can query individual recorders. Once again, IMAPI uses COM to enumerate recorders and once again I've hidden the mechanics, this time using an enumerator class to make life swell:

CDiscRecorder dr;
CDiscRecorderIterator itr(dm); // dm=CDiscMaster
while (itr.Next(dr)) {
 // do something
}

Pretty easy, huh? Each time you call Next, the iterator fetches the next recorder into CDiscRecorder. CDiscRecorder encapsulates the other big IMAPI interface, IDiscRecorder, which represents a recordable CD device. IDiscRecorder provides methods to open the recorder, query its type (CD-R or CD-RW) and path name, get device-specific properties, eject the CD, and so on. Figure 4 lists the IDiscRecorder methods. CDINFO shows how to use CDiscRecorder to get all sorts of information about recorders. The code is mostly straightforward so I'll spare you the details. Download the source if you want the full scoop.

One thing I left out of CDINFO is code to actually burn data, so let me at least give you a brief overview of how you'd do it. To write to disc, you need either IJolietDiscMaster or IRedbookDiscMaster, which you get by QueryInterface-ing your IDiscMaster. Or you can use IMAPITools:

dm.SetActiveDiscRecorder(dr); // select recorder
CJolietDiscMaster jdm(dm); // get joliet interface

Now jdm holds the IJolietDiscMaster interface and you can call any of the IJolietDiscMaster methods. AddData is the one to write data; it requires a COM IStorage pointer, which is a pain for simple purposes like files but provides utmost generality since your data can come from anywhere in the known universe as long as you have an IStorage for it. Recording audio is similar, except you use IRedbookDiscMaster and AddAudioTrackBlocks to add raw audio data (44.1 KHz, 16-bit RAW, same as in WAV files). You can create multiple audio tracks with Create/CloseAudioTrack. AddData and AddAudioTrackBlocks don't actually write to disc; they write to the staging area. To really move the bits, you have to call RecordDisc, as shown here:

BOOL bSimulate=FALSE;
BOOL bEjectAfterBurn=TRUE;
dm.RecordDisc(bSimulate, bEjectAfterBurn);
dm.Close();

You can call RecordDisc with bSimulate=TRUE to simulate the burn without actually writing. Windows goes through its entire pre-flight checklist and burning conniptions, doing everything it would normally do except actually writing. This lets you test and debug your software without running through a pile of CDs or waiting for hours while the laser burns the tiny plastic pits.

Well!—That was quite a whirlwind tour of IMAPI, but it should be enough to get you started. Most of you will probably never need to call IMAPI unless you're writing a sophisticated backup program or audio recorder. For the everyday purpose of copying files, all you need is ICDBurn. But even if all you want to do is display a list of recordable drives, you'll need IMAPI. In that case I encourage you to download CDINFO and IMAPITools, which provide more details in the form of working code. And if the code doesn't work, then I must not have written it—Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


Talking To...: Robert Green Talks About Communities for Visual Basic .NET

Talking To...
Robert Green Talks About Communities for Visual Basic .NET

MSDN Magazine recently talked to Robert Green, the Community Lead Program Manager for Visual Basic, about what to expect in the next version and how the Visual Basic team hopes to work more closely with the Visual Basic community.

MSDN What projects are the Visual Basic team working on these days?

Green We are working on the next release of Visual Basic, code-named "Whidbey." We're planning on releasing a Beta in the middle of the year.

MSDN What can we expect in the Whidbey release?

Green More productivity and more power. For example, the My namespace provides access to many features of the .NET Framework with much less code. For example, My.Computer.DefaultPrinter lets you work with the default printer on your computer. You can, of course, still talk to the default printer using the appropriate .NET Framework classes, but using My offers you a shortcut. There is also My.Application, My.User, and My.Forms. We'll ship with pre-built code snippets for performing common tasks, such as reading and writing to files, reading the Registry, sending e-mail, and hundreds more. Each code snippet will show where you enter application-specific information, such as the name of the file or the Registry key. And you can easily create your own code.

Several IDE enhancements will improve your productivity. Smart tags recognize basic syntax mistakes, such as misspelling a variable name. Exception helpers provide more information on runtime errors and should make it easier to understand why your code is failing. And "Edit and Continue" is back, letting you break while your code is running, make changes, and then continue running.

The "ClickOnce" deployment technology greatly improves the deployment model for client-side applications, which can be downloaded from the Web or a network share and automatically installed with no user action required. The application (all or part of it) can be automatically updated, and updates can be rolled back.

Whidbey is not just about productivity, though. We're also doing quite a bit of work on extending the language. We're adding XML comments, operator overloading, unsigned data types, partial types, and generics.

MSDN What are you doing to entice more developers to move to Visual Basic .NET?

Green We're going to make Visual Basic .NET more attractive to them and the Whidbey release will help. But we also want to do a better job of helping developers get the most out of Visual Basic .NET, whether they are new to it or seasoned experts. We plan to have more content specific to Visual Basic on MSDN and are working on organizing the information so it is easier to find. We'll continue to have webcasts and chats. We'll have more articles and code walkthroughs that focus on the fundamentals of using Visual Basic .NET.

We want the Visual Basic Developer Center to be a great starting point for developers who want to learn Visual Basic .NET and then become more proficient. Suppose you want to learn how data binding works in a Windows Form, or how to print, or add controls to a form at run time. In the Dev Center you'll find content that is in the MSDN Library, as well as pointers to third-party Web sites, book recommendations, training classes, and so on.

MSDN What kind of community efforts are you planning?

Green First, we will continue to listen to our customers. We've made good progress on this over the past year. Everyone on the Visual Basic team spends time on some sort of community effort. Some answer questions in newsgroups while others visit with customers or participate in Web chats. We want to do more of these kinds of activities because it is the best way we can listen to our customers and understand their likes and dislikes.

We want to help those in the Visual Basic community meet and work with each other. We can do more to promote the full range of resources available to these developers. Go to http://www.asp.net and you'll see a list of community sites and recommended books. You can expect to see things like that on the Visual Basic Dev Center.

We're planning a Visual Basic Newsletter, offered both online and in e-mail, that will highlight news, resources, case studies, upcoming conferences, as well as where to find the Visual Basic team on the road so you can talk to us in person. We're going to work more closely with the MVPs, Regional Directors, user groups, third-party Web sites, and the Microsoft folks in the field and subsidiaries to have more opportunities to talk with, listen to, and help our customers.

We are working on several other ideas and will roll those out in the months ahead. We want to work and live more closely withthe Visual Basic community, both online and as often as we can in person. To learn more about what we're doing, take a look at http://msdn.microsoft.com/vbasic/community.